for

Welcome to Test Clutch for the curl project

Regression test data and analysis (pre-alpha release)

Recent Reports

This report shows all the test runs performed recently curl's master branch. More recent runs are on the left side of the page, while older ones are on the right (you may need to scroll to see them all). Successful test runs are highlighted in green and show the number of passed tests, while unsuccessful ones are red and show the number of failed tests. Test runs that were aborted are coloured yellow and show the number of passed tests. Clicking on the number will take you to the test log that will show you details of the run. Different shades of colour are used for runs older than a few days. Obsolete test run configurations that are no longer being used have their names with a grey background. Test configurations that have experienced flaky tests in the time frame shown say flaky while those that are currently always failing say permafail. Hovering over any coloured square will show you more details about that test or run.

Counts of recent test failures

This report shows a list of tests that have recently failed. It is ordered with the tests experiencing the most failures at the top and those with the fewest failures at the bottom. The kind of failure is shown in the Result column. FAIL is a normal failure, FAILIGNORE is a failure that was suppressed (usually for tests that are known not to work in a configuration), and UNKNOWN is for tests whose result could not be determined, such as when a test run aborted or timed out before its status could be determined.

Test run statistics

This report shows many statistics about recent test runs.

Test run metadata summary

This report shows all the metadata fields set in recent tests. Not all fields are set for every test run; some are only available in certain configurations. Clicking on the field name will show you all the values set in the tests, except for certain fields where that would show too many values that are not very interesting (such as the time each test started). More information is available about the meaning of some metadata fields.

What is Test Clutch?

Test Clutch is a test run aggregation and analysis system. It continually collects the results of test runs from Continuous Integration (CI) systems that run regression tests on newly-committed source code. Those test results are analyzed and displayed so developers can extract useful information from them that isn't easily obtained by looking at test runs one at a time.

It is designed especially for projects with many tests running on (potentially) many different CI systems. The curl project is a good example. It has two independent test suites totalling thousands of test cases that are run on dozens of different build configurations. Every single code change can cause over 100,000 individual tests to be run on over 100 different servers. With that many tests being run, even a tiny false positive rate makes it hard to tell if a failed test is a genuine problem or just a flaky test. This site is a Test Clutch installation tailored for curl.

Test Clutch is intended to make testing code just a little bit easier. See the README file for a sense of where we're going with this. It's early days, so come back often to see how things are progressing! It's all Open Source, so feel free to jump in and help out if you share the vision of improving presentation of test results.

cURL home page

Test Clutch development page