Stop re-running the build
to make the flaky test pass.
Datadog Test Optimization traces every test run, finds the ones that fail for no reason, quarantines them without hiding them, and skips the tests a change cannot have touched. We instrument the suites, scope it to the committer meter, and run it every month so the quarantine list is a queue rather than a graveyard.
Quick facts
Related
- CI Visibility, the pipeline around the tests
- Code Coverage, what the tests actually exercise
- Continuous Testing, Synthetic tests as a pipeline gate
The test that fails on Tuesdays, named, with its bill attached
Everyone knows the suite is flaky. Nobody knows which tests, because the failure is different each time and the re-run passes. Test Optimization traces every test on every run, so the test that fails one time in twelve has a name, a failure rate, and a figure for the pipeline hours it has cost.
Every test, traced. Name, duration, result and the commit it ran on, for every run. An integration test that calls an instrumented service shows the distributed trace underneath, so the slow test opens the slow query.
Flaky tests found per configuration. A test that fails only on one OS or one runtime version is tracked for that combination. The list shows failure rate and pipeline time wasted, so the worst offender is first, not loudest.
Quarantine without hiding. A flaky test can keep running while its failures stop breaking the build, and it is moved to fixed only when it has passed consistently. The build is green again and the test is still visible.
Only the tests the change can affect. Test Impact Analysis uses coverage and the git diff to skip tests that cannot be affected by the commit. New tests are retried early to catch flakiness before it reaches the default branch.
A quarantine nobody empties, retries that hide the bug, and a span pool spent on a matrix
How it bills. Test Optimization is billed per committer per month, on the same committer definition as CI Visibility. A committer is an active git author, identified by email, who makes at least three commits in the month. Each committer brings one million test spans a month, pooled across all Test Optimization committers, where a span is one test run, and spans beyond the pool are sold separately. Test spans are kept for three months, against 15 months for pipeline spans, and that retention cannot be changed. Committers can be split between Test Optimization and CI Visibility. Datadog publishes the rates. We publish nothing here, because the price is not the problem. How many times each test runs is.
The configuration traps. A test matrix across four runtimes and three operating systems runs every test twelve times per commit, and every one is a span. Automatic retries are switched on with a generous count and the flaky test now passes on its fourth attempt, every time, invisibly, while the race condition it was reporting ships. Quarantine is used as a fix rather than a queue. Test Impact Analysis is enabled on a suite with no coverage data, so it skips nothing, and the team concludes it does not work. None of these raises an error. They show up as a span pool spent by the middle of the month and a green build that means less than it did.
Field noteThe most common gap we see is the quarantine list growing and never shrinking. Each test was quarantined for a good reason on a busy day. By the time anyone looks, the suite is green, the quarantine holds a sizeable share of it, and the tests that would have caught the last regression are in there.
Field noteAutomatic retries are set at the organisation level to make the build reliable. They do. The test that fails because of a real timing bug in the product now passes on retry, the bug reaches production, and the test's own history is the only record that it ever complained.
Field noteThe pattern we keep finding is JUnit XML upload where native instrumentation was available. Results and durations arrive, so the dashboards look right. Impact analysis, per-test retries and quarantine do nothing, because the library that does them is not in the build.
Field noteThe fix for a flaky test is made on a branch, and the branch is renamed during review, or deleted before Datadog has seen the merge. Datadog can no longer tell that the fix landed, so the test sits in fix in progress and the board says somebody is on it. The team meets it again on the day it blocks a release, which is the day it has least patience for it.
Native libraries first. Retries with a limit. Quarantine with an owner and a date.
If test visibility is not on yet, the Software Delivery Accelerator instruments the suites, switches on flaky detection and reports the tests costing the most pipeline time in four weeks, fixed scope. If it is already on, a Datadog HealthScan reports the committer count, the span pool and the quarantine list before we change anything. Either way, the sequence is the same.
- Inventory the suites. Which test services exist, which language each runs, whether native instrumentation is available, and how many times each test runs per commit across the matrix.
- Instrument natively wherever possible. The Datadog test library in each build, with git metadata on every run. JUnit XML only where no library exists, and recorded as a known gap.
- Set retries and early flake detection with limits. A small retry count, scoped to the suites that need it, with the retried failures kept visible. New tests get early flake detection so flakiness is caught before it merges.
- Switch on Test Impact Analysis where it pays. Per test service, after coverage data exists, on suites where most tests are unrelated to most changes. We measure the skipped share for a fortnight and report it.
- Write the quarantine policy. Who may quarantine, for how long, and who owns the fix. A policy that quarantines automatically by failure rate, paired with a monitor on the size of the list.
- Hand over, in writing. Which suites are instrumented and how, the retry and impact analysis settings per suite, and the quarantine policy. The committer and span counts, and the flaky tests with the most pipeline time, each with an owner.
Access is admin on the Datadog organisation plus a pull request into each repository's build configuration, and a short session with each team that owns a suite. We do not need production shell access to implement Test Optimization, and we do not ask for it.
A quarantine is a queue. Somebody has to work it.
New suites appear without the library. A repository is restructured and impact analysis loses its map. The quarantine fills. Managed Observability is the monthly service that keeps Test Optimization honest about what the green build means.
Every month, we review the meters. The committer count against the org chart, the test span pool against the suites and the matrix, and the share of runs that were retries. Anything that moved gets a reason or a fix.
Coverage follows the repositories. New suites get the native library in the month they appear. A suite that moved from native to XML upload in a build refactor is caught in the review, not in the next release.
The quarantine is kept honest. Each month the list is reviewed against the policy. Tests past their date get an owner and a ticket. Tests that have passed consistently are moved to fixed and returned to the build.
The flaky test gets a recommendation, then an owner. We report the tests costing the most pipeline time with a likely cause from the traces. Fixing them is engineering work on your code, scoped through Catalyst when it is more than an afternoon.
We operate the stack. You own the product.
An implementation partner adds the library and hands it over. A managed partner is still there when the quarantine list is a tenth of the suite in March. We are the second kind. The instrumentation, the retry settings, the quarantine policy and the decisions stay in your Datadog organisation and your repositories, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Test Optimization, answered plainly
How is Datadog Test Optimization billed?
Test Optimization is billed per committer per month, using the same definition as CI Visibility: an active git author, identified by email, who makes at least three commits in the month. Each committer brings one million test spans a month, pooled across all Test Optimization committers, where a span is one test run, and spans beyond the pool are sold separately. Test spans are kept for three months, against 15 months for pipeline spans. A team can have some committers on Test Optimization and others on CI Visibility. We do not publish prices. Which test services are instrumented, and how many times each test runs, is what we tune.
Which languages and frameworks does it support?
Native instrumentation for .NET, Java and other JVM languages, JavaScript, Python, Ruby, Swift and Go, with Bazel rules for Go, Java and Python. Anything else can upload JUnit XML, which gives test results and durations but not the per-test retries, quarantine and impact analysis that need the native library. We check which route each suite takes before scoping, because the features worth having are on the native side.
What does Test Impact Analysis do?
It skips the tests a change cannot have affected. Using code coverage data from earlier runs and the git diff of the current one, it works out which tests touch the changed files and runs only those. It is opt-in, per test service, and it needs coverage data to exist first. On a suite where most tests are unrelated to most changes, the saving in pipeline time is large. On a suite of integration tests that touch everything, it is small, and we say so.
What happens to a test once it is marked flaky?
It appears in Flaky Tests Management with its failure rate and the pipeline time it has cost. From there it can be quarantined, so it keeps running but its failures no longer fail the build, or disabled, so it is skipped. Policies can do this automatically by failure rate or branch. A quarantined test that passes consistently is moved to fixed. A disabled test is never moved automatically, because a skipped test produces no evidence. Quarantine is a holding pen, and the fix still has an owner.
What does Critical Cloud do with Test Optimization every month?
Managed Observability runs it every month. We review the committer count and the test span pool against the suites instrumented. We report the flaky tests costing the most pipeline time, with an owner for each. We check the quarantine list for tests that have been there too long. We confirm impact analysis is still skipping what it should after a repository restructure, and report what changed. Fixing the tests is engineering work on your code, scoped through Catalyst when it is more than an afternoon.
- Datadog products: every product, and how we implement and run each one
- CI Visibility: the pipeline around the tests, on the same committer definition
- Code Coverage: the coverage data impact analysis depends on
- Software Delivery Accelerator: suites instrumented and flaky tests named in four weeks
A green build that means something. A quarantine that empties.
Tell us which suite the team re-runs most. We will tell you what Test Optimization would show, what it would skip, and what it costs to get there.