Know what the pull request
left untested, on the pull request.
Datadog Code Coverage takes the reports the pipeline already produces, merges them per commit, and puts the coverage of the changed lines on the pull request next to the test results. We implement it from the pipeline, set a gate that stops regressions without stopping work, and run it every month so the threshold keeps up with the code.
Quick facts
Related
- Test Optimization, the test runs the coverage comes from
- CI Visibility, the pipeline that uploads the report
- Static Code Analysis, the other check on the pull request
The coverage number, where the reviewer is looking
The pipeline has produced a coverage report on every build for three years. It is an artefact nobody downloads. Code Coverage reads that report, merges it with the others from the same commit, and writes the result on the pull request, where the reviewer already is.
One figure per commit. A matrix build produces a report per job. Datadog merges them, so the number on the commit is the coverage of the whole suite, not the last job to finish.
The changed lines, specifically. Patch coverage says how much of this pull request is tested. The comment lists the changed files, and inline annotations mark the lines no test reached, in the diff view.
A gate in a file. Thresholds on patch or total coverage, configured in the repository or in Datadog, evaluated on every pull request update. The rule lives next to the code it governs and changes by pull request.
No extra step where the library is in. For Ruby, JavaScript, Python and Java, the Test Optimization library uploads coverage on its own. The report exists because the tests ran, not because someone added a job.
A threshold set from a wish, a report from one job in six, and a number that went up because a file was deleted
How it bills. Code Coverage is billed per committer per month, on the maximum number of active committers in the billing period. A committer is an active Git author, identified by email address, who commits at least three times in the month. Each committer already billed for Test Optimization or Code Security brings a Code Coverage committer with it at no extra cost, so on many estates the product is already included. Datadog publishes the rates. We publish nothing here, because the price is not the problem. Whether the pipeline is uploading at all is.
The configuration traps. The gate threshold is set to the number the team would like to have, rather than the number the repository has. Every pull request fails, and the gate is removed in a week. The upload step is added to one job in the matrix, so the coverage figure is one runtime's share of the suite and the pull request comment reports a drop whenever that job is skipped. Total coverage is tracked as the success metric, and it rises when someone deletes an untested module, which is not the same as testing anything. None of these raises an error. They show up as a gate nobody respects and a trend line that measures housekeeping.
Field noteThe most common gap we see is coverage uploaded from the default branch only. The pull request has no patch figure, because the report that would give it one is only produced after merge. The review happens without the number the gate was meant to check.
Field noteThe upload step finds report files by name and extension, and a build tool upgrade renames or moves the report just enough that the search stops finding it. Nobody is watching the upload step. Coverage goes quiet for a month before anyone notices the pull request comments stopped.
Field noteThe pattern we keep finding is a second coverage product still running beside Datadog, billed per seat, because the badge in the README points at it. The two disagree by a few points on every commit, and the disagreement is discussed more than the coverage.
Field noteA monorepo uploads one report for every package, and without services or code owners defined to split them the coverage of the one package being changed is diluted by the thirty that were not. The patch figure is right. The total figure is a number about a different question.
Upload from every job. Gate on the patch. Start the threshold where the code is.
Code Coverage follows Test Optimization, and the Software Delivery Accelerator brings the pipeline, the tests and the coverage into Datadog in four weeks, fixed scope. If Test Optimization already runs, a Datadog HealthScan reports which repositories upload coverage, from which jobs, and what the gate does, before we change anything. Either way, the sequence is the same.
- Connect the source provider. The GitHub App with the permissions Datadog needs to comment and set checks, or the GitLab, Azure DevOps or Bitbucket Cloud equivalent. Scoped to the repositories in question.
- Decide the upload route per repository. Automatic from the Test Optimization library where the language allows. The datadog-ci upload step otherwise, in every job that produces a report, with the format stated explicitly.
- Check the merge. One commit, all jobs, one figure. We compare it against the number the old tool reported and explain any difference before the first comment goes on a pull request.
- Add flags for monorepos. Each package reports under its own flag, so a change to one package is judged against that package.
- Set the gate from the data. Patch coverage first, at a threshold just below where the repository is today, in a file in the repository. A date for the next increase, agreed with the team.
- Hand over, in writing. Which repositories upload, by which route, with the flags, the gate thresholds and the increase schedule. The committer count on the day we finished, with the share already covered by other products.
Access is admin on the Datadog organisation, admin on the source provider for the app installation, and a pull request into each repository's pipeline. We do not need production shell access to implement Code Coverage, and we do not ask for it.
A threshold is a promise about next quarter, not a description of today
A pipeline refactor drops the upload step. A new repository appears with no gate. The threshold that was ambitious in January is a floor by June. Managed Observability is the monthly service that keeps Code Coverage reporting and the gate moving.
Every month, we review the meter. The committer count against the org chart, and against the committers already paid for through Test Optimization or Code Security. A committer who should be included and is not gets a ticket to the account team.
Coverage follows the repositories. New repositories get an upload route and a gate in the month they appear. A repository that stopped uploading after a pipeline change is found in the review, not when the badge goes grey.
The gate is kept honest. Thresholds are raised on the agreed schedule, against the trend, and lowered only with a written reason. A gate that passed everything for a quarter is checked for an upload that quietly stopped.
The gap gets a recommendation, then an owner. Each month we report the least-covered files that change most. Writing the tests is engineering work on your code, scoped through Catalyst when it is more than an afternoon.
We operate the stack. You own the product.
An implementation partner adds the upload step and hands it over. A managed partner is still there when the step is dropped in a pipeline refactor in March. We are the second kind. The upload steps, the gate files, the thresholds and the decisions stay in your Datadog organisation and your repositories, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Code Coverage, answered plainly
How is Datadog Code Coverage billed?
Code Coverage is billed per committer per month, on the maximum number of active committers in the billing period. A committer is an active Git author, identified by email address, who commits at least three times in the month. If you already pay for Test Optimization or Code Security committers, each of those brings a Code Coverage committer with it, so for many estates the product is already included. We do not publish prices. Which repositories upload coverage, and who counts as a committer, is what we tune.
Which coverage formats and languages does it accept?
LCOV, Go coverprofile, Cobertura XML, JaCoCo XML, Clover XML, OpenCover XML and SimpleCov JSON, uploaded from the pipeline with the datadog-ci tool or its GitHub Action. For Ruby, JavaScript, Python and Java, the Test Optimization library can upload coverage automatically with no separate step. Reports for the same commit are merged by Datadog, so a matrix build that produces six partial reports gets one coverage figure.
What is the difference between total and patch coverage?
Total coverage is the share of the whole codebase that tests exercise. Patch coverage is the share of the lines changed in this pull request that tests exercise. Total coverage moves slowly and is mostly history. Patch coverage is about the change in front of you, and it is the number a reviewer can act on. We gate on patch coverage first, and we treat total coverage as a trend to report rather than a bar to clear.
Will a coverage gate block every pull request?
Only if the threshold is set without looking at the repository first. Gates are configured in the Datadog interface or in a file in the repository, on total or patch coverage, and they block a pull request that falls below the threshold. We set the first threshold just below where the repository already is, so the gate stops regressions without stopping work, and we raise it on a schedule the team agrees to. A gate that blocks every pull request gets switched off within a week, and then it is not a gate.
What does Critical Cloud do with Code Coverage every month?
Managed Observability runs it every month. We review the committer count against the org chart and the committers already paid for through Test Optimization or Code Security. We check that every repository with a gate is still uploading reports after pipeline changes. We review the gate thresholds against the trend, add new repositories, and report what changed. Raising coverage is engineering work on your tests, and when the gap is large we scope it through Catalyst.
- Datadog products: every product, and how we implement and run each one
- Test Optimization: the tests, the committer meter and the automatic upload
- Code Security: the other committer meter that includes coverage
- Software Delivery Accelerator: pipeline, tests and coverage in Datadog in four weeks
Coverage on the pull request. A gate the team keeps.
Tell us which repositories matter and what the old coverage tool costs. We will tell you whether the committers are already paid for, what the gate should start at, and what it takes to get there.