Block the deploy that would
have broken the checkout.
Datadog Continuous Testing takes the Synthetic tests that watch production and runs them against the build in the pipeline, failing the stage if they fail. We wire it into the pipeline, reach the preview environments, set the parallelism so the run meters hold, and run it every month so the gate still guards the flows that matter.
Quick facts
Related
- Synthetic Monitoring, the tests themselves, and the meters they run on
- CI Visibility, the pipeline the gate sits in
- Mobile App Testing, the same gate for the app build
The 3am alert, moved to 3pm and turned into a failed stage
The deploy passed every unit test. The checkout broke anyway, because the unit tests do not click the button and the Synthetic test that does was only watching production. Continuous Testing runs that same test against the build before it ships, so the failure is a red pipeline instead of a red status page.
One set of tests, two jobs. The browser and API tests that monitor production are selected by tag and run against the build. When a test is fixed or added, both the monitor and the gate change with it.
A gate, not a report. The pipeline stage waits for the batch and fails if any test fails. The deploy job depends on the stage. A broken checkout cannot reach production by accident, only by someone overriding the gate on purpose.
Preview environments reached. The testing tunnel lets a managed location reach an environment that exists only for the duration of the job, and a private location runs inside the network for anything behind the VPN.
After the deploy, too. The same batch runs against production once the release is live. A failure there can trigger an automatic rollback, and the results sit in the explorer next to the traces and logs from the run.
Every test on every commit, a gate everyone skips, and a tunnel that worked on the laptop
How it bills. Continuous Testing draws on the Synthetic Monitoring meters: API test runs, browser test runs and mobile app test runs, each on its own meter. A test triggered from the pipeline is a run, exactly as a scheduled one is, and so is each retry. Without the add-on, pipeline tests run one after another. Parallelisation is sold separately, per additional parallel slot, and sets how many tests a batch can run at once. Datadog publishes the rates. We publish nothing here, because the price is not the problem. The number of tests run per commit is.
The configuration traps. The first pipeline integration selects every test, on every push to every branch, and the browser run meter learns what a busy monorepo looks like. The gate is added as a job that is allowed to fail, so the pipeline goes green regardless, and the gate is decorative. The tunnel is tested from an engineer's laptop and then run from a CI runner with different egress rules, where it cannot reach anything. Parallel slots are left at the default, so a batch of forty tests runs in series and the pipeline waits half an hour for a stage that could take two minutes. None of these raises an error of its own. They show up on the invoice, in the deploy that went out anyway, and in the time the team spends waiting for the gate.
Field noteThe most common gap we see is the gate set to continue on error. It was done to unblock a release the week the tests were flaky, with a note to revert it. The note is still in the pipeline file. The gate has not stopped a deploy since.
Field noteThe tests run in the pipeline against the staging environment, which is shared, and which another pipeline is deploying to at the same moment. The gate fails for the other team's change, the failure is retried until it passes, and nobody trusts a red gate any more.
Field noteThe pattern we keep finding is tag selection that matches more than intended. A tag chosen for the gate is also on a set of tests somebody created for a load investigation, and each commit triggers a batch three times the size anyone planned.
Field noteThe post-deploy check runs against production the moment the deploy finishes, before the cache has warmed and the old pods have drained. It fails on the first request of every release, the rollback trigger fires, and the release process gains a manual step to disable it.
Select the gate tests first. Reach the environment second. Fail the stage for real.
Continuous Testing needs Synthetic tests to exist, so it follows Synthetics. The Software Delivery Accelerator wires the gate into the pipeline alongside CI Visibility in four weeks, fixed scope. If Synthetics already watches production, a Datadog HealthScan reports which tests exist, how often they run, and whether any are already triggered from CI before we change anything. Either way, the sequence is the same.
- Choose the gate tests. From the Synthetic tests that already watch production, the ones whose failure should stop a deploy. Tagged for the gate, and only for the gate. Usually fewer than the team expects.
- Decide where the build will be tested. A preview environment per pipeline run, a shared staging environment with a lock, or production after deploy. Each needs a different reach: tunnel, private location or managed location.
- Wire the trigger from a real runner. datadog-ci or the provider's integration, with the API key in the pipeline's secret store, run from the CI runner it will live on. The tunnel is proven from there, not from a laptop.
- Make the stage fail for real. No continue-on-error. The deploy job depends on the gate stage. An override path exists, requires a named approver, and is logged.
- Set parallelism and timing. Enough parallel slots that the gate takes minutes, not half an hour. A post-deploy check that waits for the rollout to settle before it runs.
- Hand over, in writing. Which tests gate which pipelines, how each environment is reached, the override process, the parallel slot count, and the CI run share at handover.
Access is admin on the Datadog organisation plus a pull request into the pipeline definitions, and the ability to run a pipeline on a branch. We do not need production shell access to implement Continuous Testing, and we do not ask for it.
A gate is only a gate while somebody checks it is closed
A flow is added to production monitoring and not to the gate. A pipeline is refactored and the gate stage loses its dependency. The preview environment changes its network. Managed Observability is the monthly service that keeps the gate guarding the flows that matter, and keeps it closed.
Every month, we review the meters. CI runs against scheduled runs on each test type, and parallel slot usage against gate duration. A gate that doubled its run count gets a reason or a smaller tag.
Coverage follows the monitors. When a new flow gets a production test, we decide whether it belongs in the gate the same month. When a flow is retired, its test leaves both.
The gate is kept honest. We check the pipeline definitions for continue-on-error and missing dependencies, and we check the override log. A gate that has not stopped anything in a quarter is either very good news or not a gate, and we find out which.
The incident is ours to take. When the post-deploy check fails against production for a real reason, Critical Support takes it from there, 24x7, with the failing test, the batch and the traces as the first evidence.
We operate the stack. You own the product.
An implementation partner adds the pipeline step and hands it over. A managed partner is still there when the step is set to continue on error in March. We are the second kind. The tests, the pipeline steps, the override process and the decisions stay in your Datadog organisation and your repositories, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Continuous Testing, answered plainly
How is Datadog Continuous Testing billed?
Continuous Testing runs draw on the same meters as Synthetic Monitoring: API test runs on one meter, browser test runs on another, and mobile app test runs on a third. A test triggered from the pipeline is a run, exactly as a scheduled one is, and so is each retry. Without the add-on, pipeline tests run one after another. Parallelisation is sold as an add-on, per additional parallel slot, which sets how many tests a pipeline batch can run at once. We do not publish prices. Which tests run per pipeline stage, and how many slots the pipeline needs, is what we tune.
How does a test failure stop the deploy?
The pipeline runs the datadog-ci command, or the provider's integration, which triggers a batch of tests and waits for the results. If a test fails, the command exits with a failure and the pipeline stage fails with it, so the deploy job that depends on that stage never runs. In a CD pipeline the same mechanism runs after the deploy, and a failure can trigger an automatic rollback. The gate is only as good as the tests selected for it, which is why selection is the first thing we agree.
Can it test a preview environment that only exists for the pipeline run?
Yes. The testing tunnel lets a Datadog managed location reach an environment that is not on the public internet. That includes one spun up for the duration of a CI job, a local server, or a staging environment behind a VPN. Alternatively a private location inside your network runs the tests directly. We pick whichever matches how your preview environments are built, and we check the tunnel works from a real pipeline run before relying on it.
Are these different tests from the ones that monitor production?
No, and that is the point. The same API and browser tests that run on a schedule against production are run on demand against the build, selected by tag. The test that would have paged you at 3am instead fails the pipeline at 3pm. Keeping one set of tests means a fix to a test fixes it in both places, and a flow that gets a new monitor gets a new gate at the same time.
What does Critical Cloud do with Continuous Testing every month?
Managed Observability runs it every month. We review CI runs against scheduled runs on each meter, and check which tests are in the gate and whether they still match the flows that matter. We confirm the tunnel or private location still reaches the preview environments after infrastructure changes, review parallel slot usage against pipeline wait time, and report what changed. When a gate stops a deploy for a real reason, that is the product working. When it stops one for a broken locator, we fix the test the same week.
- Datadog products: every product, and how we implement and run each one
- Synthetic Monitoring: the tests, the locations and the run meters
- CI Visibility: the pipeline the gate lives in, traced
- Software Delivery Accelerator: CI Visibility and Continuous Testing in four weeks
The test that would have paged you, run before the deploy instead.
Tell us which deploy you wish had been stopped. We will tell you which tests would have stopped it, how to put them in the pipeline, and what it costs to get there.