Ship the variant that won,
and know it did not break anything.
Datadog Experiments randomises users with Feature Flags and measures the result on the RUM sessions and warehouse data you already have, with latency, errors and crashes per variant beside the conversion number. We implement it so the statistics are honest, and run it every month so the flags do not outlive the experiments.
Quick facts
Related
- Feature Flags, the randomisation and the exposure record
- Product Analytics, the funnels the experiment metrics come from
- Real User Monitoring, the sessions and the guardrail signals
The winning variant, and the error rate it came with
The new checkout converted better and the experiment was called a win. The crash rate on Android went up the same week, in a different tool, and was called an incident. They were the same change. Experiments puts the conversion metric and the crash rate on the same screen, per variant, so a win that costs an incident is not called a win.
Randomisation you already have. A Datadog Feature Flag assigns the variant and records who saw what. If you randomise elsewhere, the exposure data comes in from the warehouse. Either way the assignment is the record, not a guess reconstructed later.
Metrics from the sessions or the warehouse. Conversion from Product Analytics and RUM events, without a query. Revenue from Snowflake, BigQuery, Redshift or Databricks, computed where finance already trusts it. The experiment reads both.
Guardrails from the same data. Latency, errors and crashes per variant, with checks for sample ratio mismatch and traffic imbalance. A variant that broke the instrumentation is flagged before anyone reads its result.
Statistics that survive peeking. Sequential methods let the team look as the data arrives. Variance reduction gets to an answer with fewer users. The method is chosen per experiment, and the result says which one was used.
A subject key that changes, a metric nobody defined, and a flag that stayed on after the answer came in
How it bills. Experiments has its own meter: it bills per launched experiment, per month. An experiment becomes billable once it has launched and reached at least 500 subjects and three days of data, and one that never reaches that threshold is never billed. Each experiment purchased includes one million monthly flag configuration requests, the unit Feature Flags bills on. Metrics built from client-side events need RUM or Product Analytics, which bill on their own meters. Metrics computed in the warehouse run on the warehouse, at the warehouse's cost. Online experiments on Agent Observability data are in Preview. Datadog publishes the rates for each underlying product. We publish nothing here, because the price is not the problem. The experiment design is.
The configuration traps. The flag needs a stable subject identifier so a user sees the same variant on every visit. A session ID or an anonymous cookie is not stable, and a user who sees both variants is in both arms of the result. The conversion metric needs a definition that both product and engineering agree on before the experiment starts, not after one side has seen the number. The statistical method is a choice, and changing it after the data arrives is a way to get the answer somebody wanted. And when the experiment concludes, the flag is still there, with its targeting rules, in every client that evaluates it. None of these raise an error. They produce a result that is confidently wrong, and a flag estate that grows by one every fortnight.
Field noteThe most common gap we see is the subject key set to whatever was convenient in the SDK call. Logged-out users get a cookie ID, logged-in users get an account ID, and a user who signs in half way through is two subjects who each saw one variant. Every diagnostic passes, because each subject was assigned cleanly, and the effect is diluted across both arms.
Field noteThe experiment is declared a win on the conversion metric. The guardrail metrics were never configured because the first experiment was in a hurry. The variant that won added a third-party script, and the page weight that went with it is discovered in the Core Web Vitals review a month later.
Field noteThe pattern we keep finding is a flag estate that is a graveyard of experiments. Each flag was created for a test, the test concluded, and the flag stayed at its final split because removing it needed a code change and the result was already shipped. The flag list never came down, and every client downloads the whole graveyard each time it fetches configuration.
Field noteThe revenue metric lives in the warehouse and the conversion metric lives in RUM, and the two disagree about what a day is. One runs on UTC and the other on the warehouse's local timezone. The experiment that looks like a win in one window is flat in the other, and the argument is about the calendar.
Subject key first. Metric definitions second. Guardrails before the first user is assigned.
Experiments depends on RUM, Product Analytics and Feature Flags, so it is implemented with them. If none of them are on, LaunchPad runs the rollout as a managed project, and the Digital Experience Accelerator gets the RUM foundation in place in four weeks. If RUM already runs, a Datadog HealthScan reports session volumes, user identification and action naming before we change anything.
- Agree the subject key. One stable, pseudonymous identifier per user, set in the flag evaluation context and in RUM user identity. It survives sign-in, so one person is one subject.
- Write the metric definitions down. The primary metric, where it comes from, and what counts. Secondary metrics. Guardrails on latency, errors and crashes. Signed off by product and engineering before the experiment starts.
- Connect the data sources. Product Analytics events for client-side metrics. The warehouse connection and an exposure SQL model where the truth lives in Snowflake or BigQuery. We check both agree on what a day and a user are.
- Create the flag and link the experiment. Traffic split and targeting set on the flag, the experiment linked to it, the exposure events checked in RUM before any traffic is assigned.
- Choose the statistical method and the stopping rule. Sequential if the team will look as it runs, fixed-sample if the size is agreed. A guardrail breach stops the experiment. We write the rule down before the data arrives.
- Hand over, in writing. The subject key scheme, the metric definitions, the data sources and the method per experiment. The flag resolution process, and the state of the flag estate on the day we finished.
Access is admin on the Datadog organisation, a pull request into the frontend repositories, read access to the warehouse for the metric queries, and time with the product owner and whoever answers for data protection. We do not need production shell access to implement Experiments, and we do not ask for it.
Every experiment leaves a flag behind. Somebody has to pick them up.
An experiment concludes and the flag stays. A metric definition drifts when the frontend renames an action. A new experiment reuses a subject key that was never stable. Managed Observability is the monthly service that keeps the experiment programme honest and the flag estate small.
Every month, we review the meters. Launched experiments, flag configuration requests, Product Analytics sessions and warehouse query volume, against the experiments that are actually live. A concluded experiment still generating flag requests gets a resolution ticket.
Coverage follows the roadmap. New experiments get their metric definitions and guardrails reviewed before launch. New flows get action names so they can be experimented on later.
The metrics are kept honest. After frontend releases we check the actions each live experiment depends on still fire. A metric that went to zero on release day is a naming break, and we fix it before the result is read.
The variant that broke production is ours to take. When a guardrail trips and the error rate is real, Critical Support takes the incident, 24x7, with the flag as the first thing to turn off.
We operate the stack. You own the product.
An implementation partner sets up the first experiment and hands it over. A managed partner is still there when the fourteenth experiment's flag is still live in March. We are the second kind. The subject key scheme, the metric definitions, the flags and the decisions stay in your Datadog organisation and your code, documented, and you can take them in-house whenever you choose. We do not decide what to test. That is the product team's job, and it stays theirs.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Experiments, answered plainly
How is Datadog Experiments billed?
Experiments bills per launched experiment, per month. An experiment becomes billable once it has launched and reached at least 500 subjects and three days of data, and one that never reaches that threshold is never billed. Each experiment purchased includes one million monthly flag configuration requests, the unit Feature Flags bills on. Metrics built from client-side events need RUM or Product Analytics, which bill on their own meters. Metrics computed in your data warehouse run on your warehouse, at your warehouse cost. Online experiments on Agent Observability data are in Preview. We do not publish prices, and we confirm the current meters on your contract before scoping. The number of live experiments, the flag request volume and the session sample rate are what we tune.
Do we need Datadog Feature Flags to run an experiment?
No, but it is the simplest route. With Datadog Feature Flags, the flag assigns the variant, records the exposure, and the experiment reads both. If you already randomise with another system, you send the exposure data from your warehouse using an exposure SQL model, and Datadog analyses it the same way. The thing that cannot be skipped is a stable subject identifier, so the same user sees the same variant every time.
What stops an experiment from shipping a slower product?
Guardrail metrics. Alongside the conversion metric the experiment is testing, Datadog shows latency, errors and crashes per variant from the same RUM sessions, and checks for sample ratio mismatch and traffic imbalance. A variant that converts better and crashes more is visible as both. We set the guardrails before the experiment starts, and we treat a guardrail breach as a stop, not a footnote.
Which statistical method should we use?
Datadog offers sequential and fixed-sample frequentist methods, a hybrid, and a Bayesian option. Sequential methods let you look at the results as they arrive without inflating the false positive rate, which is what most teams want when they are going to look anyway. Fixed-sample is right when the sample size is agreed up front and nobody will peek. We pick one per experiment, write down why, and do not change it after the data starts arriving.
What does Critical Cloud do with Experiments every month?
Managed Observability runs it every month. We review which experiments are live and which have concluded. We check that concluded experiments have had their flags resolved and removed. We review the flag request and session meters the experiments draw on, confirm the metric definitions still match the frontend after releases, and report what changed. When a variant breaks production, Critical Support takes the incident, 24x7.
- Datadog products: every product, and how we implement and run each one
- Feature Flags: the flag that does the randomising, and the meter it runs on
- Product Analytics: the funnels the experiment metrics are built from
- Digital Experience Accelerator: the RUM foundation in four weeks
A result you can defend. A flag estate you can read.
Tell us what the next experiment is testing. We will tell you whether the data can answer it today, what it would take to make the answer honest, and what it costs to get there.