Turn the flag off
before the incident call starts.
Datadog Feature Flags evaluates flags through OpenFeature SDKs and records every exposure beside the errors, latency and sessions it changed, so a monitor can roll a flag back on its own. We implement it with stable targeting keys and honest monitors, and run it every month so the flag estate stays small enough to read.
Quick facts
Related
- Experiments, the flag as a randomiser with the statistics done properly
- Real User Monitoring, the sessions each variant produced
- Error Tracking, the issue that started when the flag did
The flag flip on the same timeline as the error rate it caused
The error rate climbed at 14:07. Nothing was deployed. The incident call went through the deploy history first, because the flag turned on at 14:06 lived in another tool, owned by another team. Feature Flags puts the flip on the Datadog timeline, so the call starts at the flag, not the deploy history.
A standard, not a vendor API. The SDKs implement OpenFeature, so the application code calls a standard interface and Datadog is the provider behind it. Swapping providers later is a configuration change, not a rewrite.
Every exposure, correlated. Which variant each user saw is an event beside their RUM session, their traces and their logs. Errors per variant and latency per variant are a filter, not an investigation.
Rollouts driven by monitors. A canary moves from one per cent to all while the error and latency monitors stay green, and rolls back when they do not. A circuit breaker turns a flag off when a monitor fires, before a person has been paged.
A flag estate you can read. Lifecycle automations fire on creation, enablement and archive. Stale flags are listed, and Bits AI can propose the cleanup. The estate stays the size of the product, not the size of its history.
A targeting key that is not stable, a breaker wired to a monitor that never fires, and a flag that is now just an if statement
How it bills. Feature Flags is billed on monthly flag configuration requests: each time an SDK requests the flag configuration file from Datadog. The first million a month are included, and usage above that is billed per million in volume tiers. Flag evaluations inside the application do not count. Client-side SDKs request configuration when they start, so their volume follows app opens and page loads. Server-side SDKs poll on an interval, each instance on its own, and Datadog bills server-side requests at ten times their raw count. Feature Flags does not require another Datadog product, though it uses the APM and RUM SDKs where present. Datadog publishes the rates. We publish nothing here, because the price is not the problem. The polling interval across a fleet of servers is.
The configuration traps. The targeting key is set to whatever identifier the first integration had to hand. If that is a session ID, a user gets a different variant on every visit, and every rollout percentage is a lie. A server SDK left at its default polling interval on every container in a large fleet fetches configuration far more often than any flag changes, and server-side requests bill at ten times their count. A circuit breaker is wired to a monitor on the wrong service, or to one with a threshold nobody has reviewed. It either never fires, or fires on a Monday morning for an unrelated reason. And every flag that reached one hundred per cent and stayed there is an if statement with a network dependency. None of these raises an error. They show up as a rollout that was never really gradual, a meter that follows the fleet, and a flag list longer than the feature list.
Field noteThe most common gap we see is the flag flip missing from the incident timeline. The flag tool knows, the monitoring tool does not, and the two are joined only by a message in a chat channel that was posted after the call started. The correlation is the whole reason to put flags in Datadog, and it is the part that was not wired.
Field noteThe targeting key on the server side is the account ID and on the client side it is a device ID. A user with two devices and one account is three subjects. The gradual rollout to a small share of users reached far more of them than intended, and the on-call engineer found out from the error rate.
Field noteThe pattern we keep finding is a circuit breaker that was demonstrated once and never tested again. The monitor it depends on was deleted and rebuilt in a cleanup, the breaker now points at nothing, and the next bad flag was rolled back by a human at the speed humans roll things back.
Field noteThe flag estate contains flags whose feature has shipped, flags whose feature was cancelled, and flags nobody can place. Removing one needs a code change in a service with no active owner, so none are removed, and the stale list is a report rather than a queue.
Targeting key first. Monitors before breakers. A removal date on every flag.
If flags are new, or moving from another provider, the Software Delivery Accelerator gets the provider wired, the first flags correlated and the first breaker tested in four weeks, fixed scope. If flags already run in Datadog, a Datadog HealthScan reports the request meter, the targeting keys in use and the stale flag list before we change anything. Either way, the sequence is the same.
- Agree the targeting key. One stable, pseudonymous identifier per subject, the same on the client and the server, the same one RUM uses for user identity. Written down before the first flag is created.
- Wire the OpenFeature provider. Client SDKs in the web and mobile apps, server SDKs in the services. Server SDKs fetch configuration from Datadog directly, or through Agent Remote Configuration where the Agent is already there. Server polling intervals set per environment, not left at the default.
- Correlate before the first rollout. Exposure events checked in RUM and APM for one flag on one service. Errors per variant is a working filter before anyone depends on it.
- Build the monitors the breakers will use. Error rate and latency per service the flags touch, with thresholds from real data. A breaker is only as honest as its monitor.
- Wire and test the breakers and canaries. A circuit breaker on each flag that can hurt production, tested by tripping the monitor in staging. A canary policy that advances on green and rolls back on red.
- Hand over, in writing. The targeting key scheme, the provider configuration and polling interval per platform, and the breaker and monitor pairs. A removal date on every flag, and the request meter on the day we finished.
Access is admin on the Datadog organisation plus a pull request into the application repositories for the provider, and a working session with the teams that own the flags. We do not need production shell access to implement Feature Flags, and we do not ask for it.
Every feature adds a flag. Not every feature removes one.
A flag reaches everyone and stays. A monitor is rebuilt and a breaker goes dark. A new service ships with the default polling interval. Managed Observability is the monthly service that keeps the flag estate small and the breakers wired.
Every month, we review the meter. Flag configuration requests against the client count and the polling intervals per platform. A release that changed the interval shows up here, and gets a reason or a revert.
Coverage follows the services. A new service gets the provider, the targeting key and a breaker monitor in the month it ships, so its first flag is observed from the first exposure.
The breakers are kept honest. Each month we check every breaker still points at a monitor that exists and means something, and we trip one in staging to prove the path still works.
The incident is ours to take. Under Critical Support our engineers answer the alert the flag caused, 24x7, and the flag is the first thing we turn off. Datadog has published that our work reduces mean time to resolve incidents by 40 to 60 per cent.
We operate the stack. You own the product.
An implementation partner wires the provider and hands it over. A managed partner is still there when the breaker monitor is renamed in March. We are the second kind. The flags, the targeting key scheme, the breakers and the decisions stay in your Datadog organisation and your code, documented, and you can take them in-house whenever you choose. Which features ship behind which flags is your call, and it stays yours.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Feature Flags, answered plainly
How is Datadog Feature Flags billed?
Feature Flags is billed on monthly flag configuration requests: the number of times an SDK requests the flag configuration file from Datadog. The first million requests a month are included, and usage above that is billed per million in volume tiers. Flag evaluations inside the application do not count. Client-side SDKs request configuration when they start, so their volume follows app opens and page loads. Server-side SDKs poll on an interval, each instance separately, and Datadog bills those requests at ten times their raw count. Feature Flags does not require any other Datadog product, though it uses the APM and RUM SDKs where they are present. We do not publish prices. The server polling interval and the client count are what we tune.
Is Datadog Feature Flags generally available?
Datadog lists Feature Flags with its own pricing and does not label it as Preview in the documentation we checked at the time of writing. Specific capabilities around it move faster than the core product. We confirm the status of any feature you are depending on before we scope the work. If any part of your design relies on something still in Preview, the handover says so.
We already use LaunchDarkly or Split. Does this still help?
Partly. RUM Feature Flag Tracking records which variant each session saw from another provider's flags, so you can compare errors and performance by variant without moving the flags. What you do not get from another provider is the flag change itself appearing as an event beside your APM and RUM telemetry, monitor-driven rollouts, or Datadog turning a flag off when a monitor fires. Whether that is worth a migration is a question we answer per estate, not in general.
What is a circuit breaker on a feature flag?
A rule that turns a flag off, or rolls a canary back, when a Datadog monitor fires. The flag that increased the error rate is disabled by the error rate monitor, in seconds, before anyone has joined the incident call. The same mechanism drives a canary forward in stages while the monitors stay green. It needs monitors that mean something, scoped to the service the flag touches, which is why we build those before we build the breaker.
What does Critical Cloud do with Feature Flags every month?
Managed Observability runs it every month. We review the flag configuration request meter against the client count and polling intervals. We list the flags that have sat at one state long enough to be code. We confirm the circuit breaker monitors still match the services they guard. We check new services have the provider wired and a stable targeting key, and report what changed. Under Critical Support our engineers also take the incident a flag caused, 24x7, and the flag is the first thing we turn off.
- Datadog products: every product, and how we implement and run each one
- Experiments: the same flags as randomisers, measured properly
- Workflow Automation: the automations that fire on a flag's lifecycle
- Software Delivery Accelerator: the delivery stack, flags included, in four weeks
The flip on the timeline. The rollback before the call.
Tell us which flag you would least like to see flipped at 2am. We will tell you what the telemetry would show, what would turn it off, and what it costs to get there.