The log line you wish you had written.
Added to production now, without a redeploy.
Datadog Dynamic Instrumentation adds log lines, metrics, spans and span tags to a running service from the Datadog UI or the IDE, with no code change and no restart. We put the prerequisites and the guardrails in place, decide who can place a probe where, and run it every month so the probes that were useful in March are not still running in September.
Quick facts
Related
- APM, the tracer the probes are placed through
- Log Management, where the snapshots land
- Error Tracking, Exception Replay and the variables at the moment it threw
The question answered this afternoon, not after the next release
One customer in ten thousand gets the wrong total at checkout. The code that computes it has no log line at the point that matters, because nobody expected to need one. The usual path is a pull request adding logging, a review, a release window, and a week of waiting for the case to recur. Dynamic Instrumentation puts the log line there now, with the variables, and takes it away when the answer is in.
A log line on any line of code. Placed from the Datadog UI or the IDE on a method or a line in the running service. Add a snapshot of the local variables if you want one, and a condition so it fires only for the case you are chasing.
A metric from a value in the code. A count, gauge or histogram emitted from a variable the application never exposed, which is often the quickest way to find out how often the odd case actually happens.
A span on the method nobody traced. Span probes wrap a method in a new span and span tag probes add a value to the one that is already there, so the trace gains the detail the tracer did not know to collect.
The variables at the moment it threw. Exception Replay in Error Tracking is built on the same mechanism and captures the local state when an exception is raised, so the issue comes with the evidence attached before anyone places a probe at all.
A probe left on since March, a snapshot in the wrong index, and Remote Configuration switched off by a security review
How it bills. Datadog's pricing page lists Dynamic Instrumentation as a capability of every APM tier rather than as a separately metered product, and it needs the tracer, so the host is already an APM host. What the probes produce lands with the rest of the telemetry. Log probe output and snapshots are logs: they go through Log Management ingestion and into whichever index receives them. Span probes emit spans alongside the ones the tracer already sends. Metric probes emit new metrics, under their own prefix, every time the code runs. Datadog publishes the rates. We publish nothing here, because the price is not the problem. The configuration is.
The configuration traps. The product has prerequisites that fail silently: a recent tracer, a recent Agent, and Remote Configuration enabled on the organisation and in the Agent, without which a probe never reaches the service. Probe output is logs, and logs go to whichever index matches first, so without a dedicated index for the probe source the snapshots land in the main index with its retention and its cost. A snapshot contains whatever the variables held. A metric probe is not rate limited. Logpoints expire with their debug session, but a metric or span probe runs until somebody disables it. None of this raises an alert. All of it reaches the invoice or the data protection officer.
Field noteThe most common gap we see is the probe that outlived its question. A metric probe and a span probe were placed on the request handler during an incident in March. Logpoints expire with their debug session. These do not. The incident was resolved the same evening. Both were still firing on every request in September.
Field noteNo index was created for the probe source, so the snapshots matched the catch-all index and inherited its retention. The variables in the snapshot included a customer record. The scrubbing rules that would have redacted it had not been configured because nobody had reached that page of the setup.
Field noteRemote Configuration was disabled across the organisation after a security review of what the Agent could receive. Dynamic Instrumentation depends on it. Probes were created in the UI and never reached the services, and the product was written off as not working on that stack.
Field noteWrite access was granted to every engineer in every environment, because nobody knew Datadog has a separate pre-production-only write permission. Anyone could place a probe with a snapshot on the production payment handler. Nobody had yet, which is not the same as nobody being able to.
The index and the scrubbing rules before the first probe. The permissions before the second.
We start with a Datadog HealthScan. It is read-only and fully automated, and it reports tracer and Agent versions per service, whether Remote Configuration is enabled, whether a probe index exists, and which probes are already running and since when. Dynamic Instrumentation runs through the same tracer as APM, so a service without a tracer is an APM conversation first.
- Confirm the prerequisites, service by service. Tracer version against the minimum for its language, Agent version, Remote Configuration on the organisation and in the Agent, unified service tags, and Source Code Integration so a probe knows which commit it is on.
- Create the probe index and set its retention. A dedicated Log Management index for the probe source, placed ahead of the others, with a retention chosen for debugging rather than inherited from the main index.
- Configure sensitive data scrubbing. The redaction rules for variable names and types that must never appear in a snapshot, agreed with whoever owns data protection, before anyone places a probe.
- Set the permissions. Read for everyone who debugs. Write in pre-production for the engineers who need it. Write in production for a short, named list.
- Agree the probe conventions. Every probe carries an owner and a reason, a condition where the case is rare, and an expiry. Metric probes are placed with their tags reviewed first. Span probes are indexed deliberately or not at all.
- Hand over, in writing. The prerequisites met per service, the index and the scrubbing rules, the permission lists, the conventions, and the list of probes running on the day we finished.
Access is admin on the Datadog organisation plus read access to the tracer configuration and the log index configuration. We do not need production shell access to implement Dynamic Instrumentation, and we do not ask for it. The whole point of the product is that nobody does.
A probe is easy to add. The product is only as good as the habit of taking it away.
A probe is placed during an incident and forgotten. A new service ships with an older tracer that cannot receive probes. A scrubbing rule is bypassed by a new field name. Managed Observability is the monthly service that keeps Dynamic Instrumentation useful and keeps its output off the wrong meters.
Every month, every probe is listed. Owner, reason, date placed, and what it has produced on the log, metric and span meters against last month. The ones whose question has been answered are removed. The ones nobody claims are removed faster.
Coverage follows the deployments. A service that ships with a tracer below the minimum, or on an Agent without Remote Configuration, cannot receive a probe. The monthly prerequisite check finds it before the incident that needs one.
The guardrails are kept honest. Scrubbing rules checked against the fields that appear in recent snapshots, the probe index retention confirmed, the production write list reviewed against who still needs to be on it.
The probe is part of the incident. Under Critical Support our engineers place the probe that answers the question during the incident, 24x7, and remove it in the postmortem. Datadog has published that our work reduces mean time to resolve incidents by 40 to 60 per cent.
We operate the stack. You own the product.
An implementation partner enables Remote Configuration and hands over. A managed partner is still there in September, asking who owns the probe from March. We are the second kind. The index, the scrubbing rules, the permission lists, the conventions and the decisions behind them stay in your Datadog organisation, documented, and you can take them in-house whenever you choose. The code stays yours, untouched, which is rather the point.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Dynamic Instrumentation, answered plainly
How is Datadog Dynamic Instrumentation billed?
Dynamic Instrumentation is listed as a capability of APM across its tiers rather than as a separately metered product, and it needs the tracer, so the host it runs on is already an APM host. What it produces is metered where it lands. The snapshots and log lines from log probes are logs, and go through Log Management ingestion and whichever index receives them. Span probes emit spans alongside the ones the tracer already sends. Metric probes emit new metrics, under their own prefix, every time the code runs. We do not publish prices. The meters are what we tune, and with this product the tuning is mostly about what is switched off again.
Which languages does Datadog Dynamic Instrumentation support?
Java, Python, .NET and PHP, each with a minimum tracer version, plus a minimum Agent version with Remote Configuration enabled. Three limits matter before anyone plans around it. Kotlin coroutines are not supported in Java. PHP does not support probes placed by file and line. The product does not run on Azure App Service or on serverless platforms. Unified service tagging is required and Source Code Integration is strongly recommended, because the probe is placed on a line of code and the product needs to know which commit that line belongs to.
What kinds of probe can Dynamic Instrumentation add?
Four. Log probes emit a log line, with a snapshot of local variables if asked, when a line or method is hit; these are the ones the Live Debugger in the IDE plugins places as logpoints. Metric probes emit a count, gauge or histogram from a value in the code. Span probes wrap a method in a new span. Span tag probes add a tag to the active span or the service entry span. All four are added from the Datadog UI or the IDE while the service runs, and all four can carry a condition so they fire only when the case you are chasing occurs.
What are the risks of using Dynamic Instrumentation in production?
Three, and each has a control. A probe captures what the variables hold, which can include personal data, so the sensitive data scrubbing rules are configured before the first probe is placed and the output goes to an index with a deliberate retention. A probe on a hot path fires a lot, so probes carry conditions and the log output is rate limited, while metric probes, which are not, are placed with care. A probe that is forgotten keeps running, so every probe has an expiry or an owner, and the monthly review removes the ones nobody claims. Write permission for production probes is held by a short list of people, with a wider list for pre-production.
What does Critical Cloud do with Dynamic Instrumentation every month?
Managed Observability runs it. Every month we list every active probe with its owner and its reason, remove the ones left on after the question was answered, review the volume of logs, metrics and spans that probes produced against the meters they landed on, check the scrubbing rules and the probe index retention are still what was agreed, confirm new services meet the tracer and Remote Configuration prerequisites, and report what changed. Under Critical Support our engineers also place the probe during an incident, and take it away afterwards.
- Datadog products: every product, and how we implement and run each one
- Datadog APM: the tracer every probe is delivered through
- Datadog Log Management: the index the snapshots land in, and the retention that decides what they cost
- Datadog HealthScan: read-only and fully automated, the prerequisite and running-probe check before anything changes
The answer this afternoon. The probe gone by Friday.
Tell us which bug only happens in production. We will tell you whether Dynamic Instrumentation can reach it on your stack, what has to be in place first, and how to keep the probes from becoming the next bill.