The time goes somewhere inside the service.
Find the line.
Datadog Continuous Profiler shows where CPU, memory and wall time go inside a running service, by method, class and line, in production, all the time. We choose the services, enable it through the tracer, keep the profiled host count to the list you agreed, and run it every month so the profiles are read rather than merely collected.
Quick facts
Related
- APM, the span that points at the profile
- Error Tracking, the exception the profile sits behind
- Dynamic Instrumentation, the log line added once the profile names the method
The trace says the span was slow. The profile says which line.
The checkout span takes four hundred milliseconds and makes no external calls. APM has done its job: the time is inside this service. The next question is where inside, and the usual answer is a week of guessing, a local reproduction that does not reproduce, and a fix that moves the problem. The profiler answers it from production, with the method name.
The method, not the service. CPU, memory and IO broken down by method, class and line, as a flame graph, a call graph or a thread timeline, from the running process rather than a reproduction.
The leak with a name. Allocation and heap profiles show which code path is holding the memory that grows every day until the restart, which is usually a cache nobody bounded or a list nobody cleared.
Linked to the trace, both ways. With APM and the profiler on the same process, a slow span opens the profile of that moment, and a hot method shows which endpoints are paying for it.
Always on, so the evidence exists. The profiler runs continuously with low overhead by design, which means the profile from the incident at 3am is already there in the morning, rather than needing to be captured during the next one.
A flag in the base image, a replica count that breaks the allotment, and a flame graph that expired on day nine
How it bills. The profiler has its own meter. Profiled hosts are counted hourly and billed on the high-water mark of the lower 99 per cent of usage, the same method as APM hosts. Each profiled host carries an allotment of profiled containers, pooled across all profiled hosts, and only containers running the profiler count. Containers above the pooled allotment are billed per container. On Kubernetes the host is the node. The profiler is bundled in the APM Enterprise tier and is available on its own. Datadog publishes the rates. We publish nothing here, because the price is not the problem. The configuration is.
The configuration traps. On most runtimes the profiler is switched on by an environment variable in the tracer configuration. Go starts it in code. Tracer configuration tends to live in a base image or a shared Helm value. Set there, it profiles every service on every host, and every host becomes a profiled host. The container allotment is per host and pooled, so a service deployed as many small replicas uses it up faster than the same service deployed as a few large ones. Flame graphs are kept for a limited window, so an investigation that starts late finds the evidence gone unless somebody exported it. None of this raises an alert. All of it reaches the invoice or empties the investigation.
Field noteThe most common gap we see is the base image. The profiling flag was set in the shared Docker image when the first service needed it. Every service built from that image is now profiled, including the cron runners and the admin tools nobody has opened a profile for. The profiled host count follows the image, not the need.
Field noteThe pattern we keep finding is a service deployed as many small replicas across a handful of nodes. The pooled container allotment for those nodes runs out long before the replica count does, and every replica past it is billed as an extra container. The same service as fewer, larger replicas sits inside the allotment.
Field noteThe memory investigation was scheduled for the sprint after the incident. The flame graphs from the incident had expired by the time it started. The profiles that remained showed the service after the restart, which is the one state in which it had no problem.
Field noteThe profiler has been on for a year and nobody has opened it. The slow span is still investigated with log statements, because the link from the span to the profile is one tab over and nobody showed the team it was there.
Pick the services. Enable per service, never per image. Make the profile the first place anyone looks.
We start with a Datadog HealthScan. It is read-only and fully automated, and it reports which hosts and containers are already profiled, which services have a tracer the profiler can be enabled from, and which profile types are on. The profiler runs from the same tracer as APM, so a service without a tracer is an APM conversation first.
- Inventory the services with a tracer. Language, tracer version, deployment shape, and whether the service has had a latency or memory incident in the last year. The ones that have are the shortlist.
- Choose what to profile and why. Each service on the list has a question attached: which method is slow, what is holding the memory, why does CPU climb through the day. A service with no question is not profiled yet.
- Enable per service, in the deployment configuration. The profiling flag set on the service's own manifest or task definition, never in the base image or the shared values, with the default profile types first.
- Check the deployment shape against the allotment. Replica counts and node spread compared with the pooled container allotment, and the choice between more small replicas and fewer large ones made with the meter in view.
- Verify the links and set up the export habit. From a slow span to the profile, from a hot method to its endpoints, confirmed on the first service. A notebook per service for the profiles that must outlive the retention window.
- Hand over, in writing. Which services are profiled, with which profile types and which question, which were excluded and why, and what the profiled host and container counts were on the day we finished.
Access is admin on the Datadog organisation plus read access to the tracer configuration and the deployment repositories. Configuration changes go through your pipeline, not ours. We do not need production shell access to implement the profiler, and we do not ask for it.
A profiler nobody reads is a meter with a flame graph attached
A new service ships from the base image with the flag inherited. A replica count is doubled for a launch and the container allotment is exceeded. A profile that explained last month's incident expires before the postmortem. Managed Observability is the monthly service that keeps the profiler scoped to the services that need it and makes sure someone reads what it collects.
Every month, we review the meters. Profiled hosts and profiled containers against the agreed service list, overage against the pooled allotment. Anything that moved gets a reason or a fix.
Every month, we read the profiles. The top CPU and memory consumers per profiled service, whether they changed since last month, and what shipped in between. The list goes to the service owner with the method names on it.
Evidence is kept before it expires. The flame graphs and timelines from any incident are exported to a notebook within the retention window, so the postmortem has the profile and not a description of it.
The incident is ours to take. Under Critical Support our engineers answer the page the profile explains, 24x7, and fix the cause rather than report it. Datadog has published that our work reduces mean time to resolve incidents by 40 to 60 per cent.
We operate the stack. You own the product.
An implementation partner sets the flag and hands over. A managed partner is still there when the replica count doubles in March and the overage line appears. We are the second kind. The tracer configuration, the service list, the notebooks and the decisions behind them stay in your repositories and your Datadog organisation, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Continuous Profiler, answered plainly
How is Datadog Continuous Profiler billed?
Continuous Profiler has its own meter, separate from APM. Profiled hosts are counted every hour and billed on the high-water mark of the lower 99 per cent of usage, the same method as APM hosts. Each profiled host carries an allotment of profiled containers, pooled across all profiled hosts, and only containers actually running the profiler count against it. Containers above the pooled allotment are billed per container. On Kubernetes the host is the node, not the pod. The profiler is bundled in the APM Enterprise tier and is also available on its own. We do not publish prices. The meters are what we tune, and the first decision is which services are profiled at all.
Which languages does Datadog Continuous Profiler support?
Go, Java, Node.js, PHP, Python, Ruby and .NET. GraalVM native images, and C, C++ and Rust through the native profiler, are in Preview. Each is enabled through its own tracer or profiling library. CPU is common to all. Wall time, allocation, heap, lock and exception profiles depend on the language, and Go and the native profiler have no wall time profile. The implementation lists which types are on for which service, because each one is a decision about overhead and about what question the profile is meant to answer.
Does the Continuous Profiler slow down production?
It is designed to run in production all the time, using the runtime's own low-overhead profiling mechanisms such as JDK Flight Recorder for Java, and Datadog's guidance is that the impact on host CPU and memory is minimal. The exceptions are specific optional profile types on specific runtimes, which is why we enable the defaults first, measure on the service itself, and add the optional types one at a time where the question needs them. We do not switch everything on in the base image.
What is the difference between Continuous Profiler and APM?
APM follows the request between services and tells you which service and which span took the time. The profiler looks inside one service and tells you which method, which class and which line of code the CPU, memory or wall time went to while that span was running. They run from the same tracer and link to each other. A slow span opens the profile of the process at that moment, and a hot method links back to the endpoints that call it. One without the other leaves you with either a suspect and no evidence or evidence and no suspect.
What does Critical Cloud do with Continuous Profiler every month?
Managed Observability runs it. Every month we review the profiled host and container counts against the list of services that were meant to be profiled. We read the profiles and record the top CPU and memory consumers per service, and whether they changed. We export the flame graphs that matter to a notebook before the retention window closes. We bring newly shipped services into scope or deliberately leave them out, and we report what changed. Under Critical Support our engineers also take the page the profile explains, 24x7.
- Datadog products: every product, and how we implement and run each one
- Datadog APM: the tracer the profiler runs from, and the span that points at the profile
- Datadog Error Tracking: the exception, grouped, with the profile behind it
- Datadog HealthScan: read-only and fully automated, the profiled host and tracer check before anything changes
The line, not the service. The evidence, not the guess.
Tell us which service needs more memory every quarter. We will tell you what the profiler would show, what it costs to profile that service and only that service, and how to make the answer stick.