Every step the agent took,
and what each one cost.
Datadog Agent Observability, formerly LLM Observability, traces each LLM call and each agent step as a span: the prompt, the tool call, the retrieval, the answer. Every span carries tokens, latency and cost, and evaluators score the answer for quality, safety and leaked data. We instrument your AI application, choose the evaluations that matter for it, and run it every month so the cost per conversation and the rate of bad answers are numbers rather than feelings.
Quick facts
Related
- AI Guard, the control on prompt injection and agent poisoning
- GPU Monitoring, the hardware the models run on
- Datadog for AI, the group page for observing your AI
The conversation, replayed step by step, with a price on each line
A customer says the assistant gave them the wrong refund policy. The logs show a request and a response. What happened in between, the retrieval that pulled last year's policy, the tool call that timed out, the retry that doubled the tokens, is invisible. Agent Observability records the whole chain as a trace, so the wrong answer has a cause, and the cause has a cost.
A trace per request, a span per step. One LLM inference, a fixed workflow or a dynamic agent loop: each choice the agent made is a span with its input, its output, its latency and its errors. The flame graph of an agent run reads like the one for a slow checkout.
Tokens and cost, attributed. Token usage and model cost on every span, rolled up by application, feature, model and user through the operational dashboards. The one-line model bill becomes a breakdown somebody can act on.
Evaluations that score the answer. Managed Evaluators for quality, safety and privacy, custom LLM-as-a-judge evaluators written for your use case, external frameworks such as DeepEval and NeMo, and end-user feedback. A bad answer is a metric, with a threshold and a monitor.
Patterns and Insights for what you did not look for. Patterns clusters production traffic by topic so the conversations you never anticipated show up as a group. Insights analyses traces for recurring cost and reliability problems and recommends a fix. Both are Datadog's, and Patterns is in Preview.
Every span ingested, no evaluators configured, and a prompt that carries the customer's address into storage
How it bills. Agent Observability is metered on LLM spans, the calls to a model, and not on the tool, retrieval or workflow spans around them. Datadog's pricing list bills a first block of LLM spans per month with a default retention and further blocks per volume, with longer retention tiers available, and states a monthly free allotment. One agent workflow produces several LLM spans, so an agent that loops produces a bill that loops. Custom LLM-as-a-judge evaluators run through an LLM provider account you connect, so their model calls are charged by that provider. We publish no rates. The levers are which applications are instrumented, which spans are sampled and which evaluators run on which fraction of traffic.
The configuration traps. The SDK is enabled on every environment including the load-test one, and the load test is the biggest customer. Evaluators are left at the default, so the bad answer is in the trace and nobody is scoring it. Sensitive Data Scanner is not applied to prompts, so the customer's address is now in a span, retained for the whole retention period. The agent loop has no step limit and a single conversation produces hundreds of spans. Prompt Management is not used, so the prompt that changed on Tuesday is not the prompt the trace shows, and nobody can reproduce the regression. Every one of these is a configuration left at the trial setting, on a product where the trial setting is "record everything".
Field noteThe most common gap we see is tracing without evaluation. The traces are complete, the cost is attributed, and the question the product owner actually asks, "how often is it wrong?", has no number because no evaluator was ever configured for it.
Field noteThe pattern we keep finding is the agent loop with no budget. A planning step that calls the model again on every retry produces a trace with dozens of LLM spans for one user question. The span bill is explained by one conversation that should have stopped after three steps.
Field notePrompts and completions are stored verbatim because that is what makes the trace useful, and the redaction that would make it safe was assumed to be somewhere else. It was not. The first data-protection review finds it.
Field noteCorrelation with APM is skipped because the AI team and the platform team instrumented separately. The agent trace ends at the tool call, the service trace starts at the HTTP request, and the timeout that caused the bad answer lives in the gap between them.
Instrument the chain. Redact before storage. Choose the three evaluations that answer the owner's question.
The AI Observability Accelerator is a four-week, fixed-scope delivery that instruments your AI application with Agent Observability, sets up the evaluators and dashboards, and covers AI Guard and GPU Monitoring where they apply. If Agent Observability is already on, a Datadog HealthScan reports span volumes, evaluator coverage and redaction first. It is read-only and fully automated.
- Inventory the AI applications. Every LLM-backed feature, its framework, its models and providers, and which environment it runs in. Production is instrumented first. Load-test environments are sampled or excluded.
- Instrument with the SDK or OpenTelemetry. Auto-instrumentation for OpenAI, Anthropic, Bedrock, LangChain and CrewAI where it applies, OpenTelemetry GenAI conventions where it does not. Custom spans for the steps the frameworks do not see.
- Correlate with APM and set the span budget. Agent traces joined to service traces so the timeout in the tool call is one trace, not two. A step limit on agent loops and a sampling rule per application, agreed against what an incident needs.
- Redact before anything is stored. Sensitive Data Scanner on prompts and completions, and span processors for the fields the scanner cannot infer. Data Access Control on who may read raw conversations.
- Configure the evaluators that answer the question. Managed Evaluators for safety and privacy, and one or two custom LLM-as-a-judge evaluators for the quality the product owner cares about. End-user feedback wired in, and a monitor on each with thresholds set from a fortnight of data.
- Hand over with a cost and quality baseline. Dashboards by feature and model, with the evaluator results and span volumes on the day we finished. Prompt Management holding the prompts that were live, and the sampling and retention decisions written down.
Access is admin on the Datadog organisation plus a short instrumentation change in the application, which your engineers make with us or we make through Engineering Drawdown with your review. Custom LLM-as-a-judge evaluators need a connection to your own LLM provider account, which you create.
The model changed on Tuesday, the prompt changed on Thursday, and the evaluator was written for the old one
AI applications change faster than the services around them: a provider swaps a model version, a prompt is edited in production, a new feature ships with its own agent loop. Managed Observability is the monthly service that keeps the tracing, the evaluators and the cost model matched to the application as it is now.
Every month, the span meter is reviewed. LLM spans per application against the budget, spans per conversation, retention against what anyone reads. An agent loop that grew gets a step limit. A feature that was retired gets its instrumentation retired too.
Coverage follows the features. A new LLM-backed feature without tracing, without evaluators or without redaction is found in the monthly review and instrumented before it is the feature a customer complains about.
The evaluators are kept honest. Evaluator scores checked against end-user feedback and against the sampled conversations an engineer reads. An evaluator that passes answers the users rejected is rewritten. Experiments run before a prompt or model change ships, not after.
The bad answer is an incident when it needs to be. A quality or safety evaluator crossing its threshold pages like any other monitor. Under Critical Support our engineer takes the alert, finds the span that caused it and fixes what sits in the runtime we operate.
We operate the stack. You own the product.
We operate the stack your AI runs on. We do not touch your model, your prompts or your product logic, and we do not pretend to be an AI consultancy. What we do is make the AI feature observable, evaluated and costed in production, and keep it that way every month. The instrumentation, the evaluators and the dashboards stay in your Datadog, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Agent Observability, answered plainly
Is Agent Observability the same as LLM Observability?
Yes. Datadog renamed LLM Observability to Agent Observability as the product grew from tracing single model calls to tracing whole agent workflows, and the documentation and install paths still use the old name in places. The product is one thing: a trace per request, a span per LLM call or agent step, with tokens, latency, cost and evaluations attached. If you searched for LLM Observability, this is the page.
How is Datadog Agent Observability billed?
On LLM spans ingested per month. Datadog's pricing list bills a first block of spans with a default retention and further blocks by volume, offers longer retention tiers, and states a monthly free allotment. One agent workflow produces several LLM spans, so the number of model calls an agent makes drives the bill. Custom LLM-as-a-judge evaluators run through an LLM provider account you connect, at that provider's rates. We do not publish prices. We set sampling per application, a step limit on agent loops, and run evaluators on a fraction of traffic agreed with you.
Do we have to change our application code?
A little. The Python and Node.js SDKs auto-instrument OpenAI, Anthropic, Amazon Bedrock, LangChain and CrewAI calls once the SDK is enabled, and applications already emitting OpenTelemetry GenAI semantic conventions are ingested without a Datadog SDK. Custom spans are needed for the steps the frameworks do not see, and for the business context, such as the feature or the tenant, that makes the cost attributable. We make or review those changes during the implementation.
Can it catch a bad answer before a customer does?
It can score one. Managed Evaluators check safety, privacy and quality dimensions on live traffic, custom LLM-as-a-judge evaluators score the thing your product owner actually cares about, and end-user feedback is attached to the trace. Each evaluator is a metric with a monitor, so a rising failure rate pages before the complaint arrives. Patterns, in Preview, clusters production conversations by topic so the kind of question you never tested for shows up as a group. Experiments let you test a prompt or model change against a dataset before it ships.
What does Critical Cloud do with Agent Observability every month?
Under Managed Observability we review the LLM span volume per application against the budget, and the retention against what anyone reads. We add tracing, evaluators and redaction to new AI features before they matter. We check evaluator scores against end-user feedback and sampled conversations, and run experiments before prompt or model changes ship. Under Critical Support an evaluator crossing its threshold pages our engineer like any other monitor, and we fix what sits in the runtime we operate.
- Datadog products: every product, and how we implement and run each one
- Datadog for AI: observing your AI, and the AI inside Datadog, on one page
- AI Observability Accelerator: Agent Observability, AI Guard and GPU Monitoring in four weeks
- MCP Server: your own agents reading Datadog, with access scoped and audited
The wrong answer has a cause. The cause has a cost.
Tell us which AI feature is in production and what the model bill was last month. We will show you what the traces say, what the evaluators should score, and where the spend actually goes.