The queue is not slow.
The consumer behind it is three minutes behind.
Datadog Data Streams Monitoring follows a message from producer to consumer across Kafka, RabbitMQ, SQS and the rest, and shows the lag, latency and throughput at every hop. We implement it from the tracer you already run, on both ends of every queue that matters, and keep the host meter honest.
Quick facts
Related
- APM, the tracer this runs from
- Data Observability, the tables the pipelines land in
- Serverless Monitoring, the Lambda consumers on the SQS side
The message, followed from the producer to the service that finally read it
An order is placed and the confirmation email arrives twenty minutes later. The API was fast. The queue depth looked fine. Somewhere between the producer and the third consumer, a batch job took the partition for itself. Data Streams Monitoring draws that path and puts a latency on every segment of it.
Lag in seconds, per consumer group. Offset lag says how many messages are waiting. Lag in seconds says how late the customer is. The second is the one the business asks about.
A map of the pipeline as it runs. Producers, topics, queues and consumers drawn from traffic, including the service nobody knew was subscribed and the topic nobody knew was dead.
Latency across the whole pathway. Time from the first produce to the last consume, split by hop, so the slow segment is named rather than argued about.
Throughput and payload, in and out. Bytes per topic and per service, so a producer that started sending the full document instead of an ID is visible before the broker is.
A host meter on the services that emit it, a latency that needs single-threaded consumers, and a flag set on half the fleet
How it bills. Data Streams Monitoring is billed per host, on the services that run a tracer with it switched on. It is available on its own and it is bundled in the APM Pro and APM Enterprise tiers, so the first question is which tier the APM contract sits on before a separate line appears. Datadog publishes the rates. The configuration decides which hosts count.
The configuration traps. The product is switched on in the tracer, per service. Where it is enabled on the producers and not the consumers, the pathway ends at the topic and the lag is never computed. Latency for a pathway needs single-threaded message processing; a consumer that fans work out to a thread pool reports an approximate latency for that segment until it is instrumented by hand. Kafka lag in seconds needs a recent enough tracer and a Kafka client library that supports lag generation. Each gap looks like a healthy pipeline with fewer hops.
Field noteThe most common gap we see is coverage by team. The team that owns the producer turned the flag on, the team that owns the consumer did not know it existed, and the map shows every message disappearing into the topic.
Field noteThe pattern we keep finding is a lag monitor on offset count. The threshold is a round number of messages. Traffic grows, the number is reached every lunchtime, and the alert is muted the week it would have meant something.
Field noteA consumer that batches work across a thread pool reports an approximate latency for its segment. The dashboard shows a number that nobody trusts, and the slowest hop in the estate is the one whose figure is a guess.
Field noteThe flag is switched on everywhere the tracer runs, including the batch workers that only exist for an hour at night. They are hosts on the meter for that hour, and the high-water mark remembers them.
Both ends of every queue, before any dashboard
We start with a Datadog HealthScan. It is read-only and fully automated, and it reports which services have Data Streams Monitoring enabled, which pathways are complete, which end early, and which lag monitors exist, before anything changes. Then the sequence follows the queues, not the services.
- Map the pipelines that matter. Every topic and queue on the path from a customer action to the outcome they wait for, with the producer and consumer teams named.
- Check the tracer versions. Each language has a minimum SDK version for Data Streams Monitoring. Services below it are upgraded first, or the pathway will have holes.
- Enable it at both ends. The tracer flag on every producer and every consumer of each pathway from step one, not on the whole fleet.
- Instrument the fan-out consumers. Where a consumer processes across threads, the manual checkpoints that let latency be computed, written with the team that owns it.
- Build the lag and latency monitors. Lag in seconds per consumer group, pathway latency against what the business promised, thresholds from a fortnight of real traffic.
- Hand over, in writing. Which pathways are covered, which services carry the flag, the host count on the day we finished, and why each decision was made.
Access is admin on the Datadog organisation plus a tracer configuration change on each producer and consumer, usually an environment variable in the deployment. We do not need broker access, and we do not ask for it.
New consumers subscribe. Old producers change their payload. The pathway drifts.
A team adds a consumer group without the flag. A producer starts sending a larger message. A topic is retired and its monitor is not. Managed Observability keeps Data Streams Monitoring matching the pipelines it watches.
Every month, we review the meter. Hosts emitting Data Streams data against the high-water mark, and which of them sit on a pathway anyone asked for.
Coverage follows the pipeline. A new consumer group or a new topic appears in the map review and is brought under the flag and a monitor before it is the one everyone is looking for.
Monitors are kept honest. A lag monitor that fires every lunchtime is rebuilt on seconds and real traffic. One that has not fired for a real reason in a quarter is retired.
The incident is ours to take. Under Critical Support a lag monitor pages our engineers, 24x7, and they find the slow consumer, the stuck partition or the oversized payload rather than forward the alert.
We operate the stack. You own the product.
A pathway is only monitored when both of its ends are, and the two ends usually belong to different teams. We do the work across those teams, write down which pathways are covered and why, and keep the list current. The tracer flags, the monitors and the decisions live in your Datadog organisation and your deployments, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Data Streams Monitoring, answered plainly
How is Datadog Data Streams Monitoring billed?
Per host, on the services that run a tracer with Data Streams Monitoring switched on. It can be bought on its own and it is bundled in the APM Pro and APM Enterprise tiers, so the APM contract decides whether a separate line appears. We do not publish prices. Which services carry the flag is what we tune.
Which queues and streams does Datadog Data Streams Monitoring support?
Apache Kafka, whether self-hosted, Amazon MSK or Confluent Cloud, Amazon Kinesis, SQS and SNS, RabbitMQ, Google Pub/Sub, Azure Service Bus and IBM MQ, with Java, Python, .NET, Node.js, Go and Ruby tracers. Support varies by language. OpenTelemetry-instrumented services are supported where APM already receives their traces. Datadog's documentation carries the current list and the minimum SDK versions.
Why does our pathway end at the topic?
Because the consumer does not have Data Streams Monitoring enabled. The producer side is reporting and the consumer side is not, so Datadog can see messages leave but not arrive. Enabling the flag on the consumer fixes it. A consumer that processes messages across threads is a different problem: its segment appears, but its latency is approximate until manual checkpoints are added.
What is the difference between this and the Kafka integration?
The Kafka integration reports broker and consumer-group metrics: partitions, offsets, offset lag. Data Streams Monitoring follows the messages themselves across every hop, computes lag in seconds and latency per pathway, and draws the map. Both are useful. The integration tells you the broker is healthy. Data Streams Monitoring tells you the customer is still waiting.
What does Critical Cloud do with Data Streams Monitoring every month?
Managed Observability reviews the hosts on the meter and adds the consumers and topics that appeared since last month. It rebuilds lag monitors on seconds and real traffic, retires the ones that no longer fire for a real reason, and reports what changed. Under Critical Support our engineers take the lag incident, 24x7.
- Datadog products: every product, and how we implement and run each one
- Datadog APM: the tracer Data Streams Monitoring runs from
- Data Observability: the warehouse tables at the end of the pipeline
- Datadog HealthScan: read-only, fully automated, and the first thing we run
- Datadog monitoring and alerting guides: monitors built on real traffic, step by step
Lag you can read in seconds. A pathway with both ends on it.
Tell us which queue the customer waits on. We will show you where the pathway is complete, where it ends early, and what it takes to close it.