Request, error and latency for every service.
No tracer. No code change. No redeploy.
Datadog Universal Service Monitoring takes the golden signals from the Agent, at the kernel, for every service it can see, and puts them on the Service Map and in the Software Catalog. We check the prerequisites, fix the tags, scope the hosts, and run it every month so the services nobody wants to touch still have a monitor and an owner.
Quick facts
Related
- APM, the trace, for the services that earn one
- Service Level Objectives, built on the signals USM provides
- Software Catalog, where the discovered services appear
A monitor on the service nobody is allowed to redeploy
The billing service was written in 2017 by someone who has left. It works. Nobody will add a tracer to it, because adding a tracer means a deployment, and a deployment means a risk nobody wants to own. It is also the service that pages the most. Universal Service Monitoring watches it from outside.
The golden signals from the kernel. The Agent's system probe sees the requests a process sends and receives and reports rate, errors and latency per service, with nothing installed in the process and nothing restarted.
The services that have no tracer. Vendor binaries, legacy runtimes, languages the tracer does not cover, and the sidecar nobody owns. Each one becomes a service with a name, an owner and a graph.
The map, filled in. Discovered services appear on the Service Map and in the Software Catalog next to the traced ones, so the dependency that was a dotted line to an IP address becomes a service with a page.
Monitors and SLOs on day one. Rate, error and latency are enough for an alert, an error budget and a deployment marker, which is most of what a service needs before anyone looks at a flame graph.
A probe that never got its privileges, a service with no env tag, and a Go service behind TLS that the map cannot see
How it bills. Universal Service Monitoring is included on any host where APM is purchased, whatever the APM tier, and a host with both pays only for APM. Without APM it is an add-on to Infrastructure Monitoring, priced per infrastructure host and bought for all the Pro or Enterprise hosts on the plan. Covering only some hosts means buying APM on them. Datadog publishes the rates. We publish nothing here, because the price is not the problem. The configuration is.
The configuration traps. The product has a short list of needs and each one fails quietly. The system probe needs elevated capabilities on the Agent pod, and a cluster security policy that forbids them leaves the setting on and the probe off. The env tag has to be present before the process starts, and a service whose name is set in code at runtime is named by the Agent from its image instead. HTTP and OpenSSL are covered; Go and Node TLS, gRPC and Kafka have their own Agent and kernel minimums and some are still in Preview. Platforms that refuse the probe, such as GKE Autopilot, cannot run it at all. None of this is reported as an error. All of it is reported as a service that is missing.
Field noteThe most common gap we see is unified service tagging. The workloads were never given service and env tags because nothing needed them, so the Agent names the discovered services from whatever it finds, the app label or the image name. The Software Catalog gains a dozen services called app, and the SLOs are built on whichever one someone guessed.
Field noteThe newest services are written in Go and talk TLS to each other. Go TLS decoding is a Preview feature that has to be switched on separately, so the Service Map shows the old services in full and a hole where the new platform is. The hole is read as a product limitation and the rollout stops there.
Field noteService monitoring was enabled in the Agent configuration and a cluster security policy stripped the capabilities the system probe asked for. The setting says on. The probe never started. Nothing on the service pages says either of those things, and the absence of services was put down to low traffic.
Field noteUniversal Service Monitoring was bought as a cheaper APM. The latency monitors were built, the first one fired, and the on-call engineer opened the service page looking for the trace. There is no trace. The product was doing exactly what it does, and nobody had said what that was.
Tags before the probe. The probe before the monitors. The tracer only where it is earned.
We start with a Datadog HealthScan. It is read-only and fully automated, and it reports Agent versions, which hosts have service monitoring enabled, whether the system probe is actually running on them, and how the discovered services are named in the Software Catalog. Universal Service Monitoring is usually the second thing we switch on after Infrastructure Monitoring, and the sequence is the same every time.
- Inventory the services and sort them. Every service, its language, its platform, whether it speaks TLS and which library, and whether it will ever get a tracer. The services that will not are the ones this product is for.
- Confirm the prerequisites, host by host. Kernel version, Agent version, the capabilities the system probe needs, and whether the platform allows them. Each one checked, not assumed.
- Set unified service tags before anything is enabled. env on every workload, service and version wherever they can be set, applied by the deployment tooling before the process starts.
- Enable the probe by environment and check the catalog. Non-production first. Every discovered service checked for a real name, an owner and a sensible dependency list before production follows.
- Build the monitors and the SLOs. Rate, error and latency per service, with thresholds and error budgets set from a fortnight of real data rather than a round number.
- Hand over, in writing. Which services are watched this way and which are traced instead, and why. Which protocols are covered and which are not yet. What the catalog looked like on the day we finished.
Access is admin on the Datadog organisation plus read access to the Agent configuration and the deployment manifests. Tagging changes go through your pipeline, not ours. We do not need shell access to the hosts to implement Universal Service Monitoring, and we do not ask for it.
The catalog fills up on its own. Keeping it true is the work.
A new service ships without the env tag and appears under its image name. A kernel upgrade changes which protocols the probe can decode. A service that was fine on golden signals has its first real incident and needs a trace. Managed Observability is the monthly service that keeps Universal Service Monitoring matching the estate it watches.
Every month, we review the catalog. Services that appeared with an auto-generated name, services that disappeared, and the host count with the probe enabled against the host count where it is running. Anything that moved gets a reason or a fix.
Coverage follows the deployments. A new service without tags. A new node pool whose policy stripped the probe's capabilities. A new protocol the Agent now supports. Each is found in the review and corrected before it is the gap everyone is looking at.
The tracer list is kept honest. A service that has earned an incident is moved to the APM list and instrumented. A service that has been quiet for a year is left where it is. The line between the two is a decision, recorded each month.
The incident is ours to take. Under Critical Support our engineers answer the page the service raised, 24x7, and fix the cause rather than report it. Datadog has published that our work reduces mean time to resolve incidents by 40 to 60 per cent.
We operate the stack. You own the product.
An implementation partner enables the probe and hands over. A managed partner is still there when the security policy changes in March and the probe stops starting. We are the second kind. The Agent configuration, the tagging convention, the service lists and the decisions behind them stay in your repositories and your Datadog organisation, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Universal Service Monitoring, answered plainly
How is Datadog Universal Service Monitoring billed?
Universal Service Monitoring is included on any host where APM is purchased, whatever the APM tier, so on hosts that already have APM it adds nothing to the invoice. Without APM it is an add-on to Infrastructure Monitoring, priced per infrastructure host and bought for all the Pro or Enterprise hosts on the plan, and the way to cover only some hosts is to buy APM on them. We do not publish prices. What we decide with you is which of those two routes fits the estate, because it is a commercial choice as much as a technical one.
Does Universal Service Monitoring give us traces?
No. It gives request rate, error rate and latency per service, and it draws the service on the Service Map and in the Software Catalog. There is no span and no flame graph, because nothing is inside the process. The Agent watches the traffic from the kernel. That is enough for a monitor, an SLO and a dependency map. It is not enough to say which database call was slow. For that the service needs a tracer, and the usual pattern is that Universal Service Monitoring tells you which services deserve one.
What does Universal Service Monitoring need in order to work?
A recent Datadog Agent with the system probe enabled and the privileges it needs, which on Kubernetes means elevated capabilities on the Agent pod. A Linux kernel from the last several years, or Windows Server for IIS services. The env tag from unified service tagging on the workload, with service and version strongly recommended, set before the process starts. The newer protocols and TLS libraries have their own Agent and kernel minimums, and some are still in Preview. The platforms that forbid the system probe, such as GKE Autopilot, cannot run it.
Should we use Universal Service Monitoring or APM?
Both, on different services. APM goes on the services that matter at 2am and the services where latency has a business cost, because they need the trace. Universal Service Monitoring goes on everything else: the legacy service nobody will redeploy, the vendor binary with no tracer, the language the tracer does not cover, and the hundred small services that need a monitor and an owner but not a flame graph. The service inventory decides which list each service is on, and the decision is revisited when a service earns its first incident.
What does Critical Cloud do with Universal Service Monitoring every month?
Managed Observability runs it. Every month we review the Software Catalog for services that appeared with an auto-generated name, confirm the system probe is actually running on every host where it is enabled, check new deployments carry the tags, review the request, error and latency monitors and SLOs against what fired, decide which services have earned a tracer, and report what changed. Under Critical Support our engineers also take the page when a service goes wrong, 24x7.
- Datadog products: every product, and how we implement and run each one
- Datadog APM: the trace, for the services that have earned one
- Datadog Software Catalog: where every discovered service gets a page and an owner
- Datadog HealthScan: read-only and fully automated, the probe and tag check before anything is enabled
Every service with a monitor. Even the ones nobody will touch.
Tell us which services have no monitoring because nobody will redeploy them. We will tell you what Universal Service Monitoring can see of them today, what is in the way, and what it takes to get them on the map.