The most expensive hosts in the estate.
Watched like they cost something.
Datadog GPU Monitoring shows utilisation, memory, throughput, health and cost for every GPU you run, in the cloud, on-premises or rented by the hour, and attributes the idle ones to the workload holding them. We implement it alongside Agent Observability, tag the fleet by owner, and run it every month so the GPU bill has a name on every line.
Quick facts
Related
- Agent Observability, the model call that ran on the GPU
- Infrastructure Monitoring, the host the GPU sits in
- Datadog for AI, the group these products belong to
The GPU that is reserved, paid for, and doing nothing, with the name of the job holding it
A training job finished on Thursday. The notebook server that launched it is still running, still holding four GPUs, and will hold them until someone closes the tab. The cloud bills every hour. Nothing on the standard dashboards distinguishes those four from the ones doing real work. GPU Monitoring does.
The fleet, by team and by spend. The summary and the Fleet Explorer show utilisation and cost per team, cluster, namespace and job, across hyperscalers, on-premises racks and rented GPU capacity, under the same tags.
Idle capacity with an owner. The provisioning view attributes inactive devices to the workloads holding them: processes that finished and never released, pods that reserved a GPU and run on the CPU, requests that cannot be scheduled.
The hardware explaining the latency. Memory utilisation, streaming multiprocessor activity and PCIe throughput per device, next to the inference trace, so a slow model call is traced to a saturated device or cleared of the blame.
Health before it is an outage. Monitors for thermal throttling and for XID and ECC errors, with guided steps, so the device that is about to fail is replaced on a weekday rather than diagnosed at the weekend.
A flag off on the GPU node pool, a job with no owner tag, and a held device that the meter never sees
How it bills. GPU Monitoring is billed per active GPU device per month, on the monthly average of active devices rather than the fleet size. A device is active when it shows graphics engine activity in a five-minute window. Datadog publishes the rates. We publish nothing here, because the price is not the problem. The configuration is.
The configuration traps. GPU collection is a flag in the Agent configuration, and GPU node pools are usually built from a template written for the CPU pools, so the flag is off exactly where it matters. Cost attribution is only as good as the tags on the workload, and training jobs are launched by hand, from notebooks, with no owner on them. An idle device is invisible to the Datadog meter and very visible to the cloud one, which is the point, and also the reason the first fleet view is uncomfortable. None of this raises an alert. All of it reaches the cloud bill.
Field noteThe most common finding we see is the held device. A notebook server, a finished training job whose process never exited, a serving pod that reserved a GPU and runs its model on the CPU. The cloud bills all of them by the hour. The utilisation graph for each is a flat line at zero.
Field noteCost by team is one bar, and the bar is labelled untagged. Training jobs are submitted from notebooks and scripts with no owner, team or project tag, so the attribution the product can do is attribution to nobody.
Field noteThe GPU node pool was autoscaled from the same Agent configuration as the CPU pools. The flag was off, the fleet view was empty, and the conclusion drawn was that the product did not support the instance type. The instance type was fine.
Field noteThroughput on an on-premises rack dropped after a change to the room cooling. The devices were throttling. Nothing alerted, because the only monitor on the rack watched training loss, and loss kept falling, just more slowly. The run finished days late.
The flag on the right pools. The tags on the right jobs. The trace joined to the device.
GPU Monitoring is implemented through the AI Observability accelerator, four weeks and fixed scope, which puts it in place together with Agent Observability so that the model call and the hardware it ran on are on the same screen from the first day. If the Agent is already on the GPU hosts, a Datadog HealthScan comes first. It is read-only and fully automated, and it reports which hosts carry GPUs, which have collection enabled and how the workloads are tagged. Either way, the sequence is the same.
- Inventory the fleet. Every GPU host and node pool, by provider, by device type and by who uses it. That includes the on-premises racks and the rented capacity that sits in no cloud account.
- Enable collection through the deployment tooling. The Agent flag set in the configuration that builds the GPU node pools, not on the hosts by hand. An autoscaled node then arrives already monitored.
- Put an owner on every workload. Team, project and job tags applied where jobs are launched. The default is visible rather than blank, so untagged work shows up as a problem instead of disappearing.
- Join the hardware to the model call. Inference services instrumented for Agent Observability, with the same service and env tags as the GPU host. A slow trace and a saturated device are then one click apart.
- Build the monitors and the dashboard pack. Unmet GPU requests, idle devices held beyond a threshold, thermal throttling, and XID and ECC errors. Thresholds come from a fortnight of real data, not a round number.
- Hand over, in writing. Which pools are monitored and how workloads are tagged. Which devices were idle and held by whom on the day we finished, and the active device count.
Access is admin on the Datadog organisation plus read access to the node pool configuration and the job submission tooling. Tagging changes go through your pipeline, not ours. We do not need shell access to the GPU hosts to implement GPU Monitoring, and we do not ask for it.
New pools, new jobs, new people with notebooks. The fleet view is only right on the day it was built.
A new node pool arrives with a different instance type. A team starts submitting jobs from a tool that does not carry the tags. A device starts throwing errors that the monitor was not watching for. Managed Observability is the monthly service that keeps GPU Monitoring matching the fleet it watches.
Every month, we review the meter against the fleet. Active device count against the inventory, idle capacity attributed to its owners, cost by team against last month. Anything that moved gets a reason or a fix.
Coverage follows the pools. A new node pool or a new provider is found in the fleet review. It is brought under the same flag, the same tags and the same monitors before anyone is asking where the capacity went.
Idle capacity is taken to its owner. The held devices from the review are listed with the job and the team holding them, so the conversation about releasing them has a name on it and a date.
The incident is ours to take. Under Critical Support our engineers answer the page the GPU host raised, 24x7, and fix the cause rather than report it. Datadog has published that our work reduces mean time to resolve incidents by 40 to 60 per cent.
We operate the stack. You own the product.
An implementation partner sets the flag and hands over. A managed partner is still there when a new team arrives in March with notebooks and no tags. We are the second kind. The node pool configuration, the tagging convention and the decisions behind them stay in your repositories and your Datadog organisation, documented, and you can take them in-house whenever you choose. The model, the data and the product are yours. We operate the runtime underneath.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog GPU Monitoring, answered plainly
How is Datadog GPU Monitoring billed?
Datadog bills GPU Monitoring per active GPU device per month, on the monthly average of active devices rather than the total number in the fleet. A device counts as active when it shows graphics engine activity in a five-minute window. A GPU that sits reserved but idle is not on the Datadog meter, even though it is very much on the cloud one. We do not publish prices. The meters are what we tune, and the first thing the product shows you is how many of your GPUs are not active at all.
Is Datadog GPU Monitoring generally available?
Yes. Datadog announced general availability on 22 April 2026, after a preview that ran from the middle of 2025. It appears under both Infrastructure and AI in Datadog's product navigation, because it is both: hardware telemetry from the Agent, and the hardware layer of the AI stack that Agent Observability sits on top of.
Does Datadog GPU Monitoring work on-premises and with GPU cloud providers?
Yes. It covers GPUs on AWS, Google Cloud, Azure and Oracle, on-premises hosts, and GPU-as-a-service providers such as CoreWeave and Lambda. The Agent is the collector in every case, enabled with a configuration flag on the hosts that carry GPUs. Cost attribution uses your cloud bill where Cloud Cost Management is connected, and ownership tags on the workloads everywhere.
How does GPU Monitoring relate to Datadog Agent Observability?
They are two layers of the same stack. Agent Observability, formerly LLM Observability, traces the model call: the prompt, the tokens, the latency, the evaluation and the cost per request. GPU Monitoring watches the hardware the model runs on: utilisation, memory, throughput, temperature and errors. When an inference service is slow, the trace says which call was slow and the GPU view says whether the device was saturated, throttled or waiting on something else. We implement them together through the AI Observability accelerator.
What does Critical Cloud do with GPU Monitoring every month?
Managed Observability runs it. Every month we review the active GPU device count against the fleet inventory and attribute idle capacity to the workloads and owners holding it. We check that new GPU node pools have the Agent flag set and the ownership tags applied. We review the throttling and device error monitors against what fired, and report what changed. Under Critical Support our engineers also take the page when a GPU host goes wrong, 24x7.
- Datadog products: every product, and how we implement and run each one
- Datadog Agent Observability: the model call, traced and evaluated, on top of the hardware
- Datadog for AI: observability, protection and GPU for the AI workloads you run
- AI Observability Accelerator: GPU Monitoring and Agent Observability operational in four weeks
Every GPU with an owner. Every idle one with a name.
Tell us how many GPUs you pay for. We will tell you how many did any work last week, who was holding the rest, and what it takes to keep that view honest every month.