Every page has a reason.
Or it stops being a page.
Datadog monitors are the alerting layer for every product on the platform: metric, log, APM, anomaly, forecast and composite, routed to the people who can act. We build them from real data, route them to owners, and review every one of them each month. Alert fatigue is a configuration problem, and configuration is what we run.
Quick facts
Related
- Incident Management, what happens after the page
- Service Level Objectives, the error budget behind the threshold
- Event Management, so one cause raises one case
The alert that means something, sent to the person who can do something
The on-call phone goes at 3am. It is the disk monitor on the build server, again, at a threshold somebody set years ago. The engineer silences it and goes back to sleep. Forty minutes later the checkout service fails, the monitor for it was muted the previous week, and nobody is woken. Alerting is not a volume problem. It is a configuration problem.
A monitor type for every signal. Metric, log, APM, synthetic, process, network, error tracking, SLO, cloud cost and more. The right type means the condition is evaluated where the data lives.
Composite monitors that page on the combination. High latency and high error rate together is an incident. Either alone is a graph. A composite monitor pages on the first and records the second.
Anomaly and forecast, where a threshold is wrong. Traffic that is normal on Monday and alarming on Sunday needs a monitor that knows the week. Disk that will fill on Thursday needs one that says so on Tuesday.
Notifications that carry the next step. The message names the owner, the runbook, the dashboard and the conditional text for warning versus critical, so the first minute of the incident is reading, not searching.
A threshold from years ago, a mute that became permanent, and a notification to a channel nobody reads
How it bills. Monitors are not metered. Datadog includes them with the platform, with no charge per monitor or per notification. The costs sit underneath: a log monitor needs the logs it watches to be indexed, a custom metric monitor needs the metric to exist, and a Synthetic test monitor needs the test runs. Datadog publishes those rates. The monitor itself is free, which is why there are so many of them.
The configuration traps. Thresholds are set on the day the monitor is written, from whatever the graph showed that afternoon, and never revisited. Downtimes are created with no end date. Notifications go to a channel with hundreds of members and no owner, which is the same as going nowhere. Monitors are copied from a template into every environment, so staging pages the production rota. A monitor with no-data handling left at its default stays green when the Agent stops reporting, which is the one condition it should have shouted about.
Field noteThe most common gap we see is the monitor that will never fire. Its query references a tag that was renamed in a migration, it has been green since, and nobody checks a monitor that is green.
Field noteThe pattern we keep finding is a downtime with no end. It was set during a maintenance window, the window closed, and the monitor it covers has been muted ever since.
Field noteThe pattern we keep finding is one Slack channel for every monitor in the organisation. The channel carries more messages a day than anyone can read, so it is muted. The critical for the checkout service lands in the same stream as the warning for the build server.
Field noteNo-data handling is left at the default on the host metric monitors, which is to keep the last known status. A host loses its Agent and every metric monitor on that host stays green rather than going critical. The dashboards show a gap where the outage was.
Delete first. Then rebuild from real data. Then route to a person.
We start with a Datadog HealthScan. It is read-only and fully automated, and it reports every monitor, when it last changed state, whether its query still returns data, where it notifies, and every downtime without an end date, before anything changes. Then the sequence is subtraction before addition.
- Inventory the monitors. Every monitor, what it watches, when it last changed state, who it notifies, and whether anybody did anything the last time it fired.
- Retire the dead ones. Monitors whose queries return no data, duplicates, and monitors on services that no longer exist. Each removal recorded.
- Set thresholds from a fortnight of data. Warning and critical from real behaviour, with anomaly or forecast monitors where the pattern is weekly rather than flat.
- Build the composites. The combinations that mean an incident, paging once on the combination rather than twice on the parts.
- Route to owners. Each monitor tagged to its owning team, notifying that team's channel or rota, with a message that carries the runbook and the dashboard.
- Hand over, in writing. The monitor inventory, what was retired and why, the routing map, and the monitor count on the day we finished.
Access is admin on the Datadog organisation plus the notification integrations. We do not need access to the systems the monitors watch, and we do not ask for it.
A monitor is right on the day it is written and drifts from then on
Traffic grows past the threshold. A tag is renamed and a query goes quiet. A new service ships with no monitors at all. Managed Observability reviews every monitor, every month, so the estate pages for the reasons it should and no others.
Every month, every monitor. Fired, not fired, muted, no data. A monitor that has not fired for a real reason in a quarter is retired or rebuilt. One that fires daily is retuned, or the thing it watches is fixed.
Coverage follows the estate. A new service with no monitors, or a renamed tag that silenced old ones, is found in the review and covered before it is the service everyone is looking for.
Downtimes are kept honest. Every downtime is listed with its end date. The ones with none are closed, and the monitors under them are checked for what they missed.
The incident is ours to take. Under Critical Support the monitor pages our engineers, 24x7, and they fix the cause rather than acknowledge the alert. Datadog has published that our work reduces mean time to resolve incidents by 40 to 60 per cent.
We operate the stack. You own the product.
An alerting estate is a set of decisions about what matters at 3am, written down as monitors. We make those decisions with you, build them from data, and keep reviewing them. The monitors, the routing and the reasons live in your Datadog organisation, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog alerts and monitors, answered plainly
How are Datadog monitors and alerts billed?
They are not metered. Datadog includes monitors and notifications with the platform, with no charge per monitor. The costs sit in the data they watch. A log monitor needs the logs indexed, a custom metric monitor needs the metric, and a Synthetic monitor needs the test runs. Each of those has its own meter. We do not publish prices. The monitor count is free, which is why we keep it low.
What is the difference between a metric monitor and an anomaly monitor?
A metric monitor compares a value with a threshold you set. An anomaly monitor compares it with what the value usually is at that time of day and week, and alerts on the deviation. Flat thresholds suit signals with a hard limit, such as disk or error rate. Anomaly suits signals with a rhythm, such as traffic or queue depth. A forecast monitor is the third option: it alerts when the trend will cross the threshold, before it does.
How do we stop Datadog alert fatigue?
By treating it as a configuration problem. Every monitor is reviewed for whether it fired for a real reason, retired if not, and retuned from real data if it fires too often. Composite monitors replace pairs that page twice. Downtimes without an end are closed. Notifications are routed to an owning team rather than a shared channel. The fix is a review cadence, not a louder phone.
Can Datadog route different alerts to different teams?
Yes. A monitor's notification message can target a team handle, a Slack or Microsoft Teams channel, an email, a PagerDuty or Opsgenie service, or Datadog On-Call. Conditional variables can send warning and critical to different places. With Teams configured, the service's owner is the default target rather than a decision made per monitor.
What does Critical Cloud do with Datadog alerts every month?
Managed Observability reviews every monitor: fired, not fired, muted and no-data. It retires the ones that stopped firing for a reason, retunes the ones that fire too often, closes downtimes with no end date, covers new services, and reports what changed. Under Critical Support our engineers answer the page, 24x7.
- Datadog products: every product, and how we implement and run each one
- Incident Management: the page, the response and the record, staffed by us
- Datadog HealthScan: every monitor, its last state change and where it notifies, read-only and fully automated
A page that means something. A rota that trusts it.
Tell us what woke the on-call engineer last week. We will show you which monitors fired for a reason, which did not, and what it takes to make the phone trustworthy.