The page goes to a person.
The person fixes it.
Datadog Incident Management, On-Call and Status Pages turn a monitor firing into a declared incident with a responder, a timeline and a customer-facing status. We implement the model, wire the rota, and under Critical Support we are the engineers on it, 24x7.
Quick facts
Related
- Alerts, the monitors that start the incident
- Workflow Automation, the steps that run before a human is paged
- Service Level Objectives, what the incident cost in error budget
From a monitor firing to a written record, in the tool that raised it
Most incident processes live in three places: the paging tool, the chat channel, and whoever remembers what happened. Datadog puts the page, the response and the record in the same place as the monitor, the trace and the log that explain it.
On-Call pages the right person. Schedules, overrides, escalation policies and notification rules, with the monitor's context in the page. No separate tool to keep in sync with the monitors.
Incident Management runs the response. Declare from the monitor, assign commander and responders, keep the timeline as people work, link the dashboards and traces as they are used.
Status Pages tell customers first. Public or private pages updated from the incident itself, so the customer learns about the outage from you, not from a tweet.
The postmortem writes itself, mostly. The timeline, the monitors, the responders and the actions are already recorded. What is left is the thinking, which is the part worth a human's time.
A severity model nobody agreed, an escalation policy that ends at one person, and a postmortem template nobody opens
How it bills. Incident Management and On-Call are billed per seat, and an Incident Response seat covers both. A seat is needed to be on a schedule, receive pages or update an incident, not to declare, view or join one, so the cost follows the number of responders rather than the size of the estate. Status Pages are a feature of Incident Management, not a separate meter. Datadog publishes the meters. The lever here is not the meter. It is whether the people in the seats are the people who can fix the thing, at the hour it breaks.
The configuration traps. Every monitor pages, so the on-call learns to ignore the phone. The escalation policy has one level, so when that person is on a plane the incident waits. Severities are declared by feel, so a SEV1 on Tuesday is a SEV3 on Thursday and the reporting means nothing. The status page is updated by hand, late, by the same engineer who is fixing the outage. The postmortem template exists, and the last one written is from the quarter the template was introduced.
Field noteThe most common pattern we see is a rota that is one person. Not officially. Officially there are four names. In practice the alert goes to the one who always answers, and the other three have stopped expecting to be called.
Field noteMonitors are set to page because paging was the only notification channel configured. Warnings and informational alerts go to the same phone as the outage, and the phone is on silent by the second month.
Field noteIncidents are run in a chat thread and declared in Datadog afterwards, for the record, by someone reconstructing the timeline from scrollback. The timestamps are approximate, and the timestamps were the part that mattered.
Field notePostmortem actions are written with care and assigned to nobody. They are agreed in the meeting, recorded in the document, and discovered again at the next incident with the same cause.
Decide what an incident is before deciding who gets woken for it
The Service Management Accelerator is a four-week, fixed-scope delivery that sets up Incident Management, On-Call, Event Management and Workflow Automation together. For teams moving to Critical Support directly, the same sequence runs in the onboarding.
- Agree the severity model. Three or four levels, each defined by customer impact and a named example from your own history. Written down, so a SEV1 means the same thing at 3am as it does in the review.
- Sort the monitors. Each one is paging, ticketing or informational. Most are not paging. The ones that are get a runbook link in the message before they are allowed to page anyone.
- Build the On-Call schedules and escalation policies. Rotations that match real working patterns, overrides for leave, and an escalation path that always ends at someone who will answer. Under Critical Support, that someone is us.
- Configure Incident Management. Roles, the declaration flow from a monitor, the timeline, notification rules, and a postmortem template with the fields you will actually fill in.
- Wire the Status Page. Components mapped to services, updates driven from the incident, and a decision on what is public and what is for named customers only.
- Run a drill, then hand over in writing. One declared incident, end to end, in the first fortnight, with the gaps fixed afterwards. The handover records the severity model, the monitor classification, the rota and the drill result.
Access is admin on the Datadog organisation plus the contact details of everyone who is allowed to be on the rota. We do not need production access to implement Incident Management. The access we need to fix things under Critical Support is agreed separately, in writing.
The process is only as good as the last time it was used
People join and leave the rota. A new service ships without a paging monitor. The severity model drifts as the team changes. Managed Observability keeps the monitors and the rota in step with the estate. Critical Support is the service that runs the incident process every month, because we are the ones inside it.
We are the escalation. Our engineers are on the On-Call rota, 24x7, 365 days. The page reaches someone who can fix it, every time, and the incident is declared and run in your Datadog.
Every incident gets a review. Above the agreed severity, a postmortem within the agreed working days, with actions that have an owner and a date. The ones in our runtime go into our backlog and get done.
Monitors are reclassified monthly. Anything that paged without needing a human is downgraded. Anything that should have paged and did not is promoted. The phone stays worth answering.
The record is kept for you. Incident history, response times, postmortems and action status, in the format a customer questionnaire or an auditor asks for. Datadog has published that our work reduces mean time to resolve incidents by 40 to 60 per cent.
We operate the stack. You own the product.
Most partners configure the incident tooling and leave. We configure it and then answer the phone. The response time is a firm contractual commitment. The recovery time is a target, because recovery depends on the nature of the incident and not only on our speed, and we would rather say that than promise what nobody can.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Incident Management, answered plainly
What is the difference between Datadog Incident Management and Datadog On-Call?
On-Call is the paging layer. It holds the schedules, escalation policies and notification rules, and it decides which person's phone rings when a monitor fires. Incident Management is the response layer. It declares the incident, assigns roles, keeps the timeline, links the monitors and traces, and produces the postmortem. Status Pages tell customers what is happening. The three are designed to be used together, inside the same Datadog that raised the alert.
How is Datadog Incident Management billed?
Incident Management and On-Call are billed per seat rather than per host, and an Incident Response seat covers both. A seat is needed to be on a schedule, receive pages or update an incident, not to declare, view or join one, so the cost follows the number of responders, not the size of the estate. Status Pages are a feature of Incident Management, not a separate meter. Datadog publishes the current meters. We do not publish prices. When Critical Cloud staffs the rota, our engineers are the responders, which keeps the seat count small.
Do we still need PagerDuty or Opsgenie if we use Datadog On-Call?
Not for Datadog-sourced alerts. On-Call pages directly from monitors, with schedules, escalations and overrides built in, and the alert carries the context of the monitor that fired. Teams keep a separate paging tool when other systems page through it, or when the contract has time to run. We implement either, and we have moved teams from both.
What does Critical Cloud do with Incident Management every month?
Under Critical Support our engineers are the escalation target in your On-Call, 24x7, and they take the incident from the page to the fix. Every month we reclassify the monitors, so only the ones that need a human still page. We check the schedules and escalation policies against who has joined and left. Every incident above the agreed severity gets a postmortem, and we report the incident history and the action status. Datadog has published that our work reduces mean time to resolve incidents by 40 to 60 per cent.
What happens to the postmortem?
Every incident above an agreed severity gets one, built from the Datadog timeline, within a set number of working days. It names the cause, the contributing factors and the actions, with an owner and a date on each. The actions that sit in the runtime we operate go into our improvement backlog and get done. The rest are tracked until they are closed or formally dropped.
- Datadog products: every product, and how we implement and run each one
- Critical Support: 24x7 incident ownership, improvement engineering and runtime operations
- Service Management Accelerator: incidents, events and workflows set up in four weeks
A rota that always ends at someone who answers.
Tell us who holds the pager today and what woke them last month. We will show you what the incident process should look like, and what it takes for us to be on it.