Forty alerts, one cause,
one work item, one engineer.
Datadog Event Management takes every event the estate produces, from monitors, integrations and the alerting tools you have not retired yet, and correlates them so one cause raises one work item instead of forty notifications. We implement the patterns, decide what counts as one cause, and under Critical Support we are the engineer the work item reaches.
Quick facts
Related
- Alerts, the monitors that produce most of the events
- Incident Management, where a correlated work item becomes an incident
- Workflow Automation, the steps a new work item can trigger
The storm, reduced to the thing that started it
A node goes unhealthy. Every pod on it alerts. Every service those pods serve alerts. Latency monitors downstream alert. The on-call engineer receives thirty notifications in four minutes, and the one about the node is the eleventh. Event Management receives the same thirty, groups them by what they share, and opens one work item with the node at the top.
Everything arrives in one place. Monitor alerts, Watchdog and Error Tracking events, Kubernetes and deployment events, change requests, and alerts forwarded from the tools you are migrating away from. The Events Explorer is the one feed, and the Triage Inbox is where someone acts on it.
Pattern-based correlation you can read. A pattern groups events by the tags you choose, within a correlation window, after a minimum alert count. The preview shows how many events would match and how many work items would result, before the pattern is switched on.
Intelligent Correlation for what you did not anticipate. Datadog groups alerts from paging and high-priority monitors using infrastructure dependencies and telemetry, with no pattern written. It is scoped to the whole organisation and narrowed with tag filters, so it is switched on deliberately.
One work item, with the evidence attached. The correlated group lands in Work Management as a single item carrying every alert in it, so the engineer sees the node, the pods and the services as one story rather than thirty messages.
A pattern grouped by the wrong tag, a window set for a quiet week, and a correlator switched on for every monitor at once
How it bills. Datadog lists Event Management as billed per event evaluated by a correlation pattern, per month, and says the meter is not based on ingestion. Datadog-generated events, such as monitor alerts, Error Tracking and incidents, are free. The events that count are third-party events from other tools and custom sources. Custom events sent to Datadog also draw on a per-host allotment that comes with Infrastructure Pro and Enterprise. We publish no rates. The lever is which events a pattern evaluates, which is why a pattern that evaluates everything is the expensive one.
The configuration traps. The pattern groups by host when the cause is a service, so one deployment produces a work item per host. The correlation window is set from a quiet week and the next real storm spills across three items. The minimum alert count is one, so every single alert opens a work item and the queue becomes the alert feed. Intelligent Correlation is enabled organisation-wide with no exclude filter, so staging alerts and production alerts correlate into the same case. A pattern is edited during an incident, which stops every live work item it owned. Each of these is a parameter with a sensible value, and each was set once by someone who did not have the storm in front of them.
Field noteThe most common gap we see is correlation configured before the monitors were sorted. The patterns are good, the windows are right, and the storm still arrives, because the monitors feeding it were never meant to page in the first place.
Field noteThe pattern we keep finding is grouping by the tag that is easiest to see rather than the one that names the cause. Host is on every event. Service is on most. Deployment is on the one that mattered, and it was not in the dropdown.
Field noteAlerts from a legacy tool are forwarded into Datadog without a service tag. They arrive, they are real, and no pattern can group them with anything, so they sit in the Events Explorer alongside the Datadog alerts for the same outage, uncorrelated.
Field noteIntelligent Correlation is enabled on a Friday to reduce noise, and it does. On Monday the team finds a work item that correctly grouped a production database failover with the staging load test that caused the same symptoms, because nobody added the exclude filter.
Sort the monitors first. Then write patterns for the storms you have actually had.
The Service Management Accelerator is a four-week, fixed-scope delivery that sets up Incident Management, Event Management, Workflow Automation and Work Management together. Event Management comes in the second week, after the monitors are classified and the severity model is agreed, because correlating alerts that should not have paged is tidying the wrong thing. Where events are already flowing, a Datadog HealthScan reports the sources and volumes first.
- Inventory the sources. Every integration, every monitor class and every forwarded alert tool, with the tags each one carries. Sources without a service tag get one in the pipeline before anything is correlated.
- Replay the last three storms. From the incident history: which events arrived, in what order, and what the cause was. The first patterns are written to collapse those storms, not hypothetical ones.
- Write patterns grouped by cause. Group by the tag that names the cause, usually service or deployment. Size the window from the real storms and set the minimum alert count above one. The preview panel is checked against the replay before activation.
- Scope Intelligent Correlation. Enabled deliberately, with include filters for production and exclude filters for staging, load tests and known-noisy monitors. Its output goes to its own project so it can be judged separately.
- Route work items to owners. Each pattern sends to a Work Management project owned by the team that would fix the cause. The templated title names the service and the trigger. Under Critical Support that project is ours.
- Hand over the pattern set. Every pattern with its grouping, window, project and the storm it was written for. Plus the event volumes per source on the day we finished.
Access is admin on the Datadog organisation. For forwarded alerts from other tools we need a webhook or API integration from those tools, not access to them.
The patterns were right for the estate that existed when they were written
New services ship with new tags. A dependency moves and the storm changes shape. A pattern that collapsed last quarter's outages splits this quarter's into three items. Under Managed Observability we tune the patterns against what actually fired, and under Critical Support our engineers take the work item the correlation opens.
Every month, the correlations are reviewed. Work items per pattern, alerts per work item, events evaluated against the meter. A pattern that opened a hundred one-alert items is rewritten. One that never fired is questioned.
Patterns follow the estate. A new service, a renamed tag or a new source is checked against the pattern set in the monthly review, so the next storm from it is grouped rather than scattered.
The correlator is kept honest. Intelligent Correlation's project is reviewed for groups that were wrong, and the exclude filters updated. A correlator that is right most of the time is only useful if someone is counting the rest.
The work item is ours to take. Under Critical Support the correlated item reaches our on-call engineer, 24x7, with every alert in it attached. We fix the cause at the top of the group, not the thirty symptoms beneath it.
We operate the stack. You own the product.
Correlation is a judgement about what counts as one cause, and judgements need revisiting. An implementation partner writes the patterns from a workshop and leaves. A managed partner reads the work items every month and is the engineer who receives them at night. We are the second kind. The patterns, the pipelines and the decisions stay in your Datadog, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Event Management, answered plainly
How is Datadog Event Management billed?
Datadog lists Event Management as billed per event evaluated by a correlation pattern, per month, and says the meter is not based on ingestion. Datadog-generated events, such as monitor alerts, Error Tracking and incidents, are free, and the events that count are third-party events from other tools and custom sources. Custom events also draw on a per-host allotment that comes with Infrastructure Pro and Enterprise. We do not publish prices. We scope each pattern to the sources it needs, so the meter counts the events worth correlating rather than everything that arrives.
What is the difference between pattern-based correlation and Intelligent Correlation?
A pattern is a rule you write: group events by these tags, within this window, once this many alerts arrive, and send the result to this project. You can preview it against real events before activating it. Intelligent Correlation is Datadog grouping alerts from paging and high-priority monitors using infrastructure dependencies and telemetry, with no rule written. It applies to the whole organisation and is narrowed with include and exclude tag filters. We use both, patterns for the storms you have had and the correlator for the ones you have not.
Can Event Management take alerts from PagerDuty, Opsgenie or our old monitoring tool?
Yes. Event Management ingests events from any source, including third-party alert events, change requests and deployment events, alongside the hundred-plus integrations that send events natively. The condition for correlating a forwarded alert is that it carries the same tags as the Datadog alerts for the same service, which is a pipeline and processor job we do during the implementation.
Where does a correlated group go?
Into Work Management as one work item carrying every alert in the group, in the project the pattern names. From there it can be assigned to a person, or to a Datadog AI agent (Datadog lists that integration as in Preview), trigger a Workflow Automation run, or be declared as an incident in Incident Management. Under Critical Support the project is ours and the item reaches our on-call engineer.
What does Critical Cloud do with Event Management every month?
Under Managed Observability we review the correlations: work items per pattern, alerts per item, and events evaluated against the meter. Patterns that split a storm or that open one item per alert are rewritten. New services, tags and sources are checked against the pattern set. Intelligent Correlation's groups are reviewed and its filters updated. Under Critical Support our engineers take the work item the correlation opens, 24x7, and fix the cause at the top of it.
- Datadog products: every product, and how we implement and run each one
- Work Management: the queue a correlated group lands in
- Service Management Accelerator: incidents, events, workflows and work items set up in four weeks
One cause, one work item, one engineer who fixes it.
Tell us about the last alert storm and how many notifications it produced. We will show you what the same storm looks like after correlation, and who should receive the one item it becomes.