The first four runbook steps
happen before anyone is paged.
Datadog Workflow Automation runs a sequence of actions when a monitor, a security signal, an incident or a work item says so. It gathers the context, restarts the thing that always needs restarting, opens the ticket, and only then pages a person. We implement it with approvals where they belong and run budgets where they are needed, and under Critical Support we are the person it pages.
Quick facts
Related
- Incident Management, the incident the workflow declares
- Event Management, the correlated work item that can trigger a run
- Alerts, the monitors that start most workflows
The runbook, executed instead of read
Most runbooks are a wiki page with six steps. The first four are always the same: pull the recent deploys, check the queue depth, restart the worker, post to the channel. The engineer does them by hand, at night, and then starts the actual investigation. Workflow Automation does the first four before the page is sent, and sends the page with the results attached.
Triggered by the thing that already knows. A monitor, a Cloud SIEM signal, a declared incident, a work item changing state, a dashboard button or a schedule. The trigger carries its context into the run, so the workflow starts with the service name, the tags and the alert query already in hand.
Actions across the tools you run. The Action Catalog covers Datadog itself, the cloud providers, Kubernetes, GitHub, Jira, ServiceNow, Slack and the rest. HTTP and JavaScript steps cover anything it does not. Private Action Runner reaches the systems that are not on the internet.
Branches, approvals and waits. A run can check a condition, wait for a human to approve in Slack or Teams, retry, or stop. The scale-up that needs a nod gets one. The restart that never did gets none.
Every run is a record. Each step, its input, its output and its duration, in the run history and on the Workflows Overview dashboard. When the runbook was wrong, the evidence of where it went wrong is already there.
A connection with admin rights, a trigger with no filter, and a remediation that triggers itself, one billed run at a time
How it bills. Workflow Automation is billed per run. A run is counted when a published workflow is triggered, by a person, by the API or by a monitor, however many steps it then takes. Draft and test runs are not counted. Failed, cancelled and timed-out runs are. When one workflow triggers another, billing follows the root trigger. Datadog documents that workflows triggered by Incident Management, On-Call and App Builder events are free, and that some security SKUs carry a per-host run allotment. We publish no rates. The meter is runs, and a trigger with no filter is a meter with no limit.
The configuration traps. The connection is created with the credentials of whoever built the first workflow, which is usually an admin. The trigger is attached to a monitor group rather than a monitor, so a noisy host fires the run hundreds of times. A remediation step restarts a pod, the monitor re-evaluates, fires again, and the workflow restarts the pod again, all night. An approval step is added, nobody is assigned to approve, and the run waits until it times out. A workflow written during an outage stays published after it, pointing at a service that was renamed. None of these errors is visible until the run history is read, and the run history is read once a quarter.
Field noteThe most common gap we see is a monitor trigger with no guard on how often it fires. The monitor flaps, each time it crosses the threshold starts a run, and by morning the same remediation has been applied again and again to a service that recovered after the first.
Field noteA connection is created for one workflow with a broad role because scoping it would have taken an afternoon. Every workflow written afterwards reuses it, and the blast radius of a bad step is the whole account.
Field noteThe pattern we keep finding is the remediation that masks the cause. A worker is restarted automatically every few hours for months. The restart works, the alert resolves, and the memory leak underneath it is never found because nobody is woken by it.
Field noteA workflow published during an incident carries a hard-coded namespace. The namespace is renamed in a later release. The workflow still runs, succeeds against nothing, and the alert it was meant to handle pages a person who assumes it was handled.
Scope the connection. Filter the trigger. Approve the step that needs it. Then publish.
The Service Management Accelerator is a four-week, fixed-scope delivery that sets up Incident Management, Event Management, Workflow Automation and Work Management together. Workflows are written in the third week, once the monitors and the severity model are settled, because a workflow attached to a monitor that will be reclassified is a workflow that will be rewritten. If workflows already exist, a Datadog HealthScan reports the published set, the connections and the run volumes first.
- Pick the runbooks that are actually run. From the last quarter's incidents and pages: which manual steps were repeated, by whom, how often. The first workflows automate those, and nothing speculative.
- Create connections with least privilege. One connection per system per purpose, with a role that can do the step and nothing else. Private Action Runner for anything inside your network, so no credential leaves it.
- Attach triggers with filters and cooldowns. Each trigger is bound to a specific monitor or signal, with a tag filter and a rate limit. A condition stops the run if the alert has already resolved. Scheduled triggers get a window, not a cron that runs forever.
- Put approvals where a human would want one. Any step that changes capacity, data or access waits for a named approver in Slack or Teams, with a timeout that fails safe. Restarts and context-gathering steps run without one.
- Test as a draft, then publish with a budget. Every workflow is run as a draft against a real alert before publishing. A run budget per workflow is agreed and monitored, so a surprise is an alert to us, not a line on the invoice.
- Hand over the runbook as code. Each workflow documented with its trigger, its connection, its approver and the manual steps it replaced. The Workflows Overview dashboard is the record of what ran.
Access is admin on the Datadog organisation plus the ability to create scoped service accounts on the systems the workflows act on. Where a step needs to reach inside your network, we install the Private Action Runner with you rather than asking for a route in.
A workflow written for March is wrong by September
Services are renamed, monitors are rebuilt, the person who approved the scale-up changes team. A published workflow does not notice any of it. Under Critical Support our engineers are the escalation when a workflow cannot resolve the alert, and under Managed Observability we review every published workflow against what it actually did.
Every month, the run history is read. Runs per workflow against its budget, failures and timeouts, approvals that waited past their timeout. A workflow that ran a hundred times for one alert is investigated, not celebrated.
Workflows follow the estate. When a monitor is rebuilt, a service renamed or a namespace moved, the workflows attached to it are updated in the same change. We check this in the monthly monitor review, because a stale trigger is a silent one.
The remediation that hides a cause is retired. A restart that runs on a schedule is a symptom with a timer. When a workflow is doing the same fix repeatedly, the fix goes into the improvement backlog and the workflow comes off.
The page still reaches an engineer. When the workflow has gathered the context and the restart has not worked, the page goes to our on-call engineer, 24x7, with the run attached. The fix is ours to make.
We operate the stack. You own the product.
Automation is only as safe as the person who reviews what it did. An implementation partner publishes the workflows and leaves. A managed partner reads the run history every month and is the engineer the workflow escalates to when it fails. We are the second kind. The workflows, the connections and the decisions stay in your Datadog, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Workflow Automation, answered plainly
How is Datadog Workflow Automation billed?
Per run. A run is counted each time a published workflow is triggered, by a person, by the API or automatically, regardless of how many steps it takes. Draft and test runs are not billed. Failed, cancelled and timed-out runs are. When a workflow triggers another workflow, billing follows the root trigger. Datadog documents that runs triggered by Incident Management, On-Call and App Builder events are free, and that some security SKUs include a per-host allotment. We do not publish prices. We set a run budget per workflow and alert on it.
Can a workflow fix the problem without paging anyone?
Sometimes, and that is the point. Gathering context, restarting a worker, scaling a group or rolling back a deploy can all run automatically with a condition that checks whether the alert resolved. Anything that changes capacity, data or access gets an approval step first. When the automated steps do not resolve the alert, the workflow pages a person with the run attached. Under Critical Support that person is one of our engineers.
What is the Private Action Runner for?
Steps that need to reach systems inside your network: a Kubernetes cluster without a public endpoint, an internal API, a database host. The runner is installed in your environment and polls Datadog for work, so no inbound route and no credential leaves your network. We install it with you during the implementation rather than asking for access from outside.
Is Workflow Automation the same as App Builder?
They are two products that share connections and the Action Catalog. Workflow Automation runs a sequence of actions from a trigger. App Builder builds a small internal tool with a form, a table and a button that a person uses. A common pattern is an App Builder page with a button that starts a workflow, so the engineer sees the context and chooses the action in one place. We cover App Builder on its own page.
What does Critical Cloud do with Workflow Automation every month?
Under Managed Observability we read the run history for every published workflow: runs against budget, failures, timeouts and approvals that expired. We update workflows when the monitors or services they touch change, and we retire any workflow that is applying the same fix repeatedly, moving the cause into the improvement backlog instead. Under Critical Support our engineers are the escalation when a workflow cannot resolve the alert, 24x7.
- Datadog products: every product, and how we implement and run each one
- Datadog Apps: the button a person presses to start the workflow
- Service Management Accelerator: incidents, events, workflows and work items set up in four weeks
The runbook that runs itself, and an engineer when it cannot.
Tell us which five commands your on-call runs most often at night. We will show you which of them should be a workflow, which need an approval, and who should be paged when they fail.