The dashboard was right.
The table behind it stopped updating on Tuesday.
Datadog Data Observability watches the warehouses and the jobs that fill them: Snowflake, Databricks and BigQuery tables, and the Spark, Airflow and dbt runs behind them. We implement it beside the application telemetry, so a stale table and the pipeline failure that caused it are one investigation.
Quick facts
Related
- Data Streams Monitoring, the queues that feed the pipelines
- Database Monitoring, the transactional stores upstream of the warehouse
- Cloud Cost Management, the compute the jobs consume, on the same bill
The table, the job that filled it, and the run that did not
The revenue dashboard shows yesterday's number. Nobody notices for two days, because the number looks plausible. The Airflow run failed on a timeout, the downstream dbt model ran on stale input, and every chart kept rendering. Data Observability puts the failed run, the stale table and the dependent dashboards in the same place.
Quality monitors on the tables that matter. Freshness, row volume and rule-based checks on warehouse tables, with Datadog's monitors learning the normal cadence rather than enforcing a threshold someone guessed.
Jobs with their cost attached. Spark, Databricks, Airflow and dbt runs with duration, failure and the compute they consumed, so the job that doubled in cost is as visible as the one that failed.
Lineage that answers the second question. Which upstream table fed the broken one, and which dashboards and models sit downstream, before anyone starts tracing SQL by hand.
Caught in the pull request. The CI/CD component checks a data change before it merges, so the schema change that would have broken the model is a review comment rather than an incident.
A meter that counts tables and jobs, monitors on every column, and a warehouse role that cannot read the tables it was meant to
How it bills. Data Observability runs on its own meters, separate from the Infrastructure and APM host counts. Quality Monitoring is metered per monitored table, per month. Jobs Monitoring is metered per host, per hour for Spark and Databricks clusters. Airflow and dbt runs are metered per job, per hour, and Databricks Serverless on Databricks units. Datadog publishes the rates. The lever is scope: which tables and which jobs are worth a monitor, and which are not.
The configuration traps. The first instinct is to monitor every table. The warehouse has thousands, most of them staging, and a monitor on each produces alerts nobody owns and a meter nobody expected. The warehouse role granted to Datadog can list the schema but not sample the tables, so the volume checks return nothing and look healthy. Job monitoring is enabled on the Airflow instance but not on the Databricks workspace the heavy jobs moved to last year. None of this fails loudly. Each produces a green dashboard over a gap.
Field noteThe most common gap we see is scope. Quality monitors are created on the tables the data team knows best, which are the staging tables they built, rather than the handful of gold tables the business reads. The alerts arrive about the wrong layer.
Field noteThe pattern we keep finding is a freshness monitor on a weekly table, trained during a week when it updated daily. It alerts every day for a month and is muted. The week it should have fired, it is still muted.
Field noteThe warehouse role is created with the metadata grants and without read access to the tables. Schema and lineage arrive. The column-level and custom SQL checks, which have to query the data, never return a value.
Field noteJobs Monitoring covers the scheduler that was in use when it was set up. The heavy jobs migrate to a new workspace or a new runner, and the cost chart stays flat because the expensive runs are no longer in it.
Gold tables first. Then the jobs that fill them. Then everything else, if it earns it.
Where Data Observability is already connected, we start with a Datadog HealthScan. It is read-only and fully automated, and it reports which tables and jobs are monitored, how many monitors exist, which have fired and which have been muted since they were made. Where it is not connected yet, the sequence below starts at step one with your data platform team.
- List the tables the business reads. The reports, models and exports that make a decision, and the tables they depend on, with an owner for each.
- Grant the warehouse role properly. A read role that can sample the tables as well as list them. We create it with your data platform team and test it with the queries Datadog runs.
- Connect the job runners. Airflow, dbt, Spark and Databricks, including the workspace the heavy jobs moved to, so a failed run and its cost both land in Datadog.
- Set quality monitors on the gold layer. Freshness and volume on the tables from step one, with an owner on every monitor. The cadence is learnt from a full cycle, not a busy week.
- Wire lineage to the owners. So a monitor on a downstream table notifies the team that owns the upstream job, not the team that noticed the dashboard.
- Hand over, in writing. Which tables and jobs are monitored, why, who owns each monitor, and what the monitored counts were on the day we finished.
Access is admin on the Datadog organisation plus a read-only warehouse role and read access to the job schedulers. We never need write access to the warehouse, and we do not ask for it.
Pipelines change faster than the monitors watching them
A new dbt model ships every week. A table moves from staging to production without telling anyone. The heavy jobs migrate to a workspace nobody connected. Managed Observability keeps Data Observability matching the data platform it watches.
Every month, we review the scope. Tables and jobs under monitoring against the meter, monitors that fired and monitors that never did, and the gold tables that gained a dependency since last month.
Coverage follows the data platform. A new dbt model, a new workspace or a table that moved from staging to production is found in the review and brought under a monitor with an owner.
Monitors are kept honest. A freshness monitor that has been muted for a month is retrained or retired. A job whose cost doubled gets a reason or a conversation with the team that owns it.
The incident is ours to take. Under Critical Support a failed job or a stale gold table pages our engineers, 24x7, and they find the run, the input and the owner rather than forward the alert.
We operate the stack. You own the product.
Data Observability is only useful when the tables it watches are the ones the business reads and the jobs it watches are the ones that fill them. We make that scope with your data team, write it down, and keep it current as the platform moves. The monitors, the warehouse role and the decisions live in your Datadog organisation and your warehouse, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Data Observability, answered plainly
How is Datadog Data Observability billed?
On its own meters, separate from the Infrastructure and APM host counts. Quality Monitoring is metered per monitored table, per month. Jobs Monitoring is metered per host, per hour for Spark and Databricks clusters, and per job, per hour for Airflow and dbt. We do not publish prices. The scope, which tables and which jobs earn a monitor, is what we tune.
Which warehouses and job runners does Datadog Data Observability support?
Quality Monitoring covers Snowflake, Databricks, BigQuery, Redshift and ClickHouse Cloud. Jobs Monitoring covers Spark and Databricks jobs and clusters, Airflow and dbt runs. Lineage and the Data Catalog sit across them. Datadog's documentation carries the current list, which has grown with each release.
Do we need Datadog APM or Infrastructure Monitoring first?
No. Data Observability connects to the warehouse and the job runners directly. It is more useful beside them, because a failed Spark job then sits next to the cluster metrics and the trace of the service that consumed the table, but neither is a prerequisite.
How is this different from the data quality tests already in dbt?
dbt tests run inside the job and fail the run. Data Observability watches the tables between runs and after them, learns their normal cadence and volume, and connects a failure to the dashboards downstream and the team upstream. The two work together. The dbt test catches the bad row; the monitor catches the run that never happened.
What does Critical Cloud do with Data Observability every month?
Managed Observability reviews the tables and jobs under monitoring against the meter, adds the models and workspaces that appeared since last month, retrains or retires monitors that stopped being useful, and reports what changed. Under Critical Support our engineers take the failed-job or stale-table incident, 24x7.
- Datadog products: every product, and how we implement and run each one
- Database Monitoring: the transactional stores the warehouse is loaded from
- Data Streams Monitoring: the queues and streams between the services and the pipeline
- Datadog HealthScan: read-only, fully automated, and the first thing we run
- Datadog monitoring and alerting guides: monitors that fire for a reason, step by step
The table is fresh. The job ran. Somebody would know if not.
Tell us which reports the business runs on. We will show you which tables and jobs behind them are watched today, which are not, and what it takes to close the gap.