The card number redacted at ingest,
and the proof that it was.
Datadog Sensitive Data Scanner reads logs, APM spans, RUM events and Agent Observability traces as they arrive. It matches them against rules for personal data and credentials, and redacts, hashes or masks the match before it is indexed. We scope it to the sources that carry the data, write the rules for your own identifiers, run it where your residency commitments require, and keep the findings as Runtime Evidence.
Quick facts
Related
- Log Management, the pipelines the scanner sits inside
- Audit Trail, who changed the rules and who unmasked what
- AI Guard, the same rules applied to the conversations an agent has
Telemetry the whole engineering team can read, because the customer's data is not in it
Logs are written by developers who were debugging. A request body, a full user record, a token in a header, all logged because it was useful at the time. Six months later every engineer with log access can read it. The scanner catches the pattern at ingest, replaces it, and records that it did.
Scanned before it is stored. Logs are scanned during processing, before indexing and before any routing decision. Spans, RUM events and events are scanned on their attribute values. The redacted version is the only version Datadog keeps.
An action per rule. Redact replaces the match with a token you choose. Partial redact replaces a set number of characters, in the part of the match you choose. Hash gives a stable identifier for correlation without the value. Mask hides it except from people holding the unmask permission.
Rules for the common patterns, and for yours. The library covers card numbers, email addresses, national identifiers and cloud credentials. Custom rules, with keywords that must appear nearby, cover your own account numbers and references.
Findings kept as evidence. The findings page shows matches by rule and scanning group over time, with the overview dashboard installed on enablement. When the questionnaire asks whether personal data reaches the logs, the answer is a chart with dates.
A wildcard scanning group, sampling left on from the trial, and a residency promise the architecture does not keep
How it bills. Sensitive Data Scanner is billed per gigabyte of data it scans. The scanning group's filter and product toggles set that volume. Datadog publishes the meters and rates. The lever is the filter. A group that matches every log scans every log, and the bill tracks the chattiest service rather than the services that handle personal data. We set the groups by source and by service, and the scanned volume is reviewed against the scope that was agreed.
The configuration traps. A scanning group created during the trial with a wildcard filter and never narrowed. Sampling left on at a low percentage, which keeps the volume down and silently removes the redact, partial redact and hash actions, so the control that was bought is reporting matches and protecting nothing. A rule for a sixteen-digit number with no keyword, matching every order ID and timestamp in the estate. The scanner configured in Datadog while the data processing agreement says the data must be redacted before it leaves the network, which only the Observability Pipelines processor does. Masking chosen for convenience, with the unmask permission granted to a role that most of engineering holds.
Field noteThe most common gap we see is sampling left on. The group was set to scan a fraction during setup to see what matched. It was never switched to full, so the obfuscating actions were never available, and the findings dashboard has been reporting personal data in the logs without any of it being redacted.
Field noteThe pattern we keep finding is a residency answer in the questionnaire that the architecture does not support. The scanner runs in Datadog. The data arrives, then is redacted. The answer said it never arrives. Closing the gap means either moving the rules into the Observability Pipelines processor or correcting the answer that was given.
Field noteCustom rules without keywords are the ones that match everything. An eight-digit customer reference pattern also matches every order ID and invoice number of the same length. The keyword requirement, the field name that sits next to the value, is what turns a pattern into a rule.
Field noteThe service that logs the most personal data is usually the one that was never in the scanning group, because it was added to the estate after the groups were written. New sources default to unscanned, and nothing on the dashboard says so.
Find what is there first. Decide where redaction must happen. Then the rules, then full scan, then the evidence.
We start with a Datadog HealthScan. It is read-only and fully automated, and it reports which log sources and services are already in the estate and how they are indexed, which is the scope the scanning groups will be written against. The configuration then runs under Managed Observability, or through Catalyst where the pipelines need rebuilding first.
- Agree what counts as sensitive and where it must be caught. The data classes your policy names, your own identifier formats, and whether any commitment requires redaction before the data leaves your network. That last answer decides the architecture.
- Run a discovery group in sampling mode. A scanning group over the candidate sources at a sampled rate, with the library rules enabled, to see which rules match and in which services. The results are read with you before anything is redacted.
- Write the scanning groups by source. One group per set of services that handle the same data class, each with a filter that matches only those services, and the product toggles set for logs, spans, RUM and events as each source needs.
- Write and test the custom rules. Your identifiers as regular expressions with the keywords that sit next to them in real events. Each rule tested against a sample before it is enabled. The organisation's limits on groups and rules respected.
- Switch sampling off and set the actions. Redact, partial redact or hash per rule, chosen for what the engineers need to keep correlating. Mask only where a named role must see the value, with the unmask permission limited to that role. Where redaction must happen before egress, the same rules in the Observability Pipelines processor.
- Hand over, in writing. Groups and their filters, rules and their actions, which groups run in Datadog and which in the pipeline, who holds unmask, the findings baseline on the day we finished, and where the evidence lives.
Access is admin on the Datadog organisation. We do not hold the Data Scanner Unmask permission unless you decide we should, and we record who does. Where the Observability Pipelines processor is used, we need read access to the deployment tooling that runs it, not shell access to the hosts.
A scanning group written for last year's services scans last year's services
A new service ships and logs a request body. A data model change adds an identifier the custom rules do not know. A group is set to sampling for a test and left there. Managed Observability is the monthly service that keeps the groups, the rules and the actions matching the estate, and the evidence continuous.
Every month, the findings are read by rule and group. A rule that starts matching in a service where it did not before is raised with that service's owner. The data is in the code before it is in the logs. The scanner is the last line, not the fix.
Coverage follows the sources. New services and log sources are checked against the group filters. Anything that falls outside every group is either added or recorded as deliberately out of scope with a reason.
The control is kept honest. No protective group is left in sampling mode. The unmask permission is reviewed against the role list. Scanned volume is read against the agreed scope, so the filter that drifted to a wildcard is found before the invoice finds it.
The evidence is kept, dated. Findings over time per rule, the groups and actions in force, and who could unmask. We operate and evidence the runtime controls that support your compliance obligations. When the auditor asks, the record already exists.
We operate the stack. You own the product.
We are not a security company, and we are not a data protection adviser. We are a managed runtime operator. We run the scanner inside your Datadog and, where required, inside your pipeline. We keep its scope matching the estate and read what it finds. What sits in the runtime we operate, we fix. The rest goes to the owner with the finding attached. The data classification, the privacy policy and the decision about who may unmask stay yours. Runtime Evidence is what we produce: the dated record that the control was configured, ran, and caught what it was meant to catch.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Sensitive Data Scanner, answered plainly
Where does Datadog Sensitive Data Scanner redact the data?
In the default deployment, in Datadog, at ingestion. The log, span or RUM event arrives, is scanned during processing, and is redacted before it is indexed, archived or shown in the interface. The data has left your network by then. Where a residency or contractual commitment says it must not, the same rules run in the Sensitive Data Scanner processor for Observability Pipelines, inside your environment, before anything is sent. We ask which commitment you have made before choosing, because the two answers lead to different architectures.
How is Datadog Sensitive Data Scanner billed?
Sensitive Data Scanner is billed per gigabyte of data it scans. Scanning groups decide that volume: each one has a query filter and a toggle per product, and only the data matching the filter is scanned. Datadog publishes the current meters and rates. We do not publish prices. We do decide, with you, which sources go through a scanning group, because a scanning group with a wildcard filter scans every log in the organisation and sets the bill by the noisiest service.
Does the Sensitive Data Scanner sampling option still redact?
No. A scanning group can be set to scan a percentage of its matching data, which reduces the scanned volume while the rules are being tuned. With sampling on, the redact, partial redact and hash actions are not available. Sampling is a discovery mode: it tells you which rules match and where. For a scanning group that is meant to protect data rather than measure it, sampling is switched off, and we record which groups are in which state.
Can Datadog Sensitive Data Scanner find our own identifiers?
Yes. The scanning rule library covers the common patterns: email addresses, card numbers, national identifiers, cloud provider keys, tokens. Your own customer reference, account number or internal ID needs a custom rule, written as a regular expression with keywords that must appear nearby to reduce false matches. We write those from your data model, test them against a sample of real events before they are enabled, and keep them under the organisation's limits for groups and rules.
What does Critical Cloud do with Sensitive Data Scanner every month?
Managed Observability reviews the findings by rule and by scanning group every month. Any rule that starts matching in a source where it should not is raised with the owner. New services and new log sources are checked against the group filters. No protective group is left in sampling mode. Scanned volume is reviewed against the agreed scope. Custom rules are re-tested when the data model changes. We report what was found, where, and what was redacted, and the findings record is kept, dated, as evidence that the control ran.
- Datadog products: every product, and how we implement and run each one
- Log Management and Audit Trail: the pipelines the scanner sits in, and the record of who changed it
- Observability Pipelines: redaction before the data leaves your network
Personal data out of the telemetry. The evidence that it stayed out.
Tell us which services handle customer data and what you have promised about where it is redacted. We will show you what is in the logs today, which rules would catch it, and what it takes to keep the record.