Logs that answer the question,
without indexing the internet.
Datadog Log Management collects everything and lets you choose what to keep. Most estates never make the choice. We implement the collection, design the indexes, set the filters and archives, and run it every month so the logs stay useful and the bill stays flat.
Quick facts
Related
- BYOC Log Management, logs that stay in your own cloud
- Observability Pipelines, shaping logs before Datadog sees them
- Audit Trail, who changed what in Datadog
Every log line, parsed once, next to the trace that wrote it
The log line that explains the outage exists. It is on one of four hundred containers, in a format nobody agreed, three minutes before the alert. Log Management puts it one search away, with the trace and the host beside it.
Collection from everything. The Agent, cloud forwarders, serverless extensions and the API all land in the same place. Kubernetes, AWS and Azure sources are a configuration, not a project.
Pipelines that parse once. Processors turn each source into structured fields on the way in. A status, a service and a duration become facets the whole team can search and alert on.
Keep what matters, where it belongs. Standard indexes for the logs that drive monitors. Flex Logs for the ones audits need later. Archives in your own bucket for the rest, rehydrated when an investigation calls for them.
Logs joined to traces and metrics. With unified service tags and trace ID injection, the slow span opens the exact log lines written during it. No copying timestamps between tabs.
One index, default retention, and a container that logs its own health check every second
How it bills. Ingestion is billed per gigabyte. Indexing is billed per million events, at a rate set by the retention period on the index. Flex Logs is a lower-rate indexing tier for high-volume, rarely-queried logs. Archives are not indexed and cost only the storage. Rehydration is billed on the compressed data scanned, and the events it brings back are indexed at the usual rate. Datadog publishes the rates. The lever is not the rate. It is how many events reach an index, and for how long they stay.
The configuration traps. A trial sends every log to one index at the default retention. That configuration then goes to production. No daily quota, so a bad deploy that logs a stack trace per request indexes until someone notices the invoice. No exclusion filters, so health checks, debug output and load balancer access logs are indexed at the same rate as the error that mattered. An archive configured once and never rehydrated, so nobody knows whether it works until the audit asks for March.
Field noteThe most common cost pattern we see is a handful of Kubernetes system containers producing more indexed events than the whole application. Nobody chose to index them. They were simply never excluded.
Field noteMulti-line logs that are not configured as multi-line arrive as one event per line. A Java stack trace becomes forty events, each billed, none searchable as the error it was.
Field noteRetention is set to the longest value anyone asked for, on every index, because one index was easier than five. The security team needed a year. The application team needed a week. Everything gets a year.
Field noteLog-based monitors are written against an unparsed message string. They work until the log format changes in a minor release, and then they go quiet without failing.
Decide what to keep before collecting more of it
For an estate already sending logs, we start with a Datadog HealthScan. It is read-only and fully automated, and it reports volume per source, per index and per retention tier before anything changes. For a new estate, LaunchPad runs the rollout. The sequence is the same either way.
- Inventory the sources. Every service, container, cloud service and forwarder that emits logs, with its daily volume and who reads it. Most lists have a surprise in the top five.
- Set unified service tagging and trace ID injection. So each log line carries the same env, service and version as its trace, and the trace ID to join them.
- Build the pipelines. One pipeline per source type, with processors for status, service, duration and the fields the monitors will need. Multi-line handling configured at the Agent, not patched later.
- Design the indexes. Separate indexes by retention need, each with a daily quota and a warning before it. Exclusion filters for health checks, debug output and known noise, with a sample kept so the pattern is still visible.
- Set Flex Logs and archives. Flex for the logs an investigation or an audit will want months later. An archive to your own bucket for everything, with a rehydration test run and recorded on the day.
- Build the monitors and hand over in writing. Log-based monitors on parsed facets, not message strings. A handover note with the index design, the filter list, the volumes on the day we finished, and why each decision was made.
Access is admin on the Datadog organisation plus write access to the storage bucket for archives. We do not need production shell access, and we do not ask for it.
The index design that was right in January is wrong by June
New services log in new formats. A dependency upgrade changes the message a monitor was parsing. Traffic grows and the quota set in a quiet month is hit in a busy one. Managed Observability keeps the log configuration matching the estate that produces the logs.
Every month, we review the volumes. Ingested gigabytes and indexed events per source, against last month. Anything that moved gets a reason, a filter or a conversation with the team that owns it.
Filters and quotas are kept honest. Exclusion filters that stopped matching are removed. Noise that appeared since last month is excluded. Quotas are raised deliberately, not hit by accident.
Coverage follows the estate. A new service logging in a format no pipeline parses is found in the volume review and parsed before its first incident. The archive rehydration test runs on the same schedule, so the first request for last March is not the first attempt.
The incident is ours to take. Under Critical Support a log-based monitor pages our engineers, 24x7, and they fix the cause rather than forward the alert.
We operate the stack. You own the product.
A log configuration is a set of decisions about what the business needs to see and for how long. We make those decisions with you, write them down, and keep them current. The pipelines, indexes and archives live in your Datadog organisation and your cloud account, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Log Management, answered plainly
How is Datadog Log Management billed?
Two meters. Ingestion is billed per gigabyte sent to Datadog. Indexing is billed per million events, at a rate set by the retention period on the index. Flex Logs is a separate, lower-cost indexing tier for high-volume logs that are queried rarely. Archives to your own cloud storage are not indexed and cost only what the storage costs. Rehydrating from an archive is billed on the compressed data scanned, and the events it brings back are indexed at the usual rate. We do not publish prices. The index design and the exclusion filters are what we tune.
What is the difference between Standard indexing and Flex Logs?
Standard indexing keeps logs hot, searchable in Log Explorer and usable in monitors, for a retention period you set per index. Flex Logs keeps logs searchable for longer at a lower rate, with query performance that suits investigations and audits rather than live alerting. The usual split is Standard for the logs that drive monitors and incidents, Flex for the logs that compliance or forensics need months later, and archives for everything else.
Why do our Datadog logs not link to our traces?
The log line and the span carry different service, env or version tags, or the trace ID is not being injected into the log. Unified service tagging fixes the first. Enabling trace ID injection in the tracer and parsing it in the log pipeline fixes the second. Once both are in place, a slow span links to the exact log lines written during it.
Should we use Datadog Observability Pipelines for logs?
When the volume is large enough that filtering at the source saves real money, or when sensitive fields must be redacted before they leave your network. Observability Pipelines runs in your environment and shapes the logs before Datadog ever sees them. For smaller estates, exclusion filters and index quotas inside Datadog do the same job with less to run. We help you decide which, and we run either.
What does Critical Cloud do with Log Management every month?
Managed Observability reviews ingested and indexed volume per source and checks every index against its quota and retention. It removes exclusion filters that stopped matching, adds the ones that should, tests that archives still rehydrate, and reports what changed. When a log-based monitor fires under Critical Support, our engineers take the incident.
- Datadog products: every product, and how we implement and run each one
- Datadog logs pricing: why the log bill grows and the controls that stop it
- Infrastructure Observability Accelerator: logs, APM and infrastructure operational in four weeks
- Log management guides: setup, parsing, retention, archiving and enrichment, step by step
Keep the log that matters. Stop paying for the one that does not.
Tell us where the logs come from. We will show you what is indexed today, what should be, and what it takes to get there.