Why the number is absurd
Audit logs are high volume and low urgency. Every statement against a busy database is an event. Almost none of them will ever be read, and the ones that will be read will be read months later, by someone with a specific question and a deadline. That is the opposite of what a live log index is for. Live indexing is designed for troubleshooting and alerting on recent data, and it is priced accordingly, because it is doing the most work the platform does.
Send the audit stream to that tier and you are paying the most expensive rate for the least urgent data. The bill is not telling you that audit logging is expensive. It is telling you that you sent the wrong logs to the wrong place.
Retrievable is not searchable
The usual compromise is the archive. Send the audit logs to cheap object storage, keep them for seven years, tick the box. The logs are retrievable. They are not searchable. When the auditor asks what happened to a specific record on a specific date in March, or an incident needs to know who ran what on a Sunday, the answer is a rehydration job, a wait of hours or days, and then a search. In the estates we review, archives with no lookup attributes are a consistent finding: the logs exist, and the clock they were supposed to satisfy cannot be met from them.
An audit clock measures time to an answer. A control that produces an answer in two days when the obligation says one is a control on paper.
Three tiers, three jobs. Live index for what you troubleshoot and alert on this week. Searchable long retention for what you must be able to answer questions about for months or years. Archive for what you must keep and will almost never read. The mistake is one tier for everything, and it is a default.
The design
- Decide what each log is for before deciding where it goes. Application logs are for troubleshooting: short retention, live index. Audit logs are for answering questions later: long retention, searchable. Bulk infrastructure logs are for completeness: archive, with the security-relevant subset filtered out before storage.
- Put audit logs in a searchable long-retention tier. Datadog's Flex Logs is built for this: retention of three, six, twelve or fifteen months on the starter option, or thirty to 450 days with scalable compute, searchable in the explorer without live-index pricing. Monitors do not run on it, which is fine, because audit logs are for questions, not alerts.
- Filter the security-relevant subset to the detection platform. Failed logins, privilege changes, schema changes, access to the sensitive tables. That is the slice a SIEM needs in real time. It is a fraction of the audit stream, and it is the fraction that pages someone.
- Archive the rest with lookup attributes. If a log must be kept beyond the searchable tier, archive it with the attributes you would search on, so rehydration is a targeted job and not a dredge.
- Redact before you retain. Audit logs copy the data they audit. Sensitive-data detection and redaction at ingest, so a seven-year retention of audit logs is not a seven-year retention of card numbers. We wrote about this in logs as a data-protection surface.
- Let the compliance owner decide the obligation, not the bill. Whether a specific audit log is required by your regulator is a question for whoever owns that obligation. The engineering job is to make any answer to that question affordable to act on, so the decision is made on the requirement and not on the quote.
Where we fit
Critical Cloud designs log tiering as part of Managed Observability: classify the sources, route each to the tier that matches its job, filter the security subset to detection, redact before retention, and keep the evidence that it ran. A payments customer came to us for the size of their log bill and left with audit logs searchable for as long as their compliance team required and a bill that reflected what each log was for. The wider approach is on our Datadog cost optimisation page.
We do not tell you which logs your regulator requires. We operate and evidence the runtime controls that support your obligations, and we make sure the quote never becomes the reason a control stays off.
Frequently asked questions
Why is database audit logging so expensive in a log platform?
Because audit logs are high volume and low urgency, and most estates send them to the tier built for the opposite: live indexing, designed for troubleshooting and alerting on recent data. Every statement against a busy database becomes an indexed event at the most expensive rate the platform has, for a log that will be read perhaps twice a year. The control is not expensive. The tier is wrong.
What is the difference between retrievable and searchable?
A log in cold archive is retrievable: it exists and can be rehydrated into the platform, which takes a job and a wait. A log in a searchable long-retention tier can be queried now. When an auditor or an incident asks what happened on a given date, the first answer is a two-day delay and the second is an afternoon. Datadog's Flex Logs tier exists for the second case, with retention from thirty days to fifteen months depending on the option, searchable without live-index cost.
Does Critical Cloud design log tiering as part of Managed Observability?
Yes. We classify log sources by what they are for, route application logs to live indexing with short retention, audit logs to a searchable long-retention tier, and raw bulk to archive, with the security-relevant subset filtered to the detection platform. The outcome is the control switched on, the obligation met, and a bill that reflects what each log is for. We do not decide which logs your regulator requires; your compliance owner does, and we make that decision affordable to act on.
Sources
- Datadog Docs, Flex Logs: retention options, intended use and the monitor limitation.
- The "priced out" conversation and the archive pattern are from our own work in customer estates. No customer is identified and no figures are published.