Know which prefix is costing the money.
And when anyone last read it.
Datadog Storage Management breaks Amazon S3 down to the prefix, with size, cost, access and age. Azure Blob Storage and Google Cloud Storage are in Preview. We choose the buckets, configure the inventories, and run it every month so the lifecycle decisions are made with evidence and the object count on the meter is one you chose.
Quick facts
Related
- Cloud Cost Management, the bill the prefix belongs to
- Log Management, where access logs are metered
- Infrastructure Monitoring, the cloud integration it depends on
The bucket stops being one number on the bill
The cloud bill says object storage went up again. The console says the bucket is large. Neither says that most of it is a prefix of export files from a job that was switched off two years ago, written once and never read. Storage Management says exactly that, with the date it was last touched.
Cost at the level a rule can act on. Each bucket is broken into prefixes with their size, object count and cost, so growth has an owner: a workload, a team or an environment, not a bucket name.
Cold data, found. Read, write and age metrics show which prefixes were last used and when, which is the evidence for moving them to a cheaper storage class or letting them expire.
Freshness as a check, not a guess. The age of the newest object in a prefix tells you whether last night's backup ran and whether the export the regulator expects is current. A monitor on it tells you before anyone asks.
Three clouds, one view. S3 today, with Azure Blob Storage and Google Cloud Storage in Preview. Each gets the same prefix breakdown, the same tags and the same monitors, instead of three consoles with three ways of saying nothing.
An inventory in the wrong format, a prefix spelt two ways, and a billion tiny objects nobody wanted to count
How it bills. Storage Management is billed per million objects monitored per month. The meter is the object count, not the bytes, so the bucket that costs most to watch is the one with the most objects in it, which is often not the one that costs most to store. Optional S3 access logs add request and latency metrics and are ingested through Log Management, on that meter. Datadog publishes the rates. We publish nothing here, because the price is not the problem. The configuration is.
The configuration traps. Storage Management reads inventory reports, and the inventory has to be set up the way it expects. That means CSV format and one destination bucket per account. It also means a destination prefix spelt identically in the inventory configuration, the bucket policy, the IAM permissions and the API registration. An inventory that excludes non-current versions makes the most expensive part of a versioned bucket invisible. An inventory enabled on every bucket in the account brings every bucket into scope, whether or not anyone will act on what it shows. None of this raises an error. All of it shows up as an empty dashboard or a surprising line on the invoice.
Field noteThe most common finding we see is the non-current versions. The bucket is versioned, the inventory was set to current versions only, and the largest prefix in the bucket is a decade of overwritten copies that no view has ever shown.
Field noteThe destination prefix was typed with a trailing slash in the bucket policy and without one in the API registration. Inventory files arrive every day. Datadog reads none of them. The ticket says the product does not work with our buckets.
Field noteThe inventory was enabled on the log archive bucket because it was the biggest. It is the biggest in objects, by a factor nobody had calculated, and the least interesting in cost per object. The meter went to the bucket with the least to say.
Field noteAccess logging was switched on for every bucket to get request metrics on all of them. The log ingestion bill for the access logs arrived a month before the first storage saving did, and was larger.
Choose the buckets. Set the inventory once, correctly. Then wait a day.
We start with a Datadog HealthScan. It is read-only and fully automated, and it reports which buckets the cloud integration can see, which already have inventory configurations, and whether resource collection is enabled on the integration. Storage Management depends on that integration being right, so if it is not, we fix that first.
- Inventory the buckets, per account and per cloud. Every bucket and container, its size, its object count, whether it is versioned, and who owns the data in it.
- Choose which buckets are worth watching. Ranked by storage cost and by object count, because one decides the value and the other decides the meter. The list is written down with the reason for each exclusion.
- Configure the inventories through your tooling. The CloudFormation stack or the Terraform module, with CSV format, daily frequency, all versions included, and one destination bucket per account under the agreed prefix.
- Register and confirm delivery. The destination is registered, the first report is awaited, and the prefix view is checked against the console before anyone relies on it.
- Decide access logs per bucket, and build the monitors. Request metrics only where the question is about traffic. Then growth per prefix, freshness on the prefixes a backup or an export should be writing to, and age on the ones that should be shrinking.
- Hand over, in writing. Which buckets are monitored and which are not, the lifecycle changes recommended with the evidence for each, and what the object count on the meter was on the day we finished.
Access is admin on the Datadog organisation, the cloud integration role with resource collection enabled, and a pipeline that can apply the stack or the module. Lifecycle rules are applied by you, in your account. We do not need write access to the data, and we do not ask for it.
The recommendation is the easy part. The rule has to be applied, and the bill has to be read afterwards.
A new bucket appears with a new export job writing to it. An inventory stops arriving because a bucket policy was tightened. A lifecycle rule was agreed in a review and never made it into the Terraform. Managed Observability is the monthly service that keeps Storage Management matching the storage it watches, and makes sure the decisions it supports actually happen.
Every month, we review the meter. The object count against the bucket list, the buckets whose inventory stopped arriving, the access log volume against the buckets that were meant to have it. Anything that moved gets a reason or a fix.
Coverage follows the accounts. A new bucket, a new account or a bucket that crossed a size threshold is found in the review and either brought into scope or deliberately left out, in writing.
Recommendations are followed through. Each lifecycle or storage class change recommended last month is checked: applied or not, and if not, why. A recommendation nobody acts on is a dashboard, not a saving.
The bill is read with the storage. With Cloud Cost Management in place, the prefix view and the storage line on the invoice are reviewed together. The change applied in April is visible in May's bill, or the question is asked.
We operate the stack. You own the product.
An implementation partner sets up the inventories and hands over. A managed partner is still there when the bucket policy is tightened in March and the reports stop arriving. We are the second kind. The stack, the module, the bucket list and the recommendations stay in your repositories and your Datadog organisation, documented, and you can take them in-house whenever you choose.
Datadog is our proof, not our headline. Datadog certified one managed provider globally. We are it. The Powered by Datadog accreditation means Datadog's own engineers reviewed how we instrument, alert, respond and report. We are also a Datadog Advanced Partner, ISO 27001 certified and Cyber Essentials Plus certified, with offices in Cardiff, London and Dublin.
Datadog Storage Management, answered plainly
How is Datadog Storage Management billed?
Datadog bills Storage Management per million objects monitored per month. The meter is the object count, not the bytes, so a bucket holding billions of small log files or thumbnails costs more to watch than a bucket holding a few thousand large backups, whatever either costs to store. Optional S3 access logs, which add request and latency metrics, are ingested through Log Management and land on that meter. We do not publish prices. The meters are what we tune, and the first decision is which buckets are worth an inventory at all.
Which storage services does Datadog Storage Management cover?
Amazon S3, with Google Cloud Storage and Azure Blob Storage in Preview. For each, it breaks the bucket or container down to the prefix, with size, object count, cost, read and write activity and the age of the data, so the question of which team or workload is driving growth has an answer at the level where a lifecycle rule can act on it.
Do we need S3 Inventory for Datadog Storage Management?
For anything below the bucket, yes. Bucket-level views appear once the AWS integration is connected, but the prefix breakdown and the recommendations come from S3 Inventory reports, so each monitored bucket needs an inventory configuration in CSV format, ideally daily and including all versions. The reports are delivered to one destination bucket per AWS account, under a prefix Datadog is allowed to read. Datadog provides a CloudFormation stack and a Terraform module that do this, or it can be configured by hand and registered through the API. The first inventory report can take up to a day to appear, so the prefix views are empty on day one by design.
Does Datadog Storage Management move or delete data?
No. It shows which prefixes are cold, how old they are and what they cost, so that a lifecycle rule or a storage class change can be made with evidence. The rule itself is applied in your cloud account, through your tooling, by someone who owns the data. We prepare the change with the prefix, the figures and the reason, and you decide.
What does Critical Cloud do with Storage Management every month?
Managed Observability runs it. Every month we review the object count on the meter against the bucket list, bring new buckets into scope or deliberately leave them out, confirm inventory reports are still arriving for every registered bucket, check which lifecycle recommendations were acted on and what the bill did afterwards, and report what changed.
- Datadog products: every product, and how we implement and run each one
- Datadog Cloud Cost Management: the storage line on the bill, next to the prefix that explains it
- Datadog Log Management: the meter access logs land on
- Datadog HealthScan: read-only and fully automated, the bucket and integration check before any inventory is created
Every prefix with a cost. Every cold one with a date.
Tell us which bucket everyone suspects. We will tell you what Storage Management would show about it, what it takes to set the inventory up correctly, and what to do with the answer.