Using Datadog Observability Pipelines
for Cost Control
The most effective way to control Datadog log costs is to decide what reaches Datadog before it arrives. Observability Pipelines give you that control at the ingestion layer, without changing application code.
What Observability Pipelines are used for
Datadog Observability Pipelines is a Datadog capability that intercepts telemetry (primarily logs, but also metrics and traces) between your sources and your destinations. A pipeline runs as a component between the agent and the backend, applying rules you define: drop this, transform that, route this source to that destination.
For cost control, the primary use is pre-ingest filtering. Events that would not be queried, searched or alerted on are dropped at the pipeline rather than ingested, indexed and retained at full Datadog cost. For security and compliance, pipelines can also redact sensitive data before it ever reaches the platform's storage layer.
The crucial point is that pipeline decisions are permanent cost controls. Unlike retention tuning, which reduces the cost of data you have already ingested, pipeline filtering prevents that cost from occurring at all.
Why pre-ingest control matters
Every governance control applied after ingestion operates on data that has already been counted against your usage. Retention tuning, exclusion filters in Datadog's index configuration and archiving all reduce downstream cost, but the ingestion has already occurred.
Pre-ingest control with Observability Pipelines is the furthest upstream you can push that decision. Data that is dropped at the pipeline never touches Datadog's ingestion metering. For very high-volume sources where only a fraction of events have operational value (container infrastructure logs, health check responses, low-level debug output) this distinction is significant.
Route, reduce and transform
The three operations that make Observability Pipelines useful for cost and governance.
Operation 1
Route
Send different log streams to different destinations based on source, content or attributes. Operational logs go to indexed Datadog logs for active monitoring. Compliance logs go to S3 or Azure Blob for long-term retention at archive cost. Security logs go to a SIEM. Each stream reaches the destination appropriate to its value and compliance requirement.
Operation 2
Reduce
Drop events that do not justify indexing cost before they are ingested. Health check responses, readiness probe logs, verbose library output and debug events from non-production workloads are common candidates. Reducing at the pipeline is permanent: those events never enter the usage meter.
Operation 3
Transform
Modify events before they reach their destination: redact sensitive fields (PII, credentials, tokens), restructure log format for consistent parsing, enrich with context from other sources or strip high-cardinality attributes that add indexing weight without adding query value. Transformation happens without application code changes.
Use cases for cost control
Reduce noisy infrastructure logs
Kubernetes control plane, kube-proxy, container runtime and node-level logs can generate enormous volumes with low operational signal density. A pipeline rule that drops repetitive infrastructure events, or samples them at a rate appropriate to their value, can reduce log volume substantially without removing the coverage that matters for incident investigation.
Redact sensitive data before ingest
Applications sometimes emit log events that contain sensitive data: authentication tokens, partial credit card numbers, personally identifiable information, internal API keys. A pipeline with redaction rules removes that data at the agent layer before the event reaches Datadog's storage, simplifying compliance and reducing the scope of data governance controls needed in the platform itself.
Route security logs differently
Security-relevant log sources often need different treatment from operational logs: longer retention, different access controls and routing to a SIEM or compliance archive. A pipeline lets you tag and route security sources to appropriate destinations without sending everything through the same index with the same retention settings.
Send low-value logs to cheaper storage
Logs with retention requirements but low query frequency (compliance archives, audit trails that are checked rarely) do not need to be in an indexed Datadog log index. A pipeline can route them to S3 or Azure Blob Storage at a fraction of the indexed log cost, with rehydration available if they are ever needed for investigation or audit.
Preserve high-value operational signals
Cost reduction through pipelines should be targeted, not blunt. The goal is to drop what has no value and preserve what does: application errors, slow request logs, authentication failures, deployment events and infrastructure anomalies. A well-configured pipeline improves signal-to-noise ratio alongside cost, making what remains in Datadog more useful, not less.
Normalise log format across sources
Inconsistent log formats from different services and infrastructure components create parsing overhead and make log-based alerts harder to maintain. A pipeline can normalise log structure across sources before ingest, reducing the parsing configuration required in Datadog's pipeline processing layer and making log queries more consistent across the platform.
Log costs
Datadog Logs Pricing: How to Control Log Costs
Ingestion, indexing and retention explained, alongside the full set of log cost-control techniques including pipelines.
Read the logs guideFull guide
Datadog Pricing & Cost Optimisation
The complete Datadog cost governance guide: all cost drivers, the Observability FinOps framework and the 30-day review plan.
Read the full guideRelated Datadog cost guides
Frequently asked questions
What are Datadog Observability Pipelines?
Datadog Observability Pipelines is a Datadog capability that lets you intercept, route, transform and filter telemetry data (primarily logs) before it reaches its destination. Pipelines run as an agent-layer component that sits between your log sources and Datadog (or other destinations), giving you control over what is sent, in what form and to which destination.
How do Observability Pipelines reduce Datadog costs?
By intercepting log data before it reaches Datadog, Observability Pipelines let you drop events that have no operational value before they consume indexing budget. They also let you route low-value logs to cheaper archive storage rather than to indexed Datadog logs, and route security-relevant logs to a separate destination with appropriate retention. The result is that expensive Datadog destinations receive only the data that justifies their cost.
Can Observability Pipelines help with data privacy and redaction?
Yes. Pipelines can redact sensitive data (PII, credentials, tokens) at the pipeline layer before log events reach Datadog. This means sensitive data never enters the platform's storage and search layer, which simplifies compliance and reduces the risk of sensitive data being accessible through Datadog's query interface.
Do Observability Pipelines work with non-Datadog destinations?
Yes. Datadog Observability Pipelines support multiple destinations including S3, Azure Blob Storage, Elasticsearch, Splunk and others alongside Datadog itself. This lets teams route different log streams to different destinations based on content: operational logs to Datadog for active monitoring, compliance logs to archive storage for retention and security logs to a SIEM.
Control telemetry cost at the source
We design and operate Datadog Observability Pipelines as part of your cost governance strategy: routing, reducing and transforming telemetry before it reaches expensive destinations.