AWS data retention policies: Set them and stop worrying about compliance

Data that you keep longer than necessary is both a cost and a liability. Under UK GDPR, keeping personal data beyond the period for which it was collected is non-compliant. Under PCI DSS, keeping cardholder data beyond the transaction processing window creates audit scope. Under FCA rules, keeping certain records shorter than the mandatory retention period is also non-compliant.

AWS provides four tools for managing data retention automatically: S3 Lifecycle rules for object storage, S3 Object Lock for immutable retention, CloudWatch Logs retention policies for log data, and Data Lifecycle Manager for EBS snapshots. This guide covers how to configure each.

S3 Lifecycle rules: automate the storage tier ladder

S3 Lifecycle rules transition objects between storage classes based on age and optionally delete them when the retention period expires. The rule applies to all objects in a bucket or to objects matching a specific prefix or tag filter.

The standard lifecycle pattern for compliance-related data:

  1. Days 0-90: S3 Standard or S3 Standard-IA - actively queryable, low retrieval cost
  2. Days 90-365: S3 Glacier Instant Retrieval - accessible within minutes, 90% cheaper than Standard
  3. Days 365+: S3 Glacier Deep Archive - accessible within hours, the cheapest storage available
  4. Expiry (end of mandatory retention period): Delete the object

Create a lifecycle rule in the S3 console:

{
  "Rules": [{
    "ID": "compliance-archive",
    "Filter": {"Prefix": "logs/"},
    "Status": "Enabled",
    "Transitions": [
      {
        "Days": 90,
        "StorageClass": "GLACIER_IR"
      },
      {
        "Days": 365,
        "StorageClass": "DEEP_ARCHIVE"
      }
    ],
    "Expiration": {
      "Days": 2555
    }
  }]
}

This rule moves logs/ prefix objects to Glacier Instant Retrieval after 90 days, to Glacier Deep Archive after one year, and deletes them after 7 years (2,555 days), satisfying FCA log retention requirements for most record categories.

Apply different rules to different prefixes based on the data category and its retention requirement. Application logs, database backups, and customer data each have different retention obligations.

S3 Object Lock: enforce immutable retention

S3 Object Lock prevents objects from being deleted or overwritten for a fixed period. It is the correct tool for data that must be retained and must not be tampered with: audit logs, financial records, regulatory compliance evidence.

Two retention modes:

Governance mode: Prevents deletion or modification except by principals with the s3:BypassGovernanceRetention IAM permission. Useful for environments where administrators need the ability to correct mistakes.

Compliance mode: No user, including the root account, can delete or modify locked objects until the retention period expires. Appropriate for regulatory records where the immutability requirement is absolute.

Enable Object Lock when creating the bucket (it cannot be enabled on an existing bucket):

aws s3api create-bucket \
  --bucket audit-logs-immutable \
  --region eu-west-2 \
  --create-bucket-configuration LocationConstraint=eu-west-2 \
  --object-lock-enabled-for-bucket

Set a default retention period on the bucket so all objects uploaded to it are automatically locked:

aws s3api put-object-lock-configuration \
  --bucket audit-logs-immutable \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Years": 7
      }
    }
  }'

Every object uploaded to this bucket is now locked in Compliance mode for 7 years. It cannot be deleted by anyone, including you, until the retention period expires.

CloudWatch Logs retention policies

CloudWatch Logs groups have no retention limit by default: logs accumulate indefinitely and you pay for storage indefinitely. This creates both unnecessary cost and potential GDPR liability (keeping personal data in application logs longer than necessary).

Set retention on all log groups. Find groups without a retention policy:

aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].logGroupName' \
  --output text

Set retention on all unmanaged log groups:

# Apply 90-day retention to all log groups without a policy
aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].logGroupName' \
  --output text | \
  tr '\t' '\n' | \
  xargs -I{} aws logs put-retention-policy \
    --log-group-name {} \
    --retention-in-days 90

Choose retention periods by log category: - Application logs: 30-90 days (incident investigation window) - Security logs (CloudTrail, VPC Flow Logs via CloudWatch): 90 days online; archive to S3 for longer retention - Access logs (ALB, API Gateway): 30-90 days online; archive if compliance requires longer - Debug/trace logs: 7-14 days (high volume, short utility window)

For logs that must be kept longer than 90 days for compliance, export to S3 using CloudWatch Logs subscription filters or export tasks, and apply S3 Lifecycle rules for longer-term archiving. CloudWatch Logs storage is significantly more expensive than S3 Glacier; do not keep compliance archives in CloudWatch.

Data Lifecycle Manager for EBS snapshots

EBS snapshots accumulate if not managed. A server decommissioned a year ago may still have daily snapshots in the account, each costing money for data that will never be needed.

Data Lifecycle Manager (DLM) creates and manages EBS snapshot schedules automatically. Navigate to EC2 > Lifecycle Manager > Create lifecycle policy.

For production volumes, a typical policy: - Daily snapshot at 02:00 UTC - Retain for 30 days - Weekly snapshot (Sunday) retained for 12 months - Monthly snapshot (first Sunday) retained for 7 years

DLM also supports cross-region snapshot copy for DR purposes and automatically applies consistent tags to snapshots for cost allocation tracking.

For development and staging volumes where the retention requirement is shorter: - Daily snapshot retained for 7 days - No weekly or monthly snapshot

Tag volumes with DLMPolicy: production or DLMPolicy: development and configure DLM policies filtered by tag. When a new volume is created and tagged, it automatically falls under the correct lifecycle policy.

AWS Config for continuous compliance monitoring

Configure AWS Config rules to detect data retention policy gaps:

  • s3-lifecycle-policy-check: alerts when S3 buckets do not have a lifecycle policy configured
  • cloudwatch-log-group-encrypted: ensures CloudWatch log groups are encrypted
  • Custom Config rule: alert when new S3 buckets are created without Object Lock (if your policy requires it for all new buckets containing regulated data)

Route Config non-compliance findings to an SNS topic that alerts the relevant team. New buckets without lifecycle rules should be addressed within days, not discovered months later.

Where Critical Cloud comes in

Data retention compliance is an intersection of legal obligation, cost management, and technical implementation. Getting the lifecycle rules, Object Lock configurations, and retention periods right across a complex AWS estate requires understanding both the regulatory requirements and the AWS tooling. We configure and operate data retention governance for regulated businesses as part of the managed platform, with Config rules enforcing compliance continuously. See how Critical Support works.