AWS Cost Optimization: The $40,000 S3 Bandwidth Bill

Financial Impact: $42,500 in unexpected AWS Data Transfer charges over 7 days. MTTR to mitigate billing hemorrhage: 4 hours.

The Incident Trigger: Going Viral

Symptom

At 08:00 UTC on Tuesday, our new global marketing campaign launched. The frontend React application was highly interactive, featuring heavy 4K background videos and uncompressed high-resolution hero images. By Wednesday evening, our AWS Billing Dashboard triggered a critical anomaly alert: our Daily Spend had spiked by 5,000%.

AWS Billing Dashboard Anomaly Detection Spike

Timeline of the Cost Hemorrhage

Timeline

1

Tuesday 08:00 UTC monitoring

Campaign goes live globally, heavily promoted in the APAC region.

— Marketing
2

Wednesday 14:00 UTC investigating

Users in Asia report slow page load times (10+ seconds for the hero video).

— Users
3

Wednesday 18:30 UTC critical

Automated billing anomaly alert fires: $15,000 threshold breached.

— AWS Cost Explorer
4

Wednesday 19:15 UTC investigating

Identified that 98% of the cost is "Data Transfer Out to Internet" from the US-East-1 S3 bucket.

— SRE Team
5

Wednesday 21:00 UTC resolved

CloudFront distribution deployed, OAC enabled, and frontend URLs hotfixed.

— DevOps

Architecture Deep Dive: S3 vs CDN

Amazon S3 is incredibly durable (11 nines) for object storage, but it is fundamentally **not** a Content Delivery Network. When users request assets directly from S3, two critical architectural problems emerge:

High Latency (The Physics Problem): S3 buckets are regional. A user in Tokyo requesting a 50MB video from us-east-1 (Virginia) must wait for packets to cross the Pacific Ocean, resulting in terrible Time-To-First-Byte (TTFB).
Massive Costs (The Billing Problem): AWS charges premium rates for "Data Transfer Out" to the internet directly from S3 (typically ~$0.09 per GB). If 10,000 users download a 50MB video, that's 500GB of egress. Viral traffic can easily push this into petabytes.

We assumed S3 was cheap storage. We completely misunderstood the difference between Storage Pricing and Network Egress Pricing.

The Root Cause Analysis

Root Cause

To meet a tight deadline, the frontend team bypassed the infrastructure review process. They made the S3 bucket public by disabling "Block Public Access" and hardcoded the raw S3 bucket URLs directly into the React codebase.

Code javascript

// The fatal flaw: Hardcoding the direct S3 URL in the React Component
export default function HeroSection() {
  return (
    <video 
      src="https://production-marketing-assets.s3.us-east-1.amazonaws.com/hero-4k-bg.mp4" 
      autoPlay loop muted 
    />
  );
}

Because the campaign went viral in Asia, petabytes of data were pulled directly from the US-East-1 region across the public internet, completely bypassing edge caching and racking up astronomical egress fees.

5 Whys Analysis

5 Whys Root Cause Drill-Down

Why did the AWS bill spike to $42,500?

We incurred massive "Data Transfer Out" fees from our marketing S3 bucket.

Why was there so much Data Transfer Out?

Global users were downloading heavy 4K videos directly from the S3 bucket.

Why were users downloading directly from S3?

The frontend code used raw S3 URLs instead of routing traffic through a CDN (CloudFront).

Why wasn't CloudFront used?

The frontend team lacked AWS infrastructure knowledge and made the bucket public to "get it working quickly".

Why was the team able to make the bucket public?

We did not have Organization-wide SCPs (Service Control Policies) enforcing "Block Public Access" at the account level.

The Resolution: CloudFront + OAC

Resolution

The SRE team immediately placed Amazon CloudFront in front of the S3 bucket. CloudFront caches heavy assets at hundreds of edge locations (PoPs) worldwide.

Data transfer from S3 to CloudFront is $0.00. By serving the assets from the edge, latency dropped by 80%, and bandwidth costs plummeted.

To secure the bucket and prevent anyone from bypassing the CDN, we implemented Origin Access Control (OAC), which strictly limits S3 access to only our specific CloudFront distribution:

Code json

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "cloudfront.amazonaws.com"
    },
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::production-marketing-assets/*",
    "Condition": {
      "StringEquals": {
        "AWS:SourceArn": "arn:aws:cloudfront::123456789012:distribution/EDFDVBD632BHDS5"
      }
    }
  }
}

Finally, the frontend code was updated to use the custom CDN domain with aggressive Cache-Control headers.

Code javascript

// The fix: Using the custom CDN domain
export default function HeroSection() {
  return (
    <video 
      src="https://cdn.devsolved.com/hero-4k-bg.mp4" 
      autoPlay loop muted 
    />
  );
}

Post-Incident Action Items

Preventive Action Items

critical Deploy CloudFront distributions for all existing static asset buckets @DevOps Team

high Enforce "Block Public Access" globally via AWS Organizations SCPs @Security Team

medium Implement aggressive Cache-Control headers (max-age=31536000) for immutable assets @Frontend Team

medium Set up AWS Budgets with lower thresholds (e.g. $100/day anomaly detection) @FinOps Team

Join DevSolved