Search... ⌘K
Write Log In Sign Up

Join DevSolved

Share war stories & investigate outages.

Sign Up for Free Log In to Account
ESC
CRITICAL Resolved

Next.js API Rate Limiting: The Vercel Timeout Bill

Avatar Devsolved May 18, 2024 — 03:00 UTC MTTR: 5 hours 2 min read 290000 views
Executive Summary

A malicious botnet bypassed our CDN rate limiting by targeting highly dynamic, uncacheable search API routes. This exhausted our serverless function execution limits, taking the site offline and generating a massive surprise bill.

Financial Impact: Serverless function execution timeouts resulted in an unexpected $4,500 overage charge from Vercel in 6 hours.

The Symptom

Symptom

At 3:00 AM, PagerDuty alerted us that the production website was returning HTTP 429 Too Many Requests globally. When we logged into the Vercel dashboard, our account had been temporarily suspended for exceeding our Enterprise bandwidth and Serverless Function execution limits by 4000%.

Root Cause Analysis

Root Cause

We had configured Cloudflare rate limiting on our root domain to block IPs making more than 100 requests per minute. However, an attacker realized our `/api/search?q=[term]` endpoint in Next.js (which queries an external ElasticSearch cluster) was both highly uncacheable and very slow (taking ~2 seconds per request). The attacker used a distributed botnet of 50,000 unique IP addresses to send 1 request per minute to the search API. This completely bypassed Cloudflare's per-IP rate limit. The Vercel serverless functions spun up thousands of instances, each staying alive for 2 seconds. This exhausted our concurrent execution limits, effectively causing a Denial of Service (DoS) for legitimate users, while simultaneously racking up massive per-millisecond billing charges.

5 Whys Root Cause Drill-Down

1
Why #1
2
Why #2
3
Why #3
4
Why #4
5
Why #5

Resolution

Resolution

We implemented an emergency Application Firewall (WAF) rule in Cloudflare to challenge all requests to `/api/search` with a Turnstile Captcha. This instantly dropped bot traffic to zero. Long term, we implemented Upstash (Redis) global rate limiting inside the Next.js edge middleware. By running the rate limit check at the Edge (which executes in 5ms), we blocked the bad requests before they ever triggered the expensive, 2-second serverless function.
Code javascript
// The fix: Edge Middleware Rate Limiting with Upstash
import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";

const ratelimit = new Ratelimit({ redis: Redis.fromEnv(), limiter: Ratelimit.slidingWindow(10, "10 s") });

export async function middleware(request) {
  const ip = request.ip ?? "127.0.0.1";
  const { success } = await ratelimit.limit(ip);
  if (!success) return new Response("Rate limit exceeded", { status: 429 });
}

Preventive Action Items

critical Implement Edge Rate Limiting on all uncacheable API routes @Frontend Team
high Configure Vercel Spend Limits to hard-cap billing and prevent surprise overages @DevOps
|

Discussions 0

Most recent