DevSolved Logo DevSolved
Search... ⌘K
Write Log In Sign Up
Community Feed

Join DevSolved

Share war stories & investigate outages.

Sign Up for Free Log In to Account
ESC
↵ Search all
ESC Close
Pro tip: Type at least 2 characters
CRITICAL Resolved

DNS Propagation Nightmare: The Global Routing Blackhole

aws
Avatar Devsolved Jan 10, 2024 — 08:00 UTC MTTR: 24 hours 2 min read 250000 views
Executive Summary

A seemingly innocuous update to our authoritative nameservers on Route53 propagated incorrectly due to a stale glue record at the registrar level, causing 40% of global users to see a "Site Not Found" error for 24 hours.

Availability Impact: 40% of global internet traffic was blackholed. Revenue loss estimated at $120,000.

The Symptom

Symptom

At 09:00 UTC, we migrated our DNS hosting from Cloudflare to AWS Route53. We updated the NS records at our domain registrar. By 11:00 UTC, our monitoring indicated a massive drop in traffic. Users on Twitter reported that the domain simply did not exist. However, our internal ping tests and Dig commands from our office network resolved the domain perfectly. The outage was heavily localized to specific ISPs globally.

Timeline

  • 1
    09:00 update

  • 2
    09:15 update

  • 3
    11:00 update

  • 4
    12:30 update

  • 5
    14:00 update

Root Cause Analysis

Root Cause

When migrating DNS, updating the NS records in the zone file is not enough. The Top Level Domain (TLD) servers (.com) need to know which IP addresses correspond to those nameservers. These are called "Glue Records". We updated the NS records in the Route53 zone, and we updated the registrar's nameserver list. However, because we previously used custom vanity nameservers (ns1.ourdomain.com), the registrar had hidden glue records caching the OLD Cloudflare IP addresses for those nameservers. When an ISP recursively resolved our domain, it hit the .com TLD, which handed back the old IPs. The ISP then tried to ask the old IPs (which Cloudflare had already deactivated), resulting in an NXDOMAIN (domain does not exist) error.

DNS is always the culprit, even when it looks like a network layer issue. Never underestimate the power of stale glue records.

5 Whys Root Cause Drill-Down

1
Why #1
2
Why #2
3
Why #3
4
Why #4
5
Why #5

Resolution

Resolution

We had to log back into our domain registrar, navigate to the advanced "Registered Name Servers" panel, and manually delete the IP address mappings for `ns1.ourdomain.com`. Once deleted, the TLD servers stopped handing out the bad glue records. Unfortunately, because TLD records often have a 48-hour TTL, we had to wait for ISP caches globally to expire. We mitigated the immediate impact by temporarily reactivating the Cloudflare zone to catch the stale traffic.
Code bash 
// How to verify glue records via the terminal
dig +trace devsolved.com
// Look for the "Received [ip] from [TLD server] in [ms]" line. If that IP is wrong, it's a glue record issue.

Preventive Action Items

critical Document complete DNS migration runbook including Glue Record verification @DevOps
high Implement synthetic global DNS monitoring using ThousandEyes @SRE Team
|

Discussions 0

Most recent