Concise Cyber

Subscribe below for free to get these delivered straight to your inbox

Advertisements
Anatomy of the Fastly Outage: A Dormant Bug and a Global Internet Disruption
Advertisements

On June 8, 2021, a widespread internet outage rendered many of the world’s most popular websites inaccessible. High-traffic destinations for news, e-commerce, streaming, and government services suddenly went dark. The origin of the disruption was traced to Fastly, a major content delivery network (CDN) that provides critical infrastructure for a large portion of the web.

The Global Impact of the Service Disruption

The outage began at approximately 9:47 UTC and affected services globally. Websites that went offline included major news outlets like The New York Times, CNN, and The Guardian, as well as platforms such as Reddit, Twitch, and even the UK government’s primary website, gov.uk. Fastly reported that the issue caused 85% of its network to return errors. For nearly an hour, users attempting to access these sites were met with connection failure messages, causing a significant and immediate global disruption.

Root Cause: A Single Customer Triggered a Latent Bug

The massive failure was not the result of a security breach or malicious attack. Instead, the root cause was a previously undiscovered bug in a software update Fastly had pushed to its network on May 12. This bug remained latent and harmless within the system for weeks. The outage was triggered when a single, unnamed customer made a valid change to their service configuration. This specific, legitimate action exposed the latent bug, which created an error condition that rapidly propagated across Fastly’s global points of presence (POPs), causing them to fail. Fastly’s engineers detected the issue within one minute of its onset and, after identifying the source, began rolling back the problematic configuration. Service restoration began at 10:36 UTC, and within 49 minutes, 95% of Fastly’s network was operating normally.

Source: https://biztoc.com/x/87490126248509ce