A glitch with US-based cloud computing services provider Fastly, which forced major websites such as Spotify, Reddit, the Guardian and the New York Times offline earlier this week, was caused by a single customer.
Fastly is a content-delivery network (CDN) and, alongside Cloudflare and Akamai, is one of the largest providers of internet services.
In a blog post published yesterday Nick Rockwell, Fastly's senior vice president of engineering and infrastructure revealed the global outage was due to an undiscovered bug, triggered by a customer making a valid change to their account.
The bug, which could only be triggered by a specific configuration under specific circumstances, was actually introduced to the system on May 12 with a software update but wasn't triggered until June 8 in the US.
That caused 85 percent of Fastly's network to return errors, forcing websites around the globe offline.
"We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration," Rockwell wrote.
"Within 49 minutes, 95 percent of our network was operating as normal. This outage was broad and severe, and we're truly sorry for the impact to our customers and everyone who relies on them."
Even if you've never encountered the name Fastly, chances are you've interacted with it unknowingly - a common service is operating servers around the world which enables local caching of content so it can be delivered to devices quicker than by going to the source.
"Serving it really close to the user takes away a lot of the load time, and it enables everyone to have a really great experience when they're surfing the web," Angelique Medina, director of product marketing at the network monitoring firm ThousandEyes, told Wired.
A similar thing happened in 2019 when Cloudflare deployed "bad software", forcing sites like Medium, Buzzfeed, Discord and Pinterest offline.
