Soon, the company behind this week’s major global Internet outage said on Wednesday that the incident was caused by a bug in its software, which was triggered when its customers changed settings.
Tuesday’s outage raised questions about the Internet’s reliance on a handful of infrastructure companies. Quickly publish obsolete high-traffic websites, including news providers such as The Guardian and The New York Times, as well as British government websites, Reddit and Amazon.
The company said in a blog post written by Nick Rockwell, its senior engineering and infrastructure executive officer: “This power outage is widespread and serious. It is causing serious damage to our customers and all those who depend on them. We apologize for the impact.”
He said that this problem should be expected.
Fastly operates a set of servers strategically distributed around the world to help customers quickly and safely move and store content near their end users.
The company’s post gave a timeline of the incident and promised to check and explain why Fastly failed to detect software errors during its own testing.
Fastly stated that the bug appeared in a software update sent to customers on May 12, but it was not triggered until an unidentified customer made a setting change that caused an issue that “caused 85% of our network to return errors” .
The power outage was quickly noticed within one minute of 0947 GMT (3:17 PM U.S. Standard Time), and the engineer found out the cause at 1027 GMT (3:57 PM U.S. Standard Time). Once they disabled the setting that caused the problem, most of the company’s network was quickly restored.
“In 49 minutes, 95% of our network was operating normally,” the company said.
Fastly said its network was fully restored at 1235 GMT (6:05 PM U.S. Standard Time), and a permanent software repair program began at 1725 GMT (10:55 PM U.S. Standard Time).
© Thomson Reuters 2021