An accidental typo by an Amazon employee took down a large swath of the internet on Feb. 28. Some websites reported extremely slow load times, while others failed to load completely.
Service delays and outages were reported across a broad spectrum of web-based companies, such as online payments app Venmo, retailer Target, the U.S. Securities and Exchange Commission, and—somewhat ironically—Down Detector, a website that tracks internet service outages.
It turned out Amazon Web Services (AWS), the widely used cloud computing and hosting arm of internet giant Amazon.com Inc., was the reason for the problem.
The service disruption was a rude awakening for companies embracing the recent trend of moving information technology (IT) infrastructure to the cloud. It shows the cloud has many benefits, but also risks.
A lengthy and technical mea culpa published by Amazon on March 2 says the error was caused by technicians aiming to fix an issue within the billing system of AWS’s Simple Storage Service (S3).
Engineers needed to take down a small number of servers, but a wrong command was accidentally entered that took down far more—some of which were critical to the infrastructure of major websites.
So a mere typo caused major websites to crash.
The error and service outage affected AWS’s Virginia operations, which host data centers for many companies located in the eastern United States. The issue lasted around four hours from initial failure to full recovery on Feb. 28.
Lost Retail Sales
There’s no concrete data from which to estimate revenues lost due to the outage, but the amount is likely in the hundreds of millions of dollars.
Cyence, a startup that tracks the economic impact of cyber risks and attacks, estimates the outage cost Standard & Poor’s 500 companies $150 million in lost revenues, according to a Wall Street Journal report. It was unclear what data sources Cyence used to estimate the impact.
But numerous e-commerce retailers experienced significant slowdowns and outages. More than half of the major U.S. e-retailers tracked by Apica Systems, a provider that monitors cloud and mobile application performance, were impacted by the AWS outage.
Fifty-four out of the 100 e-retailers in Apica’s 100 Web Performance Cyber Monday Index were negatively affected, marked by a 20 percent or more decrease in website performance. Three sites—Lululemon, Express, and One Kings Lane—were completely unavailable. The biggest delays in load times were experienced by the Disney Store, with a whopping 1,165 percent delay, or 10 times the usual wait.
