Amazon’s Ohio datacenter experienced a significant outage today.  Amazon reported that “some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power”.

This was likely a non-event for big users because they’d have multiple availability zones, etc. and would probably just detect the failure and carry on.  But it’s interesting to see that 38 different services was affected – everything from deployment tools to EC2 images to elastic containers to their blockchain DB (Quantum Ledger).

The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.

One sentence I found interesting was:

While the vast majority of Lambda functions continue operating normally, some functions are experiencing invocation failures and latencies, but we expect this to improve over the next 30 minutes.

This doesn’t mean your Lambda functions were failing, even if you selected US-East as your region.  According to Amazon:

Lambda runs your function in multiple Availability Zones to ensure that it is available to process events in case of a service interruption in a single zone.

Overall, I suspect a lot of people running in US-East are going to have meetings over the next 24 hours to discuss parts of their infrastructure that they thought were HA and they weren’t.

Oddly, this kind of outage is probably going to increase AWS sales.  They’ve got legions of locked-in users who will now pony up for more HA.

• Author
• Recent Posts

raindog308

Dread Lord of LowEnd Content at LowEndBox

I'm raindog308, techno polymath and long-time LowEndTalk community Administrator. My technical interests include all things Unix, perl, python, golang, shell scripting, vintage operating systems such as MVS, and relational database systems such as Oracle, PostgreSQL, and MySQL.

I also enjoy using LEBs! I have an empire of little guys for VPNs, larger guys for apps, and big guys for databases and storage. When I'm not in front a screen I'm into German Shepherd dogs, quality knives, target shooting, theology, tabletop roleplaying games, and forest hiking.

I enjoy writing technical articles here on LowEndBox to help people get more out of their systems. You can find me on LowEndTalk @raindog308

Latest posts by raindog308 (see all)

• The Best Felware Offer Yet!Cheap, Awesome Dedicated Servers in Madison, Wisconsin - January 25, 2025
• Bonus Code Friday!Can You Match The Legendary Pac-Man High Score on the Arcade You Could Win from RackNerd? - January 24, 2025
• We Called It Wrong: Ross Ulbricht, Who Masterminded Multiple Attempted Murders, Has Been Pardoned - January 22, 2025