What we learned from the AWS S3 Outage

Read full article on TechCrunch. Per Amazon’s website: “Amazon Simple Storage Service (Amazon S3) is object storage with a simple web service interface to store and retrieve any amount of data from anywhere on the web. It is designed to deliver 99.999999999% durability, and scale past trillions of objects worldwide.” Unfortunately that durability had a lapse in uptime for 5 consecutive hours. Our team was notified within minutes of the outage and a recovery process, as the news spread across the web. Within 2 hours most of our web properties were up and running. Our JS APIs use S3 but run on a cache which protected us from the S3 outage. Unfortunately some of our assets like icons, and AdvancedInfowindows were impacted until Amazon resolved the issue. So what did we learn from this outage? 99.999999999% may be accurate over many years for uptime, but 5 consecutive hours in the middle of business can be crippling. As such, for our asset delivery processes, we now operate with 48 hour cache period and have implemented a nightly S3 asset backup process, so should S3 go down again, we can restore services within minutes.]]>