In October 2025, the largest Amazon Web Services (AWS) region in the world suffered an outage lasting 15 hours, which created a global impact as thousands of sites and apps crashed or degraded – including Amazon.com, Signal, Snapchat, and others.

AWS released an incident summary three days later, revealing the outage in us-east-1 was started by a failure inside DynamoDB’s DNS system, which then spread to Amazon EC2 and to AWS’s Network Load Balancer. The incident summary overlooked questions such as why it took so long to resolve, and some media coverage sought to fill the gap.

The Register claimed that an “Amazon brain drain finally sent AWS down the spout”, because some AWS staff who knew the systems inside out had quit the company, and their institutional knowledge was sorely missed.

For more clarity and detail, the « Pragmatic Engineer newsletter » went to an internal source at Amazon : Senior Principal Engineer, Gavin McCullagh, who was part of the crew which resolved this outage from start to finish. In this article, Gavin shares his insider perspective and some new details about what happened, and we find out how incident response works at the company.

Incident Response team at AWS according to senior principal Engineer Gavin McCullagh.

Amazon is made up of many different businesses of which the best known are :

1) Amazon Retail : operates the shopping websites Amazon.com, and 21 regional versions including Amazon.co.uk, Amazon.de, and others.

2) AWS : everything to do with Amazon Web Services. Retail is a major customer of AWS.

These organizations operate pretty much independently with their own processes and set ups. Processes are similar but not identical, and both groups evolve how they work over time, separately. In this deepdive, our sole focus is AWS, but it could be assumed Retail has some similar functions, like separate Incident Response teams.

Regions and AZs : AWS is made up of 38 cloud regions. Each one consists of at least 3 Availability Zones (AZs), which are at least one independent data center, connected via a low-latency network.

Note : “Region” and “AZ” mean slightly different things among cloud providers, as covered in Three cloud providers, three outages, three different responses.

Incident Response is a dedicated team at AWS, staffed by experienced support and infrastructure engineers, who do the following :

A) Monitor all 38 AWS regions, continuously. Health KPIs (Key Performance Indicators) are set up, so it’s easy to tell if a service is healthy or not. These KPIs vary by service, but typical ones include monitoring API success rates and response latencies, as well as synthetic canaries.

B) Get alerted when a region becomes unhealthy, or KPIs suggest it might be.

C) Coordinate incident calls when an outage happens within a region or AZ.

D) Oncall rotation 24/7 : someone is always available to respond to an alert.

All AWS Service teams and products run their own independent oncall teams. The AWS Incident Response group is a “safety net” which coordinates large-scale events.

AWS has a “you build it, you run it” mentality : teams have autonomy to decide which service they build, the technologies used, and how they structure it. In return, they are accountable for the service meeting the uptime target ; for production services, this means having an active oncall.

19 Oct, 11 :48 PM : the incident begins and regional health indicators degrade, triggering an alert. Minutes later, AWS Incident Response oncall is paged. The AWS call leader also joins the call. Gavin joins despite not actually being oncall because he figures he might be able to help – which turned out to be a good hunch.

The Incident Response team begins by going through a checklist in roughly this order :

1) Investigate whether there are power failures or significant network failures

2) Check if the software-defined network and load balancing layers are healthy

3) Look for issues within DNS and authentication/ authorization services

4) Confirm that core services are healthy which most AWS products build on ; like KMS, DynamoDB and S3.

That check list permit them to solve the problem. And that problem was : a failure inside DynamoDB’s DNS system, which then spread to Amazon EC2 and to AWS’s Network Load Balancer.