Post-mortem: AWS Kinesis cascade 2020

Takeaway

The November 2020 Kinesis event is a compact lesson in cascading failure: a capacity change exposed a resource-limit assumption, retries amplified the loss of capacity, and hidden dependencies degraded services that did not appear related to Kinesis. The durable response is to map dependencies, canary capacity changes, keep monitoring independent, and design recovery for backlog and retry load.

Scope and Evidence

5-Minute Incident Walkthrough

1. Trigger: A routine Kinesis front-end capacity addition interacted badly with a resource-limit assumption.
2. Failure: New front-end instances failed during initialization, reducing serving capacity.
3. Amplification: Traffic redistribution and client retries increased load on the remaining fleet.
4. Cascade: Services that used Kinesis internally—including monitoring and authentication paths—degraded as the dependency failed.
5. Recovery: Engineers used alternative probes, reduced the unstable fleet, and restored capacity gradually before dependent backlogs drained.

Causal chain: capacity addition → thread exhaustion → front-end crashes → traffic redistribution and retry amplification → dependent services degrade and monitoring loses visibility → staged recovery and backlog drain.

Incident Summary

Date: November 25, 2020 Duration: ~8 hours of significant customer impact Region: US-East-1 (Northern Virginia) Services affected: Kinesis, CloudWatch, Cognito, Lambda, API Gateway, EventBridge, ECS, Elastic Beanstalk, Personal Health Dashboard, and dozens more Root cause: Capacity addition to Kinesis front-end fleet triggered OS thread limit exhaustion, causing a cascading failure across all services that depend on Kinesis internally

This was not merely a "Kinesis outage." It is a useful demonstration of what happens when a foundational service fails and many other services have hidden dependencies on it.

The incident also exposed a fundamental assumption in AWS's architecture: that internal platform services (Kinesis, in this case) are always available. When that assumption broke, the blast radius was not proportional to Kinesis's importance. It was proportional to the number of services that had quietly built dependencies on it over the years.

For your interview preparation: this is the canonical example of a cascading failure caused by hidden dependencies. If an interviewer asks "tell me about a real-world cascading failure," this is the one to cite.

What Happened: The Timeline

Here is the minute-by-minute sequence of how a routine capacity operation became an 8-hour multi-service outage.

The total customer impact was roughly 8 hours. But the real story is in the cascade, not the timeline.

Several things stand out about this timeline. The time from trigger (07:44) to customer impact (08:15) was only 31 minutes. The time from first impact to root cause identification (08:15 to 11:00) was nearly 3 hours. And the time from identifying the root cause to full recovery (11:00 to 16:00) was 5 hours. This ratio (fast to break, slow to diagnose, even slower to fix) is characteristic of cascading failures.

The diagnosis delay was not because the engineers were slow. It was because their primary diagnostic tool (CloudWatch) was down. They were trying to fix a system while the system they use to understand system health was itself broken. This is the operational equivalent of trying to find a leak in a dark room when the flashlight batteries are dead.

This timeline distribution is something interviewers look for when you discuss incident response. Engineers who mention only "we found the bug and fixed it" miss the operational reality. A strong answer breaks the timeline into three phases: detection (how fast did you know something was wrong?), diagnosis (how did you find the root cause, and what slowed you down?), and recovery (why did the fix take so long even after you knew the cause?). This incident hits all three phases hard, which is what makes it such a useful reference.

Another subtlety: the 07:44 trigger was not a deployment. It was a capacity addition. In an interview setting, this matters because it demonstrates that operational changes (not just code changes) can cause production outages. Capacity additions, configuration changes, certificate rotations, DNS updates: these operational actions often lack the same safety nets (CI/CD pipelines, automated rollback) that code deployments have. When an interviewer asks "what kinds of changes can cause outages?", the answer should include operational changes, not just code pushes.

The Architecture Before the Incident

Kinesis is a streaming data service: producers push records, Kinesis stores them in shards, consumers pull records. The front-end fleet is the API layer that accepts all Kinesis API calls and routes them to the correct back-end shards.

The critical detail: those "producer" boxes at the top are not just customer workloads. CloudWatch, Cognito, and Lambda all use Kinesis internally to move data. This dependency is invisible to customers and, as we learned, partially invisible to AWS engineers operating those services.

"Hidden dependency" means something specific here. The Kinesis API documentation does not say "CloudWatch requires Kinesis." The CloudWatch documentation does not say "this service will degrade if Kinesis is unavailable." The dependency existed in the implementation but not in the service's operational model. This is the most dangerous kind of dependency: one that is architecturally real but operationally invisible.

In any large organization, these hidden dependencies accumulate over time. A team building CloudWatch's metrics ingestion pipeline five years ago chose Kinesis because it was the best tool for the job. That was a good technical decision. But no one updated the operational runbooks to say "if Kinesis goes down, CloudWatch will be impaired." Over the years, the dependency became infrastructure: invisible, assumed, and untested for failure.

Root Cause: Thread Limit Exhaustion During Capacity Addition

The Kinesis front-end fleet needed additional capacity. AWS operations added new servers to the fleet. This is a routine scaling operation that had been performed many times before.

The problem: the new servers were a larger instance type than the existing fleet. More CPUs per server.

Each front-end server initializes its routing cache at startup. The cache initialization code creates threads proportional to the number of CPU cores available. The formula roughly:

threads_created = num_cpus × threads_per_cpu × num_routing_entries

On the old instance type (fewer CPUs), this formula produced a thread count well within the Linux OS thread limit. On the new instance type (more CPUs), it produced a thread count that exceeded the OS ulimit for maximum threads per process.

To put numbers on this: if the old instance type had 16 CPUs and the initialization created 50 threads per CPU per routing entry, and there were 100 routing entries, that is 16 x 50 x 100 = 80,000 threads. The OS limit may have been set at 100,000, leaving comfortable headroom. The new instance type with 32 CPUs creates 32 x 50 x 100 = 160,000 threads, well past the same 100,000 limit. (These are illustrative numbers; AWS did not publish the exact figures.)

The result: new servers started, began initializing their cache, hit the thread limit, and crashed. But it was worse than that. The operating system thread limit exhaustion did not just affect the new servers. As the fleet tried to redistribute work, the cascading restarts and retry storms pushed existing servers past their thread limits too.

Within minutes, a significant portion of the Kinesis front-end fleet was down. The servers that remained healthy were now handling the load of the entire fleet, which pushed their thread counts higher from increased API handling. Some of those surviving servers crossed their own thread limits and went down too.

Here is the core failure sequence: