Chaos engineering

TL;DR

• Chaos engineering intentionally injects controlled failures into production systems to discover weaknesses before real outages do. It applies the scientific method to reliability.
• Every experiment starts with a steady-state hypothesis: measurable conditions that define "the system is healthy." Without this, you're just breaking things.
• Blast radius control (canary chaos, automated halt conditions, incremental scope) makes production experiments safe, not reckless.
• The failure injection taxonomy spans six layers: infrastructure, network, application, dependency, resource, and data. Start small and work up.
• Organizations progress through a chaos maturity model: manual one-off experiments, then automated scheduled runs, then continuous chaos integrated into CI/CD pipelines.

The Problem It Solves

Imagine a microservices platform with 40 services. Every service passes CI/CD, staging is green, and code coverage is high. The team is confident.

Then a Redis cluster node gets terminated during routine maintenance. Failover takes 8 seconds. During those 8 seconds, 12 services that cache session data in Redis start throwing connection errors. Five have circuit breakers, but two have timeouts set to 30 seconds (above the 8-second window), so the breakers never trip. Three services have no circuit breakers at all.

Those three services retry aggressively: 5 retries, 100ms backoff, 200 RPS each. That's 3,000 extra Redis connections per second during recovery. Redis recovers at second 8, but the retry storms keep it saturated until second 22.

Meanwhile, two session-dependent services fall back to the primary PostgreSQL database. The database absorbs an extra 4,000 reads/second for 6 seconds, then starts timing out under connection pool exhaustion. Now all 40 services lose their primary data store.

In this illustrative scenario, an 8-second Redis failover becomes a 47-minute cascading outage.

The postmortem identifies the root cause: untested failure modes. Every component was tested individually, but nobody tested what happens when Redis is unavailable for 8 seconds under representative load with the deployed retry configurations. Unit tests do not cover cross-service failure propagation, integration tests may use mocked dependencies, and staging traffic may differ substantially from production.

The lesson is that failures often come from interactions between components when one degrades unexpectedly. Traditional testing covers the happy path and known failure paths, but may miss emergent behavior under realistic load.

What Is It?

Chaos engineering is the discipline of experimenting on a distributed system to build confidence that it can withstand turbulent conditions in production. It applies the scientific method to reliability: form a hypothesis, run an experiment, observe the results, learn from the outcome.

Think of it like a fire drill. A fire drill doesn't prevent fires. It verifies that when a fire happens, people know where the exits are, the alarms actually work, and the sprinkler system activates. Nobody questions the value of fire drills even though they disrupt the building for 20 minutes. Chaos engineering runs fire drills for your infrastructure.

Netflix's Chaos Monkey helped popularize the practice by terminating production instances. The hypothesis was simple: if a service cannot survive a controlled instance termination, the architecture has a resilience gap. Passing that experiment proves only that one failure mode was handled under the tested conditions.

The lifecycle is iterative. Each experiment generates findings that inform the next hypothesis. Over time, the system's resilience grows not because of any single experiment, but because the practice continuously surfaces and eliminates hidden failure modes.

In an interview, describe chaos engineering as "controlled failure injection with a steady-state hypothesis." That distinguishes it from an unmeasured exercise that merely breaks a server.

How It Works

Running a chaos experiment follows a deliberate protocol. Here is a walkthrough for a single experiment: testing whether a checkout service survives a payment-provider outage.

1. Define the steady-state hypothesis. Before touching anything, define what "healthy" looks like in measurable terms: "When the payment provider is unreachable, the system queues payments for retry, returns a 202 Accepted to the user within 500ms, and processes the queued payment within 5 minutes of provider recovery."

2. Capture the baseline. Record current SLIs: error rate, p99 latency, queue depth, successful payment rate. This is your control group.

3. Configure the fault injection. Specify the fault type (dependency failure), target (payment provider API), scope (5% of traffic), duration (60 seconds), and halt conditions (abort if error rate exceeds 10%).

4. Inject the failure. The chaos tool blocks outbound requests to the payment provider for the targeted traffic cohort. Only 5% of checkout requests are affected.

5. Observe. Compare real-time SLIs against the baseline. Does the error rate stay under 5%? Does the payment queue grow as expected? Is p99 latency within bounds?

6. Halt or conclude. If halt conditions trigger, the experiment stops automatically and the team investigates. If the experiment runs its full duration, compare final metrics against the hypothesis.

7. Learn and share. Document what happened. Did the system meet the hypothesis? If not, what failed? File tickets for discovered weaknesses. Share findings in a team retro.

# Chaos experiment specification (Gremlin / Litmus style)
experiment:
  name: "payment-provider-outage"
  hypothesis:
    description: "Checkout survives payment provider unavailability"
    conditions:
      - metric: "checkout.error_rate"
        operator: "<"
        threshold: 5.0
      - metric: "checkout.p99_latency_ms"
        operator: "<"
        threshold: 500
      - metric: "payment.queue_depth"
        operator: ">"
        threshold: 0  # Queue should be growing

  fault:
    type: "dependency-failure"
    target: "payment-provider.external.svc"
    method: "block-outbound"
    scope:
      percentage: 5       # 5% of traffic
      duration: "60s"

  halt_conditions:
    - metric: "checkout.error_rate"
      operator: ">"
      threshold: 10.0     # Abort if error rate exceeds 10%
    - metric: "system.p99_latency_ms"
      operator: ">"
      threshold: 2000     # Abort if p99 exceeds 2 seconds

  schedule:
    window: "Tuesday 14:00-16:00 UTC"  # Off-peak
    notify: ["#sre-chaos", "@oncall"]

Without a steady-state hypothesis before the experiment, the team is just breaking things. The hypothesis defines "recovery" and "acceptable degradation" in measurable terms before the fault is injected. A vague experiment such as "let's kill a pod and see what happens" leaves the team without a baseline or a clear result to compare against.

Key Components

Types / Variations

Failure Injection Taxonomy

The six layers of failure injection, ordered from lowest blast radius to highest: