How connection draining prevents dropped requests

Why connection draining is necessary

During a deployment or scale-down, a backend can be stopping while a load balancer still believes it is healthy. New requests routed during that window fail with connection resets or 502 Bad Gateway, while requests already in flight may be abandoned. A health check alone only detects the problem after a polling interval; it does not coordinate a planned shutdown.

Scope and assumptions

This article covers HTTP services behind a load balancer, including Kubernetes ingress or service routing, and then extends the model to HTTP/2, WebSockets, gRPC streams, and SSE. The timing values are illustrative. Set drain and termination timeouts from measured request and stream lifetimes, health-check behavior, and the deployment controller's parallelism.

30-second mental model

Connection draining is a three-state handoff:

1. Deregister: mark the old backend unready and remove it from new-request routing.
2. Drain: stop accepting new work, but let existing requests or streams finish.
3. Terminate: after the work completes or a bounded deadline expires, close remaining connections and exit.

The critical ordering is deregister → drain → terminate. Readiness, load-balancer deregistration, application signal handling, and client reconnect behavior must agree on that ordering.

The design has four layers:

1. The race condition: Why naive server shutdown causes dropped requests
2. The draining lifecycle: Stop accepting new connections, finish in-flight requests, timeout, force close
3. Load balancer draining: How ALB deregistration delay, Nginx upstream, and HAProxy drain mode work
4. Kubernetes graceful shutdown: preStop hooks, SIGTERM handling, terminationGracePeriodSeconds, and readiness probe coordination

The Architecture

Five-minute end-to-end flow

Here is what happens during a healthy rolling update with connection draining:

1. The orchestrator decides to terminate Server v1 (OLD). Before killing it, the server is removed from the load balancer's rotation.
2. The load balancer stops sending new requests to Server v1. But the 12 requests that are already being processed continue to completion.
3. Meanwhile, Server v2 (NEW) starts up, passes health checks, and joins the LB rotation.
4. Once Server v1 finishes all in-flight requests (or the drain timeout expires), the process shuts down cleanly.

The problem happens when step 1 and step 4 are collapsed into a single "kill the process" command. The server dies while those 12 requests are mid-flight, and each one gets a 502.

The Race Condition That Causes 502s

This is the core of the problem: the load balancer's view can be stale while the process is already exiting.

The failure sequence is:

1. T=0s: The orchestrator sends SIGTERM to the old server process.
2. T=0.01s: The process exits immediately (no signal handler, default behavior).
3. T=0.5s: A new client request arrives at the load balancer.
4. T=0.5s: The LB's health check has not run yet (it runs every 5-10s), so the LB still thinks the old server is healthy.
5. T=0.5s: The LB forwards the request to a dead process. Connection refused. 502.

The root cause is a timing gap: the server is dead, but the load balancer does not know yet. Health checks are periodic, not instant. There is always a window where the LB's view of the world is stale.

The Draining Lifecycle

Connection draining solves this by inverting the sequence. Instead of "kill then deregister," you "deregister then drain then kill."

The four phases of connection draining:

Phase 1: Signal and deregister (0-2 seconds) The server receives a shutdown signal. It immediately starts failing health checks (returns 503 on the health endpoint). The load balancer detects the failure and removes the server from rotation. No more new requests arrive.

Phase 2: Drain in-flight requests (2-30 seconds) The server continues processing all requests that are already in progress. A typical HTTP request takes 50-200ms, so most requests finish within 1 second. But some requests are slow: large file uploads, long database queries, or SSE streams.

Phase 3: Timeout enforcement (at drain timeout) If requests are still running after the drain timeout (typically 30 seconds), the server force-closes them. This is a safety net. You do not want a single stuck request to block deployments indefinitely.

Phase 4: Process termination The server process exits with code 0. The orchestrator confirms the instance is gone and proceeds with the next server in the rolling update.

Deep Dive 1: Load Balancer Draining Mechanisms

Different load balancers implement draining differently, and the details matter.

How ALB deregistration delay actually works:

When you deregister a target from an ALB target group, the ALB enters a draining state for that target:

1. The ALB stops sending new requests to the target.
2. Existing connections continue until they complete naturally or the deregistration delay expires.
3. If the target has sticky sessions enabled, the ALB redirects sticky session requests to other healthy targets.
4. After the delay, any remaining connections are force-closed.

The key insight: the ALB tracks connection count at the target level. This is not a guess or a fixed sleep. It is active monitoring of real connections.

Deep Dive 2: Kubernetes Graceful Shutdown

Kubernetes has the most complex draining mechanism because it coordinates multiple systems: the kubelet, the API server, kube-proxy/iptables, and the application itself.

There are two critical race conditions in Kubernetes pod termination:

Race condition 1: Endpoint removal vs SIGTERM

When a pod is deleted, the API server sends two signals in parallel:

• It tells the kubelet to terminate the pod (which sends SIGTERM).
• It removes the pod from the Endpoints object (which tells kube-proxy to update iptables rules).

These happen asynchronously. The SIGTERM might arrive before kube-proxy has updated the iptables rules. If the app shuts down immediately on SIGTERM, traffic is still being routed to it via the old iptables rules.

The fix: add a preStop hook with a small sleep.

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

This 5-second sleep gives kube-proxy enough time to update iptables rules before the app starts its shutdown sequence.

Race condition 2: Readiness probe timing

Even with a preStop hook, external load balancers (like an ALB using target groups with pod IPs) have their own health check interval. If the ALB's health check runs every 10 seconds, it might keep routing traffic to the pod for up to 10 seconds after the pod stops accepting connections.

The fix: set the readiness probe to fail as soon as the pod starts shutting down.

terminationGracePeriodSeconds: 45
containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "5"]
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 2
      failureThreshold: 1

A properly instrumented application handles SIGTERM like this:

import signal
import sys

shutting_down = False

def handle_sigterm(signum, frame):
    global shutting_down
    shutting_down = True
    server.stop_accepting()
    server.wait_for_drain(timeout=30)
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)

@app.route('/healthz')
def health():
    if shutting_down:
        return '', 503
    return '', 200

Deep Dive 3: Long-Lived Connection Draining

HTTP request/response cycles are the easy case. Each request takes 50-500ms, and draining a server with only HTTP traffic is straightforward. The hard case is long-lived connections.

gRPC streaming connections have similar challenges. A gRPC server can send a GOAWAY frame to signal that clients should open new connections. The gRPC client library handles this automatically.

// gRPC graceful shutdown in Go
server.GracefulStop()  // sends GOAWAY, waits for streams to finish
// If GracefulStop takes too long, force it:
time.AfterFunc(30*time.Second, func() { server.Stop() })

Server-Sent Events (SSE) are the simplest long-lived case. The server can send a custom event telling the client to reconnect, and the EventSource API has built-in reconnection logic with configurable retry intervals.

Bottlenecks and failure modes

1. Health check propagation delay: Even after your server starts failing health checks, the LB needs at least one check cycle to detect it. With a 10-second interval and 2 failure threshold, that is 20 seconds of stale routing. Some teams lower the health check interval during deployments, but that adds load to the health check system.

2. Connection reuse and keep-alive: HTTP/2 multiplexes many requests over one TCP connection. "Draining connections" does not mean "draining requests." A single HTTP/2 connection might have 100 concurrent streams. The server needs to stop accepting new streams on existing connections, not just stop accepting new connections.

3. Sticky sessions break draining: If the LB uses cookie-based session stickiness, it might keep routing requests to a draining server because the cookie says so. The LB must override stickiness for draining targets, which ALB does natively but Nginx requires explicit configuration.

4. Database connection pools in the app: When your app shuts down, it also needs to drain its connection pool to the database. Closing a database connection while a transaction is in progress causes that transaction to roll back. The app must wait for all active transactions to commit or roll back before closing the pool.

5. Cascading drain during scale-down: If you are scaling from 10 servers to 5, and you start draining all 5 at once, the remaining 5 servers suddenly get double the load. Always drain one at a time, or limit parallel drain to a fraction of the fleet (Kubernetes maxUnavailable controls this).

Common mistakes

Practical checklist

• Mark the backend unready before terminating the process, and verify that the load balancer and service-discovery layer honor the state.
• Stop new requests and new HTTP/2 streams while allowing in-flight work to finish.
• Install a real SIGTERM or equivalent shutdown handler; do not assume the runtime will drain application work automatically.
• Set the deregistration and termination deadlines from measured P99 request duration plus a safety margin, with a hard upper bound.
• Ensure preStop, readiness propagation, load-balancer deregistration, and terminationGracePeriodSeconds leave enough overlap for the handoff.
• Drain database pools, queues, background workers, and other downstream resources after request admission stops.
• Give WebSocket, gRPC, and SSE clients an application-level reconnect or resumption protocol with jittered backoff.
• Limit the number of targets draining at once and monitor 502s, connection resets, in-flight count, drain duration, forced closes, and capacity on the remaining fleet.

Test Your Understanding

Quick Recap

1. 502 errors during deployment happen because the load balancer routes requests to servers that have already shut down, before health checks detect the failure.
2. Connection draining inverts the shutdown sequence: deregister from LB, finish in-flight requests, enforce timeout, then terminate.
3. AWS ALB uses "deregistration delay" (default 300 seconds, lower it to 30-60 for typical services).
4. Kubernetes requires a preStop hook (sleep 5 seconds) to handle the race condition between SIGTERM and endpoint removal via kube-proxy.
5. Long-lived connections (WebSockets, gRPC streams) need application-level drain signaling because HTTP-level draining does not help.
6. Drain timeout should be P99 request duration plus a safety margin, not an arbitrary large number.
7. Rolling updates should drain one server at a time (maxUnavailable=1) to avoid overloading remaining servers.
8. Connection draining is proactive (planned shutdown), while circuit breaking is reactive (server misbehaving).

Related Concepts

• Rolling updates and blue-green deployments use connection draining as the mechanism that prevents errors during the transition between old and new versions.
• Circuit breakers complement draining by handling the case where a server becomes unhealthy unexpectedly, rather than being intentionally removed.
• Health checks and readiness probes are the signaling mechanism that load balancers use to detect draining servers.
• Kubernetes pod lifecycle (preStop hooks, SIGTERM, terminationGracePeriodSeconds) is the orchestration layer that coordinates draining across containers, sidecars, and the infrastructure.
• Service mesh drain semantics in Envoy/Istio use the same principles but implement them at the sidecar proxy level, draining the Envoy connection pool separately from the application.