Load balancing

TL;DR

• A load balancer sits in front of your server pool and distributes incoming requests so no single instance bears all the traffic.
• It is one way to make horizontal scaling useful: adding servers requires a routing mechanism that can send traffic to them.
• Layer 4 (TCP/UDP) load balancers route using transport information; Layer 7 (HTTP) load balancers can inspect URLs, headers, or cookies and can terminate TLS. The right layer depends on the protocol and requirements.
• The algorithm matters: round-robin can fit homogeneous short requests, least-connections can fit long-lived sockets, and session affinity should be used only when its failure modes are acceptable.
• The load-balancer tier is part of the failure domain and needs high availability, health checks, and an operational plan.

The Problem It Solves

Suppose a service receives 50,000 requests at once while only one app server is running. The process reaches its CPU or connection limit.

The request queue fills. New connections get rejected with 503 Service Unavailable. Your on-call phone rings.

You start a second server. It is healthy, but the clients still resolve and connect to the first server, so the first server continues to receive the traffic.

The second server receives no traffic because there is no routing layer that knows about both servers.

That's the problem. Having more servers solves nothing if traffic has no mechanism to reach them.

The fix isn't more servers alone. The fix is a component that knows all your servers exist and can distribute traffic across all of them.

What Is It?

A load balancer is a reverse proxy or routing service that sits in front of a pool of servers and distributes incoming requests across them. It can monitor server health and route traffic only to instances that pass the configured health checks.

Analogy: Think of an airport departure terminal with 20 check-in counters. Without a dispatcher, every passenger walks to counter 1. Counter 1 is overwhelmed; counters 2–20 are empty.

With a dispatcher at the entrance directing passengers—“Counter 5 has the shortest queue, go there”—each counter can handle a share of the work. The dispatcher does not perform the check-in; its role is to route traffic.

The load balancer aims to distribute work across the pool and can hide an individual server failure when enough healthy capacity remains. A server going down does not have to take down the service; the balancer can stop routing to it. Stateless app servers plus a load balancer are a common foundation for a scalable app tier.

How It Works

Here is a simplified request flow for a load balancer:

1. DNS resolution — The client resolves api.yoursite.com to a single Virtual IP (VIP) address. The VIP is owned by the load balancer, not any backend server. This decoupling is what allows backend instances to be added, removed, or replaced without any DNS change.

2. Connection established — A Layer 4 design may forward or terminate the transport connection depending on the implementation. A Layer 7 design terminates enough of the connection to parse the HTTP request before making a routing decision.

3. Algorithm selects a backend — The load balancer runs its assignment algorithm to pick one healthy server from the pool. (Algorithms are covered in the next section.)

4. Health check gate — Before routing, and continuously during operation, the LB probes each backend. Only servers that pass health checks are eligible for traffic. A server that returns errors or fails to respond within the timeout is removed from rotation automatically.

5. Request forwarded — The request is proxied to the selected backend. For L7 balancers, headers are injected here: X-Forwarded-For: <client-ip>, X-Request-ID: <trace-id>.

6. Response returned — The backend responds to the LB; the LB returns the response to the original client. From the client's perspective, it's talking to one server. The load balancer is completely transparent.

7. Connection tracking — For long-lived connections such as WebSockets or gRPC streams, the LB keeps the established connection associated with the selected backend until it closes or the backend fails.

In this L7 example, the client connects to the VIP, TLS terminates at the LB, and the backend address is hidden from the client. Header parsing and proxying add latency, so measure the actual overhead for the chosen product and workload rather than assuming a fixed 1–3ms.

Here's what a minimal NGINX upstream config looks like in practice:

# nginx.conf — upstream pool with algorithm and health configuration
upstream api_servers {
  least_conn;  # Route to the instance with fewest active connections

  server app-server-1.internal:3000 weight=1 max_fails=3 fail_timeout=30s;
  server app-server-2.internal:3000 weight=1 max_fails=3 fail_timeout=30s;
  server app-server-3.internal:3000 weight=2 max_fails=3 fail_timeout=30s; # 2× capacity

  keepalive 32;  # Keep up to 32 idle upstream connections warm
}

server {
  listen 443 ssl;
  server_name api.yoursite.com;

  # SSL terminates here — backends get plain HTTP internally
  ssl_certificate     /etc/ssl/certs/api.crt;
  ssl_certificate_key /etc/ssl/private/api.key;

  location / {
    proxy_pass http://api_servers;
    proxy_set_header X-Forwarded-For $remote_addr;  # Pass real client IP
    proxy_set_header X-Request-ID    $request_id;   # For distributed tracing
    proxy_connect_timeout 5s;
    proxy_read_timeout    30s;
  }
}

Here's what the health check decision loop looks like on the LB side:

// Pseudocode — active health check loop per backend instance
async function healthCheckLoop(server: BackendServer): Promise<void> {
  while (true) {
    try {
      const res = await fetch(`http://${server.host}/health`, {
        signal: AbortSignal.timeout(2000), // 2s timeout
      });

      if (res.ok) {
        server.consecutiveFailures = 0;
        if (server.status === "unhealthy") {
          server.status = "healthy";
          addToRotation(server); // Re-admit after recovery
        }
      } else {
        server.consecutiveFailures++;
      }
    } catch {
      server.consecutiveFailures++;
    }

    // Remove from pool after 3 consecutive failures
    if (server.consecutiveFailures >= 3 && server.status === "healthy") {
      server.status = "unhealthy";
      removeFromRotation(server); // No traffic until next recovery check
    }

    await sleep(5000); // Re-probe every 5 seconds
  }
}

The state machine that drives this logic:

The two-step removal (DEGRADED → UNHEALTHY) gives a flaky instance a chance to recover before removal. A slow-start policy—reduced weight while a recovered instance warms up—can prevent a sudden traffic surge when it returns. Failure thresholds should be tuned to the service rather than copied blindly.

Key Components

Types / Variations

Layer 4 vs. Layer 7

The most useful first classification is the layer at which routing decisions are made. For an HTTP workload, Layer 7 is often convenient because it can inspect the request; Layer 4 is useful when the protocol is raw TCP/UDP or when transport-level forwarding is the requirement.

Load Balancing Algorithms

Hardware vs. Software vs. Cloud-Managed

For a new cloud deployment, a managed load balancer can reduce operational work because the provider supplies HA and scaling features. Self-hosted HAProxy, NGINX, or Envoy can be appropriate for on-premises systems, special routing needs, cost control, or teams that want direct control; the trade-off is that you own upgrades, capacity, and failover.

Trade-offs

The fundamental tension here is availability vs. complexity. A load balancer solves the single-point-of-failure problem for your app tier, but introduces itself as a new component that must be made highly available, monitored, and correctly configured.

The mistake teams commonly make: candidates draw the load balancer in their diagram without mentioning that it now needs HA too. Address both in the same breath — the interviewer will ask if you don't.

When to Use It / When to Avoid It

So when does this actually matter in an interview? Almost always — any system with more than one server needs one. Here's the practical guide.

Use a load balancer when:

• You have 2+ backend instances that should share traffic.
• You need fault tolerance — one instance failing must not take down the service.
• You need zero-downtime rolling deployments (draining connections from instances one at a time).
• You need SSL termination at a single point rather than managing certificates on every server.
• You need path-based routing to multiple services from a single entry point.

Avoid or simplify when:

• You're in a development environment — local port forwarding or a single process is sufficient.
• You have a monolith with no traffic redundancy requirement — a plain reverse proxy (NGINX) is often enough.
• You're routing internal service-to-service (east-west) traffic at high volume — consider a service mesh (Istio, Linkerd) rather than a centralised LB per route.
• You're prototyping — get the system working first, then add the LB tier before any production deploy.

Real-World Examples

Google — Maglev Google built a custom software load balancer called Maglev that runs on commodity servers and handles over one million packets per second per machine. Maglev uses consistent hashing over a connection table of 65,537 buckets — a prime number chosen for uniform distribution — so the same connection always reaches the same backend even when backends are added or removed.

The design handles up to 640 Gbps per cluster and sits in front of every Google service globally. Maglev is a Layer 4 LB — the routing decision happens before any HTTP parsing.

Netflix — AWS ALB + Eureka + Ribbon Netflix uses AWS ALBs as primary Layer 7 load balancers in front of their microservice clusters. They supplement this with client-side load balancing via Eureka (service registry) and Ribbon (in-process LB library) — each service instance picks a backend directly, eliminating a full network hop for all internal traffic.

New instances use Ribbon's slow-start: reduced traffic weight for the first 90 seconds while their JVM JIT warms up, preventing cold instances from being overwhelmed. The slow-start pattern is worth stealing for any JVM or interpreted-language service you build.

Cloudflare — Geographic load balancing Cloudflare's load balancer operates at the DNS level. When a client resolves your API hostname, Cloudflare returns the IP of the nearest healthy origin — informed by both geographic proximity and measured round-trip latency from multiple global vantage points.

Health checks run from multiple locations every 60 seconds. If an origin starts failing, DNS responses switch to the next healthy origin in well under a minute — geographic failover with zero backend changes required.

How This Shows Up in Interviews

Every system design interview with a backend tier needs a load balancer in the first diagram you draw — not as an afterthought after the interviewer asks "but what about availability?" Draw it immediately, name the layer, and state your algorithm choice in one sentence. The load balancer signals that you understand horizontal scaling isn't just "add more servers."

Depth expected at senior/staff level:

• Name the algorithm and justify it for this specific workload. WebSockets → least-connections. Stateless REST → round-robin. Mixed capacity → weighted.
• Proactively address the LB as a potential SPOF. Mention active/standby HA or the cloud-managed equivalent.
• Know when to use L4 vs. L7 and what that changes about the design.
• Understand that stateless app servers are a prerequisite for the LB to route correctly — not a follow-on optimisation.
• Know what connection draining is and why it matters for zero-downtime deploys.

Common follow-up questions and strong answers:

Test Your Understanding

Quick Recap

1. A load balancer distributes incoming traffic across a pool of healthy backend servers, eliminating the single-server bottleneck and making horizontal scaling possible. Without one, adding servers doesn't reduce load on the original.
2. Layer 4 load balancers route by TCP/IP address and port — fast, protocol-agnostic. Layer 7 route by HTTP content (URL, headers, cookies) and terminate SSL — smarter, with per-request observability. Default to L7 for HTTP workloads.
3. Algorithm choice drives real outcomes: round-robin for homogeneous stateless services; least-connections for long-lived sockets (WebSockets, gRPC streaming); weighted round-robin for mixed-capacity pools.
4. Stateless app servers — sessions in Redis, not in process memory — are the prerequisite for the LB to route correctly. IP Hash (sticky sessions) breaks when server count changes and causes mass session loss on instance failure.
5. Active health checks (GET /health) detect dead servers, not degraded ones. Passive outlier detection is needed to catch slow or error-prone instances before they saturate and cause visible failures.
6. The load balancer itself must be made highly available: active/standby with a shared Virtual IP, or a cloud-managed LB that is inherently multi-AZ. A single LB node is just a new single point of failure.
7. In interviews, name the algorithm and justify it, proactively address LB HA, explain stateless design as a prerequisite, and describe connection draining for zero-downtime deploys — these four together signal staff-level depth.

In System Design Interviews

When discussing load balancing in interviews:

1. Place load balancers between every critical tier (client→webserver, webserver→app, app→db)
2. Mention redundant load balancers (active-passive) to avoid SPOF
3. Discuss which algorithm and why
4. Consider geographic load balancing (DNS-based) for global systems

Related Concepts

• Scalability — Load balancing is the mechanism that makes horizontal scaling of the app tier possible. A stateless app tier behind a load balancer is the core pattern for handling 10× traffic spikes.
• API Gateway — An API Gateway includes a load balancer but adds routing, auth, rate limiting, and protocol translation. Know when a gateway adds value vs. when a bare load balancer is sufficient.
• Caching — Even a perfectly load-balanced app tier gets overwhelmed if every instance makes separate database reads. Caching in Redis keeps DB load flat as the number of instances grows.
• Rate Limiting — Rate limiters live at the load balancer or immediately behind it. Redis-backed distributed rate limiting prevents any single client from overwhelming the backend pool regardless of which instance handles their requests.
• Service Mesh — A service mesh handles load balancing for east-west (service-to-service) traffic, with retries, circuit breaking, and mTLS baked in. A front-end load balancer handles north-south (client-to-service) traffic. You typically need both.
• Envoy — Modern, microservices-focused