Post-mortem: Slack message delays 2022
A post-mortem of a Slack incident where database connection pool exhaustion caused message delivery delays affecting millions of users, with lessons on connection pool sizing.
Takeaway
Connection pools are an amplification point. A slow query holds connections, queued requests consume application resources, and retries can overload a recovering shard. The durable response is bounded waiting, query timeouts, priority lanes, shard-level circuit breakers, and a recovery plan that limits queue growth.
Scope and Evidence
Composite analysis: facts versus illustrative reasoning
The frontmatter identifies this as a composite analysis of documented Slack connection-pool incidents from the 2022 period. It does not claim that the exact timeline below occurred as one event. The durations, user counts, shard topology, query timings, pool sizes, traffic multipliers, and calculations are representative teaching values unless explicitly identified as public reporting. The causal model is illustrative and should not be read as unpublished Slack telemetry or as an attribution of a specific internal change.
5-Minute Incident Walkthrough
- Trigger: A slow query or lock extends database service time while traffic rises.
- Queue: App-server pools fill, new requests wait, and queued work consumes memory and runtime capacity.
- Cascade: GC pauses and failed health checks remove servers, pushing more traffic onto the remaining pools.
- User impact: Message sends, workspace loads, search, and integrations become delayed or fail for affected shards.
- Recovery: Kill or constrain the slow work, fail fast, restart or drain safely, and control client retries while verifying backlog recovery.
Causal chain: slow query + traffic spike → connection pool saturation → request queue and memory pressure → GC/server removal → traffic redistribution and retry amplification → delayed recovery.
Incident Summary
Date: Composite analysis of Slack's documented connection pool incidents (2022 period) Duration: Representative composite events lasted 1-4 hours; the modeled longest event degrades message delivery for approximately 6 hours Systems affected: In the representative model, the message send/receive pipeline, workspace loading, channel lists, search, and file uploads Impact: In the composite model, messages are delayed from milliseconds to minutes. Users see "sending..." indicators stuck for 30+ seconds, some messages fail, workspace loading rises from under 1 second to 10-15 seconds, and the scenario uses approximately 10 million daily active users as its scale—not as a confirmed single-event count. Root cause: In this composite model, database connection pool exhaustion is caused by a combination of a traffic spike and slow queries. When connection pools fill, new requests queue waiting for a connection. The queue grows faster than it drains, creating a cascading delay from the database layer through the message delivery pipeline.
Connection pool exhaustion is a common cause of database-related production incidents, alongside replication lag, disk exhaustion, and primary failure. It is mundane and preventable, which is exactly why it catches teams off guard. The pool is sized for average conditions, tested under average conditions, and then deployed into a world where traffic can spike to several times average on a Monday morning. The day traffic hits that spike while a slow query is running is the day the pool's assumptions meet production reality.
The recurring pattern is: something makes queries slow (a missing index, a lock, or replication lag), the pool fills up, requests queue, memory grows, GC pauses begin, and the system enters a death spiral. The most reusable controls are aggressive pool wait timeouts, circuit breakers, and automated slow-query termination.
What Happened: The Timeline
The timeline below is a representative reconstruction for the composite analysis. Its timestamps, traffic multipliers, pool sizes, and user-impact figures are illustrative unless linked to a public incident record.
| Time | Event |
|---|---|
| 9:00 AM | Monday morning traffic surge begins; traffic reaches 2.5x weekend baseline |
| 9:15 AM | A slow query (unindexed JOIN) begins executing on a hot database shard |
| 9:20 AM | Average query latency on the affected shard rises from 2ms to 200ms |
| 9:25 AM | Connection pool on app servers hitting the affected shard reaches capacity (100/100 connections) |
| 9:28 AM | New requests begin queuing for available connections |
| 9:30 AM | Pool wait time alerts fire; message send p99 latency exceeds 5 seconds |
| 9:35 AM | Request queue depth grows; memory pressure increases from buffered requests |
| 9:40 AM | GC pauses begin on app servers, holding connections longer, further slowing pool drain |
| 9:45 AM | Message send failures begin; HTTP 503 error rate exceeds 5% |
| 10:00 AM | On-call engineer identifies and kills the slow query |
| 10:15 AM | Rolling restart of affected app servers to clear request queues and reset pools |
| ~10:30 AM | Message latency returns to normal levels; incident resolved |
The total impact was roughly 90 minutes of degraded message delivery, with about 30 minutes of that being severe (messages failing or delayed by over 30 seconds). For a platform where "real-time messaging" is the core product, even 30 seconds of delay breaks the user experience.
What users actually experienced. The symptoms were not uniform. Some workspaces (those on the affected shard) saw severe delays. Others were completely unaffected. Within affected workspaces, the experience depended on timing: messages sent right before pool exhaustion went through fine, messages sent during the peak saw 10-30 second delays, and messages sent after the queue filled up either failed silently (the client showed "sending..." for 30 seconds and then gave up) or succeeded with a noticeable lag.
The inconsistency made the incident harder to triage. Support received reports like "Slack is slow" alongside reports like "Slack is working fine for me." That is the signature of a shard-specific database issue: only a subset of users are affected, and the subset corresponds to a database partition that is invisible to the user.
Downstream cascading. Message delivery delays also affected:
- Slack bots and integrations: Webhook deliveries backed up because bots waiting for database reads to construct responses timed out. CI/CD notifications from GitHub, Jira ticket updates, and PagerDuty alerts were all delayed.
- Thread loading: Opening a thread requires reading message history. Slow reads meant threads took 10+ seconds to load, and some timed out entirely.
- Search indexing: The search indexer, which reads new messages to index them, fell behind. Recently sent messages did not appear in search results for 30+ minutes.
- Unread counts: Badge counts and unread markers depend on DB reads. During the incident, unread counts were incorrect, showing stale values. Some users had channels marked "unread" with no new messages, or had new messages with no unread indicator.
Shard-specific failures look random to users
When only one database shard is affected, only the workspaces assigned to that shard experience degradation. To users, this looks random: "Why is it slow for my team but my friend at another company says it is fine?" The answer is that different workspaces live on different shards. This is important for interviews: sharded databases isolate failure to the affected shard, which is a benefit of sharding. But it also makes diagnosis harder because the symptoms are inconsistent.
Representative Slack-Scale Message Delivery Architecture
To understand why connection pool exhaustion cascades so quickly, see where the pool sits in this representative Slack-like message delivery path.
Each app server maintains a fixed-size connection pool (e.g., 100 connections) to each database shard it communicates with. Under normal conditions, the pool is partially utilized: maybe 20-30 connections active at any moment, with the rest idle and ready.
The critical path for sending a message is:
- Client sends message via HTTPS
- App server checks out a connection from the pool
- App server executes an INSERT into the message table on the appropriate shard
- App server returns the connection to the pool
- App server enqueues a fanout job for real-time delivery
- Client receives acknowledgment
Steps 2-4 typically take 2-5ms. The connection is borrowed for that tiny window and returned. At 2ms per message and 100 connections, a single pool can theoretically support 50,000 messages per second. In practice, some queries take longer, so real throughput is lower, but the point is: under normal conditions, pool utilization is low and everything flows.
The critical insight for system design is Little's Law: the number of connections in use equals the arrival rate multiplied by the average service time. If you receive 1,000 queries per second and each takes 2ms, you need 2 connections. If each takes 200ms (due to a slow query), you need 200 connections. The pool size does not change, only the query latency changes. This means a 100x increase in query latency causes a 100x increase in pool utilization. Understanding this relationship is essential.
Little's Law for connection pools:
connections_in_use = arrival_rate * avg_query_time
Normal: 1000 q/s * 0.002s = 2 connections in use (2% of 100-connection pool)
Slow Q: 1000 q/s * 0.200s = 200 connections needed (200% of 100-connection pool → queue)
Spike: 5000 q/s * 0.002s = 10 connections in use (10% of pool, still fine)
Both: 5000 q/s * 0.200s = 1000 connections needed (1000% of pool → catastrophic queue)
Continue Reading with Premium
Unlock this article and every other in-depth system design guide on the platform with SDEpedia Premium.