Design a text summarization API

30-second answer

• The core architectural decision is document length detection and routing: documents under 128K tokens go through a single-pass summary, documents over 128K tokens go through chunked map-reduce with hierarchical merging.
• Use GPT-4o-mini ($0.15/1M input tokens) for 90%+ of requests. Reserve GPT-4o ($2.50/1M input) for high-fidelity summaries only when the caller explicitly requests premium quality.
• Semantic chunking (splitting at paragraph or section boundaries) produces 30-40% better coherence scores than fixed-size token splitting, because it preserves the author's logical structure.
• Stream partial results using Server-Sent Events (SSE). Users see the first summary tokens in under 2 seconds even for 100-page documents.
• Many summarization failures are pipeline failures rather than model failures. A key argument can span two chunks and be lost at the boundary.

Requirements and assumptions

Functional requirements

• Users can submit a document (plain text, PDF, or URL) and receive a text summary within a specified length (short, medium, or detailed).
• Users can stream the summary response in real time as it is generated, instead of waiting for the full result.
• The system handles documents from 500 tokens to 500K tokens without manual intervention from the user.
• Users can specify a target language for the summary, enabling cross-lingual summarization.
• The system provides a faithfulness score with each summary, indicating how accurately the summary reflects the source document.

Non-functional requirements

• P95 latency under 5 seconds for documents under 10K tokens (single-pass path).
• P95 latency under 30 seconds for documents up to 500K tokens (chunked path).
• Cost per summary under $0.01 for documents under 50K tokens using the default model tier.
• Throughput: 500 concurrent summarization requests with graceful degradation under load.
• Availability: 99.9% uptime with automatic failover between model providers.
• Faithfulness: ROUGE-L score above 0.35 on the CNN/DailyMail benchmark for short summaries.

The hardest engineering problem here: maintaining coherence when a 500K-token document is split into chunks. Each chunk is summarized independently, and the merge step must produce a summary that reads like it was written from the full document, not stitched from fragments.

Assumptions

• The original document remains available for retries, citations, and faithfulness checks; summaries are derived artifacts.
• Short documents can use a single pass, while long documents use an asynchronous map–reduce workflow with a bounded merge context.
• “Short,” “medium,” and “detailed” are explicit output contracts. The service may reject an impossible combination of length, latency, and quality instead of silently ignoring the request.

5-minute approach

Route by token count and input type. Normalize text, choose a single-pass or chunked path, summarize chunks in parallel, merge hierarchically, stream progress or partial output, and score the final result against the source.

• Split at document structure where possible and use overlap only to preserve boundary context.
• Reserve context for the merge and response; do not let one large chunk or verbose intermediate summary consume the full budget.
• Keep standard and premium model tiers explicit, with cache keys that include source/version, target length, language, and model configuration.
• Return a job ID and status events for long work, while keeping the synchronous path small and cancellable.

The entities and endpoint below define the request, chunk, intermediate summary, final result, and quality-score lifecycle.

Core entities

SummarizationRequest

• request_id, user_id, input_type (text/pdf/url), raw_content, token_count, target_length (short/medium/detailed), target_language, model_tier (standard/premium), created_at

DocumentChunk

• chunk_id, request_id, chunk_index, text, token_count, overlap_tokens, section_boundary (boolean)

SummaryResult

• result_id, request_id, summary_text, model_used, total_input_tokens, total_output_tokens, cost_usd, faithfulness_score, latency_ms, chunks_processed

ModelConfig

• model_id, provider, model_name, max_context_tokens, cost_per_1m_input, cost_per_1m_output, avg_latency_ms, is_active

API design

POST /api/summarize - submit a document for summarization

Request: {
  "content": "full document text or base64-encoded PDF",
  "input_type": "text" | "pdf" | "url",
  "target_length": "short" | "medium" | "detailed",
  "target_language": "en",
  "model_tier": "standard",
  "stream": true
}
Response (stream=false): {
  "request_id": "req_abc123",
  "summary": "The document discusses...",
  "model_used": "gpt-4o-mini",
  "token_count": { "input": 45200, "output": 512 },
  "cost_usd": 0.0072,
  "faithfulness_score": 0.82,
  "latency_ms": 3400
}
Response (stream=true): SSE stream
  data: {"type": "chunk", "text": "The document "}
  data: {"type": "chunk", "text": "discusses "}
  data: {"type": "done", "request_id": "req_abc123", "faithfulness_score": 0.82, "cost_usd": 0.0072}

GET /api/summarize/:request_id - retrieve a previously generated summary

Response: {
  "request_id": "req_abc123",
  "status": "completed" | "processing" | "failed",
  "summary": "...",
  "metadata": { "model_used": "gpt-4o-mini", "chunks_processed": 1, "cost_usd": 0.0072 }
}

GET /api/summarize/:request_id/chunks - inspect chunk-level details for debugging

Response: {
  "request_id": "req_abc123",
  "chunks": [
    { "chunk_index": 0, "token_count": 3800, "chunk_summary": "...", "section_boundary": true },
    { "chunk_index": 1, "token_count": 4100, "chunk_summary": "...", "section_boundary": false }
  ]
}

45-minute interview approach

Treat this as a document-processing design question with a clear short-path/long-path split. The hard part is preserving meaning under a context limit while meeting cost and latency targets.

• 0–5 min — Clarify scope: ask input formats, maximum size, summary lengths, target languages, streaming/progress needs, faithfulness expectations, and retention policy.
• 5–10 min — Requirements and estimates: state token distribution, concurrent jobs, P95 latency for short and long documents, cost per summary, availability, and output-quality target.
• 10–18 min — Interfaces and data model: define submit/status/stream endpoints and model requests, chunks, intermediate summaries, final summaries, model versions, and scores.
• 18–28 min — High-level design: draw gateway, parser/tokenizer, router, object storage, queue, parallel map workers, hierarchical reducer, quality checker, cache, and stream/job store.
• 28–38 min — Deep dive: explain semantic chunking, overlap, merge coherence, prompt/output length control, PDFs/URLs, cache safety, and provider failover.
• 38–42 min — Scale and operations: cover concurrency limits, queue fairness, cancellation, retries, token budgets, model routing, observability, and cost controls.
• 42–45 min — Trade-offs and close: compare single-pass with map–reduce, quality with cost, and early streaming with final coherence; state how an unavailable provider affects each tier.

High-level design and data flow

The summarization service has two main paths determined by document length. The "short path" handles documents that fit within a single context window (under 128K tokens for GPT-4o-mini). The "long path" handles everything else using map-reduce chunking with hierarchical merge.

Every request enters through an API gateway that authenticates, rate-limits, and routes to the summarization service. The service first detects the document length by running a fast tokenizer count. If it is under the context window limit, the document goes straight to the LLM. If it exceeds the limit, the chunking pipeline splits it, summarizes each chunk independently, then merges the chunk summaries into a final output.

Start with length detection instead of calling one model for every document. A 200-page PDF needs a chunking and merge path, so the routing decision is part of the design.

The pipeline comes alive when you see the step-by-step execution. Here is the full flow for a long document that requires chunking: