Design a real-time AI translation service

30-second answer

• Use a tiered model architecture: fine-tuned NMT models (Marian/NLLB) at <100ms for high-traffic pairs like EN-ES and EN-ZH, NLLB-200 for mid-traffic pairs, and LLM fallback for the long tail. This strategy cuts cost from ~$25/1M chars (DeepL API) to ~$2-5/1M chars for 80% of traffic.
• Streaming translation uses chunk-based speculative output. Buffer input by sentence boundary detection (punctuation + 300ms pause heuristic), speculatively translate partial buffers, then replace with the refined output when the full sentence arrives. Users see translations appearing within 200ms of typing.
• Glossary enforcement through constrained decoding (logit biasing during beam search) achieves 98%+ term compliance, compared to 75-80% with naive prompt injection and ~60% with post-processing string replacement.
• Quality estimation without human references combines COMETkiwi (0.85+ correlation with human judgment), a glossary compliance checker, and back-translation semantic similarity. Translations scoring below the threshold route to a human post-editing queue automatically.
• Language-pair difficulty varies widely. EN-FR has mature parallel data, while EN-JA or EN-ZH requires handling different word order, writing systems, and formality registers. Model-tier routing should reflect those differences.

Requirements and assumptions

Functional requirements

• Users submit text (up to 5,000 characters per request) and receive a translation in their selected target language, with support for at least 50 language pairs.
• The system detects the source language automatically when not specified by the user.
• Translation output streams to the client in real time as the user types, with partial translations appearing within 500ms.
• Customers can configure per-account glossaries (brand names, technical terms) that the translation engine must preserve exactly in the target language.
• The system provides a quality score for each translation, flagging low-confidence outputs for optional human review.
• The system supports document-level context, maintaining consistency in terminology and formality across paragraphs within a single session.

Non-functional requirements

• P95 latency under 500ms for single-sentence translation (excluding network round-trip).
• P95 latency under 100ms for high-traffic language pairs (EN-ES, EN-FR, EN-DE, EN-ZH) using optimized NMT models.
• Support 10,000 concurrent translation sessions over WebSocket connections.
• Translation quality: COMET score within 2 points of DeepL on WMT23 benchmarks for top-10 language pairs.
• Cost per 1M characters under $5 for NMT-served traffic and under $20 for LLM-served traffic.
• Availability: 99.9% uptime. Translation is a real-time user-facing feature with zero tolerance for extended outages.

The hardest engineering problem here: streaming translation coherence. Unlike LLM chat where you stream tokens sequentially, translation must handle word reordering across languages (English SVO to Japanese SOV, for example). You cannot translate word-by-word because the target word order may be completely different. The system must buffer intelligently, produce speculative partial translations, and then gracefully replace them when the full sentence context is available.

Assumptions

• A translation session may emit provisional text and later revisions. The client understands segment IDs and replacement events, so a corrected sentence does not appear as duplicated text.
• Language-pair quality, latency, and cost determine model routing. High-volume pairs can use optimized NMT while lower-volume or difficult pairs use a broader model tier.
• Glossaries and document context are tenant-scoped. They are applied as policy inputs and must not leak between accounts or sessions.

5-minute approach

Design translation as a stateful streaming session. Detect language, buffer enough source text to form a stable segment, route by language pair and quality target, enforce glossary terms, and emit versioned partial/final translations.

• Keep a per-session buffer and boundary detector; do not translate each keystroke as an independent sentence.
• Use translation memory and glossary lookup before inference, then choose NMT, a multilingual model, or an LLM fallback based on pair coverage and load.
• Attach quality estimates and model/version metadata to each segment, and let the client replace speculative output when more context arrives.
• Separate connection/session state from model workers so workers can scale horizontally without losing ordering guarantees.

The entities and endpoint below define the session, segment, glossary, and quality-result contracts.

Core entities

TranslationRequest

• request_id, session_id, source_text, source_lang (nullable, auto-detected), target_lang, glossary_id (nullable), context_window (previous sentences for consistency), stream (boolean), created_at

TranslationResponse

• response_id, request_id, translated_text, model_used, model_tier (nmt/nllb/llm), latency_ms, quality_score, glossary_terms_enforced (count), is_speculative (boolean), created_at

LanguagePair

• pair_id, source_lang, target_lang, model_tier (nmt_optimized/nllb_general/llm_fallback), model_id, avg_latency_ms, avg_comet_score, daily_request_volume

Glossary

• glossary_id, account_id, source_lang, target_lang, entries (list of source_term/target_term pairs), entry_count, created_at, updated_at

QualityEstimate

• estimate_id, response_id, comet_score, glossary_compliance_pct, fluency_score, back_translation_similarity, aggregate_score, flagged_for_review (boolean)

TranslationMemory

• memory_id, account_id, source_hash, source_lang, target_lang, translated_text, model_used, quality_score, hit_count, created_at, ttl

API design

POST /v1/translate - single request translation (REST)

Request: {
  "text": "Kubernetes orchestrates containerized applications across clusters.",
  "source_lang": "en",
  "target_lang": "ja",
  "glossary_id": "gloss_abc123",
  "context": ["Previous sentence for consistency."],
  "quality_check": true
}
Response: {
  "request_id": "tr_req_xyz789",
  "translated_text": "Kubernetesは、クラスター全体でコンテナ化されたアプリケーションをオーケストレーションします。",
  "source_lang_detected": "en",
  "model_used": "nmt-en-ja-v3",
  "model_tier": "nmt_optimized",
  "latency_ms": 82,
  "quality": {
    "comet_score": 0.91,
    "glossary_compliance": 1.0,
    "aggregate_score": 0.94,
    "flagged": false
  },
  "glossary_terms": [
    { "source": "Kubernetes", "target": "Kubernetes", "enforced": true }
  ]
}

The REST endpoint handles batch and single-shot requests. Teams integrating translation for document processing or email workflows use this. The context field passes up to 3 previous sentences for cross-sentence consistency.

WebSocket /v1/translate/stream - real-time streaming translation

Client sends: {
  "type": "text_chunk",
  "session_id": "sess_abc",
  "text": "Kubernetes orchestrates",
  "source_lang": "en",
  "target_lang": "ja",
  "glossary_id": "gloss_abc123",
  "is_final": false
}
Server sends: {
  "type": "partial_translation",
  "session_id": "sess_abc",
  "translated_text": "Kubernetesはオーケストレーション...",
  "is_speculative": true,
  "confidence": 0.72
}
Server sends (after sentence completes): {
  "type": "final_translation",
  "session_id": "sess_abc",
  "translated_text": "Kubernetesは、コンテナ化されたアプリケーションをオーケストレーションします。",
  "is_speculative": false,
  "quality_score": 0.91
}

The WebSocket endpoint powers the real-time typing experience. Clients send text chunks as the user types. The server returns speculative partial translations (marked is_speculative: true) that get replaced by final translations once the full sentence context is available. This is the critical UX differentiator over batch translation APIs.

POST /v1/glossaries - create or update a customer glossary

Request: {
  "account_id": "acct_456",
  "source_lang": "en",
  "target_lang": "ja",
  "entries": [
    { "source": "Kubernetes", "target": "Kubernetes" },
    { "source": "microservice", "target": "マイクロサービス" },
    { "source": "pod", "target": "Pod" }
  ]
}
Response: {
  "glossary_id": "gloss_abc123",
  "entry_count": 3,
  "created_at": "2026-04-11T10:00:00Z"
}

Glossaries are scoped per account and per language pair. Enterprise customers maintain glossaries with 500-2,000 terms covering brand names, product features, and domain-specific vocabulary.

GET /v1/quality/report - translation quality analytics

Response: {
  "period": "2026-04-01 to 2026-04-10",
  "total_translations": 2450000,
  "avg_comet_score": 0.88,
  "by_tier": [
    { "tier": "nmt_optimized", "volume_pct": 65, "avg_comet": 0.91, "avg_latency_ms": 72 },
    { "tier": "nllb_general", "volume_pct": 25, "avg_comet": 0.84, "avg_latency_ms": 145 },
    { "tier": "llm_fallback", "volume_pct": 10, "avg_comet": 0.87, "avg_latency_ms": 380 }
  ],
  "flagged_for_review": 3200,
  "glossary_compliance_avg": 0.97
}

Quality analytics for monitoring translation accuracy across tiers. Operations teams use this to decide when to promote a language pair from NLLB to a fine-tuned NMT model.

45-minute interview approach

Keep this as a real-time translation design question. Establish the segmentation and revision protocol before discussing model names.

• 0–5 min — Clarify scope: ask typing versus completed-text translation, language pairs, glossary behavior, document context, revision tolerance, and human-review needs.
• 5–10 min — Requirements and estimates: state characters per request, concurrent sessions, message rate, latency targets by pair, quality target, availability, and cost ceilings.
• 10–18 min — Interfaces and data model: define session, source segment, provisional/final translation, glossary, model route, quality estimate, and replacement events.
• 18–28 min — High-level design: draw WebSocket ingress, session buffer, language detector, boundary detector, cache, model router, inference pools, quality estimator, and client stream.
• 28–38 min — Deep dive: explain word-order divergence, partial-sentence handling, glossary enforcement, formality/context, and consistency across paragraphs.
• 38–42 min — Scale and operations: cover sticky-session trade-offs, regional model pools, GPU saturation, backpressure, model canaries, and per-pair quality dashboards.
• 42–45 min — Trade-offs and close: compare early partial output with retractions, one multilingual model with pair-specific models, and quality with cost; state behavior during a pool outage.

High-level design and data flow

The translation service is built around a tiered model architecture. Not all language pairs are equal: EN-ES and EN-FR have decades of parallel corpora and highly optimized NMT models, while EN-MY (Burmese) or EN-AM (Amharic) have sparse training data and need a different approach entirely. Trying to serve all 50 language pairs through a single model is either expensive (GPT-4o for everything) or low quality (a single multilingual model optimized for nothing).

The architecture splits into three planes. The real-time plane handles WebSocket connections, input buffering, and streaming output. The translation plane manages model routing, inference, and glossary enforcement. The quality plane scores every translation asynchronously and routes low-confidence output to human review.

A thin wrapper around a hosted translation API can work for a prototype, but custom glossaries, streaming, quality control, and cost management may justify self-hosted models for high-traffic pairs.

The model router is the decision point. It maps each language pair to a tier based on available model quality, latency requirements, and cost. Here is the routing logic: