Design an AI document processing pipeline

30-second answer

• Tiered OCR routing (digital PDF text extraction, Tesseract for clean scans, Google Document AI for complex layouts) cuts OCR costs by 60-70% while maintaining 95%+ accuracy across all document types.
• Multimodal LLMs like GPT-4o with structured output can provide a strong baseline for field extraction on invoices and contracts, while deterministic rules and templates remain useful for high-value fields.
• Multi-layer validation (JSON schema checks, business rule validation, cross-document consistency, LLM logprob confidence scoring) reduces human review volume from 100% to 5-15% of documents while catching 99.5%+ of extraction errors.
• Layout-aware extraction using models like LayoutLM and TableTransformer preserves table structure and spatial relationships that flat OCR text destroys, increasing table extraction accuracy from 40% to 90%+.
• The central operational problem is confidence-aware routing: deciding which documents can be approved, which need human review, and which should retry with a more capable processor.

Requirements and assumptions

Functional requirements

• The system ingests documents in multiple formats (PDF, scanned images, TIFF, DOCX) and produces structured JSON output with extracted fields mapped to a configurable schema.
• The system classifies each document by type (invoice, contract, receipt, tax form, medical record) and applies the corresponding extraction template.
• The system extracts key-value fields, line items, tables, signatures, and dates with per-field confidence scores attached to every output.
• The system validates extracted data against business rules (totals match line items, dates are plausible, required fields are present) and flags inconsistencies.
• The system routes low-confidence documents to a human review queue with the original image, extracted data, and highlighted areas of uncertainty.
• The system supports schema versioning so new document types or fields can be added without reprocessing the entire backlog.

Non-functional requirements

• Throughput: process 10,000 documents per hour (approximately 3 per second sustained).
• P95 latency: under 30 seconds per document end-to-end (intake to structured output).
• Field extraction accuracy: 95%+ across all document types, 99%+ for high-value fields like invoice totals and dates.
• Cost per document: $0.01-0.10 depending on complexity (versus $2-5 for manual data entry).
• Human review rate: under 15% of documents require human intervention.
• Availability: 99.9% uptime with graceful degradation (queue documents during outages, process when recovered).

The hardest engineering problem here: every document is different. An invoice from Vendor A looks nothing like an invoice from Vendor B. Scanned documents vary wildly in quality, rotation, and resolution. The system needs to handle this variance without a custom template per vendor, and it needs to know its own confidence level well enough to route uncertain documents to humans instead of silently producing wrong data.

Assumptions

• The original file is retained in object storage and is the source of truth; derived text, layout, and extracted fields can be regenerated.
• A schema is versioned per document type. Adding a field must not silently reinterpret records that were extracted with an older schema.
• Human review is part of the design. Low-confidence fields are highlighted with their source coordinates so a reviewer can correct the result efficiently.

5-minute approach

Design document processing as a staged pipeline that preserves both text and geometry. Route each document to the cheapest processor that can meet the required quality, then validate the structured result before publishing it.

• Inspect the file and classify its type, language, page quality, and whether it already contains embedded text.
• Extract native text when possible; otherwise route through OCR and retain word boxes, tables, and page coordinates.
• Apply a document-type schema and use a layout-aware extractor for fields, line items, signatures, and tables.
• Combine schema checks, business rules, and calibrated confidence to approve, retry, or send the document to human review.

The entities and endpoint below define the durable job and its asynchronous processing contract.

Core entities

Document

• doc_id, source (upload, email, s3_sync), file_type (pdf, tiff, png, docx), file_size_bytes, page_count, status (queued, preprocessing, classifying, extracting, validating, review, completed, failed), uploaded_at, completed_at

DocumentClassification

• classification_id, doc_id, doc_type (invoice, contract, receipt, tax_form, medical_record, unknown), confidence, model_used, classified_at

ExtractionResult

• extraction_id, doc_id, schema_version, fields (JSON map of field_name to value + confidence), line_items (array of row objects), tables (array of table objects with headers and rows), model_used, tokens_consumed, cost_usd, extracted_at

OCRResult

• ocr_id, doc_id, page_number, engine (native_text, tesseract, document_ai), raw_text, word_boxes (array of {text, x, y, width, height, confidence}), avg_confidence, processing_time_ms

ValidationResult

• validation_id, extraction_id, checks_passed, checks_failed, errors (array of {field, rule, message, severity}), overall_confidence, routing_decision (auto_approve, human_review, retry_with_upgrade), validated_at

HumanReview

• review_id, doc_id, extraction_id, reviewer_id, corrections (JSON diff of changed fields), review_time_seconds, status (pending, in_progress, completed), created_at, completed_at

ExtractionSchema

• schema_id, doc_type, version, fields (array of {name, type, required, validation_rules}), active, created_at

API design

POST /v1/documents - upload a document for processing

Request: {
  "source": "upload",
  "file": "<binary>",
  "file_type": "pdf",
  "priority": "normal",
  "callback_url": "https://acme.com/webhooks/doc-processed",
  "extraction_schema": "invoice_v3"
}
Response: {
  "doc_id": "doc_abc123",
  "status": "queued",
  "estimated_completion_seconds": 25,
  "queue_position": 12
}

GET /v1/documents/{doc_id} - check processing status and results

Response: {
  "doc_id": "doc_abc123",
  "status": "completed",
  "classification": {
    "doc_type": "invoice",
    "confidence": 0.97
  },
  "extraction": {
    "vendor_name": { "value": "Acme Corp", "confidence": 0.99 },
    "invoice_number": { "value": "INV-2026-0042", "confidence": 0.98 },
    "total_amount": { "value": 15420.00, "confidence": 0.95 },
    "line_items": [
      { "description": "Cloud hosting (March)", "quantity": 1, "unit_price": 12000.00, "amount": 12000.00 }
    ]
  },
  "validation": {
    "checks_passed": 8,
    "checks_failed": 0,
    "overall_confidence": 0.96,
    "routing_decision": "auto_approve"
  }
}

POST /v1/documents/batch - submit multiple documents for processing

Request: {
  "documents": [
    { "s3_uri": "s3://acme-docs/invoices/batch-march/*.pdf", "extraction_schema": "invoice_v3" }
  ],
  "priority": "bulk",
  "callback_url": "https://acme.com/webhooks/batch-complete"
}
Response: {
  "batch_id": "batch_xyz789",
  "document_count": 342,
  "estimated_completion_minutes": 8,
  "status": "processing"
}

GET /v1/documents/{doc_id}/review - get human review interface data

Response: {
  "review_id": "rev_def456",
  "doc_id": "doc_abc123",
  "original_image_url": "/v1/documents/doc_abc123/image?page=1",
  "extracted_fields": {
    "vendor_name": { "value": "Acme Corp", "confidence": 0.99, "bounding_box": [120, 45, 380, 72] },
    "total_amount": { "value": 15420.00, "confidence": 0.72, "bounding_box": [400, 890, 520, 915], "flagged": true }
  },
  "validation_errors": [
    { "field": "total_amount", "rule": "line_item_sum_match", "message": "Line items sum to $15,320.00 but total reads $15,420.00" }
  ]
}

PUT /v1/schemas/{doc_type} - create or update an extraction schema

Request: {
  "doc_type": "invoice",
  "version": "v4",
  "fields": [
    { "name": "vendor_name", "type": "string", "required": true },
    { "name": "invoice_number", "type": "string", "required": true },
    { "name": "total_amount", "type": "number", "required": true, "validation": "must_match_line_item_sum" },
    { "name": "due_date", "type": "date", "required": false }
  ]
}
Response: {
  "schema_id": "schema_inv_v4",
  "doc_type": "invoice",
  "version": "v4",
  "active": true,
  "fields_count": 4
}

45-minute interview approach

Treat this as a pipeline design question with an accuracy and review-capacity budget, not simply an OCR vendor comparison.

• 0–5 min — Clarify scope: ask about formats, languages, handwriting, document types, synchronous versus asynchronous results, schema flexibility, and data-residency constraints.
• 5–10 min — Requirements and estimates: state pages per document, documents per hour, latency, field-accuracy targets, cost per document, review-rate target, and storage retention.
• 10–18 min — Interfaces and data model: define upload/job/status APIs and model documents, pages, layout elements, extracted fields, schema versions, confidence, and review tasks.
• 18–28 min — High-level design: draw object storage, queue, classifier, native-text/OCR router, layout analysis, field extraction, validation, result store, and review queue.
• 28–38 min — Deep dive: cover poor scans, tables and coordinates, domain-specific extraction, confidence calibration, idempotent retries, and how reprocessing works.
• 38–42 min — Scale and operations: discuss worker pools, GPU/CPU separation, queue lag, document-size limits, observability, retention, and cost routing.
• 42–45 min — Trade-offs and close: compare hosted versus self-hosted OCR, automation versus review, and accuracy versus cost; state the fallback for model or queue failures.

High-level design and data flow

The system operates as two pipelines. The real-time pipeline processes individual documents on upload: preprocess, classify, OCR, extract, validate, and route. The batch pipeline handles bulk imports by parallelizing across a worker pool, processing up to 10K documents per hour on a cluster of 20 workers. Both pipelines share the same extraction and validation logic.

The architecture has five layers. The intake layer handles uploads and queuing. The perception layer handles OCR and layout analysis (turning pixels into structured text). The intelligence layer uses LLMs for classification and field extraction. The validation layer checks the output against business rules. The routing layer decides whether to auto-approve, retry, or send to humans. Each layer is stateless and scales independently behind the task queue.

Draw the five layers and explain that each document flows through them in sequence. This makes the pipeline pattern and likely bottlenecks visible; in this design, the LLM extraction layer is often the slowest stage at 5-15 seconds per document.