Web Crawler

What is a web crawler?

A web crawler fetches pages from the internet and stores their content for downstream indexing. Downloading HTML is the easy part. The real challenge is doing it at a billion-page scale while staying polite to target servers, deduplicating URLs across a very large history, and keeping pages fresh without re-crawling the entire index on every cycle. The question tests distributed queues, Bloom filters, scheduling algorithms, and rate limiting.

The baseline design is periodic and pull-based. A push supplement can consume sitemap notifications, RSS updates, or WebSub events, but a general-purpose crawler still needs a scheduled frontier because many sites do not publish usable change events. The frontier therefore combines priority, freshness, and per-domain politeness rather than treating the web as one unbounded FIFO queue.

TL;DR

Use a durable, two-tier frontier: per-domain URL queues plus a domain-priority queue. A Domain Selector chooses a ready domain, and domain-partitioned workers fetch its URLs while enforcing robots rules and crawl delays locally with a persisted handoff state.

Gate newly discovered URLs with a shared Bloom filter for compact probabilistic deduplication, but keep Crawl DB as the audit and scheduling source of truth. Store raw HTML in object storage, content hashes in the crawl database, and let a Recrawl Scheduler reinsert due pages without consulting the new-URL Bloom filter.

The key trade-off is best-effort coverage versus memory and throughput. A Bloom false positive can skip a new URL, so seeds and high-value URLs need a durable lookup path. At-least-once fetching is acceptable when writes and downstream indexing are idempotent; politeness and security controls are mandatory.

Scope and assumptions

The following are illustrative interview planning assumptions, not a statement about any particular search engine or target site:

• About 1 billion pages in the crawl corpus, up to 100 million new or changed pages/day, and roughly 1,000 fetches/second sustained at the stated capacity.
• The crawler respects robots.txt and uses a default minimum of one request per domain per second, with longer delays for sensitive or slow domains. Site-specific policies and legal review remain required.
• High-change pages are revisited within about 24 hours and low-change pages within about 30 days. Re-crawls bypass the new-discovery Bloom filter and are scheduled from durable state.
• Raw HTML is retained in object storage. A 50 KB average page implies about 50 TB for one compressed-free snapshot; a 1 PB budget is a conservative allowance for multiple crawl versions, replicas, metadata, and larger pages.
• HTML fetching, link extraction, URL canonicalization, freshness scheduling, and downstream handoff are in scope. JavaScript rendering, authenticated content, binary media processing, and full-text ranking are separate tiers.
• Bloom-filter false positives are accepted for ordinary discovery at an illustrative 0.1% rate. Seeds, contractual pages, and other guaranteed URLs use a durable Crawl DB check instead.

Functional Requirements

Core Requirements

1. Crawl all pages reachable from a seed list of URLs.
2. Store the raw HTML and extracted links for downstream indexing.
3. Discover new URLs continuously and re-crawl stale pages on a schedule.

Below the Line (out of scope)

• Full-text search indexing and ranking (downstream system)
• JavaScript rendering (Puppeteer/headless browser tier)
• Login-gated content crawling
• Media file (image/PDF) extraction

Full-text indexing is a downstream concern that consumes the raw HTML this crawler produces. If it becomes in scope, an Index Service can read from the HTML store, tokenize content, compute TF-IDF or BM25 scores, and write to an inverted index store. The crawler and indexer can evolve independently through the raw HTML interface.

JavaScript rendering requires a headless browser tier (such as Puppeteer or Playwright) whose cost and throughput are materially different from a plain HTTP fetch. If in scope, route detected JavaScript-heavy pages to a separate rendering queue with lower throughput and pass the rendered HTML back into the main pipeline.

Login-gated content requires per-site credential management, session handling, and CAPTCHA handling. It is sufficiently different from general-purpose crawling to warrant a separate, targeted crawl service rather than a generalized main crawler.

Media extraction (images, PDFs) is excluded because the storage and processing requirements are different: blob storage, OCR pipelines, and image recognition. The crawler still encounters these URLs during link extraction, but it discards the binary payload and stores only the URL reference.

The hardest part in scope: URL deduplication at petabyte scale is the single most challenging problem here. At 1 billion pages, a memory-resident hash set requires hundreds of gigabytes of RAM. A database lookup per URL becomes a write-path bottleneck at 1,000 pages per second. Getting deduplication right determines everything else about the crawler's performance.

Non-Functional Requirements

Core Requirements

• Scale: 1 billion pages crawled total; 100 million new or updated pages per day.
• Throughput: 1,000 pages per second sustained (86.4 million pages per day at full capacity).
• Politeness: Maximum 1 request per domain per second; 1 per 10 seconds for sensitive or slow-responding domains.
• Deduplication: No URL crawled twice within a single crawl cycle.
• Freshness: High-change pages (news sites, live feeds) re-crawled within 24 hours; low-change pages within 30 days.
• Storage: Raw HTML stored durably; estimated 1 TB per 1 million pages = 1 PB total at full scale.
• Availability: The crawler runs continuously; transient failures must not lose queued URLs.

Below the Line

• Sub-millisecond deduplication latency (seconds-scale latency is acceptable for the URL queue)
• Exactly-once crawl guarantees (at-least-once with idempotency is sufficient)

Read/write ratio: This system is almost entirely writes. For every URL fetched and stored, there is 1 deduplication check, 1 HTML write to object storage, and several URL queue operations. The only significant read workload is the downstream indexer reading from HTML storage. Design every component for write throughput, not read latency. This is one of the few large-scale systems where you can deprioritize read optimization almost entirely.

30-second answer / outline

1. Accept seeds into a durable, domain-aware URL frontier and canonicalize URLs before admission.
2. Use a shared Bloom filter for ordinary new-link deduplication, with a Crawl DB fallback for guaranteed seeds and re-crawls.
3. Let a Domain Selector choose the highest-priority domain whose robots policy and cooldown permit a request; route that domain to one worker owner.
4. Fetch with bounded timeouts, store raw HTML, extract links, update content hashes and next_crawl_at, and enqueue new links idempotently.
5. Monitor frontier depth, domain delay compliance, fetch errors, robots failures, Bloom fill/FPR, storage growth, and scheduler lag.

5-minute explanation

Start with the write-heavy shape: each fetch creates an HTML object, a crawl-state update, and many link-admission operations. A durable frontier prevents process failure from losing queued work, while a two-tier design prevents one prolific domain from monopolizing the queue.

The Domain Selector ranks domains by authority, staleness, and recent reliability, but only dispatches a domain after its cooldown. Domain-partitioned workers make local rate limiting globally effective while the assignment is stable. Worker failure requires a lease or handoff that carries the last-crawl timestamp so reassignment does not reset politeness.

New links pass through canonicalization and a Bloom filter. The Bloom filter is compact and has no false negatives, but false positives can skip legitimate pages, so the system treats it as a best-effort admission optimization rather than the audit source of truth. Re-crawl scheduling uses Crawl DB directly, compares content hashes, adapts intervals between 24 hours and 30 days, and writes the next due time.

The detailed high-level architecture and critical flows below show seeds, frontier selection, robots and rate limits, fetching, storage, deduplication, freshness, and failure recovery. The deep dives justify the filter, queue, and worker-assignment choices.

45-minute interview approach

This is a time-boxed plan for answering the design question in an interview, not a claim that the article should be read in 45 minutes.

• 0–5 minutes — Clarify the contract: Confirm seed scope, crawl depth, robots and legal policy, HTML versus rendered content, freshness, retention, guaranteed URLs, and downstream indexing needs.
• 5–10 minutes — Establish scale: Use the illustrative page, fetch-rate, link fan-out, page-size, retention, domain-delay, and freshness assumptions. Separate fetch throughput from URL-admission volume.
• 10–15 minutes — Define APIs and records: Walk through seed submission, status, jobs, canonical URL, content hash, crawl state, next-crawl time, robots policy, and failure statuses.
• 15–22 minutes — Draw the basic loop: Show seed submitter, durable frontier, fetcher, HTML object store, Crawl DB, link extraction, and at-least-once retry behavior.
• 22–30 minutes — Deep dive on the frontier: Compare FIFO, per-domain round robin, and the two-tier priority queue. Explain domain cooldown, priority scoring, caps, and starvation prevention.
• 30–35 minutes — Deep dive on deduplication: Compare in-memory sets, Redis keys, and Bloom filters; do the 1.8 GB estimate, explain false positives, atomic admission, and re-crawl bypass.
• 35–41 minutes — Reliability, security, and operations: Cover worker reassignment, robots refresh, DNS and SSRF controls, timeouts, storage failures, frontier recovery, scheduler lag, and observability.
• 41–45 minutes — Trade-offs and close: Compare pull versus push supplements, exact versus probabilistic deduplication, and centralized versus partitioned politeness. Recap the frontier lifecycle and invite follow-ups.

Core Entities

• CrawlJob: A top-level crawl task with a seed URL list, status, and configuration (depth limit, domain scope, re-crawl enabled flag).
• URLFrontier: A prioritized queue entry representing a URL to be crawled, with a scheduled crawl time and a numeric priority score.
• CrawledPage: The stored result of one crawl: raw HTML, extracted outbound links, HTTP status code, crawl timestamp, and content hash.
• DomainPolicy: The cached per-domain rules: parsed robots.txt directives, crawl delay setting, last-crawl timestamp, and domain authority score.
• URLFingerprint: A compact record (canonical URL string and its hash) used for Bloom filter deduplication lookups.

Full schema and indexing strategy are deferred to the deep dives. These five entities are enough to drive the API and High-Level Design.

API Design

A web crawler is primarily internal, but operator APIs are needed for seed submission and observability.

FR 1: Submit seed URLs to start a crawl:

POST /v1/crawl/seeds
Body: {
  urls: ["https://example.com", "https://news.ycombinator.com"],
  config: { max_depth: 5, scope: "domain", recrawl_enabled: true }
}
Response: { job_id: "cj_abc123", queued_count: 2, status: "queued" }

POST because this creates a new CrawlJob. The config block lets callers scope the crawl to a single domain or the full reachable web, and enables or disables freshness-based re-crawling per job. The job_id is returned for status polling; callers do not wait synchronously for the crawl to complete.

FR 2: Check the crawl status of a specific URL:

GET /v1/crawl/status/{encoded_url}
Response: {
  url: "https://example.com",
  status: "crawled",
  last_crawled_at: "2026-03-29T12:00:00Z",
  next_scheduled_at: "2026-03-30T06:00:00Z",
  http_status: 200,
  content_hash: "sha256:abc..."
}

The URL is URL-encoded in the path. next_scheduled_at lets downstream integrations know when refreshed content will be available. content_hash reveals whether a re-crawl produced any actual change, which matters for incremental indexers.

FR 3: List active crawl jobs:

GET /v1/crawl/jobs?status=running&cursor=eyJ0c...&limit=20
Response: {
  jobs: [
    { job_id: "cj_abc", seed_count: 2, pages_crawled: 15420, status: "running", started_at: "..." }
  ],
  next_cursor: "eyJ0c..."
}

Use cursor-based pagination because the job list is a time-ordered stream. Offset pagination skips jobs when new crawls start mid-page. Filter by status (queued, running, completed, failed) to limit result set sizes in production.

High-Level Design

1. Basic crawl loop: frontier, fetcher, and storage

The core pipeline: dequeue a URL, fetch the page, parse links, store HTML, and enqueue discovered URLs.

This satisfies FR 1 and FR 2 end-to-end on a small seed set. It has no politeness enforcement, no deduplication, and no priority control. Establishing correctness first gives us a clear baseline before adding complexity.

Components: