System Design Interview: Design a Large-Scale Web Crawler

Interview Question and Scope

Design a large-scale web crawler that supplies HTML to a search index. It tracks 10 billion known URLs and may issue at most 1 billion fetches per day. Design the URL frontier, host-level politeness, deduplication, recrawling, failure recovery, and validation plan, including capacity estimates.

This is a senior system design question for backend, infrastructure, search, and data-platform engineers. The output contains compressed raw HTML, fetch metadata, and newly discovered links. Full-text indexing, search ranking, authenticated pages, images and video, and default JavaScript rendering are out of scope. Coverage is best effort; the system does not promise to traverse the entire web.

The following numbers are interview assumptions, not production measurements: an average successful response body is 200 KB; the daily fetch cap includes successes, 304 responses, failures, and retries; average load uses the full daily cap, and the planned peak is 25,000 fetches per second. A newly discovered URL is durable within 60 seconds. When capacity is available, 99% of overdue high-priority URLs receive a lease within 10 minutes. Host politeness is a hard constraint and cannot be relaxed to catch up on throughput.

A public interview report describes a 25-minute design round in which the crawler code already existed and the candidate had to design a scalable architecture. Public 2026 system design material also treats a distributed crawler as a standalone exercise. One report cannot establish a company's fixed question bank or the question's frequency, so this article treats it as a representative system design problem and makes no company attribution.

What the Interviewer Evaluates

The first signal is scope and estimation. A strong candidate defines the crawl target, freshness goals, downstream consumer, and failure semantics before calculating throughput and storage. Drawing a queue, crawler workers, and a database immediately does not explain why those components are needed.

The second signal is the URL frontier's scheduling invariant. One global FIFO can distribute work, but it cannot enforce concurrency, delay, and backoff across all consumers for the same host. A strong design separates “which host is ready?” from “which URL should this host fetch next?” and gives each host_key one logical owner for its token state.

The third signal is distinguishing three kinds of duplicates: the same normalized URL, different URLs with byte-identical content, and pages with small content differences. They require exact URL state, a content hash, and a near-duplicate fingerprint respectively. One Bloom filter cannot serve all three roles.

The final signal is the failure model. External fetching inevitably encounters timeouts, 429, 5xx, DNS errors, redirect loops, huge responses, and malicious pages. A good answer accepts at-least-once execution, limits duplicate side effects with leases, version-conditional writes, and idempotent completion, and proposes tests and metrics that could falsify the design.

Questions to Clarify Before Answering

• What consumes the output? A search index needs HTML, fetch time, status, and a canonicalized URL. An archive also needs immutable versions. A training corpus would emphasize quality and licensing filters. This problem sends output only to a search index.
• What content is in scope? This design fetches public HTTP/HTTPS HTML. PDFs, media, authenticated sessions, or JavaScript rendering would change the fetcher, parser, cost model, and security isolation.
• How should coverage trade off against freshness? Of the 10 billion known URLs, this problem refreshes 100 million high-value URLs daily and targets a 30-day interval for the other 9.9 billion. Refreshing every page every day is mathematically incompatible with a 1-billion-fetch daily budget.
• At what boundary is politeness enforced? The design forms a host_key from scheme + authority and centralizes robots policy, concurrency, minimum delay, and server-directed backoff for that key. A negotiated allowance changes only that host's policy, not the global invariant.
• How strict is “no duplicates”? URL discovery must not silently lose a URL because of a Bloom-filter false positive, so a durable unique key is the source of truth. Network fetches may repeat; storage and downstream events must be idempotent.
• How long do deleted and failed pages remain? A 404, a 410, repeated failures, and a temporary 5xx need different revisit intervals. This design retains a tombstone and the latest status so rediscovery does not create a new URL.

The 30-Second Answer

“One billion fetches per day is about 11,600 per second on average, so I would plan for a 25,000-per-second peak and divide freshness into 100 million URLs refreshed daily and 9.9 billion refreshed every 30 days. Discovery performs conservative normalization and exact unique-key deduplication. The frontier is sharded by host: a shard first chooses a host whose next_allowed_at has arrived, then takes the highest-priority URL from that host's queue. That gives robots policy, concurrency, and backoff one owner. Fetchers use leases and conditional requests, write HTML to object storage, and send it to parsers that feed links back through discovery. Execution is at least once; URL versions and idempotent completion absorb duplicates. I would focus validation on per-host rate limits, lease expiry, 429/503, unreachable robots files, redirect loops, and crawler traps.”

Step-by-Step Deep Dive

Step 1: Prove that the goals fit within the budget

One billion fetches divided by 86,400 seconds is about 11,574 fetches per second on average. Allowing for traffic variation and catch-up work, round the planned peak to 25,000 per second. If every response returned a 200 KB body, ingress would be at most about 200 TB per day, or 2.31 GB per second on average. A 304 Not Modified has no response content, so actual ingress should be lower than this conservative bound and must be calibrated with load tests and observed distributions.

The daily revisit plan requires:

100,000,000 + 9,900,000,000 / 30 = 430,000,000 fetches

That leaves roughly 570 million fetches for newly discovered pages, retries, and fast-changing pages. At an assumed 200 bytes of raw logical state per URL, 10 billion URL records require about 2 TB. Replication, indexes, LSM amplification, and object storage are excluded. This order of magnitude calls for horizontally partitioned metadata and separate object storage for HTML; page bodies do not belong in the frontier.

Step 2: Build a staged data flow

The full path is: seeds and Sitemaps → URL discovery and normalization → exact seen state → URL metadata → frontier scheduler → robots and host-politeness check → DNS/HTTP fetcher → HTML object storage → parser → discovered links back into discovery. Parsed output and fetch-completion events then go to the search index and revisit calculator.

A Sitemap supplements seeds; it does not guarantee coverage. One Sitemap file may contain at most 50,000 URLs and may be at most 50 MB uncompressed. Large sites split them behind a Sitemap index. Link discovery, Sitemaps, and operator-provided seeds all use the same deduplication entry point so three state machines cannot disagree.

Separating fetch from parse has two direct benefits. Slow external I/O does not occupy parser CPU, and a parser crash can replay stored HTML without contacting the site again. Every stage needs a bounded queue and backpressure so a temporary download rate above parsing or storage capacity cannot exhaust memory.

Step 3: Make host politeness the frontier's scheduling primitive

The frontier uses two queue levels. The upper level stores each host's next_allowed_at and priority and selects only hosts that are ready and not in backoff. The lower level is a priority queue of URLs for that host, ordered by signals such as business value, due time, link depth, and historical change rate. Leasing one URL atomically updates the host's in_flight count and next eligible time.

Hashing host_key assigns a host to one scheduler shard. Even if a host has a million pending URLs, one logical owner grants its tokens while fetch work can run on many machines. A hot host may have multiple concurrent connections, but the same host state still controls its allowance. Adding workers increases parallelism across hosts; it cannot legitimately exceed one host's allowance.

robots.txt is fetched from the service's top-level /robots.txt. A successful fetch requires the crawler to follow parseable rules. When the file is unavailable with 400–499, the protocol permits access; when network errors or 500–599 make it unreachable, the crawler assumes complete disallow. A cached copy normally should not be used for more than 24 hours unless the file is unreachable. “One concurrent request and a one-second delay per host” is only this interview's configurable default; the protocol defines no universal rate. On 429 or 503, honor Retry-After; when it is absent, apply jittered exponential backoff and reduce that host's allowance.

Step 4: Separate URL deduplication from content deduplication

Normalization performs only semantics-preserving transformations: resolve relative references, remove the fragment, normalize scheme and hostname case, handle default ports, and resolve path dot-segments. Do not globally delete or reorder query parameters; some sites assign meaning to order and repeated keys. A page-declared canonical URL may influence scoring and clustering, but it must not overwrite the observed URL as fact.

Store canonical_url, or a collision-safe unique key for it, in the partitioned metadata store. A Bloom filter is only a negative accelerator: on “definitely absent,” try the insert directly; on “possibly present,” still check the durable unique key. A false positive therefore adds one read instead of dropping a page. Resolve hash collisions by comparing the full URL or a second fingerprint.

Compute a content hash only after fetching. Byte-identical content can reuse one object while preserving each URL's metadata. Near-duplicate pages can be grouped with a fingerprint such as SimHash. Research has demonstrated this class of fingerprint at multi-billion-page scale, but the near-duplicate signal is safer as an input to storage, indexing, or revisit priority. Dropping a page outright may also drop links that are unique to that page.

Step 5: Recover with versioned state and leases

Keep the core records compact:

UrlState( urlid, canonicalurl, hostkey, stateversion, lastfetchat, nextfetchat, priority, etag, lastmodified, contenthash, failure_count )

HostState( hostkey, robotspolicy, robotsexpiresat, nextallowedat, inflight, backoffuntil, policy_version )

FetchLease(leaseid, urlid, urlversion, expiresat, attempt)

The scheduler issues a bounded lease carrying url_version. A fetcher can crash after writing HTML but before acknowledging the task, so lease expiry can cause another fetch. Completion uses a conditional write on (url_id, url_version). An old lease or duplicate acknowledgement returns the existing outcome and does not publish another index event. A new revisit increments the version first, so the previous round's idempotency key cannot suppress legitimate new work.

When an ETag is available, send If-None-Match; otherwise a stored Last-Modified can drive If-Modified-Since. A 304 updates fetch time and the next schedule without writing an empty body. DNS timeout, connection failure, and 5xx enter a capped retry policy. A durable 404/410 creates a tombstone and a much longer revisit interval. Redirects have a hop limit and loop detection.

Step 6: Put revisits, trap protection, and security under one budget

Revisit priority combines page value, recent change interval, status, and site allowance. A content change shortens the interval; repeated unchanged results lengthen it, clamped between one and 30 days. This shifts budget from stable pages to changing pages while retaining a minimum freshness guarantee.

A maximum link depth alone does not stop calendars, faceted navigation, or unbounded query combinations. Add a per-host daily budget, URL-template growth limits, query-parameter count, repeated-path detection, response-body and decompressed-size limits, parser-time limits, and redirect-hop limits. When a pattern consumes its budget, pause that pattern and keep a sample without blocking unrelated hosts.

Fetchers process untrusted input. Reject loopback, private, link-local, and cloud-metadata addresses from DNS results and recheck them immediately before connecting to reduce SSRF and DNS-rebinding risk. Run parsers with memory and CPU limits and isolate compression bombs and malformed HTML. Robots rules express crawl preferences; they are not access authorization.

Step 7: Validate invariants with fault injection

Begin with a deterministic scheduling simulation. Give three hosts different rates, robots rules, and Retry-After values; advance a virtual clock; assert that no time window exceeds an allowance and a disallowed path never receives a lease. Then inject “process crashes after HTTP success,” “lease acknowledgement is lost,” “robots cache expires,” “DNS resolves to a private address,” and “parser queue stops.” Tasks must recover, index events must remain unique, and the fetch stage must apply backpressure.

Capacity tests should cover 25,000 lease grants per second, partition skew across a 10-billion-URL keyspace, and one hot host with a million pending URLs. Key metrics include eligible lag, fetch and byte rates, 2xx/304/429/5xx ratios, host-policy violations, lease retries, URL and content duplicate rates, robots-cache age, parser backlog, and budget-trigger rate. Host-policy violations must remain zero; meeting average throughput while violating politeness is a failed test.

Alternatives and their boundaries

At a few million fetches per day against owned sites, a relational database can index next_fetch_at, use SKIP LOCKED to claim tasks, and update a host token in the same transaction. It is simpler to deploy and debug. At a billion fetches per day, global index scans, hot updates, and cleanup become bottlenecks, making the two-level partitioned frontier a better fit.

Using a Bloom filter as the seen set saves reads, but false positives permanently reduce coverage. This design demotes it to a cache and keeps the durable unique key as the source of truth. Skipping the confirmation read is reasonable only when the product explicitly accepts a quantified false-positive budget.

Example of a Strong Answer

“I would start with two invariants: budget and politeness. One billion fetches per day is about 11,600 per second on average and a 25,000-per-second peak. Refreshing 100 million URLs daily plus 9.9 billion every 30 days schedules about 430 million fetches per day, leaving room for discovery, retries, and change-driven refreshes. At 200 KB per response, 200 TB per day is a conservative network upper bound; 304 responses reduce actual traffic.

On entry, a URL receives only safe normalization and is checked against a durable unique key. A Bloom filter only avoids reads for obviously unseen URLs. The frontier is partitioned by scheme + authority; each shard maintains host readiness plus a per-host URL priority queue. Claiming work atomically consumes a host token, so multiple fetchers cannot collectively overload one site. An unreachable robots file pauses the host, and 429/503 triggers Retry-After or jittered backoff.

A fetcher receives a bounded lease, makes a conditional GET, writes HTML to object storage, and hands parsing to the next stage. Parsed links return through the same discovery entry point. Lease expiry may repeat a request, but URL-versioned completion prevents an old lease from overwriting state or emitting a second index event. A content hash reuses exact duplicate objects, while SimHash affects near-duplicate priority instead of discarding potentially unique links.

I would prove the per-host interval with a virtual clock and inject crashes after write, lost acknowledgements, expired robots state, DNS rebinding, and parser backpressure. Acceptance requires both the 25,000-per-second peak and zero host-policy violations.”

Common Mistakes

• Mistake: use one global message queue. Why it fails: independent consumers cannot jointly enforce a host's next eligible time, so higher throughput increases the risk of violating politeness. Fix: assign scheduling ownership by host_key and use a host-ready queue plus per-host URL queues.
• Mistake: use a Bloom filter as the only seen set. Why it fails: a false positive permanently drops an unseen URL, and the filter cannot store status, version, or revisit time. Fix: use it only as a cache and keep durable unique-key state.
• Mistake: treat URL deduplication as content deduplication. Why it fails: different URLs can return the same content, and one URL can change over time. Fix: deduplicate exact URLs during discovery, then compute a content hash and near-duplicate fingerprint after fetching.
• Mistake: promise exactly-once crawling. Why it fails: external HTTP success and internal acknowledgement cannot form one atomic transaction, leaving a crash window. Fix: accept at-least-once execution and control internal side effects with leases, version-conditional writes, and idempotent events.
• Mistake: continue on every robots error. Why it fails: the protocol distinguishes unavailable from unreachable; a network error or 5xx requires complete disallow. Fix: implement an explicit state machine and separately test cache age, redirects, and failure classes.
• Mistake: rely only on maximum depth for crawler traps. Why it fails: one depth can contain unbounded faceted, calendar, and query-parameter combinations. Fix: combine host budgets, URL-pattern growth, parameter count, response-size, and parser-time limits.

Follow-up Questions

What if 50% of pending URLs belong to one host?

First establish the concurrency and rate that the site permits. With a fixed allowance, more workers cannot increase that host's legitimate throughput; they only improve parallelism across other hosts. The host's URL queue can be partitioned to reduce a storage hotspot, but every partition still requests capacity from one logical token service. If the business needs more speed, negotiate a dedicated feed or larger allowance with the site and version the new policy.

Why not guarantee exactly once?

A fetcher can crash after receiving the HTTP response and before acknowledging its lease, and the external site does not participate in an internal transaction. A distributed transaction cannot undo the GET that already happened. The achievable contract is at-least-once claim, possible duplicate fetch, and idempotent internal completion. Measure the duplicate rate and reduce its cost with conditional requests.

What if the page needs JavaScript to expose its content?

Keep normal HTTP fetching as the first tier. Only when parsing is empty, site policy permits it, and page value clears a threshold should a URL enter a separate rendering queue. Renderers have lower concurrency and stricter CPU, memory, and time budgets, and share the original host token. Otherwise expensive rendering would bypass politeness and consume the global budget.

How can the crawler run in multiple regions without hitting a host twice?

Assign each host_key a home region and allow only that region to grant host tokens; other regions may parse and store. On a regional failure, transfer ownership with a lease carrying a fencing token. The recovered old region must hold the new epoch before granting work. Active scheduling for the same host in multiple regions would violate the politeness invariant.

Storage cost suddenly exceeds budget. What should be reduced first?

First improve conditional-request hits, compression, and reuse of exact duplicate objects. Then shorten raw-HTML retention by content value. Do not discard URL metadata and fetch audit records with it; revisits, deduplication, and compliance investigations depend on that state. Near-duplicate detection can lower priority or choose a storage tier, but it should not trigger unvalidated bulk deletion.