Prompt and context
Assume each job invokes about 25 actions and the average new output is 300 MB. Repeated builds often share inputs, but a wrong hit can ship an artifact compiled with the wrong toolchain or secret. The cache is an optimization: a miss or outage must fall back to execution, while a false hit is a correctness incident. You should separate action-result lookup from immutable artifact storage and explain how remote execution changes the trust and capacity model.
What the interviewer is assessing
- Whether you model an action key from every input that affects a deterministic result.
- Whether you separate mutable action metadata from an immutable content-addressed store (CAS).
- Whether you make writes atomic, verify digests, and prevent cross-tenant cache poisoning.
- Whether capacity, garbage collection, observability, outage fallback, and migration are explicit.
- Whether you distinguish a build cache from a generic key-value cache: correctness beats stale-read tolerance.
Clarifying questions to ask
Ask whether builds are hermetic, which operating systems and architectures are supported, and whether remote execution is required or optional. Clarify the retention target, tenant and repository boundaries, maximum artifact size, expected hit-rate objective, data residency, and whether secrets or proprietary source may enter an output. Confirm whether a result may be shared across branches, toolchain versions, or only within a commit and platform tuple. Also ask whether CI may write while developer machines are read-only.
A 30-second answer framework
I would hash a canonical action description containing the command, toolchain and platform identity, declared input Merkle root, relevant environment, and build configuration. The action cache maps that key to output digests and result metadata; the CAS stores immutable blobs by digest. Readers verify metadata and every blob before materializing. Successful local or remote execution uploads blobs first and publishes the action result last, so partial work cannot become a hit. Namespaces, authenticated writes, quotas, and sandboxing prevent tenant leakage. Cache failures return misses and use controlled execution, while metrics and sampled clean rebuilds detect false hits.
Step-by-step deep dive
1. Define keys and storage boundaries
Canonicalize the action command, compiler and linker versions, platform, declared inputs, relevant flags, whitelisted environment, and external dependency lockfiles. Hash the input tree as a Merkle root. Exclude secrets and nondeterministic timestamps; if an action is not hermetic, mark it non-cacheable or give it a deliberately narrow scope. Store action results separately from the CAS: the result contains output filenames, digests, sizes, exit code, and optional stdout/stderr digests. CAS objects are immutable and addressed only by their digest.
2. Design hit, miss, and publish paths
On a read, route the tenant and action-key namespace to a replicated metadata service, fetch the result, then fetch missing CAS blobs in parallel. Verify size and digest before exposing files. On a miss, execute locally or on a sandboxed worker. Upload verified blobs with idempotent digest operations, then commit the action result in one conditional publish. Concurrent writers may upload the same blob, but only a complete result with all referenced blobs is visible. A corrupted blob or metadata mismatch is a miss plus an alert, never a successful hit.
3. Add distribution, isolation, and security
Use consistent hashing or a metadata-service partition map for action-result lookups, with replicas across failure zones. Put large CAS data in an object store or sharded blob tier and keep hot metadata on low-latency storage. Authenticate every request, authorize repository and tenant namespaces, and default developer machines to read-only. Encrypt transit and storage, apply per-tenant quotas, and sandbox remote actions. Do not deduplicate across tenants unless policy explicitly permits it; a digest alone must not bypass authorization.
4. Plan capacity and lifecycle management
The stated workload is 20,000 jobs/day × 25 actions = 500,000 lookups/day, about 5.8 requests/second on average. A 20× burst is roughly 120 requests/second, before parallel blob reads. If 10% of actions create a new 300 MB output, the uncompressed ingress is 1.5 TB/day; compression and deduplication reduce storage, but the design should reserve multi-terabyte object capacity and bandwidth headroom. Garbage collection starts from live action results and manifests, follows their digest references, applies a grace period, then combines size quotas with LRU or age policies. Never delete a reachable blob, and enforce fair quotas per tenant.
5. Make outages and correctness observable
Treat cache timeout, permission failure, missing blob, digest mismatch, and backend unavailability as separate outcomes. A miss can execute; a prolonged outage needs admission control, local-cache preference, and bounded retries to avoid turning CI into a retry storm. Track hit rate by repository, action class, platform, and toolchain; lookup latency, blob bandwidth, upload aborts, evictions, corruption, and tenant denials. Periodically perform a clean rebuild with local caches removed and compare execution logs or output digests to detect nondeterminism and false hits.
Example of a strong answer
I would expose two services: an authenticated action-cache index and an immutable CAS. An action key covers the canonical command, compiler and platform identity, declared-input Merkle root, whitelisted environment, flags, and locked external dependencies. A hit returns output digests; clients verify and download those blobs before materializing. A miss executes in a sandbox, uploads blobs idempotently, verifies them, and publishes the result only after every reference exists. Action metadata is replicated by tenant and repository, while CAS data is sharded or placed in object storage across zones. Reads can fall back to execution during cache failures; writes are restricted to trusted CI, with quotas, encryption, and no cross-tenant reuse by default. I would size for about 120 peak lookups/second and multi-terabyte storage, then validate hit rate, digest mismatches, nondeterministic actions, concurrent writers, toolchain upgrades, GC reachability, and outage recovery.
Common mistakes
- Hashing only source files while omitting compiler, flags, platform, environment, or locked dependencies.
- Treating an action result and its output blobs as one mutable record, allowing partial publishes.
- Sharing a digest across tenants without checking namespace authorization.
- Designing cache unavailability as a total build outage instead of a controlled miss and fallback.
- Using one global LRU and deleting blobs without tracing references from live action results.
- Calling stdout or stderr volume a cache-hit metric; execution strategy and explicit hit counters are needed.
- Claiming high hit rate without testing clean builds, cross-machine reproducibility, and nondeterministic actions.
Follow-up questions and answers
What if the compiler toolchain is upgraded but the command line is unchanged?
The toolchain identity must be part of the action key, usually through a pinned digest or versioned execution image. A migration can dual-read old namespaces for rollback but should write to the new namespace and measure misses. Never reuse old outputs merely because source and flags match.
How do you recover after cache poisoning?
Stop untrusted writes, quarantine the affected namespace, and identify bad action results and reachable blobs from audit logs. Invalidate the action index, rebuild trusted outputs, and repopulate from verified executions. Keep cache use optional during recovery and preserve evidence for incident review.
What if a build action downloads an unpinned dependency during execution?
It is non-hermetic: mark it non-cacheable until the dependency is pinned and its fetched bytes are represented in the input closure. A short, repository-scoped TTL can be an explicit emergency exception, but it must not be presented as deterministic reuse.
How would you migrate from local caches without a CI outage?
Run remote reads in shadow mode, compare local and remote keys and output digests, then enable remote hits for a small repository cohort. Keep local execution and local cache fallback, gate writes to trusted CI, and expand only after hit-rate, latency, and false-hit checks pass.