Prompt and scope
Each view is a query over source tables or other views. A source change invalidates descendants, but recomputing every descendant immediately is too expensive. The system should serve a versioned result, expose freshness, and never combine incompatible generations of parent views. Assume refresh work can be asynchronous and that a full rebuild remains available for recovery.
What the interviewer is testing
- Modeling dependency edges, versions, invalidation, and topological refresh order.
- Choosing incremental versus full recomputation from change volume and query shape.
- Handling stale results, partial failures, backfills, hot keys, and cache eviction.
- Proving correctness with lineage, manifests, checksums, and replayable events.
Clarifying questions to ask
Ask whether every query has a freshness SLO, whether joins can be incrementally maintained, how updates and deletes arrive, and whether readers prefer a stale answer to an error. If a view contains non-invertible aggregates, a changed row may require a broader recomputation; if a view is append-only, delta refresh is cheaper.
The 30-second answer
I would store a versioned manifest for each view: dependency versions, output location, row count, checksum, and freshness timestamp. Source changes append invalidation events; a scheduler computes affected descendants in topological order, using delta refresh when the query supports it and full rebuild otherwise. Publish a new manifest atomically only after all required parents match the target generation. Readers select a complete generation, may use a bounded stale generation when policy permits, and expose its age. Replay, checksums, and periodic full rebuilds repair drift.
Step-by-step deep dive
1. Represent the DAG and generations
Give each view a stable ID, query definition, parent IDs, and generation. A refresh plan carries a target source watermark and records which parent generation it consumed. Reject publication if a parent changed mid-refresh; retry from a new watermark instead of silently mixing results.
2. Choose delta or full refresh
Use change volume, join shape, and aggregate invertibility as the decision rule. Incremental refresh reads only changed partitions or rows when the engine can prove the delta is sufficient; full refresh is simpler for broad joins or deletes. Keep the old generation available until the new manifest is validated, so a failed refresh does not remove the last good answer.
3. Schedule invalidation and control hot views
Coalesce many source events into one target watermark, then process affected nodes once per generation. Prioritize views by query demand and freshness debt, but cap concurrent work per source to prevent a hot upstream from exhausting compute. Cache popular results by view parameters and generation; invalidate by generation rather than deleting every key individually.
4. Recover, backfill, and prove correctness
Persist invalidation events and refresh manifests so workers can resume after crashes. A backfill runs under a separate target generation and publishes only after comparison with the current generation. Compare row counts, checksums, sampled aggregates, and lineage watermarks; alert when a view exceeds its five-minute freshness target or its parents disagree. A periodic full rebuild provides an oracle for detecting incremental drift.
A strong sample answer
I would clarify freshness by view, update/delete behavior, and whether stale reads are acceptable. Each view has a manifest with parent generations, source watermark, output location, checksum, and freshness. Invalidation events feed a scheduler that coalesces work and refreshes descendants topologically. Use delta refresh for provably incremental queries, full rebuild for broad joins or deletes, and publish a new generation atomically. Readers never mix generations; they may receive a bounded stale generation. Replay, backfill generations, checksums, and periodic full rebuilds make correctness testable.
Common mistakes
- Refresh every descendant immediately → bursts create duplicate work → coalesce events by target watermark.
- Overwrite the only result in place → a failed job leaves readers with partial data → publish immutable generations atomically.
- Assume every aggregate is incremental → deletes or non-invertible functions drift → use a query-shape decision rule and full rebuild fallback.
- Cache without generation metadata → parent and child answers can disagree → bind cache keys to a complete generation.
- Let one hot view consume all workers → other freshness SLOs fail → use per-source and per-view concurrency caps.
- Trust one row count as proof → silent corruption survives → compare checksums, samples, lineage watermarks, and full rebuilds.
Follow-up questions and responses
A parent view refreshes while a child is running. What happens?
The child records the parent generation it read. If that generation is no longer current at publish time, discard or retry the child against a new target watermark; never publish a mixed generation.
Deletes arrive in a supposedly incremental view. Can you still use delta refresh?
Only if the change log and query semantics retain enough information to subtract the old contribution. Otherwise widen the affected partitions or schedule a full rebuild, and state the freshness trade-off.
How do you prevent a backfill from replacing newer data?
Assign the backfill its own generation and source watermark. Publish only when it covers the requested range and does not supersede newer partitions; merge manifests by explicit range and generation rules.
What if the view is queried far more often than it is refreshed?
Serve the last complete generation with its age, prioritize it by freshness debt, and optionally precompute popular parameter keys. Do not hide staleness or let read demand bypass refresh concurrency limits.