Question and scenario
Design a vector-search extension for an Iceberg table. Data files keep the row data and the query engine reads a snapshot; the vector index lives in Puffin sidecar files and is referenced by snapshot metadata. The design must support approximate nearest-neighbor queries without breaking snapshot isolation, time travel, or data-file maintenance.
Assume daily batch appends and small updates. Approximate results are acceptable only when the index version is visible to the caller. Distinguish the capabilities of the Apache Puffin file format from a particular ANN layout proposed by research; an experimental graph design is not automatically Iceberg behavior.
What the interviewer is testing
Snapshot and index consistency
A strong answer binds the data snapshot, Puffin blob, and index metadata in one visible commit. Uploading an index file alone does not make it queryable.
Approximate search and file pruning
Explain that the vector index retrieves candidates, while final distance and visibility checks still read data rows. Handle missing, stale, and low-recall indexes explicitly.
Incremental maintenance and deletes
Cover appends, updates, deletes, merges, and compaction. Do not stop at a one-time offline build.
Operability with disaggregated compute and storage
State how object storage holds Puffin, how a coordinator schedules shards, how garbage is bounded, and how freshness and fallback are monitored.
Clarifying questions before answering
- What are embedding dimension, distance function, query latency, and minimum recall?
- Must a query use the newest snapshot, or is a bounded index lag acceptable?
- Are updates and deletes append-only CDC, Iceberg equality deletes, or data-file rewrites?
- Is one engine building the index, or must Spark, Flink, and Trino share it?
- Are vectors sensitive, and who owns access control and encryption?
- Must time-travel queries reuse historical indexes, or is indexing only required for the current snapshot?
30-second answer framework
“I would keep Iceberg data files, Puffin index blobs, and snapshot metadata separate, but publish the binding in one commit for a single snapshot ID. A query chooses a visible snapshot, reads its index references, performs ANN candidate retrieval, then validates row versions and exact distances from the data files. Missing or stale indexes fall back to partition or file scans and expose quality status. Appends can create delta indexes; updates and deletes are filtered by tombstones or delete layers; compaction rebuilds a baseline. Index jobs use optimistic snapshot commits, and we monitor freshness, recall samples, fallback rate, and Puffin garbage.”
Step-by-step deep answer
Step 1: Estimate data and index boundaries
As an illustrative assumption, 1 billion vectors with 768 float32 dimensions require 1 billion times 768 times 4 bytes, about 3 TB for raw vectors before columnar compression or index overhead. The estimate rules out putting the index in one manifest or coordinator memory; shard it in object storage and load by query partition.
Step 2: Bind Puffin to a snapshot
Puffin stores index or statistics blobs that an Iceberg manifest cannot directly carry. Each blob includes metadata such as type, fields, partition, or data-file references. An index builder writes Puffin, then commits a new Iceberg snapshot whose summary records index location, version, and coverage. A reader accepts a reference only when it is visible with that snapshot.
snapshot S42
data files: D100, D101
summary:
vector.index.version = v7
vector.index.puffin = s3://table/metadata/puffin-v7
vector.index.covers = D100,D101Step 3: Design the query path
Resolve the current branch or time-travel request to snapshot S, then select Puffin blobs by coverage. An ANN graph or shard returns candidate row identifiers and approximate distances. The engine reads those data files, checks row visibility in S, applies authorization and predicates, and recomputes exact distances. Return snapshotid and indexversion so callers can reason about freshness.
Step 4: Handle appends, updates, and deletes
Appends can write a delta Puffin index and declare its files in the same snapshot commit. Before a rebuild, updates or deletes filter old candidates with equality deletes, position deletes, or a delta tombstone; a query must never return a deleted row. Merge delta indexes into a new baseline in the background, then atomically publish a new snapshot binding. On failure, keep the old index and fallback path.
Step 5: Handle concurrent commits and compaction
An index job reads baseline S42 and builds v7. If a data commit advances the table to S43, optimistic concurrency decides whether to retry, merge a delta, or abandon v7. When compaction changes data-file paths, the old index cannot claim coverage of the new files. Build a new blob bound to the rewritten file set, then garbage-collect the old blob after reference tracking and a grace period.
Step 6: Operate, isolate, and fall back
Object storage holds Puffin; a coordinator schedules builds by partition or file set; query nodes cache small routing or centroid structures. Missing blobs, incompatible versions, authorization failures, or low recall samples trigger a file-scan fallback or a response containing only verified candidates and a reason. Track index freshness, build lag, query p95, recall samples, Puffin bytes, fallback rate, and unreferenced blobs.
High-quality sample answer
“I would keep vector columns in Iceberg data files, write ANN structures to Puffin, and publish their references as part of a snapshot. With an illustrative assumption of 1 billion 768-dimensional float32 vectors, raw vectors are about 3 TB, so the index must be sharded in object storage rather than held by a coordinator.
The builder reads S42, creates Puffin v7 covering D100 and D101, and uses an optimistic commit to publish a new snapshot. A query fixes the snapshot for a time-travel request, reads only indexes declared for that snapshot, and validates candidate visibility, authorization, and exact distance by reading the data files. While the index is stale, appends use delta indexes and updates or deletes are filtered by delete files or tombstones; compaction later rebuilds the baseline.
If the table has advanced to S43, v7 cannot be called current without a retry or an explicit stale marker. Missing blobs, incompatible formats, or low recall trigger a scan fallback, with snapshotid, indexversion, and reason in metrics. Reclaim old Puffin only after no historical snapshot or branch references it.”
Common errors
- Treat Puffin as a new primary table format → the query cannot prove which data version an index covers → bind coverage to a snapshot.
- Make an uploaded index immediately readable → data files and index may belong to different snapshots → publish the reference in one optimistic commit.
- Return ANN candidates directly → deletes, permissions, or distance error leak wrong rows → recheck visibility and recompute exact distance.
- Keep using the old graph after updates → deleted rows can be retrieved → filter with delete layers, merge deltas, and rebuild a baseline.
- Reuse old file paths after compaction → the index claims to cover files that no longer exist → build a new blob and snapshot for the rewritten set.
- Treat a research graph layout as Puffin standard → engines cannot interoperate → use Puffin for storage and metadata, with the graph algorithm replaceable.
- Fail every query when an index is missing → service is unavailable during new partition rollout → scan as fallback and expose freshness and reason.
- Delete Puffin without reference tracking → time-travel or branch reads break → wait for every snapshot, branch, and grace period to release it.
Follow-up questions and responses
Follow-up 1: How do you guarantee the right index for time travel?
Store the reference in the corresponding snapshot summary with its file coverage and version. Fix the snapshot first, reject blobs covering later or different files, and scan on absence instead of silently using the current index.
Follow-up 2: Will a large update stream create unbounded delta indexes?
Set a delta-layer limit per partition or file set and schedule a merge rebuild when it is exceeded. Atomically switch to a new snapshot after the merge, retaining old deltas until historical snapshots no longer reference them.
Follow-up 3: Can two engines exchange their ANN indexes?
Only when blob type, distance function, vector encoding, row identifier, and version protocol are compatible. Puffin defines the container and metadata boundary; the graph needs a capability declaration. Otherwise ignore it and fall back.
Follow-up 4: How do you measure recall without scanning the whole table?
Sample a small set of live queries and compute approximate ground truth offline with exact search or a trusted baseline. Compare recall@k by partition, vector version, and distance function. Mark an index stale when samples fall below the threshold instead of watching p95 alone.
Follow-up 5: What if a Puffin file or object store is temporarily unavailable?
Verify blob checksums and metadata, retry from a replica, then scan or reject approximate search with an explicit status after timeout. Never publish a new snapshot that references a corrupt blob.
Source 1: Apache Puffin specification
The Puffin specification defines a file format for indexes and statistics that cannot be stored directly in Iceberg manifests, including blob metadata and data-file references. That supports the sidecar, coverage, and snapshot boundary in this answer.
Source 2: Puffin-backed vector-index research
The 2026 paper proposes attaching approximate-nearest-neighbor structures to Iceberg snapshots and discusses compute-storage disaggregation, snapshot-level index management, and billion-vector settings. This answer treats it as a replaceable research implementation and adds delete, fallback, and concurrent-commit boundaries.
Source 3: Data-engineering interview preparation guide
The public data-engineering guide highlights SQL, data modeling, pipelines, batch and streaming systems, and reliability. The answer maps those signals to snapshot consistency, index maintenance, sharding, verification, and failure fallback.