Prompt and scope
An orders lake table is read by batch jobs, streaming jobs, and ad hoc analysts. The business wants a nested field, a column rename, and a gradual change from monthly to daily partitioning. Old jobs cannot upgrade at once, and rewriting all historical files is too expensive. Explain how Iceberg records these changes and keeps readers correct while old and new layouts coexist.
Assume an Iceberg Catalog and engines that support the target Iceberg version. The interview tests table-format schema and partition evolution, not a particular Spark SQL spelling.
What the interviewer is evaluating
- Whether you distinguish field IDs, Schema IDs, Partition Spec IDs, and snapshots.
- Whether you explain why add, drop, rename, and selected type promotions need not rewrite old files, plus their limits.
- Whether you explain coexistence of partition layouts and planning across multiple specs, including when rewrite is still useful.
- Whether you propose compatibility, performance, concurrent-commit, and rollback validation.
Questions to clarify first
- Are readers using the Iceberg table format, or merely treating a directory as a Hive table? The latter does not automatically provide field-ID semantics.
- Is the change top-level, nested, or a partition transform? Nested and partition fields have extra constraints.
- Do old readers bind columns by position or cache an old schema? Confirm that engines honor Iceberg field mapping.
- Is the goal fewer scans, a hotspot fix, or only logical schema change? Partition benefits need query evidence.
A 30-second answer framework
“Iceberg stores table state in versioned metadata and maps columns with non-reused field IDs instead of positions or recycled names. A schema change creates a Schema ID; a partition change creates a Partition Spec ID. Old files keep their layout, new writes use the new spec, and readers plan each spec while using hidden partition pruning. I would run compatibility checks, publish the metadata atomically, then test old and new readers, rename correctness, files scanned, failed retries, and snapshot rollback. ‘No file rewrite’ is a migration property, not a promise of zero performance cost.”
Step-by-step analysis
1. Use field IDs for column identity
Iceberg assigns each field an ID that is never reused in a table. A rename changes the name while readers still find the original field by ID. Re-adding a dropped name gets a new ID, so values from old files cannot silently come back. Position-based formats cannot safely handle deletes and reorders; name reuse can also mis-map data.
Old schema: id=17, name="customer_id"
New schema: id=17, name="account_id"
Added field: id=42, name="region"2. Separate Schema ID from field ID
A field ID answers “which column is this?” A Schema ID answers “which version of the table structure is this?” An evolution creates a new schema object and makes its Schema ID current; snapshots record the schema used when they were written. Readers cannot safely rely on a cached array of column names.
3. Decide which schema changes are safe
Add, drop, rename, reorder, and selected widening operations are supported, but not every type change is safe. Check format version, value range, and partition transforms. A field used by a bucket transform may not be promotable when the transform result changes. Map-key structural changes also have equality constraints.
Safe candidate: add an optional field, rename a non-partition field, int -> long when transform output is unchanged
Block or redesign: narrowing a type, changing bucket input semantics, dropping a field required by critical readers4. Let old and new partition specs coexist
Partition evolution creates a new Partition Spec ID. Old files retain the old spec while new files use the new default. A reader must interpret each file using its spec and combine the results. Hidden partitioning lets queries express predicates on data values instead of hard-coding a date directory.
5. Evaluate “no rewrite” against query performance
Metadata evolution avoids rewriting data files, lowering migration cost, but old files still have the old physical layout. Multiple specs can create multiple split plans with different pruning quality. Compare files scanned, planning time, bytes read, task skew, and small-file counts. If the old layout remains a hotspot, schedule a bounded rewrite rather than pretending logical evolution reorganized physical data.
6. Protect publication with atomic commits and snapshots
Table state is represented by metadata files and snapshots; an update atomically replaces the current metadata pointer. Read the current version, build new metadata from it, and commit. On conflict, refresh and retry. Tie the change record to a snapshot ID so a bad rollout can return to a verified snapshot while preserving a compatibility window for old readers.
High-quality sample answer
I would separate column identity, structural version, and physical layout. Field IDs prevent a rename or reorder from mapping old-file values incorrectly; a Schema ID records the structural version; a Partition Spec ID records the partition transform. Adding region or renaming customer_id is metadata work, not a file rewrite, but I would first verify that every engine reads by field ID.
For monthly-to-daily partitioning, I would create a new spec and use it for new writes while retaining the old spec on existing files. The planner must apply each spec’s partition expression and merge the splits. Before publishing, I would run an old-reader/new-reader matrix, compare values across the rename, measure files and bytes scanned, and commit from an isolated branch or catalog transaction. If conflicts or query regressions appear, roll back by snapshot; if old layout remains slow, run a budgeted rewrite later.
Common mistakes and improvements
- Mistake → Treat a column rename as a file-position change → Why it fails → Position binding can attach the wrong value → Improvement → Explain stable field identity and verify engine mapping.
- Mistake → Scan only new directories after partition evolution → Why it fails → Old files remain part of the table and results can be incomplete → Improvement → Keep every Partition Spec and use spec-aware planning.
- Mistake → Treat “no rewrite” as “no performance cost” → Why it fails → Multiple splits and old layout can still increase scans → Improvement → Measure planning, files, bytes, and skew; rewrite in controlled batches if needed.
- Mistake → Overwrite current metadata directly → Why it fails → Concurrent commits or retries can lose updates → Improvement → Commit atomically from a read version, refresh on conflict, and retain rollback points.
Follow-up questions and responses
Why can’t you drop a field and then reuse the same name?
You can add the name again, but it must receive a new field ID. Reusing the old ID could make values from the old file appear as the new field, violating drop semantics. Test old files, new files, and the ID assigned to the re-created name.
Is int-to-long promotion always safe?
No. Check format version, value range, downstream types, and whether the field feeds a partition transform. If a bucket or another transform changes its output, old and new partition semantics may diverge; block the change or design a new spec first.
Can monthly old files and daily new files cause missed rows?
Not in a correct implementation. The reader interprets each file with its Partition Spec, prunes within that spec, and combines the results. Test range queries crossing the evolution boundary and compare the row set with a full-scan reference.
When do you still rewrite data files?
Rewrite when old layout causes persistent scans, skew, small files, or storage cost. Schema or partition metadata evolution itself does not require it. Give a rewrite snapshots, concurrency controls, and a budget so a physical cleanup remains reversible.