Prompt and scope
An event lake receives webhooks from several vendors. Core fields are stable, extensions change often, and some events contain dates, timestamps, binary values, and decimals. Design an Iceberg v3 storage model, compare Variant with JSON strings and structs, and give a migration and acceptance plan.
This tests semi-structured data modeling, not a decision to put every field in Variant. The Iceberg spec defines Variant as a value whose structure and types may vary across rows and files, with richer primitives than JSON. It is a v3 capability, so format version and reader compatibility are part of the design.
What the interviewer is testing
A strong answer promotes stable, frequently filtered fields to top-level columns and keeps low-frequency, fast-changing extensions in Variant. It distinguishes Variant arrays and objects from fixed-type lists and structs, then discusses statistics, predicate pushdown, projection cost, and engine support.
The interviewer also looks for data contracts, naming, type conflicts, privacy, backfills, and fallback paths. A great answer provides a dual-write or view strategy from raw payload to canonical columns and states what happens when an older reader cannot understand v3.
Clarifying questions
Which fields are keys and filters
Confirm whether tenant ID, event type, event time, and idempotency key are stable and queried often. They should be typed columns, not paths parsed from Variant on every query.
What query guarantee do extensions need
For audit replay, Variant can preserve the original shape. For low-latency aggregates or partition pruning, validated paths should become columns or materialized views.
Do all readers support v3
Inventory Spark, Flink, Trino, service SDKs, and export jobs. If a v2 reader remains, define a JSON compatibility view, an isolated v3 table, or a delayed upgrade.
A 30-second answer
“I keep stable, frequently filtered fields as typed columns and put only fast-changing, low-frequency extensions in Variant. Variant preserves more types than a JSON string, but it does not automatically provide column statistics or pushdown. I use a path registry, quality rules, and materialized columns to control cost. Before upgrading to v3 I inventory readers, give v2 jobs a compatibility view with explicit precision loss, and measure scan bytes, latency, type conflicts, and backfill success.”
Step-by-step solution
Step 1: Separate canonical columns from extensions
Put tenant ID, event name, event time, source, and idempotency key in top-level struct fields with consistent types, optionality, and field IDs. Put vendor-specific low-frequency objects in Variant, retaining source version and raw event ID for replay and audit.
Step 2: Compare the three representations
A fixed struct fits stable schemas, typed computation, and column statistics. A JSON string is broadly compatible but reparsed for every query, and date, decimal, and binary semantics depend on the parser. Variant allows changing objects and arrays plus richer primitives, at the cost of engine support, statistics, and governance.
Step 3: Define a Variant contract
Create a registry for allowed paths: path, expected type, sensitivity, owning team, first-seen version, and whether it is eligible for promotion. Reject or quarantine unknown high-risk types instead of silently converting them to strings.
event_id: string
event_time: timestamptz
payload: variant
payload_registry:
vendor.order.total: decimal(18,2)
vendor.order.shipped_at: timestamptzThe registry governs data quality; it should not hard-code every Variant path into the table schema. Promote a path through schema evolution or a materialized column only after it becomes a core query dimension.
Step 4: Control query cost
Avoid unbounded wildcard traversal of Variant in large scans. Build projected views or materialized columns for stable paths, partition by event type and time, and record scan bytes, parse CPU, and hit rate. Sample and profile unknown paths offline before promoting them.
Step 5: Handle versions and readers
Variant is allowed in Iceberg v3. Before release, check each reader's format version, Parquet or Avro mapping, and SDK support. A v2 job can use a compatibility view that serializes Variant as JSON, but the view must document type and precision loss and fields that are no longer efficiently queryable.
Step 6: Design backfills, conflicts, and privacy
Backfill a new canonical column from Variant while retaining raw payload and transformation version. If one path changes from a decimal to a string, do not overwrite silently: version the path, emit a conflict metric, and quarantine invalid records when needed. Apply field-level masking, deletion handling, and access audit to Variant too.
Step 7: Build an acceptance matrix
Test nulls, mixed types, deep arrays, time zones, precision, unknown fields, old readers, concurrent writes, and retries. Track scan bytes, parse CPU, p95 latency, type-conflict rate, replay consistency, and v2/v3 reader success rate.
Model high-quality answer
I would not store every webhook as a JSON string. Tenant, event type, time, and idempotency key become typed top-level columns; fast-changing vendor extensions go into Variant under a registry of paths, types, and sensitivity. Variant preserves dates, timestamps, and decimals better than a string, but I would not assume every engine can push predicates into it efficiently.
Before upgrading to Iceberg v3, I inventory readers and provide v2 jobs with a JSON compatibility view that records precision loss. The query layer materializes high-value paths and profiles unknown ones. Type conflicts enter a quarantine stream, and backfills retain versions and raw events. I accept the design only after measuring scan bytes, p95 latency, conflict rate, replay consistency, and cross-engine success.
Common mistakes
- Symptom → Put every field in Variant → Why it fails → Core filters lose typed statistics and query cost becomes unpredictable → Fix → Promote stable fields and reserve Variant for changing extensions.
- Symptom → Treat Variant as a JSON string → Why it fails → Dates, decimals, and binary values lose type semantics → Fix → Preserve primitives under a registry.
- Symptom → Upgrade to v3 without reader testing → Why it fails → Older engines may fail to read or silently degrade → Fix → Build a version matrix and compatibility view.
- Symptom → Force a type conflict to string → Why it fails → Downstream aggregates and constraints break → Fix → Version the path or quarantine and measure conflicts.
- Symptom → Overwrite the raw payload during backfill → Why it fails → Conversion differences cannot be replayed or audited → Fix → Retain raw events, transformation versions, and idempotent jobs.
Follow-up questions and responses
Follow-up 1: Why not keep a JSON string and parse it at query time?
Strings maximize compatibility, but every query pays parsing cost and type precision depends on the parser. That can be acceptable for audit-only replay. Aggregates, filters, and cross-engine consistency benefit from moving the type contract into Variant or canonical columns.
Follow-up 2: A path is numeric today and a string tomorrow. What do you do?
Use the registry to reject or quarantine the write and record the vendor version. If both types are legitimate, version the path or define an explicit union; do not make the query engine guess.
Follow-up 3: An old reader only supports Iceberg v2. How do you migrate?
Keep a v2-compatible table or view that serializes Variant as JSON and documents precision loss. Shadow-read v3 with new readers, then cut jobs over gradually. Every job declares its minimum format version in the upgrade gate.
Follow-up 4: When should a Variant path become a top-level column?
Promote it when the path is stable, frequently accessed, low-conflict, and profiling shows less scanning or parsing. Keep the raw Variant for a verification and replay window before considering cleanup.