Data engineering interview: How can Iceberg v3 nanosecond timestamps avoid cross-engine distortion?

Prompt and scope

An event lake must retain capture timestamps at nanosecond precision. The table uses Iceberg v3 and is read by batch, streaming, and BI engines. Explain the distinction between timestamp types with and without a timezone, write-time checks, old-reader compatibility, and a reversible migration.

What the interviewer is testing

• Knowing that v3 adds timestamp_ns and timestamptz_ns, rather than renaming a millisecond field.
• Distinguishing civil time from an absolute instant and handling offsets correctly.
• Recognizing truncation, negative times, and serialization risks.
• Designing a cross-engine capability matrix and a rollback path.

Clarifying questions

1. Is nanosecond precision required for ordering, or only for audit display?
2. Does the value represent one global instant or a local calendar value?
3. Which Iceberg v3 versions do writers, catalog services, and readers support?
4. If an old system reads only milliseconds, may it use an explicitly degraded column?

A 30-second answer

Choose by meaning: timestamp_ns has no timezone, while timestamptz_ns represents an absolute instant with a +00:00 offset. Reject implicit truncation at the writer, normalize input to canonical ISO-8601 or an explicit epoch-nanosecond value, and validate range, sign, and rounding. Build a capability matrix before migration; readers without v3 support should use a compatibility view or a dual-written millisecond column, not guess at the new type. Prove the result with replay, cross-timezone, and boundary tests.

Step-by-step design

1. Establish time semantics

Birthdays and business dates are local calendar values; logs, trades, and trace spans usually represent one instant worldwide and should use timestamptz_ns. A field named created_at does not prove UTC, and a local offset must not be silently discarded.

2. Preserve the precision boundary

Parse input and retain all nine fractional digits. Do not pass through JavaScript Date or a millisecond integer first. Compare serialized and parsed values with a round trip so the nanosecond component remains unchanged.

stored_ns = parse(input)
assert format(parse(format(stored_ns))) == stored_ns

3. Handle timezone and outliers

The timezone-aware type requires a canonical offset; store a UTC representation and convert only at display time. Test negative epochs before 1970, the leap-second policy, daylight-saving boundaries, and values outside the implementation range. Reject ambiguous local strings or require the caller to provide a zone.

4. Build a compatibility matrix

Cover the catalog, writers, batch readers, streaming readers, and BI engines. Record whether each recognizes v3, preserves nanoseconds, and fails or ignores unknown types. Do not infer behavior from a library version alone; run a minimal read/write table through every engine.

5. Plan migration and degradation

Write both nanosecond types to an isolated table and replay real samples. If an old reader cannot support v3, expose an explicit millisecond-derived column with a precision disclaimer; never pretend it is nanosecond data. Use dual writes, equality checks, and per-engine cutover, with a switch back to the old table or column.

6. Monitor data quality

Track truncation, parse failures, missing zones, negative-value ratios, and cross-engine round-trip differences. Sample raw events, Iceberg files, and query results; record schema version, writer version, and timezone policy in audit metadata.

Model high-quality answer

I would first establish whether the value is an absolute instant. Absolute events use timestamptz_ns; local calendar values use timestamp_ns. The writer keeps a nanosecond representation end to end, rejects millisecond APIs, and tests nine-digit round trips, negative epochs, daylight-saving boundaries, and range limits. Before migration I build a v3 capability matrix. Older readers use an explicit millisecond-derived column or compatibility view, never silent truncation. I dual-write, cut over engine by engine, monitor loss and divergence, and retain a switch back to the old table or column.

Common mistakes

• Treating both types as UTC → changes local-calendar meaning → establish semantics first.
• Converting to milliseconds before writing v3 → precision is already gone → keep nanoseconds end to end.
• Removing a +08:00 offset → merges distinct instants → normalize to UTC, then display locally.
• Testing only a successful write → old readers may fail or truncate → test the whole matrix and replay.
• Trusting a version number → format support may differ → run minimal cross-engine reads and writes.

Follow-up questions and responses

Can timestamp_ns replace timestamptz_ns?

No. The first carries no timezone and fits defined calendar semantics; the second expresses an offset-bearing instant. Replacing one changes business meaning, not only precision.

What if an old engine cannot read a v3 table?

Determine whether it rejects unknown types or can read other columns. In production, use a compatibility view or a dual-written millisecond column with an explicit precision downgrade; do not make the old engine guess.

Does nanosecond ordering equal causal ordering?

No. Values from clocks on different machines can be skewed. Ordering still needs event IDs, source sequence numbers, or a logical clock; the timestamp is an observation.