Prompt and scope
A company emits application text logs, container stdout, and legacy JSON events with different field names, time precision, and severity conventions. It wants the OpenTelemetry Logs Data Model while preserving historical search and avoiding misleading TraceId links. Design an offline backfill plus real-time dual-write plan.
OpenTelemetry separates Timestamp, ObservedTimestamp, TraceId, SpanId, SeverityNumber, Body, Resource, and Attributes. The migration is a traceable data contract, not a decision to put every line into Body. A strong answer handles parser failures, time zones, duplicates, sensitive fields, and replay watermarks.
What the interviewer evaluates
- Distinguish event time, observation time, resource attributes, and event attributes.
- Design schema mappings, versions, unknown-field retention, and parse-failure paths.
- Preserve the truth and optionality of TraceId, SpanId, and request identifiers.
- Handle tenant isolation, redaction, duplicates, reordering, and backfill cost.
- Give replayable, reconcilable quality gates rather than naming an ETL tool.
Clarifying questions
- Are legacy timestamps local, UTC, or mixed, and are they millisecond or nanosecond precise?
- Which sources have stable schemas, and which require regex or sample-driven parsing?
- Are TraceId, SpanId, and request IDs produced by applications or guessed by collectors?
- Must raw payloads be retained, for how long, and who may read them?
- Do backfill and dual-write share storage and indexes, and how much query divergence is acceptable?
30-second answer
I would create versioned mapping contracts and retain raw payloads with parse status. Timestamp is event time and ObservedTimestamp is observation time; Resource holds stable source facts such as service, host, and tenant, Attributes hold event fields, and Body keeps structured business content. I would populate TraceId and SpanId only from trusted context. Backfill runs alongside dual-write, reconciliation uses source and time partitions, and failures enter replayable dead letters. Gates cover parse success, field completeness, time skew, duplicates, redaction hits, and query equivalence.
Step-by-step solution
1. Fix the data contract first
Define parser version, required fields, defaults, and unknown-field policy for every source. Generate a stable event ID and record the source, file offset, or message position. Unknown fields may stay in Attributes or the raw payload, but must not disappear silently; semantic changes require a mapping-version bump.
{
"timestamp": "2026-08-02T02:00:00.123Z",
"observedTimestamp": "2026-08-02T02:00:00.800Z",
"severityNumber": 17,
"severityText": "ERROR",
"body": {"message": "payment declined", "code": "CARD_DECLINED"},
"resource": {"service.name": "checkout", "tenant.id": "t-7"},
"attributes": {"region": "us-east-1"}
}2. Preserve time semantics
Normalize timezone-aware timestamps and retain the original string and parse status. Timestamp is when the event occurred; ObservedTimestamp is when the collector observed it. If event time is missing, use observation time only as an explicit fallback and mark it, so collection delay is not mistaken for business latency. Validate future times, excessive age, and precision truncation.
3. Map Resource, Attributes, and Body
Resource describes the entity producing logs, such as service, version, host, cluster, and tenant. Attributes describe an event instance, such as region, request type, or experiment group. Body contains structured content or an unparsed message. Misplacing fields changes aggregation, indexing, and cost, so mappings should record reasons and downstream consumers.
4. Map severity and context
Map legacy WARN, ERR, and numeric levels to SeverityNumber while retaining original SeverityText. Accept TraceId, SpanId, and TraceFlags only when format and injection point are trusted. Keep missing context empty with a reason; never invent a trace ID. A request ID may be an ordinary Attribute but must not be presented as TraceId.
5. Design dual-write and backfill
The real-time path writes old and new stores with the same event ID. The offline path slices files, partitions, or message positions and records checkpoints. Both paths share parsers and redaction rules but may use different batch sizes. After backfill, reconcile by source, time window, and event ID, then gradually shift queries to the new model.
6. Handle failure, duplicates, and reordering
Write parse failures to dead letters containing raw data, error code, and parser version; replay them from a checkpoint after a fix. Deduplicate with event ID plus source position and content hash. Do not rewrite event time because records arrive out of order; let indexes support event and observation time separately. If identity is uncertain, mark it instead of silently overwriting.
7. Isolation, redaction, and cost
Take tenant ID from a trusted Resource attribute, not arbitrary client input. Redact secrets, tokens, and personal data before persistence, recording redaction version and hit counts. Encrypt raw payloads separately with restricted access and shorter retention. Budget indexes for high-cardinality Attributes so one normalized model does not create runaway cost.
8. Quality gates and rollback
Sample old and new queries and compare event counts, severity distribution, time skew, and critical fields. Gate on parse success, required-field completeness, duplicates, time skew, redaction misses, and query equivalence. Keep old writes during canary; on field drift or tenant leakage, stop new writes and route queries back using checkpoints without deleting replayable raw data.
Sample strong answer
I would separate the migration into contract, parsing, dual-write, backfill, reconciliation, and cutover. Every source gets a versioned parser and stable event ID. Timestamp is event time and ObservedTimestamp is collection time. Resource holds stable service, host, and tenant facts; Attributes hold event fields; Body keeps structured content; SeverityNumber maps old levels; TraceId is accepted only from trusted context.
During the live phase, old and new stores receive the same events. Backfill runs by position, and failures go to dead letters with raw data and error codes. Reconcile by event ID and position while tracking reorder and duplicate rates. Redact before persistence and isolate raw payloads. Canary on event counts, completeness, time skew, and query results; fail gates stop the new writer and restore old queries.
Common mistakes
- Put everything in Body → downstream loses resource and attribute semantics → map fields by the data model and retain unknowns.
- Replace event time with collection time → business latency becomes unmeasurable → retain both Timestamp and ObservedTimestamp.
- Invent TraceId when context is missing → creates fake traces → keep it empty and record why.
- Dual-write without reconciliation → history and live results cannot be proven equivalent → reconcile by position, event ID, and time window.
- Drop parse failures → parser fixes cannot repair history → use replayable dead letters.
- Persist before redaction → raw data has a larger exposure window → redact at a controlled ingress and isolate originals.
Follow-up questions and responses
What if there is no event timestamp?
Use ObservedTimestamp as an explicit fallback with a missing-time marker. Do not present it as business time; report it separately in quality metrics.
How do you prove TraceId was not fabricated?
Trust only application SDK or controlled-proxy context, validate format and scope, and treat a same-named client field as an ordinary Attribute.
How do you handle duplicates from dual-write?
Generate a stable event ID and use source position plus content hash for idempotent writes. If identity remains uncertain, preserve a duplicate marker and explain it at query time.
What if field meaning changes during migration?
Bump parser and schema versions, retain old mappings and version metadata, allow a compatibility window, and test downstream queries against both versions.
Why retain raw payloads?
They enable parser fixes, disputes, and replay. Encrypt and restrict them, audit access, shorten retention, and keep them separate from normalized indexes.