How Do You Handle Late and Out-of-Order Events in a Streaming Aggregation?

1. Question

An order-event stream aggregates amount and order count by customerId in five-minute windows. Device clocks may drift, network retries create out-of-order delivery, and some Kafka partitions may temporarily have no new messages. Results should appear quickly while allowing late data to correct a window for a bounded period. Design the event-time handling.

2. Constraints and clarifications

• Confirm whether windows use event time, write time, or processing time; this problem uses event time.
• Set a maximum disorder bound, such as 30 seconds, and define what happens beyond it.
• Decide whether results are append-only events or updateable snapshots; downstream consumers must identify revisions.
• Discuss idle partitions, invalid timestamps, duplicate events, and replay jobs so that no-message input is not mistaken for end-of-stream.

3. Core concepts

Event time comes from the record itself. A watermark says that the system believes event time has advanced to a position. A window commonly fires when the watermark passes its end; a record arriving after that watermark whose timestamp still belongs to the window is late. When parallel inputs are combined, an operator commonly waits for the minimum input watermark, so an inactive partition can hold back global progress. Idle detection or a per-partition timeout policy is therefore required.

4. Reference flow

onRecord(event):
  ts = extractEventTimestamp(event)
  key = canonicalKey(event.customerId)
  updateWatermarkGenerator(ts)

  window = floorToFiveMinutes(ts)
  if ts <= currentWatermark + allowedLateness:
    state[window, key] = aggregate(state[window, key], event)
    emitUpsert(window, key, state[window, key], revision + 1)
  else:
    routeToLateData(window, key, event)

onWatermark(wm):
  finalizeWindowsBefore(wm)
  expireStateAfterRetention()

The source extracts event timestamps and generates a bounded-out-of-orderness watermark. Mark a partition idle after a prolonged period without data so it cannot hold back the merged watermark. Retain window state through the allowed-lateness period; update results with an upsert or correction event. Route data beyond the deadline to a side output for review or offline backfill.

5. Accuracy and latency trade-offs

Longer allowed lateness makes the event-time view more complete but increases state and correction volume. Emitting one final result minimizes downstream latency but cannot represent late revisions. Update events carrying a window key, revision, and reason require downstream idempotent merges. For non-loss-tolerant measures such as money, retain raw events and schedule offline recomputation; for a real-time leaderboard, it may be acceptable to refresh only in the next batch after the deadline.

6. Verification and observability

• Generate ordered, out-of-order, sub-30-second-late, and beyond-deadline events; compare each window with an offline baseline.
• Inject idle partitions, clock jumps, duplicate events, and task restarts; verify that watermarks do not stall or move backward.
• Record current watermark, processing-time versus event-time lag, window-state size, late-event rate, side-output volume, and correction count.
• Include window key, revision, and input event ID in every update; replay the same log and compare final snapshots.

7. Common mistakes

• Replacing event-time windows with processing-time windows while claiming resistance to network disorder.
• Treating a watermark as an absolute completion marker; in practice it can be a heuristic based on a lateness assumption.
• Ignoring idle partitions, allowing a silent partition to block every window from closing.
• Dropping late data without a side output, version, or offline backfill path.

8. Interview scoring points

Distinguishes the three notions of time

The candidate should define event, ingestion, and processing time and explain how window choice changes accuracy and latency.

Generates and merges watermarks

The answer should cover bounded out-of-orderness, the minimum watermark across parallel inputs, and idle handling, while noting that a watermark is progress estimation rather than an absolute promise.

Designs late-data and correction paths

The candidate should define an allowed-lateness deadline, side output, revision, and downstream idempotent merge for both in-deadline and post-deadline data.

Verifies the result with replay

The candidate should test disorder, idleness, restarts, and duplicates against an offline baseline and live metrics, rather than checking only whether the job stays running.