Prompt and scope
This is a common design question for data-platform, real-time analytics, and senior data-engineering roles. Assume a peak of 50,000 events per second, a 60-second dashboard freshness target, and a report for the previous day due at 07:00. The interview is testing when a shared execution graph forces every consumer to inherit the strictest SLO, and how independent progress and resource pools reduce coupling.
What the interviewer is evaluating
- Whether you turn “real time” and “on time” into end-to-end SLOs before naming tools.
- Whether you distinguish branches inside one Beam graph, independent subscriptions to one source, and fully independent compute resources.
- Whether you can explain the trade-offs among duplicate reads, cost, backpressure, replay, late events, and failure domains.
- Whether you provide metrics, a canary, rollback, and reconciliation that prove the split improves user outcomes.
Clarifying questions to ask first
Ask whether 60 seconds means event-time-to-queryable freshness or only processor latency; whether the finance report may backfill late partitions; whether both consumers can share immutable raw storage; whether events need tenant or priority fairness; what duplicate, loss, and ordering guarantees are required; and whether the budget favors cost or dashboard tail latency. If the report is T+1 but the dashboard has a hard low-latency target, isolation is already a credible default to evaluate.
A 30-second answer structure
I would write down the two consumers’ end-to-end SLOs, correctness guarantees, and recovery targets. If one shared pipeline must satisfy the strict 60-second target, I would establish a shared baseline, then use independent subscriptions to separate real-time and batch progress. I would share expensive common parsing, but split compute when resource pressure or failure propagation is material. Each path gets its own freshness, backlog, lateness, duplicate, and report-completion metrics, and replay and fault injection decide whether the extra isolation is worth the cost.
Deep-dive answer
1. Derive structure from SLOs
Define the real-time SLO as p99 event-time-to-queryable freshness under 60 seconds. Define the batch SLO as completing valid previous-day events by 07:00, with late data entering a bounded repair window. Google’s Dataflow guidance notes that a single pipeline serving mixed SLOs must meet the stricter target, which lets lower-priority work consume real-time capacity. Different error budgets, alert routes, or autoscaling policies are strong reasons to separate paths.
2. Compare three topologies
An in-graph branch is useful for shared decoding and lightweight routing. Apache Beam documents that multiple transforms can read the same PCollection, but each transform processes the input again; a single multi-output transform can process each element once for common work.
Independent subscriptions to one topic let real-time and batch consumers own separate acknowledgements, backlog, and replay positions. Google’s Dataflow guidance describes multiple pipelines using separate subscriptions so each job pulls and acknowledges independently. Fully independent jobs additionally isolate CPU, memory, release cadence, and failure domains, at the cost of duplicate reads, serialization, and operational ownership.
3. Design the recommended path
Keep one immutable raw-event layer and create a real-time subscription plus a batch subscription from the same source. The real-time job performs lightweight aggregation into a low-latency serving store. The batch job reads retained data by event date and writes partitioned tables. If common parsing is more than 20% of total CPU, normalize it once at ingress and write versioned events to the raw layer; do not treat the real-time consumer’s progress as proof that the batch path has committed.
raw-events
-> realtime-subscription -> stream-aggregate -> serving-store
-> batch-subscription or retained-raw -> daily-transform -> partitioned-lake4. Handle backpressure, priority, and cost
Give the real-time path its own concurrency cap and backlog alert. Let batch reduce concurrency when real-time resources are tight, but do not put both behind one unbounded queue. If two complete computations are too expensive, share raw decoding and landing, then isolate downstream stages. When real-time backlog exceeds 60 seconds, stop expanding batch capacity and recover the real-time SLO first. Attribute input bytes, CPU, backlog age, and cost per million events to each path rather than comparing job counts.
5. Handle lateness, replay, and recovery
Use a watermark and bounded allowed lateness for real-time windows. Route events beyond the window to a late-data queue or raw layer, then let batch repair affected partitions with a versioned, idempotent write. Replay from a saved source position by creating a new subscription; never rewind the production consumer’s acknowledgement position. A batch path should still recover from raw data after a real-time failure. If raw storage is unavailable, both paths need explicit degradation and alerts.
6. Let an acceptance experiment decide
Run the shared baseline first, then enable isolated resources for a small tenant slice. Compare dashboard p50/p95/p99 freshness, report completion, backlog age, duplicate rate, replay time, CPU, storage, and cost per million events. Inject batch bursts, processor restarts, duplicate messages, late partitions, and paused subscriptions. If the split only improves processor latency while increasing duplicate data or exceeding the cost budget, keep the shared plan and optimize the common stage.
Example of a strong answer
I would define two end-to-end SLOs first: dashboard p99 event-time-to-queryable freshness under 60 seconds, and a previous-day finance report complete by 07:00 with a bounded late-data repair window. I would build a shared baseline but never share the acknowledgement position between real-time and batch. My default design is one immutable raw-event layer, two independent subscriptions, and two downstream jobs. The real-time job performs lightweight aggregation; the batch job reads retained data by event date. Common parsing can be versioned once at ingress, while full jobs are separated only when resources or failure domains require it. Each path owns freshness, backlog, lateness, duplicates, report completion, and unit-cost metrics; replay uses a new subscription and idempotency keys. I would inject bursts, restarts, and late events to verify the SLOs. If isolation does not improve user outcomes, I would keep shared compute and optimize the common stage.
Common mistakes
- Symptom: Copying two complete pipelines just because there are two consumers. Why it fails: Common parsing and landing costs double without improving isolation. Correction: Share immutable raw data first, then isolate downstream compute by SLO.
- Symptom: Driving both consumers with one global offset. Why it fails: A slow consumer blocks the fast one and replay cannot be independent. Correction: Use independent subscriptions or independently verifiable progress.
- Symptom: Measuring processor latency only. Why it fails: Storage, query, and refresh time can still violate the user target. Correction: Measure end-to-end freshness and percentiles.
- Symptom: Writing late events directly into the live result. Why it fails: Counts may duplicate or already-published reports may change silently. Correction: Use a watermark, repair window, version, and idempotency key.
- Symptom: Rewinding the production consumer to replay history. Why it fails: Online progress is disturbed and traffic can amplify. Correction: Create a rate-limited replay subscription from retained data.
Follow-up questions
What if both paths need the same expensive feature computation?
Make the feature stage versioned and replayable, land its output once, and let both paths read it. Accept duplicate computation only when state must stay online and cannot be shared. Compare a shared intermediate layer with duplicated compute using CPU, latency, and consistency experiments.
Will independent subscriptions double input cost?
They add read and acknowledgement overhead, but immutable landing, compression, retention windows, and on-demand replay can control it. Evaluate cost per million events together with real-time SLO and failure isolation; storage cost alone is incomplete.
Can batch borrow real-time capacity when it falls behind?
Use bounded, preemptible low-priority capacity with a lease and automatic reclamation. Real-time keeps a hard cap and an independent backlog metric, so borrowed capacity cannot push its p99 beyond 60 seconds.
How do you prove the two paths eventually agree?
Build reconciliation sets from the same event version, business key, and time boundary. Compare counts, amounts, missing records, duplicates, and late corrections. If real-time output is approximate, define the convergence window and explainable difference budget instead of treating one equal total as proof.