Prompt and setting
The same usage event feeds operational limits and financial reporting. Events can arrive late, duplicate, or be corrected, and one noisy tenant must not delay everyone else or read another tenant’s data.
What the interviewer tests
- Separating immutable facts from mutable projections.
- Defining idempotency, event-time windows, corrections, and invoice finality.
- Designing tenant isolation, backpressure, and reconciliation evidence.
Clarifying questions before answering
- What are the event rate, dimensions, retention, and invoice finalization deadline?
- Is usage measured at request acceptance, completion, or successful business effect?
- Can producers resend an event ID, and how are corrections or refunds represented?
- Which quota decisions need seconds-level freshness, and which reports may be delayed?
30-second answer framework
I would capture a tenant-scoped immutable usage fact with a stable event ID, event time, unit, source, and schema version. An append-only log feeds separate quota and billing projections. Deduplication is keyed by tenant and event ID; corrections are new facts, not overwrites. Quota reads use a fast bounded projection, while invoices close only after a watermark and reconciliation pass. Every aggregate remains traceable to source facts.
Step-by-step deep dive
1. Capture the fact
Authenticate the producer, validate tenant scope and units, and persist the raw event before acknowledging. Require a producer event ID, source, event time, and measurement version. Reject malformed or cross-tenant writes and retain a digest for audit.
2. Make writes idempotent
Enforce a uniqueness constraint on (tenant, source, event_id). A duplicate with the same digest returns the existing outcome; a different digest is quarantined as a conflict. Do not use an invoice row as the idempotency store because operational retries happen before invoicing.
3. Project quotas and billing separately
The quota projection keeps short windows, current counters, and a fail-closed policy when freshness is unknown. Billing aggregates by the contractual unit and price version, preserving event-time buckets and correction links. Neither projection edits the immutable fact log.
4. Handle late data and close invoices
Use a watermark based on observed event time plus an agreed lateness bound. Events after the watermark enter an adjustment ledger and the next billing cycle or credit workflow. Before finalization, reconcile counts and sums from the log against projections and record the calculation version.
5. Operate safely
Partition queues and storage by tenant or fair-share key, apply quotas and backpressure, and isolate dead letters. Monitor ingestion lag, duplicate and conflict rates, watermark age, projection drift, correction volume, and invoice-close latency. Encrypt data, restrict exports, and make every correction auditable.
High-quality sample answer
“I would store an immutable, tenant-scoped usage fact with event ID, source, event time, unit, schema and price version before acknowledging. An append-only log feeds independent quota and billing projections. Same-ID same-digest retries are idempotent; a changed digest is quarantined. Quotas use a fast bounded projection, while invoices close only after a lateness watermark and reconciliation against source facts. Late events become adjustment records, and tenant partitioning, backpressure, encryption and drift metrics protect correctness and fairness.”
Common mistakes
- Overwrite the aggregate on every event → late data and corrections become invisible → retain immutable facts and projections.
- Use wall-clock arrival as usage time → delayed events land in the wrong invoice → bucket by event time and watermark.
- Share one global queue → a noisy tenant creates head-of-line blocking → partition or enforce fair share.
- Finalize without reconciliation → projection drift becomes a billing dispute → compare projections with source facts and version the calculation.
Follow-up questions and responses
How do you support a refund or correction?
Append a compensating fact linked to the original event and contract version. Recompute the affected projection or create an adjustment ledger entry; never mutate the original evidence.
What if quota and billing disagree temporarily?
Declare separate freshness and finality guarantees. Quota can be approximate within a bounded window, while billing remains provisional until watermark and reconciliation checks pass; expose the state to operators and customers.
How do you prevent a tenant from reading another tenant’s usage?
Enforce tenant scope in the write path, storage partitions, projection queries, exports, and support tooling. Test authorization at each boundary and log access without including sensitive event payloads.