Prompt and context
You own a data platform that ingests user events for several real-time consumers. Traffic is steady most of the day and spikes during campaigns. The business cannot tolerate sustained throttling, but it does not want to prepay capacity for the trough. Assume you can read producer throughput, read latency, throttling, and consumer lag, and can switch capacity mode during a maintenance window.
What the interviewer evaluates
The interviewer is testing whether you understand that capacity mode changes operating responsibility and cost, not delivery semantics. A strong answer uses historical peaks and burst shape to decide whether automatic scaling is valuable, then checks per-shard limits, consumer count, retries, and lag recovery. A weak answer simply says to choose on-demand whenever traffic varies.
Clarifications to ask first
- How long do peaks last, and are they predictable? Short, unpredictable spikes favor automated capacity management.
- Are writes and reads constrained at the same time? Measure bytes, records, reads, and consumer behavior separately.
- Is there a hard cost ceiling or capacity budget? Stable traffic with a tight budget is easier to optimize in provisioned mode.
- Do consumers share throughput or need dedicated reads? Multiple consumers change read quotas and cost.
- Can the team tolerate a short state transition? Switching and burst retries need an explicit runbook.
30-second answer framework
“I would measure hourly write MB/s, record rate, read MB/s, lag, and throttling, separating predictable peaks from bursts. Stable, predictable traffic can use provisioned capacity with a scaling schedule; fast or unpredictable changes favor on-demand, but I would validate ramp-up, quotas, and cost. Either choice must pass guardrails for throttling, lag recovery, end-to-end latency, and monthly spend.”
Step-by-step deep answer
- Build a capacity baseline. Aggregate writes and reads by minute, record P50, P95, P99, peak duration, and hot partition keys; averages hide bursts.
- Check throughput limits. AWS documents a default shard limit of 1 MB/s or 1,000 records per second for writes and 2 MB/s for reads. Convert record size and batching into both bytes and records.
- Choose a mode. Use provisioned capacity with a scaling plan for stable load; prefer on-demand when load changes quickly or is hard to predict, and record automatic-adjustment latency.
- Model consumers. Shared reads, enhanced fan-out, retries, and duplicate processing affect read quotas. Consumer lag belongs in capacity planning, not only producer metrics.
- Model cost. For provisioned mode, include shard-hours and scaling headroom. For on-demand, include actual throughput and peak billing, plus replay and burst double-writes.
- Pilot and roll back. Switch one noncritical stream, watch throttling, lag recovery, P99 latency, and monthly spend. If a guardrail fails, return to the validated mode while preserving ordering and retry behavior.
Alternatives include splitting hot partition keys, batching writes, shrinking events, buffering with Firehose, or pre-scaling for predictable campaigns. Capacity mode cannot repair skewed keys, slow consumers, or infinite retries.
Model answer
“I would inspect 30 days of per-minute write and read curves, calculate P95/P99, peak duration, and partition-key skew. I would translate record size through AWS shard limits into write MB/s and record rate, then check shared consumer throughput and lag recovery. For predictable peaks lasting hours, I would use provisioned capacity and pre-scale before campaigns. For short, unpredictable peaks, I would pilot on-demand on a noncritical stream with throttling below 0.1%, lag-recovery p99 under 5 minutes, end-to-end latency within 20% of baseline, and a monthly cost cap. I would monitor hot partitions and duplicates in both modes; changing capacity mode must not hide the root cause.”
Common mistakes
- Mistake: Estimate from average throughput → Why it fails: Bursts and hot keys cause local throttling → Fix: Use P95/P99, peak duration, and key distribution.
- Mistake: Treat on-demand as unlimited throughput → Why it fails: Service quotas and ramp-up still matter → Fix: Validate ramp behavior, quotas, and burst tests.
- Mistake: Count only producer cost → Why it fails: Consumers, retries, and replay amplify throughput → Fix: Build end-to-end cost scenarios.
- Mistake: Ignore delivery semantics → Why it fails: Capacity mode does not remove at-least-once and duplicate-processing risks → Fix: Keep idempotency, checkpoints, and lag recovery.
Follow-up questions and responses
On-demand still throttles; what do you tune first?
Separate total-throughput shortage, hot partition keys, and a lagging consumer. Check record rate, key distribution, and scaling events before choosing backoff, repartitioning, or more consumers.
When is provisioned capacity cheaper?
When writes and reads are stable, peaks can be scheduled, and utilization stays high, model shard-hours and scaling headroom. Do not compare only the unit price.
What if campaign traffic grows tenfold?
Verify on-demand quotas and historical ramp behavior first. Pre-scale predictable campaigns, add producer backoff and lag alerts, and define degradation for noncritical events.
How do you prove the choice is correct?
Compare throttling, lag-recovery p99, end-to-end latency, duplicate rate, and monthly cost before and after the pilot with the same definitions. Expand after two business cycles meet the guardrails.