Data Engineering Interview: How Do You Diagnose and Fix a Hot Kafka Partition?

Prompt and When It Applies

A Kafka topic has 24 partitions and receives 120,000 records per second at peak. One tenant produces 45% of the traffic, and the producer partitions by tenant_id, so one partition keeps accumulating lag while most other consumers are idle. One consumer can sustain 8,000 records per second with the current handler. The business requires ordering only within an order_id, not across every event from the tenant, and partitions cannot be added during the incident.

Explain how you would prove that key skew, rather than a consumer, broker, or downstream failure, caused the symptom. Then describe how you would reduce backlog growth today, redesign the partition key and migration, and commit offsets safely after adding asynchronous processing. Finish with the metrics and failure tests that demonstrate the fix.

This is a data engineering and streaming-platform troubleshooting question. A current public Kafka interview guide presents essentially the same combination: one hot partition, its consumer falling behind, idle consumers elsewhere, a keyed producer, and no immediate partition-count change. It asks candidates to connect producer partitioning, consumer parallelism, and ordering. Apache Kafka's documentation states that the default producer chooses a partition by hashing a present key. Semantic partitioning preserves locality and order within the chosen partition, but it also concentrates a key's traffic there. Huawei Cloud's Kafka guidance likewise states that one partition can be consumed by only one consumer at a time and that adding partitions temporarily is not a quick way to clear an existing partition's backlog.

The throughput, traffic share, consumer capacity, and ordering scope in this prompt are interview assumptions, not production figures attributed to a particular company.

What the Interviewer Evaluates

The first signal is whether you can move from aggregate metrics to partition-level evidence. Topic-wide lag, average consumer CPU, and consumer count can all hide one hot partition. A strong answer aligns each partition's produce rate, consume rate, lag slope, leader broker, key distribution, and downstream processing time on the same timeline. That separates producer skew from a slow handler, repeated rebalances, or broker resource pressure.

The second signal is whether you quantify the incident:

120,000 × 45% = 54,000 records/second

If the consumer assigned to that partition can process only 8,000 records per second, backlog grows by:

54,000 - 8,000 = 46,000 records/second
46,000 × 600 = 27,600,000 records in 10 minutes

That calculation explains why adding ordinary consumers does not change this partition's ceiling and why changing an alert threshold does not mitigate the incident.

The third signal is whether you identify the real ordering invariant. The current key expands the ordering domain to an entire tenant, while the business needs order only within an order. Changing the key to order_id increases cardinality and distributes new orders, but only if the migration prevents one order from crossing two partitions or topics.

The fourth signal is offset correctness. Once records from one partition run in a worker pool, completion order can differ from offset order. If offset 105 finishes while 104 is still running, committing through 105 can skip 104 after a crash. A strong design tracks the highest contiguous completed offset and makes downstream effects idempotent because completed but uncommitted records can be replayed.

The final signal is whether you state the hard boundary. More partitions create more parallel slots, but they do not split a giant entity that remains bound to one key. More consumers in a traditional consumer group do not let two consumers own the same partition simultaneously. If one order alone exceeds safe single-partition capacity and must remain strictly ordered, the remaining levers are optimizing the serial path, throttling it, or redefining independent business sequences.

Questions to Clarify Before Answering

• Is ordering required per tenant, per order, or for some smaller event stream? If tenant-wide order is mandatory, splitting a tenant is invalid. If order-wide order is sufficient, order_id is the more accurate partition boundary.
• Does 45% describe record count, bytes, or processing cost? Large records or expensive downstream writes can create cost skew even when record counts appear balanced. Inspect records, bytes, and handler time.
• Did produce rate rise, or did consume capacity fall? A steady 54,000 records per second against an 8,000-record capacity indicates key skew and insufficient per-key capacity. If input is unchanged but consumption drops from 8,000 to 2,000, investigate the downstream system, garbage collection, network, disk, and rebalances first.
• Where does the consumer write? A Kafka-to-Kafka pipeline can atomically commit output and input offsets with Kafka transactions. A database, object store, or external API usually needs at-least-once processing plus a business idempotency key or topic-partition-offset.
• How long can an order remain active? Short-lived orders can stay on the legacy route until completion while new orders use the new route. Long-lived orders require an explicit barrier, sequence, or routing state.
• Can the incident response throttle or degrade traffic? A tenant quota, delayed noncritical events, or coalesced state updates can reduce input faster than a code and partition migration.
• Are consumers exceeding max.poll.interval.ms? If heavy work blocks the poll thread, rebalances amplify lag. Separate polling from processing and bound in-flight work before merely increasing the timeout.

30-Second Answer Framework

“I would break aggregate lag into per-partition produce rate, consume rate, and lag slope, then align those with key frequency, bytes, handler time, rebalance logs, and the leader broker's metrics. The hot tenant produces 54,000 records per second while one consumer handles 8,000, so backlog grows by roughly 46,000 per second; adding ordinary consumers cannot accelerate that partition. Today I would throttle or degrade the hot tenant, isolate the partition on a dedicated instance, and exploit the true ordering boundary by serializing each order while processing different orders concurrently. I would commit only the highest contiguous completed offset and use idempotent downstream writes. Long term, new orders would move to a new topic keyed by order_id, while existing orders remain on the legacy route until they finish. I would validate partition-level lag slope, end-to-end p99, duplicates, and order violations under skewed load, consumer crashes, and rebalances.”

Step-by-Step Deep Answer

Step 1: Prove which layer created the hotspot

Use one peak-time window and align:

1. produce records per second, bytes per second, and high-watermark growth by partition;
2. consume records per second, committed offset, and lag slope by partition;
3. key frequency, bytes, and estimated processing-cost distribution;
4. CPU, network, disk wait, and request latency on the hot partition's leader broker;
5. poll intervals, batch sizes, handler latency, garbage collection, errors, and rebalance logs for the assigned consumer;
6. partition-correlated latency or throttling in the downstream database, storage system, or API.

The decision rule follows from the numbers. The hot partition receives about 54,000 records per second. The other 23 partitions share the remaining 66,000, averaging about 2,870 records per second if that remainder is reasonably uniform. An 8,000-record consumer has spare capacity on an ordinary partition but cannot match the hot input. That fully explains one growing partition and idle capacity elsewhere. If hot-partition input is normal while consumption degrades in step with downstream latency, key design is not yet the proven root cause.

Check broker placement as well. A partition whose leader sits on an overloaded broker can suffer slower production and fetching. Moving leadership or rebalancing replicas can remove that placement bottleneck, but it does not change the fact that one tenant_id still maps to one partition.

Step 2: Reduce backlog slope before trying to drain it

The first incident objective is:

hot-partition input rate ≤ hot-partition safe processing rate

The fastest lever is often admission. Apply an explicit quota to the hot tenant, delay reconstructable analytics events, coalesce updates for which only the latest state matters, or place noncritical work on a declared degraded path. Every measure needs explicit loss, delay, and replay semantics. Silently dropping records is not a throttling strategy.

On the consumer side, a controlled maintenance window can stop the existing group and replace it with an exclusive, non-overlapping explicit assignment: one adequately provisioned instance owns only the hot partition, and the remaining instances own the other partitions. Starting a second ordinary consumer group is not help; it independently consumes the full topic and duplicates business effects. Isolation prevents the hot partition from starving ordinary partitions on the same process, although it does not raise the original serial handler above 8,000 records per second.

Because the business needs order only within an order, the hot consumer can dispatch by order_id: one serial queue per active order and a bounded worker pool across different orders. The pool must have an in-flight limit. When it is full, pause the partition or reduce the amount released to workers so that Kafka lag does not become unbounded process memory. The poll loop must remain responsive; otherwise exceeding max.poll.interval.ms triggers a rebalance and adds another pause.

Step 3: Commit a contiguous completion watermark

Intra-partition concurrency changes completion order, but it must not change commit order. Maintain this state for each partition:

nextCommitOffset = smallest unfinished offset
completed = offsets that finished but still have a gap before them

onComplete(offset):
  add offset to completed
  while completed contains nextCommitOffset:
    remove nextCommitOffset from completed
    increment nextCommitOffset
  commit nextCommitOffset

Kafka commits the next position to read. Therefore, only after 104, 105, and 106 all finish may the committed position advance to 107. If 105 finishes while 104 retries, the commit point stays at 104. A crash before the next commit replays some finished records, so database writes should use event_id or another business-unique key for an idempotent upsert. If no business key exists, topic-partition-offset can identify the source record.

Do not describe an offset commit and an external side effect as naturally atomic. A Kafka-to-Kafka application can put output records and consumed offsets in one Kafka transaction. With an external database, a more common contract is at-least-once consumption plus an idempotent write, or a database transaction containing both the deduplication record and the business mutation.

Step 4: Match partition scope to the real ordering domain

The average capacity of 24 partitions is not necessarily insufficient. If 8,000 records per second is a measured safe maximum and planned utilization is capped at 70%, planned capacity per partition is 5,600:

120,000 ÷ 5,600 ≈ 21.4

With an even distribution, 24 partitions cover the assumed peak with modest headroom. The failure comes from putting 45% of traffic behind one low-cardinality key, not from the aggregate partition count. A durable key should represent the smallest required ordering domain, have enough cardinality, and remain predictably distributed at peak. Here, order_id is the natural choice. A stable encoding of (tenant_id, order_id) is also possible if tenant locality has a real operational value.

Controlled salting is valid only when records within the original key may be reordered or a downstream stage can restore order. Randomly salting one order violates this prompt's invariant because that order can arrive concurrently from several partitions. If one order alone exceeds single-partition capacity, greater key cardinality does not help; optimize or throttle that order's serial path, or redesign the business protocol into explicitly independent sequences.

Step 5: Migrate with versioned routing

Adding partitions to the existing topic and changing keys immediately creates two risks. The default hash mapping can move existing keys when the partition count changes. Events for one order can also land in different partitions before and after cutover, while Kafka does not guarantee cross-partition order.

A safer design creates a new topic partitioned by order_id and versions producer routing:

• orders created after the cutover use the new topic;
• orders that already existed remain on the old topic and legacy key until they close;
• every producer uses the same order-routing state instead of comparing its local clock with a cutover time;
• consumers read both routes, but one order belongs to only one active route at any moment;
• the old topic is retired after legacy orders drain and retention requirements are met.

If orders do not naturally finish, create a per-order migration barrier: pause new events for that order, wait until the old route reaches a recorded final sequence or offset, change the routing version, and resume. A zero-pause alternative can carry monotonic sequence numbers and merge both routes downstream, but that introduces buffering, timeouts, and gap recovery. It is justified only if the business requirement pays for that complexity.

Step 6: Validate with skewed traffic and failures

Aggregate throughput is not sufficient evidence. Test at least:

• a distribution in which one tenant produces 45% of traffic and order distribution resembles the real peak;
• input, consumption, lag slope, and maximum lag by partition;
• end-to-end p50, p95, and p99 plus estimated drain time;
• in-flight worker count, oldest-task age, retries, and dead letters;
• duplicate effects, per-order ordering violations, and idempotency conflicts;
• a consumer crash while completed offsets contain a gap;
• whether long processing triggers a rebalance and how long recovery takes;
• whether an order appears on only one route at the old/new topic boundary.

The pass condition includes sustained behavior: hot-partition lag slope is no longer positive at steady peak; a crash may replay work but cannot lose a business effect; no order is observed out of sequence; new orders distribute across partitions; and legacy orders drain on schedule. If aggregate throughput rises while one large order repeatedly creates a hotspot, the ordering domain or business admission problem remains unresolved.

High-Quality Sample Answer

“I would not start by adding consumers because the prompt already tells us one partition is behind while consumers elsewhere are idle. I would first prove skew using per-partition records per second, bytes per second, lag slope, and key frequency, while ruling out the leader broker, rebalances, and downstream latency.

The hot tenant produces 54,000 records per second. A single partition consumer processes 8,000, so lag grows by about 46,000 records per second, or 27.6 million in 10 minutes. If the other 55% is roughly spread over 23 partitions, each averages about 2,870 records per second. That explains both the hot consumer's deficit and spare capacity elsewhere. In a traditional consumer group, one partition belongs to one consumer at a time, so more ordinary instances do not accelerate it.

Today I would first reduce the input slope with a documented tenant quota and move delay-tolerant or coalescible events onto a degraded path. In a controlled window, I would isolate the hot partition on a dedicated instance so it does not starve ordinary partitions. Since only per-order ordering matters, I would dispatch by order_id to a bounded pool: serial within an order, concurrent across orders. I would not commit whichever task finishes fastest. I would track the highest contiguous completed offset and stop at any gap. A crash can replay completed but uncommitted records, so the sink uses event_id or a business-unique key for idempotency.

Long term, adding partitions is not the whole fix. At 70% planned utilization, each 8,000-record partition contributes about 5,600 records per second, so 24 evenly loaded partitions can cover the assumed 120,000-record peak. The problem is that tenant_id pins 45% to one partition. I would create a new topic keyed by order_id. New orders after cutover use it, while active legacy orders stay on the old route until completion. Shared routing state ensures that one order never spans both topics.

Before rollout, I would replay the same 45% tenant skew and inspect per-partition throughput, lag slope, end-to-end p99, duplicates, and order violations. I would crash the consumer while offsets complete out of order and verify that restart causes only idempotent replay, trigger rebalances and measure recovery, and verify that every order at the route boundary appears on one topic only. If one order itself exceeds single-partition capacity, I would state the hard limit: neither more consumers nor more partitions solves it without optimizing, throttling, or changing that order's sequencing model.”

Common Mistakes

• Looking only at topic-wide lag → An average hides the input and consume slope of one partition → Graph records, bytes, lag, and key distribution per partition.
• Adding consumers whenever lag rises → One partition in a traditional group has one consumer owner at a time → Compare partition count, assignment, and per-partition capacity first.
• Adding partitions immediately → Existing backlog does not redistribute automatically, and default key mapping can change → Stop backlog growth first, then use a versioned topic and migration boundary.
• Randomly salting the hot key → One order can cross partitions and arrive out of sequence → Salt only when reordering is acceptable; use order_id for this prompt's true ordering scope.
• Committing an offset as soon as its worker finishes → A lower unfinished offset can be skipped after a crash → Commit only the contiguous completion watermark.
• Calling an offset commit exactly-once processing → An external database mutation and Kafka offset are usually not one transaction → State the at-least-once, idempotency, and transaction boundaries.
• Starting a second consumer group to help → The second group reads its own complete copy and duplicates effects → Use exclusive explicit assignment or a controlled processing redesign.
• Testing only uniform traffic → A passing average does not prove that the hot key is gone → Replay a realistic skew and inspect the maximum partition, not just the mean.
• Ignoring the single-entity limit → One strictly ordered entity cannot be parallelized across partitions for free → State the hard trade-off among order, throttling, and serial processing.

Follow-Up Questions and Responses

Follow-up 1: Why not increase the partition count from 24 to 48 immediately?

New partitions create future parallel slots, but they do not split the backlog already stored in an old partition, nor do they make one tenant_id map to several partitions. With default key hashing, changing the partition count can also remap existing keys and place one order on different partitions across the cutover. Fix the ordering domain and migration first, then use a skewed load test to decide whether more aggregate partitions are necessary.

Follow-up 2: How do you prevent unbounded memory after adding intra-partition concurrency?

Set a maximum in-flight record count and maximum uncommitted offset window per partition. When either limit is reached, pause that partition or release fewer records to the worker pool while keeping polling separate from heavy processing. Track the age of the oldest unfinished offset. If one order remains blocked, isolate, retry, or route it for intervention rather than allowing every later offset to consume memory indefinitely.

Follow-up 3: What if the downstream database does not support an idempotent upsert?

Insert a deduplication record and perform the business mutation in one database transaction. Use a business event_id or topic-partition-offset as a unique key; a uniqueness conflict means the event has already been applied. For a nontransactional external API, use its idempotency key, an outbox, or a queryable operation state. If no idempotency boundary can be built, you cannot promise that crash replay has no duplicate effect.

Follow-up 4: What changes if the business later requires strict tenant-wide order?

Then tenant_id is the indivisible ordering domain, and cross-order concurrency is no longer valid. If one tenant exceeds single-partition capacity, optimize that serial path, throttle the tenant, or renegotiate which event sequences are independent. A globally sequenced sharded stream with a downstream merger is possible, but it moves ordering waits, gap recovery, and availability cost into the consumer; it is not free scale.

Follow-up 5: How long will it take to drain the 27.6-million-record backlog?

Input must first fall below processing capacity. If throttling lowers hot-partition input to 3,000 records per second and optimization raises processing capacity to 12,000, net drain rate is 9,000:

27,600,000 ÷ 9,000 ≈ 3,067 seconds ≈ 51 minutes

That is still a steady-rate estimate. A real recovery plan adds retries, downstream throttling, record-size variance, and safety margin, then continuously revises the estimate from the observed lag slope.

Follow-up 6: How do you prove that no order crosses the old and new topics?

Store one partitioning_version for each order. Every producer reads or caches the same versioned routing record, and the version changes only after the migration barrier succeeds. Consumers record the first topic and version seen for an order, alert on an order appearing on both active routes, and stop automatic progression for that order. During load tests and canary rollout, reconcile producer logs, offsets on both topics, and downstream order sequences rather than checking only equal total record counts.