Problem and scope
Order notifications, image transforms, and billing calculations are often independent work items. The team already writes events to Kafka, but a traditional Consumer Group assigns a partition to one member at a time; adding more workers than partitions does not directly increase parallelism. Design a way for multiple consumers to cooperate on work, with per-record acknowledgement, retries, and observable delivery attempts, while identifying business flows that still require partition ordering.
This question uses the Share Group model described by Apache Kafka KIP-932. The KIP describes a new group type for cooperative consumption on regular topics; it does not make Kafka identical to RabbitMQ. A strong answer verifies the deployed broker version, client support, and API availability before recommending production use.
What the interviewer is testing
- Can you contrast exclusive partition assignment with cooperative record acquisition?
- Can you map acknowledge, release, reject, and lock expiry to processing states?
- Do you know that a Share Group can have more consumers than partitions without preserving the usual key-order intuition?
- Do retries, poison records, processing deadlines, and concurrency limits fit one failure model in your answer?
- Will you verify client, broker, ACL, monitoring, and rollback details instead of naming a KIP?
A weak answer says “Kafka can also be a queue.” A strong answer names the queue-like benefit, the guarantees that change, and the gates required before rollout.
Clarifying questions to ask first
- Are jobs truly independent? If events for one order must be applied in key order, Share Groups may be the wrong primitive.
- Is a failure transient, manually recoverable, or permanently invalid? That determines release, reject, and quarantine behavior.
- What are p99 processing time, maximum concurrency, and the tolerated duplicate side effects? They shape the acquisition lock and idempotency design.
- Does the business require Kafka transactions end to end? Do not assume a Share Group inherits the existing Consumer Group transaction plan.
- Does the topic still serve broadcast or replay consumers? Changing one group must not change another group’s read contract.
A 30-second answer
“I would first confirm whether jobs may complete out of order and whether the deployed Kafka and clients support KIP-932. A Share Group lets members cooperatively acquire records from a topic, allows the member count to exceed the partition count, and supports per-record acknowledgement, release, and rejection. A Consumer Group remains a better fit for partition-local ordering and offset-based reasoning. I would make each side effect idempotent, set lock and retry policy from processing latency, monitor acquire, acknowledge, release, reject, and timeout states, and quarantine poison records. If ordering, transaction boundaries, or client support are unresolved, I would retain a Consumer Group and validate a separate work topic with a small cohort before migrating.”
Step-by-step reasoning
1. Draw the two assignment models
A Consumer Group normally assigns partitions to members; one member reads a given partition within that group, so parallelism is bounded by partition count. A Share Group lets members cooperatively acquire records from subscribed topics. Multiple members can process different records from one partition, and the member count can exceed the partition count. That is useful for independent jobs, but it does not imply global ordering.
Consumer Group: partition-0 -> worker-A
partition-1 -> worker-B
extra workers wait for another partition
Share Group: partition-0 records -> worker-A, worker-B, worker-C
each acquired record is locked for one consumerThe reason to choose a Share Group should be elastic work acquisition and per-record completion, not simply “there are too few partitions.” If events for one customer must be applied in order, keep a Consumer Group or add an application-level serialized state machine.
2. Model the record lifecycle
KIP-932 describes a time-limited acquisition lock. After acquiring a record, a consumer can acknowledge success, release it for another delivery, reject it as unprocessable, or do nothing until the lock expires. The KIP describes a 30-second default, but production behavior must use the deployed broker setting; a default is not an SLA.
available -> acquired -> acknowledged
-> released -> available
-> rejected -> terminal or quarantine
-> lock timeout -> availableThe handler should register an idempotency key before an external side effect. Otherwise a client crash or lock expiry can charge, ship, or notify twice. An acknowledgement says that this acquisition completed; it cannot roll back a side effect already committed by another system.
3. Bound retries, poison records, and concurrency
Delivery-attempt counts help separate transient failures from permanently invalid records. Back off and release for a network failure. Reject deterministic schema or validation errors into a quarantine topic or manual queue. Do not release forever: one poison record can consume locks and downstream capacity indefinitely.
Set the lock above normal p99 processing time with an explainable jitter margin. Too short causes overlapping redelivery; too long delays recovery. Also bound acquired records per partition and coordinate worker semaphores, database pools, and external API quotas. Monitor active locks, lock timeouts, attempt distribution, rejects, and end-to-end completion latency together.
4. Re-state ordering and duplicate guarantees
Kafka explanations often turn “ordered inside a partition” into “business processing is ordered.” Share Group members can acquire records concurrently, so completion order for one key may differ from write order; release and redelivery amplify the difference. If order matters, encode key-level serialization, version checks, or a state machine in the application. Do not answer with “Kafka is ordered” alone.
Exactly-once also does not appear automatically from the group type. Trace the boundaries between record acquisition, business writes, and acknowledgement. An external database or payment service still needs idempotency keys, a deduplication constraint, or a transactional outbox. If a combination is unsupported, state that the design is at-least-once with idempotency instead of calling it exactly-once.
5. Plan migration and rollback
Verify broker version, client API, group configuration, ACLs, metrics, and operational commands. Then load-test a separate topic or small workload. Inject failures: crash after acquisition, processing beyond the lock, repeated rejects, broker restart, and coordinator movement. Record business key, attempt, state, and timestamps for every record.
If old consumers depend on ordering or transactions, do not change the same group in place. Copy work to a dedicated topic and let a new group take traffic gradually; keep the old path replayable until error rate, duplicate side effects, and latency meet gates. Rollback stops new acquisition and leaves the old path to consume unmigrated records. Two active paths must not execute the same side effect without an explicit deduplication boundary.
High-quality sample answer
“I would start by asking whether jobs may complete out of order, whether processing is idempotent, and whether the deployed broker and client support KIP-932. A Share Group treats independent topic records as cooperative work: multiple members can acquire different records from one partition, the member count can exceed the partition count, and each record has acknowledge, release, reject, and lock-expiry paths. It changes the allocation and ordering intuition of a traditional group, so I would keep a Consumer Group or add version checks when a business key requires order.
I would assign an idempotency key to every record, set the acquisition lock from p99 processing time, and bound active locks and downstream concurrency. Transient failures release with backoff; deterministic bad data is rejected to quarantine; an attempt threshold stops automatic retry. I would monitor acquire, acknowledge, release, reject, timeout, duplicate side effects, and completion latency. Before migration I would verify versions, ACLs, client behavior, and injected failures on a separate topic. Unless record acquisition, business writes, and acknowledgement share one proven transaction boundary, I would call the design at-least-once plus idempotency, not exactly-once.”
Common mistakes
- Calling a Share Group a RabbitMQ clone → storage, replay, and administration differ → promise only the cooperative acquisition and acknowledgement semantics documented by KIP-932.
- Capping workers at partition count → Share Groups allow multiple members to process one partition → bound concurrency by locks, downstream capacity, and end-to-end latency.
- Assuming key order remains intact → concurrent acquisition and redelivery change completion order → serialize keys or check versions when order is a requirement.
- Acknowledging without idempotency → a crash or lock expiry can redeliver the record → deduplicate by business key before acknowledgement.
- Releasing poison records forever → retries exhaust locks and downstream budget → stop automatic retry by error type, attempt count, and quarantine policy.
- Treating 30 seconds as a guarantee → broker configuration and processing latency differ → test the deployed lock setting against p99.
Follow-up questions and responses
What if events for one order must be strictly ordered?
Do not switch directly to a Share Group. Keep a Consumer Group partitioned by order ID, or use an application-level serialized state machine. If shared acquisition is mandatory, add version checks, prerequisite validation, and failure reordering, and acknowledge the extra complexity.
What if a consumer hangs for two minutes after acquisition?
Set a lock slightly above normal p99 and alert on lock timeout. Allow redelivery after expiry, but require idempotent business handling. For long work, split into resumable steps or use an external lease instead of extending the lock indefinitely.
How do you handle five consecutive schema failures?
Treat them as deterministic errors: reject after a threshold and write the payload, schema version, and reason to quarantine. After fixing the consumer, replay under a controlled procedure. Do not keep the main workflow retrying forever.
The current system relies on Kafka transactions. Can it switch directly?
List the transaction’s read, processing, and write boundaries, then verify actual Share Group client and transaction support. If an external side effect is outside the same transaction, use an outbox, idempotency key, and compensation. Keep a Consumer Group when support is unproven.
How do you prove migration did not double-charge customers?
Record every execution under a unique business key with a deduplication constraint. Inject crashes, lock expiry, retries, and rollback, then compare execution and acknowledgement counts. Expand traffic only when side-effect count, latency, and error-rate invariants hold.