Question and When It Applies
A platform uses Kafka for event streams and task queues. A traditional Consumer Group assigns each partition to one member, while the team wants Kafka 4.1 Share Groups for more queue-like concurrency. Decide which workloads fit and how to control preview-feature risk.
What the Interviewer Evaluates
- Distinguishing partitioned stream processing from shared-queue delivery semantics.
- Explaining acquisition limits, acknowledgment, failure redelivery, and ordering.
- Combining multi-tenant fairness, lag, idempotency, and observability in one design.
- Setting compatibility, canary, replay, and rollback gates for a preview feature.
Clarifying Questions Before You Answer
- Does the work require key ordering and partition-local state, or only eventual processing of each record?
- Should a failed record retry immediately, later, or through a dead-letter queue?
- Do tenants share topics and consumer capacity, and is there a stable tenant identity?
- Are downstream side effects idempotent, and can processing be concurrent or duplicated?
- Do the Kafka version, client, admin tools, and managed platform support Share Groups?
30-Second Answer Framework
Consumer Groups use partitions as parallelism and ordering boundaries, fitting streaming aggregates and keyed state. Share Groups are closer to a shared queue: multiple consumers can acquire different records from one topic-partition, while the cluster limits how many records can be acquired per partition. I would classify by business semantics, validate acknowledgment, redelivery, idempotency, and fairness, then canary the preview feature while retaining a Consumer Group rollback path.
Step-by-Step Deep Dive
Step 1: Start with processing semantics, not the API name
If processing depends on partition order, window state, or keyed aggregation, Consumer Group assignment is easier to reason about. If tasks are independent, need more concurrency, and accept queue-style acknowledgment, Share Groups are candidates. Do not migrate merely because a benchmark promises more throughput.
Step 2: Compare concurrency and acquisition boundaries
Traditional group parallelism is mainly bounded by partition count; one member handles a partition at a time. Share Groups allow multiple consumers to acquire records from one topic-partition, but the cluster still limits the number acquired per partition. Measure the product of batch size, processing time, and downstream capacity.
Step 3: Define acknowledgment, failure, and redelivery
Before migration, define when a record succeeds, whether failure makes it available to another consumer, and whether redelivery can overlap old work. Use idempotency keys, a deduplication table, or repeatable transactions for external writes. Send unrecoverable failures to a dead-letter queue with reason, tenant, and attempt count.
Step 4: Handle ordering and state
Share-queue semantics can invalidate assumptions about partition order. Keep keyed-serial work in Consumer Groups, or implement key-level serialization and version checks in the application. State storage should record event version, processor, and retry status so concurrent updates cannot silently overwrite one another.
Step 5: Build tenant fairness and backpressure
A tenant burst on a shared topic can create a noisy neighbor. Carry a stable tenant identity and observe wait time, throughput, and failure rate by tenant. Add application quotas, per-batch acquisition caps, topic separation, or downstream bulkheads when needed. Databases and external APIs need their own concurrency limits.
Step 6: Validate preview and operations paths
Kafka documentation labels Share Groups as preview and not enabled by default. Verify broker, client, admin-tool, metrics, failure-recovery, and upgrade compatibility first. Use synthetic events to test restarts, fewer consumers, duplicate acknowledgments, broker changes, lag, and dead letters.
Step 7: Design migration and rollback
First mirror a small non-critical workload into a Share Group. Compare throughput, p99 wait, duplicates, redelivery, tenant fairness, and downstream errors. Preserve the original topic or replayable offset boundary. If ordering, duplicate side effects, or preview components regress, pause new traffic and switch back to the Consumer Group.
High-Quality Sample Answer
I would split by semantics. Events requiring partition order, window state, or keyed aggregation stay in Consumer Groups; independent, concurrent, idempotent tasks can be candidates for Share Groups. Traditional groups use a partition as the parallelism boundary, with one member handling it; Share Groups behave more like a shared queue, allowing several consumers to acquire different records from one topic-partition while retaining a per-partition acquisition limit. Before migration I would define acknowledgment and redelivery, idempotency keys, dead letters, and tenant-fairness metrics, with downstream bulkheads. Because Share Groups are preview in Kafka 4.1 documentation, I would verify version and operations compatibility, canary non-critical tenants, and compare wait, duplicate, lag, redelivery, per-tenant throughput, and downstream errors. I would preserve replayable data and the Consumer Group rollback path; any ordering or side-effect regression pauses and reverts.
Common Mistakes
- Treating Share Groups as merely “more consumers” and ignoring delivery and acknowledgment semantics.
- Moving partition-ordered state streams directly to a shared queue.
- Accepting redelivery and concurrent processing without idempotency.
- Watching total throughput while missing tenant wait, duplicates, and downstream saturation.
- Ignoring preview compatibility across clients, tools, and upgrades.
- Deleting the original data after migration and losing replay and rollback.
Follow-Up Questions and Responses
Follow-up 1: Do Share Groups remove partitions?
No. A topic-partition remains the storage and replication boundary. The change is that several share-group members can acquire different records from one partition, subject to a cluster acquisition limit.
Follow-up 2: Can the same key still be ordered?
Do not assume it. Keep order-dependent work in a Consumer Group, or serialize by key in the application and use version checks to prove concurrent updates cannot overwrite one another.
Follow-up 3: What happens to a failed record?
Confirm from the implementation and configuration whether it becomes available again, is delayed, or can be redelivered concurrently. Protect side effects with idempotency keys, attempt counts, and dead-letter reasons.
Follow-up 4: How do you prevent a tenant burst from filling capacity?
Carry tenant identity, observe per-tenant wait and throughput, and combine application quotas, acquisition caps, topic separation, or downstream bulkheads. Turn fairness into alertable objectives.
Follow-up 5: Why not migrate everything?
Stream and queue workloads need different ordering, state, retry, and operations guarantees. A preview feature adds version and failure risk, so classify workloads instead of forcing one model.
Follow-up 6: How do you roll back records already processed?
Keep replayable events and processing versions, stop new Share Group traffic, and resume from the Consumer Group boundary. Compensate or reconcile external side effects idempotently; do not blindly write them twice.