How should Kafka Streams migrate to the broker-driven rebalance protocol?

Prompt and scope

You own a stateful Kafka Streams application using the classic group protocol. After upgrading to Kafka 4.2, the team wants the broker-driven Streams Rebalance Protocol to reduce global coordination pauses when instances join, leave, or fail. The interviewer asks for a migration plan, compatibility boundary, and rollback.

Assume Kafka Streams 4.2.x, existing changelog and repartition topics, and no unplanned full state rebuild. The official guide says the new protocol continuously computes task assignments on brokers and uses a dedicated streams group. New Kafka 4.2 clusters enable the feature by default, but clients still set group.protocol=streams.

What the interviewer evaluates

• Whether you explain how broker-driven coordination removes a client-side global barrier instead of reciting a setting.
• Whether you separate a new streams group, an online classic-group upgrade, and the supported offline migration.
• Whether you check broker and client versions and the KAFKA-20254 risk in 4.2.0.
• Whether you know that committed offsets are preserved while other group metadata is rebuilt.
• Whether you convert missing static membership, topology updates, and regex support into release gates.

A weak answer says “change the setting and roll.” A strong answer inventories version and feature gaps, chooses a new group or maintenance-window migration, and records offsets, changelog, repartition topics, and recovery objectives.

Clarifications before answering

1. Are Kafka and the Streams client both at least 4.2? Otherwise the complete protocol cannot be enabled safely.
2. Does the application rely on static membership, online topology updates, regex subscriptions, or standby/rack-aware assignment? Any dependency can block migration.
3. Can every instance stop while the group becomes empty? The official 4.2 path supports offline migration only.
4. Is the version 4.2.0 or 4.2.1 and later? 4.2.0 has a known broker bug in offline migration, fixed in 4.2.1.
5. Can a new application.id be used? A new group isolates risk but rebuilds state and changes offset management.

These answers change the plan: if downtime is impossible, do not claim online migration; if an unsupported feature is required, stay on the classic protocol or refactor first.

A 30-second answer framework

“I first verify that brokers and clients are on 4.2.x and inventory features that the new protocol does not support. The Streams Rebalance Protocol moves task coordination to brokers and removes a client-side global barrier, but the migration is not a normal rolling deployment. The documented path makes the group empty, sets group.protocol=streams, and starts the instances. Only committed offsets are preserved; changelog and repartition topics remain, while other group metadata is rebuilt. I would avoid 4.2.0 and use 4.2.1 or later, record offsets and state checkpoints, verify recovery, latency, and rebalance metrics, and roll back to classic or rebuild with a new application ID if checks fail.”

Step-by-step solution

1. Explain what changes

Classic Streams groups calculate member task assignments on clients, which can create a global coordination point during membership changes. The new protocol stores streams-group metadata and task assignment on brokers; applications coordinate through a dedicated heartbeat and streams group. The official guide describes this as broker-driven and provides separate streams-group states and Admin APIs.

2. Inventory capability gaps

Kafka 4.2 documents clear limits: static membership is unavailable; significant topology updates require a new streams group; only the sticky task assignor is supported, so warmup tasks and rack-aware assignment are unavailable; pattern subscriptions are unsupported; and online migration between classic and streams groups is unavailable. Put these on the release checklist before changing a protocol.

3. Choose the migration path

The documented offline path is: stop every instance, wait for session.timeout.ms or explicitly leave so the group is empty, set group.protocol=streams, and start the instances. Only committed offsets are retained on the broker. Changelog and repartition topics remain ordinary internal topics; other group metadata is rebuilt.

Stop all instances
      ↓
Confirm an empty streams group and record committed offsets
      ↓
Upgrade brokers and clients to a compatible version
      ↓
Set group.protocol=streams
      ↓
Start instances and observe recovery and rebalance metrics

If a maintenance window is unacceptable, keep the classic protocol or use a new application.id for shadow validation. Do not transfer the rolling-upgrade assumptions of classic consumers to the Streams Rebalance Protocol.

4. Handle the version risk

The Kafka upgrade guide warns that classic-to-streams offline migration in 4.2.0 is affected by the KAFKA-20254 broker-side bug and recommends against it. The fix is in 4.2.1. In an interview, make 4.2.1 the minimum migration version instead of saying only that “Kafka 4.2 supports it.”

5. Design state and offset checks

Before migration, record committed offsets for each input topic, changelog status, and processing latency. After migration, verify that the new group resumes from expected offsets, state stores restore from changelogs, repartition topics still exist with the same partition counts, and duplicate or missing records match the agreed processing semantics. Compare with a pre-migration baseline rather than checking only process startup.

6. Monitor and roll back

Use streams-group state, rebalance count/rate, recovery duration, processing latency, and error rate as the observation surface. If recovery times out or result checks fail, stop the new group, preserve offsets and logs, and revert configuration to classic. If the classic group was already emptied, recovery requires a backup or a new application ID; group metadata will not magically return.

High-quality sample answer

“I would not treat this as a normal rolling release. First I verify 4.2.x brokers and clients and check for static membership, online topology updates, regex subscriptions, warmup, or rack-aware assignment. Because the official path is offline, I choose 4.2.1 or later, stop every instance in a maintenance window, confirm an empty group, record committed offsets and state-store checkpoints, and then set group.protocol=streams.

“After the change I verify that offsets continue, changelogs restore state stores, repartition topics remain, and streams-group state, rebalance metrics, recovery time, and business latency are healthy. Only committed offsets are preserved; other group metadata is rebuilt. 4.2.0 carries KAFKA-20254 risk, so I would not call it a safe migration version. If validation fails, I stop the new group, return to classic, or rebuild with a new application ID and retain the evidence for review.”

Common mistakes

• Mistake: treating group.protocol=streams as a rolling switch → Why it fails: online migration is unsupported → Fix: schedule an empty-group maintenance window.
• Mistake: migrating on 4.2.0 → Why it fails: the official upgrade guide records KAFKA-20254 → Fix: use 4.2.1 or later with the fix.
• Mistake: promising every group state is preserved → Why it fails: only committed offsets remain and other metadata is rebuilt → Fix: record separate offset, state-store, and topic checks.
• Mistake: ignoring static membership or topology updates → Why it fails: the new protocol does not support them yet → Fix: inventory features and stay on classic when necessary.

Follow-ups and responses

The business cannot stop. Can two instance batches switch gradually?

Do not describe that as the documented Streams migration. If downtime is impossible, keep classic or create a new application ID for shadow validation, then let the business layer absorb the state-rebuild cost of a cutover.

Why validate the state store if committed offsets survive?

An offset says where to read next, not that local state is complete. Changelog replay, incompatibility, or processing failures can leave the state store inconsistent with the offset, so validate state and business results.

4.2.0 is GA. Why avoid it?

GA means the feature was released, not that every migration path is free of known defects. The official guide identifies KAFKA-20254 in offline migration and says it is fixed in 4.2.1; choose the fix version, not the GA label.

The application uses regex subscriptions. What now?

The new streams protocol does not support pattern-based topic subscriptions. Stay on classic or change discovery to an explicit topic list before reconsidering migration; changing only the group protocol is insufficient.

How do you decide whether to use a new application ID?

Use one when parallel validation is required, the old group cannot be safely emptied, or state-recovery risk must be isolated. The cost is reprocessing, state-store rebuild, and extra resources, so estimate recovery time and storage first.

References

• Apache Kafka Streams Rebalance Protocol developer guide.
• Apache Kafka 4.2 Streams Upgrade Guide.
• Apache Kafka 4.2.0 Release Announcement.