Kafka Tiered Storage: How Do You Design Local and Remote Retention?

Prompt and scope

An interviewer may ask: “If a Kafka topic must retain years of history, how would you design Tiered Storage's local and remote retention policies?”

The core is not memorizing a configuration name. It is explaining the lifecycle of log segments across local and remote tiers. KIP-405 keeps a local tier on Kafka brokers while uploading completed log segments to external storage; local retention can be shorter than remote retention to reduce broker disk pressure. A complete answer also covers object-store failures, historical-read latency, deletion ownership, and recovery.

What the interviewer is testing

• Whether you understand tiered storage as log-segment hot/cold placement, not Kafka becoming an object store.
• Whether you can separate cluster capability, topic-level remote.storage.enable, and retention policies.
• Whether you can estimate hot-disk capacity, remote capacity, upload lag, and replay bandwidth.
• Whether you consider remote outages, metadata consistency, partition movement, and consumer replay.
• Whether you assign clear ownership for remote deletion without accidental or unbounded growth.

Clarifying questions

• Which topics need historical storage, and should every topic be enabled?
• What are the hot-read latency and historical-replay throughput targets?
• What are the local-disk budget, remote storage class, regional, and compliance requirements?
• During an object-store outage, how do production, real-time consumers, and historical readers degrade?
• Is deletion driven by time, size, a compliance event, or tenant policy?

A 30-second answer

You can say:

I would treat Kafka's local tier as a low-latency hot cache, upload completed segments to a remote tier, and define local and remote retention separately. Enable remote.storage.enable only for topics that need it; use local retention to control broker disks and remote retention to meet audit and replay requirements. I would quantify upload lag, remote-read latency, object-store failures, and partition movement, then assign an owner for remote cleanup. Real-time consumers stay on local paths; historical replay accepts extra latency with throttling and monitoring.

Step-by-step reasoning

Establish the two-tier lifecycle

Kafka still uses partition log segments as the basic unit. Active segments are written locally; after rolling, the remote-log component uploads segments and index information to configured remote storage. A client must not assume every historical byte remains on broker disks:

producer -> leader broker local segment
                    | segment roll
                    v
             remote object store
consumer <---- local cache or remote fetch

The local tier serves low-latency consumption and the remote tier carries longer retention. Keep an observable safety window between upload completion and local deletion; object creation alone is not proof that indexes and metadata are usable.

Scope the feature per topic

Apache Kafka documents that after broker-side configuration, a topic still opts in with remote.storage.enable. State whether the governance model is opt-in or default-on, because enabling every topic can multiply cost and the failure surface.

Separate local and remote retention

Set local retention by time or size to preserve hot data and the read capacity needed for reassignment; set remote retention for audit, replay, and compliance. The invariant is that the only local copy is not deleted until remote objects, indexes, metadata, and recovery tests are verified. Remote deletion needs independent audit and retry handling.

Design reads and degradation

Real-time consumers read the local tier first; an offset beyond the local window triggers a remote read. When remote storage slows, throttle historical replay to protect live traffic. When it is unavailable, state which offsets are temporarily unreadable, how alerts fire, and how recovery works. Test metadata and cache rebuild time during partition movement, leader changes, and broker restarts.

Model high-quality answer

I would first confirm the topic's hot-read window, replay throughput, retention period, and compliance requirements. Following KIP-405, the broker's local tier is a hot cache and rolled segments are uploaded to remote object storage; only topics that need history enable remote.storage.enable. Local retention follows disk budget and real-time replay needs, while remote retention follows the audit period. The deletion gate is verified remote objects, indexes, and metadata, not merely an upload request. Consumers keep low latency inside the local window; older replay uses remote reads with throttling. I would monitor upload lag, remote-read latency, cache hits, object/metadata mismatches, deletion failures, and unreadable offsets, and exercise object-store outage, broker restart, and partition-movement recovery. Remote cleanup needs an owner, audit trail, and restore procedure; deleting broker files does not make that responsibility disappear.

Common mistakes

• Claiming tiered storage streams every Kafka record directly to object storage and ignoring segment rolling.
• Giving only a cluster switch and omitting topic-level remote.storage.enable.
• Calculating object cost while skipping remote-read latency, upload lag, and replay bandwidth.
• Assuming deleting broker files automatically deletes remote objects.
• Letting historical replay consume resources needed by live traffic during remote failure.
• Omitting monitoring and recovery drills for object, index, and metadata consistency.

Follow-up questions and responses

1. How do you choose local retention time?

Work backward from the largest real-time rewind window, disk budget, reassignment recovery time, and failure-period safety margin. Include peak replay and maintenance windows rather than only average consumer lag.

2. What if the remote object store is briefly unavailable?

Pause or throttle reads beyond the local window, preserve the verified local range, and alert on upload and read failures. Production and real-time consumption continue within validated local capacity; historical reads catch up after recovery. Expose an explicit state for unreadable offsets instead of returning empty data.

3. How do you prove deletion is safe?

Before deleting a local segment, verify that its remote object, indexes, and metadata are readable and record the segment range. For remote deletion, audit the retention policy, sample reads, and run restoration drills; alert on failed deletion, orphan objects, and metadata gaps.