Question and When It Applies
A service sees low throughput and queueing delay on high-bandwidth, long-RTT, or shared links. The team wants to change Linux TCP congestion control from CUBIC to BBR but worries about competing flows, bursts, and kernel-version differences. Explain the mechanisms and propose a rollout.
What the Interviewer Evaluates
- Separating congestion control, reliable delivery, and application retries.
- Explaining loss-based window growth versus sending-rate control from bandwidth and RTT estimates.
- Identifying queue buildup, shared-link fairness, app-limited traffic, and implementation differences.
- Validating with real workloads, tail latency, and rollback gates instead of one throughput run.
Clarifying Questions Before You Answer
- Is the traffic long-lived bulk transfer, short RPC, or frequently app-limited?
- Where is the bottleneck, and do tenants or different congestion algorithms share the queue?
- Which kernel, NIC offload, queue discipline, and BBR version are available?
- Is the goal throughput, p99 latency, cost, or stability on lossy paths?
- Can the change be canaried by host, service, or connection percentage with fast rollback?
30-Second Answer Framework
CUBIC mainly adjusts from congestion-window and loss feedback; it is mature and predictable on shared networks. BBR estimates bottleneck bandwidth and minimum RTT and tries to approach capacity while controlling the queue. BBR can reduce queueing delay but depends more on versions, workload, and fairness. I would establish a CUBIC baseline, canary BBR by host, compare goodput, p99 RTT, retransmissions, queue depth, CPU, per-flow share, and errors, and keep an immediate rollback.
Step-by-Step Deep Dive
Step 1: Separate TCP control responsibilities
TCP provides ordered reliable delivery. Congestion control limits sending from network feedback, while flow control reflects receiver capacity. Application timeouts and retries do not replace congestion control; retries can amplify congestion.
Step 2: Explain CUBIC signals and trade-offs
CUBIC uses congestion-window and loss signals to infer congestion, with a window-growth function that recovers efficiently on high-bandwidth, long-RTT paths. It is mature, widely deployed, and well understood with existing devices and operations. Its downside is that it may build a deeper queue before a clear loss-driven reduction.
Step 3: Explain the BBR model
BBR estimates bottleneck bandwidth from delivered rate and propagation delay from minimum RTT, then uses their bandwidth-delay product to control sending. It alternates between probing bandwidth and draining the queue, aiming for high goodput with fewer losses. Measurements are sensitive to noise, app-limited traffic, and path changes.
Step 4: Discuss fairness and queue risk
Different algorithms sharing a bottleneck do not guarantee equal bandwidth. BBR version, parameters, queue management, and flow count all matter. An overestimated sending rate can increase queue delay. Measure each flow’s share and RTT distribution; total throughput can hide one class being squeezed out.
Step 5: Design an experiment matrix
Cover short RPCs, long downloads, app-limited traffic, varied RTT and loss, single and many flows, and homogeneous and mixed algorithms. Use fixed content and matched hosts. Record goodput, p50 and p99 RTT, retransmissions, loss, queue depth, CPU, and completion time.
Step 6: Canary by host and keep rollback
Enable BBR in isolation first, then canary by service, zone, or a small host percentage. Version kernel and queue settings, monitor anomalies, and pause automatically. Roll back when p99 RTT, errors, bandwidth fairness, or downstream completion exceeds a threshold, preserving comparison data.
Step 7: State the evidence boundary
BBR results do not generalize to every path, kernel, or application. CUBIC’s maturity does not make it optimal on every long-RTT path. Bind the choice to workload, network operator, queue management, and business goal, then retest.
High-Quality Sample Answer
I would first separate responsibilities: TCP delivers reliably, congestion control regulates sending, and application retries cannot replace it. CUBIC mainly uses congestion-window and loss feedback. It is operationally mature but may build a deep queue before reducing. BBR estimates bottleneck bandwidth from delivered rate and propagation delay from minimum RTT, then controls sending from the bandwidth-delay product. It can reduce queueing delay but depends more on version, queue, and mixed-flow fairness. Before rollout I would establish a matched CUBIC baseline covering long flows, short RPCs, app-limited traffic, varied RTT, single flows, and mixed algorithms. During a host canary I would compare goodput, p99 RTT, retransmissions, queue depth, CPU, completion time, and per-flow share. Any tail-latency, error, or fairness regression triggers automatic rollback to CUBIC.
Common Mistakes
- Saying BBR has no loss or CUBIC only looks at bandwidth.
- Replacing a workload matrix with one iperf throughput result.
- Ignoring app-limited flows, mixed algorithms, or the queue manager.
- Monitoring total bandwidth without per-flow fairness and p99 RTT.
- Changing sysctl without confirming the kernel, NIC, and queue settings took effect.
- Skipping a small canary and a verifiable rollback path.
Follow-Up Questions and Responses
Follow-up 1: Is BBR always faster than CUBIC?
No. Results depend on RTT, bandwidth, loss, queue, flow count, and whether the application is limited. Define the goal and compare under matched conditions rather than judging by the algorithm name.
Follow-up 2: Why measure minimum RTT?
It approximates propagation delay and helps separate path delay from queueing delay. If the baseline is polluted by queueing, bandwidth-delay and queue decisions are biased.
Follow-up 3: What happens when BBR and CUBIC share a link?
Competition may be unfair, depending on version, queue management, flow count, and path. Measure per-flow share, RTT, and loss at the shared bottleneck instead of BBR throughput alone.
Follow-up 4: Should short RPCs switch too?
First check whether they are persistently app-limited and how much connection reuse and handshake cost dominate. The benefit may not justify the configuration risk, so canary by service instead of switching globally.
Follow-up 5: How do you attribute a p99 regression to congestion control?
Compare the same host, path, and application version while correlating RTT, queue depth, retransmissions, congestion window, and completion time. Exclude CPU, TLS, server queueing, and application-retry changes.
Follow-up 6: What happens after rollback?
Confirm new connections use the original algorithm and decide whether old connections need recreation. Remove canary settings, preserve experiment data and trigger thresholds, and fix queue or kernel causes before scheduling a separate algorithm experiment.