Prompt and context
A service is moving from Go 1.25 to Go 1.26. Green Tea GC is enabled by default, but the team is concerned about different heap sizes, allocation patterns, and CPU architectures. Explain how you would understand the change, build a baseline, canary it, observe GC metrics, and roll back if needed.
What the interviewer evaluates
- Separating a runtime implementation change from an application defect.
- Validating throughput, pauses, CPU, and memory on real workloads instead of repeating release-note numbers.
- Understanding the scope and cost of
GOEXPERIMENT=nogreenteagc. - Designing compatible builds, canaries, alerts, and an upgrade plan.
Questions to clarify before answering
- What are allocation rate, object size, heap goal, and latency SLO?
- Is the platform amd64, arm64, or mixed, and does the service use cgo?
- What are the current Go version, GOGC, CPU use, and GC baselines?
- Which signals decide continue, pause, or rollback, and who owns the release switch?
- Is there replayable traffic, an isolated canary pool, and a rollback path?
30-second answer framework
Go 1.26 enables Green Tea GC by default to improve locality and CPU scalability when marking and scanning small objects. I would compare the old and new versions on representative load, measuring p95/p99 latency, GC CPU, allocation rate, heap size, and throughput. I would canary with explicit rollback thresholds. GOEXPERIMENT=nogreenteagc is useful for an A/B diagnostic or temporary rollback, not as an unexamined permanent default.
Step-by-step deep dive
Step 1: Describe the change precisely
The Go 1.26 notes say Green Tea GC became the default after feedback, focusing on locality for marking and scanning small objects and CPU scalability. It is a runtime implementation change, not a change to Go language semantics or memory safety.
Step 2: Establish a comparable baseline
Run Go 1.25 and 1.26 with the same optimization, configuration, traffic, and hardware. Record throughput, end-to-end latency, GC CPU, allocation rate, heap goal, RSS, and tail latency rather than one average.
Step 3: Cover allocation patterns
Include short-lived small objects, long-lived objects, burst allocation, and low-allocation workloads. Different object graphs and heap sizes can behave differently; include caches, serialization, traffic peaks, and background jobs.
Step 4: Analyze runtime signals
Use runtime/metrics, pprof, logs, and business metrics to distinguish GC from lock contention, scheduling, or downstream latency. Align the sample window with GC cycles so a cold start or traffic shift is not misattributed.
Step 5: Design a canary and alerts
Start with replayable traffic and a small instance percentage, grouped by version and architecture. Set thresholds for GC CPU, p99 latency, OOMs, heap growth, and error rate; stop expansion when a threshold is crossed.
Step 6: Use the rollback switch diagnostically
GOEXPERIMENT=nogreenteagc disables Green Tea GC at build time for an A/B comparison or temporary rollback. It creates another build variant and operational cost, so record the supported toolchain, expiry date, and removal plan.
Step 7: Close the upgrade loop
Record baselines, canary results, rollback criteria, and the decision. Continue observing real traffic after upgrade, then remove the temporary switch once benefits and risks are understood instead of running a permanent runtime fork.
High-quality sample answer
I would collect Go 1.25 baselines for latency, GC CPU, allocation rate, heap goal, and RSS, then compare Go 1.26 on identical hardware and representative traffic. Tests cover allocation-heavy small objects, long-lived caches, and bursty requests, split by amd64 and arm64. The canary starts on a small instance set and stops if p99 latency, GC CPU, or OOMs exceed thresholds. To isolate Green Tea GC, I build a comparison with GOEXPERIMENT=nogreenteagc, limiting it to diagnosis and temporary rollback. After the result is clear, I update the upgrade record and remove the fork.
Common mistakes
- Promising that every service gets the release-note “10–40%” improvement.
- Looking only at GC count or average latency and ignoring tail latency, CPU, RSS, and throughput.
- Comparing versions with different hardware, traffic, or GOGC settings.
- Treating the rollback environment variable as a permanent production configuration.
- Skipping a canary and losing the ability to separate runtime from application changes.
Follow-ups and responses
Follow-up 1: Does Green Tea GC change Go semantics?
It is a runtime optimization for marking, scanning, and CPU scaling; it does not change Go language semantics. Performance and resource behavior still require validation.
Follow-up 2: When is a benchmark insufficient?
When the service has bursts, complex caches, multiple architectures, or cgo, a microbenchmark is not representative. Combine replayed traffic with canary metrics.
Follow-up 3: How do you attribute a p99 regression to GC?
Align GC cycles with GC CPU, allocation, and heap changes, compare the disable experiment, and rule out scheduler, lock, and network jitter using business metrics.
Follow-up 4: What risks come with the rollback switch?
It creates a second build variant and can complicate images, caches, and upgrades. Set an expiry and verify toolchain compatibility.
Follow-up 5: How do you canary mixed amd64 and arm64?
Pool by architecture and establish separate baselines and thresholds so one architecture's gain does not hide a regression in the other.
Follow-up 6: When do you end the canary?
After covering peaks and valleys, completing a stable business window, meeting error and resource thresholds, and retaining a tested rollback path.