How would you use Go's FlightRecorder for production diagnostics?

Problem and context

Go 1.25 adds runtime/trace.FlightRecorder, which continuously keeps a recent execution-trace window in memory and can snapshot it when a problem is detected. It is intended for rare events in long-running services where starting a full trace after the symptom is too late.

Assume a service has a latency trigger, a bounded diagnostic budget, and an object store for trace files. The design must avoid unbounded memory, duplicate snapshots, sensitive-data leakage, and a slow incident path.

What interviewers evaluate

Interviewers look for an accurate lifecycle: create a recorder, start it once, trigger a bounded WriteTo, and stop it during shutdown. Strong answers discuss one-recorder limits, single-writer snapshot behavior, age and size configuration, sampling, redaction, and backpressure.

An ordinary answer says “turn on tracing when latency spikes.” A strong answer explains why a rolling window must already be running and how to keep the trigger from causing a thundering herd of snapshots.

Questions to clarify first

• What latency or error signal triggers a snapshot, and how noisy is it?
• How many instances may trigger at once, and is there a fleet-wide capture budget?
• What window age and byte size fit the memory budget and incident latency?
• Can trace files contain request data, credentials, or tenant identifiers?
• Where are snapshots uploaded, retained, encrypted, and access-audited?

If the signal is noisy, add sampling and cooldown before capturing. If the service cannot protect trace contents, restrict the recorder or use a safer diagnostic signal.

A 30-second answer

“I would run one FlightRecorder per process with a bounded age and memory configuration, then trigger a snapshot only on a sampled, debounced SLO violation. A single writer serializes WriteTo; the request path should enqueue work and return quickly. I would redact or encrypt trace files, cap fleet-wide captures, monitor recorder errors and upload latency, and stop the recorder cleanly during shutdown.”

Step-by-step design

1. Create and start once. Construct FlightRecorderConfig with a bounded window, call Start, and expose startup failure as a metric rather than silently assuming capture works.
2. Choose the window. Pick minimum age and buffer size from the expected diagnosis interval and memory budget. A larger window improves context but increases memory and upload cost.
3. Design the trigger. Use a latency, error, or health signal with sampling, cooldown, and per-process and fleet-wide quotas. Do not let every request call WriteTo.
4. Serialize snapshots. WriteTo permits only one concurrent writer. Put capture jobs on a bounded channel, drop or coalesce duplicates, and keep the hot path non-blocking.
5. Protect data. Treat traces as sensitive operational data. Encrypt in transit and at rest, limit retention and access, and attach incident metadata without copying raw payloads into logs.
6. Operate safely. Record capture success, bytes, duration, dropped triggers, upload failures, and recorder state. Stop during graceful shutdown and verify that in-flight writes finish.

Alternatives include conventional start/stop tracing for controlled tests, profiles for CPU or heap questions, and structured request logs for business context. A flight recorder is strongest when the symptom is rare and the useful evidence precedes detection.

Example answer

“I would start one recorder per process with a five-second bounded window sized from memory limits. A sampled p99 violation can enqueue one capture per minute per instance, while a fleet quota prevents an incident from filling object storage. The worker calls WriteTo serially, encrypts the trace, and uploads it asynchronously; the request path only records that a capture was requested. I would expose capture errors, dropped triggers, upload latency, and bytes, and stop the recorder gracefully so concurrent writes complete.”

Common mistakes

• Error: Starting tracing after the alert fires → Why it fails: the preceding evidence is gone → Fix: keep a bounded rolling window active.
• Error: Letting every request snapshot → Why it fails: concurrent writes and storage storms overload the service → Fix: sample, debounce, and queue one writer.
• Error: Ignoring trace sensitivity → Why it fails: operational evidence may expose tenant data → Fix: encrypt, restrict, redact, and retain briefly.
• Error: Treating Start or WriteTo errors as impossible → Why it fails: capture silently disappears during incidents → Fix: publish explicit metrics and fallback behavior.

Follow-up questions and responses

What if two goroutines call WriteTo simultaneously?

Serialize calls through a single capture worker. The API returns an error for an in-progress concurrent write, so callers should coalesce triggers rather than retry in a tight loop.

How do you choose the buffer size?

Start from the time between root cause and detection, then bound memory per process and across the fleet. Validate with representative trace volume and observe dropped context.

Could a trace snapshot block the request path?

Do not write to a slow network destination synchronously. Queue a bounded job, snapshot to a controlled buffer or file, and upload asynchronously with a timeout and drop policy.

What happens during shutdown?

Stop accepting new captures, call Stop, wait for in-flight writes to finish, and close the upload path. Record whether the final snapshot completed or was intentionally dropped.