Prompt and scope
A multithreaded, asynchronous Python service has intermittent CPU spikes at peak load. The team wants call stacks without instrumenting every function call or stopping the process, then wants developers to replay the result. Python 3.15 adds profiling.sampling; PEP 799 organizes tracing and sampling under one namespace.
Explain when to choose sampling or deterministic tracing, how to choose CPU or wall clocks, and how to cover threads and free-threaded builds without treating estimates as exact timings.
What the interviewer evaluates
The interviewer looks for an understanding of statistical sampling error and the difference between its low overhead and cProfile's instrumentation and coverage.
A strong answer covers attach permissions, sensitive data, sampling frequency, short-lived work, profile versioning, and replay rather than giving one command.
Clarifications to ask first
- Is the issue CPU saturation, I/O wait, lock contention, or a short spike?
- How much CPU, memory, and disk overhead is allowed in production?
- Do you need threads, async tasks, GIL state, or native-extension stacks?
- May an operator attach to an existing process, and who may access the data?
- Is the goal hotspot discovery, version comparison, or proving a regression is gone?
A 30-second answer
“I would use low-overhead sampling to find sustained hotspots, then confirm short paths with deterministic tracing in a small canary or offline reproduction. profiling.sampling reports sample estimates, not exact function time. I would collect CPU and wall views, retain thread and build metadata, encrypt and redact profiles, and monitor the profiler's own cost. If overhead or privacy risk is unacceptable, revoke attach access and use offline profiling.”
Step-by-step solution
Define the sampling question
Sampling estimates hotspots over time with low intrusion, but cannot guarantee a very short function is observed or provide exact per-call timing. Define CPU, wall, lock-wait, and tail-latency signals before choosing clock and duration.
Separate sampling and tracing
PEP 799 places deterministic tools under profiling.tracing and keeps cProfile as a compatibility alias; sampling lives under profiling.sampling. Tracing records every call and suits short flows or call counts at higher cost. Sampling observes stacks periodically and suits production hotspots and long requests.
python -m profiling.sampling record --pid 1234 --clock cpu --duration 30 --output profile.bin
python -m profiling.sampling replay profile.bin --view flamegraphVerify command options against the target 3.15 build; the snippet describes a workflow, not a promise that every beta has identical flags.
Choose CPU and wall clocks
CPU answers how much processor time a thread consumed; wall includes sleep, I/O, and waiting. CPU-only sampling can miss an end-to-end latency problem, while wall-only sampling can mislabel waiting as computation. Store the clock type in profile metadata.
Cover threads, async, and free-threaded builds
Aggregate stacks by thread or task instead of looking only at the main thread. For async services, distinguish event-loop computation from I/O waiting. Free-threaded builds need contention and native-extension context. Record instance, interpreter build, and thread identifiers so deployments can be compared.
Control overhead and missed short work
Higher frequency improves resolution but increases stack-reading and write cost. A short task may finish between samples; zero samples do not prove zero execution. Extend the window, aggregate instances, or use offline tracing for critical paths, while measuring dropped samples and profiler CPU.
Protect data and permissions
Attach requires process permission, and profiles may contain module names, paths, and business functions. Limit attach operators, keep request data out of labels, encrypt binaries, set TTLs, and share only redacted copies. Operational logs contain profile ID, version, and configuration, not keys or payloads.
Replay, compare, and set regression gates
Save clock, rate, duration, interpreter version, commit hash, and load window. Compare hotspot share, thread distribution, wall/CPU difference, and sample count; percentages from different rates are not directly comparable. Pair profiles with the same benchmark load and block releases only when a predefined threshold is exceeded.
Canary, stop, and roll back
Enable a short, revocable window on one instance first. If CPU, memory, permission, or privacy risk exceeds the budget, stop new attaches, revoke temporary access, and delete expired files. Keep the service running and switch later analysis to offline reproduction and tracing.
Model high-quality answer
“Sampling is a low-intrusion locator, not an exact timer. I would collect CPU and wall views with profiling.sampling on one instance, then confirm short paths with profiling.tracing or a controlled benchmark. Every profile carries build, commit, clock, rate, and load metadata; files are encrypted, redacted, access-controlled, and short-lived. Comparisons use identical parameters and focus on hotspot share, thread distribution, and thresholds. If the profiler costs too much or exposes data, revoke attach access and return to offline analysis.”
Common mistakes
- Treating sample time as exact time → optimization targets are wrong → explain sampling estimates and limits.
- Looking only at CPU → I/O wait is missed → collect CPU and wall views for the question.
- Sampling only the main thread → worker and event-loop hotspots disappear → retain thread, task, and build metadata.
- Assuming higher frequency is always better → collection cost rises → measure profiler overhead.
- Comparing percentages from different rates → results are not comparable → standardize parameters and load.
- Keeping profiles forever → paths and sensitive data leak → redact, encrypt, restrict, and expire.
Follow-up questions and answers
Follow-up 1: When must you use tracing?
Use tracing for per-call counts, exact call relationships, or very short paths, preferably offline or on a small canary. Use sampling for long-running production hotspots where intrusion must stay low.
Follow-up 2: What if sampling misses a short task?
Longer windows or more instances raise probability but do not guarantee capture. Use benchmarks, timestamped logs, or offline tracing to cross-check; zero samples do not mean zero execution.
Follow-up 3: Why record the free-threaded build?
Scheduling, lock contention, and stack shape can change with the build mode. Without it, profile differences cannot be explained and an optimization may appear valid only on one interpreter.
Follow-up 4: How can profiles gate releases?
Fix load, clock, rate, and duration, then compare distributions rather than one sample. Block only when hotspot share, tail latency, or profiler cost exceeds a predefined threshold; send other differences to review.