Prompt and context
How would you deploy an eBPF observability program across Linux distributions and kernel versions? Explain what the verifier protects, how you debug load failures, and how you build a testable compatibility matrix.
This fits platform, Linux, SRE, security, and systems interviews. It tests kernel boundaries, static verification, the user-space loader lifecycle, and release discipline. An eBPF program passes through the verifier before it attaches to a kernel hook. Passing verification does not prove business correctness or compatibility with every kernel, permission set, helper, map, or BTF environment.
What interviewers assess
- Separating verifier safety constraints from business-result correctness.
- Understanding register types, pointer bounds, initialized stack, loops, and helper checks.
- Explaining the libbpf open, load, attach, and teardown lifecycle.
- Using BTF, CO-RE, feature detection, and a kernel matrix for version differences.
- Providing a debugging path for verifier logs, permissions, map resources, and attach failures.
- Knowing that eBPF remains subject to kernel bugs, helper semantics, and observation cost.
A 30-second answer
“The verifier performs static checks before execution: it proves bounded memory access and valid register and pointer states, limits helpers, and rejects paths it cannot prove safe. It cannot prove that my metrics mean the right thing. I would make object open, load, verify, attach, and teardown observable and use verifier logs to debug. BTF, CO-RE, feature detection, and a CI matrix covering kernel, architecture, permissions, and hooks establish compatibility. If a target cannot load, the service keeps a user-space or existing-metrics fallback rather than bypassing the check.”
Step-by-step solution
Step 1: Define the eBPF safety boundary
An eBPF program runs in a kernel-controlled execution environment; it cannot arbitrarily read kernel memory or call unauthorized functions. The verifier follows control flow while tracking registers, stack, map pointers, and packet pointers, rejecting paths whose safety cannot be proven. It is a safety gate, not a complete formal proof, and it does not review sampling meaning, privacy, or resource cost.
Start with the program type and hook: tracepoint, kprobe, cgroup, XDP, and others have different inputs, helpers, and return constraints. The same source can fail when moved to another program type because the context and capabilities changed.
Step 2: Recognize common verifier failures
Uninitialized registers, stack out-of-bounds access, unchecked nullable pointers, insufficient packet bounds, unprovable loops, invalid helper arguments, and invalid pointer arithmetic can all cause rejection. Read the first state failure in the log instead of patching only the last message.
/* Pseudocode: check bounds before reading packet fields */
if (cursor + sizeof(struct header) > data_end)
return DROP;
struct header *h = cursor;
return handle(h->kind);Keep paths simple, put bounds checks next to reads, and split complex parsing into small verifiable functions. Bound loops explicitly, check map lookup results, and use pointer conversions allowed by the current context.
Step 3: Explain the libbpf lifecycle
The user-space loader opens the BPF object and discovers programs, maps, and globals. Loading creates maps, resolves relocations, and asks the kernel verifier to check and load programs. Attaching connects programs to hooks; teardown detaches and frees resources. Record object name, program type, kernel version, error code, and verifier log at every phase.
A successful load is not a successful attach. Permissions, locked-memory limits, missing BTF, absent hooks, map limits, and ring-buffer resources can fail at different phases. The launcher should expose the failing phase and state which capabilities remain when only some programs load.
Step 4: Design maps, events, and resource limits
Maps provide the main shared state between kernel and user space. Select hash, array, per-CPU, or ring-buffer maps based on key, value, update concurrency, lifecycle, and capacity. High-frequency events need sampling, batching, and loss counters; observation must not exhaust kernel resources.
Minimize sensitive fields in the kernel and send a hash, category, or count when possible. User-space consumers must handle ring-buffer overflow, restart, and version changes. Lost events are explicit uncertainty, not zero observations.
Step 5: Handle BTF, CO-RE, and compatibility
BTF supplies type metadata so loaders and tools can understand programs, maps, and kernel symbols. CO-RE relocates by type and reduces per-kernel recompilation, but it does not remove every difference. Helpers, kfuncs, program types, architecture, compiler, and permissions still require feature checks.
The compatibility matrix should cover distribution, kernel version, architecture, BTF presence, feature flags, container permissions, and target hook. CI must run compile, verifier load, attach, event replay, and teardown on real or virtual kernels. Compilation alone is not compatibility.
Step 6: Build debugging, fallback, and release
Debug in this order: confirm kernel and permissions; inspect program type and hook; capture the full verifier log; check BTF and helpers; then check map resources and user-space consumption. Classify the failure as source safety, missing capability, permission, runtime resource, or business logic so the repair matches the cause.
Release first on shadow nodes or a small host canary. Monitor CPU, memory, lost events, verifier rejects, kernel logs, and the target signal. If loading fails, the user-space service keeps its existing metrics or logs. Keep the previous object and detach operation so rollback does not depend on the new program.
Information gain and boundaries
The key benefit of eBPF is constrained kernel programmability through a verifiable and observable loading path. The verifier proves a set of safety properties, not program intent; CO-RE improves portability but does not guarantee cross-version success. A strong answer covers safety, compatibility, resources, and business correctness together.
Model answer
“I would start with the program type and hook because the context defines inputs, return values, and helper capabilities. Before execution, the verifier tracks registers, stack, maps, and packet pointers and rejects uninitialized registers, out-of-bounds access, unchecked nulls, unbounded loops, and invalid helper arguments. Passing proves those paths are safe to execute; it does not prove that sampling or business metrics are correct.
The user-space loader uses libbpf to make open, load, attach, and teardown observable. On load failure, I preserve the complete verifier log and distinguish source safety, missing helpers, BTF, permissions, and resources. BTF and CO-RE reduce recompilation, but I still test a real matrix of distributions, kernels, architectures, hooks, permissions, and features across compile, load, attach, replay, and teardown.
I limit map capacity and event rate, record loss, and minimize sensitive fields in the kernel. Production starts with shadow nodes and a small canary, monitoring CPU, memory, lost events, and kernel logs. A failed load keeps the existing metrics or log path and rolls back to the previous object with detach. This uses eBPF without confusing the verifier or CO-RE with absolute safety or compatibility.”
Common mistakes
- Treating the verifier as business-proof → it does not check metric meaning or privacy → add replay, sampling, and data audits.
- Checking only compilation → load, attach, permissions, and BTF can still fail → test the full lifecycle on a kernel matrix.
- Fixing the last log line → the root cause is often an earlier state failure → start at the first register or bounds error.
- Writing unlimited maps and events → observation can exhaust kernel resources → limit capacity, sample, and count loss.
- Assuming CO-RE removes version differences → helpers, hooks, permissions, and architecture still vary → feature-detect and degrade.
- Having no user-space fallback → a failed load can affect the main service → retain metrics, logs, and detach paths.
Follow-up questions
How do you fix a possibly null pointer verifier error?
Check it explicitly on every control-flow path before dereferencing, keeping the check close to the read. If the branch is complex, split the function and inspect verifier state again rather than hiding the issue with an unsafe cast.
What if the same program loads on one kernel and fails on another?
Compare program type, helpers, kfuncs, BTF, architecture, configuration, permissions, and verifier logs instead of only comparing version numbers. Feature-detect an alternative and record the result in the compatibility matrix and release gate.
How do you prevent lost events from looking healthy?
Count loss, ring-buffer overflow, and consumer restarts in both kernel and user space. Publish sample rate, loss rate, and coverage. A dashboard should show uncertainty when events are incomplete, not zero.
Is a security review still needed after verifier approval?
Yes. Review program permissions, data minimization, the user-space loader, upgrade and rollback, kernel exposure, and resource limits. The verifier is necessary, but it does not replace threat modeling or runtime monitoring.