Linux interview: How would you diagnose high I/O wait?

Prompt and context

A Linux host reports %iowait rising from 3% to 38%, while application p99 latency doubles and total CPU utilization remains 30%. Explain how you would distinguish a local block device, NFS or another remote store, memory reclaim, hypervisor steal, and a mismatch between host and service measurements.

Start with the meaning and limits of iowait in /proc/stat. Then build evidence with vmstat, iostat, pidstat, PSI, task states, cgroup metrics, and kernel logs. The numbers are fictional practice data, not universal alert thresholds.

What the interviewer is testing

Accounting boundary

The candidate should know that iowait is a CPU accounting field, not device utilization or the total time that all tasks spend waiting. Multicore scheduling makes attribution imperfect.

Evidence layers

A strong answer moves from host averages to per-CPU, thread, cgroup, device, and business-tail-latency evidence, and explains what each observation changes.

Wait paths

The answer should separate local block I/O, NFS/FUSE, database RPC, memory reclaim, and hypervisor steal. Each path has different evidence and mitigation.

Safe closure

The candidate preserves evidence, chooses low-risk mitigation tied to the hypothesis, and verifies recovery instead of restarting or increasing concurrency by default.

Questions to clarify first

• Is iowait from the host or a container? Host /proc/stat and service cgroup metrics may cover different populations.
• Is every CPU high, or only one? Averages hide affinity, NUMA, and quota-induced local saturation.
• Is the request waiting on a file, block device, or network RPC? NFS, database, and HTTP waits have different evidence.
• Do p99 and I/O signals share the exact window? Align sampling, deployment, backups, and traffic changes.
• Are swap, major faults, memory PSI, or steal time rising?
• Can you read thread stacks and cgroup files? Use least privilege during production investigation.

A 30-second answer

“Thirty-eight percent iowait is a clue, not proof of a disk failure. I would align the host, service, and p99 windows, then inspect per-CPU mpstat, vmstat r/b, CPU/IO/memory PSI, cgroup quota, and steal time.

If device queues, latency, process reads/writes, and kernel errors are abnormal together, I would investigate block storage. If threads wait in NFS or filesystem paths while local devices are healthy, I would inspect mounts, network, and the server. If memory PSI, swapping, and major faults rise, I would investigate the working set; if steal rises, I would investigate the hypervisor. I would verify business tail latency, waiters, PSI, and the resource after mitigation.”

Step-by-step deep answer

Step 1: Define the iowait boundary

/proc/stat iowait is CPU-time accounting associated with waiting for I/O completion. Another task may run while one task waits, and multicore accounting cannot always attribute the wait precisely. Low iowait does not rule out remote storage or kernel waits; high iowait does not prove a disk failure.

Step 2: Check per-CPU data and queues

mpstat -P ALL 1 10
vmstat 1 10
cat /proc/loadavg

Sustained r with busy CPUs supports CPU queueing; sustained b points to uninterruptible waits. Per-CPU data, affinity, cgroup quota, and throttling can reveal “30% overall, saturated locally.”

Step 3: Measure stalled time with PSI

for r in cpu io memory; do echo "[$r]"; cat /proc/pressure/$r; done

PSI some measures time when at least some work is stalled; full measures time when all non-idle work is stalled. avg10/60/300 show trends. Read the target cgroup’s pressure files to separate service pressure from host noise.

Step 4: Locate threads and wait points

ps -eLo state,pid,tid,wchan:32,comm --sort=state
pidstat -d -p ALL 1 10

Group by R/D state, command, cgroup, and wchan. If permitted, inspect /proc/PID/stack for representative threads. A wchan is only a current sleep location; require temporal correlation, PSI, and business latency before calling it the cause.

Step 5: Enter the relevant subsystem

iostat -xz 1 10
dmesg -T | tail -200
cat /proc/meminfo | egrep 'Swap|Dirty|Major'

For block devices, inspect await, queue depth, throughput, and errors. For NFS/FUSE, inspect mounts, retransmits, network, and server health. For memory, correlate memory PSI, swap, and major faults. Database or HTTP RPC waits require tracing and pool metrics; iostat cannot see them alone.

Step 6: Mitigate and verify

Capture thread, PSI, device, and log snapshots, then rate-limit, pause batch work, switch to a healthy replica, or repair a mount according to the evidence. Do not repeatedly kill -9 D-state tasks; they normally need to leave the uninterruptible wait before handling the signal. Verify p99, errors, R/D counts, PSI, resource latency, and backlog together.

Model answer

“Thirty-eight percent iowait is a clue, not a conclusion. It is CPU accounting affected by multicore scheduling, so I would not call it a disk failure immediately. I would align scope and time windows, then inspect per-CPU mpstat, vmstat r/b, CPU/IO/memory PSI, cgroup quota, and steal time.

If D-state threads and I/O PSI rise, wchan points to block I/O, and iostat shows higher latency and queues, I would inspect the device, filesystem, and kernel errors. If local devices are healthy but threads cluster in NFS waits, I would inspect mounts, retransmits, network, and the server. If memory PSI, swap, or major faults rise, I would control the working set; if steal rises, I would investigate the hypervisor.

I would pause I/O-amplifying batch work, rate-limit, or switch to a healthy replica while preserving evidence. Recovery means business p99 and errors, R/D waiters, PSI, resource latency, and backlog return toward baseline, not merely that iowait falls.”

Common mistakes

• Treating iowait as disk utilization; correlate it with iostat and a device baseline.
• Adding CPU as soon as iowait rises; check per-CPU data, R/D states, and PSI first.
• Ruling out I/O because iowait is low; inspect D states, remote storage, and I/O PSI.
• Calling every D state a local-disk problem; NFS, filesystems, and drivers can block too.
• Looking only at host metrics; the service cgroup may have different limits and pressure.
• Taking one sample; use continuous, timestamped samples aligned with the request curve.
• Restarting or killing immediately; preserve evidence and assess data safety.
• Waiting only for iowait to decline; verify user latency and resource queues.

Follow-up questions

Follow-up 1: Why can iowait be high while disk util is low?

The wait may be in NFS, FUSE, a database, or a network RPC, or the device mapping and windows may differ. Follow the thread wait point, tracing, I/O PSI, mount, and network signals.

Follow-up 2: Why can iowait be low while requests are stuck?

Tasks may wait on locks, pools, CPU quota, memory reclaim, or remote RPC. Compare R/D states, stacks, CPU/memory PSI, and dependency latency; CPU accounting does not classify every wait as iowait.

Follow-up 3: How do you tell whether a container has its own I/O pressure?

Read the target cgroup’s io.pressure, I/O statistics, and throttling events, then compare them with host /proc/pressure/io and service p99. High host pressure with low service pressure does not imply that scaling the service will help.

Follow-up 4: Can you kill a D-state thread?

You can send a signal, but it normally cannot be handled until the uninterruptible wait returns. Capture the stack and wait point, repair the device, mount, or driver, and consider a restart only after redundancy and data safety are confirmed.

Follow-up 5: How would you alert on iowait?

Avoid a universal percentage. Combine iowait with I/O PSI, local or remote storage latency, D-state counts, the business SLO, and duration, calibrated to the workload’s baseline.