Data engineering interview: How do you prove DataFusion nested-field pushdown reduces scan cost?

Question

DataFusion 53 can push expressions such as get_field into a data source. How would you design queries, plans, and benchmarks to prove lower I/O and decoding cost without changing results?

Context and boundaries

Assume a Parquet table has a wide struct column s, while the query needs only s['label'] and filters on s['value']. Cover batch scans, statistics, nulls, schema evolution, and safe fallback when pushdown does not apply. Do not treat “projection appears in the plan” as end-to-end evidence.

What the interviewer is testing

The skill is separating semantic correctness, plan rewriting, data-source capability, and observable benefit. DataFusion 53 moves nested-field access closer to the scan so a full struct need not be read; its configuration docs describe enable_leaf_expression_pushdown extracting get_field from filter, sort, or join expressions and pushing it toward leaf nodes.

Clarify these points first:

• Does the source support field-level projection, and is the file format Parquet?
• What are the types, null semantics, and missing-field rules for s['label'] and s['value']?
• Is the baseline disabled optimization, an older DataFusion version, or a query that reads the whole struct?
• Should the comparison focus on scan bytes, decoding CPU, peak memory, latency, or object-store requests?

30-second answer

Start with SQL containing nested projection and filtering. Then compare logical and physical plans and the Parquet reader projection before and after optimization. Finally, use identical data, cache state, and concurrency to measure scan bytes, decode time, peak memory, and result checks, while stating when to fall back.

Step-by-step deep dive

1. Build data: create Parquet files with identical partitions, row groups, and statistics while controlling struct width, null rate, and field selectivity.
2. Define baselines: hold DataFusion version, thread count, object-store latency, and cache state constant for full-struct reads, disabled pushdown, and leaf-field reads.
3. Inspect plans: confirm get_field is near the scan and the projection contains only id, s.label, and filter field s.value; SQL text alone is insufficient.
4. Observe the source: record Parquet columns read, row-group pruning, bytes fetched, and decode batches, separating projection benefit from predicate benefit.
5. Check results: compare sorted hashes or rows, covering missing fields, nulls, type changes, empty structs, and duplicate rows.
6. Define fallback: if the source lacks field pushdown, the rewrite is unsafe, or benefit is below a threshold, keep the correct full-read path and record the reason.

Model answer

I would build three baselines from one fixed Parquet dataset: read the full s, disable enable_leaf_expression_pushdown, and enable the optimization while selecting leaf fields. The query is:

SELECT id, s['label']
FROM events
WHERE s['value'] > 150;

I would save logical and physical plans and verify that get_field sits near the scan and the scan projection no longer includes all of s. I would run multiple cold-cache and warm-cache iterations, recording bytes read, object-store requests, Parquet decode CPU, peak memory, end-to-end latency, and output rows. I would compare a stable sorted hash with the full-struct baseline, explicitly covering nulls, missing fields, and old and new schemas. If the source cannot project fields, I would keep the full read and emit a metric rather than sacrifice correctness for a prettier plan. Before rollout, scan bytes and result hashes would become regression gates.

Common mistakes

• Comparing only end-to-end latency without controlling cache, concurrency, and file layout.
• Combining field projection, predicate pushdown, and row-group pruning into one unexplained number.
• Testing only ordinary values and ignoring nulls, missing fields, and schema evolution.
• Declaring success from a rewritten plan without checking the columns and bytes the reader actually consumed.
• Forcing a rewrite when pushdown fails instead of keeping a correctness-first fallback.

A strong answer connects SQL, plan, source, and metrics; gives a reproducible baseline and result check; and explains attribution and fallback. A weak answer only says “projection pushdown is faster” without experimental controls or correctness evidence.

Follow-up questions and responses

Why might leaf-field reads still show no benefit?

The file may be row-oriented, the struct may not be physically separable, object-store requests may dominate, or the source may not implement field projection. Inspect actual bytes and decode time rather than only the plan.

If s['value'] is mostly null, how do you preserve results?

Fix the SQL null and type semantics first, then compare rows against a full-read baseline. Optimization may avoid unneeded fields but must not treat null as missing or erroneous.

What happens to old files after a nested field is added?

The reader should resolve fields by name and apply the defined null or default semantics to old files. The benchmark must include old and new files and validate the combined scan.

Interview checklist

One-sentence takeaway

Prove pushdown by checking plan placement, actual scan behavior, result equivalence, and a safe fallback when the source cannot optimize.