Prompt and context
Upstream jobs produce several data assets each day. Reports and quality checks should run after their dependencies update. Explain how you would model producers, consumers, partitions, and recovery with Airflow asset-aware scheduling, and where it differs from time schedules and external sensors.
What the interviewer is testing
- Treating an asset as a logical data dependency that a task updates, not just a filename or cron label.
- Distinguishing asset events, DAG timetables, expression composition, and event triggers.
- Accounting for duplicate events, late data, partition granularity, quality gates, and backfills.
- Explaining monitoring, authorization, idempotency, retries, and pause/resume behavior.
Clarifying questions to ask
- Does an asset represent a whole table, a partition, or an object-storage path? Do events carry partition and batch identity?
- Must the consumer wait for every upstream asset, or can any one update trigger it? Are cross-DAG asset expressions needed?
- If an event arrives but quality checks fail, should the consumer be blocked, the producer retried, or a human release it?
- Do we need historical backfills, event replay, or compatibility with an existing cron schedule? What does a duplicate trigger cost?
A 30-second answer
I would model the data product as assets with update contracts, then let a producer DAG emit an asset update only after an atomic write and quality checks succeed. A consumer DAG uses asset dependencies or expressions to define its trigger condition instead of guessing readiness with a shorter cron interval. I would define partition, idempotency, duplicate-event, late-update, and backfill rules, then monitor event latency, waiting DAGs, retries, and freshness. If updates originate outside Airflow, I would evaluate the recoverability and security boundaries of an event-driven trigger.
Step-by-step deep dive
1. Establish an asset contract
The official Airflow Assets documentation defines assets as data dependencies shared between DAGs. A producer should update an asset only after the data commit and contract checks succeed; creating a temporary file, starting a task, or partially writing data is not readiness. Keep the asset URI, owner, and partition granularity stable so consumers can distinguish new data from old data.
2. Select the trigger logic
A consumer DAG can depend on one or more assets. The official Asset-Aware Scheduling documentation supports logical combinations that express conditions such as all assets updated or any asset updated; a timetable can remain an independent constraint. Define what happens when event and time conditions coincide, and do not mistake an expression for a filter over row contents.
3. Handle partitions and duplicates
An asset event does not replace a data-partition watermark. Carry a traceable batch or partition identity, and use a watermark table, unique key, or transactional write for idempotency. Duplicate events, task retries, and scheduler recovery can all re-evaluate dependencies, so consumers must be safely rerunnable instead of assuming exactly one event.
4. External events, quality, and recovery
When updates come from a queue or another system, use the official event-driven scheduling documentation to choose a trigger, then verify recoverable checks, connection credentials, and resource release. Keep an asset unready when quality fails, and retry or require an explicit release. For backfills, decide whether to emit events, isolate historical partitions, and prevent a replay from overwriting the real-time result.
Model answer
I would first define the asset contract: URI, partition key, producer, quality gate, and batch identity. A producer DAG updates the asset only after an atomic write and quality checks. A consumer DAG uses asset dependencies or expressions for all-upstream versus any-upstream conditions, optionally combined with a timetable. Events carry partition and batch identity; the consumer uses watermarks and idempotency keys for duplicates, retries, and scheduler recovery. External updates use a controlled trigger with bounded credentials and resource lifetime. I monitor event latency, waiting tasks, freshness, and replay failures. Backfill and real-time paths use separate batch boundaries so historical replay cannot overwrite the latest result.
Common mistakes
- Treating an asset as an arbitrary path without defining update ownership and readiness.
- Shortening a cron interval without a data-ready signal, partition contract, or quality gate.
- Assuming asset events are delivered exactly once and ignoring retries, duplicates, or scheduler recovery.
- Treating an asset expression as a row-content filter and omitting the actual partition watermark.
- Letting an external trigger hold connections or credentials indefinitely without timeout and cleanup.
- Reusing the real-time DAG for backfills and allowing historical replay to overwrite live output.
Follow-up questions and responses
What if two upstream assets arrive at different times?
Decide whether the consumer waits for all assets or can publish a partial result. For all-assets semantics, track each partition's arrival watermark and alert on timeout. For partial results, publish a version so consumers know a later asset may update it.
How would you test duplicate events?
In a test environment, publish the same asset update twice, retry the producer, and restart the scheduler. Verify consumer unique keys, watermarks, and transaction boundaries, including row counts, versions, and external side effects such as notifications or charges.
When would you keep an external sensor?
Keep one temporarily when the dependency system cannot emit Airflow asset events, only exposes a controlled polling interface, or must remain compatible during migration. Record the migration deadline and polling cost, then move the upstream system toward a verifiable asset update signal.