Backend Interview: How Do You Design a Cancellable Node.js stream.compose Pipeline?

Prompt and context

A Node.js service receives large uploads and must run decompression, virus scanning, format validation, and object-storage writes in sequence. The old implementation listens to data events manually, sometimes grows memory, keeps running after a client disconnects, and leaves temporary files when an intermediate step fails.

Use stream.compose() or an equivalent pipeline composition approach. Explain how readable, writable, and Transform stages connect, and how backpressure, AbortSignal, error propagation, and final cleanup keep one job controllable.

What the interviewer tests

• Whether you distinguish the boundaries and lifecycles of pipe, pipeline, and compose.
• Whether you can explain how backpressure limits production instead of merely increasing queue limits.
• Whether cancellation, exceptions, and client disconnects propagate through the whole pipeline.
• Whether you handle async generators, resource release, idempotent writes, and observability.

Questions to clarify

1. Does input come from an HTTP request, a file, or an object-storage SDK? Are multipart output and retries supported?
2. Is each stage a Node stream, Web Stream, AsyncIterable, or ordinary function?
3. Do scanning and validation create a child process, temporary file, or database state?
4. Can object storage abort a multipart upload, and must a cancelled job resume?

30-second answer

I would define each stage as a readable, writable, Transform, or AsyncIterable and compose them into a Duplex with stream.compose(), then let pipeline drive the final destination. Producers continue only when downstream can accept data, so backpressure bounds memory. Request disconnects, deadlines, and business cancellation share one AbortSignal passed to supported stages; any error fails the chain. Temporary files, child processes, and multipart uploads are cleaned in finally or abort handlers, and writes use idempotency keys. Metrics cover throughput, queue depth, peak RSS, cancellation reason, and cleanup outcome.

Step-by-step deep dive

Define each stage contract

Specify input and output types, chunk size, whether null is allowed, blocking behavior, and resource ownership at completion. An async generator must consume its source and yield on demand; an ordinary function must not silently read the whole file into memory.

Compose reusable stages

compose connects streams, AsyncIterables, or functions into a new Duplex and handles adjacent stages with pipeline semantics. Example:

import { compose } from 'node:stream';

async function* validate(source) {
  for await (const chunk of source) {
    checkChunk(chunk);
    yield chunk;
  }
}

const processing = compose(decompress(), validate, scan());

The real implementation must connect processing to the destination writable and centralize completion and errors rather than swallowing them per stage.

Make backpressure the default control plane

When downstream is not ready, Readable and Transform stages should stop producing. Do not push without bounds in a data callback or hide a slow consumer by endlessly raising the high-water mark. Load tests should record each queue, throughput, and RSS so the slowest stage determines total speed.

Propagate cancellation and disconnects

Combine request disconnect, deadline, and manual cancellation in one AbortController. Pass its signal to supported compose stages and external SDKs; after abort, stop reading, destroy downstream, and await close. Logs must distinguish normal abort, business failure, and network error.

Centralize errors and cleanup

One owner should invoke pipeline/compose, receive the first error, and destroy the chain. Temporary files, scanners, sockets, and multipart uploads must be released on success, failure, and cancellation; cleanup failures alert with the job ID without replacing the original error.

Design idempotency and metrics

Generate idempotency keys from upload ID and stage version, then commit business state only after the object write completes. Record input and output bytes, duration, peak RSS, cancellation reason, failed stage, and cleanup time. Retry only from a recoverable boundary to avoid duplicate writes.

Model answer

I would define upload, decompression, scanning, validation, and storage as stages with explicit inputs, outputs, and ownership, compose them into a readable, writable, or async-iterable pipeline, and let one pipeline owner drive it. Downstream consumption controls backpressure; unbounded buffering in data handlers is forbidden. Disconnect, timeout, and manual cancellation share an AbortSignal passed to supported stages and the storage SDK. The owner handles the first error and destroys the chain; temporary files, scanners, and multipart uploads are cleaned on success, failure, and cancellation. Idempotency keys protect writes, metrics cover throughput, queues, RSS, cancellation, and cleanup, and retries start only at safe boundaries.

Common mistakes

• Collecting the whole stream into a Buffer and still claiming to use compose.
• Listening only to the final writable’s error, missing async-generator or child-process failures.
• Continuing to read and write after the client disconnects.
• Solving backpressure by endlessly increasing the high-water mark.
• Deleting temporary files only on success, ignoring abort and exceptions.
• Retrying without idempotency keys and duplicating objects or business state.

Follow-up questions

How do compose and pipeline differ?

Compose creates a reusable Duplex from stages; pipeline drives the end-to-end connection, propagates errors, and waits for closure. They can be combined, but ownership and error handling should be centralized.

What happens when an async generator throws?

The composed stream should fail and destroy adjacent stages. The caller still waits for pipeline cleanup and records the failing stage instead of relying on process exit alone.

How do you verify backpressure works?

Load-test with a producer faster than a controlled slow sink, observe bounded queue depth and RSS, and confirm throughput follows the slowest stage rather than unlimited buffering.

Can you retry immediately after cancellation?

First confirm all resources are closed and temporary state is identifiable. Retry from an idempotent recoverable boundary; compensate or mark unknown state for external writes that cannot be interrupted.