Process Large Files in Node.js with Streams Instead of Loading Them Whole

A practical approach

Step 1

Identify code that reads whole files or responses into memory, such as fs.readFile on large files or building one huge string before sending.

Step 2

Replace it with a readable stream, such as fs.createReadStream, and process chunks or lines as they arrive.

Step 3

Connect stages with pipeline from node:stream/promises, which forwards errors and closes every stream on failure.

Step 4

For line-based files, use the readline module or a streaming parser for formats such as CSV or JSON Lines.

Step 5

Stream responses to clients where possible, so the first bytes are sent before the whole result is ready.

Worked example

An admin endpoint exports all webinar registrations as CSV by querying every row, building one large string and sending it. As registrations grow, memory spikes and the process is restarted during exports. Rewriting it to stream rows from a database cursor through a CSV transform into the response keeps memory flat and starts the download immediately.

A mistake to avoid

Do not connect streams with .pipe() without handling errors on every stream. An error in one stage can leave others open and leak file handles. pipeline handles cleanup for you.

Try it yourself

Generate a test file of several hundred megabytes. Count its lines once with fs.readFile and once with a stream and readline, and compare peak memory use reported by process.memoryUsage().

Your implementation checklist

• Large files and responses are processed as streams
• Stages are connected with pipeline
• Errors in any stage clean up all streams
• Line-based data uses readline or streaming parsers
• Large responses are streamed to the client