Repository navigation
fix(gateway): stop retaining every payload we gzip - #442
Merged
Merged
Conversation
Compressing a stored Responses payload or a dump body went through `new Blob([bytes]).stream()`. A Blob is a Node `BaseObject` registered in the realm's `BaseObjectList`, which is a GC root, and its stream leaves the source buffer attached to it, so every compression pinned a full copy of its input for the life of the process. The bytes are native, so `heapUsed` stayed flat while RSS climbed — the shape of an OOM kill on a gateway that looks healthy. Feed the compression transform from a `ReadableStream` we build ourselves instead. The pipeline is otherwise identical, still uses the web-standard streams both deployment targets provide, and drops the defensive `new Uint8Array(bytes)` copy that only existed to satisfy the lib's `BufferSource` narrowing. The two call sites had grown their own copies of the same four lines; they now share one module. An ESLint rule bans `new Blob` in server-side production sources, so the retention cannot come back through a new call site. Tests and `apps/web` are exempt: the hazard is retention across a long-lived process. Measured on Node 24.18.0, 300 gzips of a 256 KiB payload with forced GC: `arrayBuffers` grew to 75.5 MiB before and stays at 0.5 MiB after. The leak only bites the affected Node line — v24.16 through v24.18 — and the shipped image pins 22.23.1, where the old code measured flat. nodejs/node#63574 nodejs/node#64105
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ReadableStreamwe build instead ofnew Blob([bytes]).stream(), which pins its source buffer to a GC root for the life of the processnew Uint8Array(bytes)copy that existed only to satisfy the lib'sBufferSourcenarrowingnew Blobin server-side production sources so the retention cannot return through a new call siteThe defect
Blobis a NodeBaseObjectregistered in the realm'sBaseObjectList, which is a GC root, andBlob.prototype.stream()leaves the source buffer attached to it. Every payload the gateway compressed therefore stayed resident for the life of the process. The retained bytes are native, soheapUsedstayed flat while RSS climbed — a gateway storing Responses payloads leaks roughly one full copy of every stored item, and nothing in the heap profile shows it.Upstream: nodejs/node#63574 (confirmed-bug), nodejs/node#64105.
The bug is present on Node v24.16.0 through v24.18.0.
docker/DockerfilepinsNODE_VERSION=22.23.1, where the old code measures flat, so the shipped image was not affected — a self-built or self-hosted deployment on the 24.x line was.Memory benchmark
Node 24.18.0 on macOS arm64, forced GC before every sample.
Helper in isolation, 300 gzips of a 256 KiB payload:
arrayBuffersWhole gateway on a real SQLite database, 200
/v1/responsesrequests withstore: trueand a 256 KiB input, stubbed Ollama upstream:arrayBuffersgrowthheapUsedgrowthThe residual RSS growth is present with and without the change and decelerates; it does not track payload size.
Guard
The ESLint rule covers
packages/,apps/platform-*,tools/andscripts/, and exempts__tests__andapps/web: tests construct Blobs deliberately as fixtures for code that must accept a caller-supplied one, the dashboard uses Blob to hand bytes to a browser download, and the hazard is retention across a long-lived process. The rule was verified to fire on a probe in production code and to stay silent onapps/web.A runtime regression test was considered and rejected: the retention only reproduces on the affected Node line, so on CI's Node the assertion would pass against the defective code as well and give false assurance. The lint rule is the version-independent guard.
Test Plan
apps/web