Repository navigation
Piping readable stream to multiple writeable streams results in crash #5820
Description
If I execute the following code snippet in node >= 4.2.0, it results in the read stream reading the data very fast but not being able to write them out to wr2 equally fast. The memory usage of the process increases very fast until the process exits with an error.
var fs = require('fs');
//M: and L: are local disks with fast read and write
// the .mov file is several gigabytes in size
var rd = fs.createReadStream('M:/A007C003_141024_R7C6.mov');
var wr = fs.createWriteStream('L:/A007C003_141024_R7C6.mov');
//N: is a NAS drive, slow
var wr2 = fs.createWriteStream('N:/A007C003_141024_R7C6.mov');
//read from fast disk, write to both fast and slow disk
rd.pipe(wr);
rd.pipe(wr2);The script ends with this error:
.\node-4.2.0 test.js
buffer.js:98
const ui8 = new Uint8Array(size);
^
RangeError: Invalid array buffer length
at new ArrayBuffer (native)
at new Uint8Array (native)
at allocate (buffer.js:98:17)
at new Buffer (buffer.js:49:12)
at Function.Buffer.concat (buffer.js:234:16)
at fromList (_stream_readable.js:840:20)
at ReadStream.Readable.read (_stream_readable.js:339:11)
at flow (_stream_readable.js:743:26)
at WriteStream.<anonymous> (_stream_readable.js:601:7)
at emitNone (events.js:67:13)
at WriteStream.emit (events.js:166:7)
Earlier versions of node (4.1.2 and below) do not show the same behaviour.
- Version: v4.2.0
- Platform: Win10 64bit
- Subsystem:
Activity
Fishrock123 commentedon Mar 21, 2016
v4.2.0 has this commit: #2325 -- #2325
Sounds like it might be related?
calvinmetcalf commentedon Mar 30, 2016
though the error in question isn't actually an error related to running out of memory which is the error you'd expect if it was actually due to improper back pressure handling between 2 streams.
kr3l commentedon Mar 31, 2016
From the error and the observed memory behaviour, maybe the 'size' variable is becoming too big and overflowing before another memory-related error can occur?
addaleax commentedon Mar 31, 2016
There’s a maximum size of 2^31-1 bytes for a Buffer, so that would definitely make sense, even without an actual overflow occurring.
addaleax commentedon Apr 2, 2016
I’ve looked a bit into this and it seems @Fishrock123 is right, #2325 effectively – and probably inadvertently – disabled the
readableState.awaitDrainfunctionality that’s supposed to keep the destination streams in sync (if I read the code correctly).Fishrock123 commentedon Apr 2, 2016
cc @mscdex here too
addaleax commentedon Apr 2, 2016
PR: #6023
stream: Fix readableState.awaitDrain mechanism
stream: Fix readableState.awaitDrain mechanism
stream: Fix readableState.awaitDrain mechanism
stream: Fix readableState.awaitDrain mechanism
stream: Fix readableState.awaitDrain mechanism
stream: Fix readableState.awaitDrain mechanism
stream: Fix readableState.awaitDrain mechanism
stream: Fix readableState.awaitDrain mechanism