Repository navigation
Cannot construct new Buffer in worker thread after transferring buffer out #32752
Description
After transferring a Buffer instance to a different thread, it becomes impossible to construct new Buffers in the current thread.
- Version: v12.16.2
- Platform: win10 x64
- Subsystem: buffer, worker_threads
What steps will reproduce the bug?
var worker_threads = require('worker_threads');
if (worker_threads.isMainThread) {
var worker = new worker_threads.Worker(__filename);
worker.on('message', function(msg) {
console.log(msg);
});
} else {
try {
sendBuf(Buffer.from('hello world'));
sendBuf(Buffer.from('hello world'));
} catch (ex) {
console.error(ex);
}
}
function sendBuf(buf) {
worker_threads.parentPort.postMessage({
buf: buf
}, [ buf.buffer ]);
}What is the expected behavior?
It should not crash node
What do you see instead?
{
buf: Uint8Array(11) [
104, 101, 108, 108,
111, 32, 119, 111,
114, 108, 100
]
}
TypeError: Cannot perform Construct on a neutered ArrayBuffer
at new Uint8Array (<anonymous>)
at new FastBuffer (internal/buffer.js:945:1)
at fromStringFast (buffer.js:427:11)
at fromString (buffer.js:452:10)
at Function.from (buffer.js:302:12)
at Object.<anonymous> (/worker-test.js:14:18)
at Module._compile (internal/modules/cjs/loader.js:1156:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1176:10)
at Module.load (internal/modules/cjs/loader.js:1000:32)
at Function.Module._load (internal/modules/cjs/loader.js:899:14)
It looks like after the first Buffer is transferred out, the allocPool contains a now-neutered ArrayBuffer.
If I call createPool() before
Line 427 in 3443e3a
then the operation succeeds.
Activity
addaleax commentedon Apr 10, 2020
I guess it’s fair to expect this to not stop Node from creating new buffers, but if you transfer a pooled Buffer’s ArrayBuffer, bad things are bound to happen either way.
daguej commentedon Apr 10, 2020
Perhaps I'm not understanding the transfer API correctly -- is there something wrong with my example code?
Bufferand put some data in it.postMessage, the buffer is transferred to a different thread.Buffer, but now the operation fails.From the perspective of user code, I think that usage should be fine. If there's pooling going on, isn't that an internal implementation detail that user code shouldn't have to worry about?
If there's something different my example code should be doing, what is it?
Either way, it should definitely not be possible to get in to a state where it's impossible to create new buffers.
addaleax commentedon Apr 10, 2020
@daguej I agree that this situation is unfortunate.
That’s what it looks like to you, but we do warn in the documentation that
bufandbuf.bufferare not the same thing and do not necessarily cover the same memory range. When you transferbuf.buffer, you will end up transferring a lot more then just the contents ofbufitself.I think what we could do here is to mark the pool for
Buffers as untransferable. That way, this footgun is removed, and performance-wise, it should be fine because pooled Buffers are small-ish by default anyway.buffer: mark pool ArrayBuffer as untransferable
daguej commentedon Apr 10, 2020
OK, I see. I had read that documentation but unfortunately it didn't really click that it meant small
Buffers get created using a shared backing allocation, and attempting to transfer that shared backing would cause problems.Thanks for the quick response and patch.
Meanwhile, to guard against doing this in unpatched versions of node, what's the best way to detect that a
Bufferhas a shared backingArrayBufferso I don't attempt to transfer it?buf.length != buf.buffer.byteLength?buf.length <= Buffer.poolSize?addaleax commentedon Apr 12, 2020
Yup, that seems just right and can handle other kinds of pooling situations as well, not just Node’s pool for
Buffer👍buffer: mark pool ArrayBuffer as untransferable
buffer: mark pool ArrayBuffer as untransferable
buffer: mark pool ArrayBuffer as untransferable
buffer: mark pool ArrayBuffer as untransferable