Repository navigation
fs.promises.readFile is 40% slower than fs.readFile #37583
Description
- Version: 15.10.0
- Platform: macOS
- Subsystem: 11.2.1
What steps will reproduce the bug?
Run this benchmark on a 1 MB file (big.file):
const Benchmark = require("benchmark");
const fs = require("fs");
const path = require("path");
const chalk = require("chalk");
const util = require("util");
const promisifed = util.promisify(fs.readFile);
const bigFilepath = path.resolve(__dirname, "./big.file");
const suite = new Benchmark.Suite("fs");
suite
.add(
"fs.readFileSync",
(defer) => {
fs.readFileSync(bigFilepath, "utf8");
defer.resolve();
},
{ defer: true }
)
.add(
"fs.readFile",
(defer) => {
fs.readFile(bigFilepath, (err, data) => {
defer.resolve();
});
},
{ defer: true }
)
.add(
"fs.promises.readFile",
(defer) => {
fs.promises.readFile(bigFilepath).then(() => {
defer.resolve();
});
},
{ defer: true }
)
.add(
"util.promisify(fs.readFile)",
(defer) => {
promisifed(bigFilepath).then(() => {
defer.resolve();
});
},
{ defer: true }
)
.on("cycle", function (event) {
console.log(String(event.target));
})
.on("complete", function () {
console.log(
"Fastest is " + chalk.green(this.filter("fastest").map("name"))
);
console.log("Slowest is " + chalk.red(this.filter("slowest").map("name")));
})
.run({ defer: true });To create a 1 MB file (~40% slower):
dd if=/dev/zero of=big.file count=1024 bs=1024To create a 20 KB file (~55% slower):
dd if=/dev/zero of=big.file count=20 bs=1024How often does it reproduce? Is there a required condition?
Always.
What is the expected behavior?
fs.promises.readFile should perform similarly to fs.readFile
What do you see instead?
Additional information
I suspect the cause is right here: https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L319-L339
Instead of creating a new Buffer for each chunk, it could allocate a single Buffer and write to that buffer. I don't think Buffer.concat or temporary arrays are necessary.
Activity
aduh95 commentedon Mar 3, 2021
Would you be interested in sending a PR to improve the implementation?
TheRamann commentedon Mar 3, 2021
But no one said
fs.promises.readFileshould perform similarly tofs.readFile🤔bluefield-creator commentedon Mar 3, 2021
Well that's right, but no one said it had to be slower either. I think all code should always be as fast as possible.
justinvanwinkle commentedon Mar 3, 2021
My guess is that the promise version is idling when IO capacity is available due to scheduling? Is it implemented in an event driven way internally, and if it is (for example using epoll) is the epoll event able to interrupt and cause the promise to continue to be evaluated or does it have to wait for the vm to get around to scheduling it again?
vigneshtdev commentedon Mar 4, 2021
What about other resources(CPU/memory) consumption?
bluefield-creator commentedon Mar 4, 2021
I dont think it's that, but cant be discarded.
The best option here it's to reproduce.
Linkgoron commentedon Mar 4, 2021
Note that it looks like there are some other differences as well, although I'm not sure how much they affect performance:
fs/promisesversion does an extra read even if it has already readsizeamount of bytes (it doesn't keep track of how much it has read in total), and only finishes reading when it reads0bytes. Thefsversion checks if it has readsizeamount and ends the read (readFileAfterReadinread_file_context.js)Also, the 'utf8' in your
readFileSyncis really slowing it down, on my machine if you remove it the sync version is the fastest by far (and theutils.promisify(fs.readFile)is usually faster thanfs.readFile).medikoo commentedon Mar 5, 2021
This signals the benchmark flaw. as it's same to stating that
decorate(fn)()is faster thanfn()(ofc assuming that in both casesfn()does exactly same job).Promises naturally will always be slower than callbacks (extra objects to garbage collect and artificial ticks are not free)
I can expect it when running 1 to 1, but I'd expect callback version to be more efficient (faster) if we start to process multiple files and take advantage of parallel execution. If that's not the case, I'd see it as other flaw on Node.js side
thelebdev commentedon Mar 5, 2021
This is an amazing finding, @Jarred-Sumner
I have a quick question, what did you use to benchmark it and come up with the tests?
fs: improve fsPromises readFile performance
bluefield-creator commentedon Mar 8, 2021
Quite sure it's npm package "benchmark", sorry if I did not understand your question and you meant something else
fs: improve fsPromises readFile performance
fs: improve fsPromises readFile performance
12 remaining items
fs: improve fsPromises readFile performance
Use `readFileSync`
Mesteery commentedon Feb 17, 2022
I guess it's fixed? Feel free to (or ask to) reopen if it isn't.
sanjarcode commentedon Jan 28, 2023
@Mesteery can you get the docs updated. #39847
benmotyka commentedon Sep 26, 2023
luisnquin commentedon Sep 27, 2023
akbor commentedon Sep 27, 2023
SyedMSawaid commentedon Sep 27, 2023
bro fixed it in bun
benjamingr commentedon Sep 27, 2023
Hey I'm locking this thread for non contributors leaving off topic comments.
I also blocked the last person who left an off topic comment.