fs#opendir does not return all files in subdirectories when bufferSize is less than the directory size

Version

Node.js v20.4.0

Platform

Linux rowlf 6.3.11-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Jul 2 13:17:31 UTC 2023 x86_64 GNU/Linux

Subsystem

fs

What steps will reproduce the bug?

If I call fs#opendir in recursive mode on a directory structure where a subdirectory has more than bufferSize files in it, than the complete results will not be returned (each subdirectory gets truncated to bufferSize entries.

I have attached a code sample that demonstrates the bug. It requires a directory structure with at least 64 files in a subdirectory of a directory named to-read:

.
├── repro-opendir.mjs
└── to-read
    └── subdir
        ├── 1
        ├── 2
        ├── ...
        └── 64

I created the above on my machine with for i in $(seq 1 64); do; touch $i; done

repro-opendir.mjs:

import assert from "assert";
import fs from "fs/promises";

async function opendir(dir, bufferSize) {
  const files = [];
  for await (const file of await fs.opendir(dir, { recursive: true, bufferSize })) {
    files.push(file);
  }
  return files;
}

async function main() {
  const filesA = await opendir("./to-read", 128);
  assert.strictEqual(filesA.length, 65);

  const filesB = await opendir("./to-read", 32);
  assert.strictEqual(filesB.length, 65);
}

await main();

How often does it reproduce? Is there a required condition?

With the directory structure as described it reproduces 100% on node.js v20.3 and v20.4. I have not tried other versions.

What is the expected behavior? Why is that the expected behavior?

Neither assertion should trigger.

What do you see instead?

With the above code sample, the second assert will fail with the following message:

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

33 !== 65

    at main (file:///home/matthew/spikes/repro-node-opendir/repro-opendir.mjs:17:10)
    at async file:///home/matthew/spikes/repro-node-opendir/repro-opendir.mjs:20:1

Additional information

No response

@Ethan-Arrowood

It looks like the problem here is caused by readSyncRecursive not looping on all results returned by the sub-handle. I added a draft pull request #48829 that naively loops on it in place but I'm not sure if there are issues with blocking the event loop for a large subdirectory if it is done that way.

original:

node/lib/internal/fs/dir.js

Lines 172 to 181 in 12a93ce

    
           const result = handle.read( 
        
             this[kDirOptions].encoding, 
        
             this[kDirOptions].bufferSize, 
        
             undefined, 
        
             ctx, 
        
           ); 
        
           if (result) { 
        
             this.processReadResult(dirent.path, result); 
        
           }

revised:
https://github.com/nodejs/node/blob/11cedfe13fb25b65c7f7f14ccb7c7983a7fb53b9/lib/internal/fs/dir.js#L172-L185

Tagging @Ethan-Arrowood as you seem to have been here most recently.

Yeah this was identified recently - i have a PR fixing it differently: #48698

WDYT?

I tried your branch out and it does not fix my issue. I think the problem is because opendir and readdir don't share the same code paths. On the upside, your branch pointed me to where I can change a test to illustrate the issue. I will update my PR branch with a change to test/sequential/test-fs-opendir-recursive.js that demonstrates my bug.

Okay cool. I'll focus on finishing up the change necessary for readdir, and then we can use your work for opendir.

This shows the test change that triggers my issue 9483314

fs: read all results for subdirectories in opendir

Any update on this? The issue is still reproducible in v20.8.0 and my proposed fix above still seems reasonable?

Interesting. I assumed #49603 would fix your issue. Feel free to send a PR with another fix though and we can get it landed asap!

@mdouglass

@mdouglass I saw you have a draft open that would just need an update?

I haven't touched it as I was still awaiting feedback on the questions I raised in the original PR.

It looks like the problem here is caused by readSyncRecursive not looping on all results returned by the sub-handle. I added a draft pull request #48829 that naively loops on it in place but I'm not sure if there are issues with blocking the event loop for a large subdirectory if it is done that way.

Besides a bug, this may be a lack of documentation.

juanarbol@b35181f#diff-5a0fa708afa6a2202dc64e5038e16c8e249cb1c3ec23c87885acc2a662a1ec00L48

It used to read up to 32 folders, now, it can read as bufferSize (can hold).

https://github.com/juanarbol/node/blob/main/lib/internal/fs/dir.js#L52

BufferSize will be 32 by default, and overwritten by the provided args.

@Ethan-Arrowood

This bug specifically happens with {recursive:true}; any subdirectory containing more items than bufferSize will get truncated. The following is a reproducer to demonstrate using bufferSize of 1 (note: it will leave an issue48820 directory in cwd):

import { mkdir, open, opendir, rmdir } from "node:fs/promises";

await mkdir("issue48820");
await mkdir("issue48820/1");
await mkdir("issue48820/1/1");
await mkdir("issue48820/1/1/1");
await mkdir("issue48820/1/1/2");
await mkdir("issue48820/1/2");
await mkdir("issue48820/2");

const dir = await opendir("issue48820", { bufferSize: 1, recursive: true });
for await (const dirent of dir)
  console.log(`${dirent.parentPath}/${dirent.name}`);

// Actual output:
//   issue48820/1
//   issue48820/1/1
//   issue48820/1/1/1
//   issue48820/2

In the root directory both items are found. However, in each subdirectory only one is found.

@Ethan-Arrowood implemented the recursive option; maybe he has more insights. (Edit: oops, he's already in this thread.)

fs: fix bufferSize option for opendir recursive

@Ethan-Arrowood

Thanks @Ethan-Arrowood, this lgtm.

PR merged 🚀

fs: fix bufferSize option for opendir recursive

May I suggest that a warning be added to the documentation ?
It would have been useful to me.

That would be a great addition until #55764 is fixed. Would you like to send a PR?

I'm not very used to doing this but here is a suggestion anyway #55876

fs: fix bufferSize option for opendir recursive

Navigation Menu

Repository navigation

fs#opendir does not return all files in subdirectories when bufferSize is less than the directory size #48820

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	const result = handle.read(
	this[kDirOptions].encoding,
	this[kDirOptions].bufferSize,
	undefined,
	ctx,
	);

	if (result) {
	this.processReadResult(dirent.path, result);
	}

fs#opendir does not return all files in subdirectories when bufferSize is less than the directory size #48820

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Activity

mdouglass commented on Jul 18, 2023

Ethan-Arrowood commented on Jul 18, 2023

mdouglass commented on Jul 18, 2023

Ethan-Arrowood commented on Jul 18, 2023

mdouglass commented on Jul 18, 2023

mdouglass commented on Oct 11, 2023

Ethan-Arrowood commented on Oct 11, 2023

BridgeAR commented on Jan 15, 2024

mdouglass commented on Jan 15, 2024

juanarbol commented on Oct 31, 2024

sjohannes commented on Oct 31, 2024

8 remaining items

mdouglass commented on Nov 7, 2024

Ethan-Arrowood commented on Nov 8, 2024

tjaisson commented on Nov 15, 2024

Ethan-Arrowood commented on Nov 15, 2024

tjaisson commented on Nov 16, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions