Repository navigation
Node 6 fs.realpath behavior changes #7175
Description
- Version: 6
- Platform: Tested on Darwin and Linux
- Subsystem: fs
Previously, looping symbolic links and long path names were handled by fs.realpath without issue. The changelog says that fs.realpath "can throw new errors", but fails to identify that it also is less powerful (and also does not identify what those new errors are).
Seems like it'd be possible to maintain backwards compatibility if Node could fall back to the slower but more resilient implementation when ELOOP or ENAMETOOLONG are raised.
If this is not desirable, then the changelog should be updated to reflect the reduction in functionality.
I have no strong opinion about whether or not Node is capable of resolving the realpath of long looping symbolic links, but I would very much like a thing to point to when people complain about glob being broken. Thanks.
$ node -v
v5.3.0
$ cat rp.js
var fs = require('fs')
var l = 128
var long = 'rptest/' + Array(l).join('a/b/') + 'a'
clean()
process.on('exit', clean)
function clean () {
try { fs.unlinkSync('rptest/a/b') } catch (e) {}
try { fs.rmdirSync('rptest/a') } catch (e) {}
try { fs.rmdirSync('rptest') } catch (e) {}
}
fs.mkdirSync('rptest')
fs.mkdirSync('rptest/a')
fs.symlinkSync('..', 'rptest/a/b')
console.log(fs.realpathSync(long))
fs.realpath(long, console.log)
$ node rp.js
/home/isaacs/rptest/a
null '/home/isaacs/rptest/a'
$ nave use 6
$ node rp.js
fs.js:1568
return binding.realpath(pathModule._makeLong(path), options.encoding);
^
Error: ELOOP: too many symbolic links encountered, realpath 'rptest/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a/b/a'
at Error (native)
at Object.realpathSync (fs.js:1568:18)
at Object.<anonymous> (/home/isaacs/rp.js:19:16)
at Module._compile (module.js:541:32)
at Object.Module._extensions..js (module.js:550:10)
at Module.load (module.js:458:32)
at tryModuleLoad (module.js:417:12)
at Function.Module._load (module.js:409:3)
at Function.Module.runMain (module.js:575:10)
at startup (node.js:160:18)
Activity
ChALkeR commentedon Jun 6, 2016
@isaacs Is this related to npm/npm#5082 fix being blocked?
jhamhader commentedon Jun 6, 2016
Introduced in #3594
By the way, what happens when you try to
fs.stat()on the link?addaleax commentedon Jun 6, 2016
#6861 and #7044 are also issues that stem from the new realpath implementation.
Falling back to a JS-based realpath implementation sounds like a good idea to me.
Fishrock123 commentedon Jun 6, 2016
@thealphanerd citgm tests glob, right? @isaacs Did glob have tests for it previously?
jasnell commentedon Jun 6, 2016
@Fishrock123 ... the primary issue with glob is two fold: (a) fs.realpath now throws errors when it previously did not and (b) when those errors are thrown is platform specific.
Previously, if I use glob with realpath on a circular symlink, it would recurse until a name too long error would occur, which is being caught and intentionally used by glob. This error is essentially the signal that glob should stop recursing and just returns the result up to that point. On certain operating systems (e.g. OSX), the libuv realpath implementation now throws ELOOP errors before the other error is thrown. Since glob does not ignore the ELOOP errors, the app dies. On Linux, the eloop and name too long errors throw at the same depth so the problem is not encountered because the ignored error comes first and the loop inside glob exits before the eloop can occur. On OSX, however, the eloop happens first.
The glob code was written to assume that fs.realpath would never throw an errors of it's own (there's no defensive error handling logic around the call to realpath).
Fishrock123 commentedon Jun 6, 2016
I think glob's assumptions here are ok in terms of errors, fwiw.
Edit: As long as you don't pass it obviously invalid parameters
Fishrock123 commentedon Jun 6, 2016
As a note, the
realpathchange is listed on the breaking change doc: https://github.com/nodejs/node/wiki/Breaking-changes-between-v5-and-v6#fsHowever it does not note this specifically, and it does not mean that there isn't still a problem or bug here.
jasnell commentedon Jun 6, 2016
Note, I'm not saying that glob's assumptions around error handling are right or wrong, just describing the situation. Previously fs.realpath did not throw, and now it does throw based on platform specific differences.
jhamhader commentedon Jun 6, 2016
It is important to note that other
fsfunctions also have this issue. I verified that about the following:stat()access()chmod()chown()open()readdir()watch()fsfunctions which uses the above functions (truncate(),writeFile(), etc)There is almost no
fsfunction which is not affected.isaacs commentedon Jun 10, 2016
@Fishrock123 Yes, glob's tests fail on Node 6. There's a PR from @thealphanerd which makes the test pass, but there are problems with the fix. It can't really be hacked around in node-glob without a very subtle and significant breaking change in node-glob that'll require people to update their code.
Imo, citgm should have prevented this change from happening in the first place. It is a bug in the process that it got out into a node release. That underlying process bug should be fixed, along with this technical bug.
@jhamhader Prior to node 6, one could use
fs.realpathto avoid ELOOP errors in other fs operations, precisely because it would walk up the path from left to right resolving symbolic links.isaacs commentedon Jun 10, 2016
See: isaacs/node-glob#259 (comment)
saghul commentedon Jun 11, 2016
Was this behavior documented? Generally speaking, all fs APIs are thing wrappers around their POSIX counterparts, and very thin wrappers at that. IMHO the general expectation is that they work just the man page says. In this line, I consider the current implementation works as expected.
FTR, the work done here was to make fs.realpath faster, and keeping the JS fallback was met with resistance back then. We waited for Windows XP support to go away in order to land it, because uv_fs_realpath doesn't work there.
The way I see it the only feasible solution here is to catch the error at the application level and handle it.
I understand that a documentation change is acceptable to you then?
ChALkeR commentedon Jun 11, 2016
@isaacs
Does this mean that other
fsfunctions have also been affected, not onlyrealpath()? Afaik onlymoduleusesfs.realpathin Node.js. What prevents from re-implementing the same functionality?Perhaps what we need here need a separate method for just that, either in
fsor in userland?isaacs commentedon Jun 15, 2016
Adequately documenting breaking changes in a platform is the bare minimum requirement. Saying that something "raises new errors", without specifying what those errors are, when they are raised, or that they are platform-specific, is inadequate.
It is not a "fix" though, and I'll be disappointed if that's the resolution.
Time and typing.
It looks like I'm going to have to fall back to the pre-node6 fs.realpath implementation anyway, I'd just prefer that it be in the platform rather than in my userland code, because I care a great deal about the stability of the Node.js platform and the community built on top of it. Stability is the point of a platform; it's what a platform is for, and breaking stability imposes a high cost. Stuff that was built on that platform falls over.
Do not be flippant about that cost. Every time something like this comes up, it imposes many person-hours of work, and reduces confidence in Node.js. Ignoring that cost is user-hostile, especially if the justification is "Well, it was wrong of the user to rely on this thing not breaking anyway". I'm not saying that breaking changes are never justified, but they are expensive. As a platform, Node has a responsibility to make sure the benefit is valuable enough to be worth it.
What is the point of making
fs.realpathfaster if most people don't use it directly, and modules that expose a realpath interface now all have to port the old (slower) implementation forward into their code? It seems like misguided priorities to me. If falling back to a JS implementation to preserve stability was discussed and rejected, then I'd argue that the discussion was with the wrong people or in the wrong context or based on the wrong set of values, because it is the wrong decision.The net impact of stuff like this is that module authors feel the need to scramble and do a bunch of busy-work to catch up to the ways that core has broken their programs. It causes resentment and reduces trust. I'm guessing that this is not the intent of the TSC or CTC, based on what I've been told every time something like this comes up and I complain about it. Citgm is a good step in the right direction. But there needs to be a clearer understanding of trade-offs much earlier in the design process. I don't see that happening. I see my code breaking, and then post-hoc justifications about why it was the right thing to do, based on values that are not clearly articulated and not shared by me or my users.
51 remaining items
saghul commentedon Jul 13, 2016
Quick update: I've tried (without success) to get Windows to work properly. No luck. One clear case where our new realpath implementation fails is with ramdisks like the ones made with ImDisk. For context, we use GetFinalPathNameByHandle with
VOLUME_NAME_DOSthat is, with the drive letter. Turns out that some tools such as ImDisk bypass the Windows Volume Manager soGetFinalPathNameByHandlefails.If there is no way around this, I don't see other solution but to revert to the old code. I'll keep digging.
fs: allow realpath to resolve deep symlinks
colltoaction commentedon Aug 12, 2016
What version will include this patch? Next release on 6.X?
jasnell commentedon Aug 12, 2016
there's a new 6.x version coming out next week but it's not clear yet if this will be included in that. /cc @cjihrig @evanlucas
fs: restore JS implementation of realpath
cjihrig commentedon Aug 15, 2016
It's in the commit list at #8070.
Shahjalal7311 commentedon Jan 18, 2017
npm run build error in my existing project
npm 3.10.8
node -v 6.9.1
i am getting this error
debug log:
how can i fix the problem?
fs: restore JS implementation of realpath