Skip to content

Improved workarounds for win32 - #119

Closed
mekwall wants to merge 18 commits into
isaacs:masterfrom
mekwall:patch-1
Closed

mekwall wants to merge 18 commits into
isaacs:masterfrom
mekwall:patch-1

Conversation

@mekwall

@mekwall mekwall commented Oct 10, 2017

Copy link
Copy Markdown

Summary

This PR improves on the existing override for fs.rename and also adds a similar but blocking override for fs.renameSync (obsoleting #23). The overrides tries to normalize the behavior of fs.rename and fs.renameSync between platforms. It does this by ensuring that the file was actually moved before resolving within a 60 second time frame.

Background

fs.rename and fs.renameSync uses MoveFileEx function on Windows. It is not an atomic operation and honors the Windows sharing modes, meaning that whenever a file or parent directory is locked (in use) the rename might fail with EACCS or EPERM errors depending on the sharing mode set on the file and/or directory.

This differs from how the api call used by Linux and OSX works where the rename operation is atomic and will go through no matter if the file is locked or not.

Disabling anti-virus is NOT a viable solution

Most active anti-virus file scanners will lock the file and/or directory during scan, so if you are trying to rename it during this time you'll get an EPERM/EACCS error thrown in your face. Just to be clear: Disabling AV is NOT a viable solution and should never be accepted as such.

Issues possibly affected by this

References

@mekwall mekwall changed the title Stat from file to make sure it was renamed Stat destination file to make sure it was moved Oct 10, 2017
@mekwall mekwall changed the title Stat destination file to make sure it was moved Stat source file to make sure it was moved Oct 10, 2017
@mekwall mekwall changed the title Stat source file to make sure it was moved Improved workarounds for win32 Oct 11, 2017
Comment thread polyfills.js Outdated
fs.stat(from, function (fromStater, fromSt) {
if (!fromSt && toSt) {
if (cb) cb(toStater ? er : null)
} else if (fromSt && toSt && fromSt.size === toSt.size) {

@mekwall mekwall Oct 11, 2017

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really fond about this, so could use some feedback/input on how it could be solved differently.

Comment thread polyfills.js Outdated
throw e
}
// Wait until destination exists and source no longer exists or that we've reached the backoff limit
while ((fs.existsSync(from) || !fs.existsSync(to)) && Date.now() < backoffUntil) {}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly the worst hack ever, but it seems to be the only real way to solve inconsistent behavior on Windows where the destination and source can exist at the same time.

@conceptualspace

conceptualspace commented Nov 7, 2017

Copy link
Copy Markdown

testing this out on a windows machine lead to crash:

"error": "TypeError: Cannot read property 'size' of undefined", "errorStackTrace": "TypeError: Cannot read property 'size' of undefined\n at node_modules\graceful-fs\polyfills.js:112:45\n at node_modules\graceful-fs\polyfills.js:331:29\n at FSReqWrap.oncomplete (fs.js:114:15)"

taking a quick look -- perhaps there needs to be error handling on the first call to fs.stat?

Node v7.9.0
Windows Server 2008

@mekwall

mekwall commented Nov 7, 2017

Copy link
Copy Markdown
Author

@conceptualspace I've added some additional sanity checks that should solve your crash. Thanks for testing and reporting the issue!

@RyanCavanaugh

Copy link
Copy Markdown

Thanks @mekwall for writing this patch! I work on TypeScript, Visual Studio, and Visual Studio Code. The underlying problem described here is coming up a lot for VS developers because the TypeScript language service puts a file watcher on the project folder (including when the user is just writing JavaScript). We typically see errors like this:

npm ERR!   stack: 'Error: EPERM: operation not permitted, rename \'C:\\foo\\node_modules\\pug-error\\package.json.731450482\' -> \'C:\\foo\\node_modules\\pug-error\\package.json\''

As the PR describes, this is definitely related to file watchers - in my testing, if we disable the file watching from the TSLS side, the error stops happening.

I've done some testing and this PR definitely fixes the issue. The particular scenario we're trying to make work has gone from failing 1 in 4 times to succeeding the last 35 times in a row.

We really need this to be merged - it's impacting a lot of developers and there's nothing that can be done on our end. Developers running npm install while VS or VS Code are open with TS or JS files in the editor end up seeing this error all the time and it's going to result in a lot of support noise on the NPM and VS side.

@rjgotten

Copy link
Copy Markdown

@mekwall

Thank you, thank you, thank you, thank you, thank yo-----

It may take longer to run a big npm install pass, but unlike any other solution including all the stuff the NPM team has been throwing up against the wall in a bid to see what sticks, this actually WORKS.

@nullstd

nullstd commented Apr 10, 2018

Copy link
Copy Markdown

NPM team really needs to merge this PR ASAP, could any guys in their team give a response?

@isaacs

isaacs commented May 15, 2018

Copy link
Copy Markdown
Owner

Hi. It seems like there's a semantic change here. If the target directory is not empty, then it should fail with ENOTEMPTY. But, the fs.rmdir call (which should error in that case) is ignoring the error.

Also, I'm not sure it's a good idea to spin on CPU for potentially 60 seconds in this case. Is it possible to make this time shorter, or allow it to be configured?

@mekwall

mekwall commented May 29, 2018

Copy link
Copy Markdown
Author

Sadly I don't have time to fix this at the moment. Any takers?

@jgoz

jgoz commented Jun 27, 2018

Copy link
Copy Markdown

@mekwall @isaacs Addressed feedback in #131.

@mekwall

mekwall commented Jul 18, 2018

Copy link
Copy Markdown
Author

Closing this in favor of #131.

@manjaneqx manjaneqx left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good

@manjaneqx manjaneqx left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Y

@manjaneqx manjaneqx left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Y

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.