Question

What is the best practice for coordinating access to files in node.js?

I’m trying to write an http based file uploader for very large files (10sGB) that is resumable. I’m trying to figure out what the best approach is to handle two people trying to upload the same file at the same time… I’m also trying to think ahead to the possibility where more than one copy of the node.js http server is running behind a load balancer, which means catching duplicate uploads can’t rely on just the code itself.

In python, for example, you can create a file by passing the correct flags to the open() call to force an atomic create. Not sure if the default node.js open new file is atomic.

Another option I thought of, but don’t really want to pursue, is using a database with an async driver that supports atomic transactions to track this state…

Answer 1

I found this on HackerNews under a response to someone complaining about some of the same things in node.js. I’ll put it here for completeness. This allows me to at least lock some file writes in node.js like I wanted to.

IsaacSchlueter 4 hours ago | link

You can open a file with O_EXCL if you pass in the open flags as a
number. (You can find them on require(“constants”), and they need to
be binary-OR’ed together.) This isn’t documented. It should be. It
should probably also be exposed in a cleaner way. Most of the rest of
what you describe is APIs that need to be polished and refined a bit.
The boundaries are well defined at this point, though. We probably
won’t add another builtin module at this point, or dramatically expand
what any of them can do. (I don’t consider seek() dramatic, it’s just
tricky to get right given JavaScript’s annoying Number problems.)