Question

When scanning a directory tree with node.js, as we utilise asynchronous callbacks with a large directory it is easy to instantly queue up more than a 1000 fs operations at once which often causes the node.js process to crash or simply for the fs operations to fail with a “EM too many open files” error.

Has anyone figured out an elegant way of solving this?

Answer 1

Update for the years:

1. I coded a https://npmjs.com/package/safefs package based on this answer
2. Isaac coded a https://npmjs.com/package/graceful-fs package
3. Safefs now uses the graceful-fs package
4. The same problem exists for processes, so I created the https://npmjs.com/package/safeps package which solves this issue among many others

—

Not sure if this is the best way to solve this issue, but what I’ve effectively done is wrapped all the asynchronous fs functions with a wrapper that checks whether or not the we have more than the allowed amount of files open (arbitrarily set to a reasonable 100). If there is more than that many files open at the time of the call, the call will idle and check again after a delay to see if there is now an opening. The code for this logic can be found here: https://github.com/balupton/bal-util/blob/master/src/lib/paths.coffee#L7-45

In terms of implementing it, you just need to change require('fs').readFile or whatever call you are using to require('bal-util').readFile and install the bal-util dependency (npm install bal-util) which is simple enough.

Hope this helps. If there is a more robust way of doing this, I’d love to know.