Question

I’m playing around with node.js, trying to re-write a particularly poorly designed part of my production system at work. So far, so good, I use rabbitmq for messaging, and my node.js part of the system runs ghostscript command line tool to convert tiff files to pdf. Obviously I need to make sure I’m not running more than some fixed amount of conversions at a time. What would be the best way to do this with node? I understand that maybe node.js isn’t really about running heavy disk IO stuff, but I’m having too much fun with it to quit.
I was considering just using a blocking call to execute command line utilities but the thing is that some messages don’t require this conversion and there’s no need to delay their processing.

Answer 1

[Update] node-batches seems more appropriate.

I think you need something like forEachLimit (the following snippet was extracted from the async library)

forEachLimit = function (arr, limit, iterator, callback) {
    callback = callback || function () {};
    if (!arr.length || limit <= 0) {
        return callback(); 
    }
    var completed = 0;
    var started = 0;
    var running = 0;

    (function replenish () {
      if (completed === arr.length) {
          return callback();
      }

      while (running < limit && started < arr.length) {
        iterator(arr[started], function (err) {
          if (err) {
              callback(err);
              callback = function () {};
          }
          else {
              completed += 1;
              running -= 1;
              if (completed === arr.length) {
                  callback();
              }
              else {
                  replenish();
              }
          }
        });
        started += 1;
        running += 1;
      }
    })();
};

Usage:

var fileToConvert = ['file1', 'file2', 'file3']
    maxConcurrency = 4;

function fnIter(item, callback){
  console.log('converting', item);
  // Convertion happen here
  require('child_process').exec("some -f "+item, function(error, stdout, stderr){
    callback(stderr); // stderr should be "null" if everything went good.
  });
}

function fnDone(){
  console.log('done !');
}

forEachLimit(fileToConvert, maxConcurrency, fnIter, fnDone);