Question

Before you start shouting “Do not use for-loops with async functions!” let me state that I do know how to program a loop with async functions calling the next iteration from within the callback.

But what, if I am not interested in the callback or in processing one iteration only if the predecessor has finished. In my case I am importing a big number of records from an xml-file and writing them into a mongoDB. The normal call to my “write”-function would be like this:

asyncWrite( record, function(err, result) {
    ...
});

Now as I said, im not interested in the callback (no errors will occur!) nor in a serial process. So I used a for-loop and left out the callback:

for( i=0; i<max; i+=1) {
    asyncWrite( record_arr[i] );
}

For small max this works perfectly. But with max getting bigger (> 100.000) I end up getting an ‘out of memory’ error. The only way to prevent this was turning back to a ‘normal’ async-loop (recursive function call from within the callback).

Any other idea?

Answer 1

The async functions will not be executed until the for-loop has done its work. It has just collected a queue that is too beg. For big max you got out of memory error because the queue is full. You can see how your memory leaks by inserting console.log(process.memoryUsage()); inside of the loop. You need to work with chunks to process this task.