Question

I am trying to read from a csv file and insert the data into an elasticsearch index. As below, I use a readstream and listen in on the “data” event. My problem is, I quickly run out of memory using this approach. I’m guessing it’s because the elasticsearch module (elastical) is making a REST every time, and the number of such requests build up.

I am pretty new, so is there a way for me to fix this so it doesn’t run out of memory? Any general patterns or techniques?

stream.on('data', function (doc) {
    // create a json from doc
    client.index('entities', 'command', json, function (err, res) {
        console.log(res);
    });
}

Answer 1

Pause the stream when you get data and resume it when the request completes.

stream.on('data', function (doc) {
    stream.pause();
    // create a json from doc
    client.index('entities', 'command', json, function (err, res) {
        stream.resume();
        console.log(res);
    });
}

Weird thing about your code is you’re not using doc anywhere in that function. I’m guessing you’re not posting your entire code.