Question

I have an application where I am sequentially downloading mp3 files from a server, storing them in my server temporarily, then streaming them directly to clients, like so:

function downloadNextTrack(){
  var request = http.get('http://mp3server.com', function(response){
    response.on('data', function(data) {
      fs.appendFile('sometrack.mp3', data, function (err) {});
    });
    response.on('end', function(){
      streamTrack('sometrack.mp3');
    }
  });
};

var clients = []; // client response objects are pushed to this array when they request the stream through a route like /stream.mp3

var stream;

function streamTrack(track){
  stream = fs.createReadStream(track);
  stream.on('data', function(data){
    clients.forEach(function(client) {
      client.write(data);
    });
  });
  stream.on('end', function(){
    downloadNextTrack(); // redoes the same thing with another track
  }
};

Apparently this code is creating a lot of buffers which aren’t being freed by the OS, when I run ‘free -M’ command, this is what I get (after about 4 hours of running the app):

                   total      used       free     shared    buffers     cached
              Mem: 750        675         75          0         12        180
-/+ buffers/cache:            481        269
             Swap: 255        112        143

The number under ‘buffers’ constantly rises (as well as the cached memory) and the OS apparently doesn’t reclaim that 180mb back, until eventually my app runs out of memory and crashes when I try spawning a small process to verify a track’s bitrate, sampling rate, id3 info, etc.

I have diagnosed with a lot of different tools (such as memwatch and nodetime) to find out if it was an internal memory leak and it isn’t, the V8 memory heap as well as the Node RSS vary +/- 10mb but stay constant for most part while the OS free memory gets lower and lower (when the Node process starts I have about 350MB of free memory).

I read somewhere that Buffer instances allocated by Node have direct access to raw memory and so V8 doesn’t have power over them (which checks out with the fact that I am not getting memory leaks from the V8 heap), the thing is, I need a way to get rid of these old buffers. Is this possible? Or will I have to restart my app every 5 hours or so (or worse, buy more RAM!)?

PS. I am running Node v0.8.16 on Ubuntu 10.04.

Answer 1

I agree with Tiago,
I think this is caused because of the recursive nature of your code.
I don’t think the streams is what gobbling up your heap, because as you said, the stream variable is being reassigned with a new ReadStream with every iteration.
However, the http.get’s request and response (and whatever Buffers they use) in line 2 are never being released before calling the next iteration; they are scoped within the downloadNextTrack function. You end up with a recursive stack trace that has a set of request and response objects (and some underlying buffers) per file.

In general, if this code needs to run many, many times, why not opt-out of the recursion and do it all iteratively? a never-ending recursion will always gobble more and more memory, until the program crashes, even if there’s no memory leaks on your part.