Question

I’ve got a fairly simple NodeJS app that is nothing but a shell that accepts requests and parses content using a form of Readability (based on https://github.com/arrix/node-readability). The problem is that I’ve noticed the app leaks memory – I’ve run node --trace-gc and the memory slowly builds up with each request with the occasional gc pass doing nothing.

$ node --trace-gc server.js 
Scavenge 2.3 -> 2.3 MB, 1 ms.
Mark-sweep 3.5 -> 2.9 MB, 2 ms.
Scavenge 4.4 -> 3.8 MB, 0 ms.
16 Feb 10:57:51 - Server started on PORT 8000
Scavenge 5.9 -> 5.0 MB, 2 ms.
Mark-sweep 5.0 -> 4.2 MB, 3 ms.
Mark-compact 4.2 -> 4.1 MB, 7 ms.
Scavenge 5.2 -> 4.7 MB, 1 ms.
Scavenge 5.5 -> 5.1 MB, 1 ms.
Scavenge 5.9 -> 5.5 MB, 1 ms.
Scavenge 7.1 -> 6.3 MB, 2 ms.
Scavenge 7.8 -> 7.0 MB, 2 ms.
Mark-sweep 8.5 -> 7.6 MB, 10 ms.
Scavenge 11.7 -> 9.7 MB, 4 ms.
Scavenge 12.7 -> 11.2 MB, 5 ms.
Mark-sweep 14.2 -> 12.4 MB, 21 ms.
Scavenge 20.5 -> 16.5 MB, 10 ms.
Scavenge 22.5 -> 19.5 MB, 11 ms.
Mark-sweep 25.5 -> 22.4 MB, 38 ms.
Scavenge 36.6 -> 29.6 MB, 25 ms.
Scavenge 41.6 -> 35.6 MB, 24 ms.
Mark-sweep 46.8 -> 41.1 MB, 75 ms.
Scavenge 46.8 -> 44.0 MB, 33 ms.
Mark-sweep 57.2 -> 50.6 MB, 92 ms.
Scavenge 62.3 -> 56.5 MB, 26 ms.
Scavenge 68.5 -> 62.6 MB, 24 ms.
Scavenge 74.6 -> 68.6 MB, 26 ms.
Mark-sweep 80.6 -> 74.5 MB, 130 ms.
Scavenge 80.5 -> 77.6 MB, 25 ms.
Mark-sweep 77.6 -> 77.4 MB, 112 ms.
Mark-compact 77.4 -> 77.4 MB, 260 ms.

Even after the requests stop and the GC has time to run over and over the memory usage does not decrease.

I’m more or less convinced that I must be assigning a global variable somewhere but I can’t figure out how to fix it. The code for this project lives here:

https://github.com/erskingardner/Readable

Does anyone have any suggestions (and reasoning) of what I should do?

Answer 1

I’ll answer my own question here in hopes that it might help out other people with the same issue.

The problem was actually to do with the event.emitter’s I’d used in creating the server. When just a single connection was being used there was no memory leak (think development) but when the app started to get hit a lot memory started to climb and climb and the GC wasn’t doing anything. I had placed an event listener one step too high in the chain and because of that I was creating an event listener that was never being destroyed.

==

BROKEN CODE

Readable.prototype.createHTTPServer = function() {
  var self = this;

  var server = http.createServer(function(request, response) {  
    request.addListener('end', function() {
      var  location  = url.parse(request.url, true)
          ,params    = (location.query || request.headers)
          ,body      = "";

      if (location.pathname == '/' && request.method == "GET"){
        if (params["url"] == null){
          response.writeHead(200, {
            'Content-Type': 'text/html'
          });
          response.end("Good to go, you might want to try adding a url param though.");
        }
        else if (params["url"] != null){
          self.fetchAndParse(params["url"], params);
        }
        var listener = emitter.on("readability", function(result) {
          response.writeHead(200, {
            'Content-Type': 'text/html'
          });
          if (result == "error"){
            response.end("error");
          } else {
            response.end(result.content);          
          }
        });
      }
    });
  });
  return server
};

The listener variable was the part that was never being destroyed. I tried to force the delete of the variable but then the response was never making it to the browser. The answer was that all I had to do was move that listener var inside the else if statement and it was automatically GC’ed when the function was finished running.

==

PROPER CODE

Readable.prototype.createHTTPServer = function() {
  var self = this;

  var server = http.createServer(function(request, response) {  
    request.addListener('end', function() {
      var  location  = url.parse(request.url, true)
          ,params    = (location.query || request.headers)
          ,body      = "";

      if (location.pathname == '/' && request.method == "GET"){
        if (params["url"] == null){
          response.writeHead(200, {
            'Content-Type': 'text/html'
          });
          response.end("Good to go, you might want to try adding a url param though.");
        }
        else if (params["url"] != null){
          self.fetchAndParse(params["url"], params);
          var listener = emitter.on("readability", function(result) {
            response.writeHead(200, {
              'Content-Type': 'text/html'
            });
            if (result == "error"){
              response.end("error");
            } else {
              response.end(result.content);          
            }
          });
        }
      }
    });
  });
  return server
};