Question

First – a little bit about my background: I have been programming for some time (10 years at this point) and am fairly competent when it comes to coding ideas up. I started working on web-application programming just over a year ago, and thankfully discovered nodeJS, which made web-app creation feel a lot more like traditional programming. Now, I have a node.js app that I’ve been developing for some time that is now running in production on the web. My main confusion stems from the fact that I am very new to the world of the web development, and don’t really know what’s important and what isn’t when it comes to monitoring my application.

I am using a Joyent SmartMachine, and looking at the analytics options that they provide is a little overwhelming. There are so many different options and configurations, and I have no clue what purpose each analytic really serves. For the questions below, I’d appreciate any answer, whether it’s specific to Joyent’s Cloud Analytics or completely general.

QUESTION ONE

Right now, my main concern is to figure out how my application is utilizing the server that I have it running on. I want to know if my application has the right amount of resources allocated to it. Does the number of requests that it receives make the server it’s on overkill, or does it warrant extra resources? What analytics are important to look at for a NodeJS app for that purpose? (using both MongoDB and Redis on separate servers if that makes a difference)

QUESTION TWO

What other statistics are generally really important to look at when managing a server that’s in production? I’m used to programs that run once to do something specific (e.g. a raytracer that finishes running once it has computed an image), as opposed to web-apps which are continuously running and interacting with many clients. I’m sure there are many things that are obvious to long-time server administrators that aren’t to newbies like me.

QUESTION THREE

What’s important to look at when dealing with NodeJS specifically? What are statistics/analytics that become particularly critical when dealing with the single-threaded event loop of NodeJS versus more standard server systems?

I have other questions about how databases play into the equation, but I think this is enough for now…

Answer 1

We have been running node.js in production nearly an year starting from 0.4 and currenty 0.8 series. Web app is express 2 and 3 based with mongo, redis and memcached.

Few facts.

• node can not handle large v8 heap, when it grows over 200mb you will start seeing increased cpu usage
• node always seem to leak memory, or at least grow large heap size without actually using it. I suspect memory fragmentation, as v8 profiling or valgrind shows no leaks in js space nor resident heap. Early 0.8 was awful in this respect, rss could be 1GB with 50MB heap.
• hanging requests are hard to track. We wrote our middleware to monitor these especially as our app is long poll based

My suggestions.

• use multiple instances per machine, at least 1 per cpu. Balance with haproxy, nginx or such with session affinity
• write midleware to report hanged connections, ie ones that code never responded or latency was over threshold
• restart instances often, at least weekly
• write poller that prints out memory stats with process module one per minute
• Use supervisord and fabric for easy process management

Monitor cpu, reported memory stats and restart on threshold