Question

I wrote a admin script that tails a heroku log and every n seconds, it summarizes averages and notifies me if i cross a certain threshold (yes I know and love new relic — but I want to do custom stuff).

Here is the entire script.

I have never been a master of IO and threads, I wonder if I am making a silly mistake. I have a couple of daemon threads that have while(true){} which could be the culprit. For example:

# read new lines
f = File.open(file, "r")
f.seek(0, IO::SEEK_END)
while true do
  select([f])
  line = f.gets
  parse_heroku_line(line)
end

I use one daemon to watch for new lines of a log, and the other to periodically summarize.

Does someone see a way to make it less processor-intensive?

Answer 1

This probably runs hot because you never really block while reading from the temporary file. IO::select is a thin layer over POSIX select(2). It looks like you’re trying to block until the file is ready for reading, but select(2) considers EOF to be ready (“a file descriptor is also ready on end-of-file”), so you always return right away from select then call gets which returns nil at EOF.

You can get a truer EOF reading and nice blocking behavior by avoiding the thread which writes to the temp file and instead using IO::popen to fork the %x[heroku logs --ps router --tail --app pipewave-cedar] log tailer, connected to a ruby IO object on which you can loop over gets, exiting when gets returns nil (indicating the log tailer finished). gets on the pipe from the tailer will block when there’s nothing to read and your script will only run as hot as it takes to do your line parsing and reporting.

EDIT: I’m not set up to actually try your code, but you should be able to replace the log tailer thread and your temp file read loop with this code to get the behavior described above:

IO.popen( %w{ heroku logs --ps router --tail --app my-heroku-app } ) do |logf|
  while line = logf.gets
    parse_heroku_line(line) if line =~ /^/
  end
end

I also notice your reporting thread does not do anything to synchronize access to @total_lines, @total_errors, etc. So, you have some minor race conditions where you can get inconsistent values from the instance vars that parse_heroku_line method updates.