Question

I have written a very simple webserver in Scala (based on Actors). The purpose
of it so to log events from our frontend server (such as if a user clicks a
button or a page is loaded). The file will need to be rotated every 64-100mb or so and
it will be send to s3 for later analysis with Hadoop. the amount of traffic will
be about 50-100 calls/s

Some questions that pops into my mind:

1. How do I make sure that all actors can write to one file in a thread safe way?
2. What is the best way to rotate the file after X amount of mb. Should I do this
   in my code or from the filesystem (if I do it from the filesystem, how do I then verify
   that the file isn’t in the middle of a write or that the buffer is flushed)

Answer 1

One simple method would be to have a single file writer actor that serialized all writes to the disk. You could then have multiple request handler actors that fed it updates as they processed logging events from the frontend server. You’d get concurrency in request handling while still serializing writes to your log file. Having more than a single actor would open the possibility of concurrent writes, which would at best corrupt your log file. Basically, if you want something to be thread-safe in the actor model, it should be executed on a single actor. Unfortunately, your task is inherently serial at the point you write to disk. You could do something more involved like merge log files coming from multiple actors at rotation time but that seems like overkill. Unless you’re generating that 64-100MB in a second or two, I’d be surprised if the extra threads doing I/O bought you anything.

Assuming a single writing actor, it’s pretty trivial to calculate the amount that has been written since the last rotation and I don’t think tracking in the actor’s internal state versus polling the filesystem would make a difference one way or the other.