Question

For a project I was working on I need a queue which will be too large to hold in normal memory. I had been implementing it as a simple file where it would read the whole file take the first few (~100) lines, process them, then write back the updated queue with new instructions added and the old ones removed. However, since the queue became too large to hold in memory like this I need something different. Preferably someone can tell me a way to peel off just the first few lines of a file without having to look at the rest of the data. I had thought about using a database (MySQL probably with sorted insert timestamps) but I would heavily prefer to do it without for load and bandwidth reasons (several servers would have to all be sending and receiving a lot of data from the DB). The language I’m working in is PHP but really this question is more about unix files I suppose. Any help would be appreciated.

Answer 1

Sucking out the first line of a file is pretty trivial (fopen() followed by an fgets()). Re-writing the file to remove completed jobs would be very painful, especially if you’ve got multiple concurrent servers working off the same queue file.

One alternative would be to use a seperate file for each job. If you have some concurrency-safe method of generating an incrementing ID for these files, then it’d be a simple matter of picking out the file with the lowest id for the oldest job, and generating a new id for each new job. You’d have to figure out some file locking, though, to keep two+ servers grabbing the same file at the same time, however.