Question

I have a Windows service written in C# which monitors a folder for files to process. When files are added to this folder, the service should pick up the file and perform a task with it, consuming the file in the process.

I would like to distribute the work over multiple physical servers for fault-tolerance. The files will be placed on NAS accessible to all service instances.

The significant requirement is that I would like each service to pick up a file exclusively; a file should not be processed by more than one service.

Are there any good strategies for working with files in this way?

Answer 1

The simplest solution, it seems to me, would be to create a .lock file. So if ServiceA sees a file called myfile.dat, it would look for a myfile.dat.lock file. If it doesn’t find it, it would create one; subsequence services would see the myfile.data.lock file and skip over that file.

There’s still the potential that two services would attempt to create the .lock file at the exact same time, but one of those services would receive an exception for attempting to create a duplicate file. So you could handle that exception and retry the .lock file check (or just skip it) and continue from there.