Question

I need to create a service which is basically responsible for the following:

1. Watch a specific folder for any new files created.
2. If yes , read that file , process it and save data in DB.

For the above task, I am thinking of creating a multi threaded service with either of the following approach:

1. In the main thread, create an instance of filesystem watcher and as soon as a new file is created, add that file in the threadQueue. There will be N no. of consumer threads running which should take a file from the queue and process it (i.e step 2).

2. Again in the main thread, create an instance of filesystem watcher and as soon as a new file is created, read that file and add the data to MSMQ using wcf MSMQ service. When the message is read by the wcf msmq service, it will be responsible for processing further

I am a newbie when it comes to creating a multi threaded service. So not sure which will tbe the best option. Please guide me.

Thanks,

Answer 1

First off, let me say that you have taken a wise approach to do a single producer – multiple consumer model. This is the best approach in this case.

I would go for option 1, using a ConcurrentQueue data structure, which provides you an easy way to queue tasks in a thread-safe manner. Alternatively, you can simply use the ThreadPool.QueueUserWorkItem method to send work directly to the built-in thread pool, without worrying about managing the workers or the queue explicitly.

Edit: Regarding the reliability of FileSystemWatcher, MSDN says:

The Windows operating system notifies your component of file changes
in a buffer created by the FileSystemWatcher. If there are many
changes in a short time, the buffer can overflow. This causes the
component to lose track of changes in the directory, and it will only
provide blanket notification. Increasing the size of the buffer with
the InternalBufferSize property is expensive, as it comes from
non-paged memory that cannot be swapped out to disk, so keep the
buffer as small yet large enough to not miss any file change events.
To avoid a buffer overflow, use the NotifyFilter and
IncludeSubdirectories properties so you can filter out unwanted change
notifications.

So it depends on how often changes will occur and how much buffer you are allocating.