Question

I am writing a tool that monitors a network directory and is running off of a Windows Server 2008 machine, the OnChanged event for the FileSystemWatcher is being fired correctly from files placed on the network drive by any computer that is not using Windows 7, for some reason if the amount of files copied is more than 19 on a windows 7 computer (at once) then no events are fired although it works if files are done individually. Is there a workaround for this or is that just how the Windows 7 kernel behaves with FSW events?

Just to clarify it works for thousands of files when copied from an XP machine. (The software is still on the 2008 server machine).

Answer 1

From MSDN:

The Windows operating system notifies your component of file changes in a buffer created by the FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with the InternalBufferSize property is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.

If increasing the buffer size is not sufficient and you cannot control how many files are triggering events at a time you would have to add additional polling.

See also this related question:

FileSystemWatcher does not work properly when many files are added to the directory at the same time…

Update:

It might be tempting to simply increase the buffer size but this should be done with care. In fact, there is a 64k limitation when it comes to network access. The FileSystemWatcher class is using the Windows API function ReadDirectoryChangesW underneath which has this limit:

ReadDirectoryChangesW fails with ERROR_INVALID_PARAMETER when the buffer length is greater than 64 KB and the application is monitoring a directory over the network. This is due to a packet size limitation with the underlying file sharing protocols.

If you want to get a deeper understanding on the cost of modifying the buffer size you should have a look at the post of Walter Wang of Microsoft here:

FileSystemWatcher across the network (full post quoted below)

I’m sorry that the documentation of
FileSystemWatcher.InternalBufferSize
didn’t state very clear about the
buffer size when monitoring network
path. It’s recommended not exceeds 64K
when monitoring network path.

FileSystemWatcher is basically a .Net
wrapper for the Win32
ReadDirectoryChangesW API. To use
ReadDirectoryChangesW, you create and
specify a buffer that the OS will
populate with the changes. However,
what is not mentioned in the
ReadDirectoryChangesW documentation
(but is hinted in the
FileSystemWatcher docs) is that the
file system creates an internal kernel
buffer to store the change information
temporarily until it has the chance to
update the user buffer. The size of
the kernel buffer that is created is
the same size that is specified in
ReadDirectoryChangesW and is created
in non-paged pooled memory. Every time
a FileSystemWatcher /
ReadDirectoryChangesW is created /
called, a new kernel buffer is also
created.

The kernel memory pools (paged and
non-paged) are set aside in the system
address space for device drivers and
other kernel components to use. They
grow and shrink dynamically as
necessary. The current size of the
pools can be easily seen by going to
the Performance tab of the Task
Manager. The pools will grow
dynamically until they hit a maximum
value which is calculated at boot time
and depends on available system
resources (mostly RAM). You do not
want to hit this maximum value or else
various system services and drivers
will start failing. However, this
calculated maximum value is not easily
available. To determine the maximum
pool sizes, you need to use a kernel
debugger. If you are interested in
further information about the system
memory pools, I recommend that you
look at Chapter 7 in the MSPress book
Inside Windows 2000 by Solomon and
Russinovich.

With this in mind, there is no
recommendation on what size buffers
you can use. The current and maximum
size of the system pools are going to
be varied from client to client.
However, you probably should not go
over 64k for each FileSystemWatcher /
ReadDirectoryChangesW buffer. This
stems from the fact that there is a
64k limitation with network access as
documented in ReadDirectoryChangesW.
But in the end you are going to have
to test the application on a variety
of expected target systems so that you
can tune your buffer.

There is overhead associated with .Net
applications and I imagine that a
Win32 ReadDirectoryChangesW program
might be able to achieve better
performance with the same buffer size.
However, with very fast and numerous
file changes, buffer overruns will be
inevitable and the developer is going
to have to handle the case when an
overrun occurs such as manually
enumerating the directory to detect
the changes.

In conclusion, FileSystemWatcher and
ReadDirectoryChangesW are a
lightweight file change detection
mechanism that is going to have its
limitations. Change Journals is
another mechanism which we would
consider a medium-weight solution, but
would still have limitations:

http://msdn.microsoft.com/en-us/library/aa363798%28VS.85%29.aspx

Heavy-weight solutions would be to
write a dedicated file system filter
driver that sits in the file system
stack and monitors file system
changes. Of course this would be the
most complex approach. Most virus
scanners, backup software, and file
system monitoring utilities such as
filemon (www.sysinternals.com)
implement a filter driver.

I hope above explanation helps you to
understand the root cause of the issue
you’re experiencing. Please reply to
let us know whether or not you need
further information. Thank you.