Question

We were faced with very strange issue that made us crazy. Sometimes newly created files on our File Share PC were “absent” for some period of time. To reproduce a problem you should have at least two computers, call them alpha and beta. Create file share on beta PC (\\beta\share\bug) and run this PowerShell script from alpha PC:

param(
  $sharePath="\\beta\share\bug"
)
$sharePC = ($sharePath -split '\\')[2]
$session = New-PSSession -ComputerName $sharePC
$counter = 0
while ($true) {
  $fileName = $sharePath + "\$counter.txt"
  Invoke-Command -Session $session -ScriptBlock {
    param(
      $fileName
    )
    "" > $fileName
  } -ArgumentList $fileName
  if (Test-Path $fileName) {
    Write-Host "File $fileName exists" -fore Green
  } else {
    Write-Host "!!! File $fileName does NOT exist!" -fore Red
  }

  $counter = $counter + 1
  Start-Sleep 2
}

After starting this script you should be able to see these messages:

File \\beta\share\bug\1.txt exists
File \\beta\share\bug\2.txt exists
...

And now:
Open cmd.exe and run this command:

if exist \\beta\share\bug\foo.txt echo 1

After this during approx 10 seconds you’ll see following messages:

!!! File \\beta\share\bug\3.txt does NOT exist!
!!! File \\beta\share\bug\4.txt does NOT exist!

We’ve discovered that bug is caused by enumerating shared directory where new files are being created. In Python call os.listdir('//beta/share/bug') to reproduce a bug. In C#: Directory.GetDirectories(@"\\beta\share\bug"). You can even simply navigate to share directory by shell and call ls or dir.

Bug were found on Windows Server 2008 R2

Note, that you cannot watch directory content on alpha PC in Windows Explorer in real time, because if you open this directory in Explorer bug would not occur! So ensure to close all such windows before attempts to reproduce a bug. After each script restart you should manually remove all already created files from share (because script is rather stupid and always starts from 0.txt).

We currently have 2 workarounds for this issue:

1. If client sees this situation, it creates some temporary file in problematic directory – after this files magically appear.
2. Disable SMB 2.0: http://www.petri.co.il/how-to-disable-smb-2-on-windows-vista-or-server-2008.htm

Does anybody have ever discovered similar problem and can explain why it occurs and how “correctly fix” it?

Thanks

Answer 1

I was experiencing a similar problem and, eventually, I found the cause of this issue. The specific problem is the SMB2 Directory Cache, which is one of the SMB2 Client Redirector cache components:

This is a cache of recent directory enumerations performed by the client. Subsequent enumeration requests made by client applications as well as metadata queries for files in the directory can be satisfied from the cache. The client also uses the directory cache to determine the presence or absence of a file in the directory and uses that information to prevent clients from repeatedly attempting to open files which are known not to exist on the server. This cache is likely to affect distributed applications running on multiple computers accessing a set of files on a server – where the applications use an out of band mechanism to signal each other about modification/addition/deletion of files on the server.

The default value for this wonderful little cache is 10 seconds, which is producing the behavior you’re seeing. When your code asks the system about that directory/file it’s getting the cached result, which is 10 seconds old, so it says the file does not exist. Setting HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstation\Parameters\DirectoryCacheLifetime (DWORD) to a value of 0 will disable the cache and resolve the file doesn’t exist issue. Surprisingly this change does not require a restart of the client machine! This will also allow you to keep SMB2 enabled, which should be better for a number of reasons vs. forcing SMB1.

Furthermore, the cache is not used when the share in question is opened in Windows explorer, as having it open tells the system to bypass the cache to keep a live view going. Modifying something in the share via code, however, does not.

I think this whole issue is fixed in Windows 2008 R2/7 and higher, but I cannot absolutely confirm it. This is still an issue in modern versions of Windows. See the comments below for details.