Question

Follow up to: How to safely update a file that has many readers and one writer?

In my previous questions, I figured out that you can use FileChannel’s lock to ensure an ordering on reads and writes.

But how do you handle the case if the writer fails mid-write (say the JVM crashes)? This basic algorithm would look like,

WRITER:
  lock file
  write file
  release file

READER:
  lock file
  read file
  release file

If the JVM crashes during write file, sure the lock would be released, but now I have an incomplete file. I want something complete to always be readable. Either the old content the new content and nothing in between.

My first strategy was to write to a temporary file, and then copy the contents into the “live” file (while ensure good locking). The algorithm for this is,

WRITER:
  lock temp file
  write temp file
  lock file
  copy temp to file
  release file
  release temp
  delete temp

READER:
  lock file
  read file
  release file

One nice thing is the delete temp won’t delete the temp if it has already been locked by another writer.

But that algorithm doesn’t handle if the JVM crashes during copy temp to file. So then I added a copying flag,

WRITER:
  lock temp file
  write temp file
  lock file
  create copying flag
  copy temp to file
  delete copying flag
  release file
  release temp
  delete temp

READER:
  lock file
  if copying flag exists
    copy temp to file
    delete copying flag
    delete temp 
  end
  read file
  release file

There won’t ever be two things accessing the copying file as it is guarded by the file lock.

Now, is this the way to do it? It seems very complicated to ensure something very simple. Is there some Java library that handles this for me?

EDIT

Well, I managed I make a mistake in my third attempt. The reader doesn’t hold the lock to temp when it does copy temp to file. Also its not a simple fix to simply lock the temp file! That would cause the writer and reader to acquire locks in different orders and can lead to deadlock. This is getting more complicated all the time. Here’s my fourth attempt,

WRITER:
  lock file
  write temp file
  create copying flag
  copy temp to file
  delete copying flag
  delete temp
  release file

READER:
  lock file
  if copying flag exists
    copy temp to file
    delete copying flag
    delete temp 
  end
  read file
  release file

This time the temp file is guarded by main lock, so it doesn’t even need its own lock.

EDIT 2

When I say JVM crash, I actually mean say the power went out and you didn’t have a UPS.

EDIT 3

I still managed to make another mistake. You shouldn’t lock on the file you are writing to or reading from. This will cause problems, since you can’t get both the read and write lock unless you use RandomAccessFile in Java, which does not implement Input/Output stream.

What you want to do instead is just lock on a lock file that guards the file you are read or writing. Here’s the updated algorithm:

WRITER:
  lock
  write temp file
  create copying flag
  copy temp to file
  delete copying flag
  delete temp
  release

READER:
  lock
  if copying flag exists
    copy temp to file
    delete copying flag
    delete temp 
  end
  read file
  release

lock and release guards the file, the temp file and the copying flag. The only problem is now the reader lock can’t be shared, but it never could be really. The reader always had a chance to modify the file, therefore it would have been wrong to make a shareable lock in the first place.

Answer 1

despite the fact that there is no bullet-proof, cross-OS, cross-FS solution, the “write to unique temp file and rename” strategy is still your best option. most platforms/filesystems attempt to make file renaming (effectively) atomic. note, you want to use a +separate+ lock file for locking.

so, assuming you want to update “myfile.txt”:

• lock “myfile.txt.lck” (this is a separate, empty file)
• write changes to a unique temp file, e.g. “myfile.txt.13424.tmp” (use File.createTempFile())
• for extra protection, but possibly slower, fsync the temp file before proceeding (FileChannel.force(true)).
• rename “myfile.txt.13424.tmp” to “myfile.txt”
• unlock “myfile.txt.lck”

on certain platforms (windows), you need to do a little more dancing due restrictions on file ops (you can move “myfile.txt” to “myfile.txt.old” before renaming, and use the “.old” file to recover from if you need to when reading).