Question

I am working on an application that involves maintaining consistency between two local directories. Specifically, the directories should be identical, with the exception that all files in one of the directories are modified in some particular way (this part is not important to my question).

While running, my application runs two processes that listen for changes occurring under each of the paths, and performs relevant operations to bring them back in sync when necessary.

In terms of my specific question: I’m looking for advice on the tricker situation of when one starts the application. At this point, each process needs to check all files/folders under both the path that it is looking after, to see if anything has changed in anyway whilst the application was not running. (Let us assume that the application cannot be notified by the OS of anything that happened while it was shutdown, and thus will need to directly check every file/folder.)

Each process will have access to (and maintain) a persistent data-structure of all files/folder under its designated path. I was thinking that the following should be held within the data-structure for each of the files and folders:

• File/folder name;
• File hash (CRC32);
• File/folder last mod data; and
• File/folder size.

These pieces of information will obviously help to check for any changes to files/folder, but what is the best way to store them?

It seems to me that one sensible way to approach the situation of an application start is for each process to recursively scan through all files/folders under its designated path, and compare the metadata for each file scanned to the metadata stored in its data-structure. Then the processes should also iterate through the data-structures to look for things that have been removed from the paths. Some cases that may be encountered during this process are:

• file modified (file name found in data-structure, but hash differs);
• file added (no identical filename or hash found in data-structure);
• file renamed (file with same hash exists in data-structure, but not with same filename);
• folder added (no folder name in data-structure);
• folder removed (folder name in data-structure, but not under path);
• folder renamed (tricky one).

So, what’s the best data-structure to use for this task? In my head I’m thinking some form of sorted associative array, e.g., a red-black tree, which store file and folder objects. Each file object contains name, hash and mod-date attributes , while each folder object contains name and children attributes, where children stores another associative array with everything underneath. Given the path to an arbitrary file, e.g., /foo/bar/file.txt, you begin at the root (foo), check for bar and so on until you get to file.txt‘s parent object.

Another alternative I can think of is to merely store everything flatly, such that there is one red-black tree where each key is the full path to each file/folder, and the value is the file / folder object. This would probably be quicker for retrieval, but it won’t be possible to detect renamed files/folders without iterating through all values anyway, which sounds expensive. In the first approach, it may be the case that identifying a rename would only involves checking a portion of the data-structure rather than all of it.

Sorry the above ideas aren’t terribly well thought out. What’s the state of the art in this area, and are there any well-trodden approaches to these types of problems?

Answer 1

You’re modelling a filesystem, so it’s quite natural to use a hierarchical data structure. After all, you don’t need to compare the file at dir1\dir2\foo.txt to dir3\bar.txt, right? You didn’t mention file moves between directories as something you’re tracking.

So, the data structure could be:

interface IFSEntry {
  string name
  datetime creationDate
  pure virtual bool Compare(IFSEntry other)
  pure virtual void UpdateFrom(IFSEntry other)
  pure virtual bool WasRenamed(Dictionary<string,IFSEntry> possibleOriginals, out string oldName)
  ...
} 

class File : IFSEntry {
  ...
} 

class Directory : IFSEntry {
  private Dictionary<string,IFSEntry> children;
  ...
}

The Directory implementations of UpdateFrom and Compare would recurse down their children.

File renames would be relatively easy by comparing CRC’s. You’d miss files that changed in both places and were renamed. You could add a CRC dictionary to the Directory class if the time to run the comparisons proves a performance problem.

For directory moves, if the child files also changed, then you’ve got a fuzzy logic situation. It would be best to have a merge tool that the user would operate for that situation.

If a file changes in both places, you also need a user-facing merge strategy if conflicting changes occur. I’d argue that is always a good idea, just to let the user eyeball that the document didn’t lose coherence.