Question

I don’t know enough about VB.Net (2008, Express Edition) yet, so I wanted to ask if there were a better way to find files with different names but the same contents, ie. duplicates.

In the following code, I use GetFiles() to retrieve all the files in a given directory, and for each file, use MD5 to hash its contents, check if this value already lives in a dictionary: If yes, it’s a duplicate and I’ll delete it; If not, I add this filename/hashvalue into the dictionary for later:

'Get all files from directory
Dim currfile As String
For Each currfile In Directory.GetFiles("C:\MyFiles\", "File.*")
    'Check if hashing already found as value, ie. duplicate
    If StoreItem.ContainsValue(ReadFileMD5(currfile)) Then
        'Delete duplicate
    'This hashing not yet found in dictionary -> add it
    Else
        StoreItem.Add(currfile, ReadFileMD5(currfile))
    End If
Next

Is this a good way to solve the issue of finding duplicates, or is there a better way I should know about?

Thank you.

Answer 1

You can optimise this by the following

• Iterate all the files and record the filename and length
• Then compare (MD5) each file only with those that are the same length
• This is one of those tasks that is called embarrassingly parallel, so you should be able to use mulitple threads to do this and more efficiently, and since each comparison is independent
• You only need to compare one file to another once not both ways round, ie. if you do compare(f1, f2) then you don’t need to do compare(f2, f1)

I’ve sure there are many others.