Question

if a single image has been saved twice with two different filenames, is there a way to compare them to see if they’re the same..?

I’m hoping a basic hash or CRC type check could work..?

File size might not, as there are millions of images in the pool and different images could have the same size..

Hoping there’s an easy way to do it in C#..

Answer 1

If the file contents are identical, then a cryptographic hash would at least give a very good indication of equality. The SHA256 class would be a good candidate here, although it’s possibly a little over the top. For example:

static byte[] Sha256HashFile(string file)
{
    using (SHA256 sha256 = SHA256.Create())
    {
        using (Stream input = File.OpenRead(file))
        {
            return sha256.ComputeHash(input);
        }
    }
}

The simplest way to compare the two returned byte arrays is probably to convert them both to strings using Convert.ToBase64 and then compare the strings. Ugly but easy 🙂 You could also use Enumerable.SequenceEqual:

byte[] hash1 = Sha256HashFile("file1.png");
byte[] hash2 = Sha256HashFile("file2.png");
bool same = hash1.SequenceEqual(hash2);    

If you want to store the hashes as a set or dictionary, you could implement your own IEqualityComparer<byte[]> but frankly it would be easiest to use a base64 string. For example, this will print out the duplicate files:

var hashToNameMap = new Dictionary<string, string>();
foreach (string file in files)
{
    byte[] hash = Sha256HashFile(file);
    string base64 = Convert.ToBase64(hash);
    string existingName;
    if (hashToNameMap.TryGetValue(base64, out existingName))
    {
        Console.WriteLine("{0} is a duplicate of {1}", file, existingName);
    }
    else
    {
        hashToNameMap[base64] = file;
    }
}

A few notes:

• This isn’t guaranteed to be accurate, but the chances of getting a collision are very small, particularly if the files also have to be valid images.
• This involves reading all of every file – even if there are no other files with the same size (and therefore no possible duplicates). This may or may not be an issue for you.
• Even if there are multiple files of the same size, you only need to read all of them to find duplicates… you could potentially read the files and compute the hashes as you go, stopping as soon as you find that files are different.

How you approach this depends on whether your goal is absolute speed, simplicity of code, etc. It may also depend on whether the pool will grow over time – for example, you may want to hash files as soon as you get two or more files of the same size, so that when you add another file of the same size you can hash that and add it without ever rereading the existing data.