Question

Background

This is an optimization problem. Oracle Forms XML files have elements such as:

<Trigger TriggerName="name" TriggerText="SELECT * FROM DUAL" ... />

Where the TriggerText is arbitrary SQL code. Each SQL statement has been extracted into uniquely named files such as:

sql/module=DIAL_ACCESS+trigger=KEY-LISTVAL+filename=d_access.fmb.sql     
sql/module=REP_PAT_SEEN+trigger=KEY-LISTVAL+filename=rep_pat_seen.fmb.sql 

I wrote a script to generate a list of exact duplicates using a brute force approach.

Problem

There are 37,497 files to compare against each other; it takes 8 minutes to compare one file against all the others. Logically, if A = B and A = C, then there is no need to check if B = C. So the problem is: how do you eliminate the redundant comparisons?

The script will complete in approximately 208 days.

Script Source Code

The comparison script is as follows:

#!/bin/bash

echo Loading directory ...

for i in $(find sql/ -type f -name \*.sql); do
        echo Comparing $i ...

        for j in $(find sql/ -type f -name \*.sql); do
                if [ "$i" = "$j" ]; then
                        continue;
                fi

                # Case insensitive compare, ignore spaces
                diff -IEbwBaq $i $j > /dev/null

                # 0 = no difference (i.e., duplicate code)
                if [ $? = 0 ]; then
                        echo $i :: $j >> clones.txt
                fi
        done
done

Question

How would you optimize the script so that checking for cloned code is a few orders of magnitude faster?

Idea #1

Remove the matching files into another directory so that they don’t need to be examined twice.

System Constraints

Using a quad-core CPU with an SSD; trying to avoid using cloud services if possible. The system is a Windows-based machine with Cygwin installed — algorithms or solutions in other languages are welcome.

Thank you!

Answer 1

Your solution, and sputnick’s solution, both take O(n^2) time. This can be done in O(nlog n) time by sorting the files and using a list merge. It can be sped up further by comparing MD5 (or any other cryptographically-strong hash function) of the files, instead of the files themselves.

Assuming you’re in the sql directory:

md5sum * | sort > ../md5sums
perl -lane 'print if $F[0] eq $lastMd5; $last = $_; $lastMd5 = $F[0]' < ../md5sums

Using the above code will report only exact byte-for-byte duplicates. If you want to consider two non-identical files to be equivalent for the purposes of this comparison (e.g. if you don’t care about case), first create a canonicalised copy of each file (e.g. by converting every character to lower case with tr A-Z a-z < infile > outfile).