Question

I work a lot with WiX XML files and just about every object in WiX requires a GUID. To avoid copy-paste errors, I’ve set about on a way to sort and display all duplicate GUIDs given a list like this (created with find and egrep):

./A2.Spam.EggsMgrSvc/__A2.Spam.EggsMgrSvc.wixproj:3A206536-FBCC-4911-AF2B-CBCD76E2C23E
./A2.Spam.TrojanBunnies/Files/Files.wxs:1F372E8A-95B9-49AC-84A6-998E7F5B0689
./A2.Spam.TrojanBunnies/Files/Files.wxs:4BB4FBAD-032A-4FBA-8B81-8AA2876E6765
./A2.Spam.TrojanBunnies/Files/File1.wxs:E289D834-4421-4DCE-B0A8-94C09978058A
./A2.Spam.TrojanBunnies/Files/Files.wxs:083863F1-70DE-11D0-BD40-00A0C911CE86
./A2.Spam.TrojanBunnies/Files/File1.wxs:E289D834-4421-4DCE-B0A8-94C09978058A
./A2.Spam.TrojanBunnies/Files/Files.wxs:083863F1-70DE-11D0-BD40-00A0C911CE86
./A2.Spam.TrojanBunnies/Files/File2.wxs:E289D834-4421-4DCE-B0A8-94C09978058A

in a format like this:

  3 E289D834-4421-4DCE-B0A8-94C09978058A
       2 ./A2.Spam.TrojanBunnies/Files/File1.wxs
       1 ./A2.Spam.TrojanBunnies/Files/File2.wxs
  2 083863F1-70DE-11D0-BD40-00A0C911CE86
       2 ./A2.Spam.TrojanBunnies/Files/Files.wxs

The total number of occurences of the GUID are counted next to the GUID, then the number of occurences of that GUID are counted in each file.

I’ve come up with the following script (which produced the above output). I’m still new to Python and am really trying to understand dictionaries and their practical uses. Was using nested dictionaries the right way to go? I picked dictionaries because I thought it was the easiest way to add/track unique entries. Though, using syntax like parent_dict['child_dict_key']['value_key'] feels a bit odd, like maybe I could make use of items() or other iterable methods/techniques:

#!/usr/bin/env python

guids = {}
f_and_g = open( 'files-and-guids.txt', 'r')

for fg in f_and_g.readlines():
    fname, guid = map( str.strip, fg.split(':') )

    if guid not in guids:
        guids[guid] = { 'count': 1, 'files': {} }
    else:
        guids[guid]['count'] += 1

    ## Count how many times a GUID was used in a given file
    if fname not in guids[guid]['files']:
        guids[guid]['files'][fname]  = 1
    else:
        guids[guid]['files'][fname] += 1

## Sort by total count for a given GUID
for guid in sorted( guids, key=lambda x:guids[x]['count'], reverse=True):
    ## Skip printing if count is below threshold
    if guids[guid]['count'] < 2:
        continue
    guid_dict = guids[guid]
    print '{:>3} {}'.format( guid_dict['count'], guid )
    ## Sort by filename counts
    for fname in sorted( guid_dict['files'],
                         key=lambda x: guid_dict['files'][x], reverse=True ):
        fname_cnt = guid_dict['files'][fname]
        print '{:>8} {}'.format( fname_cnt, fname)

Answer 1

Yet another variation:

#!/usr/bin/env python
import fileinput
from collections import defaultdict, Counter

# count guids
perfile = defaultdict(Counter)
total = Counter()
for line in fileinput.input():
    fname, guid = map(str.strip, line.split(':'))
    perfile[guid][fname] += 1
    total[guid] += 1

# print most common guid first
for guid, count in total.most_common():
    if count < 2: continue # skip printing if count is below threshold
    print '{:>3} {}'.format(count, guid)
    # sorting by filename counts
    for fname, fname_cnt in perfile[guid].most_common():
        print '{:>8} {}'.format(fname_cnt, fname)

Example

$ python2.7 count-guid.py  input 
  3 E289D834-4421-4DCE-B0A8-94C09978058A
       2 ./A2.Spam.TrojanBunnies/Files/File1.wxs
       1 ./A2.Spam.TrojanBunnies/Files/File2.wxs
  2 083863F1-70DE-11D0-BD40-00A0C911CE86
       2 ./A2.Spam.TrojanBunnies/Files/Files.wxs

Don’t overthink it if the script is clear and it works for you.