Question

New to python here. I’ve been pulling my hair for hours and still can’t figure this out.

I have a list of dictionaries:

[ {'FX0XST001.MID5': '195', 'Name': 'Firmicutes', 'Taxonomy ID': '1239', 'Type': 'phylum'}
  {'FX0XST001.MID13': '4929', 'Name': 'Firmicutes', 'Taxonomy ID': '1239','Type': 'phylum'},
  {'FX0XST001.MID6': '826', 'Name': 'Firmicutes', 'Taxonomy ID': '1239', 'Type': 'phylum'},
                                        .
                                        .
                                        .
                                        .

  {'FX0XST001.MID6': '125', 'Name': 'Acidobacteria', 'Taxonomy ID': '57723', 'Type': 'phylum'}
  {'FX0XST001.MID25': '70', 'Name': 'Acidobacteria', 'Taxonomy ID': '57723', 'Type': 'phylum'}
  {'FX0XST001.MID40': '40', 'Name': 'Acidobacteria', 'Taxonomy ID': '57723', 'Type': 'phylum'} ]

I want to merge the dictionaries in the list based on their Type, Name, and Taxonomy ID

  [ {'FX0XST001.MID5': '195', 'FX0XST001.MID13': '4929', 'FX0XST001.MID6': '826', 'Name': 'Firmicutes', 'Taxonomy ID': '1239', 'Type': 'phylum'}
                                        .
                                        .
                                        .
                                        .

    {'FX0XST001.MID6': '125', 'FX0XST001.MID25': '70', 'FX0XST001.MID40': '40', 'Name': 'Acidobacteria', 'Taxonomy ID': '57723', 'Type': 'phylum'}]

I have the data structure setup like this because I need to write the data to CSV using csv.DictWriter later.

Would anyone kindly point me to the right direction?

Answer 1

You can use the groupby function for this:

http://docs.python.org/library/itertools.html#itertools.groupby

from itertools import groupby

keyfunc = lambda row : (row['Type'], row['Taxonomy ID'], row['Name'])

result = []

data = sorted(data, key=keyfunc)
for k, g in groupby(data, keyfunc):
    # you can either add the matching rows to the item so you end up with what you wanted
    item = {}        
    for row in g:
        item.update(row)
    result.append(item)

    # or you could just add the matched rows as subitems to a parent dictionary
    # which might come in handy if you need to work with just the parts that are
    # different
    item = {'Type': k[0], 'Taxonomy ID' : k[1], 'Name' : k[2], 'matches': [])
    for row in g:
        del row['Type']
        del row['Taxonomy ID']
        del row['Name']
        item['matches'].append(row)
    result.append(item)