Question

Using: Python 2.4

Currently, I have a nested for loop that iterates over 2 lists and makes a match based on two elements that exists on both lists. Once a match has been found, it the element from the r120Final list and puts in a new list called “r120Delta”:

for r120item in r120Final:
    for spectraItem in spectraFinal:
    if(str(spectraItem[0]) == r120item[2].strip()) and (str(spectraItem[25]) == r120item[10]):
        r120Delta.append(r120item)
        break

The problem is that this is SO SLOW and the lists aren’t that deep. The R120 is about 64,000 lines and the Spectra is about 150,000 lines.

The r120Final list is a nested array and it looks like so:

r120Final[0] = [['xxx','xxx','12345','xxx','xxx','xxx','xxx','xxx','xxx','xxx','234567']]
...
r120Final[n] = [['xxx','xxx','99999','xxx','xxx','xxx','xxx','xxx','xxx','xxx','678901']]

The spectraFinal list is essentially the same, a nested array and it looks like so:

spectraFinal[0] = [['12345','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','234567']]
...
spectraFinal[0] = [['99999','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','xxx','678901']]

Finally, the reason for the “r120Delta” is so that way I can then do a list differential between r120Final and r120Delta and retrieve r120 data elements that were NOT matched. This is the function I defined for this very task, and again, slow:

def listDiff( diffList, completeList ):
    returnList = []
        for completeItem in completeList:
            if not completeItem  in diffList:
                returnList.append(completeItem)
    return returnList

Basically, I’m knowledgeable in Python but by no means an expert. I’m looking for some experts to show me how to speed this up. Any help is appreciated!

Answer 1

spectra_set = set((str(spectraItem[0]), str(spectraItem[25])) for spectraItem in spectraFinal)

returnList = []
for r120item in r120Final:
    if (r120item[2].strip(), r120item[10]) not in spectra_set:
       returnList.append(r120item)

This will add all items that didn’t match to the returnList.

You can do it in one line (if you really want) as

returnList = [r120item for r120item in r120Final 
                 if (r120item[2].strip(), r120item[10]) not in 
                     set((str(spectraItem[0]), str(spectraItem[25])) 
                         for spectraItem in spectraFinal)]

If you need the full spectraItem:

spectra_dict = dict(((str(spectraItem[0]), str(spectraItem[25])), spectraItem) for spectraItem in spectraFinal)
returnList = []
for r120item in r120Final:
    key = (r120item[2].strip(), r120item[10])
    if key not in spectra_dict:
        returnList.append(r120item)
    else:
        return_item = some_function_of(r120item, spectra_dict[key])
        returnList.append(return_item)