Question

I have two lists of strings that are passed into a function. They are more or less the same, except that one has been run through a regex filter to remove certain boilerplate substrings (e.g. removing ‘LLC’ from ‘Blues Brothers LLC’).

This function is meant to internally deduplicate the modified list and remove the associated item in the non-modified list. You can assume that these lists were sorted alphabetically before being run through the regex filter, and remain in the same order (i.e. original[x] and modified[x] refer to the same entity, even if original[x] != modified[x]). Relative order must be maintained between the two lists in the output.

This is what I have so far. It works 99% of the time, except for very rare combinations of inputs and boilerplate strings (1 in 1000s) where some output strings will be mismatched by a single list position. Input lists are ‘original’ and ‘modified’.

# record positions of duplicates so we're not trying to modify the same lists we're iterating
dellist_modified = []
dellist_original = []

# probably not necessary, extra precaution against modifying lists being iterated.
# fwiw the problem still exists if I remove these and change their references in the last two lines directly to the input lists
modified_copy = modified
original_copy = original

for i in range(0, len(modified)-1):
    if modified[i] == modified[i+1]:
dellist_modified.append(modified[i+1])
dellist_original.append(original[i+1])

for j in dellist_modified:
    if j in modified:
        del modified_copy[agg_match.index(j)]
        del original_copy[agg_match.index(j)]

# return modified_copy and original_copy

It’s ugly, but it’s all I got. My testing indicates the problem is created by the last chunk of code.

Modifications or entirely new approaches would be greatly appreciated. My next step is to try using dictionaries.

Answer 1

Here is a clean way of doing this:

original = list(range(10))
modified = list(original)
modified[5] = "a"
modified[6] = "a"

def without_repeated(original, modified):
    seen = set()
    for (o, m) in zip(original, modified):
        if m not in seen:
            seen.add(m)
            yield o, m

original, modified = zip(*without_repeated(original, modified))

print(original)
print(modified)

Giving us:

(0, 1, 2, 3, 4, 5, 7, 8, 9)
(0, 1, 2, 3, 4, 'a', 7, 8, 9)

We iterate through both lists at the same time. We keep a set of items we have seen (sets have very fast checks for ownership) and then yields any results that we haven’t already seen.

We can then use zip again to give us two lists back.

Note we could actually do this like so:

seen = set()
original, modified = zip(*((o, m) for (o, m) in zip(original, modified) if m not in seen and not seen.add(m)))

This works the same way, except using a single generator expression, with adding the item to the set hacked in using the conditional statement (as add always returns false, we can do this). However, this method is considerably harder to read and so I’d advise against it, just an example for the sake of it.