Question

I am curious what would be an efficient way of uniquifying such data objects:

testdata =[ ['9034968', 'ETH'], ['14160113', 'ETH'], ['9034968', 'ETH'], ['11111', 'NOT'], ['9555269', 'NOT'], ['15724032', 'ETH'], ['15481740', 'ETH'], ['15481757', 'ETH'], ['15481724', 'ETH'], ['10307528', 'ETH'], ['15481757', 'ETH'], ['15481724', 'ETH'], ['15481740', 'ETH'], ['15379365', 'ETH'], ['11111', 'NOT'], ['9555269', 'NOT'], ['15379365', 'ETH']
]

For each data pair, left numeric string PLUS the type at the right tells the uniqueness of a data element. The return value should be a list of lists as same as the testdata, but with only the unique values kept.

Answer 1

You can use a set:

unique_data = [list(x) for x in set(tuple(x) for x in testdata)]

You can also see this page which benchmarks a variety of methods that either preserve or don’t preserve order.