Question

I have one list (list a) that increases based on size and has items removed from this list and put into another list (list b) that maintains what has already been checked.

The list that is maintaining (list b) what has been done needs to be compared with the list that has content (list a) that hasn’t been checked already; what is the most efficient way to do this check in python? Even when the contents of each list can go well over 10k items.

while listA:
    for a in listA:
        #do something

        listB.append(url)
        listA.remove(url)

This is the furthest I can get.

Answer 1

Your code will produce inconsistent output. You shouldn’t iterate and modify a list in the same loop context.

You could use listA as a Queue:

 while len(listA):
     element = listA.pop()
     #do something with element
     listB.add(element)

this code is consistent because even tough the list is being modified within the while loop there is no iterator created for listA. Only the len of the list is checked to make sure that the program terminates.