Question

I’m working on a simple web crawler in python and I wan’t to make a simple queue class, but I’m not quite sure the best way to start. I want something that holds only unique items to process, so that the crawler will only crawl each page once per script run (simply to avoid infinite looping). Can anyone give me or point me to a simple queue example that I could run off of?

Answer 1

I’d just use a set, it doesn’t maintain order but it will help you maintain uniqueness:

>>> q = set([9, 8, 7, 7, 8, 5, 4, 1]) >>> q.pop() 1 >>> q.pop() 4 >>> q.pop() 5 >>> q.add(3) >>> q.add(3) >>> q.add(3) >>> q.add(3) >>> q set([3, 7, 8, 9]