Question

I am designing a little soft which involves:

1. Fetching a resource on the internet,

2. Some user interaction (quick editing of the resource),

3. Some processing.

I would like to do so with many resources (they are all listed in a list). Each is independent from the others. Since the editing part is quite weary, I would like to make life easier for the user (probably me) so that he does not have to wait for the download of each ressource. For simplicity we forget the third task here.

My idea was to use the threading or multiprocessing module. Some thread (say thread 1) would do the “download” in advance while another (say thread 2) one would interact with the user on an already downloaded resource.

Question: How can I make sure that thread 1 is always ahead of at least ahead_min resources and at most ahead_max (ahead_max>ahead_min) at all times?

I typically would need something similar to Queue.Queue(ahead_max) (or multiprocessing.Queue(ahead_max)) except that when ahead_max is attained then insertion is blocked until there are at most ahead_min elements left in the queue (in fact it blocks until the queue is empty, see http://docs.python.org/library/queue.html#module-Queue). Popping should also be blocked until at least ahead_min+1 elements are in the queue (at the end of the sequence of resources I can then insert some dummy objects to ensure even the last resource is processed).

Any idea? If you think of any simpler alternative, please share!

Answer 1

In this case I would suggest to subclass Queue and implement your own logic. This should be an easy task as the implementation of the Queue class is already in Python.

You can use this as template

from queue import Queue

class MyQueue(Queue):

    def put(self, item, block=True, timeout=None):
        ...

    def get(self, block=True, timeout=None):
        ...