Question

could anyone please provide on how to achieve below scenario ?

2 queues – destination queue, response queue
thread picks task up from destination queue
finds out needs more details
submits new task to destination queue
waits for his request to be processed and result appear in response queue
or
monitors response queue for response to his task but does not actually pick any response so it is available to the other threads waiting for other responses ?

thank you

Answer 1

If a threads waits for a specific task completion, i.e it shouldn’t pick any completed task except that one it put, you can use locks to wait for the task:

  def run(self):
    # get a task, do somethings, put a new task
    newTask.waitFor()
    …

class Task:
…
def waitFor(self):
self._lock.acquire()

def complete(self):
self._lock.release()

def failedToComplete(self, err):
self._error = err
self._lock.release()

This will help to avoid time.sleep()-s on response queue monitoring. Task completion errors handling should be considered here. But this is uncommon approach. Is it some specific algorithm where the thread which puts a new task, should wait for it? Even so, you can implement that logic into a Task class, and not in the thread that processes it. And why the thread picks a task from the destination queue and puts a new task back to the destination queue? If you have n steps of processing, you can use n queues for it. A group of threads serves the first queue, gets a task, processes it, puts the result (a new task) to the next queue. The group of final response-handler threads gets a response and sends it back to the client. The tasks encapsulate details concerning themselves, the threads don’t distinguish a task from another. And there is not need to wait for a particular task.