Question

I have a problem with using Twisted for simple concurrency in python. The problem is – I don’t know how to do it and all online resources are about Twisted networking abilities. So I am turning to SO-gurus for some guidance.

Python 2.5 is used.

Simplified version of my problem runs as follows:

1. A bunch of scientific data
2. A function that munches on the data and creates output
3. ??? < here enters concurrency, it takes chunks of data from 1 and feeds it to 2
4. Output from 3 is joined and stored

My guess is that Twisted reactor can do the number three job. But how?

Thanks a lot for any help and suggestions.

upd1:

Simple example code. No idea how reactor deals with processes, so I have given it imaginary functions:

datum = 'abcdefg'

def dataServer(data):
    for char in data:
        yield chara

def dataWorker(chara):
    return ord(chara)

r = reactor()
NUMBER_OF_PROCESSES_AV = 4
serv = dataserver(datum)
id = 0
result = array(len(datum))

while r.working():
    if NUMBER_OF_PROCESSES_AV > 0:
        r.addTask(dataWorker(serv.next(), id)
        NUMBER_OF_PROCESSES_AV -= 1
        id += 1
    for pr, id in r.finishedProcesses():
        result[id] = pr

Answer 1

It seems to me that you are misunderstanding the fundamentals of how Twisted operates. I recommend you give the Twisted Intro a shot by Dave Peticolas. It has been a great help to me, and I’ve been using Twisted for years!

HINT: Everything in Twisted relies on the reactor!

_{(source: krondo.com)}