Question

I am interested in developing self-documenting pipelines.

Can I wrap Ruffus tasks in Pweave chunks?

Pweave and Ruffus
==============================================================

**Let's see if Pweave and ruffus can play nice**


<<load_imports>>=
import time
from ruffus import *
@

**Do this**
<<task1>>=
task1_param = [
                    [ None, 'job1.stage1'], # 1st job
                    [ None, 'job2.stage1'], # 2nd job
              ]
@files(task1_param)
def first_task(no_input_file, output_file):
    open(output_file, "w")
@

I get the feeling the Ruffus decorators are throwing Pweave off:

$ Pweave ruffus.Pnw
Processing chunk 1 named load_imports
Processing chunk 2 named task1
<type 'exceptions.TypeError'>
("unsupported operand type(s) for +: 'NoneType' and 'str'",)

Perhaps there is a workaround?

Answer 1

I am the author of Ruffus and have just checked in changes to ruffus to allow it to cooperate with pweave into the google source code repository. I will be in the next release.

You can get the latest (fixed) source with the following command line if you are impatient:

hg clone https://[e-mail]/p/ruffus/ 

Leo

The details are as follows:

Ruffus uses the full qualified name (with module name) of each ruffus task function to uniquely identify code so that pipeline tasks can be referred to by name.

The Pweave code was very straightforward. Nice! Pweave sends chunks of code at a time to the python interpretor to be exec-ed chunk by chunk. Of course chunks do not belong to any “module” and task functions have function.__module__ values of None rather than any string.

A single judicious str() converting None to "None" seems to have solved the problem.

Leo