Question

I’ve recently jumped into Web2py framework which I find very good. However, I’m now stuck with a ‘basic’ problem.

Context

The website I’m building is an interface for scientific code: people fills-in a form and submit it. The data (written in a file inside a shared folder) is then being processed by the code, running as a daemon in the background (the code has no link with the website, excepts this shared folder and a JSONRPC link between code->website).
The code generates files and images that I would like to make available to the website users (ie. returning them back the results they asked for).
When a job is finished, the code ‘push’ the results to the website using a JSONRPC service link, which works perfectly. The results are in a folder with a generated name which have basically the following structure:

unique_folder_name/
        file1.txt
        file2.csv
        file3.xls
        image1.png
        image2.png

The current implementation (not working properly)

Currently I have 3 DBs:

# Job database
db.define_table("job",
    Field('owner', 'string', length=60, required=True, writable=True, readable=True),
    Field('uniq_id', "string", length=60, required=True, unique=True, writable=False, readable=True))

# Result database
db.define_table("result",
    Field("job", "reference job"),
    Field("x", "integer", default=1),
    Field("y", "integer", default=0),
    Field("z", "integer", default=0),
    Field("data", "string", length=500),
    Field("message", "string", length=200))

# File database
db.define_table("file",
    Field("job", "reference job"),
    Field("file", "upload"),
    Field("isimage", "boolean", default=False))

When the code ‘push’ the results, some modules in Web2py create the entry in the ‘result’ db, and entries in the ‘files’ db associated to the job.
Since I did not find a way to make files (already on the filesystem) available to Web2py without it copying to the upload folder, I currently store the files like that (in a module):

stream = open(os.path.join(directory, _file), 'rb')
current.db.file.insert(job=jid, file=current.db.file.file.store(stream, _file), isimage=isimg)

Then, when I want to create a view with the images, I do (in a module):

rows = current.db((current.db.file.job==jid) & (current.db.file.isimage==True)).select()
for row in rows:
    div.append(I(_src=URL(c="default", f="download", args=os.path.join(directory, row.file)), _width="500", _height="500"))

and in the view: `{{=div}}“

Problems

This is just not working… The source code of the displayed page is like:

<i height="500" src="/mycode/default/download//path/to/directory/file.file.9b7d3a0367de0843.6d732d72732e706e67.png" width="500"></i>

If I type this URL in the address bar, the file download properly, but otherwise the image is not displayed on the webpage.
Also, I have to give the path of the file, even if Web2py did copy the file into ‘upload’ folder (with its new safe ugly name:)), otherwise the link is just not working.
So: no image displayed, plus the files are copied into ‘upload’ folder anyway 🙁

I’m lost, and dont see how to fix this. (I’ve tried also to add the request object when building the image URL, and also tried a custom download function… none worked so far).

EDIT (SOLUTION)

Well, there is an obvious bug in my code that I’ve missed: images tag helper is not I, but IMG 🙂 Simple mistake arising from the confusion with the I tag used in twitter bootstrap for icons… So that solves the display issue.
For streaming files without copying them in the upload folder (ie. not using an upload field in the DB), rochacbruno (many thanks to him) has put me on the right track. See my own answer for the complete solution.

Answer 1

So here is the complete solution to my problem.

In the db.py file, I’ve replaced the ‘file’ table by this definition (so files stay where they are, and are not copied into the web2py upload folder):

# File database
db.define_table("file",
    Field("job", "reference job"),
    Field("name", "string", length=30, required=True),   # stores the filename (without path)
    Field("isimage", "boolean", default=False))

Then, in a controller (eg. ‘mycontroller’), I’ve defined this function to stream files:

from mymodule import OUTPUT  # this is the base directory of the files
from gluon.contenttype import contenttype

def export():
    # allow to download files
    jid = request.args(0)
    fid = request.args(1)

    if None in (jid, fid):
        res = 'Invalid URL'
    else:
        # get the file row
        row = db.file(fid)

        # some checks (not necessary if you know what you're doing)
        jrow = db.job(jid)
        if row is None:
            res = "unknown file ID"
        elif jrow.id  is None:
            res = "unknown job ID"
        else:
            filename = row.name
            # jrow.perma_id, is a field in the 'job' DB, that I use to create a unique
            # directory name, so the files of job ID 'jid' are under: OUTPUT/perma_id/
            fullname = os.path.join(OUTPUT, jrow.perma_id, filename)

            ext = os.path.splitext(filename)[1]
            response.headers['Content-Type'] = contenttype(ext)
            response.headers['Content-disposition'] = 'attachment; filename=%s' % filename
            res = response.stream(open(fullname, "rb"), chunk_size=4096)

    return res

Please note, that I ran into another problem at this point: I first thought to pass the full path as a request argument (eg. URL(c='mycontroller', f='export', args=(path, filename))),
but it did not work since path was containing ‘/’ that were split into as many arguments…
If you don’t have an easy path like me (ie. just one component change), you can store the path into the ‘file’ DB for instance.

Then, for the view (using either a module or whatever you like):

rows = current.db((current.db.file.job==jid) & (current.db.file.isimage==True)).select()
for row in rows:
    div.append(IMG(_src=URL(c="mycontroller", f="export", args=(jid, fid), _width="500", _height="500"))

Notice that the I helper tag has been replaced by the correct IMG one. ‘jid’ is the job ID, and ‘fid’ is the file ID you want to display/download.