Question

I’m building an internal webapp to process DNA sequence data. I need to be able to accept a zipped file of several sequences that is often larger than 30MB and can be as large as 80MB.

I’ve followed the pattern I’ve used for file uploads in the past:

class ABIUploadHanlder(Handler):
    def POST(self):
        x = web.input(abifile = {})
        filedir = '/usr/local/www/wsgi-scripts'
        if 'seqfile' in x:
            time_key = str(time.time()).replace('.','')

            filepath=x.seqfile.filename.replace('\\','/')
            filename = time_key + 'abizipped.zip'

            filetype = filename.split('.')[-1]
            if filetype == 'zip':
                pass
            else:
                raise web.seeother('/export?err=type')
            fout = open(filedir +'/uploads/'+ filename,'w')
            fout.write(x.seqfile.file.read())
            fout.close()

        raise web.seeother('/abiprocess')

When I post a 30MB file to the web form, I get odd behaviors. Occasionally the upload will get to about 30% then restart. Sometimes the upload will fail after a single attempt. In all attempts the upload never completes and I eventually get a “web page not available” error.

My hunch is that somewhere I need to tell the site to keep the connection alive during upload, even though it’s taking a long time. I have not been able to find how to control this so far.

Any help is appreciated.

Answer 1

I had an error in the code that ran after a certain number of chunks were transmitted. The way web.py and apache are configured on my app (and I don’t think I’m using an odd configuration) result in a weird behavior if an error occurs in the app before the client has finished the file upload. In particular, if an error occurs the browser, instead of displaying the typical “Internal Server Error” page, attempts to upload the file again, cycling until it gives up.

Because of numerous steps and statuses I’m currently logging, the error messages were buried in the log. I finally came across the problem after going through it very carefully.