Question

I am trying to use Valum’s Ajax Upload to do file uploads on a Django-
based site I am making. Currently I am avoiding a form simply because
AU sends the upload as the entirety of the POST data in an ajax
request. Right now I have a very naive approach to doing this:

upload = SimpleUploadedFile( filename, request.raw_post_data )
...then I loop through the chunks to write to disk...

This works great… on small files. I’ve tested with PDFs, various
other files, and up to the ~20MB Google Chrome deb package and they
all do great. However, if I move up to something like a CD or DVD iso
it bombs horribly. Often Django sends back an Out of Memory
response. On the surface this makes sense since SimpleUploadedFile is
an in-memory version of the upload classes. I cannot see how to use
TemporaryUploadedFile because it does not take actual content in its
constructor. As a side note: I would think after using up the
available RAM it would go to virtual memory, but whatever.

So, my question is, how do I get this to work? Is there a better way
to read in the file? I tried directly reading the raw_post_data via
Python’s IO (system uses 2.6.5) but FileIO’s ascii encoder/decoder
will obviously complain about non-ascii characters when working with
binary files. I have not been able to find info on changing the
encoder/decoder.

I wouldn’t mind passing the data into a form and having Django do the
work of picking the right upload class and so on, but I cannot figure
out how to pass it because something like

upload_form = UploadForm( request.POST, request.FILES )

will not work because the POST contains the file and not the normal
Django info and FILES does not exist.

As I said, I’m not worried about the method of the solution, just that
I get something that works! Thanks!

Answer 1

Well I found two solutions if anyone is interested.

The first is a pure-Python way of doing it that is moderately successful.

with BufferedReader( BytesIO( request.raw_post_data ) ) as stream:
  with BufferedWriter( FileIO( "/tmp/foo.bar", "wb" ) ) as destination:
    foo = stream.read( 1024 )
    while foo:
      destination.write( foo )
      foo = stream.read( 1024 )

It worked on testing for small files (up to 20MB) but failed when I tried it with ISOs (~600MB) or larger files. I didn’t try anything between the 20MB and 600MB so not sure where the break point is. I’ve copied the bottom of the trace below, I’m not sure what the root problem is in this situation. There seemed to be a struggle with memory, but I had enough RAM+swap to hold the file three times over so not sure why there was an issue. Not sure if using other forms of Python read/write or not using buffers would help here.

[error] [client 127.0.0.1]   File "/usr/local/lib/python2.6 /dist-packages/django/core/handlers/wsgi.py", line 69, in safe_copyfileobj, referer: http://localhost/project/
[error] [client 127.0.0.1]     buf = fsrc.read(min(length, size)), referer: http://localhost/project/
[error] [client 127.0.0.1] TemplateSyntaxError: Caught IOError while rendering: request data read error, referer: http://localhost/project/

The solution that has worked with everything I’ve thrown at it, up to 2GB files at least, required Django 1.3. They have added file-like support for reading directly from HttpRequest so I took advantage of that.

with BufferedWriter( FileIO( "/tmp/foo.bar", "wb" ) ) as destination:
  foo = request.read( 1024 )
  while foo:
    destination.write( foo )
    foo = request.read( 1024 )