Question

I’m in the process of writing a python module to POST files to a server , I can upload files of size of upto 500MB but when I tried to upload a 1gb file the upload failed, If I were to use something like cURL it won’t fail. I got the code after googling how to upload multipart formdata using python , the code can be found here. I just compiled and ran that code , the error I’m getting is this

Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    opener.open("http://127.0.0.1/test_server/upload",params)
  File "C:\Python27\lib\urllib2.py", line 392, in open
    req = meth(req)
  File "C:\Python27\MultipartPostHandler.py", line 35, in http_request
    boundary, data = self.multipart_encode(v_vars, v_files)
  File "C:\Python27\MultipartPostHandler.py", line 63, in multipart_encode
    buffer += '\r\n' + fd.read() + '\r\n'  
MemoryError

I’m new to python and having a hard time grasping it. I also came across another program here , I’ll be honest I don’t know how to run it. I tried running it by guessing based on the function name , but that didn’t work.

Answer 1

The script in question isn’t very smart and builds the POST body in memory.

Thus, to POST a 1GB file, you’ll need 1GB of memory just to hold that data, plus the HTTP headers, boundaries, and python and the code itself.

You’d have to rework the script to use mmap instead, where you first construct the whole body in a temp file before handing that file wrapped in a mmap.mmap value to passing it to request.add_data.

See Python: HTTP Post a large file with streaming for hints on how to achieve that.