Question

I am quite new to GAE and web development in general, and I am trying to work with GAE BlobStore. I am using python can not seem to find any resources on uploading a large amount of files into BlobStore.

I have read a number of resources and examples, but non of them seem to be helpful in this particular matter.

Does anyone have any resources or experience regarding this ?

Answer 1

I had a similar requirement in my project, when moving to appengine from dedicated server.
We didn’t have to move 1 million, but it was about 100000.

It took some time, but at the end was done.

It was done using pretty simple solution. Just created a file upload service, just like in the examples on blogstore docs.

class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
  def get(self):
    upload_url = blobstore.create_upload_url('/upload')
    self.response.write(upload_url)

  def post(self):
    upload_files = self.get_uploads('file')
    blob_info = upload_files[0]
    self.response.write(blob_info.key())

And then using upload script to upload the files from localhost.

Such a script can be written in python, or even in shell script, by combination of find, curl, and xargs.

For speedup using several threads and doing 32 uploads in paralel helps.

For robustness, after uploading a file, we can try to download it and compare the sizes/md5, and re-uploading if something went wrong.