Question

I’m trying to create and store 20000 random codes in my local datastore, before trying this in appspot… This is the model

class PromotionCode (db.Model):
  code = db.StringProperty(required=True)

And this is the class that handles the populate request (only a logged admin may use it). It creates random alphanumeric codes and tries to store 20000 of them in the datastore:

class Populate(webapp.RequestHandler):

  def GenerateCode(self):
    chars = string.letters + string.digits
    code = ""
    for i in range(8):
      code = code + choice(chars)
    return code.upper()

  def get(self):
    codes = "";
    code_list = []
    for i in range(20000):
      new_code = self.GenerateCode()
      promotion_code = PromotionCode(code=new_code)
      code_list.append(promotion_code)
      codes = codes + "<br>" + new_code
    db.put(code_list)
    self.response.out.write("populating datastore...<br>")
    self.response.out.write(codes)

I thought I could try batching all those put(), so I created a list of codes (code_list). It takes 2-5 minutes to do it locally.

Is it possible to do it faster without using the bulkuploader option? Because I’m getting the 500 server error, obviously. Or maybe doing it in consecutive calls or steps…

Answer 1

Why not just change your code above to insert 100 at a time, and just run something like:

for i in {1..200}
  do
    curl --cookie "ACSID=your-acsid-cookie" http://your-app-id.appspot.com/populatepath
    sleep(5)
  done

from your command line? The entries are random anyway, you don’t need to remember any state.

You can get the ACSID cookie by logging in manually and inspecting the cookies from your browser.

The sleep between requests will prevent you from spinning up a gigantic number of instances or hitting short-term quotas.

The task queue suggestion is good if this is something you need to automate, but if it’s a one-time thing you might as well keep it simple.