Question

Hi i’ve searched the App Engine Groups but couldn’t find a definitive solution.

I want to put 1000 entities into my App Engine Server (Production Server), but I get timeout errors and also 500 Server errors. It worked perfectly in local development.

To reduce the Put load, I made the code sleep for 5 seconds after every 10 puts. I still got the same errors 🙁

Please point me in the right direction. Really appreciate your insights.

Code:

class User(db.Model)
   nickname = db.StringProperty()
   feed = db.StringListProperty()

class Initialize(webapp.RequestHandler):
  def get(self):
    self.response.headers['Content-Type'] = 'text/plain'
    start = 0
     end = 10

  #add 1000 users ( 10 Puts x 100 rows )
  for row in range(1,100):
    for i in range(start,end):
      nickname = "nickname" + str(i)
      feed = ["string1","string2","string3","string4","string5"]

      User(key_name=str(i),nickname=nickname,feed=feed).put()

      self.response.out.write('Entry %s done\n' % i)

    #add counters by 10
    start = start + 10
    end = end + 10

    #sleep
    time.sleep(5)

  self.response.out.write('Initialized 1000 users for Datastore.')

Answer 1

I would suggest that you need to break up the writes into batches as write operations use a lot of time, and you will otherwise exceed AppEngine’s maximum CPU usase per request. The correct way to break into into batches is to have multiple, separate requests that each add in a small number of records.

So code the Initialize handler such that you can call it multiple times, each call accomplishing a small piece of the total work.