Question

I’m writing an optimizer for my application, so db.put() invoked as rarely as possible. I stuck with following problem:

I have a number of classes derived from db.Model. The instances of those classes stored in list:

class DBPutter:
    data = [] # list of instances
    def add(self, model):
        # HERE I WANT TO CHECK THAT self.data IS NOT EXEEDING 1MB
        self.data.append(model)
        if len(self.data) == 1000:
            self.flush()  # actual call to db.put() using deferred

With this approach I receive alot of RequestTooLargeError exceptions. How do I check that my data is not exeeding 1MB?

Answer 1

Pympler has a asizeof method, and should run in python 2.5: http://code.google.com/p/pympler/

I think you’re over-optimizing though. If an instance is shut down before 1000 objects are in your putter you could lose data. Also, I think using the deferred library with a large amount of data would result in at least two db.puts. One when the task is submitted (because the payload is over 10k), and one inside the task, actually writing your models.