Question

Cheers all. I’m running Ubuntu 10.04 and the latest Google AppEngine SDK. I’m working on a simple website which has posts and comments to posts. I basically implemented a simple tree to store my comments with a parent_comment, left and right values.

I created an event which fires before a new (not is_saved()) comment is put() into the Google Datastore, which calculates the left and right values for the new comment, as well as updates older comments for a valid hierarchy. I basically followed Managing Hierarchical Data in MySQL and implemented it in Python.

Everything seems to work fine, new comments are added, threading looks good, but…
A cycle that submits 40 comments during startup works, but when I increase that cycle to 80 or more, I’m left with an IOError:

IOError: [Errno 24] Too many open files: '/tmp/tmp0agXqU'

My code for generating 60 comments looks like this:

for k in range(0, 4):
    comments = {0: None}
    for i in range(1, 21):
        j = random.randrange(0, len(comments))
        pc = comments[j]

        comments[i] = Comment(
            name=lipsum(count=1),
            email=lipsum(count=1, make_slug=True) + '@email.com',
            url='http://' + lipsum(count=2, make_slug=True) + '.com',
            content=lipsum(count=random.randrange(10, 50)),
            object_link=p.key(),
            parent_comment=pc
        )
        comments[i].put()

The lipsum function simply returns a piece of lorem ipsum text.

Any ideas on how to solve this? Thanks!

Answer 1

I had the same problem for a massive taskqueue worker.

Switching to SQLite for your local Datastore could solve the problem:

dev_appserver.py --use_sqlite 

Since App Engine SDK 1.3.3, the Python
SDK has a new experimental feature
that gives the option to use SQLite
as the datastore stub backend. Using
SQLite within the dev_appserver should
speed up performance of your local
datastore when testing on large
datasets.

For the dev_appserver documentation, have a look here