Question

I recently went to my App Engine site which has been running just fine and no errors and got the big ugly error page. In the admin tool App Engine was logging this error:

2012-09-06 10:53:43.938
Traceback (most recent call last):
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 189, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 227, in _LoadHandler
    handler = __import__(path[0])
  File "/base/data/home/apps/s~myapp/1.361555922666090832/main.py", line 3, in <module>
    from controllers.routes import api_routes, web_routes, admin_routes
  File "/base/data/home/apps/s~myapp/1.361555922666090832/controllers/routes/api_routes.py", line 3, in <module>
    from ..api import api_obj_controller, api_app_controller, api_path_controller, api_user_controller
  File "/base/data/home/apps/s~myapp/1.361555922666090832/controllers/api/api_obj_controller.py", line 2, in <module>
    from ..handlers.api_handler import ApiRequestHandler
  File "/base/data/home/apps/s~myapp/1.361555922666090832/controllers/handlers/api_handler.py", line 2, in <module>
    from ..handlers.content_handler import BaseRequestHandler
  File "/base/data/home/apps/s~myapp/1.361555922666090832/controllers/handlers/content_handler.py", line 3, in <module>
    from webapp2_extras import jinja2
  File "/base/data/home/apps/s~myapp/1.361555922666090832/webapp2_extras/jinja2.py", line 15, in <module>
    import jinja2
  File "/base/python27_runtime/python27_lib/versions/third_party/jinja2-2.6/jinja2/__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "/base/python27_runtime/python27_lib/versions/third_party/jinja2-2.6/jinja2/environment.py", line 13, in <module>
    from jinja2 import nodes
ImportError: cannot import name nodes
W 2012-09-06 10:53:43.967
After handling this request, the process that handled this request was found to have handled too many sequential errors, and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you are likely returning errors continously from your application.

So it continued to get many errors, but then just started working again without any new code published. My concerns obviously is how can I prevent this in the future? Why did this happen? And how is it possible it corrected itself without me deploying any code fixes? This error makes me nervous that it will randomly happen for my customers.

Edit:

Also, the very first error I received was a DeadlineExceededError error, which includes the message “This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.”

So from this message and what Tim commented on makes complete sense, but how do I fix it so it doesn’t happen again? What can I do so the DeadlineExceededError doesn’t basically error out the entire site until a new instance is spun up? (that is if I’m understanding it correctly)

Answer 1

If I understand correctly, you saw DeadlineExceededErrors in some loading requests, but the instances which handled that request were not killed.

It is probably because you have some naked except blocks like:

try:
    do something
except:
    do other things

I think, what happened to your instances is that one of such except clauses caught the DeadlineExceededErrors and the code continued to run instead of throwing the DeadlineExceededError. If that DeadlineExceededError is thrown during importing jinja2, jinja2 module can remain half-loaded state. Therefore the instance keeps living in a corrupt state.

If DeadlineExceededError is thrown correctly, the instance should be killed, so that there won’t be a corrupted instance like you have.

It’s just my assumption, let me know if that’s likely the case.