Question

My django application takes forever to load so I’d like to find a way to measure the import latencies so I can find the offending module and make it load lazily.

Is there an obvious way to do this? I was considering tweaking the import statement itself to produce the latencies but I’m not sure exactly how to do that. I guess that it would be ideal to have the import tree (or DAG?) along with the latencies but just a list of latencies and import statements would be enough.

Answer 1

You can redefine the __import__ built-in function to log the start and end times (delegating everything else to the original built-in __import__). That’s exactly the way to “tweak the import statement” in Python: redefine __import__!

Edit: here’s a simple example…:

import sys

import __builtin__
_orgimp = __builtin__.__import__

import logging
FORMAT = "%(asctime)-15s %(message)s"
logging.basicConfig(format=FORMAT, level=logging.INFO)

def __import__(name, *a):
  r = sys.modules.get(name)
  if r is not None: return r
  logging.info('import bgn %s', name)
  r = _orgimp(name, *a)
  logging.info('import end %s', name)
  return r
__builtin__.__import__ = __import__

import pyparsing

This shows, on my system:

2010-08-24 08:36:39,649 import bgn pyparsing
2010-08-24 08:36:39,652 import bgn weakref
2010-08-24 08:36:39,652 import bgn _weakref
2010-08-24 08:36:39,653 import end _weakref
2010-08-24 08:36:39,653 import end weakref
2010-08-24 08:36:39,654 import bgn copy
2010-08-24 08:36:39,655 import bgn org.python.core
2010-08-24 08:36:39,656 import end copy
2010-08-24 08:36:39,675 import end pyparsing

Of course you can parse this log to show the nesting (what module first-imported what other modules) and “attempted imports” that failed (here, that of org.python.core from copy, no doubt in a try/except statement) as well as the time at which each import begins and concludes (the latter only if it does conclude, of course, not if it fails — easy to tweak with try/finally or whatever to get the exact behavior you prefer!-).