Question

I’m having a module import issue.

using python 2.6 on ubuntu 10.10

I have a class that subclasses the daemon at http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/ . I created a python package with a module containing code that imports some models from a django project. The code works when used from a class, not subclassing the daemon. The structure looks something like:

my_module
    __init__.py
   - bin
   - cfg
   - core
     __init__.py
     collection.py
     daemon.py

The ItemRepository code:

class ItemRepository(object):
    """This class provides an implementation to retrieve configuration data 
    from the monocle database
    """
    def __init__(self, project_path):
        if project_path is not None:
            sys.path.append(project_path)

        try:
            from django.core.management import setup_environ
            from someproj import settings
            setup_environ(settings)
            from someproj.someapp.models import ItemConfiguration
        except ImportError:
            print "Could not import models from web app.  Please ensure the\
            PYTHONPATH is configured properly"

    def get_scheduled_collectors(self):
        """This method finds and returns all ItemConfiguration objects 
        that are scheduled to run
        """
        logging.info('At the error path: %s' % sys.path)
        # query ItemConfigs from database
        items = ItemConfiguration.objects.filter(disabled=False) #ERROR OCCURS AT THIS LINE
        return [item for item in items if item.scheduled]

The daemon code (in /usr/local/bin/testdaemon.py):

import sys
from my_module.core.daemon import Daemon
from my_module.core.collection import ItemRepository
import logging
import time

class TestDaemon(Daemon):
    default_conf = '/etc/echodaemon.conf'
    section = 'echo'

    def run(self):
        while True:
            logging.info('The echo daemon says hello')
            ir = ItemRepository(project_path=self.project_path)
            items = ir.get_scheduled_collectors() #TRIGGERS ERROR
            logging.info('there are %d scheduled items' % len(items))
            time.sleep(1)

if __name__ == '__main__':
    TestDaemon().main()

The error I get is “NameError: global name ‘my_module’ is not defined” It get’s past the import but then fails when trying to call a method on the object. I’m assuming it has to do with sys.path / PYTHONPATH, but I know my django project is on the path, as I’ve printed it out. Nothing so far in the python docs or Learning Python has helped yet. Does anyone have any insights or know of a good reference to module imports?

UPDATE:

Now I’ve attempted to simplify the problem to make it easier to understand. Now I have a directory structure that looks like:

/home
  /username
    /django
      /someproj
         /web
           models.py
      /my_module
         daemon.py

I have set the $PYTHONPATH variable in /etc/bash.bashrc to ‘/home/username/django’.

Inside the testdaemon.py file the imports look like:

import logging
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj.web.models import ItemConfiguration

But now I get an ImportError: No module named ‘someproj’. So then I appended the path.

import sys
sys.path.append('/home/username/django')
import logging
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj.web.models import ItemConfiguration

And now the ImportError says: No module named ‘web’. Here’s the traceback:

Traceback (most recent call last):
  File "testdaemon.py", line 77, in <module>
    TestDaemon('/tmp/testdaemon.pid').run()
  File "testdaemon.py", line 47, in run
    scheduled_items = [item for item in ItemConfiguration.objects.filter(disabled=False) if collector.scheduled]
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 141, in filter
    return self.get_query_set().filter(*args, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 550, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 568, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 1128, in add_q
    can_reuse=used_aliases)
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 1026, in add_filter
    negate=negate, process_extras=process_extras)
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 1179, in setup_joins
    field, model, direct, m2m = opts.get_field_by_name(name)
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/options.py", line 291, in get_field_by_name
    cache = self.init_name_map()
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/options.py", line 321, in init_name_map
    for f, model in self.get_all_related_m2m_objects_with_model():
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/options.py", line 396, in get_all_related_m2m_objects_with_model
    cache = self._fill_related_many_to_many_cache()
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/options.py", line 410, in _fill_related_many_to_many_cache
    for klass in get_models():
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 167, in get_models
    self._populate()
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 61, in _populate
    self.load_app(app_name, True)
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/loading.py", line 76, in load_app
    app_module = import_module(app_name)
  File "/usr/local/lib/python2.6/dist-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
ImportError: No module named web

So going from the earlier comments I tried adding:

import sys
sys.path.append('/home/username/django')
import logging
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj import web
from someproj.web import models
from someproj.web.models import ItemConfiguration

But that didn’t help. So I created a very simple file:

#!/usr/bin/python

import logging
import time
import sys
sys.path.append('/home/username/django')
from django.core.management import setup_environ
from someproj import settings
setup_environ(settings)
from someproj.web.models import ItemConfiguration

if __name__ == '__main__':    
    print sys.path
    items = ItemConfiguration.objects.all()
    for item in items:
        print item

And this works! Which really only further confuses me. So now I’m thinking maybe it has to do with the daemon. It uses os.fork() and I’m not sure if the path is still set. This is why I set the $PYTHONPATH variable in the /etc/bash.bashrc file.

Any more insights? I really need the daemon, I don’t have much of a choice as I need a long running process.

Answer 1

It ended up being that I needed to reference the fully qualified name of my app in my Django project’s settings.py file in the INSTALLED_APPS setting. “someproj.web” instead of just “web”. Using the shorter version works fine within a Django project, just not so well from the outside.