Question

I have two directories in my project:

project/
  src/
  scripts/

“src” contains my polished code, and “scripts” contains one-off Python scripts.

I would like all the scripts to have “../src” added to their sys.path, so that they can access the modules under the “src” tree. One way to do this is to write a scripts/__init__.py file, with the contents:

scripts/__init__.py:
  import sys
  sys.path.append("../src")

This works, but has the unwanted side-effect of putting all of my scripts in a package called “scripts”. Is there some other way to get all my scripts to automatically call the above initialization code?

I could just edit the PYTHONPATH environment variable in my .bashrc, but I want my scripts to work out-of-the-box, without requiring the user to fiddle with PYTHONPATH. Also, I don’t like having to make account-wide changes just to accommodate this one project.

Answer 1

Even if you have other plans for distribution, it might be worth putting together a basic setup.py in your src folder. That way, you can run setup.py develop to have distutils put a link to your code onto your default path (meaning any changes you make will be reflected in-place without having to “reinstall”, and all modules will “just work,” no matter where your scripts are). It’d be a one-time step, but that’s still one more step than zero, so it depends on whether that’s more trouble than updating .bashrc. If you use pip, the equivalent would be pip install -e /path/to/src.

The more-robust solution–especially if you’re going to be mirroring/versioning these scripts on several developers’ machines–is to do your development work inside a controlled virtual environment. It turns out virtualenv even has built-in support for making your own bootstrap customizations. It seems like you’d just need an after_install() hook to either tweak sitecustomize, run pip install -e, or add a plain .pth file to site-packages. The custom bootstrap could live in your source control along with the other scripts, and would need to be run once for each developer’s setup. You’d also have the normal benefits of using virtualenv (explicit dependency versioning, isolation from system-wide configuration, and standardization between disparate machines, to name a few).

If you really don’t want to have any setup steps whatsoever and are willing to only run these scripts from inside the ‘project’ directory, then you could plop in an __init__.py as such:

project/
    src/
        some_module.py
    scripts/
        __init__.py # special "magic"
        some_script.py

And these are what your files could look like:

# file: project/src/some_module.py
print("importing %r" % __name__)

def some_function():
    print("called some_function() inside %s" % __name__)
--------------------------------------------------------
# file: project/scripts/some_script.py
import some_module

if __name__ == '__main__':
    some_module.some_function()
--------------------------------------------------------
# file: project/scripts/__init__.py
import sys
from os.path import dirname, abspath, join

print("doing magic!")
sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'src'))

Then you’d have to run your scripts like so:

[~/project] $ python -m scripts.some_script
doing magic!
importing 'some_module'
called some_function() inside some_module

Beware! The scripts can only be called like this from inside project/:

[~/otherdir] $ python -m scripts.some_script
ImportError: no module named scripts

To enable that, you’re back to editing .bashrc, or using one of the options above. The last option should really be a last resort; as @Simon said, you’re really fighting the language at that point.