Question

The project I’m working in uses fabric for many build steps and requires a offline build as fallback.

I’m currently stuck at installing python packages provided in tarballs.

The thing is I have trouble getting into the newly extracted directory and running setup.py install in there.

@task
def deploy_artifacts():
    """Installs dependencies from local path, useful for offline builds"""
    #TODO: Handle downloading files and do something like this bellow
    tmpdir = tempfile.mkdtemp()
    artifacts_path = ''
    if not 'http' in env.artifacts_path:
        artifacts_path = env.artifacts_path

    with lcd(artifacts_path):

        for f in os.listdir(artifacts_path):
            if 'gz' in f:
                put(f, tmpdir)
                tar = os.path.join(tmpdir, f)
                target_dir = os.path.join(tempfile.gettempdir(), normalize(f))
                if not files.exists(target_dir):
                    run('mkdir %s' % target_dir)
                else: 
                    run('rm -rf %s' %target_dir)
                    run('mkdir %s' % target_dir)
                run('tar xf %s -C %s' % (tar, target_dir))
                run('rm %s' % tar)
                with cd(target_dir):
                    sudo('python setup.py install')

I come from reading the tar man page for the bazillion time and I got nowhere near to getting what I want.

Did some of you face a situation like this? is there some other (read: better) approach to this scenario?

Answer 1

There’s nothing wrong (in principle) with what you’re trying do. Maybe just take smaller steps getting there. Rather than using temporary directories, it might make debugging easier if everything was put in a systematic location that has known permissions that nothing else writes to by convention. At least that would let you use some combination of fabric and manual intervention to check what is going wrong.

In the longer term, there are a few alternatives that I see. For simplicity you want the online and offline versions to work the same way, and that means fetching packages using easy_install / pip for both cases.

One way to do this is to build a mirror of PyPi. The right way to do this if you’ve got plenty of storage space (30Gb) is to use software that implements PEP381 (Mirroring Infrastructure for PyPI), there is already a client that does this (pep381client). A number of other projects are available that do similar things (basketweaver, djangopypi2, chishop).

An alternative is to consider a lighter weight proxying scheme. I’ve been looking a pip2pi and pipli. I’m unsure if they will work directly with easy_install, but it would be worth a try.

It’s also worth noting that if you were using pip, you could have installed directly from the tarballs.