Question

I am writing code to deploy using Fabric to all the web machines and was wondering how rsync vs. upload project works in terms of parallelizing and shortest time.

Are there any benchmarks?

Can I rsync to 100 machines in parallel? What’s the limiting factor?

  rsync_project(
        env.root,
        exclude=RSYNC_EXCLUDE,
        delete=True,
        extra_opts=extra_opts,
    )

Similarly what’s the limiting factor for upload_project? What’s the sftp limit in terms of number?

@parallel
def testapp():
    with cd('~/projects'):
        upload_project('./receiver', '/home/sysadmin/projects')

From a hunch stand point, upload project should be better since the tar needs to be done only once and then its sftp. Or is it multiple times for an example shown above?

Does fabric do some sort of throttling to ensure the network is not choked to the limit?

Can someone help with this?

Answer 1

Before benchmark rsync and upload_project, you have to know that rsync transmit only differential data. Rsync will become very efficient than upload_project when your deployment only contains few changes.

delete=True means that files that have been deleted locally will be deleted remotely. It’s probably what you want.

If you insist, I have to say the benchmark result will depend on the file numbers and size of them. For example, if you have 1K-sized file of 1G, rsync will be much slower than upload_project. Because the later one always pack a tar/gzip and then transmit this large file.

At last, fabric doesn’t have the “tar cache”, it will repeat tar on every deploy because the code is written as:

    finally:
        run("rm -f %s" % tar_file)
finally:
    local("rm -rf %s" % tmp_folder)

But you could add some cache or just comment out them manually.

For the network thing, fabric leaves them to sftp which have Congestion window to make
sure the network is not choking.