Question

I have a backup server which receives many rsync connections every hour. Since having too many open rsync instances might crash it, I want to limit the number of concurrent instances using a Semaphore. What I have in mind is something like:

ssh root@backup_server "get_semaphore"     #Will hold until semaphore released
rsync -avzrL --super --delete local_directory root@backup_server:`localhost`

Any ideas?

Answer 1

In my opinion, you should be looking at limiting the number of concurrent connections directly on the server side, so that you don’t have to do anything extreme on the client side:

(1) If you run rsync in daemon mode, there is a max connections option for rsyncd.conf.

(2) On Linux, you can use iptables and its connlimit module to limit the number of concurrent connections to a port from one or more remote hosts. You may have to force any rsync clients to use a different port than your normal ssh users and have an sshd instance listen to that as well.

(3) Replace your rsync binary on the server with a wrapper script that will catch the –server parameter that is used internally by rsync and stall until a slot is available. Said wrapper script should make sure that no more than N instances of itself are executed at the same time. E.g.

#!/bin/bash

N=5

mutex_hold() {
    while ! mkdir /var/lock/rsync/mutex 2>/dev/null; do
       sleep 1
    done
}

mutex_release() {
    rmdir /var/lock/rsync/mutex
}

if [[ "$1" = "--server" ]]; then
    shopt -s nullglob

    while mutex_hold && A=(/var/lock/rsync/[0-9]*) && [[ "${#A[@]}" -ge "$N" ]] && mutex_release; do
       sleep 1
    done

    touch /var/lock/rsync/$$

    mutex_release

    rsync.bin "$@"

    rm -f /var/lock/rsync/$$
else
    rsync.bin "$@"
fi

Please note that this script is mostly untested and that it lacks the necessary trap code to remove the lockfile even if interrupted. It also does not take care of any stale lock files etc or creating the lock directory.

If you are set on making your own semaphore implementation, you might be interested in the mutex I used above, in order to avoid the race condition between counting the number of lockfiles and creating a new lockfile, which might allow two (or more) instances to run in a single slot.

In all of these cases you should make sure that your clients can handle connection timeouts or rejected connections gracefully.

If you really want to use a semaphore-style system as you proposed, the script above might serve with a few modifications.