I created a program that iterates over a bunch of files and invokes for some of them:
scp <file> user@host:<remotefile> However, in my case, there may be thousands of small files that need to transferred, and scp is opening a new ssh connection for each of them, which has quite some overhead.
I was wondering if there is no solution where I keep one process running that maintains the connection and I can send it ‘requests’ to copy over single files.
Ideally, I’m looking for a combination of some sender and receiver program, such that I can start a single process (1) at the beginning:
ssh user@host receiverprogram And for each file, I invoke a command (2):
senderprogram <file> <remotefile> and pipe the output of (2) to the input of (1), and this would cause the file to be transferred. In the end, I can just send process (1) some signal to terminate.
Preferably the sender and receiver programs are open source C programs for Unix. They may communicate using a socket instead of a pipe, or any other creative solution.
However, it is an important constraint that each file gets transferred at the moment I iterate over it: it is not acceptable to collect a list of files and then invoke one instance of scp to transfer all the files at once at the end. Also, I have only simple shell access to the receiving host.
Update: I found a solution for the problem of the connection overhead using the multiplexing features of ssh, see my own answer below. Yet, I’m starting a bounty because I’m curious to find if there exists a sender/receiver program as I describe here. It seems there should exist something that can be used, e.g. xmodem/ymodem/zmodem?
This way would work, and for other things, this general approach is more or less right.