Question

In some machine there are almost five millions of small (50KB) text files. I need to send them to another machine on our LAN. I tried doing

scp *.txt remote_machine:

since ssh connection is setup passwordless between them. But a new connection is established for each file, so it is painstakingly slow.

I wonder therefore what would be the best strategy for doing this.

Answer 1

Not sure if scp is multi-threaded. If not try something like this so better utilize all cores/ cpus and network bandwidth:

scp [A-M]*.txt remote_machine:
scp [M-Z]*.txt remote_machine:
scp [0-9]*.txt remote_machine:
...

Of course the patterns to use depend on the naming of your files.

Instead of scp you could also use rsync with the same approach.