We’re developing a windows service which will act as a sort of ‘slave’ process. This process basically downloads a pdf, splits it into several pdfs then needs to send that pdf back.
We’re currently using a http request to retrieve the pdf and a number of POSTs to send the files back. This is so the slave service can be run from pretty much any machine and more slaves can be easily added to lighten the load as necesscary.
My question is: is using http for file transfers like this significantly slower than, for example, just using copy commands (which would work only if the slave is on the same machine/network).
Using normal commands is feasible but I like the flexibility in just being able to add a new slave to anywhere.
My completely gut-based thought would be that some protocols, such as NFS, would in some circumstances be somewhat and maybe even significantly faster than HTTP. But I wouldn’t think that’s enough data to go by. I think you need to just figure out how much of a difference would matter to you, and then run some quick tests. Better yet, make a gut call on which one’s a better fit for your needs — HTTP is certainly much easier to get through firewalls and over the open internet, and maybe even over a VPN — and just try that first. If you hit a wall, experiment with other options.
Update: right after I posted this, I remembered that Backblaze, the online backup service, uses HTTPS for all its internal data transfer to and from their storage appliances. This is documented in this post: Petabytes on a budget:
How to build cheap cloud storage — jump down to “A Backblaze Storage Pod Runs Free Software”. There’s some good thinking there on the advantages of HTTPS over a lower-level protocol. And they have to transfer a lot of data, quickly. So if it works for them, there’s a good chance it’ll work for you.