Question

I have a service that requires some input files from the client. This service is run from a number of different cloud platforms (Amazon, Azure, etc.). Further, it requires that these input files be stored in their respective cloud platform’s persistent storage. What is the best way to transfer those input files from the client to the service without requiring that the client knows about the specific cloud platform’s details (e.g. secret keys)?

Given these requirements, the simple approach of the client transferring these files directly to the platform’s storage (e.g. S3) isn’t possible.

I could have the client call a service method and pass it the file blob to transfer up to the service, but that would require that the service receive the file, then do it’s own file transfer to the cloud platform’s storage (in other words, do 2 file transfers for 1 file).

So, the question is, what is the best way to transfer a file to storage given that we don’t know what cloud platform the service is running on?

Answer 1

In this setting your proposed proxy solutions seems to be the only one and although the double transfer seems to be bad, in practice its not that evil as the cloud storage is commonly in the same datacenter as the compute nodes and therefore very fast to access.