Question

I’m writing an API using php to wrap a website functionality and returning everything in json\xml. I’ve been using curl and so far it’s working great.
The website has a standard file upload Post that accepts file(s) up to 1GB.

So the problem is how to redirect the file upload stream to the correspondent website?

I could download the file and after that upload it, but I’m limited by my server to just 20MG. And it seems a poor solution.

Is it even possible to control the stream and redirect it directly to the website?

Answer 1

I preserverd original at the bottom for posterity, but as it turns out, there is a way to do this

What you need to use is a combination of HTTP put method (which unfortunately isn’t available in native browser forms), the PHP php://input wrapper, and a streaming php Socket. This gets around several limitations – PHP disallows php://input for post data, but it does nothing with regards to PUT filedata – clever!

If you’re going to attempt this with apache, you’re going to need mod_actions installed an activated. You’re also going to need to specify a PUT script with the Script directive in your virtualhost/.htaccess.

http://blog.haraldkraft.de/2009/08/invalid-command-script-in-apache-configuration/

This allows put methods only for one url endpoint. This is the file that will open the socket and forward its data elsewhere. In my example case below, this is just index.php

I’ve prepared a boilerplate example of this using the python requests module as the client sending the put request with an image file. If you run the remote_server.py it will open a service that just listens on a port and awaits the forwarded message from php. put.py sends the actual put request to PHP. You’re going to need to set the hosts put.py and index.php to the ones you define in your virtual host depending on your setup.

Running put.py will open the included image file, send it to your php virtual host, which will, in turn, open a socket and stream the received data to the python pseudo-service and print it to stdout in the terminal. Streaming PHP forwarder!

There’s nothing stopping you from using any remote service that listens on a TCP port in the same way, in another language entirely. The client could be rewritten the same way, so long as it can send a PUT request.

The complete example is here:

https://github.com/DeaconDesperado/php-streamer

I actually had a lot of fun with this problem. Please let me know how it works and we can patch it together.

Begin original answer

There is no native way in php to pass a file asynchronously as it comes in with the request body without saving its state down to disc in some manner. This means you are hard bound by the memory limit on your server (20MB). The manner in which the $_FILES superglobal is initialized after the request is received depends upon this, as it will attempt to migrate that multipart data to a tmp directory.

Something similar can be acheived with the use of sockets, as this will circumvent the HTTP protocol at least, but if the file is passed in the HTTP request, php is still going to attempt to save it statefully in memory before it does anything at all with it. You’d have the tail end of the process set up with no practical way of getting that far.

There is the Stream library comes close, but still relies on reading the file out of memory on the server side – it’s got to already be there.

What you are describing is a little bit outside of the HTTP protocol, especially since the request body is so large. HTTP is a request/response based mechanism, and one depends upon the other… it’s very difficult to accomplish a in-place, streaming upload at an intermediary point since this would imply some protocol that uploads while the bits are streamed in.

One might argue this is more a limitation of HTTP than PHP, and since PHP is designed expressedly with the HTTP protocol in mind, you are moving about outside its comfort zone.

Deployments like this are regularly attempted with high success using other scripting languages (Twisted in Python for example, lot of people are getting onboard with NodeJS for its concurrent design patter, there are alternatives in Ruby or Java that I know much less about.)