Project Requirements:
I need to build a web service that will receive chunks of text data, any where from a few kilobytes up to 10 Megabytes, but the average hovering around a few hundred kilobytes. There will be a high volume of transactions and I’d like to secure the service with authentication, SSL, and eventually some encryption.
I’d like the service to support XML and JSON if possible, and I’d like to build it using open source tools – most likely PHP/MySQL.
Suggestions regarding possible PHP frameworks are welcome.
I had already started down the path to build a REST web service in PHP with Tonic framework, but I thought that I may be limited to the amount of data I could post in a URI and posting so much data in a URL sounded cumbersome. So I’ve been looking at SOAP solutions.
Questions:
Am I limited in the amount of data I can post to a REST service?
Are there data limits to the SOAP protocol?
Do you recommend I use SOAP or REST based on the project requirements and why?
HTTP isn’t limited at all at the protocol level whether you’re using the REST architecture or the SOAP protocol tunneled over HTTP.
You major limit will be memory, especially if you’re streaming several multi-megabyte payloads. With a clever enough handler, and a compliant request, you could choose to stream smaller requests to memory only and larger ones directly to disk before processing.
Details like that likely limit how much whatever framework you choose can do for you, since most will likely consume straight in to memory. Whether you can change that at all, or on a request by request basis is up to the framework and possibly your desire to change it.
The choice of SOAP as a protocol is likely irrelevant. It’s just a simple XML envelope around your payload, unless you also choose to encode your messages with the SOAP encodings.
Since you want to support JSON as well, a SOAP stack would likely simply get in the way over a raw HTTP processing backend.
You don’t mention the complexity of your API, but you could likely get by with a framework for much of it and just write a custom handler for the large, heavy transactions.
Your transactions per sec isn’t going to be high, since the transaction themselves will be so large. If you’re streaming to disk, you’ll have a large I/O impact as well (you’re going to have that anyway unless you’re converting the data in RAM and spitting it back out the socket rather than saving it). So, I/O throughput is likely going to be your major bottleneck in terms of what’s going to limit your overall performance.
If you’re getting uploads from consumers, you’ll probably be getting about 150K bytes per second (lots of consumer upload speeds suck — mine for example). If you have an I/O channel that can push, say, 60MB/sec sustained, that’s 400 simultaneous connections. So that should give you an idea of machine scaling based on anticipated transaction volumes (of course, you’ve already done this analysis — I’m just making this stuff up as I go). With each transaction uploading 300K, that’s 2s per transaction, so 200 TPS. That’s another number to play with (in terms of perhaps what your database can sustain, etc. especially if you’re going to have several machines feeding it).
Modern CPUs are disgusting, so PHP or whatever should be fine — that won’t be your problem.