Question

I have a PHP SOAP web service, using CakePHP, which is returning a variable amount of data back to clients. The amount of data returned can be very little, or huge ( > 10MB but < 100MB).

The problem I’m having is that with larger responses (usually greater than 5 MB), the application runs out of memory and things come crashing down. The first thing I’ve tried changing is using raw SQL queries rather than Cake’s data access level. This way, when I retrieve 10,000 records from the database, Cake won’t try to keep it all in memory.

That has worked really well, although it’s a bad convention. The bigger problem however, is once I have the results, it still runs out of memory transmitting the response. I’m not sure if this is happening when the SoapServer compiles the response, or if the client runs out of memory storing the response, or if it runs out of memory just serializing/rendering the data on the screen.

In any case, it seems like if there were some way to stream the SOAP response back to the client, everything would work a lot better. No matter what the issue is, the data could be broken down into manageable chunks. I don’t know how to do this in PHP though, or if it’s even possible.

Please advise.

Answer 1

You should be getting an error message telling you what exhausted the memory. In theory, this should help you determine what is failing. My guess is that SOAPServer is storing the response in memory before sending a response, likely because it is building an XML DOM using PHP’s DomDocument. The downside of DomDocument (or, possibly SimpleXML) is that the entire document must be processed into memory and valid as a DOM document before the data can be serialized into XML.

A better way to stream large chunks of XML data would be to use PHP’s XMLWriter extension. Unlike DOM, it does not need to store the entire document into memory and it expects you to handle document validation. You can use this extension to output directly to the HTTP response using the XMLWriter::openUri function. This means that you can perform the following operations in order and have minimal memory overhead:

1. Open a connection to the database
2. Search for the stuff you need
3. Create an instance of XMLWriter using the XMLWriter::openUri function with the parameter php://output for the URI. This will push everything to the output (similar to print/echo) and store (almost) nothing in memory. Make sure you’ve already pushed out any headers, cookies, etc you need first, just like any other outputting check.
4. Create the SOAP preamble.
5. Loop through the items you are going to return in the soap response, adding the elements to the output.
6. Create the SOAP postamble

As a side note, the reverse of XMLWriter is XMLReader, if you are interesed in reading large XML files that cannot be opened using normal methods due to memory limits.