Question

I am writing a PHP framework which can be used to visualize huge amounts of data. I cannot fetch all the data into memory and process it (like sort,filter etc). So I have a sql query builder which constructs sql queries and pushes all the processing part to the sql server. Is there a way I can do it with PHP? Or is there some kind of C library available which stores data in binary and reduces the memory usage while processing?

Answer 1

I make heavy use of a caching engine in PHPExcel (using APC, serialization of objects, php://temp, disk storage, in-memory SQLite, memcache and other options), which can work with huge numbers (several millions) of cell objects. The problem is reducing the memory overhead by factoring as much out of php’s limited memory as possible, while trying to keep execution speed reasonable: any cell cache that stores data outside of PHP memory will invariable add execution speed overhead.

Serialization of data using igbinary is a good trade-off, but you can still hit memory limits. It’s about as fast as PHP’s built in serialize, but a lot more efficient in terms of compressing the data.

I’d say to do as much on the database as you possibly can, because databases are designed for this kind of work.