Question

Let say i have 100k records in table, after fetching that records from table i am pushing it to an array with some calculations, and then send them to server for further processing.

I have test the scenario with(1k) records, its working perfectly, but worrying about if there is performance issue, because the page which do calculation and fetching records from db run after each 2 mins.

My Question is can I use array for more than 2 Millions records?

Answer 1

My Question is can I use array for more than 2 Millions records?

Yes you can, 2 Million array entries is not a limit in PHP for arrays. The array limit depends on the memory that is available to PHP.

ini_set('memory_limit', '320M');
$moreThan2Million = 2000001;
$array = range(0, $moreThan2Million);
echo count($array); #$moreThan2Million

You wrote:

The page is scheduled and run after 2 min, so I am worrying about the performance issue.

And:

But I need to fetch all, not 100 at time, and send them to server for further processing.

Performance for array operations is dependent on processing power. With a fast enough computer, you should not run into any problems. However, keep in mind that PHP is an interpreted language and therefore considerably slower than compiled binaries.

If you need to run the same script every 2 minutes but the runtime of the script is larger than two minutes, you can distribute script execution over multiple computers, so one process is not eating the CPU and memory resources of the other and can finish the work in meantime another process runs on an additional box.

Edit

Good answer, but can you write your consideration, about how much time the script will need to complete, if the there is no issue with the server processor and RAM.

That depends on the size of the array, the amount of processing each entry needs (in relation to the overall size of the array) and naturally the processor power and the amount of RAM. All these are unspecified with your question, so I can specifically say, that I would consider this unspecified. You’ll need to test this on your own and building metrics for your application by profiling it.

I have 10GB RAM and More than 8 Squad processor.

For example you could do a rough metric for 1, 10, 100, 1000, 10000, 100000 and 1 million entries to see how your (unspecified) script scales on that computer.

I am sending this array to another page for further processing.

Metric as well the amount of data you send between computers and how much bandwidth you have available for inter-process communication over the wire.