Question

I am creating a website in the CMS/framework called Processwire now I do want to fill a few arrays with values i collected from database results (called pages in processwire).
Now what happens is that i get a memory error and I know that i should raise the memory limit but that doesn’t feel right because the pages will even get more and more then it is already therefor the load will get bigger and i might end up increasing the memory limit even more. Now I would like kind of pick batches from the big pile and process them and then pick the next batch.
this is the part that causes to crash:

    for($i = 0; $i<=$count_log; $i+=100)
{
    $all_log_files = $pages->find("template=logs, start=$i, limit=100");
    foreach($all_log_files as $log_file)
    {
        $u++;
        if(empty($actions["$log_file->action"]))
        {
            $actions[$log_file->action] = 1;
        }
        if(empty($ips["$log_file->ip"]))
        {
            $ips[$log_file->ip] = 1;
        }
        if(empty($values["$log_file->value"]))
        {
            $values[$log_file->value] = 1;
        }
        if(empty($results["$log_file->result"]))
        {
            $results[$log_file->result] = 1;
        }
    }
}

Now as you can see i already tried to make “batches” but I failed since i’d still get the error…

What can i do this fix this or isn’t there anything i can do and should i just raise the memory limit?

EDIT:

The specific error:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 6 bytes) in /xxx/xxx/xxx/xxx/wire/core/DatabaseQuery.php on line 47

Answer 1

You’re having a memory issue because the data that you’re trying to store in memory all at once is too big. You can either increase the memory limit to hold all the data or store less data and change your algorithms to work with batches of the data.

For example, if you have a function that uses the values stored in $results, then you can try to change this function to work just as well with only part of the results, and call the function for each iteration, then empty the array after the iteration finishes.