Question

I have a phrase counter function as part of a class that I would like to adapt to run very large data sets

<?php
private static function do_phrase_count($words, $multidim, $count, $pcount){
        if($multidim === false){
            $words = array($words);
        }

        $tally = array();
        $arraycount = 0;
        foreach($words as $wordgroup){
             $max = count($wordgrounp) - $pcount;
             for($x = 0; $x < $max; $x++){
                $cutoff = $x + $pcount;
                $spacekey = false;
                $phrase = '';
                $z = 0;
                for($y = $x; $y < $cutoff; $y++){
                    if($spacekey) $phrase .= ' ';
                    else $spacekey = true;
                    $phrase .= $wordgroup[$y + $z];
                    $z++;
                }
                if(isset($tally[$phrase])){
                     $tally[$phrase]++;
                     $arraycount++;
                 }
                else $tally[$phrase] = 1;
                if($arraycount > 99999){
                    arsort($tally);
                    $tally = array_slice($tally, 0, 50000);
                    $arraycount = 49999;
                }
            }
        }
        arsort($tally);
        $out = array_slice($tally, 0, $count);
        return $out;
}

• $words is an array of words to check
• $multidim is a boolean showing if the array is cascading or flat
• $count is the number of elements to be returned
• $pcount is the number of words in a phrase

With every iteration, array_key_exists gets slower, so at a certain point I need to decrease the size of the tally array.

I was considering using a limit (100K) to stop the script from adding new array elements to $tally, or even using a percentage of total words, but after I stop adding new elements to the array I lose the ability to track trends that may pop up. (If I’m analysing data from a whole year, by the time I got to June, I wouldn’t be able to see “summer time” as a trend).

Anyone have a solution as to how to limit my tally array to keep the script zinging without losing the ability to track trends?

UPDATE: I changed the script per your suggestions. Thank you for helping. I also figured out a solution to cut down on the size of the array.

Answer 1

if(isset($tally[$phrase])) 
   $tally[$phrase]++;
else 
   $tally[$phrase] = 1;

It should be faster than array_key_exists

ps: test sample

function genRandomString($length) {
    $characters = '0123456789abcdefghijklmnopqrstuvwxyz';
    $string = '';    
    for ($p = 0; $p < $length; $p++) {
        $string .= $characters[mt_rand(0, strlen($characters)-1)];
    }
    return $string;
}
function microtime_float()
{
    list($usec, $sec) = explode(" ", microtime());
    return ((float)$usec + (float)$sec);
}
$len = 1000000; $str = genRandomString($len);
$tally = array();
$stamp1 = microtime_float();
for($i=0; $i<$len; $i++)
{
    if(array_key_exists($str[$i], $tally)) 
        $tally[$str[$i]]++;
    else 
        $tally[$str[$i]] = 1;
}    
echo microtime_float() - $stamp1 . '<br />';
$tally = array(); $stamp1 = microtime_float();
for($i = 0; $i<$len; $i++)
{
    if(isset($tally[$str[$i]])) 
       $tally[$str[$i]]++;
    else 
       $tally[$str[$i]] = 1;
}    
echo microtime_float() - $stamp1 . '<br />';

Result:

0.80751395225525
0.44111108779907

Of course, the amount of keys is limited here.