Question

This is in relation to my post here but taken in a completely different direction
Charset detection in PHP

essentially, i’m looking to reduce the memory that many huge arrays cause.

These arrays are just full of integers but seeing as PHP uses 32bit and 64 bit integers internally (depending which version you have compiled for your CPU type), it eats the memory.

is there a way to cheat PHP into using 8bit or 16bit integers?

I’ve thought about using pack(); to accomplish this so I can have an array of packed binary values and just unpack them as I need them (yes I know this would make it slower but is much faster than the alternative of loading and then running through each array individually as you can stream the text through so they all need to be in memory at the same time to keep speed up)

can you suggest any better alternatives to accomplish this? i know it’s very hacky but I need to prevent huge memory surges.

Answer 1

Don’t tell nobody!

class IntegerstringArray IMPLEMENTS ArrayAccess {

    var $evil = "0000111122220000ffff";

                 // 16 bit each


    function offsetExists ( $offset ) {
        return (strlen($this->evil) / 4) - 1 >= $offset;
    }

    function offsetGet ( $offset ) {
        return hexdec(substr($this->evil, $offset * 4, 4));
    }

    function offsetSet ( $offset , $value ) {

        $hex = dechex($value);
        if ($fill = 4 - strlen($hex)) {
            $hex = str_repeat("0",  $fill) . $hex;
        }

        for ($i=0; $i<4; $i++) {
            $this->evil[$offset*4+$i] = $hex[$i];
        }
    }

    function offsetUnset ( $offset ) {
        assert(false);
    }

}

So you can pretty much create an array object from this:

$array = new IntegerstringArray();
$array[2] = 65535;
print $array[2];

It internally stores a list and accepts 16-bit integers. The array offsets must be consecutive.

Not tested. Just as an implementation guide.