Question

What I try to do:

I want to store very much data in RAM. For faster access and less memory footprint I need to use an array of struct values:

MyStruct[] myStructArray = new MyStruct[10000000000];

Now I want to store unsigned integer values with one, two, three or four bytes in MyStruct. But it should only use the less possible memory amount. When I store a value it one byte it should only use one byte and so on.

I could implement this with classes, but that is inappropriate here because the pointer to the object would need 8 bytes on a 64bit system. So it would be better to store just 4 bytes for every array entry. But I want to only store/use one/two/three byte when needed. So I can’t use some of the fancy classes.

I also can’t use one array with one bytes, one array with two bytes and so on, because I need the special order of the values. And the values are very mixed, so storing an additional reference when to switch to the other array would not help.

Is it possible what want or is the only way to just store an array of 4 byte uints regardless I only need to store one byte, two byte in about 60% of the time and three bytes in about 25% of the time?

Answer 1

You could get close to that what you want if you are willing to sacrifice some additional CPU time and waste additional 2 or 4 bits per one stored value.

You could just use byte byte[] and combine it with BitArray collection. In byte[] you would then just sequentially store one, two, three or four bytes and in BitArray denote in binary form (pairs of two bits) or just put a bit to value 1 to denote a new set of bytes have just started (or ended, however you implement it) in your data array.

However you could get something like this in memory:

byte[]   --> [byte][byte][byte][byte][byte][byte][byte]...
BitArray --> 1001101...

Which means you have 3 byte, 1 byte, 2 bytes etc. values stored in your byte array.

Or you could alternatively encode your bitarray as binary pairs to make it even smaller. This means you would vaste somewhere between 1.0625 and 1.25 bytes per your actual data byte.

It depends on your actual data (your MyStruct) if this will suffice. If you need to distinguish to which values in your struct those bytes really corresponds, you could waste some additional bits in BitArray.

Update to your O(1) requirement:

Use another index structure which would store one index for each N elements, for example 1000. You could then for example access item with index 234241 as

indexStore[234241/1000]

which gives you index of element 234000, then you just calculate the exact index of element 234241 by examining those few hundred elements in BitArray.

O(const) is acheieved this way, const can be controlled with density of main index, of course you trade time for space.