Question

in Matlab, I’m trying to pack arbitrary bit-length unsigned integers (e.g., an array of 3-bit integers) into an array of uint8. Given the tip here, I can generate code that works for “small” arrays (say 10,000 elements), but it consumes all memory for large arrays (such as 16 million elements). The code I use is below, borrowing from previous postings:

function x_bytes = stuff_bits(x, n)
    r = dec2bin(x,n);                 % bitstring for each uint in x
    s = reshape(r',[],1);             % one continuous string of bits
    t = reshape(str2num(s),8,[])';    % array of 8-bit numbers (stuffed)
    u = t*(2.^(size(t,2)-1:-1:0))';   % array of bytes representing all the bits stuffed together
    x_bytes = uint8(u);              % should be compressed byte stream
end

I realize that I’m taking a uint, converting it to a string, then converting it back to a bit; I’ve also read that dec2bin is not terribly efficient.

When I try this with 16mil elements (on a 64-bit windows box with 8 GB of memory), all memory is consumed. Blah. So I loop over subsections and it requires about 10 minutes to complete the 16mil elements. So, something is very inefficient.

Anyone got a better way to generate bit strings like python’s BitArray?

thanks,

Answer 1

It seems to be similat to this one and this one

At the first one, it was suggested to use dec2bitvec inside a for loop. This may be enought to you (altought slow).

The second one sugests to create a lookup table using bitget and then use it (instead of using dec2bit or dec2bitvec)

You can try to use something ‘in the middle’.

B = 3; % Number of bits per int.
A = randi(7, 16000000, 1); % 16M random elements between 1 and 7 (3bits).

tic
% get each group of bits in a column of K.
K = cell2mat(arrayfun(@(bit)bitget(A, B+1-bit), 1:B, 'UniformOutput', 0))';
% reshape to have them in 8 packs
K = reshape(K, [8, numel(K)/8])';
% get the uint8 vec.
U = K*(2.^(size(K,2)-1:-1:0))';
toc

Mine was elapsed in 3.5 seconds. (Win8 64bits, i5 4GB ram)

Instead of creating a lookup table, this code is creating a matrix (K) with bit values of each integer (stored in columns), reshaping it (to create 8bin value) and then using the same math as you used before to create the uint8 vector.