Question

I’m looking for something resembling an STL vector but can handle integers that are, for example, 12, 16, 20, 24, 32, and 40 bits long. The 16-bit and 32-bit cases are handled wonderfully by vector<uint16_t> and vector<uint32_t>, but I haven’t been able to find any way to handle the other ones. Note that the whole purpose of going this way is to save memory and bandwidth, so padding is not an option.

My data structure can infer the most significant bits of the integers (which are int64s), hence I only want to store the LSBs. The bits-per-integer and number of integers are known at creation time, but not compile time. Ideally bits per integer can be any value between 12 and 40, but tiers are ok for performance reasons or to work with a structure where bits-per-integer needs to be set at compile time.

vector<bool> and dynamic_bitset can create bitfields, but they’re limited to 1-bit integers. Anyone know of something else out there?

Answer 1

There is none in the STL.

I understand you performance concern, and notably the memory issue. However unaligned reads (requiring bit-shifting) can be slower than regular ones (on bytes boundaries), so I would suggest keeping to whole bytes. This means:

• 12 bits, 16 bits: uint16_t
• 20 bits, 24 bits: either 3 uint8_t or 1 uint32_t (tradeoff speed/memory)
• 32 bits: uint32_t
• 40 bits: either 5 uint8_t, 3 uint16_t or 1 uint64_t

Assuming you choose the easier solution (ie, always bumping to the next available integer), you could use something like:

boost::variant<
  std::vector<uint16_t>,
  std::vector<uint32_t>,
  std::vector<uint64_t>
>

This let you choose the exact vector to be used at runtime, based on the number of bytes you need to store.

If you want to spare as much as possible, then you simply need to add a std::vector<uint8_t> to the list. But I would refrain from doing so and perhaps only “pack” the 40 bits case (in uint16_t) as the others never waste much memory anyway… and I would probably profile to check both alternatives.

Note: you might also wish to tune the vector memory usage by checking that the capacity does not exceed the size too much.