Question

I am playing around with xor decoding via a small C file, and am running into issues with endianness …I am a bit stuck on how to work around them. This is really the first time I’ve played this deeply with bitwise operations in C.

If I use a one-byte xor key and pick up several xor-encoded values into a uint8_t pointer, my basic code works fine. Walk each byte, xor it against the key, and store the result in a decoded byte array/buffer and then print it back to the console.

However, if I try a two-byte xor key, then endianness starts to get in the way. I currently stick the key into a uint32_t, because I don’t plan on dealing with xor keys greater than 32bits. On a little-endian system, a xor key of 0xc39f gets stored as 0x9fc3. The bytes to be decoded are big-endian if I play them back one byte at a time, but they too, get flipped to little-endian if I try to play them back two-bytes at a time (same size as the xor key).

I am tempted to #include <byteswap.h> and then call bswap_32(). But while this will work on little endian, it might have the opposite effect on big-endian. I assume then I’d need ugly #ifdef’s to only use bswap_32() for little-endian archs. I figure, there has got to be a more portable way for this to work.

Random sample string:

g   e   n   e   r   a   t   e
67  65  6e  65  72  61  74  65

Xor 0xc39f

a4  fa  ad  fa  b1  fe  b7  fa

If I play back the xor-encoded buffer with two-byte (uint16_t) pointers, I get this (via a basic printf):

0xfaa4 0xfaad 0xfeb1 0xfab7

And with four-byte pointers (uint32_t):

0xfaadfaa4 0xfab7feb1

I would expect for the above, to get instead for two-byte pointers:

0xa4fa 0xadfa 0xb1fe 0xb7fa

And four-byte pointers:

0xa4faadfa 0xb1feb7fa

Thoughts?

Edit: Any takers? Current answers aren’t adequate to my needs.

Answer 1

You’re overthinking this—just treat your xor key as an endianless binary blob, and convert it to a native uint32_t for performance:

void xor_encrypt_slow(uint8_t *data, size_t len, uint8_t key[4])
{
    // key is a 4-byte xor key
    size_t i;
    for(i = 0; i < len; i++)
        data[i] ^= key[i % 4];
}

void xor_encrypt_fast(uint8_t *data, size_t len, uint8_t key[4])
{
    // Convert key to a 32-bit value
    uint32_t key32 = *(uint32_t *)key;

    // This assumes that data is aligned on a 4-byte boundary; if not, adjust
    // accordingly
    size_t i;
    for(i = 0; i + 3 < len; i += 4)
        ((uint32_t *)data)[i] ^= key32;
    // Handle the remainder, if len is not a multiple of 4
    for( ; i < len; i++)
        data[i] ^= key[i % 4];
}