Question

I did some googling and couldn’t find any good article on this question. What should I watch out for when implementing an app that I want to be endian-agnostic?

Answer 1

What should I watch out for when implementing an app that I want to be endian-agnostic?

You first have to recognize when endian becomes an issue. And it mostly becomes an issue when you have to read or write data from somewhere external, be it reading data from a file or doing network communication between computers.

In such cases, endianess matters for integers bigger than a byte, as integers are represented differently in memory by different platforms. This means every time you need to read or write external data, you need to do more than just dumping the memory of your program, or read data directly into your own variables.

e.g. if you have this snippet of code:

unsigned int var = ...;
write(fd, &var, sizeof var);

You’re directly writing out the memory content of var, which means the data gets presented to wherever this data goes just as it is represented in your own computer’ memory.

If you write this data to a file, the file content will be different whether you run the program on a big endian or a little endian machine. So that code is not endian agnostic, and you’d want to avoid doing things like this.

Instead focus on the data format. When reading/writing data, always decide the data format first, and then write the code to handle it. This might already have been decided for you if you need to read some existing well defined file format or implement an existing network protocol.

Once you know the data format, instead of e.g. dumping out an int variable directly, your code does this:

uint32_t i = ...;
uint8_t buf[4];
buf[0] = (i&0xff000000) >> 24;
buf[1] = (i&0x00ff0000) >> 16;
buf[2] = (i&0x0000ff00) >> 8;
buf[3] = (i&0x000000ff);
write(fd, buf, sizeof buf);

We’ve now picked the most significant byte and placed it as the first byte in a buffer, and the least significant byte placed at the end of the buffer. That integer is represented in big endian format in buf, regardless of the endian of the host – so this code is endian agnostic.

The consumer of this data must know that the data is represented in a big endian format. And regardless of the host the program runs on, this code would read that data just fine:

uint32_t i;
uint8_t buf[4];
read(fd, buf, sizeof buf);
i  = (uint32_t)buf[0] << 24;
i |= (uint32_t)buf[1] << 16;
i |= (uint32_t)buf[2] << 8;
i |= (uint32_t)buf[3];

Conversely, if the data you need to read is known to be in little endian format, the endianess agnostic code would just do

uint32_t i ;
uint8_t buf[4];
read(fd, buf, sizeof buf);
i  = (uint32_t)buf[3] << 24;
i |= (uint32_t)buf[2] << 16;
i |= (uint32_t)buf[1] << 8;
i |= (uint32_t)buf[0];

You can makes some nice inline functions or macros to wrap and unwrap all 2,4,8 byte integer types you need, and if you use those and care about the data format and not the endian of the processor you run on, your code will not depend on the endianess it’s running on.

This is more code than many other solutions, I’ve yet to write a program where this extra work has had any meaningful impact on performance, even when shuffeling 1Gbps+ of data around.

It also avoids misaligned memory access which you can easily get with an approach of e.g.

uint32_t i;
uint8_t buf[4];
read(fd, buf, sizeof buf);
i = ntohl(*(uint32_t)buf));

which can also incur a performance hit (insignificant on some, many many orders of magnitude on others) at best, and a crash at worse on platforms that can’t do unaligned access to integers.