Question

Redis keys are binary safe. I’d like to mess around and put binary into redis using C#. My client of choice doesn’t support writing binary keys it uses keys and it make sense. However i am just fooling around so tell me how i can do this.

How do i convert a raw byte[] into a string? At first i was thinking about converting a byte[] to a utf8 string however unicode has some checks to see if its valid or not. So raw binary should fail.

Actually i tried it out. Instead of failing i got a strange result. My main question is how do i convert a raw byte[] to the equivalent string? As in have the raw byte[] as a string and not encoding as base32/64/hex/whatever. My unimportant question is why did i get a 512 byte string instead of an exception saying this is not a valid UTF8 string?

code

var rainbow = new byte[256];
for (int i = 0; i < 256; i++)
{
    rainbow[i] = (byte)i;
}
var sz = Encoding.UTF8.GetString(rainbow);
var szarr = Encoding.UTF8.GetBytes(sz);
Console.WriteLine("{0} {1} {2}", ByteArraysEqual(szarr, rainbow), szarr.Length, rainbow.Length);

Output

False 512 256

Answer 1

You have to use some kind of encoding to convert bytes to a string. The encoding iso-8859-1 will give the correct result:

var sz = Encoding.GetEncoding("iso-8859-1").GetString(rainbow);
var szarr = Encoding.GetEncoding("iso-8859-1").GetBytes(sz);
Console.WriteLine("{0} {1} {2}", ByteArraysEqual(szarr, rainbow), szarr.Length, rainbow.Length);

True 256 256

The thing is that UTF8 requires more than one bytes per character. It can encode the first 128 characters with one byte:

Console.Write(Encoding.UTF8.GetBytes(Encoding.UTF8.GetString(new byte[] { 127 })).Length);

1

But the rest require three bytes:

Console.Write(Encoding.UTF8.GetBytes(Encoding.UTF8.GetString(new byte[] { 128 })).Length);

3

So, when you convert bytes 0-255 to a string and back with UTF8, the first 128 come back as one byte, but the last 128 come back as 3. 128 + 3*128 = 512, hence your result.

ASCII doesn’t know what to do with bytes past 128, so they just get encoded as ?, and come back as one byte also.