Question

I’m currently reverse engineering a network protocol and I wrote a small decryption protocol.

I used to define the bytes of the packet into an unsigned character array, as so:

unsigned char buff[] = "\x00\xFF\x0A" etc.

In order to not recompile the program multiple times per packet I made a small GUI tool where it would get the bytes in \xFF notation from a string. I did this the following way:

int length = int(stencString.length());
unsigned char *buff = new unsigned char[length+1];
memcpy(buff, stencString.c_str(), length+1);

When I call my function it gives me a proper decryption when I hardcode it using the prior method but it gives me garbage then the rest of my string when I memcpy from the string to the array. The creepy part? They both have the same print output!

Here’s how I’m using it:
http://pastie.org/private/kndfbaqgvmjiuwlounss9g

Here’s kdxalgo.h (c) Luigi Auriemma:
http://pastie.org/private/7dzemmwyyqtngiamlxy8tw

Can someone point me in the right direction?

Thanks!

Answer 1

See what happens when you use the following for the hardcoded version of buff.

unsigned char buff[] =
"\\xd3\\x8c\\x38\\x6b\\x82\\x4c\\xe1\\x1e"
"\\x6b\\x7a\\xff\\x4c\\x9d\\x73\\xbe\\xab"
"\\x38\\xc7\\xc5\\xb8\\x71\\x8f\\xd5\\xbb"
"\\xfa\\xb9\\xf3\\x7a\\x43\\xdd\\x12\\x41"
"\\x4b\\x01\\xa2\\x59\\x74\\x60\\x1e\\xe0"
"\\x6d\\x68\\x26\\xfa\\x0a\\x63\\xa3\\x88";

I have a suspicion that it will produce the same output as you entering the following: \xd3\x8c\x38\x6b\x82\x4c\xe1\x1e\x6b\x7a\xff\x4c\x9d\x73\xbe\xab\x38\xc7\xc5\xb8\x71\x8f\xd5\xbb\xfa\xb9\xf3\x7a\x43\xdd\x12\x41\x4b\x01\xa2\x59\x74\x60\x1e\xe0\x6d\x68\x26\xfa\x0a\x63\xa3\x88.

The compiler automatically takes “\xd3” and converts it into the expected underlying binary representation. You need to have a method of converting the characters backslash, x, d, 3 into the same binary representation.

---

If you are certain that you will receive properly formated input, then the answer isn’t too hard:

unsigned char c2h(char ch)
{
    switch (ch)
    {
        case '0': return  0;
        case '1': return  1;
        case '2': return  2;
        case '3': return  3;
        case '4': return  4;
        case '5': return  5;
        case '6': return  6;
        case '7': return  7;
        case '8': return  8;
        case '9': return  9;
        case 'a': return 10;
        case 'b': return 11;
        case 'c': return 12;
        case 'd': return 13;
        case 'e': return 14;
        case 'f': return 15;
    }
}

std::string handle_hex(const std::string& str)
{
    std::string result;

    for (size_t index = 0; index < str.length(); index += 4) // skip to next hex digit
    {
        // str[index + 0] is '\\' and str[index + 1] is 'x'
        unsigned char ch = c2h(str[index+2]) * 16 + c2h(str[index+3]);
        result.append((char)ch);
    }

    return result;
}

Again assuming perfect formatting, so there is not error handling. I know that I’ll lose some points for this answer because it’s not the best way of doing this, but I want to make the algorithm as easy to understand as possible.