Question

signed int _rotr( int a1, int a2 )
{
    return (a1 << a2 % 0x20u) | ((unsigned int)a1 >> (char)(32 - a2 % 0x20u));
}
int _encrypt_password( int a1, int a2, signed int a3 )
{
    signed int  v10; // [sp+10h] [bp-8h]@3
    int         result; // eax@1
    int         v4; // ebx@3
    int         v5; // edi@3
    int         v6; // esi@3
    int         v7; // eax@4
    int         v8; // [sp+14h] [bp-4h]@2
    int         v9; // [sp+Ch] [bp-Ch]@2

    result = 8 * a3 / 8;
    a3 = 8 * a3 / 8;
    if ( result > 0 )
    {
        result = a2;
        v8 = 0;
        v9 = a2;
        while ( a3 / 8 > v8 )
        {
            v4 = *(_DWORD *)v9 + *(_DWORD *)(a1 + 16);
            v6 = *(_DWORD *)(v9 + 4) + *(_DWORD *)(a1 + 20);
            v10 = 1;
            v5 = a1 + 24;
            do
            {
                v7 = *(_DWORD *)v5 + _rotr(v6 ^ v4, v6);
                v4 = v7;
                v6 = *(_DWORD *)(v5 + 4) + _rotr(v7 ^ v6, v7);
                ++v10;
                v5 += 8;
            }
            while ( v10 <= 12 );
            result = v9;
            *(_DWORD *)v9 = v4;
            *(_DWORD *)(v9 + 4) = v6;
            ++v8;
            v9 += 8;
        }
    }
    return result;
}

Above is a reverse engineered set of functions which provides the encryption of a string (password) for authentication of a game client to a game server. The original encryption is no longer available and it was reverse engineered for this purpose.

The code runs perfectly in 32bit, however, when compiling for 64bit systems I run into some problems. The code builds successfully as it does in 32bit, however upon execution of the function it segfaults (EXC_BAD_ACCESS) when it hits the first v4 assignment line:

v4 = *(_DWORD *)v9 + *(_DWORD *)(a1 + 16);

I have attempted to understand the function and figure out what sort of encryption its actually providing however I’ve failed at that miserably. I was running with RC5 at first, but that led to about 2 days of math that ultimately proved extremely valuable to my cryptography skills but futile to the problem at hand.

I’d like to know one or both things about this code:

First and easiest, does anyone recognize the algorithm or cryptographic method that this is using? Is it even using a standard method? It looks like RC5/ROT13/DES/AES in some way shape or form to me, but I can’t tell.

Second, is there any indication as to why this would fail to work in 64bit?

** Sorry, _DWORD is an unsigned long.

Answer 1

Try replacing all the occurrences of int and _DWORD with int32_t (you’ll want to #include <stdint.h>. You said _DWORD is defined as unsigned long, which on many 32-bit machines will be a 32-bit quantity but on 64-bit machines will often be 64.

Or if the function is actually called with addresses as its arguments, which it looks like it might be, you may want to try using int64_t everywhere. Do you have an example of how the function is invoked, and some example inputs on a 64-bit system?