Question

Does anyone know how to decrypt the following encryption routine? Basically I have an encryption key, I enter it, then I am asked to enter a 6 letter word which then gets encrypted. How shall I decrypt this?
Thanks

encrypt:

      push edx 
      push ecx 
      not eax 
      add eax,0x04 
      mov edx,eax 
      pop eax 
      xor eax,edx 
      pop edx 
      rol al,1 
      rol al,1
      rol al,1 
      sub al,0x02 
      ret 

Answer 1

edit: for the new code, see bottom

This code is weird, but it seems to be doing something like this: (not tested)

char encrypt(char a, int c)
{
    int t = 4 + ~a;        // the NOT and the ADD
    int t2 = (c ^ t) & 0xFF;  // the XOR
    int t3 = ((t2 << 3) | (t2 >> 5)) & 0xFF;  // the three ROL's
    return (char)(t3 - 2);   // the SUB
}

The corresponding decrypt would, I think, look like this: (not tested)

char decrypt(char a, int c)
{
    int t = (a + 2) & 0xFF;
    int t2 = ((t >> 3) | (t << 5)) & 0xFF;
    int t3 = t2 ^ c;
    return (char)~(t3 - 4);
}

Which in assembly could be this: (not tested, and no clutter)

add al, 2
ror al, 3   ; or three times ror al, 1
xor al, cl
sub al, 4
not al
ret

Or you could do it in “mostly 32 bit”: (also not tested)

add eax, 2
ror al, 3
xor eax, ecx
sub eax, 4
not eax
movzx eax, al  ; or just ignore everything but the low byte
ret

Nothing at all was tested, but the general strategy I used is this: figure out what the code is doing, and then step by step think about how to undo these things, starting at the end. If there is a rotate left by 3, make a rotate right by 3. If they add 4, subtract 4. XOR and NOT are their own inverses.

---

Because I got the key and the data mixed up I got it wrong. Actually, it should be this: (also not tested)

; eax = EKey, cl = char
decryptB:
  add ecx, 2   // undo sub 2
  ror cl, 3    // undo rol
  not eax      // actually do not
  add eax, 4   // actually do add 4
  xor eax, ecx // undo xor
  ret

Because the operations done on the key should not be the inverses.