Question

I have a C++ program that crashes on an access violation when I run it in the Debugger with Visual Studio. The problem is that in the past it normally would take me to a line where this happens, but for this problem it says there is no source available and takes me to the disassembly. I have no idea how to decipher it and get any useful information. It takes me to the following line:

0000000057A93F0F  cmp         dword ptr [rcx+11BCh],0  

Then if I run it again it will stop at:

0000000058673F0F  cmp         dword ptr [rcx+11BCh],0  

Then if I run it again it stops at the first one and then again the second one if I re-run it… it just keeps going in this loop. I find it strange that this happens at different addresses alternating times but then again I don’t even know if that is for sure the address.

How can I use this information to find the problem?

(I would post code but I have no real idea what section is causing the problem so don’t know what to post… I should not that I am trying to compile flash-to-directx with an x64 platform).

Thanks

EDIT

Here are the lines before the crash happens:

0000000057A93EF3  test        dl,1  
0000000057A93EF6  je          0000000057A93EFD  
0000000057A93EF8  call        0000000057FC8024  
0000000057A93EFD  mov         rax,rbx  
0000000057A93F00  add         rsp,20h  
0000000057A93F04  pop         rbx  
0000000057A93F05  ret  
0000000057A93F06  int         3  
0000000057A93F07  int         3  
0000000057A93F08  sub         rsp,28h  
0000000057A93F0C  mov         ecx,r8d  

and then the crash happens at:

0000000057A93F0F  cmp         dword ptr [rcx+11BCh],0  

Also the last item on my call stack is:

Flash64_11_1_102.ocx!0000000058673f0f()

Answer 1

Classic 64-bit pointer truncation in Adobe Flash dll. Probably some function receives DWORD userData instead of void* userData through some structure and then casts it to object pointer. This works ok in 32-bit environment, but crashes in 64-bit.

Explanation

mov         ecx,r8d

First operation copies only low 32-bits from R8D to ECX (ECX is 32-bit).

cmp         dword ptr [rcx+11BCh],0

Second operation accesses 64-bit register, where low 32-bits contains correct address and high 32-bits contains some junk. Leading to crash, of course.

How to fix

This is pretty easy, if you don’t mind editing dll using hex editor. You will need an old school MASM.exe (you can get one from Microsoft web-site).

Create .asm file, or get one from examples and modify, with code like this:

nop
nop
nop

mov ecx,r8d
cmp dword ptr [rcx+11BCh],0

nop
nop
nop

mov rcx,r8d   // I've replaced ecx with rcx here
cmp dword ptr [rcx+11BCh],0

Create .obj file by submitting this to MASM. Open resulting file with any hex editor and notice sequence 90 90 90 first_sequence_of_bytes 90 90 90 second_sequence_of_bytes. All you need to do is to find in original dll first sequence of bytes and replace it with second sequence of bytes. I’m pretty sure difference will be only in first byte.

This will fix your crash.