I have the following MyType::Is_Inst () function which is throwing an invalid memory access

Question

0

Asked: May 22, 20262026-05-22T23:14:47+00:00 2026-05-22T23:14:47+00:00

I have the following MyType::Is_Inst () function which is throwing an invalid memory access

0

I have the following MyType::Is_Inst () function which is throwing an invalid memory access error on return in 64-bit mode but not in 32-bit:

MyType MyType::Is_Inst () {
    unsigned char Bar=0;    
    MyType Foo={0};    
    return Foo;
    }

Looking at the disassembly + step-through, the program crashes at the line

mov  dword ptr [rax],ecx

…when the program is basically trying to dereference the original value of %rdx (from when the function was first called), which is now in %rax. However, %rdx is just junk left over from the previous function call.

The last time I had an issue like this, it was because I was missing some compilation flags or the like. Are there any settings I should be aware of for x64 unmanaged c++ projects? Are there any other reasons I might see this behavior?

I can post more of the disassembly if you need it.

The class definition for MyType looks like this:

class __declspec(dllexport) MyType {
public:
    union {
        struct {
            unsigned int    Id     : 23;
            unsigned int    Flag   : 1;  
            unsigned int    Type   : 4;  
            unsigned int    Unused : 4;  /* 32 bits total */
            };

        unsigned int    All_Bits;          /* Full 32 bits of MyType */
        };

     /* There are some function definitions here, but no other 
        variables, aside from some statically defined ones. */
     };

UPDATE: This is the more succinct, optimized version of Is_Inst()’s disassembly, which shows the problem. I’ve removed the old version that was here before for brevity.

// MyType MyType::Is_Inst () {
// uchar Bar=0;
// MyType Foo={0};
   mov         dword ptr [rdx],0 /* %rdx is 0x17, from a prev fn call. */ 

// return Foo;
   mov         rax,rdx  
// }
   ret

The code leading up to Is_Inst() getting called:

...
for (Counter=0; Counter<N_Items; Counter++) {
    ...
    myOther32BitType = arrayOfMyOtherTypes [Counter]; /* Debugger shows this is ok. */ 

    if (myOther32BitType.8BitField==UNEQUAL_ENUM_VALUE) {
        /* Some stuff that doesn't happen. */
        }

    /* myOther32BitType.8BitField==0x17, so %rdx gets set to 0x17. */
    else if (strchr((char*)Uchar_Table_Of_Enum_Values, myOther32BitType.8BitField)) continue;

    /* %rdx gets set to 0x17 again. */
    else if (strchr((char*)Other_Uchar_Table_Of_Enum_Values,  myOther32BitType.8BitField)) continue;

    else if ( myOther32BitType.8BitField==EQUAL_ENUM_VALUE) {
        if (myType.Is_Inst ().All_Bits) { /* Is_Inst() called here. */
            return false;
            }
        ...
        }
    ...
    }