I have a library I am trying to use with a Dot Net 4 App in C#. This is a binaray runtime so I dont have access to the library source. I am getting the error
A call to PInvoke function ‘XXXXXXXXXXX.WinAPI::GetDCEx’ has
unbalanced the stack. This is likely because the managed PInvoke
signature does not match the unmanaged target signature. Check that
the calling convention and parameters of the PInvoke signature match
the target unmanaged signature.
If I disable the PInvokeStackImbalance MDA the application runs because it is not longer thrown by the debugger. Whilst this gets around the issue of the application not running there is obviously an issue under the hood.
I recently read about NetFx40_PInvokeStackResilience an element you can place in the app.config which tells the runtime to use a marshaling layer to detect and fix incorrect platform invoke declarations (like in previous .Net versions).
<configuration>
<runtime>
<NetFx40_PInvokeStackResilience enabled="1" />
</runtime>
.....
</configuration>
The problem is this doesn’t seem to do anything regarding the debugger, the debugger still invokes the error.
My questions are should I still have to disable the PInvokeStackImbalance MDA?
And how can I tell that NetFx40_PInvokeStackResilience has worked and the marshaling layer is doing its job?
PS I am using Visual Studio 2010 Express
PPS I have written to the owner of the library with a possible solutions amending his DllImports etc. Although the marshaling layer should get around the problem there may be a cost to performance. Plus its confusing and not ideal. However, in the meantime I still need a solution until the library gets updated.
An imbalanced stack is a nasty kind of runtime problem. You tend to get away with it in the Debug build because the way a method sets up and tears down the stack frame can hide the problem with the incorrect stack pointer value. The stack pointer is restored at method exit. You’ll have fewer odds for that kind of luck in the Release build. Particularly due to method inlining, it removes the stack setup/teardown code.
The NetFx40_PInvokeStackResilience element was needed in .NET 4 because they optimized the pinvoke marshaller. Which can break bad code that worked in earlier releases, adding the element disables that optimization. It has a global effect on the entire program. It was not meant to disable the MDA, that would be hiding a very real problem. It merely gives IT staff a way to work around a compatibility problem.
You can lower the risk by putting the pinvoke call into a separate method that does nothing but make the call. You must decorate it with [MethodImpl(MethodImplOptions.NoInlining)] so the optimizer can’t inline it. That helper method cannot have any out or ref arguments and must not return a value of a struct type and should not have any additional code after the call.
Clearly you’ll want to invest as much time as possible to really fix this problem. There’s somebody out there that can help you with the original C declaration, we can’t help you find him however. Not posting the pinvoke declaration also removes any odds of somebody having a guess at it.