Question

After weeks of effort I have managed to write F# programs that use LLVM for JIT compilation. However, whenever I run my programs in Visual Studio 2010 with the debugger attached (i.e. by pressing F5) I get the following warning:

Now, I get this warning for every single PInvoke call when using my Windows 7 netbook but I only get it for some calls when using my Windows Vista desktop.

Other people hitting this problem seem to have solved it by adding attributes to the PInvoke calls requesting ANSI strings or the CDecl calling convention. I found that changing the calling convention fixes the warnings on my Windows Vista desktop but none of the available calling conventions (or ANSI format strings) fix the warnings on my Windows 7 netbook. Any ideas how to fix this?

Note that both machines are entirely 32-bit x86.

EDIT

People are posting comments asking for repros. The simplest way to reproduce this problem is to install LLVM and llvm-fs following the instructions I documented here and run any of the example programs given. They all exhibit this issue on all calls to LLVM on my netbook.

Alternatively, the following code (derived from llvm-fs) should repro the issue without requiring llvm-fs:

open System.Runtime.InteropServices

[<DllImport("LLVM-3.0.dll",
            EntryPoint="LLVMModuleCreateWithName",
            CharSet=CharSet.Ansi,
            CallingConvention=CallingConvention.Cdecl)>]
extern void *moduleCreateWithNameNative(string ModuleID)

let mdl = moduleCreateWithNameNative "foo"

Note that the corresponding definitions in the original C header file are:

typedef struct LLVMOpaqueModule *LLVMModuleRef;
...
LLVMModuleRef LLVMModuleCreateWithName(const char *ModuleID);

Answer 1

Are you targeting .NET 4.0 or an earlier version?

The reason I ask is there’s a security/stability feature for the CLR which performs extra-strict checking of Pinvoke signatures; it’s been around since .NET 2.0, but was off by default until .NET 4.0.

The switch in behavior has resulted in a number of developers reporting the same issue as you are; their bindings worked just fine on .NET 2.0/3.5, but started throwing errors when compiled for .NET 4.0. In reality, the problem is that prior versions of .NET allowed slightly-buggy PInvoke signatures to work without issue; now that the strict checking is on by default, the bugs are starting to show up.

Another thing to note is that even if you change the configuration on your machine to disable this behavior in .NET 4.0, Visual Studio will still always use it when debugging a project. Worse yet, the strict checking is only on by default in the x86 version of .NET 4.0, not the x64 version, so an assembly which works just fine on a 64-bit machine may possibly crash on a 32-bit machine.

MSDN has more information on the pInvokeStackImbalance MDA and this blog post also provides more detail about why the problem crops up during the debugging process.

EDIT: I just noticed you edited your question to include a code example. This sort of confirms my suspicion about the PInvoke signature being slightly wrong. What happens if you change the signature from extern void *moduleCreateWithNameNative(string ModuleID) to extern LLVMModuleRef* moduleCreateWithNameNative(string ModuleID)?

It also looks like there’s a compiler bug at work here — F# shouldn’t allow you to define a method called *moduleCreateWithNameNative. I’d guess it is allowing it (for whatever reason), so the return type of your function is compiled as void — and when the native method tries to return a value (a pointer to an LLVMModuleRef struct) the CLR gets tripped up and crashes.