Question

I have a C++ DLL that is being called like below from a C# console app.

When run in Visual Studio to debug it, it throws an exception saying the stack is unstable and to check that the method arguments are correct. However, if I run the *.exe outside of VS from Windows Explorer it retuns data to the screen as expected.

How can I get this to run within Visual Studio?

Thanks

**From the C++ header file:**
#ifdef RFIDSVRCONNECT_EXPORTS
#define RFID_CONN_API __declspec(dllexport)
#else
#define RFID_CONN_API __declspec(dllimport)
#endif

RFID_CONN_API BSTR rscListDevices( long showall ) ;


[DllImport("MyDLL.dll")]
[return: MarshalAs(UnmanagedType.BStr)]
public static extern string rscListDevices(int showall);

static void Main(string[] args)
{
  string data= rscListDevices(0);
  Console.WriteLine(data);
  Console.ReadKey();
}

Answer 1

Firstly, make sure you’re using the same calling convention in both C++ and C#.

I suspect that the /Gd compiler option is set (since it is set by default), so __cdecl is used as default calling convention for unmarked functions.

You can fix crashes by either specifing the same calling convention in your C# code:

[DllImport("MyDLL.dll", CallingConvention=CallingConvention.Cdecl))]
[return: MarshalAs(UnmanagedType.BStr)]
public static extern string rscListDevices(int showall);

Or changing rscListDevices‘s calling convention to __stdcall (which is the default in C#):

RFID_CONN_API BSTR __stdcall rscListDevices( long showall ) ;

You can also set __stdcall as the default calling convention for the unmarked functions in your C++ DLL by changing compiler option from /Gd to /Gz in manually or using the Project Properties dialog:

But if you really want to disable MDA, you can go Debug->Exceptions and uncheck Managed Debugging Assistance.

You can read more about pInvokeStackImbalance and MDA here and here.