Question

I’m trying to call functions of a C DLL.

But I got a StackOverflowException so I think something is wrong with the function as parameter.

In detail it looks like this.

C DLL (header file):

typedef struct
{
  MyType aType;     /* message type */
  int nItems;       /* number of items */
   const MyItems    *lpItem;    /* pointer to array of items */
} MyStruct;

typedef void (__stdcall *MyCbFunc) (HWND, const MyStruct  *);

API(BOOL) RegisterCbFunc (ARGS, MyCbFunc);

In C# I tried this:

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct MyStruct
{
  MyType aType;
  int nItems;
  MyItems[] lpItem;
}

[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate void CallbackDelegate(MyStruct mStruct);

[DllImport("MY.dll", CallingConvention=CallingConvention.StdCall)]
private static extern int RegisterCbFunc(IntPtr hWnd, Delegate pCB);

public static void MyCbFunc(MyStruct mStruct)
{
  // do something
}

static void Main(string[] args)
{
  CallbackDelegate dCbFunc = new CallbackDelegate(MyCbFunc);
  int returnValue = RegisterCbFunc(IntPtr.Zero, dCbFunc);
  // here, returnValue is 1
}

This runs as long until the DLL calls the callback function. Then I got an error:

An unhandled exception of type 'System.StackOverflowException' occurred in Microsoft.VisualStudio.HostingProcess.Utilities.dll

Thanks for help.

ANSWER:
I don’t know why, but there was an answer which is now deleted?!

I try to recover it. The solution was to use call by reference instead of call by value for the function parameter.

public delegate void CallbackDelegate(ref MyStruct mStruct);

Answer 1

Your C function is expecting a pointer to a MyStruct, but you’re telling the C# that it wants a pointer to a function. The difference between a function and a struct is… significant. Perhaps try something like

[DllImport("MY.dll", CallingConvention=CallingConvention.StdCall)]
private static extern int RegisterCbFunc(IntPtr hWnd, Delegate pCB);

static void Main(string[] args)
{
MyStruct mStruct;
int returnValue = RegisterCbFunc(IntPtr.Zero, mStruct);
}

If your C function is filling in the lpItem member of the MyStruct with something it allocates itself, I’ve got no idea what happens next, but at least it’s not going to be trying to overwrite your code.