Question

I’m currently exploring DLL exported functions and P/invoke in C#.
I’ve created very simple .dll:

Test.h

#ifndef TEST_DLL_H
#define TEST_DLL_H
extern "C" __declspec(dllexport) const char * __cdecl hello ();
extern "C" __declspec(dllexport) const char * __cdecl test ();    
#endif  // TEST_DLL_H

Test.cpp

#include <stdlib.h>
#include "test.h"
#include <string.h>

const char* hello()
{
    char *novi = (char *)malloc(51);
    strcpy(novi, "Test.");

    return novi;
}

const char * test()
{
    return "Test.";
}

I’ve compiled it and used in C# project like this:

    [DllImport("test.dll", CallingConvention = CallingConvention.Cdecl)]
    public static extern IntPtr hello();

    [DllImport("test.dll", CallingConvention = CallingConvention.Cdecl)]
    public static extern string test();

    private void button1_Click(object sender, EventArgs e)
    {
        MessageBox.Show(test());
        IntPtr a =  hello();
        MessageBox.Show(Marshal.PtrToStringAnsi(a));
    }

But it isn’t working. test() gets called successfully and I get back correct string. But hello() just hangs up the program. If I remove malloc line from the hello() definition and return constant, everything works, so I guess there’s a problem with malloc that I’m now aware of.

Also, somewhere I’ve seen that string shouldn’t be used when return type is char*. If that’s true, why should we use IntPtr?

Answer 1

Functions that return strings across a DLL boundary are very difficult to call reliably from C or C++, it doesn’t get any better when you do it from C#. At issue is how the caller is going to release the string buffer. That needs to be done for hello() but not for test(). Something that’s very hard to guess at. The hello() function requires using the free() function, using the exact same allocator that was used to call malloc(). That can only work when the DLL and the caller share the same CRT implementation. The odds for which are slim.

The pinvoke marshaller also releases the string buffer, it has to. And does so with the only reasonable choice, CoTaskMemFree(). Which uses the default allocator used by COM. This does not come to a good end, your C code didn’t use CoTaskMemAlloc(). The possible outcome of this depends on the operating system. On Vista and up, your program will die with an AccessViolation, those Windows versions use a strict heap allocator designed to crash misbehaving programs. On XP, you’ll get something between a memory leak and heap corruption, sounds like you hit the second option.

Declaring the return value as IntPtr will come to a good end. Well, your program won’t crash, you still have a memory leak that you cannot plug. There is no way to call free() reliably. Or use CoTaskMemAlloc() in your C code so that the pinvoke marshaller’s release call will work.

But realistically, just don’t write C code like this. Always use memory allocated by the caller so there is never a guess who owns the memory. Which requires a function signature similar to this:

extern "C" __declspec(dllexport) 
int hello(char* buffer, int bufferSize);