Question

I’m trying to call a Dll function which looks like this (in the C/C++ Dll):

__declspec(dllexport) void execute(char** ReturnMsg, char* serverAddress, char* commandLine)

The VB ‘wrapper’ function looks like:

<DllImport("TPClient.dll", EntryPoint:="execute", CallingConvention:=CallingConvention.Cdecl, CharSet:=CharSet.Auto, ExactSpelling:=True)> _
Public Shared Sub tg_execute(<Out()> <MarshalAs(UnmanagedType.LPStr)> ByRef returnString As System.Text.StringBuilder, _
                             <MarshalAs(UnmanagedType.LPStr)> ByVal serverAddress As String, _
                             <MarshalAs(UnmanagedType.LPStr)> ByVal commandLine As String)
End Sub

The parameters are:
returnString: a string I need to get back from the function, result of the command sent;
serverAddress: a string, input only (an IP or DNS name); and
commandLine: a string, input only (any command)

To call the function, I make a StringBuilder object with some sufficient capacity to use as the returnString variable:

Dim returnString As New System.Text.StringBuilder(128)

tg_execute(returnString, TextBox_serverName.Text.Trim, TextBox_Command.Text.Trim)

When I run the code, I do get the expected string in the returnString (as I can see in the debugger), however I also get an AccessViolationException. So, I get for example “2.6.30.8-x86” in returnString when I use the command “uname -r” in commandLine. But the code hangs due to the memory error.

Now I’m not too familiar with VB and P/Invoke, and I had to do some trial and error to get the arguments passed to the DLL (which I’m also writing and debugging). This is also how I ended up using the “MarshalAs(UnmanagedType.LPStr)” attributes. However now I don’t know why I’m getting these memory errors.

I made some other attempts using IntPtr arguments, but I also couldn’t get this working and gave up on that approach, as to my understanding the marshaling should be handled automatically (is that correct?).

Any help is much appreciated.

Answer 1

The return value char** ReturnMsg would suggest that ReturnMsg is a pointer to C string. This would imply that the native code was in charge of allocating the buffer. So StringBuilder is not appropriate here.

There is not actually enough information here to know how to call this function. What is missing is knowledge of which party is responsible for deallocating the string. It could be either party and I’m going to assume that the C code will do so, probably by means of the strings being statically allocated, e.g. constants.

Now, I have no experience with VB p/invoke so I hope you don’t mind if I give you a C# version. I expect you can translate easily enough.

[DllImport("TPClient.dll", CallingConvention=CallingConvention.Cdecl,
    CharSet=CharSet.Ansi, EntryPoint="execute", ExactSpelling=true)]
private static void tg_execute(out IntPtr returnString, 
    string serverAddress, string commandLine)

You call the function like this:

IntPtr returnStringPtr;
tg_execute(out returnStringPtr, serverAddress, commandLine);
string returnString = Marshal.PtrToStringAnsi(returnStringPtr);

Note that your character set was incorrect in the question. You have to use Ansi because the native code uses char. I also think that your MarshalAs attributes are spurious since you are just re-stating the default marshalling for those parameter types.

Now, if the native code expects the caller to deallocate the memory, then the native code would have to export a function to do so. If that’s the case then you would call it passing returnStringPtr by value.