Question

I’m attempting to call a method on the ssdeep fuzzy.dll

The .h file is here and a friendly reference is here

Specifically, I’m trying to call this method….

int fuzzy_hash_filename (
        const char * filename,
        char * result 
)   

I’ve got the following…

<DllImport("C:\SSDeep\Fuzzy.dll", EntryPoint:="fuzzy_hash_filename")>
Private Shared Function fuzzy_hash_filename(
                                           <InAttribute(),
                                           MarshalAsAttribute(UnmanagedType.LPStr)>
                                           ByVal Filename As String, ByVal Result As StringBuilder) As Integer
End Function


Public Shared Function FuzzyHash(Filename As String) As String
    Dim Ret As New StringBuilder
    Ret.Capacity = NativeConstants.FUZZY_MAX_RESULT
    Dim Success = fuzzy_hash_filename(Filename, Ret)
    If Success <> 0 Then
        Throw New Exception("SSDeep fuzzy hashing failed")
    End If
    Return Ret.ToString
End Function

If I run this code, VS gives me a modal dialogue

A call to PInvoke function ‘(Blah)::fuzzy_hash_filename’ has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.

(FWIW The call seems to succeed if I ignore the warning so I must be close)

What change do I need to make to my definition to get this going?

Answer 1

I found someone that had the same issue on MSDN forums:

1. Concerning the PInvokeStackImbalance.

1.1 This is usually due to mismatch of the calling convention used by the API and that declared for the API in the C# code.

1.2 By default, if the CallingConvention argument for the DllImportAttribute is not set, then StdCall is used by default.

1.3 If the DoSomething() API is to use __cdecl (as is the default in C++ projects), then you should use the following declaration for
DoSomething() in the C# code : [DllImport(@”dll.dll”,
CallingConvention=CallingConvention.Cdecl)]

1.4 Also, I suggest that you declare the API as extern “C” otherwise it will be subject to name mangling by the C++ compiler.