Question

Alright so I have this code, and I pass it to an unmanaged dll, to which I only know the exports, and have some sample code. I’m getting back the correct string, but it’s followed by garbage bytes.

I’m basically translating code verbatim from a c++ example program that doesn’t have this issue. I’m assume there is something fundamental I am missing here, so if anyone could tell me what that is, I’d appreciate it.

Example C++ Code

void CDUKPT_TESTDlg::OnButton4() 
{
    // TODO: Add your control notification handler code here
    unsigned char dataout[1024],tmp[1024],ksn[20],keyval[20];
    int nRet,len;
    memset(dataout,0,sizeof(dataout));
    memset(ksn,0,sizeof(ksn));
    memset(keyval,0,sizeof(keyval));
    memset(tmp,0,sizeof(tmp));
    UpdateData(TRUE);

    two_one((unsigned char *)m_strCURKSN.GetBuffer(m_strCURKSN.GetLength()),m_strCURKSN.GetLength(),ksn);
    two_one((unsigned char *)m_strMACK.GetBuffer(m_strMACK.GetLength()),m_strMACK.GetLength(),keyval);
    two_one((unsigned char *)m_EncryptDat.GetBuffer(m_EncryptDat.GetLength()),m_EncryptDat.GetLength(),dataout);

    len=m_EncryptDat.GetLength()/2;
    //extern int __stdcall ExtractDat(unsigned char *input,
    //unsigned short len,unsigned char *output,unsigned char *key,
    //unsigned char *ksn);
    nRet=ExtractDat(dataout,len,tmp,keyval,ksn); //External Call
    //Good string+bad trailing data comes back in tmp
    m_Result=tmp;
    UpdateData(FALSE);
}

This code spits out this ܉Òdÿo 

Here is my VB.Net Code

Public Function Encrypt(ByVal inp As String) As String
    Dim tmpSB As New StringBuilder
    Dim i As Integer
    Dim tKsn As Char() = TwoOne(StrCurKsn)
    For i = tKsn.Length To 19
        tKsn = tKsn + Chr(0)
    Next
    Dim tMack As Char() = TwoOne(StrMack)
    For i = tMack.Length To 19
        tMack = tMack + Chr(0)
    Next
    Dim tEnc As Char() = TwoOne(inp)
    For i = tEnc.Length To 1023
        tEnc = tEnc + Chr(0)
    Next
    Dim len As Integer = tEnc.Length / 2

    Dim tmpStr(1023) As Char
    Array.Clear(tmpStr, 0, 1024)
    Dim tmpPtr = Marshal.StringToHGlobalAnsi(tmpStr)

    Dim nRet = ExtractDat(tEnc, len, tmpPtr, tMack, tKsn)

    tmpStr = Marshal.PtrToStringAnsi(tmpPtr)
    Dim tsl = tmpStr.Length
    Encrypt = tmpStr
End Function

This code spits this out

܉Òdÿo ålUäÙålUäÙålUäÙålUäÙålUäÙålUäÙålUäÙ

So I get the right string, but it’s followed by a repeating string of garbage characters.
I’m hoping I’ve done something blatantly wrong here, but I’ve tried pulling the data as bytes, and chars, and converting in many different methods, and I can’t seem to get rid of those characters…Also, ExtractDat doesn’t return the length of the string(not a problem, as it’s not supposed to, which is really annoying).

Answer 1

Turns out the dll was bad, so after I got a fresh compile from the vendor it seemed to work.