Question

I’m developing an IM provider for Outlook 2010. In order to do that, I have to implement some IMessenger* COM interfaces that outlook loads on startup. I want to do this using C# and an out-of-proc server for which I followed the MSDN example here.

This is what I have already achieved:

• Register everything (CLSID, type libraries, etc.) correctly and configure Outlook so that it tries to load my server upon startup
• Implement the COM server as a native ATL/C++ server and make sure it works
• Implement the basic interface in C#/.NET and made sure it is loaded

Right now, I can basically start my .EXE server, then start outlook and watch the console windows of my server while it displays all the methods calls that come from the COM subystem (I’ve added logging to every method). The problem is, that at some point (only in the .NET implementation!) I hit an error that looks like this in the Outlook logs:

CMsoIMProviderOC20::HrEnsureServiceData !failed!  Line: 402  hr = 0x8000FFFF

I know exactly where this happens in my program, but I can’t do anything about that (I’ve tried to find a solution for this for days). Please remember, that if I implement it using C++/ATL it actually works, so I guess it must have something to do with the way .NET Interop marshaling works.

Here are the details:

There are basically 3 interfaces related to this issue: IMessenger, IMessengerServices and IMessengerService:

[
    uuid(D50C3186-0F89-48f8-B204-3604629DEE10), // IID_IMessenger
    helpstring("Messenger Interface"),
    helpcontext(0x0000),
    dual,
    oleautomation
]
interface IMessenger : IDispatch
{
    ...
    [id(DISPID_MUAM_SERVICES), propget, helpstring("Returns services list."), helpcontext(0x0000)]
    HRESULT Services([out, retval] IDispatch ** ppdispServices);
    ...
}

Whereas I know that this Services property should actually return this inteface:

[
 uuid(2E50547B-A8AA-4f60-B57E-1F414711007B), // IID_IMessengerServices
 helpstring("Messenger Services Interface"),
 helpcontext(0x0000),
 dual,
 oleautomation
]
interface IMessengerServices : IDispatch
{
    ...
    [id(DISPID_NEWENUM), propget, restricted, helpstring("Enumerates the services."), helpcontext(0x0000)]
    HRESULT _NewEnum([out, retval] IUnknown **ppUnknown);
    ...
}

This interface in turn is expected to return IMesengerService interfaces. However, this one not important for the purpose of this question.

In C++/ATL I implement the property Services like this:

STDMETHOD(get_Services)(IDispatch ** ppdispServices)
{
    (*ppdispServices) = (IDispatch*)new CComObject<CMessengerServices>();
    (*ppdispServices)->AddRef();
    return S_OK;
}

And here is the C# implementation:

public object Services
{
    get { return new MessengerServices(); }
}

Up until here everything is okay in both implementations! Now the problem begins…

First thing that is strange is that in C++/ATL the function get__NewEnum(IUnknown **pUnkown) of my CMessengerServices class will never be executed. Instead other functions are called. This is good.

In C# however, the first member of my MessengerServices class that is called is the GetEnumerator() method. I can set a breakpoint there and clearly see that it is the last line in the C# code (under my control) which is executed before Outlook stops the startup and reports the 0x8000FFFF error in its log. After than, no other line of code in my .EXE server is called.

There are the most important parts of the MessengerServices class implementation:

[ClassInterface(ClassInterfaceType.None)]
[ComVisible(true), Guid("DF394E2C-38E2-4B70-B707-9749A4F857B0")]
public class MessengerServices : IMessengerServices
{
    IEnumerator IMessengerServices.GetEnumerator()
    {
        return messengerServices.GetEnumerator();
    }

    private readonly ArrayList messengerServices;

    public MessengerServices()
    {
        messengerServices = new ArrayList { new MessengerService() };
    }
    ...
}

And here the CCW proxy generated by tlbimp.exe:

[ComVisible(true), Guid("2E50547B-A8AA-4F60-B57E-1F414711007B")]
public interface IMessengerServices : IEnumerable
{
    [TypeLibFunc(TypeLibFuncFlags.FRestricted)]
    [DispId(-4)]
    [MethodImpl(MethodImplOptions.InternalCall, MethodCodeType = MethodCodeType.Runtime)]
    [return: MarshalAs(UnmanagedType.CustomMarshaler, MarshalType = "System.Runtime.InteropServices.CustomMarshalers.EnumeratorToEnumVariantMarshaler, CustomMarshalers, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")]
    new IEnumerator GetEnumerator();
    ...
}

One last hint if that maybe helps: When I set a breakpoint inside the GetEnumerator() implementation and the hit F10 to step-over, this is the last line in the Output window of Visual Studio:

Stepping over non-user code ‘System.Runtime.InteropServices.CustomMarshalers.EnumeratorToEnumVariantMarshaler.MarshalManagedToNative

Another interesting thing (at least for me) is that if I completely remove the Enumerator thing from the Interface declaration and my class definition, Outlook still calls my interface, but uses another method (PrimaryService for the sake of completeness) instead but then basically has the same error.

To be honest, I have absolutely no idea what to do about this error? Thanks in advance for any kind of help!

Answer 1

I found a solution for the problem now. Hans Passants answer led me into the right direction (thank you!), however, the problem was that the following sentence is true:

Tlbimp.exe does not preserve the order of the methods in the vtable like they were originally defined in the IDL/TLB. Instead, the methods and properties are re-sorted alphatetically!

So, when you implement an interface and reference the type-library using Visual Studio, chances are that the vtable that is created by the CCW will be screwed up. The solution is to copy the Tlbimp (the tool used by Visual Studio generate the interop assembly) generated wrapper into your own code and re-order the methods and properties to be identical with the order in the type-library.