Question

One of our largest old VB6 apps has some code in it to allow other apps (including some dotNET ones) to pass an ID to it via a Windows message – this ID is then used by the VB6 app to load an entry in a regular Windows form. The message hook is added after the user is logged in and authenticated, and removed once they logout.

Public Sub HookClaimFinderCall()
    lpPrevWndProc = SetWindowLong(gHW, GWL_WNDPROC, AddressOf WindowProc)
End Sub

Public Sub UnhookClaimFinderCall()
    Dim temp As Long
    If gHW <> 0 Then temp = SetWindowLong(gHW, GWL_WNDPROC, lpPrevWndProc)
End Sub

Private Function WindowProc(ByVal hw As Long, ByVal uMsg As Long, ByVal wParam As Long, ByVal lParam As Long) As Long
    If uMsg = WM_FINDCLAIM Then
        MasterFindClaim lParam
    End If
    WindowProc = CallWindowProc(lpPrevWndProc, hw, uMsg, wParam, lParam)
End Function

However, there are two issues with this. The first relates to Visual Studio 6. If the code is being debugged, and an error occurs to bring up the “Continue End Debug Help” dialog box, pressing End quits Visual Studio instantly (losing any unsaved changes). This does not happen if the message hook has not been activated yet. What causes this, and is there anything I can do to stop it short of commenting out the code that loads the hook?

Secondly, if the users quit the app without logging out properly (by whatever means), what happens to the message hook?

I hope I’ve got all the terms right in the above…

Answer 1

1. You are going to get crashing (or disappearing VB6) if you initiate a global hook and attempt to Debug in VB6. VB6 is kind of like a simulator, it doesn’t exactly duplicate the runtime of a VB6 application and hooking is one of the areas it fails miserably at (though it can’t really be blamed if you understand what is going on). For all of the global hooks we use in our applications we check to see if VB6 is running in IDE mode (there are several ways to do this) and if it is, don’t run the global hook. If you absolutely have to run the global hook, do not stop the application in the debugger – use debug.print or some other means, but don’t stop the application otherwise you will have milliseconds to seconds before it “goes away”.
2. Although you should unhook before exiting the application, when the message pump hits the hook and the handle of the application where the hook takes place no longer exists, it is a fairly cheep operation to ignore that hook. Now if you ran the application and exited thousands of times, it would probably build up, but that I think is the least of your concerns.