Question

I have a few quick questions about the Microsoft Detours Library. I have used it before (successfully), but I just had a thought about this function:

LONG DetourUpdateThread(HANDLE hThread);

I read elsewhere that this function will actually suspend the thread until the transaction completes. This seems odd since most sample code calls:

DetourUpdateThread(GetCurrentThread());

Anyway, apparently this function “enlists” threads so that, when the transaction commits (and the detours are made), their instruction pointers are modified if they lie “within the rewritten code in either the target function or the trampoline function.”

My questions are:

When the transaction commits, is the current thread’s instruction pointer going to be within the DetourTransactionCommit function? If so, why should we bother enlisting it to be updated?

Also, if the enlisted threads are suspended, how can the current thread continue executing (given that most sample code calls DetourUpdateThread(GetCurrentThread());)?

Finally, could you suspend all threads for the current process, avoiding race conditions (considering that threads could be getting created and destroyed at any time)? Perhaps this is done when the transaction begins? This would allow us to enumerate threads more safely (as it seems less likely that new threads could be created), although what about CreateRemoteThread()?

Thanks,

Paul

For reference, here is an extract from the simple sample:

// DllMain function attaches and detaches the TimedSleep detour to the
// Sleep target function.  The Sleep target function is referred to
// through the TrueSleep target pointer.
BOOL WINAPI DllMain(HINSTANCE hinst, DWORD dwReason, LPVOID reserved)
{
    if (dwReason == DLL_PROCESS_ATTACH) {
        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        DetourAttach(&(PVOID&)TrueSleep, TimedSleep);
        DetourTransactionCommit();
    }
    else if (dwReason == DLL_PROCESS_DETACH) {
        DetourTransactionBegin();
        DetourUpdateThread(GetCurrentThread());
        DetourDetach(&(PVOID&)TrueSleep, TimedSleep);
        DetourTransactionCommit();
    }
    return TRUE;
}

Answer 1

How embaressing:
I forgot that the source was available!

DetourUpdateThread silently ignores the enlisting of the current thread. Otherwise, the given thread is suspended. I wonder why ALL code examples enlist the current thread anyway! This answers the first 2 questions.

As for the 3rd question:
I found another detouring library that attempts to suspend all threads by doing the following:

1. Get snapshot of all threads

2. Loop through the snapshot and suspend threads that we have not already suspended.

3. If threads were suspended, then go back to 1 (we still keep track of threads that we have suspended). If no threads were suspended then we are done.

I think the assumption is that if we can loop through all threads and they are all already suspended (i.e. from before we took the snapshot), then no more threads can have been created. Not so sure about CreateRemoteThread though!

Edit: Re: CreateRemoteThread.

“Only one thread in a process can be in a DLL initialization or detach routine at a time.”
CreateRemoteThread “results in a call to the entry point of each DLL in the process”.
http://msdn.microsoft.com/en-us/library/ms682437%28VS.85%29.aspx

A new thread cannot start executing if you are in a DllMain function (as long as the new thread has not yet caused the calling the entry point of each DLL in the process).
So if you apply your detours within a DllMain function, you may just be safe from the race condition of a new remote thread being created and having its instruction pointer within your rewritten target/trampoline function.

Thanks,

Paul