Question

I need the simplest way from C using Win32 to get the process handle of another process by its executable file name.

The process I am looking for does not have any registered window classes. I also know that if it is running there will be only one instance of it running.

Answer 1

Use CreateToolhelp32Snapshot, Process32First, and Process32Next to enumerate all of the processes.

Inside the PROCESSENTRY32 you can find a szExeFile member.
You can get the process handle by calling OpenProcess with the process ID th32ProcessID within the same struct.

Once you find a process matching your exe name, you can break out of your loop and obtain the handle.

Note: If you need to enumerate EVERY process no matter what the session is, you should acquire the SE_DEBUG privilege.

At the top of your main call this:

acquirePrivilegeByName(SE_DEBUG_NAME);// SeDebugPrivilege

And here is the definition of acquirePrivilegeByName:

BOOL acquirePrivilegeByName(
                            const TCHAR     *szPrivilegeName)
{
    HANDLE          htoken;
    TOKEN_PRIVILEGES    tkp;
    DWORD           dwerr;

    if (szPrivilegeName == NULL)
    {
        SetLastError(ERROR_INVALID_PARAMETER);
        return FALSE;
    }

    if (!LookupPrivilegeValue(NULL, szPrivilegeName, &(tkp.Privileges[0].Luid)))
        return FALSE;

    tkp.PrivilegeCount = 1;
    tkp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;

    if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES, &htoken))
        return FALSE;

    if (!AdjustTokenPrivileges(htoken, FALSE, &tkp, 0, NULL, NULL) ||
        GetLastError() != ERROR_SUCCESS)    // may equal ERROR_NOT_ALL_ASSIGNED
    {
        dwerr = GetLastError();
        CloseHandle(htoken);
        SetLastError(dwerr);
        return FALSE;
    }

    CloseHandle(htoken);
    SetLastError(ERROR_SUCCESS);

    return TRUE;
} //acquirePrivilegeByName()

In addition to what I said above, there is an example on how to use the above Win32 API here.