Question

I’m trying to develop an anti-cheating system. The system will work by identifying applications which make use of the various keyboard/mouse entry APIs such as SendInput, keybd_event, mouse_event and SetKeyboardState in Windows. All applications are to be compared against a database of applications which are allowed by the system to make calls to those functions.

To do this, I need to know if there’s any way I can monitor calls to certain WinAPI functions.

I would also need to monitor calls to SetWindowsHookEx.

Answer 1

Both keybd_event and mouse_event use SendInput under the covers. Although it is possible for you to inject a usermode DLL to all processes and hook these APIs, this is not the way that most anti-cheat systems work because it is easily subverted (e.g. through the use of a trampoline).

Typically, anti-cheat programs wishing to monitor/detour these APIs do so further down. For example, they would hook the NtSendUserInput system call. GameGuard is an example of a system that hooks both SendInput and NtSendUserInput.

The first thing you have to realise when coding your anti-cheat system is that you might as well assume that your code can easily be subverted or manipulated by an attacker. You speak of comparing against a database of applications. Perhaps you intend to do so by file path, which you get through GetProcessImageFileName. Then you need to think about how you are going to stop an attacker injecting a DLL to hook your call to that API or maybe even load a driver that hooks further down.

The back and forth game of cheating vs anti-cheating is endless and constantly changing (updating code signatures, cheaters recompile with different heuristics, you hook usermode, they trampoline, you hook kernelmode, they load driver to trampoline, etc.) and if you are asking questions like this, chances are you are better suited to purchasing a solution than attempting to roll your own.