On Linux, normally I use ptrace function to trace all syscall, and kill the process if the it tries to do anything harmful to my machine, such as system("shutdown -s -t 00") or so.
Is there a way for me to do this on Windows?
EDIT: I want to write Sandbox program to limit time and memory usage of its child that can work on both Windows and Linux, and now it can only run on Linux via ptrace
If you attach your process to a Job object, you can limit the CPU time and memory of the process. That won’t allow you to prevent if from performing malicious actions but it will allow you to achieve what you asked.