There are many online judge sites which can verify your program by comparing its output to the correct answers. What’s more, they also check the running time and memory usage to make sure that your program doesn’t exceed the maximum limit.
So here is my question, since some online judge sites run several test programs at the same time, how do they achieve performance isolation? and how do they achieve same running time on same program that run at another time?
I think there are isolated environment processes like ‘VMware’ or ‘Sandbox’ that always return same result. is this correct? and any idea about how to implement these things?
Current Solution
I’m using docker for sandboxing. it’s a dead simple and the safest way.
Unfortunately it is VERY hard to actually guarantee consistent running times even on a dedicated machine versus a VM. If you do want to implement something like this as was mentioned you probably want a VM to keep all the code that will run sandboxed. Usually you don’t want to service more than a couple of requests per core so I would say for algorithms that are memory and cpu bound use at most 2 VMs per physical core of the machine.
Although I can only speculate why not try different numbers of VMs per core and see how it performs. Try to aim for about a 90% or higher rate of SLO compliance (or 98-99 if you really need to) and you should be just fine. Again its hard to tell you exactly what to do as a lot of these things require just testing it out and seeing how it does.