Question

Recently I have made a plan to develop a Online Judge system (like spoj). And I have write a judge script in python. It forks every time when it receives one submitted code, and then compile the code and use subprocess.Popen to start the program. But when I get program’s memory usage with resource.getrusage(resource.RUSAGE_CHILDREN), it returns memory of the forked python process instead of the subprocess.

Below is some of my code:

try:
    programThread = subprocess.Popen(command, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, preexec_fn = self.limit, bufsize = -1)
except OSError:
    self.status.value = -2
    return -2
self.JudgeID.value = programThread.pid
programOutput = programThread.communicate(_in)
self.status.value = 0
Res = resource.getrusage(resource.RUSAGE_CHILDREN)
self.cpuusage.value = Res.ru_utime + Res.ru_stime
self.memoryusage.value = Res.ru_maxrss * resource.getpagesize()

cpuusage and memoryusage is shared memory created by multiprocessing.Value, the value of memoryusage is 12M. But via top, i’ve found, that the memory usage of forked python process is 12M while the subprocess is 900K. Are there anything wrong with my code? Sorry for my bad English.

Answer 1

This is called maxrss because it’s the maximum of the RSS usage of the child. The problem is that subprocess uses the standard way to create the subprocess: it first forks, creating a new process id (childpid), and then execv the intended subprocess’ executable (which doesn’t change the childpid). Then resource returns the maximum of the RSS among the lifetime of this childpid. If this child ran both a forked copy of Python at 12MB and another program at 900K, then the result is 12MB.

I don’t really see how to fix this problem…