Question

Pretty simple, I’d like to run an external command/program from within a Python script, once it is finished I would also want to know how much CPU time it consumed.

Hard mode: running multiple commands in parallel won’t cause inaccuracies in the CPU consumed result.

Answer 1

On UNIX: either (a) use resource module (also see answer by icktoofay), or (b) use the time command and parse the results, or (c) use /proc filesystem, parse /proc/[pid]/stat and parse out utime and stime fields. The last of these is Linux-specific.

Example of using resource:

import subprocess, resource
usage_start = resource.getrusage(resource.RUSAGE_CHILDREN)
subprocess.call(["yourcommand"])
usage_end = resource.getrusage(resource.RUSAGE_CHILDREN)
cpu_time = usage_end.ru_utime - usage_start.ru_utime

Note: it is not necessary to do fork/execvp, subprocess.call() or the other subprocess methods are fine here and much easier to use.

Note: you could run multiple commands from the same python script simultaneously either using subprocess.Popen or subprocess.call and threads, but resource won’t return their correct individual cpu times, it will return the sum of their times in between calls to getrusage; to get the individual times, run one little python wrapper per command to time it as above (could launch those from your main script), or use the time method below which will work correctly with multiple simultaneous commands (time is basically just such a wrapper).

Example of using time:

import subprocess, StringIO
time_output = StringIO.StringIO()
subprocess.call(["time", "yourcommand", "youroptions"], stdout=time_output)
# parse time_output

On Windows: You need to use performance counters (aka “performance data helpers”) somehow. Here is a C example of the underlying API. To get it from python, you can use one of two modules: win32pdh (part of pywin32; sample code) or pyrfcon (cross-platform, also works on Unix; sample code).

Any of these methods actually meet the “hard mode” requirements above: they should be accurate even with multiple running instances of different processes on a busy system. They may not produce the exact same results in that case compared to running just one process on an idle system, because process switching does have some overhead, but they will be very close, because they ultimately get their data from the OS scheduler.