Question

There is a performance issue when using a system call after pre-allocating big amount of memory (e.g. numpy array). The issue grows with the amount of memory.

test.py :

import os
import sys
import time
import numpy

start = time.clock()
test = int(sys.argv[1])
a = numpy.zeros((test,500,500))
for i in range(test) :
    os.system("echo true > /dev/null")
elapsed = (time.clock() - start)
print(elapsed)

The per-iteration time increases dramatically :

edouard@thorin:~/now3/code$ python test.py 100
0.64
edouard@thorin:~/now3/code$ python test.py 200
2.09
edouard@thorin:~/now3/code$ python test.py 400
14.26

This should not be related to virtual memory.
Is it a known issue?

Answer 1

You seem to have narrowed the problem down to os.system() taking longer after you’ve allocated a large NumPy array.

Under the covers, system() uses fork(). Even though fork() is supposed to be very cheap (due to its use of copy-on-write), it turns out that things are not quite as simple.

In particular, there are known issues with Linux’s fork() taking longer for larger processes. See, for example:

• Faster forking of large processes on Linux?
• Linux Fork Performance Redux: Large Pages.

Both documents are fairly old, so I am not sure what the state of the art is. However, the evidence suggests that you’ve encountered an issue of this sort.

If you can’t get rid of those system() calls, I would suggest two avenues of research:

• Look into enabling huge pages.
• Consider the possibility of spawning an auxiliary process on startup, whose job would be to invoke the necessary system() commands.