Question

I’m trying to use python to create a random binary file. This is what I’ve got already:

f = open(filename,'wb')
for i in xrange(size_kb):
    for ii in xrange(1024/4):
        f.write(struct.pack("=I",random.randint(0,sys.maxint*2+1)))

f.close()

But it’s terribly slow (0.82 seconds for size_kb=1024 on my 3.9GHz SSD disk machine). A big bottleneck seems to be the random int generation (replacing the randint() with a 0 reduces running time from 0.82s to 0.14s).

Now I know there are more efficient ways of creating random data files (namely dd if=/dev/urandom) but I’m trying to figure this out for sake of curiosity… is there an obvious way to improve this?

Answer 1

IMHO – the following is completely redundant:

f.write(struct.pack("=I",random.randint(0,sys.maxint*2+1)))

There’s absolutely no need to use struct.pack, just do something like:

import os

fileSizeInBytes = 1024
with open('output_filename', 'wb') as fout:
    fout.write(os.urandom(fileSizeInBytes)) # replace 1024 with a size in kilobytes if it is not unreasonably large

Then, if you need to re-use the file for reading integers, then struct.unpack then.

(my use case is generating a file for a unit test so I just need a
file that isn’t identical with other generated files).

Another option is to just write a UUID4 to the file, but since I don’t know the exact use case, I’m not sure that’s viable.