Question

Right, I’m iterating through a large binary file

I need to minimise the time of this loop:

def NB2(self, ID_LEN):
    r1=np.fromfile(ReadFile.fid,dTypes.NB_HDR,1)
    num_receivers=r1[0][0]
    num_channels=r1[0][1]
    num_samples=r1[0][5]

    blockReturn = np.zeros((num_samples,num_receivers,num_channels))

    for rec in range(0,num_receivers):
        for chl in range(0,num_channels):
            for smpl in range(0,num_samples):
                r2_iq=np.fromfile(ReadFile.fid,np.int16,2)
                blockReturn[smpl,rec,chl] = np.sqrt(math.fabs(r2_iq[0])*math.fabs(r2_iq[0]) + math.fabs(r2_iq[1])*math.fabs(r2_iq[1]))

    return blockReturn

So, what’s going on is as follows:
r1 is the header of the file, dTypes.NB_HDR is a type I made:

NB_HDR= np.dtype([('f3',np.uint32),('f4',np.uint32),('f5',np.uint32),('f6',np.int32),('f7',np.int32),('f8',np.uint32)])

That gets all the information about the forthcoming data block, and nicely puts us in the right position within the file (the start of the data block!).

In this data block there is:
4096 samples per channel,
4 channels per receiver,
9 receivers.

So num_receivers, num_channels, num_samples will always be the same (at the moment anyway), but as you can see this is a fairly large amount of data. Each ‘sample’ is a pair of int16 values that I want to find the magnitude of (hence Pythagoras).

This NB2 code is executed for each ‘Block’ in the file, for a 12GB file (which is how big they are) there are about 20,900 Blocks, and I’ve got to iterate through 1000 of these files (so, 12TB overall). Any speed advantage even it’s it’s milliseconds would be massively appreciated.

EDIT: Actually it might be of help to know how I’m moving around inside the file. I have a function as follows:

def navigateTo(self, blockNum, indexNum):
    ReadFile.fid.seek(ReadFile.fileIndex[blockNum][indexNum],0)
    ReadFile.currentBlock = blockNum
    ReadFile.index = indexNum

Before I run all this code I scan the file and make a list of index locations at ReadFile.fileIndex that I browse using this function and then ‘seek’ to the absolute location – is this efficient?

Cheers

Answer 1

Because you know the length of a block after you read the header, read the whole block at once. Then reshape the array (very fast, only affects metadata) and take use the np.hypot ufunc:

blockData = np.fromfile(ReadFile.fid, np.int16, num_receivers*num_channels*num_samples*2)
blockData = blockData.reshape((num_receivers, num_channes, num_samples, 2))
return np.hypot(blockData[:,:,:,0], blockData[:,:,:,1])

On my machine it runs in 11ms per block.