Question

I’m new to python and am trying to read “blocks” of data from a file. The file is written something like:

# Some comment
# 4 cols of data --x,vx,vy,vz
# nsp, nskip =           2          10


#            0   0.0000000


#            1           4
 0.5056E+03  0.8687E-03 -0.1202E-02  0.4652E-02
 0.3776E+03  0.8687E-03  0.1975E-04  0.9741E-03
 0.2496E+03  0.8687E-03  0.7894E-04  0.8334E-03
 0.1216E+03  0.8687E-03  0.1439E-03  0.6816E-03


#            2           4
 0.5056E+03  0.8687E-03 -0.1202E-02  0.4652E-02
 0.3776E+03  0.8687E-03  0.1975E-04  0.9741E-03
 0.2496E+03  0.8687E-03  0.7894E-04  0.8334E-03
 0.1216E+03  0.8687E-03  0.1439E-03  0.6816E-03


#          500  0.99999422


#            1           4
 0.5057E+03  0.7392E-03 -0.6891E-03  0.4700E-02
 0.3777E+03  0.9129E-03  0.2653E-04  0.9641E-03
 0.2497E+03  0.9131E-03  0.7970E-04  0.8173E-03
 0.1217E+03  0.9131E-03  0.1378E-03  0.6586E-03

and so on

Now I want to be able specify and read only one block of data out of these many blocks. I’m using numpy.loadtxt('filename',comments='#') to read the data but it loads the whole file in one go. I searched online and someone has created a patch for the numpy io routine to specify reading blocks but it’s not in mainstream numpy.

It’s much easier to choose blocks of data in gnuplot but I’d have to write the routine to plot the distribution functions. If I can figure out reading specific blocks, it would be much easier in python. Also, I’m moving all my visualization codes to python from IDL and gnuplot, so it’ll be nice to have everything in python instead of having things scattered around in multiple packages.

I thought about calling gnuplot from within python, plotting a block to a table and assigning the output to some array in python. But I’m still starting and I could not figure out the syntax to do it.

Any ideas, pointers to solve this problem would be of great help.

Answer 1

A quick basic read:

>>> def read_blocks(input_file, i, j):
    empty_lines = 0
    blocks = []
    for line in open(input_file):
        # Check for empty/commented lines
        if not line or line.startswith('#'):
            # If 1st one: new block
            if empty_lines == 0:
                blocks.append([])
            empty_lines += 1
        # Non empty line: add line in current(last) block
        else:
            empty_lines = 0
            blocks[-1].append(line)
    return blocks[i:j + 1]

>>> for block in read_blocks(s, 1, 2):
    print '-> block'
    for line in block:
        print line


-> block
 0.5056E+03  0.8687E-03 -0.1202E-02  0.4652E-02
 0.3776E+03  0.8687E-03  0.1975E-04  0.9741E-03
 0.2496E+03  0.8687E-03  0.7894E-04  0.8334E-03
 0.1216E+03  0.8687E-03  0.1439E-03  0.6816E-03
-> block
 0.5057E+03  0.7392E-03 -0.6891E-03  0.4700E-02
 0.3777E+03  0.9129E-03  0.2653E-04  0.9641E-03
 0.2497E+03  0.9131E-03  0.7970E-04  0.8173E-03
 0.1217E+03  0.9131E-03  0.1378E-03  0.6586E-03
>>> 

Now I guess you can use numpy to read the lines…