Question

I am trying to parse a text file with the following structure:

latitude                        5.0000
number_of_data_values             9
  0.1   0.2   0.3   0.4
  1.1   1.2   1.3   1.4      
  8.1
latitude                        4.3000
number_of_data_values             9
  0.1   0.2   0.3   0.4
  1.1   1.2   1.3   1.4       
  8.1
latitude                        4.0000
number_of_data_values             9
  0.1   0.2   0.3   0.4
  1.1   1.2   1.3   1.4       
  8.1
 ...

Every different latitude number is a different array line.
number_of_data_values is the number of colomns (consistent thorough the file).

For this example I would like to read the file and output a 3 by 9 two-dimensional array like the following:

array = [[0.1,0.2,0.3,0.4,1.1,1.2,1.3,1.4,8.1],
         [0.1,0.2,0.3,0.4,1.1,1.2,1.3,1.4,8.1],
         [0.1,0.2,0.3,0.4,1.1,1.2,1.3,1.4,8.1]]

I had a try at it by iterating through the line with loops but I am looking for a more efficient way to do it as I may deal with voluminous input files.

Answer 1

A line-by-line implementation is rather easy and understandable. Assuming that your latitude always start on a new line (which is not what your example give, but it might be a typo), you could do:

latitudes = []
counts = []
blocks = []
current_block = []
for line in test:
    print line
    if line.startswith("latitude"):
        # New block: add the previous one to `blocks` and reset
        blocks.append(current_block)
        current_block = []
        latitudes.append(float(line.split()[-1]))
    elif line.startswith("number_of_data"):
        # Just append the current count to the list
        counts.append(int(line.split()[-1]))
    else:
        # Update the current block
        current_block += [float(f) for f in line.strip().split()]
# Make sure to add the last block...
blocks.append(current_block)
# And to remove the first (empty) one
blocks.pop(0)

You can know check whether all your blocks have the proper size:

all(len(b)==c for (c,b) in zip(counts,blocks))

Alternative solution

If you’re concerned about the loops, you may want to consider querying a memory-mapped version of your file. The idea is to find the positions of the lines starting with latitude. Once you find one, find the next and you have a block of text: zap the first two lines (the one starting with latitude and the one starting with number_of_data), combine the remaining ones and process.

import mmap

with open("crap.txt", "r+b") as f:
    # Create the mapper
    mapper = mmap.mmap(f.fileno(), 0)
    # Initialize your output variables
    latitudes = []
    blocks = [] 
    # Find the beginning of the first block
    position = mapper.find("latitude")
    # `position` will be -1 if we can't find it
    while (position >= 0):
        # Move to the beginning of the block
        mapper.seek(position)
        # Read the first line
        lat_line = mapper.readline().strip()
        latitudes.append(lat_line.split()[-1])
        # Read the second one
        zap = mapper.readline()
        # Where are we ?
        start = mapper.tell()
        # Where's the next block ?
        position = mapper.find("latitude")
        # Read the lines and combine them into a large string
        current_block = mapper.read(position-start).replace("\n", " ")
        # Transform the string into a list of floats and update the block
        blocks.append(list(float(i) for i in current_block.split() if i))