Question

I wasted most of my morning failing to solve this simple problem. Using python, I want to parse data files that look like this:

# This is an example comment line, it starts with a '#' character.
# There can be a variable number of comments between each data set.
# Comments "go with" the data set that comes after them.
# The first data set starts on the next line:
0.0 1.0
1.0 2.0
2.0 3.0
3.0 4.0

# Data sets are followed by variable amounts of white space.
# The second data set starts after this comment
5.0 6.0
6.0 7.0


# One more data set.
7.0 8.0
8.0 9.0

The python code I want would parse the above example into the three “blocks”, storing them as elements of a list. The individual code-blocks could themselves be stored as lists of lines, with or without the comment lines, whatever. A handraulic way is to do this:

#! /usr/bin/env python

# Read in data, seperate into rows_alldata
f=open("example")
rows = f.read().split('\n')
f.close()

# Do you haz teh codez?
datasets=[]
datasets.append(rows[0:8])
datasets.append(rows[9:13])
datasets.append(rows[15:18])

I’m looking for a more general solution that supports variable numbers and lengths of data sets. I have tried several catastrophes built off non-pythonic looking loops. I think it best not to clutter up my question with them; this is work and not “homework”.

Answer 1

Use groupby.

from itertools import groupby

def contains_data(ln):
    # just an example; there are smarter ways to do this
    return ln[0] not in "#\n"

with open("example") as f:
    datasets = [[ln.split() for ln in group]
                for has_data, group in groupby(f, contains_data)
                if has_data]