Question

In a project we are doing we encounter log files of which each line has the following structure:

2012-01-02,12:50:32,658,2,1,2,0,0,0,0,1556,1555,62,60,2,3,0,0,0,0,1559,1557,1557,63,64,65,0.305,0.265,0.304,0.308,0.309

The structure of the string is embedded in the string itself.

First we have some metadata:

• date: 2012/01/02
• time: 12:50:32
• measurement number: 658
• number of measurement groups: 2

This is then followed by the data of each group sequentially.

• Measurement group 1: 1,2,0,0,0,0,1556,1555,62,60
• Measurement group 2: 2,3,0,0,0,0,1559,1557,1557,63,64,65

Group data has the following structure (measurement group 1 used below as an example):

• number of the measurement group:1
• number of sensors in this group:2
• control field 1 to 4 (0 most of the time):0,0,0,0
• raw values of type 1 for each sensor (>1500 in the examples):1556,1555
• raw values of type 2 for each sensor (~60 in the examples),62,60

The line continues with the calculated values for all sensors mentioned above consecutively (i.e. no more control values, or raw values)

In the example, the total number of sensors = 2 + 3 = 5 so the calculated line is:

0.305,0.265,0.304,0.308,0.309

My question is this:
If we want to normalize the values for each sensor like this:

date, time, number of measurement group, number sensor in group, (raw value type 1, raw value type 2, calculated value)

What would be a flexible solution, given that a any date-time each variable is well… variable (meaning that the number of measurement group is variable, and the number of sensors in each group can also be variable?

For the example final output should be something like:

• 2012/01/02,12:50:32,1,1,(1556,62,0.305)
• 2012/01/02,12:50:32,1,2,(1555,60,0.265)
• 2012/01/02,12:50:32,2,1,(1559,63,0.304)
• 2012/01/02,12:50:32,2,2,(1557,64,0.308)
• 2012/01/02,12:50:32,2,3,(1557,65,0.309)

What I did up to now is to segment the measurement into cases over time and define “statically” which columns are to be inserted for a line belonging to a case, which group a sensor belongs to, what its sensornumber is,…

This is hardly a good solution as each change in the measurement setup results in more changes to the code.

line="""2012-01-02,12:50:32,658,2,1,2,0,0,0,0,1556,1555,62,60,2,3,0,0,0,0,1559,1557,1557,63,64,65,0.305,0.265,0.304,0.308,0.309"""
parts=line.split(",")
date=parts[0]
groupnames=[1,1,2,2,2]
sensornumbers=[1,2,1,2,3]
raw_type1_idx=[10,11,20,21,22]
raw_type2_idx=[12,13,23,24,25]
calc_idx=[26,27,28,29,30]
for i,j,k,l,m in zip(groupnames,sensornumbers,raw_type1_idx,raw_type2_idx,calc_idx):
    output_tpl= parts[k],parts[l],parts[m]
    print "%s,%s,%s,%s" % (date,i,j,output_tpl)

Is there a better Python way of doing stuff like this?

Answer 1

It’s a non-trivial task. Probably the most “pythonic” way would be to create a class.

I took the liberty and time to make an example:

from collections import namedtuple

class DataPack(object):
    def __init__(self, line, seperator =',', headerfields = None, groupfields = None):        
        self.seperator = seperator
        self.header_fields = headerfields or ('date', 'time', 'nr', 'groups')
        self.group_fields = groupfields or ('nr', 'sensors','controlfields',
                                            't1values', 't2values')
        Header = namedtuple('Header', self.header_fields)

        self.header_part = line.split(self.seperator)[:self.data_start]
        self.data_start = len(self.header_fields)
        self.data_part = line.split(self.seperator)[self.data_start:]
        self.header = Header(*self.header_part)
        self.groups = self._create_groups(self.data_part, self.header.groups)

    def _create_groups(self, datalst, groups):
        """nr, sensors controllfield * 4, t1value*sensors, t2value*sensors """        
        Group = namedtuple('DataGroup', self.group_fields)
        _groups = []
        for i in range(int(groups)):
            nr = datalst[0]
            sensors = datalst[1]
            controlfields = datalst [2:6]
            t1values = datalst[6:6+int(sensors)]
            t2values = datalst[6+int(sensors):6+int(sensors)*2]
            _groups.append(Group(nr, sensors, controlfields, t1values, t2values))
            datalst = datalst[6+int(sensors)*2:]
        return _groups

    def __str__(self):
        _return = []        
        for group in self.groups:
            for sensor in range(int(group.sensors)):
                _return.append('%s, ' % self.header.date.replace('-','/'))
                _return.append('%s, ' % self.header.time)
                _return.append('%s, ' % group.nr)
                _return.append('%s, ' % (int(sensor) + 1,))
                _return.append('(%s, ' % group.t1values[int(sensor)])
                _return.append('%s)\n' % group.t2values[int(sensor)])
        return u''.join(_return)

if __name__ == '__main__':
    line = """2012-01-02,12:50:32,658,2,1,2,0,0,0,0,1556,1555,62,60,2,3,0,0,0,0,1559,1557,1557,63,64,65,0.305,0.265,0.304,0.308,0.309"""
    data = DataPack(line)
    for i in data.header: print i,
    for i in data.groups: print '\n',i
    print '\n',data
    print 'cfield 0:2 ', data.groups[0].controlfields[2]
    print 't2value 1:2 ', data.groups[1].t2values[2]

On bigger changes to the input-data you would have to subclass and overwrite the _create_groups and __str__ methods.