Question

I have a small Python script that I need to modify because the format of the metrics file has changed slightly. I do not know Python at all and have tried to take an honest effort to fix it myself. The changes make sense to me but apparently there is still one issue with the script. Otherwise, everything else is working. Here’s what the script looks like:

import sys
import datetime

##########################################################################

now = datetime.datetime.now();
logFile = now.strftime("%Y%m%d")+'.QE-Metric.log';

underlyingParse = True;
strParse = "UNDERLYING_TICK";
if (len(sys.argv) == 2):
    if sys.argv[1] == '2':
    strParse = "ORDER_SHOOT";
        underlyingParse = False;
elif (len(sys.argv) == 3):
    logFile = sys.argv[2];    
    if sys.argv[1] == '2':
    strParse = "ORDER_SHOOT";
        underlyingParse = False;
else:
    print 'Incorrect number of arguments. Usage: <exec> <mode (1) Underlying (2) OrderShoot> <FileName (optional)>'
    sys.exit()

##########################################################################

# Read the deployment file
FIput = open(logFile, 'r');
FOput = open('ParsedMetrics.txt', 'w');

##########################################################################

def ParseMetrics( file_lines ):

    ii = 0
    tokens = []; 
    for ii in range(len(file_lines)):

        line = file_lines[ii].strip()

        if (line.find(strParse) != -1):

             tokens = line.split(",");
             currentTime = float(tokens[2])

             if (underlyingParse == True and ii != 0):
                 newIndex = ii-1
                 prevLine = file_lines[newIndex].strip()
                 while (prevLine.find("ORDER_SHOOT") != -1 and newIndex > -1):
                     newIndex -= 1;
                     tokens = prevLine.split(",");
                     currentTime -= float(tokens[2]);
                     prevLine = file_lines[newIndex].strip();

         if currentTime > 0:
                 FOput.write(str(currentTime) + '\n')

##########################################################################

file_lines = FIput.readlines()
ParseMetrics( file_lines );

print 'Metrics parsed and written to ParsedMetrics.txt'

Everything is working fine except for the logic that is supposed to reverse iterate through previous lines to add up the ORDER_SHOOT numbers since the last UNDERLYING_TICK event occurred (starting at the code: if (underlyingParse == True and ii != 0):…) and then subtract that total from the current UNDERLYING_TICK event line being processed. This is what a typical line in the file being parsed looks like:

08:40:02.039387(+26): UNDERLYING_TICK, 1377, 1499.89

Basically, I’m only interested in the last data element (1499.89) which is the time in micros. I know it has to be something stupid. I just need another pair of eyes. Thanks!

Answer 1

So, if command line option is 2, the function creates an output file where all the lines contain just the ‘time’ portion of the lines from the input file that had the “order_shoot” token in them?

And if the command line option is 1, the function creates an output file with a line for each line in input file that contained the ‘underlying_tick’ token, except that the number you want here is the underlying_tick time value minus all the order_shoot time values that occurred SINCE the preceding underlying_tick value (or from the start of file if this is the first one)?

If this is correct, and all lines are unique (there are no duplicates), then I would suggest the following re-written script:

#### Imports unchanged.

import sys 
import datetime 

#### Changing the error checking to be a little simpler.
#### If the number of args is wrong, or the "mode" arg is
#### not a valid option, it will print the error message
#### and exit.

if len(sys.argv) not in (2,3) or sys.argv[2] not in (1,2):
    print 'Incorrect arguments. Usage: <exec> <mode (1) Underlying (2) OrderShoot> <FileName (optional)>'
    sys.exit()  

#### the default previously specified in the original code.

now = datetime.datetime.now()

#### Using ternary logic to set the input file to either
#### the files specified in argv[2] (if it exists), or to
#### the default previously specified in the original code.

FIput = open((sys.argv[2] if len(sys.argv)==3 
                          else now.strftime("%Y%m%d")+'.QE-Metric.log'), 'r');

#### Output file not changed.

FOput = open('ParsedMetrics.txt', 'w');

#### START RE-WRITTEN FUNCTION

def ParseMetrics(file_lines,mode): 

#### The function now takes two params - the lines from the 
#### input file, and the 'mode' - whichever the user selected
#### at run-time. As you can see from the call down below, this
#### is taken straight from argv[1]. 

    if mode == '1':

#### So if we're doing underlying_tick mode, we want to find each tick,
#### then for each tick, sum the preceding order_shoots since the last
#### tick (or start of file for the first tick).

        ticks = [file_lines.index(line) for line in file_lines \
                                        if 'UNDERLYING_TICK' in line]

#### The above list comprehension iterates over file_lines, and creates
#### a list of the indexes to file_lines elements that contain ticks.
#### 
#### Then the following loop iterates over ticks, and for each tick,
#### subtracts the sum of all times for order_shoots that occure prior
#### to the tick, from the time value of the tick itself. Then that
#### value is written to the outfile.

        for tick in ticks:
            sub_time = float(file_lines[tick].split(",")[2]) - \
                       sum([float(line.split(",")[2]) \ 
                       for line in file_lines if "ORDER_SHOOT" in line \
                       and file_lines.index(line) <= tick]
            FOput.write(float(line.split(",")[2]))    

#### if the mode is 2, then it just runs through file_lines and
#### outputs all of the order_shoot time values.

    if mode == '2':
        for line in file_lines:
            if 'ORDER_SHOOT' in line:
                FOput.write(float(line.split(",")[2]))

#### END OF REWRITTEN FUNCTION

#### As you can see immediately below, we pass sys.argv[2] for the
#### mode argument of the ParseMetrics function.

ParseMetrics(FIput.readlines(),sys.argv[2])

print 'Metrics parsed and written to ParsedMetrics.txt' 

And that should do the trick. The main issue is that if you have any lines with “UNDERLYING_TICK” that are exact duplicates of any other such line, then this will not work. Different logic would need to be applied to get the correct indexes.

I am sure there is a way to make this much better, but this was my first thought.

It’s also worth noting I added a lot of inline line breaks to the above source for readability, but you might want to pull them if you use this as written.