Question

I’m writing a Python script that takes in a (potentially large) file. Here is an example of a way that input file could be formatted:

class1 1:v1 2:v2 3:v3 4:v4 5:v5
class2 1:v6 4:v7 5:v8 6:v9
class1 3:v10 4:v11 5:v12 6:v13 8:v14
class2 1:v15 2:v16 3:v17 5:v18 7:v19

Where class1 and class2 are some number, e.g. 1 and -1. (A curious user may notice that this is a LIBSVM-related file, but knowing the software isn’t necessary in this case.) The values v1, v2, …, v19 represent any integer or float value. Obviously, my files would be much larger than this, in terms of total lines and length per line, which is why I’m concerned about efficiency here.

I am trying to check what is the greatest value to the left of a colon. In LIBSVM, these are called “features” and are always integers here. For instance, in the example I outlined above, line 1 has 5 as its largest feature. Line 2 has 6 as its largest feature, line 3 has 8 as its largest feature, and finally, line 4 has 7 as its largest feature. Since 8 is the largest of these values, that is my desired value. I’m looking at a file with possibly thousands of features per line, and many hundreds of thousands of lines.

The file satisfies the following properties:

1. The features must be strictly increasing. I.e. “3:v1 4:v2” is allowed, but not “3:v1 3:v2.”
2. The features are not necessarily consecutive and can be skipped. In the first example I gave, the first line has its features in consecutive order (1,2,3,4,5) and skips features 6, 7, and 8. The other 3 lines do not have their features in consecutive order. That’s okay, as long as those features are strictly increasing.

Right now, my approach is to check each line, split up each line by a space, split up the final term by a colon, and then check the feature value. Following that, I do a procedure to check the maximum such featureNum.

file1 = open(...)
max = 0
for line in file1:
    linesplit = line.rstrip('\n').split(' ')
    val = linesplit[len(linesplit) - 1]
    valsplit = val.split(':')
    featureNum = valsplit[0]
    if (featureNum > max):
        max = featureNum
 print max
 file1.close()

But I’m hoping there is a better or more efficient way of doing this, e.g. some way of analyzing the file by only getting those terms that directly precede a newline character (maybe to avoid reading all the lines?). I’m new to Python so it wouldn’t surprise me if I missed something obvious.

Possible reference: http://docs.python.org/library/stdtypes.html

Answer 1

Since you don’t care about all the features in a line but just the last one, you don’t need to split the whole line. I don’t know if this is actually faster though, you need to time it and see. It definitely isn’t as Pythonic as splitting the entire line.

def last_feature(line):
    start = line.rfind(' ') + 1
    end = line.rfind(':')
    return int(line[start:end])

with open(...) as file1:
    largest = max(last_feature(line) for line in file1)