Question

I have written a little program that parses log files of anywhere between a few thousand lines to a few hundred thousand lines. For this, I have a function in my code which parses every line, looks for keywords, and returns the keywords with the associated values.

These log files contain of little sections. Each section has some values I’m interested in and want to store as a dictionary.

I have simplified the sample below, but the idea is the same.

My original function looked like this, it gets called between 100 and 10000 times per run, so you can understand why I want to optimize it:

def parse_txt(f):
    d = {}
    for line in f:
        if not line:
            pass

        elif 'apples' in line:
            d['apples'] = True

        elif 'bananas' in line:
            d['bananas'] = True

        elif line.startswith('End of section'):
            return d


f = open('fruit.txt','r')
d = parse_txt(f)
print d 

The problem I run into, is that I have a lot of conditionals in my program, because it checks for a lot of different things and stores the values for it. And when checking every line for anywhere between 0 and 30 keywords, this gets slow fast. I don’t want to do that, because, not every time I run the program I’m interested in everything. I’m only ever interested in 5-6 keywords, but I’m parsing every line for 30 or so keywords.

In order to optimize it, I wrote the following by using exec on a string:

def make_func(args):
    func_str = """
def parse_txt(f):
    d = {}
    for line in f:
        if not line:
            pass
    """

    if 'apples' in args:
        func_str += """ 
        elif 'apples' in line:
            d['apples'] = True
"""

    if 'bananas' in args:
        func_str += """ 
        elif 'bananas' in line:
            d['bananas'] = True
"""

    func_str += """ 
        elif line.startswith('End of section'):
            return d"""

    print func_str
    exec(func_str)
    return parse_txt

args = ['apples','bananas']
fun = make_func(args)
f = open('fruit.txt','r')
d = fun(f)
print d

This solution works great, because it speeds up the program by an order of magnitude and it is relatively simple. Depending on the arguments I put in, it will give me the first function, but without checking for all the stuff I don’t need.

For example, if I give it args=['bananas'], it will not check for 'apples', which is exactly what I want to do.

This makes it much more efficient.

However, I do not like it this solution very much, because it is not very readable, difficult to change something and very error prone whenever I modify something. Besides that, it feels a little bit dirty.

I am looking for alternative or better ways to do this. I have tried using a set of functions to call on every line, and while this worked, it did not offer me the speed increase that my current solution gives me, because it adds a few function calls for every line. My current solution doesn’t have this problem, because it only has to be called once at the start of the program. I have read about the security issues with exec and eval, but I do not really care about that, because I’m the only one using it.

EDIT:
I should add that, for the sake of clarity, I have greatly simplified my function. From the answers I understand that I didn’t make this clear enough.
I do not check for keywords in a consistent way. Sometimes I need to check for 2 or 3 keywords in a single line, sometimes just for 1. I also do not treat the result in the same way. For example, sometimes I extract a single value from the line I’m on, sometimes I need to parse the next 5 lines.

Answer 1

I would try defining a list of keywords you want to look for (“keywords”) and doing this:

for word in keywords:
    if word in line:
        d[word] = True

Or, using a list comprehension:

dict([(word,True) for word in keywords if word in line])

Unless I’m mistaken this shouldn’t be much slower than your version.

No need to use eval here, in my opinion. You’re right in that an eval based solution should raise a red flag most of the time.

Edit: as you have to perform a different action depending on the keyword, I would just define function handlers and then use a dictionary like this:

def keyword_handler_word1(line):
    (...)

(...)

def keyword_handler_wordN(line):
    (...)

keyword_handlers = { 'word1': keyword_handler_word1, (...), 'wordN': keyword_handler_wordN }

Then, in the actual processing code:

for word in keywords:
    # keyword_handlers[word] is a function
    keyword_handlers[word](line)