Question

I have a script in python to process a log file – it parses the values and joins them simply with a tab.

p = re.compile(
    "([0-9/]+) ([0-9]+):([0-9]+):([0-9]+) I.*"+
    "worker\\(([0-9]+)\\)(?:@([^]]*))?.*\\[([0-9]+)\\] "+
    "=RES= PS:([0-9]+) DW:([0-9]+) RT:([0-9]+) PRT:([0-9]+) IP:([^ ]*) "+
    "JOB:([^!]+)!([0-9]+) CS:([\\.0-9]+) CONV:([^ ]*) URL:[^ ]+ KEY:([^/]+)([^ ]*)"
  )

for line in sys.stdin:
  line = line.strip()
  if len(line) == 0: continue
  result = p.match(line)
      if result != None:
    print "\t".join([x if x is not None else "." for x in result.groups()])

However, the scripts behaves quite slowly and it takes a long time to process the data.

How can I achieve the same behaviour in faster way? Perl/SED/PHP/Bash/…?

Thanks

Answer 1

Im writing Perl, not Python, but recently i used this technique to parse very big logs:

1. Divide input file to chunks (for example, FileLen/NumProcessors bytes
   each).
2. Adjust start and end of every chunk to \n so you take full lines to
   each worker.
3. fork() to create NumProcessors workers, each of which reading own
   bytes range from file and writes his own output file.
4. Merge output files if needed.

Sure, you should work to optimize the regexp too, for example less use .* cus it will create many backtraces, this is slow. But anyway, 99% you will have bottleneck on CPU by this regexp, so working on 8 CPUs should help.