Question

I have been using this script for years at work to summarize log files.

 #!/usr/bin/perl

 $logf = '/var/log/messages.log';

 @logf=( `cat $logf` );
 foreach $line ( @logf ) {
      $line=~s/\d+/#/g;
      $count{$line}++;
 }

 @alpha=sort @logf;
 $prev = 'null';
 @uniq = grep($_ ne $prev && ($prev = $_), @alpha);
 foreach $line (@uniq) {
      print "$count{$line}: ";
      print "$line";
 }

I have wanted to rewrite it in Python but I do not fully understand certain portions of it, such as:

 @alpha=sort @logf;
 $prev = 'null';
 @uniq = grep($_ ne $prev && ($prev = $_), @alpha);

Does anyone know of a Python module that would negate the need to rewrite this? I haven’t had any luck find something similar. Thanks in advance!

Answer 1

 @alpha=sort @logf;
 $prev = 'null';
 @uniq = grep($_ ne $prev && ($prev = $_), @alpha);

would be equivalent to

uniq = sorted(set(logf))

if logf were a list of lines.

However, since you are counting the freqency of lines,
you could use a collections.Counter to both count the lines and collect the unique lines (as keys) (thus removing the need to compute uniq at all):

count = collections.Counter()
for line in f:
    count[line] += 1

---

import sys
import re
import collections

logf = '/var/log/messages.log'
count = collections.Counter()
write = sys.stdout.write

with open(logf, 'r') as f:
    for line in f:
        line = re.sub(r'\d+','#',line)
        count[line] += 1

for line in sorted(count):
     write("{c}: {l}".format(c = count[line], l = line))