Question

Normally I use something like this for processes I run on my servers

./runEvilProcess.sh >> ./evilProcess.log

However I’m currently using Doxygen and it produces lots of duplicate output

Example output:

QGDict::hashAsciiKey: Invalid null key
QGDict::hashAsciiKey: Invalid null key
QGDict::hashAsciiKey: Invalid null key

So you end up with a very messy log

Is there a way I can only add the line to the log file if the line wasn’t the last one added.

A poor example (but not sure how to do in bash)

$previousLine = ""
$outputLine = getNextLine()
if($previousLine != $outputLine) {
  $outputLine >> logfile.log
  $previousLine = $outputLine
}

Answer 1

If the process returns duplicate lines in a row, pipe the output of your process through uniq:

 $ ./t.sh 
one
one
two
two
two
one
one
 $ ./t.sh | uniq
one
two
one

If the logs are sent to the standard error stream, you’ll need to redirect that too:

 $ ./yourprog 2>&1 | uniq >> logfile

(This won’t help if the duplicates come from multiple runs of the program – but then you can pipe your log file through uniq when reviewing it.)