Question

I wrote a script to print specific days of a log file ! the script work properly , but I wrote the loop part with while read line and the log file contains more than 150000 items ! the script wants to read it line by line and it takes hours !so, the while should be changed in some way ! I know about for I in {1..N} but I want something better that work for any log file .

what do you suggest ?

Answer 1

You’ll have to show us what you are doing. Please edit your posting to include smallest subset of while loop code that illustrates your problem. Also please show exact text of any errors or warning msgs.

Per your comment ‘… I should press enter for each line to be read’. You are definitely doing something wrong. Reading each line should be automatic. Here is a basic outline of processing files with bash.

Would you consider using awk? I can post better solutions with that standard tool.

 inputFiles="${@}"
 cat "${inputFiles}"\
 | while read line ; do
     case ${line} in
        2[0-9][0-9][0-9]-[0-1][0-9]-[0-3][0-9]* ) 
            # matched date
             print ${line}
        ;;
        * )
            # you'll need to add other reg-expes here to extract data you want
        ;;
      esac
 done

You may have to remove the dbl-quotes on ${inputFiles}.

The purpose of using cat ${inputfiles | while read line ... is to allow processing any files you list on the cmdline.

Have you tried using grep to look thru your logfiles?

 date1='2011-09-11' ; date2='2011-09-12'; date3='2011-09-13'
 grep "${date1}|${date2}|${date3}" *.logFiles | grep -i error

You could also use the date1… as a case target in the while loop above.

 case ${line} in
   ${date1}*|${date2}*|${date3}* )
      echo $line
   ;;
 esac

You may get away with removing the ‘*‘s above OR if the date is NOT the first element in the line, you may need to add ‘*‘ in front of each ${dateN}.

Or if you’re keeping your logfiles zipped up,

 gunzip -c ${zippedLogFiles_gz) | grep "${date1}|${date2}|${date3}" | grep -i error

is often a good place to start (and only unzips your files in place (just sending the uncompressed output into the pipe)).

P.S. as you appear to be a new user, if you get an answer that helps you please remember to mark it as accepted, and/or give it a + (or -) as a useful answer.