Question

I need to write a shell script that pick all the files (not directories) in /exp/files directory. For each file inside the directory I want to find whether the last line of file is received . The last line in the file is a trailer record. Also the third field in the last line is the number of data records count i.e 2315 (Total Number of lines in the file -2 (header,trailer) ) . In my unix shell script i want to check whether the last line is a trailer record by checking T and want to check whether the number of lines in the file is equal to (2315+2). If this is successful then i want to move the file to a different directory /exp/ready.

tail -1 test.csv 
T,Test.csv,2315,80045.96

Also in the inputfile sometimes 0 or 1 more fields of trailer record can be within double quotes

"T","Test.csv","2315","80045.96"
"T", Test.csv, 2212,"80045.96"
T,Test.csv,2315,80045.96

Answer 1

You can test for the presence of the last line with the following:

tail -1 ${filename} | egrep '^T,|^"T",' >/dev/null 2>&1
rc=$?

At that point $rc will be 0 if the line started with either T, or "T",, assuming that’s enough to catch the trailer record.

Once you’ve established that, you can extract the line count with:

lc=$(cat ${filename} | wc -l)

and you can get the expected line count with:

elc=$(tail -1 ${filename} | awk -F, '{sub(/^"/,"",$3);print 2+$3}')

and compare the two.

So, tying all that together, this would be a good start. It outputs the file itself (my test files num[1-9].tst) along with a message indicating whether the file is okay or why it is not okay.

#!/bin/bash
cd /exp/files
for fspec in *.tst ; do
    if [[ -f ${fspec} ]] ; then
        cat ${fspec} | sed 's/^/   /'
        tail -1 ${fspec} | egrep '^T,|^"T",' >/dev/null 2>&1
        rc=$?
        if [[ ${rc} -eq 0 ]] ; then
            lc=$(cat ${fspec} | wc -l)
            elc=$(tail -1 ${fspec} | awk -F, '{sub(/^"/,"",$3);print 2+$3}')
            if [[ ${lc} -eq ${elc} ]] ; then
                echo '***' File ${fspec} is done and dusted.
            else
                echo '***' File ${fspec} line count mismatch: ${lc}/${elc}.
            fi
        else
            echo '***' File ${fspec} has no valid trailer.
        fi
    else
        ls -ald ${fspec} | sed 's/^/   /'
        echo '***' File ${fspec} is not a regular file.
    fi
done

The sample run, showing the test files I used:

   H,Test.csv,other rubbish goes here
   this file does not have a trailer
*** File num1.tst has no valid trailer.
   H,Test.csv,other rubbish goes here
   this file does have a trailer with all quotes and correct count
   "T","Test.csv","1","80045.96"
*** File num2.tst is done and dusted.
   H,Test.csv,other rubbish goes here
   this file does have a trailer with all quotes but bad count
   "T","Test.csv","9","80045.96"
*** File num3.tst line count mismatch: 3/11.
   H,Test.csv,other rubbish goes here
   this file does have a trailer with all quotes except T, and correct count
   T,"Test.csv","1","80045.96"
*** File num4.tst is done and dusted.
   H,Test.csv,other rubbish goes here
   this file does have a trailer with no quotes on T or count and correct count
   T,"Test.csv",1,"80045.96"
*** File num5.tst is done and dusted.
   H,Test.csv,other rubbish goes here
   this file does have a traier with quotes on T only, and correct count
   "T",Test.csv,1,80045.96
*** File num6.tst is done and dusted.
   drwxr-xr-x+ 2 pax None 0 Feb 23 09:55 num7.tst
*** File num7.tst is not a regular file.
   H,Test.csv,other rubbish goes here
   this file does have a trailer with all quotes except the bad count
   "T","Test.csv",8,"80045.96"
*** File num8.tst line count mismatch: 3/10.
   H,Test.csv,other rubbish goes here
   this file does have a trailer with no quotes and a bad count
   T,Test.csv,7,80045.96
*** File num9.tst line count mismatch: 3/9.