Question

I’ve got a problem with my shell script. I am basically trying to create a new log file from (2) files using unix (zcat, cat) commands

FileA = gzipped logfile (.gz)
FileB = non-gzipped log file (.log)

FileC = new file from FileA:FileB

My problem is with FileC.

For example:
FileA has timestamp data from Aug19-Sept3
FileB has timestamp data from Sept4-Sept17
FileC has contents from both files = Aug19-Sept3:Sept4-Sept17

Problem:
If I manually run the commands on the server, the file looks fine as I expect FileC to have timestamp data from Aug19-Sept17
But, if done programmatically with the shell script, the local copy looks like this when I copy it back remotely:
FileC = Aug19-Sept17:Aug19-Sept2

So my question is why is this happening programmatically?
Is there a better way to combine/stitch these 2 logs together to create another one?

Here is my shell script:

#!/bin/bash
if [ $# != 1 ]; then
    echo "Usage: getlogs.sh <remote-host>" 2>&1
    exit 1
fi

#Declare variables
STAMP=`date '+%Y%m%d-%H:%M'`
REMOTE_MYCNF=/var/log/mysoft/mysoft.log
REMOTE_GZ=/var/log/mysoft/mysoft.log.1.gz
REMOTE_DIR=/var/log/mysoft/
BACKUP_DIR=/home/dev/logs/
NEWLOG="foo-temp.log"

#Copy file over
echo "START..." 2>&1
test -f $BACKUP_DIR$1.mysoft.log
if [ $? = 0 ]; then
   echo "Local log file $BACKUP_DIR$1.mysoft.log exists, clean up for new copy..." 2>&1
   /bin/rm $BACKUP_DIR$1.mysoft.log
   else
        echo "File does not exist, getting a new copy..." 2>&1
fi

echo "Checking remotely in $1 for logfile $REMOTE_MYCNF $STAMP" 2>&1
if [ ! -f $REMOTE_MYCNF ]; then
   echo "File exists remotely, creating new logfile and copy here...." 2>&1
   ssh $1 "zcat $REMOTE_GZ >> $REMOTE_DIR$NEWLOG"
   ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"
   /usr/bin/scp $1:$REMOTE_DIR$NEWLOG $BACKUP_DIR$1.mysoft.log
   echo "end remote copy" 2>&1
   echo "Cleaning up remote files" 2>&1
   ssh $1 "rm $REMOTE_DIR$NEWLOG"
   exit 0
   else
        echo "Unable to get file" 2>&1
        exit 0
fi

Answer 1

Perhaps it’s possible that the cat is starting before the zcat is finished. However, it would surprise me if this were the case.

Try this:

ssh $1 "zcat ... >> ...; cat ... >> ..."

or

ssh $1 "zcat ... >> ..."
sleep 5
ssh $1 "cat ... >> ..."

or

ssh $1 "zcat -f file1 file2 >> ..."   # let zcat do both files