Some times, especially in a production environment you may want to tail logs and do something with it such as monitoring something. However this can put (heavy) load on the server and this is something you don’t ever want to do. But how do you tail a file remotely to keep the load off? You’ve come to the right place!

Here you see a code snippet – that is ideal for a crontab – you may use any of this code freely as long as you reference sshadmincontrol.com somewhere in your code. Everything is explained in the comments.

#!/bin/bash
#Code Snippet from and copyright by sshadmincontrol.com
#You may use this code freely as long as you keep this notice.
 
hostname=$1 #host name to remotlely access
log_dir=$2  #log directory on the remotehost
log_file=$3 #remote log file name
username=$3 #username to use to access remote host
log_base=$4 #where to save the log locally
 
ORIGLOG="$log_base/$hostname/${log_file}.orig"
INTERLOG="$log_base/$hostname/${log_file}.inter"
#The file below is the file you will tailing locally
FINALLOG="$log_base/$hostname/${log_file}.log"
 
#We will use rsync to keep a local copy up to date
#This creates a new i-node each time so is unsuitable to tail -f
rsync -q -e ssh $username@$hostname:$log_dir/$log_file ${ORIGLOG}
 
#Optinally remove some lines from the log file that we are not
#interested in which may slow down processing
grep -Ev ".ico|.jpg|.gif|.png|.css" > ${INTERLOG}  
 
#If the logfile for tailing doesn't exist yet then create it.
if [ ! -e $FINALLOG ]; then
   cp  ${INTERLOG} ${FINALLOG}
else
#Get the last line that we wrote to the file for tailing
   LINE=`tail -1 ${FINALLOG}`
#Add the new lines to the file for tailing
   grep -F "$LINE" -A 999999999 ${INTERLOG} \
      | grep -Fv "$LINE" >> ${FINALLOG}
fi

That’s it. Well almost.

You will have a problem if this script runs simultanously – I run this script from the crontab every minute. That is it may happen that it will start up again before the previous invocation had finished. See our other tutorial how to avoid this.

Using what we learned in that tutorial lets put it all together here:

#!/bin/bash
#Code Snippet from and copyright by sshadmincontrol.com
#You may use this code freely as long as you keep this notice.
 
PIDHOME=/a_place/to/store/flag/file
FILE=`echo ${0} | sed 's:.*/::'`
RUNFILEFLAG=${PIDHOME}/${FILE}.running
 
if [ -e $RUNFILEFLAG ]; then
   echo "Already running ${RUNFILEFLAG}"
   exit 1
else
   touch ${RUNFILEFLAG}
fi
 
hostname=$1 #host name to remotlely access
log_dir=$2  #log directory on the remotehost
log_file=$3 #remote log file name
username=$3 #username to use to access remote host
log_base=$4 #where to save the log locally
 
ORIGLOG="$log_base/$hostname/${log_file}.orig"
INTERLOG="$log_base/$hostname/${log_file}.inter"
FINALLOG="$log_base/$hostname/${log_file}.log"
 
rsync -q -e ssh $username@$hostname:$log_dir/$log_file ${ORIGLOG}
grep -Ev ".ico|.jpg|.gif|.png|.css" > ${INTERLOG}  
 
if [ ! -e $FINALLOG ]; then
   cp  ${INTERLOG} ${FINALLOG}
else
   LINE=`tail -1 ${FINALLOG}`
   grep -F "$LINE" -A 999999999 ${INTERLOG} \
      | grep -Fv "$LINE" >> ${FINALLOG}
fi
 
rm ${RUNFILEFLAG}
exit 0

Now all you need to do is tail the file defined in $FINALLOG locally and do what you want with it.

Also don’t forget to try out the monkey tool, it’s a great time saver.

Go Back to the tutorial menu page.