Vserver watchdogs
Monitoring listening TCP/UDP processes
mynetstat
#!/bin/bash
# Copyright: 2007 Philippe Teuwen <phil at teuwen dot org>
# License: GPLv3 or later
#
# Description:
# Watch for modifications in the list of listening TCP/UDP processes across all vservers
# Examples: mysql, imapproxy,.. crashes
# IP binding broken after package upgrade
# new suspicious listening process, etc
#
# Usage:
# mynetstat [update|diff|diff+update] options
#
# mynetstat
# Gives a clean list of listening processes in which vserver and with which binding
# mynetstat update
# Creates a snapshot in a logfile
# mynetstat diff
# Shows diffs compared to the snapshot
# This is the one expected to be called by cron on the host
# mynetstat diff+update
# Same as diff but takes a new snapshot in case of changes
# so cron doesn't send repetitively alerts
# Options:
# You can use numeric options of netstat: [--numeric|-n] [--numeric-hosts] [--numeric-ports]
# Changelog:
# v1.0
###### Adapt this one to your needs:
LOGFILE="$(dirname $0)/mynetstat.snapshot"
####################################
POSTUPDATE=false
case "$1" in
"update")
shift
CMD="cat > $LOGFILE"
;;
"diff")
shift
CMD="diff -U0 $LOGFILE -"
;;
"diff+update")
shift
CMD="diff -U0 $LOGFILE -"
POSTUPDATE=true
;;
*)
CMD="cat"
esac
(
for i in $(/usr/sbin/vserver-stat |awk '/^[0-9]/{print $8}'); do
/usr/sbin/vserver $i exec /bin/netstat --ip -lp|\
/usr/bin/awk -v vs=$i '/^(tcp|udp)/{if ($7==""){$7=$6};printf "%-20s%-10s%-30s%s\n", vs, $1, $4, $7}'
done
/bin/netstat --ip -lp|/usr/bin/awk -v vs=HOST '/^(tcp|udp)/{if ($7==""){$7=$6};printf "%-20s%-10s%-30s%s\n", vs, $1, $4, $7}'
) |sort -b | eval $CMD
[ $? -eq 1 ] && $POSTUPDATE && exec $0 update $*
/etc/cron.d/watch-vservers
MAILTO=root */15 * * * * root /home/vservers/tools/watch/mynetstat diff
Monitoring running processes
myps
#!/bin/bash
# Copyright: 2007 Philippe Teuwen <phil at teuwen dot org>
# License: GPLv3 or later
#
# Description:
# Watch for modifications in the list of running processes across all vservers
#
# Usage:
# myps
# Gives a clean list of processes in which vserver
# myps update
# Creates a snapshot in a logfile
# Be careful that transient processes as those ran by this script itself will be there!
# myps update-filtered
# Same a bit smarter, it filters out processes of the day and according to
# extra rules written in LOGFILE_OPT
# myps missing
# Shows missing processes apparently not running but well in the reference snapshot
# This is the one expected to be called by cron on the host
# myps new
# Shows processes running but not present in the reference snapshot
# Be careful that transient processes as those ran by this script itself will be there!
# myps new-filtered
# Same a bit smarter, it filters out processes of the day and according to
# extra rules written in EXTRAFILTER
# Note that you could miss processes forking at abnormal rate as they would be "too young"
# This is one could be interestingly called by cron too
# myps missing+new-filtered
# Guess what...
# Changelog:
# v1.0
###### Adapt this one to your needs:
LOGFILE="$(dirname $0)/myps.snapshot"
EXTRAFILTER="$(dirname $0)/myps-filter.conf"
####################################
# Default passthrough filter:
FILTER="cat"
case "$1" in
"update")
shift
CMD="cat > $LOGFILE && echo \"Done. You can clean manually the snapshot from transient processes\""
;;
"update-filtered")
shift
# Filter our processes of today:
FILTER="egrep -v \"[ ]+ +[0-9]\""
# Filter out lines matching those in EXTRAFILTER
CMD="cat |grep -v -f $EXTRAFILTER > $LOGFILE && echo \"Done. You can clean manually the snapshot from transient processes\""
;;
"missing")
shift
CMD="diff -U0 $LOGFILE - |sed \"1,2d\" |grep \"^-\""
;;
"new")
shift
# Filter our processes of today:
FILTER="egrep -v \"[ ]+ +[0-9]\""
# Filter out lines matching those in EXTRAFILTER
CMD="diff -U0 $LOGFILE - |sed \"1,2d\" |grep \"^+\"|grep -v -f $EXTRAFILTER"
;;
"missing+new-filtered")
shift
$0 missing $*
exec $0 new-filtered $*
;;
*)
CMD="cat"
;;
esac
# To get also args of the command, change "pid,command" by "pid,comm"
/usr/sbin/vps ax -o pid,start_time,command|\
sed "1d;s/^[ 0-9]\\+//"|\
eval $FILTER|\
cut -c-14,21-|\
sort|\
uniq|\
eval $CMD
Some explanation:
myps-filter.conf is supposed to contain lines used by grep -v to clean the output from processes that can run but not necessarily so you don't want to hear about them
/etc/cron.d/watch-vservers
MAILTO=root 57 * * * * root /home/vservers/tools/watch/myps missing+new-filtered
We choose once per day among others at 23:57 and 0:57 so we avoid to capture processes of less than 1 hour but "of yesterday"
Monitoring and killing CPU-hungry logins
In some situations my CPU was burning like hell for days.
This was the vcontext/login process when using the command "vserver XX enter".
If it happens that the process is disconnected from its terminal it eats all the CPU :-(
I could reproduce the problem as follow:
- ssh to the server as normal user
- sudo su -
- vserver XX enter
- from another term, kill -9 one of the 2 su processes ("su -" or "-su")
- quit the current terminal
Now from another terminal you can see that
vps ax|grep " login"
shows a "login" process not attached to any tty (cf the "?") and using top you see it's running 100% cpu.
Actually the real process is called vcontext as seen by top, pgrep and as you can see in /proc/XXX/status but its command line was overwritten with "login\0\0\0\0\0\0\0\0\0\0\0..." and commands like vps or pgrep -f show you this name "login" as you can see in /proc/XXX/cmdline.
So I wrote a script and installed it in a cron to kill any such dandling process:
pids=$(/usr/sbin/vps axc|\
egrep ' ? +[RS] +[1-9][0-9]*:[0-9]+ +vcontext'|\
sed 's/^ *\([0-9]\+\) .*$/\1/')
[ "$pids" != "" ] && \
echo "Killing dandling login" && \
kill $pids
And /etc/cron.d/watch-vservers:
MAILTO=root */5 * * * * root /home/vservers/tools/kill-vcontext
This also shows something weird: processes in vservers don't appear with ps ax, ok, they appear with vps ax.
Now if I want to kill one of them from the host:
vkill 29825 vkill: vc_ctx_kill(): No such process chcontext --ctx 1 kill 29825 <no error but nothing happen neither because ctx1 can watch all but can only watch!> kill 29825 <works!!>
It seems that on newer versions (2.6.18?), such direct kill will not work => use vkill --xid <vserver1> <pid>.
Monitoring and restarting daemons
I have a big syslog-ng aggregator + dumping into mysql, cf Syslog
But sometimes during/after cron.daily (logrotate?), syslog-ng dies
I experienced similar problems with apache2 at logrotate weekly and with denyhosts
So I decided to monitor all my daemons and try to restart them automatically when there is a problem
mywatchdog
#!/bin/bash
# Copyright: Philippe Teuwen <phil at teuwen dot org>
# License: GPLv3 or later
#
# Usage:
# mywatchdog <vserver> <space-separated list of daemons>
# Checks in vserver if such daemons are running otherwise it tries to launch them with /etc/init.d/<daemon> start
# Special vserver MAIN refers to the host
#
# Note that <daemon> must be the name of the script in /etc/init.d/
# and must be usable as a pattern to find the process otherwise add a specific filter below
VSERVER=$1
shift
for i in $*; do
FILTER=${i}
# Special daemons to tweak as the process has another name
# -f greps on the full command line otherwise filter applies only on the process name
[ "${i}" == "backuppc" ] && FILTER=BackupPC
[ "${i}" == "courier-authdaemon" ] && FILTER=authdaemond
[ "${i}" == "courier-imap" ] && FILTER=couriertcpd
[ "${i}" == "denyhosts" ] && FILTER="-f denyhosts"
[ "${i}" == "openbsd-inetd" ] && FILTER=inetd
[ "${i}" == "ozyman" ] && FILTER="-f nomce.pl"
[ "${i}" == "postgresql-8.1" ] && FILTER=postmaster
[ "${i}" == "spamassassin" ] && FILTER=spamd
[ "${i}" == "sysklogd" ] && FILTER=syslogd
[ "${i}" == "uml-utilities" ] && FILTER=uml_switch
[ "${i}" == "zope2.9" ] && FILTER="-f zope2.9"
if [ "$VSERVER" == "MAIN" ]; then
if [ ! -e "/etc/init.d/${i}" ]; then
echo "Error in using mywatchdog script, /etc/init.d/${i} does not exist in MAIN!"
exit 1
fi
pgrep ${FILTER}\|logrotate\|apt-get >/dev/null || \
{
echo "Error! ${i}@${VSERVER} seems to be down! I try to restart it..." && \
/etc/init.d/${i} start;
}
else
if [ ! -e "/home/vservers/${VSERVER}/etc/init.d/${i}" ]; then
echo "Error in using mywatchdog script, /etc/init.d/${i} does not exist in ${VSERVER}!"
exit 1
fi
/usr/sbin/chcontext --ctx ${VSERVER} pgrep ${FILTER}\|logrotate\|apt-get >/dev/null || \
{
echo "Error! ${i}@${VSERVER} seems to be down! I try to restart it..." && \
/usr/sbin/vserver ${VSERVER} exec /etc/init.d/${i} start;
}
fi
done
/etc/cron.d/watch-vservers
I added to my /etc/cron.d/watch-vservers entries similar to
*/15 * * * * root : "WD MAIN" ;/home/vservers/tools/watch/mywatchdog MAIN acpid atop exim4 munin-node ssh denyhosts sysklogd udev xinetd uml-utilities */15 * * * * root : "WD MX" ;/home/vservers/tools/watch/mywatchdog mx syslog2mysql courier-authdaemon courier-imap cron exim4 fetchmail munin-node spamassassin ssh denyhosts syslog-ng xinetd */15 * * * * root : "WD NS0" ;/home/vservers/tools/watch/mywatchdog ns0 cron dnsmasq sysklogd */15 * * * * root : "WD PUBLIC" ;/home/vservers/tools/watch/mywatchdog public apache2 cron exim4 munin-node sysklogd */15 * * * * root : "WD SQL" ;/home/vservers/tools/watch/mywatchdog sql cron munin-node mysql sysklogd [...]
"WD VSERVER" are the first strings to appear in the mails titles, this helps me to quickly identify problems
I don't monitor cron on MAIN as if it's dead, well, the watchdogs won't be called :-/
Note that if you use logcheck, you will get security warnings because our calls contain "deny" when we want to check "denyhosts" so you could need to add sth like this in /etc/logcheck/violations.ignore.d/local-logcheck-watchdogs
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ /USR/SBIN/CRON\[[0-9]+\]: \(root\) CMD \(.*mywatchdog.*denyhosts.*\)$
Monitoring imapproxy
Imapproxy is in one vserver, the one with apache2 and squirrelmail and IMAP server is in another vserver, the mail server.
Because of racing between both vservers at boot time, imapproxy can fail to start if courier was not yet there
*/15 * * * * root /usr/sbin/chcontext --ctx vserverX pgrep imapproxy >/dev/null|| { /usr/sbin/chcontext --ctx vserverY pgrep couriertcp >/dev/null && { echo "Error! Imapproxy@vserverX seems to be down! I try to restart it..." && /usr/sbin/vserver vserverX exec /etc/init.d/imapproxy start; } }