Alert notifications

This is a generic page about reporting all kind of misbehaviours from a server.
This is draft, to be implemented :-)

Data collection

• Syslog
• Munin
• acct & atop
• procinfo
• exim stats
• apache stats
• netstat -l...

What to filter for what kind of alert?

Mail alerts

• Syslog -> Logcheck
• We should send also at least what we want to report via jabber/SMS
• Vserver watchdogs -> cron

Jabber/SMS alerts

You can use sendxmpp

apt-get install sendxmpp
echo "Here is a very cool notification agent"|\
   sendxmpp -u <user> -p <passwd> -j <server> -s <subject> <destinataires>

Note that v1.13-1 is buggy, apply this patch:

--- /usr/bin/sendxmpp    2008-09-21 00:51:15.000000000 +0200
+++ /usr/bin/sendxmpp    2008-09-21 00:47:32.000000000 +0200
@@ -281,7 +281,7 @@
                connectiontype  => 'tcpip',
                componentname   => $comp
        };
-       $arghash->{port} = $port if (!$port);
+       $arghash->{port} = $port if ($port);
        if (!$port) {
                @res = $cnx->Connect(%$arghash);
                error_exit ("Could not connect to server '$host': $@") unless @res;

• Hardware damages
  • temp, fans
  • raid
• Software damages
  • HD capacity
  • CPU load 100% for more than X mins
    The easiest is to take the third field of /proc/loadavg which is a mean over the last 15 mins, here with 2 CPUs:

awk '$3 > 2 {print "alert"}' /proc/loadavg

• • network load > X for more than Y mins
  • exim load > X mails sent per min

From sendxmpp author:

if test "$cpuload" -gt "$CPULOADMAX"; then
 top -b -n 1 | sendxmpp -s "wake up! cpu load $cpuload at `hostname`" someone@jabber.org  
fi

To get all logcheck mail and mails with subject starting with JABBER I added this to my procmail:

:0 c                                                                                                                                                                      
* ^From: .*logcheck|\                                                                                                                                                     
  ^Subject: JABBER                                                                                                                                                       
* ^Subject: \/.*                                                                                                                                                          
| formail -k -X From:|sendxmpp -u sendxmpp -p xmpp -j <jabber server> -s "$MATCH" <my@jabber.account>