I created a cron job as such:
*/10 * * * * sudo -u user $somepath/bin/cluvfy comp nodecon -n `hostname -f` > /var/log/interconnect/interconnect.out 2>&1
This cron job will run the "cluvfy" binary and send the results to a log file. This will happen every 10 minutes.
Then, I created a Nagios plugin as such:
#!/bin/bash
#
#Simple script to keep a tab on interconnect
#
addinfo="some additional information"
isupdated=`find /var/log/interconnect/interconnect.out -mmin -15`
isok=`tail -1 /var/log/interconnect/interconnect.out | grep 'Verification of node connectivity was successful'`
if [[ $isupdated ]] && [[ $isok ]];then
echo "Ok - interconnect is updated within 10 minutes and did not fail"
exit 0
else
echo "Critical - $addinfo"
exit 2
fi
This plugin checks to make sure that the interconnect.out file has been updated within the last 15 minutes. We want to make sure the file is fresh.
Then, we look at the last line to see if we find "Verification of node connectivity was successful".
If these two conditions are true, then everything is ok and we can exit with a 0. Otherwise, we throw a critical alert and exit with a 2.
