check_logfiles is a Plugin for Nagios which scans log files for specific patterns.

The conventional plugins which scan log files are not adequate in a mission critical environment. Especially the missing ability to handle logfile rotation and inclusion of the rotated archives in the scan allow gaps in the monitoring. Check_logfiles was written because these deficiencies would have prevented Nagios from replacing a propritetary monitoring system.

• Detection of rotations - usually nightly logfiles are rotated and compressed. Each operating system or company has it's own naming scheme. If this rotation is done between two runs of check_logfiles also the rotated archive has to be scanned to avoid gaps. The most common rotation schemes are predefined but you can describe any strategy (shortly: where and under which name is a logfile archived).
• More than one pattern can be defined which again can be classified as warning patterns and critical patterns.
• Triggered actions - Usually nagios plugins return just an exit code and a line of text, describing the result of the check. Sometimes, however, you want to run some code during the scan every time you got a hit. Check_logfiles lets you call scripts either after every hit or at the beginning or the end of it's runtime.
• Exceptions - If a pattern matches, the matched line could be a very special case which should not be counted as an error. You can define exception patterns which are more specific versions of your critical/warning patterns. Such a match would then cancel an alert.
• Thresholds - You can define the number of matching lines which are necessary to activate an alert.
• Protocol - The matching lines can be written to a protocol file the name of which will be included in the plugin's output.
• Macros - Pattern definitions and logfile names may contain macros, which are resolved at runtime.
• Performance data - The number of lines scanned and the number of warnings/criticals is output.
• Windows - The plugin works with Unix as well as with Windows (e.g. with ActiveState Perl).

Usually you call the plugin with the -f option which gets the name of a configuration file:

nagios:~> check_logfiles -f <configfile>
OK - no errors or warnings

In it's most simple form check_logfiles can get all the essential parameters as command line options. However, not all features can be utilized in this case.

nagios:~> check_logfiles --tag=ssh --logfile=/var/adm/messages → ...
    → --rotation=SOLARIS → ...
    → --criticalpattern="Failed password for root" 
OK - no errors or warnings |ssh=1722;0;0;0
nagios:~> check_logfiles --tag=ssh --logfile=/var/adm/messages → ...
    → --rotation=SOLARIS → ...
    → --criticalpattern="Failed password for root" 
CRITICAL - (1 errors in check_logfiles.protocol-2007-04-25-20-59-20) →
    → - Apr 25 20:59:15 siapp8 sshd[10849]: → ...
    → [ID 800047 auth.info] Failed password for root → ...
    → from 172.16.224.11 port 24206 ssh2 |ssh=2831;0;1;0

In principle check_logfiles scans a log file until the end-of-file is reached. The offset will then be saved in a so-called seekfile. The next time check_logfiles runs, this offset will be used as the starting position inside the log file. In the event that a rotation has occurred in the meantime, the rest of the rotated archive will be scanned also.

For the most simple applications it is sufficient to call check_logfile with command line parameters. More complex scan jobs can be described with a config file.

• --tag=<identifier> A short unique descriptor for this search. It will appear in the output of the plugin and is used to separare the different services.
• --logfile=<filenname> This is the name of the log file you want to scan.
• --rotation=<method> This is the method how log files are rotated.
• --criticalpattern=<regexp> A regular expression which will trigger a critical error.
• --warningpattern=<regexp> The same...a match results in a warning.
• --criticalexception=<regexp> / --warningexception=<regexp> Exceptions which are not counted as errors.
• --okpattern=<regexp> A pattern which resets the error counters.
• --noprotocol Normally all the matched lines are written into a protocol file with this file's name appearing in the plugin's output. This option switches this off.
• --syslogserver With this option you limit the pattern matching to lines originating from the host check_logfiles is running on.
• --syslogserver With this option you limit the pattern matching to lines originating from the host named in this option.
• --sticky[=<lifetime>] Errors are propagated through successive runs.
• -f <configfile> The name of a configuration file. The syntax of this file is described in the next section.
• -F <configdir> The name of a configuration directory. Configfiles ending in .cfg or .conf are (recursively) imported.
• --searches=<tag1,tag2,...> A list of tags of those searches which are to be run. Using this parameter, not all searches listed in the config file are run, but only those selected. (--selectedsearches is also possible)

The definitions in this file are written with Perl-syntax. There is a distinction between global variables which influence check_logfiles as a whole and variables which are related to the single searches. A "search" combines where to search, what to search for, which weight a hit has, which action will be triggered in case of a hit, and so on...

These macros are also available in scripts called out of check_logfiles. Their values are stored in environment variables, whose names are derived from the macro's names. The preceding CL_ is replaced by CHECK_LOGFILES_. You can also access user defined macros. Their names are also prefixed with CHECK_LOGFILES_.

nagios:~> cat check_logfiles.cfg
$scriptpath = '/usr/bin/my_application/bin:/usr/local/nagios/contrib';
$MACROS = {
    MY_FUNNY_MACRO => 'hihihihohoho',
    MY_VOLUME => 'loud'
};

@searches = (
  {
    tag => 'fun',
    logfile => '/var/adm/messages',
    criticalpatterns => 'a funny pattern',
    script => 'laugh.sh',
    scriptparams => '$MY_VOLUME$',
    options => 'noprotocol,script,perfdata'
  },
);



nagios:~> cat /usr/bin/my_application/bin/laugh.sh
#! /bin/sh
if [ -n "$1" ]; then
  VOLUME=$1
fi
printf "It is %d:%d and my status is %s\n" \
  $CHECK_LOGFILES_DATE_HH \
  $CHECK_LOGFILES_DATE_MI \
  $CHECK_LOGFILES_SERVICESTATE

printf "I found something funny: %s\n" "$CHECK_LOGFILES_SERVICEOUTPUT"
if [ "X$VOLUME" == "Xloud" ]; then
  echo "$CHECK_LOGFILES_MY_FUNNY_MACRO" | tr 'a-z' 'A-Z'
else
  echo "$CHECK_LOGFILES_MY_FUNNY_MACRO"
fi  
printf "Thank you, %s. You made me laugh.\n" "$CHECK_LOGFILES_USERNAME"

The number of scanned lines as well as the number of pattern matches (critical, warning and unknown) are appended to the plugin's output in performance data format. You can suppress this by using the noperfdata option.

nagios:~> check_logfiles --logfile=/var/adm/messages → ...
    → --criticalpattern="Failed password" --tag=ssh
CRITICAL - (4 errors) - May  9 11:33:12 localhost sshd[29742] → ...
    → Failed password for invalid user8 ... |ssh_lines27 → ...
    → ssh_warnings=0 ssh_criticals=4 ssh_unknowns=0

nagios:~> check_logfiles --logfile=/var/adm/messages → ...
    → --criticalpattern="Failed password" --tag=ssh --noperfdata
CRITICAL - (2 errors) - May  9 11:58:48 localhost sshd[29813] → ...
    → Failed password for invalid user8 ... 

It is possible to execute external scripts out of check_logfiles. This can be at the startup phase ($prescript), before termination ($postscript) or every time a pattern matches a line. See example above.

With the option "smartscript" output and exitcode of the script are treated like a match in the logfile and reflected in the overall result. The option "supersmartscript" makes output and exitcode of the script replace those of the triggering match.

Pre- and Postscript declared as supersmart scripts directly influence the process of check_logfiles. The option "supersmartprescript" causes an immediate abort of check_logfiles if the prescript has a non-zero exit code. In this case output and exitcode of check_logfiles correspond to those of the prescript. With the option "supersmartpostscript" output and exitcode of check_logfiles can be determined by the postscript. Thus a more meaningful output is possible.

If you have just one service which uses check_logfiles you can hard-code the config file in your services.cfg/nrpe.cfg

define service {
  service_description   check_sanlogs
  host_name              oaschgeign.muc
  check_command       check_nrpe!check_logfiles
  is_volatile           1
  check_period          7x24
  max_check_attempts    1
  ...
}

define command {
  command_name          check_nrpe
  command_line          $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

command[check_logfiles]=/opt/nagios/libexec/check_logfiles → ...
    → -f logdefs.cfg

If multiple services are based on check_logfiles you need multiple config files. I propose to name them after the service_description. In the following example we would have a directory cfg.d with config files solaris_check_sanlogs and solaris_check_apachelogs.

define service {
  service_description  logfilescan
  register             0
  is_volatile          1
  check_period         7x24
  max_check_attempts   1
  ...
}

define service {
  service_description  solaris_check_sanlogs
  host_name            oaschgeign.muc
  check_command     → ...
      → check_nrpe_arg!20!check_logfiles!cfg.d/$SERVICEDESC$
  contact_group        sanadmin
  use                  logfilescan
}

define service {
  service_description  solaris_check_apachelogs
  host_name            oaschgeign.muc
  check_command     → ...
      → check_nrpe_arg!20!check_logfiles!cfg.d/$SERVICEDESC$
  contact_group        webadmin
  use                  logfilescan
}

define command {
  command_name         check_nrpe_arg
  command_line         $USER1$/check_nrpe → ...
      → -H $HOSTADDRESS$ -t $ARG1$ -c $ARG2$ -a $ARG3$
}

# nrpe.cfg von Host 
[check_logfiles]=/opt/nagios/libexec/check_logfiles -f $ARG1$

The corresponding line in the host's nrpe.cfg looks like that:

[check_logfiles]=/opt/nagios/libexec/check_logfiles -f $ARG1$

If you use nsclient++ under Windows, the entry in the NSC.ini looks like that:

check_logfiles=C:\Perl\bin\perl C:\libexec\check_logfiles -f $ARG1$

• After unpacking the tar-archive you have to call ./configure. With ./configure --help you can show the options if you want to modify the default settings. However, these settings can later be overridden again by variables in the config file.
• Linux systems are more restrictive regarding the permission of log files. The /var/log/messages file is not readable for non-root users. If you run check_logfiles as an unprivileged user, follow the link below and look for a trick in the examples.
• --prefix=BASEDIRECTORY Specify here the directory where you want to install check_logfiles. (default: /usr/local/nagios)
• --with-nagios-user=SOMEUSER The user which will own the check_logfiles script. (default: nagios)
• --with-nagios-group=SOMEGROUP The group (default: nagios)
• --with-perl=PATH_TO_PERL The path to your perl binary. (default: The perl in the current PATH)
• --with-gzip=PATH_TO_GZIP The path to your gzip binary. (default: The gzip in the current PATH)
• --with-trusted-path=PATH_YOU_TRUST The path where you expect your triggered scripts. (default: /sbin:/usr/sbin:/bin:/usr/bin)
• --with-seekfiles-dir=SEEKFILES_DIR The directory where status files will be kept. (default: /tmp)
• --with-protocols-dir=PROTOCOLS_DIR The directory where protocol files will be written to. (default: /tmp)
• Under Windows you build the plugin with perl winconfig.pl

If you want to scan the alert log of an oracle database without having access to the database server on the operating system level (e.g. it is a Windows server or you are not allowed to log in to a Unix server for security reasons) and therefore no access to the alert file, then this file can be mapped to a database table. The contents of the file are then visible through a database connection by executing SQL SELECT statements. If you specify the type "oraclealertlog" in a check_logfiles configuration, this method is used to scan the alert log. You need some extra parameters in the configuration.

# extra parameters in the configuration file
@searches = ({
  tag => "oratest",
  type => "oraclealertlog",
  oraclealertlog => {
    connect => "db0815",       # connect identifier
    username => "nagios",      # database user
    password => "hirnbrand",   # database password
  },
  criticalpatterns => [
...

Maping external files to database tables is possible since Version 9. Use this script to prepare your database.

Installation of the Perl-Modules DBI and DBD::Oracle (http://search.cpan.org/~pythian/DBD-Oracle-1.21/Oracle.pm).

Here you can find example configurations for several scenarios.

• Nagios Homepage
• Nagios Plugins Exchange
• German Nagios Portal
• German Nagios Wiki

• 2008-05-07 2.4 Support for Oracle Alertlogs through a database connection.
• 2008-05-06 2.3.3 Option -F which is used to search multiple configfiles in a directory.
• 2008-02-26 2.3.2.1 Bugfix to support Perl 5.10. More encoding tinkering.
• 2008-02-12 2.3.2 Support for IPMI System Event Log, Errpt Bugfix, ucs-2 encoded files for Windows.
• 2007-12-27 2.3.1.2 Can now handle very large files, $CL_PROTOCOLFILE$, $CL_SERVICEPERFDATA$, more commandline options.
• 2007-11-16 2.3.1.1 Bugfix in sticky code. Thanks Marc Richter. New option savethresholdcount. Thanks Hannu Kivimäki.
• 2007-10-16 2.3.1 Templates, bzip2 archives, scriptparam bugfix, threshold counters are inherited.
• 2007-09-10 2.3 Bugfixes. Type errpt. Okpatterns. Options sticky and syslogclient. New format for performance data.
• 2007-06-08 2.2.4.1 Bugfix (--searches)
• 2007-06-06 2.2.4 Support for "virtual" files like Linux /proc/*
• 2007-06-05 2.2.3 Bugfixes
• 2007-06-02 2.2.2 Support for supersmart scripts with empty output.
• 2007-06-01 2.2.1 Smart scripts. Scripts can be embedded perl code.
• 2007-05-21 2.1.1 Bugfixes
• 2007-05-21 2.1 Native Windows now supported. New option --selectedsearches. New rotation method mod_log_rotate.
• 2007-05-10 2.0 Complete Redesign. Official handling of non-rotating logfiles. Performancedata.

2007 Gerhard Laußer

Check_logfiles is released under the GNU General Public License. GPL

Gerhard Laußer (gerhard.lausser@consol.de) will gladly answer your questions.