Warning This page still needs heavy tweaking!

Monitoring

• Nagios
• Tivoli
• CSM

System logs

System logs are critical components of an operating system that allow us to track and debug errors. They also provide more in-depth information about system performance and behavior.

Linux syslog

The system logging utility that we use in Linux is called "syslog". The syslog bundled inside Linux is derived from BSD sources and is bundled natively into the operating system. It is turned on by default on Linux. This tool allows you to trap kernel messages and capture system messages, with capabilities to customize the level of criticality that you choose to capture.
Syslog provides two daemons as part of the package, syslogd and klogd. The syslogd daemon provides the system logging facilities, and klogd provides the kernel logging facility.

Daemons are processes that runs continuously in the operating system and are started during the runlevel processing at boot time (for example, /etc/rc.d/rc<N>.d/S06syslog).

The core configuration file in the syslog configuration is located in /etc/syslog.conf. It lists the types of errors, levels, and where in the file the errors be directed to.

The /etc/syslog.conf file contains logging parameters in the form of application.levelofseverity. The syslog.conf allows multiple entries of the same application error in different lines or separated by semicolons, as shown in the
sample syslog configuration file in Example 4-6.

Example 4-6 Sample of /etc/syslog configuration file

# /etc/syslog.conf
kern.warn;*.err;authpriv.none /dev/tty10
kern.warn;*.err;authpriv.none|/dev/xconsole
*.emerg *
*.* /dev/tty12
mail.* -/var/log/mail
news.crit -/var/log/news/news.crit
news.err -/var/log/news/news.err
news.notice -/var/log/news/news.notice
*.=warn;*.=err;*.crit-/var/log/warn
*.*;mail.none;news.none-/var/log/messages

The incoming logs are basically logged to files accordingly to their classification.
Some of the common application classifications are listed in Table 4-1.

TABLE MISSING !!!

Based on the application classification, you can filter the messages based on their severity levels. Table 4-2 lists some of the severity levels that you can define inside your /etc/syslog.conf. They are listed in increasing severity.

TABLE MISSING !!!

System logs in a centralized environment

In an environment with more than two servers, syslog can be configured to allow servers to forward syslogs to a central server where all the system logs can be stored. Syslog allows this operation by forwarding the logs from the client to the server through the port 514/UDP.

First, you need to configure the syslog server to accept incoming logs. By default, this is turned off.

'''Server side'''

In the server side:

• Update /etc/sysconfig/syslog file under the SYSLOGD_PARAMS option.

  SYSLOGD_PARAMS="-r"
  

• Restart the syslog service by using the command /etc/init.d/syslog restart or you can use the command

  kill -HUP 'cat /var/run/syslogd.pid'
  

  Now you need to configure the client to forward the logs to the server. You can still have logs stored in the client machines as if it is a standalone server.

'''Client side'''

In the client side:

• Update /etc/syslog.conf with the respective configuration.
  In Example 4-7, all the logs will be forwarded to the p630sles /var/log/messages log file. You can customize it further to forward only necessary facility and criticality.

Example 4-7 Client syslog.conf

# /etc/syslog.conf
kern.warn;*.err;authpriv.none /dev/tty10
kern.warn;*.err;authpriv.none|/dev/xconsole
*.emerg *
*.* /dev/tty12
mail.* -/var/log/mail
news.crit -/var/log/news/news.crit
news.err -/var/log/news/news.err
news.notice -/var/log/news/news.notice
*.=warn;*.=err;*.crit-/var/log/warn
*.*;mail.none;news.none-/var/log/messages
*.*;mail.none;news.none-p630sles@/var/log/messages

• Restart syslog service using the command /etc/init.d/syslog restart
• Test logging using the logger command:

  logger -p local0.crit -t TEST testing
  

On the server side, check the /var/log/messages file. You should see that the "testing" message is logged as shown in Example 4-8 on page 178.

Example 4-8 Example of logger test with output in the /var/log/messages

Oct 21 13:18:52 p630sles kernel: Kernel logging (proc) stopped.
Oct 21 13:18:52 p630sles kernel: Kernel log daemon terminating.
Oct 21 13:18:53 p630sles exiting on signal 15
Oct 21 13:18:54 p630sles syslogd 1.4.1: restart (remote reception).
Oct 21 13:18:59 p630sles kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 21 13:18:59 p630sles kernel: Inspecting /boot/System.map-2.4.21-83-pseries64
Oct 21 13:18:59 p630sles kernel: Loaded 31486 symbols from /boot/System.map-2.4.21-83-pseries64.
Oct 21 13:18:59 p630sles kernel: Symbols match kernel version 2.4.21.
Oct 21 13:18:59 p630sles kernel: Loaded 110 symbols from 8 modules.
Oct 21 13:19:44 lpar8 syslogd 1.4.1: restart.
Oct 21 13:19:47 lpar8 TEST: testing
Oct 21 13:19:49 lpar8 kernel: klogd 1.4.1, log source = /proc/kmsg started.

Besides configuring syslog by hand, SLES8 bundles a graphical YaST2 tool to help you with the configuration. Figure 4-6 on page 179 shows the YaST2 utility for configuring system logging.

Event logging - enterprise event log

Event Logging or evlog is a set of tools that implements an enterprise-level event logging facility, according the POSIX draft standard. evlog stores event records in binary and uses two daemons: evlogd and evlogrmtd. The evlogd daemon is used to log the events, and evlogrmtd is for clients to send remote logs for the evlog server.

The evlog installable RPM is located inside the first CD1 in the directory suse/RPMS/ppc/.

1. 1. Install the evlog-1.5.3-36.rpm:

   # rpm -i evlog-1.5.3-36.rpm
   

2. 2. Create an empty file inside /var/log/evlog/ called eventlog:

   # touch /var/log/evlog/eventlog
   

3. 3. Start evlog services:

   # /etc/init.d/evlog start
   

4. 4. enable forward kernel and system logs to the evlog:

   # /sbin/slog_fwd
   

The evlog command stores the files in the directory /var/log/evlog. Table 4-3 lists some of the commands you can use the query, modify or update evlog services and logs.

TABLE MISSING !!!

Tip: If Linux is behaving strangely after enabling the forwarding of system log using the command /sbin/slog_fwd, you can disable it by using /sbin/slog_fwd -r or you can remove the file libevlsyslog.so.1 from /etc/ld.so.preload and run /sbin/ldconfig. This will remove the system logging forwarding service to evlog.

evlog in a centralized environment

The remote event consolidator, or evlogrmtd, is the daemon that evlog runs to accept incoming events from other hosts in the network. All the incoming events will be consolidated and stored into a single file for easy retrieval and searching.

You optionally set filters to check for certain more alarming words like "crit", "emerg", "error" and others.

First, you need to configure the server to listen for incoming logs and to allow specific clients for access.

'''Server side'''

On the server side, perform these steps:

1. 1. Login as root and create a new evlogrmtd.conf file from the template available from /etc/evlog.d/evlogrmtd.conf.sample. Example 4-9 shows what should be inside the evlogrmt.conf file.

Example 4-9 Sample of /etc/evlog.d/evlogrmt.conf file

# evlogrmtd.conf file syntax
Password=<cleartextpassword>
TCPPort=12000
UDPPort=34000

1. 2. Update /etc/evlog.d/evlhosts similar to what is shown in Example 4-10 for each client that you want evlog to receive events from. Each entry must have a resolvable hostname and also a unique identifier for each host. If you leave the identifier blank, it will be ignored by evlog.

Example 4-10 Sample of /etc/evlog.d/evlhosts file

#evlhosts file syntax
#
# hostid hostname
# ----------------
# 1 mylinuxbox
# 10.128 mylinuxbox2
100.79 lpar3

Change the permissions to read and write only for root.

# chmod 700 /etc/evlog.d/evlhosts

1. 3. Restart the evlog daemon and check that the evlogrmtd daemon is running.

   # /etc/init.d/evlog restart
   # ps -ef | grep evlogrmtd
   

In the client side, you will need to configure it to send logs in the corresponding port that the server is listening on.

'''Client side'''

On the client side, perform these steps:

1. 1. Login in as root and go into the /etc/evlog.d/ directory.
2. 2. If you plan to use UDP, edit udp_rmtlog_be.conf to specify the IP address of the server and Port number; see Example 4-11.

Example 4-11 udp_rmtlog_be.conf configuration file

# udp_rmtlog_be.conf file syntax
Remote Host=192.168.100.84
Port=34000
Disable=no

If you plan to use TCP, edit tcp_rmtlog_be.conf to specify the IP address of the server, Port number, Password and BufferLenInKbytes, as shown in Example 4-12.

Example 4-12 tcp_rmtlog_be.conf

# tcp_rmtlog_be.conf file syntax
Remote Host=192.168.100.84
Password=<clear_textpassword_from_management_server>
Port=12000
BufferLenInKbytes=128
Disable=no

1. 3. Finally, test whether your management host can receive the logging message from the newly added client.

   # evlsend -f user -t 1 -m testing from lpar3
   

   You should see the following at the evlog management server:

   # evlview -m
   Oct 22 15:41:14 lpar8 tcp_rmtlog_be: plugin is disabled.
   Oct 22 15:41:14 lpar8 tcp_rmtlog_be: plugin unloaded.
   Oct 22 15:41:14 lpar8 udp_rmtlog_be: plugin is disabled.
   Oct 22 15:41:14 lpar8 udp_rmtlog_be: plugin unloaded.
   Oct 22 17:12:59 lpar8 evlogrmtd: This 192.168.100.79 host is successfully authenticated.
   Oct 22 17:13:06 lpar3 testing from lpar3
   

Log rotation

Most of daemons like syslog append their events and logs into a file. As a result, the file size will get larger over time. To avoid this, Linux uses logrotate to manage logs. Logrotate is run daily by the cron; it can create and then zip up the files into their respective folders. The main configuration file of logrotate is located at /etc/logrotate.conf. Example 4-13 shows logrotate.conf configuration file in a standard SLES 8 server.

Example 4-13 Sample of logrotate configuration file (/etc/logrotate.conf)

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# uncomment these to switch compression to bzip2
#compresscmd /usr/bin/bzip2
#uncompresscmd /usr/bin/bunzip2

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own lastlog or wtmp -- we'll rotate them here
#/var/log/wtmp {
#    monthly
#    create 0664 root utmp
#    rotate 1
#}
# system-specific logs may be also be configured here.

Any application that you would like logrotate to manage should be created as a file and placed into the directory /etc/logrotate.d/; it will be automatically run by the cron daemon on daily basis, /etc/cron.daily.

Nagios

http://www.nagios.org/

Overview

Nagios® is a host and service monitor designed to inform you of network problems before your clients, end-users or managers do. It has been designed to run under the Linux operating system, but works fine under most *NIX variants as well. The monitoring daemon runs intermittent checks on hosts and services you specify using external "plugins" which return status information to Nagios. When problems are encountered, the daemon can send notifications out to administrative contacts in a variety of different ways (email, instant message, SMS, etc.). Current status information, historical logs, and reports can all be accessed via a web browser.

Features

Nagios has a lot of features, making it a very powerful monitoring tool. Some of the major features are listed below:

• Monitoring of network services (SMTP, POP3, HTTP, NNTP, PING, etc.)
• Monitoring of host resources (processor load, disk and memory usage, running processes, log files, etc.)
• Monitoring of environmental factors such as temperature
• Simple plugin design that allows users to easily develop their own host and service checks
• Ability to define network host hierarchy, allowing detection of and distinction between hosts that are down and those that are unreachable
• Contact notifications when service or host problems occur and get resolved (via email, pager, or other user-defined method)
• Optional escalation of host and service notifications to different contact groups
• Ability to define event handlers to be run during service or host events for proactive problem resolution
• Support for implementing redundant and distributed monitoring servers
• External command interface that allows on-the-fly modifications to be made to the monitoring and notification behavior through the use of event handlers, the web interface, and third-party applications
• Retention of host and service status across program restarts
• Scheduled downtime for supressing host and service notifications during periods of planned outages
• Ability to acknowlege problems via the web interface
• Web interface for viewing current network status, notification and problem history, log file, etc.
• Simple authorization scheme that allows you restrict what users can see and do from the web interface

Reporting

Inventory