IBM®
Skip to main content
    Country/region [select]      Terms of use
 
 
    
     Home      Products      Services & solutions      Support & downloads      My account     
 
developerworks > My developerWorks >  Dashboard > Linux for Power Architecture > ... > Previous Home > Systems Management Integration > Information > Page Comparison
developerWorks
Log In   View a printable version of the current page.
Overview Connect Spaces Forums Wikis
Systems Management Integration
Browse space Browse Space    
Version 4 by mperzl
on Dec 14, 2005 19:11.

<< Changes from 3 to 4
compared with
Current by wburos
on Apr 14, 2009 10:41.
(show comment)
 
Key
These lines were removed. This word was removed.
These lines were added. This word was added.

View page history


There are 1 changes. View first change.

   
 {warning:title=Warning}
 This page still needs heavy tweaking!
 {warning}
  
 h3. Monitoring
  
 - Nagios
 - Tivoli
 - CSM
  
 h2. System logs
 System logs are critical components of an operating system that allow us to track and debug errors. They also provide more in-depth information about system performance and behavior.
  
 h3. Linux syslog
 The system logging utility that we use in Linux is called "syslog". The syslog bundled inside Linux is derived from BSD sources and is bundled natively into the operating system. It is turned on by default on Linux. This tool allows you to trap kernel messages and capture system messages, with capabilities to customize the level of criticality that you choose to capture.
 Syslog provides two daemons as part of the package, syslogd and klogd. The syslogd daemon provides the system logging facilities, and klogd provides the kernel logging facility.
  
 Daemons are processes that runs continuously in the operating system and are started during the runlevel processing at boot time (for example, /etc/rc.d/rc<N>.d/S06syslog).
  
 The core configuration file in the syslog configuration is located in /etc/syslog.conf. It lists the types of errors, levels, and where in the file the errors be directed to.
  
 The /etc/syslog.conf file contains logging parameters in the form of application.levelofseverity. The syslog.conf allows multiple entries of the same application error in different lines or separated by semicolons, as shown in the
 sample syslog configuration file in Example 4-6.
  
 Example 4-6 Sample of /etc/syslog configuration file
 {noformat}
 # /etc/syslog.conf
 kern.warn;*.err;authpriv.none /dev/tty10
 kern.warn;*.err;authpriv.none|/dev/xconsole
 *.emerg *
 *.* /dev/tty12
 mail.* -/var/log/mail
 news.crit -/var/log/news/news.crit
 news.err -/var/log/news/news.err
 news.notice -/var/log/news/news.notice
 *.=warn;*.=err;*.crit-/var/log/warn
 *.*;mail.none;news.none-/var/log/messages
 {noformat}
  
 The incoming logs are basically logged to files accordingly to their classification.
 Some of the common application classifications are listed in Table 4-1.
  
  TABLE MISSING !!!
  
 Based on the application classification, you can filter the messages based on their severity levels. Table 4-2 lists some of the severity levels that you can define inside your /etc/syslog.conf. They are listed in increasing severity.
  
  TABLE MISSING !!!
  
 h4. System logs in a centralized environment
 In an environment with more than two servers, syslog can be configured to allow servers to forward syslogs to a central server where all the system logs can be stored. Syslog allows this operation by forwarding the logs from the client to the server through the port 514/UDP.
  
 First, you need to configure the syslog server to accept incoming logs. By default, this is turned off.
  
 '''Server side'''
  
 In the server side:
 * Update /etc/sysconfig/syslog file under the SYSLOGD_PARAMS option.
 {noformat}
 SYSLOGD_PARAMS="-r"
 {noformat}
 * Restart the syslog service by using the command /etc/init.d/syslog restart or you can use the command
 {noformat}
 kill -HUP 'cat /var/run/syslogd.pid'
 {noformat}
 Now you need to configure the client to forward the logs to the server. You can still have logs stored in the client machines as if it is a standalone server.
  
 '''Client side'''
  
 In the client side:
 * Update /etc/syslog.conf with the respective configuration.
 In Example 4-7, all the logs will be forwarded to the p630sles /var/log/messages log file. You can customize it further to forward only necessary facility and criticality.
  
 Example 4-7 Client syslog.conf
 {noformat}
 # /etc/syslog.conf
 kern.warn;*.err;authpriv.none /dev/tty10
 kern.warn;*.err;authpriv.none|/dev/xconsole
 *.emerg *
 *.* /dev/tty12
 mail.* -/var/log/mail
 news.crit -/var/log/news/news.crit
 news.err -/var/log/news/news.err
 news.notice -/var/log/news/news.notice
 *.=warn;*.=err;*.crit-/var/log/warn
 *.*;mail.none;news.none-/var/log/messages
 *.*;mail.none;news.none-p630sles@/var/log/messages
 {noformat}
  
 * Restart syslog service using the command /etc/init.d/syslog restart
 * Test logging using the logger command:
 {noformat}
 logger -p local0.crit -t TEST testing
 {noformat}
  
 On the server side, check the /var/log/messages file. You should see that the "testing" message is logged as shown in Example 4-8 on page 178.
  
 Example 4-8 Example of logger test with output in the /var/log/messages
 {noformat}
 Oct 21 13:18:52 p630sles kernel: Kernel logging (proc) stopped.
 Oct 21 13:18:52 p630sles kernel: Kernel log daemon terminating.
 Oct 21 13:18:53 p630sles exiting on signal 15
 Oct 21 13:18:54 p630sles syslogd 1.4.1: restart (remote reception).
 Oct 21 13:18:59 p630sles kernel: klogd 1.4.1, log source = /proc/kmsg started.
 Oct 21 13:18:59 p630sles kernel: Inspecting /boot/System.map-2.4.21-83-pseries64
 Oct 21 13:18:59 p630sles kernel: Loaded 31486 symbols from /boot/System.map-2.4.21-83-pseries64.
 Oct 21 13:18:59 p630sles kernel: Symbols match kernel version 2.4.21.
 Oct 21 13:18:59 p630sles kernel: Loaded 110 symbols from 8 modules.
 Oct 21 13:19:44 lpar8 syslogd 1.4.1: restart.
 Oct 21 13:19:47 lpar8 TEST: testing
 Oct 21 13:19:49 lpar8 kernel: klogd 1.4.1, log source = /proc/kmsg started.
 {noformat}
  
 Besides configuring syslog by hand, SLES8 bundles a graphical YaST2 tool to help you with the configuration. Figure 4-6 on page 179 shows the YaST2 utility for configuring system logging.
  
 h2. Event logging - enterprise event log
 Event Logging or evlog is a set of tools that implements an enterprise-level event logging facility, according the POSIX draft standard. evlog stores event records in binary and uses two daemons: evlogd and evlogrmtd. The evlogd daemon is used to log the events, and evlogrmtd is for clients to send remote logs for the evlog server.
  
 The evlog installable RPM is located inside the first CD1 in the directory suse/RPMS/ppc/.
 # 1. Install the evlog-1.5.3-36.rpm:
 {noformat}
 # rpm -i evlog-1.5.3-36.rpm
 {noformat}
 # 2. Create an empty file inside /var/log/evlog/ called eventlog:
 {noformat}
 # touch /var/log/evlog/eventlog
 {noformat}
 # 3. Start evlog services:
 {noformat}
 # /etc/init.d/evlog start
 {noformat}
 # 4. enable forward kernel and system logs to the evlog:
 {noformat}
 # /sbin/slog_fwd
 {noformat}
  
 The evlog command stores the files in the directory /var/log/evlog. Table 4-3 lists some of the commands you can use the query, modify or update evlog services and logs.
  
  TABLE MISSING !!!
  
 Tip: If Linux is behaving strangely after enabling the forwarding of system log using the command /sbin/slog_fwd, you can disable it by using /sbin/slog_fwd -r or you can remove the file libevlsyslog.so.1 from /etc/ld.so.preload and run /sbin/ldconfig. This will remove the system logging forwarding service to evlog.
  
 h3. evlog in a centralized environment
 The remote event consolidator, or evlogrmtd, is the daemon that evlog runs to accept incoming events from other hosts in the network. All the incoming events will be consolidated and stored into a single file for easy retrieval and searching.
  
 You optionally set filters to check for certain more alarming words like "crit", "emerg", "error" and others.
  
 First, you need to configure the server to listen for incoming logs and to allow specific clients for access.
  
 '''Server side'''
  
 On the server side, perform these steps:
  
 # 1. Login as root and create a new evlogrmtd.conf file from the template available from /etc/evlog.d/evlogrmtd.conf.sample. Example 4-9 shows what should be inside the evlogrmt.conf file.
  
 Example 4-9 Sample of /etc/evlog.d/evlogrmt.conf file
 {noformat}
 # evlogrmtd.conf file syntax
 Password=<cleartextpassword>
 TCPPort=12000
 UDPPort=34000
 {noformat}
  
 # 2. Update /etc/evlog.d/evlhosts similar to what is shown in Example 4-10 for each client that you want evlog to receive events from. Each entry must have a resolvable hostname and also a unique identifier for each host. If you leave the identifier blank, it will be ignored by evlog.
  
 Example 4-10 Sample of /etc/evlog.d/evlhosts file
 {noformat}
 #evlhosts file syntax
 #
 # hostid hostname
 # ----------------
 # 1 mylinuxbox
 # 10.128 mylinuxbox2
 100.79 lpar3
 {noformat}
  
 Change the permissions to read and write only for root.
 {noformat}
 # chmod 700 /etc/evlog.d/evlhosts
 {noformat}
  
 # 3. Restart the evlog daemon and check that the evlogrmtd daemon is running.
 {noformat}
 # /etc/init.d/evlog restart
 # ps -ef | grep evlogrmtd
 {noformat}
  
 In the client side, you will need to configure it to send logs in the corresponding port that the server is listening on.
  
 '''Client side'''
  
 On the client side, perform these steps:
  
 # 1. Login in as root and go into the /etc/evlog.d/ directory.
 # 2. If you plan to use UDP, edit udp_rmtlog_be.conf to specify the IP address of the server and Port number; see Example 4-11.
  
 Example 4-11 udp_rmtlog_be.conf configuration file
 {noformat}
 # udp_rmtlog_be.conf file syntax
 Remote Host=192.168.100.84
 Port=34000
 Disable=no
 {noformat}
  
 If you plan to use TCP, edit tcp_rmtlog_be.conf to specify the IP address of the server, Port number, Password and BufferLenInKbytes, as shown in Example 4-12.
  
 Example 4-12 tcp_rmtlog_be.conf
 {noformat}
 # tcp_rmtlog_be.conf file syntax
 Remote Host=192.168.100.84
 Password=<clear_textpassword_from_management_server>
 Port=12000
 BufferLenInKbytes=128
 Disable=no
 {noformat}
  
 # 3. Finally, test whether your management host can receive the logging message from the newly added client.
 {noformat}
 # evlsend -f user -t 1 -m testing from lpar3
 {noformat}
 You should see the following at the evlog management server:
 {noformat}
 # evlview -m
 Oct 22 15:41:14 lpar8 tcp_rmtlog_be: plugin is disabled.
 Oct 22 15:41:14 lpar8 tcp_rmtlog_be: plugin unloaded.
 Oct 22 15:41:14 lpar8 udp_rmtlog_be: plugin is disabled.
 Oct 22 15:41:14 lpar8 udp_rmtlog_be: plugin unloaded.
 Oct 22 17:12:59 lpar8 evlogrmtd: This 192.168.100.79 host is successfully authenticated.
 Oct 22 17:13:06 lpar3 testing from lpar3
 {noformat}
  
 h2. Log rotation
 Most of daemons like syslog append their events and logs into a file. As a result, the file size will get larger over time. To avoid this, Linux uses logrotate to manage logs. Logrotate is run daily by the cron; it can create and then zip up the files into their respective folders. The main configuration file of logrotate is located at /etc/logrotate.conf. Example 4-13 shows logrotate.conf configuration file in a standard SLES 8 server.
  
 Example 4-13 Sample of logrotate configuration file (/etc/logrotate.conf)
 {noformat}
 # see "man logrotate" for details
 # rotate log files weekly
 weekly
  
 # keep 4 weeks worth of backlogs
 rotate 4
  
 # create new (empty) log files after rotating old ones
 create
  
 # uncomment this if you want your log files compressed
 #compress
  
 # uncomment these to switch compression to bzip2
 #compresscmd /usr/bin/bzip2
 #uncompresscmd /usr/bin/bunzip2
  
 # RPM packages drop log rotation information into this directory
 include /etc/logrotate.d
  
 # no packages own lastlog or wtmp -- we'll rotate them here
 #/var/log/wtmp {
 # monthly
 # create 0664 root utmp
 # rotate 1
 #}
 # system-specific logs may be also be configured here.
 {noformat}
  
 Any application that you would like logrotate to manage should be created as a file and placed into the directory /etc/logrotate.d/; it will be automatically run by the cron daemon on daily basis, /etc/cron.daily.
  
 h2. Nagios
 http://www.nagios.org/
  
 h3. Overview
 Nagios® is a host and service monitor designed to inform you of network problems before your clients, end-users or managers do. It has been designed to run under the Linux operating system, but works fine under most *NIX variants as well. The monitoring daemon runs intermittent checks on hosts and services you specify using external "plugins" which return status information to Nagios. When problems are encountered, the daemon can send notifications out to administrative contacts in a variety of different ways (email, instant message, SMS, etc.). Current status information, historical logs, and reports can all be accessed via a web browser.
  
 h3. Features
 Nagios has a lot of features, making it a very powerful monitoring tool. Some of the major features are listed below:
 * Monitoring of network services (SMTP, POP3, HTTP, NNTP, PING, etc.)
 * Monitoring of host resources (processor load, disk and memory usage, running processes, log files, etc.)
 * Monitoring of environmental factors such as temperature
 * Simple plugin design that allows users to easily develop their own host and service checks
 * Ability to define network host hierarchy, allowing detection of and distinction between hosts that are down and those that are unreachable
 * Contact notifications when service or host problems occur and get resolved (via email, pager, or other user-defined method)
 * Optional escalation of host and service notifications to different contact groups
 * Ability to define event handlers to be run during service or host events for proactive problem resolution
 * Support for implementing redundant and distributed monitoring servers
 * External command interface that allows on-the-fly modifications to be made to the monitoring and notification behavior through the use of event handlers, the web interface, and third-party applications
 * Retention of host and service status across program restarts
 * Scheduled downtime for supressing host and service notifications during periods of planned outages
 * Ability to acknowlege problems via the web interface
 * Web interface for viewing current network status, notification and problem history, log file, etc.
 * Simple authorization scheme that allows you restrict what users can see and do from the web interface
  
  
 h3. Reporting
  
  
 h3. Inventory

 
    About IBM      Privacy      Contact