Reloading multiple probe rules files

The $OMNIHOME/extensions/roi/probemanagement.sql file contains a reloadrules_allprobes procedure and supporting ObjectServer triggers. This code makes use of probe HTTP command interfaces. After you load the procedure into an ObjectServer, you can use it to simultaneously instruct multiple probes to reload their rules files. This facility is useful when you change the rules files for several probes and you want to implement all the changes at the same time.

Before you begin

Take the following actions:

Configure the ObjectServer to run external procedures with the process agent daemon (nco_pad).
Configure the target probes for administration with the probe HTTP interface. The reload procedure does not support authentication, so do not configure the HTTP interface to require a user name or password.

About this task

The following steps describe how to configure and run the reloadrules_allprobes procedure.

The files that are required to configure and run the procedure are stored in the $OMNIHOME/extensions/roi directory. The reloadrules_allprobes procedure uses the nco_http utility to send commands to probes.

If you require a secure connection, configure SSL encryption on the probe and on the nco_http utility, as described in steps 4-7. If you enable both HTTP and HTTPS on the same probe, HTTPS is used by default.

Note: Probes do not immediately process the reload command. When it receives the reload command, a probe flags that the rules file must be reloaded. When the probe detects a new event, it reloads the rules file and processes it with the new event.

Procedure

Setting up the environment to run the procedure

To redirect ProbeWatch events to the probe management log file, modify each probe rules file to include the following file:
$OMNIHOME/extensions/roi/probewatch.include
For example:
```
if( match( @Manager, "ProbeWatch" ) )
{
    include "$OMNIHOME/extensions/roi/probewatch.include" 
    break
}
```
Configure and start the nco_pad process control agent.
If you are running nco_pad on a different computer to the ObjectServer, use the following steps:
1. On the computer where the ObjectServer is installed, make a copy of the $OMNIHOME/extensions/roi/probemanagement.sql file.
2. Edit the copy to set the host parameter of the do_probereloadrules procedure to the host name of the computer where nco_pad is running.
Use the SQL interactive interface to run the probemanagement.sql file. For example:
- $OMNIHOME/bin/nco_sql -user administrator_user_name -password password -server ObjectServer_name < $OMNIHOME/extensions/roi/probemanagement.sql
- %OMNIHOME%\bin\isql -S ObjectServer_name -U administrator_user_name -P password -i %OMNIHOME%\extensions\roi\probemanagement.sql
  Where administrator_user_name is an administrator user with the privileges to create triggers, files, and stored procedures.
This command creates the following artifacts:
- A probe management log file, $OMNIHOME/log/ObjectServer_name_probemanagement.log.
- A probe_management trigger group with triggers that log ProbeWatch events that are related to loading rules files.
- A reloadrules_allprobes procedure that requests all probes to reload their rules files.
- An external procedure that is used by the reloadrules_allprobes procedure to call the nco_probereloadrules utility. The nco_probereloadrules utility requests the reload over the probe HTTP interface.

Optional: Configuring SSL encryption

Create the key database and certificates that are required for an SSL connection.
Edit the $OMNIHOME/etc/nco_http.props file and set the following property values:
```
NHttpd.SSLEnable : TRUE
```
Where certificate_password is the password of the SSL certificate for the computer where nco_http is installed.

The NHttpd.SSLEnable property enables SSL connections between the nco_http utility and SSL-configured probes.
Edit each probe properties file and set the following property values:
```
NHttpd.SSLCertificate : certificate_name
NHttpd.SSLEnable : TRUE
NHttpd.SSLListeningPort : port
```
Where certificate_name is the name of the SSL certificate for the computer where the probe is installed and certificate_password is the certificate password. port is the port number on which the probe listens for HTTPS traffic.

The NHttpd.SSLEnable property enables SSL connections between the probe and the nco_http utility.

Running the procedure

Use the following SQL command to request that all running probes reload their rules files:
execute reloadrules_allprobes
For example, to run the procedure with the SQL interactive interface, use the following commands:
```
1> execute reloadrules_allprobes;
2> go
```
Check the $OMNIHOME/log/ObjectServer_name_probemanagement.log file to verify that the probes reloaded their rules files.

Results

The following information is logged in the probe management log file:

The date and time at which a reload is requested.
Details of each HTTP-enabled probe to which a reload request is sent.
A summary that includes the following information:
- The number of probes with an HTTP interface enabled (managed).
- The number of probes that do not have an HTTP interface enabled (unmanaged).
- The number of probes that are listed in the registry.probes table but are not running.
The status of each reload attempt. Because the reload is processed with the next event received by a probe, these log entries can be written some time after the initial reload request.

If you configured SSL encryption, the log entries refer to HTTPS instead of HTTP.

When a reload fails, the log message indicates the reason for the failure. For more information about the failure, check the local probe log file.

If no status messages are logged after a reload command, use the following steps to troubleshoot the problem:

Verify that the log contains an entry that shows that an HTTP request was attempted. For example:
```
Tue Jul 23 12:46:27 2013 Sent HTTP reload rules request to probe_name on test_server:28888
```
If this request message was not logged, verify that the probe is running and is connected to the ObjectServer.
Verify that the nco_pad process is running and check its log file for errors.
If nco_pad is running correctly, verify that the host name logged in the probe management log file is correct (probe_name on host_name:port).
The probe might not be able to find the name of the host that it is running on when it is running in a virtualized environment, or in an environment with multiple network cards, or when naming services are not available. In such cases, specify the host name in the probe Nhttpd.ListeningHostname property.

Note: When the reload procedure is run against probes that were installed under Tivoli Netcool/OMNIbus V7.4, the messages that are written to the log differ slightly from the messages that are written by V8.1 probes. The messages reference SIGHUP instead of HTTP, whether the reload was triggered by an HTTP request or a SIGHUP signal. For example:

Tue Jul 23 12:47:28 2013 simnet2 probe on test_server:28148 : Rules file reread upon SIGHUP request successful

Also, if a rules file reload fails for any reason other than a syntax error, the rules file name is not included in the message. The failure message looks similar to the following message:

Tue Jul 23 18:20:24 2013 simnet_my probe on test_server:6790 : Rules file reread upon HTTP request failed ... (  in file  )

To resolve these problems, install the probes into a Tivoli Netcool/OMNIbus V8.1 environment.

Example

The following example shows the type of messages that are logged when the reload command is sent to three probes with active HTTP interfaces:

Tue Jul 23 12:46:27 2013 Reload all probe rules request
Tue Jul 23 12:46:27 2013 Sent HTTP reload rules request to simnet1 on test_server:28888
Tue Jul 23 12:46:27 2013 Sent HTTP reload rules request to simnet3 on test_server:28316
Tue Jul 23 12:46:27 2013 Sent HTTP reload rules request to simnet2 on test_server:28602
Tue Jul 23 12:46:27 2013 Reload all rules summary: managed probes 3, unmanaged probes 0, probes not running 0
Tue Jul 23 12:46:31 2013 simnet2 probe on test_server:28602 : Rules file reread upon HTTP request successful ...
Tue Jul 23 12:46:31 2013 simnet3 probe on test_server:28316 : Rules file reread upon HTTP request successful ...
Tue Jul 23 12:46:31 2013 simnet1 probe on test_server:28888 : Rules file reread upon HTTP request successful …

The following example shows the type of messages that are logged when some running probes have active HTTP interfaces and some do not:

Tue Jul 23 12:47:24 2013 Reload all probe rules request
Tue Jul 23 12:47:24 2013 Sent HTTP reload rules request to simnet1 on test_server:29420
Tue Jul 23 12:47:24 2013 Sent HTTP reload rules request to simnet3 on test_server:28863
Tue Jul 23 12:47:24 2013 Sent HTTP reload rules request to simnet2 on test_server:28148
Tue Jul 23 12:47:24 2013 HTTP interface not active for notmgd_simnet1 on test_server
Tue Jul 23 12:47:24 2013 HTTP interface not active for notmgd_simnet2 on test_server
Tue Jul 23 12:47:24 2013 Reload all rules summary: managed probes 3, unmanaged probes 2, probes not running 0
Tue Jul 23 12:47:27 2013 simnet3 probe on test_server:28863 : Rules file reread upon HTTP request successful ...
Tue Jul 23 12:47:28 2013 simnet1 probe on test_server:29420 : Rules file reread upon HTTP request successful ...
Tue Jul 23 12:47:28 2013 simnet2 probe on test_server:28148 : Rules file reread upon HTTP request successful …

The following example shows the type of message that is logged when a rules file contains a syntax error:

Tue Jul 23 18:20:24 2013 simnet_my probe on test_server:6790 : Rules file reread upon HTTP request failed ... ( parse error at line 6 in file simnet.rules )