Administrator toolkit

The administrator toolkit consists of a set of MBeans (managed beans), scripts, and other tools that help you administer Decision Server Insights solutions and servers.

Insight Server has a range of operational metrics. The most important metrics are related to heap usage, garbage collection, inbound queue depth (which indicates whether a cluster is keeping up with the volume of incoming events), outbound queue depth, and how long each agent is taking to process different types of events. These basic metrics give you good visibility into the health of your servers and can help you to detect and react to issues before they become critical.

Decision Server Insights provides scripts to automate deployment and administrative tasks over JMX and REST. The following files are modified when you manage solutions and administer your servers, where WLP_USER_DIR is the environment variable that points to the location of the Liberty usr directory:

  • <WLP_USER_DIR>/servers/server_name/server.xml

    The server configuration file is updated by running scripts and other management tools.

  • <WLP_USER_DIR>/servers/server_name/bootstrap.properties and jvm.options

    These files are used for various server properties.

  • <WLP_USER_DIR>/servers/server_name/grids/objectgrid.xml and objectGridDeployment.xml

    These files are used to configure WebSphere® eXtreme Scale.

Important: When you enable security the property and XML files are also likely to be modified.

The <InstallDir>/runtime/ia/etc directory contains property files that are used by the scripts in ia/bin. If you need different connection properties, do not edit the <InstallDir>/runtime/ia/etc/connection.properties file. Create a properties file that includes your connection properties and use the --propertiesFile command-line argument to reference it. The <InstallDir>/runtime/ia/etc directory is replaced when you run an update of Insight Server.

Do's and don'ts

Table 1. The do's and the don'ts

Goal

The do's The don'ts

Modify solution and system behavior

Use the solutionManager and propertyManager scripts to adjust default values and add new properties. For more information, see Properties.

You set a system property on a single server of a server topology; system property updates are pushed to all servers.

 

Manage system memory

Monitor key metrics to ensure that your runtime servers and solutions are working well.

Monitor the Java™ virtual machine (JVM) heap usage remotely by using the MBeans supplied by the JVM.

You can also use JConsole to get information on performance and resource consumption both on a local or remote computer. Connect to a JVM and then browse the JMX MBeans to see the wide variety of metrics available. The following batch file launches JConsole and connects to a remote JVM over HTTPS:

REM service:jmx:rest://hostname:9443/IBMJMXConnectorREST
REM username is tester
REM password is tester
REM you can set -J-Djavax.net.debug=ssl to help debug
REM **NOTE** SSL host name verification is disabled.

C:/IBM/ODMInsights/jdk/bin/jconsole 
-J-Djava.class.path=C:/IBM/ODMInsights/jdk/lib/jconsole.jar;
C:/IBM/ODMInsights/jdk/lib/tools.jar;
C:/IBM/ODMInsights/runtime/wlp/clients/restConnector.jar 
-J-Dcom.ibm.ws.jmx.connector.client.disableURLHostnameVerification=true 
-J-Djavax.net.ssl.trustStore=C:/IBM/Solutions/Config/key.jks 
-J-Djavax.net.ssl.trustStorePassword=tester 
-J-Djavax.net.ssl.trustStoreType=jks

If the heap usage exceeds some percentage of available heap, alarms can be generated.

Do not expect the system to automatically page entities in/out of memory.

Update entities

When you use the REST API to update an entity, agents are not notified that the entity is updated. If this lack of notification is an issue in your solution, submit an event to update your entities rather than using the REST API.

If the events are submitted by the gateway API, the events are routed to the correct partition and processed immediately. If the events are submitted through HTTP or JMS, the processing passes through the solution connectivity, which can be scaled independently of agent processing.

 

Protect access to entities

Configure grid security in a production server that is running in an unprotected network.

Do not keep test users and authorization roles in the configuration of your production server; at a minimum encode or encrypt the passwords.

Deploy solutions

In production, use the solutionManager script to deploy a solution.

In a production topology, use bash scripts over Secure Shell (SSH) to clean up the servers, start/stop the servers, deploy, and retrieve logs.

Pass a script and have it run ephemerally. For example:

cat script.sh | ssh user@remote_server /bin/bash

Or:

ssh user@remote_server "$(< localfile)"

In production, do not deploy a solution from Insight Designer.

Monitor your operations

Decision Server Insights provides comprehensive and flexible logging, and utilities to monitor the performance of Insight Server. Consider the following practices:

Centralize logging
Centralized logging provides an easy way to view and search your logs, which are normally stored locally on individual servers across a topology. Aside from the convenience of not having to log in to individual servers to read logs, centralized logging also helps to identify issues that span multiple servers by correlating the logs and metrics during a specific time frame. In a test environment, you can also use Insight Monitor, or install IBM® Log Analytics Server.
Choose which logs to enable
Gather the important logs from all of your servers, and set a time to retain the logs before they are discarded.
Use log filters
Filtering logs improves your ability to query, analyze, and graph the data in meaningful ways.
Synchronize your server clocks
Ensure that the clocks of your servers are synchronized and set to the same time zone, so that your log timeline is accurate across your topology.

Use the propertyManager script to enable system performance monitoring and capture warnings about garbage collection and available heap. The properties are automatically set on all servers when you run the script on one runtime server.

You can use the SystemMonitorMXBean API to retrieve warning messages collected by system monitoring. You can also reset the logging cache, and get notified whenever a warning message is added to the cache. The MBean returns only the warning messages of the runtime server to which the connection is established.

Create operational alerts to trigger an action, such as running a script or sending a notification, when a service or server goes down, or if a certain resource (CPU, memory, or storage) is over-utilized. These notifications help you to react to any issues as soon as they occur, which can help minimize or prevent the downtime of your application.

Build dashboards to improve operational efficiency. Some metrics are best viewed on a dashboard. IBM Log Analysis Server has a built-in index, search, and dashboard capability.

Use the pmiManager guardian utility to set conditions on the Insight Server MBeans. IBM Tivoli® Monitoring can also be used to set up alert conditions and alarms.

In production, do not enable every warning message and monitoring tool you can, unless execution speed is not an issue.

Monitoring is not always required for a production server, but the need for monitoring increases as a production environment grows in size and complexity. It provides an easy way to track your critical resources, and improve the planning and maintenance of your environment.

MBean APIs for general administration

You can manage Insight Server, deployed solutions, and outbound endpoints by using JMX MBean (managed bean) APIs. The Liberty profile provides a list of MBeans and corresponding management interfaces that help you manipulate and monitor the server. Liberty profile uses the restConnector feature to expose JMX over HTTPS. All of the MBeans enabled on a server can be found by entering the following URL: https://server_name:9443/IBMJMXConnectorREST/mbeans

The management interface is a Java interface that you use to construct a proxy object for the MXBean, as described in the Liberty documentation. Documentation of the management interface for each MBean API is provided in the Java API reference.

Table 2. MBeans for general administration
MBean When to use Supported tasks

AgentStatsMXBean

Use AgentStatsMXBean to obtain statistics about Java agents.

For more information, see AgentStatsMXBean and Monitoring agent activity.

  • Determine how many times an agent processed an event.
  • Determine which agents were triggered by a specified event type.
  • Determine which agents correspond to an event type.
  • Determine the time that is taken to process all agent calls.
  • Determine how many times an engine was retrieved from cache.
  • Determine how many times an engine was created.
  • Determine how many times an event was triggered.
  • List the invocation statistics for a specified event.
  • List the invocation statistics for a specified event type.
  • Determine the time that is taken to process all events.

DataLoadManagerMXBean

Use DataLoadManagerMXBean to recover and restore configuration information and data that is persisted to a backup database.

For more information, see DataLoadManagerMXBean

  • Load persisted data back onto the system.
  • Restart the system manually after you load persisted data back onto the system.
  • Load persisted data back onto the system, and restart the system automatically.
  • Check the progress of a persisted data reload.

GlobalPropertiesMXBean

Use GlobalPropertiesMXBean to manage global properties.

For more information, see GlobalPropertiesMXBean.

  • Set a global property.
  • Retrieve a global property value.
  • List all available global properties.

InboundEndpointMonitorMXBean

Use InboundEndpointMonitorMXBean to retrieve information about solution connectivity that can help you assess the performance of your inbound endpoints.

For more information, see InboundEndpointMonitorMXBean

  • Display the total processing time
  • List the number of times an event is invoked
  • List the number of times events are rejected for invalid format
  • List the number of times events are rejected as not supported
  • List the number of timeouts

JobManagerMXBean

Use JobManagerMXBean to manage jobs.

For more information, see JobManagerMXBean and Managing global aggregates.

  • Display job information.
  • List all jobs, active jobs, or queued jobs.
  • Start a job.
  • Update a job schedule.
  • Stop a job.

OutboundBufferManagerMXBean

Use OutboundBufferManagerMXBean to manage outbound connectivity (outbound endpoints), and outbound events that are pending delivery, for a solution.

For more information, see OutboundBufferManagerMXBean and Managing outbound endpoints with the MBean API.

  • List all outbound endpoints in a solution.
  • List all unused outbound endpoints in a solution.
  • Determine how many events are pending on an outbound endpoint.
  • Discard all pending events from an outbound endpoint.

OutboundEndpointMonitorMXBean

Use OutboundEndpointMonitorMXBean to retrieve information about solution connectivity that can help you assess the performance of your outbound endpoints.

For more information, see OutboundEndpointMonitorMXBean

 

ServerAdminMXBean

Use ServerAdminMXBean to manage the servers in your topology.

For more information, see ServerAdminMXBean.

  • List all servers.
  • Display the name and product level for the current server.
  • Determine the number of partitions on a server.
  • Determine whether a server is online.
  • Shut down a server.

SolutionsMXBean

Use SolutionsMXBean to manage solutions and solution properties.

For more information, see SolutionsMXBean and Managing deployed solutions by using MBean APIs.

  • List all currently deployed solutions.
  • Set an upgraded or redeployed solution as the current solution version.
  • Restore a previously deployed solution version as the active version.
  • Display the current version of a specified solution.
  • Determine whether a specified solution is deployed.
  • List all active, running servers.
  • Set or override the value of solution property.
  • Determine the current value of a solution property.
  • Retrieve the names of all properties that are defined for a specified solution.

SystemMonitorMXBean

Use SystemMonitorMXBean to retrieve warning messages collected by the system.

For more information, see SystemMonitorMXBean.

  • Notification when a warning is added to the logging cache.
  • Reset the logging cache.

TestDriver API for server and solution problem determination

Table 3. TestDriver API for server and solution problem determination
API When to use Supported tasks

Test driver

Use test driver to set up and automate complex solution testing, and to detect server problems.

For more information, see TestDriver and Receiving and storing debug events and Creating automated test cases.

  • Load entities into the data grid then submit test events.
  • Debug a server problem by connecting a test driver instance and capturing events.

REST APIs for entity and global aggregate administration

Table 4. REST APIs for entity and global aggregate administration
API When to use Supported tasks
Entity resources

DELETE

Use DELETE to remove a business entity from a solution on an Insight Server.

For more information, see Deleting entities.

  • Delete a single business entity.
  • Delete all business entities of a specified type.
  • Delete all entities in a solution.

DELETE state

Use the DELETE state REST method to reset the state of a solution in a production environment.

For more information, see the REST API.

Reset the solution state, including the rule agent history, cached engines, time triggers, pending events and global aggregates.

GET

Use GET to retrieve information about business entities in a solution.

For more information, see Listing entities.

  • List the business entities in all deployed solutions.
  • List the business entities in a specified solution.
  • List all business entities that are associated with a specified solution.
  • Display a single entity by key value.

POST

Use POST to update business entities in a solution. For example, use a POST request to change a flight number or customer account number.

For more information, see Creating entities.

  • Update fields or data in a single business entity.
  • Update fields or data in a set of business entities.

PUT

Use PUT to deliver business entities to a deployed solution, at the location indicated by the URI. You use the PUT request for delivering new business entities, and for delivering business entities that are updated with new fields or deleted fields. For more information, see Updating entity data.

  • Deliver a newly created business entity to a deployed solution.
  • Deliver an updated business entity to a deployed solution.
Global aggregate resources

GET

Use GET to retrieve information about global aggregates in a solution.

For more information, see the REST API.

  • Get information on a single global aggregate.
  • Get information on a all global aggregates in a solution.
Shared aggregate resources

GET

Use GET to retrieve information about shared aggregates in a solution.

For more information, see the REST requests.

  • Get information on a shared aggregate at a time point by using the when parameter.

Scripts for general administration

Table 5. Scripts for general administration
Script When to use Supported tasks

connectivityManager

Use connectivityManager to manage the connectivity assets in a solution.

For more information, see Managing solution connectivity by running connectivityManager.

  • Deploy solution connectivity.
  • Generate solution inbound connectivity.
  • Validate solution connectivity.
  • Activate solution connectivity.
  • Deactivate solution connectivity.
  • Undeploy solution connectivity.

dataLoadManager

Use dataLoadManager to recover and restore configuration information and data that is persisted to a backup database.

For more information, see Restoring data after a system shutdown by running dataLoadManager.

  • Load persisted data back onto the system.
  • Restart the system manually after you load persisted data back onto the system.
  • Load persisted data back onto the system, and restart the system automatically.
  • Check the progress of a persisted data reload.

jobManager

Use jobManager to manage and obtain information about analytics jobs.

For more information, see Managing global aggregates.

  • List all jobs, active jobs, or queued jobs.
  • Display information about a job.
  • Start a job.
  • Update a job schedule.
  • Stop a job.

outboundBufferManager

Use outboundBufferManager to manage the outbound connectivity (outbound endpoints), and the outbound events that are pending delivery, for a solution.

For more information, see Managing outbound endpoints by running outboundBufferManager.

  • List all outbound endpoints, and outbound event counts, in a solution.
  • Clear (discard) all pending events from the buffer of an outbound endpoint in a solution.

pmiManager

Use pmiManager to generate statistics about a running server. For more information, see pmiManager

  • Generate statistics that include the number of times an agent or event type is invoked, average time an agent or event time is active, and the number of times an agent or event type is invoked per second.
  • Set up a guardian utility to trigger an alert and detect anomalies when a server reaches specified thresholds.

propertyManager

Use propertyManager to manage server properties.

For more information, see Modifying server behavior by setting properties.

  • Set server properties.
  • Retrieve server properties.
  • Modify server properties.

serverManager

Use serverManager to pause and shutdown server processes.

For more information, see Creating, starting, and stopping servers.

  • Use isonline to determine whether a server is running.
  • Use shutdown to pause and shutdown server processes and components in the correct order.
  • Run suspendbalancing and resumebalancing to suspend and resume partition balancing across servers in a multiple-server production topology.

solutionManager

Use solutionManager to manage deployed solutions.

For more information, see Managing deployed solutions by running solutionManager.

  • List all active servers.
  • List all deployed solutions.
  • Display the current solution version.
  • Revert the solution to a previous version.
  • Set or modify user authentication properties.
  • Override the solution properties.

Insight Monitor

Table 6. Insight Monitor. Insight Monitor helps you to check that your runtime servers are processing events as you expect.
Features When to use Supported tasks

adminCenter-1.0 on a catalog server

monitor-1.0 on runtime servers

Use Insight Monitor to view event rates and available memory for all of the runtime servers.

Display metrics on each runtime server:

  • Events: counts, rates, distribution
  • Memory consumption
  • CPU usage

Log and trace files for checking server status

Table 7. Log and trace files for checking server status
File When to use Supported tasks

console.log

Use console.log to view standard process output and error messages from the runtime environment, for run and debug actions. Messages are redirected to the server_name/logs/console.log file when you start the server.

For more information, see Trace and logging in the WebSphere Application Server, Liberty core documentation.

View a confirmation message that a solution is deployed and ready to use, for example:
CWMBD0060I: Solution MyCreditCardSolution-0.0 ready

View other standard process output and error messages.

messages.log

Use messsages.log to view operational messages and information that is generated by the system.

For more information, see Trace and logging in the WebSphere Application Server, Liberty core documentation.

  • View INFO, AUDIT, WARNING, ERROR, and FAILURE messages.
  • View time stamps and the IDs of message threads.

trace.log

Use trace.log to gather and view debug information.

For more information, see Trace and logging in the WebSphere Application Server, Liberty core documentation.

Gather and view debug information that is obtained by basic, enhanced, or advanced tracing.