Logging service for plug-ins

The logging service is a general service to collect multiple types of information.

The information is securely transferred from the virtual machine and stored for review by a logging service implementation.

The information that is collected by the logging service is for administrative purposes and not for the application. The service can collect text and binary type file information. The file can be a single snapshot file that is never collected again or an infinitely growing file that can rotate to manage the size.

The logging service is a high-level service that supports zero to multiple registered logging service implementations. The registered implementations are the real processes that provide reports on the multiple types of information collected.

This general logging service presents a subset of the collected information in the Log Viewer page of the console and the Instance Console deployment File Viewer tab.

The log viewer displays only the information found on the virtual machine when requested. Historical information cannot be displayed when the information is removed, such as if the data is deleted, rolled over, or if the virtual machine disappears because it is terminated. The presentation of the historical information, even after the virtual machine is no longer available, remains available to the logging service implementation to extend the log viewer capabilities to access the extracted information such as the external storage system.

The logging service information is explained in more detailed in the following sections:

High-level design of the log service
Plug-in interaction with the log service
Create a log service implementation

High-level design of the log service

The logging service is automatically included on all virtual machines in a virtual application instance. It is a generic framework for plug-ins to use, notifying the logging service implementation about the multiple types of information to collect. This design provides the flexibility for different logging service implementations to be registered on a single virtual machine. When an implementation notifies the logging service about the list of files to collect, specific information is required that helps all logging service implementations. The logging service uses a logtype, such as name, type, and single event pattern, and a list of specific files or directories that are pattern-based or any file that is created in a directory to monitor. The type and single event pattern helps the logging service to understand details about the file, like how individual events are contained inside the file. The logging service provides a default list of logtypes that can be used and information about how to extend the list of logtypes for the custom event patterns that are described as follows.

Logging service default list of logtypes

This section describes the default list of logtypes. Using these logtypes reduces the need for creating custom logtypes. These logtypes can be used by external resources to monitor the files or directories.

Table 1. `Logtypes`
`Logtype` name	`Logtype` details
File	"description":"Single file that is created for log entry"
BinaryFile	"description":"Single binary file that is created for log entry"
SingleLine	"description":"Each list is a new entry"
MultiLineTimeStamp	"description":"Single or multi-line entry where brackets encase the entry" date/time, [10/8/10 16:42:54:109 EDT], notes the start of a new entry", "start": "\\[\\d{1,2}/\\d{1,2}/\\d{2}.\\d{1,2}:\\d{2}:\\d{2}:\\d{3}.\\w{1,3}\\].*"
MultiLineIP	"description":"Single or multi-line entry where IP address notes the start of a new entry", "start": "\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}.*"

Custom logtypes

External resources can create a custom logtype to assist custom event patterns in monitoring the files and directories. This custom logtype is created by using a custom JSON file that is packaged with the external resource. The plug-in must notify the logging service about this custom logtype file. The basic metadata fields for a logtype entry are as follows:

Table 2. Custom `logtypes`
Metadata field ID	Value	Required
name	Specifies a unique name.	True
description	Specifies a text description of the `logtype`	False
format	Specifies either binary or text. The default is text.	False
start	Specifies a pattern to determine where an event starts. Important: If no end tag is included, an event ends when the pattern is seen again.	False
end	Specifies a pattern to determine when an event is complete. Important: This tag determines when an event is complete, therefore, other start patterns that are found are ignored until the end pattern is found.	False

Example

"logtype-config.json" file:
{"types":[
	{
	"name": "adaptorName2",
	"description":"This is a new adapter",
	"format":"text"
"start": "\\[\\d{2}/\\w{3}/\\d{4}.*\\d{2}:\\d{2}:\\d{2}:\\d{3}.*\\-\\d{4}\\].*Start:.*",
"end": "\\[\\d{2}/\\w{3}/\\d{4}.*\\d{2}:\\d{2}:\\d{2}:\\d{3}.*\\-\\d{4}\\].*End:.*"
	}
]}

Logging service methods

Plug-in Python lifecycle scripts interact with the logging service through the maestro.loggingUtil method call. The following list includes methods that the logging service utility exposes for plug-ins to call:

maestro.loggingUtil.registerImplementation(ImplName, ImplScript)
Allows the logging service implementations to register with the general logging service. This method indicates whether the logging services are the local virtual machine implementations. When registered, the maestro.loggingUtil monitor, unmonitor and registerPluginLogtype calls are forwarded to the registered implementation.
- Input: ImplName - Specifies the name of the logging service implementation.
- Input: ImplScript - Specifies the path and file name of the Python script that is implementing the monitor, unmonitor and registerPluginLogtype" calls.
- Return: Specifies true if the call successfully completes or false if the call fails.
maestro.loggingUtil.unregisterImplementation(ImplName)
Allows the logging service implementations to unregister with the general logging service, which stops the forwarding of the monitor, unmonitor and registerPluginLogtype calls to the service implementation.
- Input: ImplName - Specifies the name of the logging service implementation.
- Return: Specifies true if successfully complete or false if fails.
maestro.loggingUtil.monitor(listjson)
Allows the plug-in to notify the logging service implementations about the list of files and directories to monitor. This object indicates the specific role and lists the files to be collected. Details about the files, such as location and logtype, are provided. The location and logtype explain the structure of a single event inside the file for indexing implementations. The implementation must handle multiple duplicate invocations of this call.
- Input: jsonData - Specifies a JSON list that contains the details about a specific role list of information to monitor. The structure of this object is as follows: { "role": "<roleName>", "types": [ { "logtype": "<logtypeName>", "type": "<file or dir>", "name": "<path with or without a file name>", "pattern": " <optional with type directory noting the pattern of files inside directory>" }] }
- Return: Specifies true if successfully complete or false if fails.
maestro.loggingUtil.unmonitor(listjson)
Allows the plug-in to remove the specified list of files and directories from being monitored. This action requires the logging service implementation to make one last extraction from these files and then stop monitoring the files even if new events are logged in the files.
- Input: jsonData - Specifies the JSON list that contains the details about a specific role list of information to monitor. The structure of this object is as follows: { "role": "<roleName>", "types": [ { "logtype": "<logtypeName>", "type": "<file or dir>", "name": "<path with or without a file name>", "pattern": "<optional with type directory noting the pattern of files inside directory>" }] }
- Return: Specifies true if successfully completes or false if fails.
maestro.loggingUtil.registerPluginLogtype(file)
Allows the plug-in to provide information about custom logtypes if the default logtype list must be enhanced.
- Input: file - Specifies the JSON file that contains the definition of the log type.
- Return: Specifies true if successfully completes or false if fails.
maestro.loggingUtil.isImplementationRegistered(ImplName)
Allows plug-ins to know whether a specific logging service implementation is registered on the local virtual machine.
- Input: ImplName - Specifies the name of the logging service implementation.
- Return: Specifies true or false.
maestro.loggingUtil.getLogTypes() maestro.loggingUtil.registerImplementation(ImplName, ImplScript)
Allows logging service implementations to retrieve information about the default logtypes that are available.
- Input: None
- Return: Specifies the JSON object of all the default logtypes that are available, represented by the default logtype-config.json file that is provided.

Plug-in interaction with the log service

A plug-in must notify the logging service with the list of directories and files to collect for the log viewer and logging service implementations.

The plug-in notifies the logging service when to start and stop monitoring the specific files. The plug-in must provide details about the files so the logging service knows the type of the file (either a binary file or text file), and the structure of a single event inside the file. The plug-in uses the logtype to describe the details of a file to the logging service so it can properly handle the events.

The plug-in uses the following methods inside the lifecycle scripts any time during the lifecycle execution:

maestro.loggingUtil.monitor(jsonData)
maestro.loggingUtil.unmonitor(jsonData)
maestro.loggingUtil.registerPluginLogtype(file)
maestro.loggingUtil.isImplementationRegistered(ImplName)

The following example shows how a plug-in interacts with the logging service. The example illustrates where the default logtypes are used so that the logging service monitors both specific files and directories for historical purposes.

Inside the example plug-in, the start.py lifecycle script first creates a JSON object that contains a list of files and directories to monitor:

listjson = '{ "role": "'+maestro.role['name']+'", "types": [ { "logtype": "SingleLine", "type": "file", 
"name": "/opt/plugin/log/instance.log"}, {"logtype": "BinaryFile", "type": "dir", "name": "'/opt/plugin/log", 
"pattern": "*.errlog"}, {"logtype": "File", "type": "dir", "name": "/opt/plugin/log"}] }'

Then, the example plug-in start.py lifecycle script notifies the logging service about the list of files and directories:

maestro.loggingUtil.monitor(listjson)

The logging service monitors these specific files to display in the log viewer and allows logging service implementations that are configured to store these files for historical purposes.

Create a log service implementation

The logging service supports custom implementations such as log backup, logging collection and analysis service, and software monitoring like Splunk. These implementations act as the underlying process for a secure information transfer from the virtual machine and information storage for data review.

The logging service implementation is required to follow these steps:

Create a plug-in that contains the logging service implementation that gets registered with the logging service.
Because the logging service implementation can be used on virtual machines where the logging service is implemented, the implementation must be a pattern type plug-in that allows the logging service implementation to be properly installed and managed by the plug-in infrastructure.
Use the plug-in lifecycle script calls to register with the logging service.
The logging plug-in implementation uses its lifecycle scripts to register when it is ready to receive forwarded logging service method calls. The method to do this registration is maestro.loggingUtil.registerImplementation(ImplName, ImplScript).

The implementation provides a name, for example, logbackup, so other plug-ins required to interact with the specific implementation know whether that implementation is active on the virtual machine. Also, this implementation helps the logging service know that the services are registered. The Python script that is provided contains the implementation of the core forwarding methods.

When the implementation is deactivated and does not take any more forwarded calls from the logging service, the unregistered method is used. The method to do this unregistration is maestro.loggingUtil.unregisterImplementation(ImplName).

This method again provides the official name of the implementation.
Implement the core forwarding methods such as monitor, unmonitor, and registerPluginLogtype.
Each logging service implementation is required to provide a Python script that implements the following methods. These methods are automatically called when the implementation is registered. Any local plug-in on the virtual machine can call these core methods. The following methods are to be implemented:
- monitor(jsonData)
  Provides the list of files and directories to be monitored with a logtype. The logtype defines the details about the text or binary file and what a single event structure looks like inside the file. If the service cares about the specific event structure, the logtype defines a generic pattern that indicates the start and end pattern of a single event for that specific file.
- unmonitor(jsonData)
  Provides the list of files and directories to stop monitoring.
- registerPluginLogtype(file)
  Provides a file that contains custom logtypes that are provided by a specific plug-in. This file explains unique event patterns for the specific plug-in role, for example:
```
{"types":[ 	
{ 		
		"name": "DB2instance", 		
		"start": "------------------------------------------------------------.*", 		
		"end": "------------------------------------------------------------.*" 	
}, 	
{ 		
		"name": "DB2StandardLog", 		
		"start": "\\d{4}\\-\\d{2}\\-\\d{2}\\-\\d{1,2}\\.\\d{1,2}\\.\\d{1,2}\\.\\d{1,6}.*" 	
		}  
]}
```
The implementation sets up these calls. For example, the logbackup plug-in creates a registry of files and file patterns that are required to regularly back up the logs to an external system.