IBM Spectrum Symphony log files
The IBM® Spectrum Symphony log files provide information on the general well-being of workload-related daemons and services. Logs are a valuable tool used for troubleshooting resources, daemons, and services: cluster and consumer administrators can retrieve and download the logs using the cluster management console. You can choose to retrieve part of a log file instead of the entire file. Each log file can be a maximum of 2 GB in size. You can access the logs for the features you have installed and enabled.
Additionally, in IBM Spectrum Symphony Developer Edition, events are logged to the log files in the logs directory mentioned in this topic (versus using to the IBM Spectrum Symphony event framework).
Default log file locations
- IBM Spectrum Symphony component log files
-
- Windows: %SOAM_HOME%\logs
- Linux®: $SOAM_HOME/logs
- IBM Spectrum Symphony API log files
- The IBM Spectrum Symphony API log file is written to the directory where the client executable resides.
Log file names
Log files are named according to the component they are logging and the host name where the
component runs. For example, a log file for the session director running on hostA
is named sd.hostA.log.
The following table lists possible log files and host applicability:
| Log file | Description | Host on which you can find the log file |
|---|---|---|
| agent.host_name.log | Only in IBM Spectrum Symphony Developer Edition. Messages, events, and errors related to startup and shut down of IBM Spectrum Symphony Developer Edition processes. | Only in IBM Spectrum Symphony Developer Edition. Found on every host on which IBM Spectrum Symphony Developer Edition runs. |
| api.host_name.log | Messages, events, and errors for the client application that submits work to the system. | The host on which the client application runs. |
| cli.log | Messages, events, and errors related to the command line. | When enabled, found on the host from which the command was issued, in the directory from which the command was issued. By default, no cli.log files exist. |
| sd.host_name.log | Messages, events, and errors for session director. | The host on which session director is running. |
| sim.host_name.application_name.log | Messages, events, and errors related to tasks that ran for the specified application. | Each compute host running tasks for the application. |
| ssm.host_name.application_name.log | Messages, events, and errors related to workload scheduling for the specified application. | The host on which session manager is running. |
The following table lists possible log files and where you can retrieve them in the cluster management console:
| Log name | Description | Console location it can be retrieved from |
|---|---|---|
| datasourcetools.log | Logs database configuration tool activities, which configures the connection between the reporting services and the relational database. | System logs, tasks, hosts, blocked hosts |
| egodeploy.log | Logs messages and errors relating to deploying and installing service packages from the repository service. | System logs, tasks, hosts, blocked hosts |
| es.log | Logs system errors and debug information relating to Elasticsearch. Elasticsearch is used for Explorer reports on IBM Spectrum Symphony Advanced Edition for Linux. | System logs, tasks, hosts. blocked hosts |
| esc.log | Logs service failures and service instance restarts based on availability plans. Errors surrounding the cluster management console startup are logged here. | System logs, tasks, hosts, blocked hosts |
| jobdt.log | Logs the job data transformer service (jobdt), which converts raw job data in the relational database into a format usable by the reporting feature. | System logs, tasks, hosts, blocked hosts |
| lim.log | Logs static and dynamic attribute information about individual hosts. Logs any error warnings generated while lim was running. If log class is set to debug, logs information that helps locate the encountered problem. | Tasks, hosts, blocked hosts |
| loader.log | Logs the data loaders activities, which collects operation data using data loader plug-ins to load data into tables in a relational database. | System logs, tasks, hosts, blocked hosts |
| messages.log | Logs system errors and debug information from web server startup. | System logs, tasks, hosts, blocked hosts |
| messages_gui.log | Logs system errors and debugging information related to the cluster management console web server startup. | System logs, tasks, hosts. blocked hosts |
| pem.log | Logs remote operations (start, stop, control activities, failures). Logs tracked results for resource utilization of all processes associated with the host, and information for accounting or chargeback. | Tasks, hosts, blocked hosts |
| plc.log | Logs the loader controller service activities, which controls the data loader plug-ins that collect data from the system and writes the data into the database. | System logs, tasks, hosts, blocked hosts |
| purger.log | Logs data purger activities, which maintains the size of the database by purging old records from the data base and archiving them. | System logs, tasks, hosts, blocked hosts |
| rest_messages.log | Logs system errors and debug information relating to the REST web server. | System logs, hosts |
| rs.log | Logs messages, events, and errors for the repository service, used for package deployment on compute hosts. | System logs, tasks, hosts, blocked hosts |
| sd.log | Logs messages, events, and errors for session director. | System logs, tasks, hosts, blocked hosts |
| sim.log | Logs messages, events, and errors related to tasks that ran for the application. | Tasks, hosts, blocked hosts |
| ssm.log | Logs messages, events, and errors for the IBM Spectrum Symphony repository service, used for package deployment on compute hosts | Tasks, hosts, blocked hosts, system logs |
| vemkd.log | Logs aggregated host information about the state of individual resources, status of allocation requests, consumer hierarchy, resources assignment to consumers, and started operating system-level process. | System logs, tasks, hosts, blocked hosts |
| wsm.log | Logs information collected by the web server monitor daemon. Failures of the WEBGUI service that runs the cluster management console are logged here. | System logs, tasks, hosts, blocked hosts |
Configuration files to set logging properties
- Configuration files location
- The logging configuration files are by default in these locations:
- Windows: %SOAM_HOME%\conf
- Linux: $SOAM_HOME/conf
In these directories, the following properties files are available:- agent.log4j.properties
- api.log4j.properties
- cli.log4j.properties
- rs.log4j.properties
- sd.log4j.properties
- ssm.log4j.properties
- sim.log4j.properties
You can edit these properties files to customize logging settings. For example, to specify the location where you want the system to write logs to, set this in the .appender.API.FILE parameter within your properties file. For example:
- Windows: .appender.API.FILE=c:/api.${log4cxx_hostname}.log (use a forward slash (/) to specify directory names in the location, regardless of operating system).
- Linux: .appender.API.FILE=/opt/mylogs/api.${log4cxx_hostname}.log
- Log file and message formats
- Log file entries follow a format that depends on the log level in which the message was logged.
The format of the log-file entries can be changed. For more details, see the log4cxx documentation: http://logging.apache.org/log4cxx/.
The messages within the log files follow this format:- INFO, WARN, ERROR, and FATAL log level messages
- The following information is included for all messages recorded at the INFO, WARN, ERROR, and
FATAL log
levels:
time_stamp log_level [process_ID:thread_ID] logger_name - message
- ERROR level messages
- Additionally, the following information is also included for some
errors:
so that error messages contain more detail, as such:Code[Internal_Code]:file_name:line_numbertime_stamp log_level [process_ID:thread_ID] logger_name - Code[Internal_Code]: file_name:line_number message
- Message attributes
- All message levels (INFO, WARN, DEBUG, ERROR, and FATAL) can include these attributes:
- time_stamp
- Displays the time when the exception was thrown. The format for the time stamp is as
follows:
Year-month-day hour:minute:second.millisecondNote: For Linux, by default, the time displayed in the logger files is GMT. The format of the timestamp can be changed by editing the related time zone settings in each log4j.properties file in the $SOAM_HOME/conf directory. The properties files contain instructions on how to implement this change. - log_level
- Displays the log level of the logger that logged the message.
Level Description FATAL Logs only those messages in which the system is unusable. ERROR Logs only those messages that indicate error conditions or more serious messages. WARN Logs only those messages that are warnings or more serious messages. This is the default level of debug information. INFO Logs all informational messages and more serious messages. DEBUG Logs all debug-level and INFO messages. ALL Logs all available messages. - process_ID
- Displays the ID of the IBM Spectrum Symphony component. The process
ID is used to differentiate between daemons when more than one daemon of the same type runs on the
host, such as when multiple session managers run on the same host. Note: The soamview app command displays the process ID of the session manager and soamview task displays the process ID of the service instance. The identity of the process that generated the message can be determined by comparing the process ID in the message with the process IDs displayed by soamview app and soamview task.
- thread_ID
- Displays the thread of the program that triggered the message.
- logger_name
- Displays the name of the logger component used to set the log level of the component that generated the message. The log4j logger components are listed in the properties files. These loggers are used to set the logging levels of specific components such as session director, session manager, service instance manager, repository service, and the CLI.
- error_message
- Displays the error message generated by the IBM Spectrum Symphony API.
- error_code
- Displays the error code that uniquely identifies the error. Error codes and their corresponding messages are listed in Error messages.
- file_name
- Displays the name of the source code file that triggered the message.
- line_number
- Displays the number of the line in the file that triggered the message.
- domain
- Displays the domain in which the message was triggered. Domains are virtual groups that
categorize messages to precisely identify the component to which the message applies.Possible domains are as follows:
- Application: Application configuration and deployment
- SOAM: Any IBM Spectrum Symphony component such as session manager and session director
- VEM: Resource management performed by EGO (not available in IBM Spectrum Symphony Developer Edition)
- OS: Operating system resource management of resources such as memory and disk capacity