Log files and trace files

Edit online

If you receive any error message during the configuration and recovery operations by using the KSYS node, analyze the log files and trace files to find the component that is causing the error.

After a log file such as host_monitor.log, host_monitor_crit.log, and host_monitor_hvncp.log reaches a maximum of 20 MB, the existing log file is backed up and a new log file is created. By default, network logging is not enabled. The spooling file that has the highest number in the file name will be the oldest log file. By default, three spooling files are created for log collection. You can configure the number of spooling files by using the ksysmgr command. The maximum value is 10 spooling files.

Analyzing the log files

Use the following log files that are available in the specified directory in your KSYS node to troubleshoot errors:

/var/ksys/log/ksysmgr.log: Contains the detailed processing information about each function when you run the ksysmgr command. The ksysmgr.log file contains the detailed processing information only when you specify the -l max flag along with the ksysmgr command.
/var/ksys/log/ksysmgr.oplog: Keeps a rolling record of all the ksysmgr operations that you ran for a specific period. All the commands that you entered are logged in this log file along with the date, time, and the transaction ID.
/var/ksys/events.log: Contains details about a specific event.
/var/ksys/log/ksys_vmm.log: Contains all the log information about each process thread such as the VM monitor daemon startup, shutdown, and restart processes, application reporting, communication between the VM monitor and host monitor daemons. The /var/ksys/log/ksys_vmm.debug file contains the same information in detail.
/var/ksys/log/ksysvmmgr.log: Contains the history of the ksysvmmgr commands that you ran in the command line interface. The ksysvmmgr.log file contains the detailed processing information only when you specify the -l 3 flag along with the ksysvmmgr command. The ksysvmmgr.critlog file is a subset of the ksysvmmgr.log file and contains only the error information.
/var/ksys/log/ksys_vmm_ame.log: Contains log information about the application management engine. The ksys_vmm_ame.debug log file contains the same information in detail.
/var/ksys/log/ksys_vmm_vmH.log: Contains log information about the VM health monitor threads in the Linux® VMs. The ksys_vmm_vmH.debug log file contains the same information in detail.
/var/ksys/log/lnx_vmmke.log: Contains log information about the kernel extensions in the Linux VMs.
/var/ksys/log/aix_vmmke.log: Contains log information about the kernel extensions in the AIX® VMs.
/var/ksys/log/agents/oracle_agent/logfiles: Contains log information about the Oracle application processes that are registered in the VM agent configuration settings.
/var/ksys/log/agents/db2_agent/logfiles: Contains log information about the DB2® application processes that are registered in the VM agent configuration settings.
/var/ksys/log/agents/saphana/logfiles: Contains log information about the SAP HANA application processes that are registered in the VM agent configuration settings.

Use the following log files to troubleshoot the VM Recovery Manager HA GUI:

/opt/IBM/ksys/ui/server/logs/uiserver.log: Contains information about the VM Recovery Manager HA GUI server.
/opt/IBM/ksys/ui/agent/logs/uiagent.log: Contains information about the agent that is installed on each KSYS node.
/opt/IBM/ksys/ui/agent/logs/notify-event.log: Contains information about all VM Recovery Manager HA events that are sent from the GUI agent to the GUI server.

Analyzing trace files

Trace files contain the details about the processes that are running as part of the operations performed by the KSYS node. If you cannot identify the cause of the error by analyzing the log files, use the following resource manager (RM) trace files for problem determination:

/var/ct/cluster_name/log/mc/IBM.VMR/trace.user.*: Contains an abstract view of functions that are run for each operation. Review this log file to check the components that participated in an operation, the processing activity, the point where the operation failed, and the component that caused the failure.
/var/ct/cluster_name/log/mc/IBM.VMR/trace.ksys.*: Contains detailed information about the functions that are run for each operation.
/var/ct/cluster_name/log/mc/IBM.VMR/trace.fde.*: Contains information about the failure detection engine (FDE) processes.
/var/ct/cluster_name/log/mc/IBM.VMR/trace.krest.*: Contains abstract information about the HMC and VIOS processes.
/var/ct/cluster_name/log/mc/IBM.VMR/trace.krestlong.*: Contains detailed information about the HMC and VIOS processes.

You can use the rpttr command to format trace files. The syntax of the rpttr command follows:

# rpttr -f -o dict trace_file_names

For example,

#rpttr -o dict /var/ct/<cluster_name>/log/mc/IBM.VMR/trace.ksys.* >   <logfilename.log>
#rpttr -o dict /var/ct/<cluster_name>/log/mc/IBM.VMR/trace.krest.* >   <logfilename.log>

VM log files of a host group

You can get the VM log files of a host group or all host groups of a site by running the export command. You can use these log files to debug issues that might occur while starting a VM. To get VM log files, the KSYS node must be in the online state, the VM must be in the managed state, and HA monitoring must be enabled for the VM.

To collect the VM log files of a host group, run the following commands from the CLI of the KSYS node:

export VMRM_HGNAME=HA_HG1

snap -z "product_name=ksys_prod"

If the export command does not contain the host group name, the VM log files of all host groups will be generated.