Using commands to gather information

Use the commands described in this section to obtain detailed information about resources and resource groups if the information provided in the system log was not sufficient to resolve the problem. For best results, you should invoke the commands in the sequence in which they are listed. For detailed information about the commands, see Commands Reference.

The following commands can be used to gather information about resources and resource groups:
lssam -V

The command lssam displays summary information about the operational states of the managed resources on a per node basis. It provides additional information about excluded or offline nodes and requests that were issued against resources or resource groups.

The option -V displays:
  • Relationships and resource details. For example the IPAddress of IBM.ServiceIP resource, or the mount point of IBM.AgFileSytem resource.
  • More verbose resource states, such as BindingState and ControlState of a resource or resource group in case a problem or non-default value is set.
  • Error messages for a non-functional sub-system of System Automation, and in situations where the user has insufficient permissions to run the command.

Example:

In this example, the command lssam was used to find out why the resource 'app1' is offline although the nominal state of the resource group 'rg1', of which it is a member, is Online. 
node1:~ # lssam
Online IBM.ResourceGroup:rg1 Nominal=Online
        |- Online IBM.ServiceIP:ip1
                |- Online IBM.ServiceIP:ip1:node1
                '- Offline IBM.ServiceIP:ip1:node2 Node=Excluded
        '- Offline IBM.Application:app1 Request=Offline
                |- Offline IBM.Application:app1:node1
                '- Offline IBM.Application:app1:node2 Node=Excluded
Offline IBM.ResourceGroup:rg2 Nominal=Offline
        '- Offline IBM.Application:app2
                '- Offline IBM.Application:app2:node2 Node=Excluded
The output shows why 'app1' is offline although the nominal state of resource group 'rg1' is Online (Nominal=Online):
  • Node 'node2' is in the list of excluded nodes (Node=Excluded), which is why all resources on the node, including 'app1', are offline.
  • An Offline request has been issued against 'app1' (Request=Offline), which explains why it is also offline on node 'node1'.
Note: The most important information is highlighted in color in the output.
Using the option -V shows also more detailed information including the relationships between the resource, for example: 
node1:~ # lssam -V
Online IBM.ResourceGroup:rg1 Nominal=Online
   |- Online IBM.AgFileSystem:fs1 MNT=/fs1                     -. <.
      |- Online IBM.AgFileSystem:fs1:node1                      |  |
      '- Offline IBM.AgFileSystem:fs1:node2                    SO SA
   '- Online IBM.Application:app1                     <. -. -. <' -'
      |- Online IBM.Application:app1:node1             |  |  |
      '- Offline IBM.Application:app1:node2            |  |  |
Online IBM.ResourceGroup:rg2 Nominal=Online            |  |  |
   |- Online IBM.ResourceGroup:rg3 Nominal=Offline     |  |  |
      |- Online IBM.Application:app3                   |  |  | <. -.
         |- Offline IBM.Application:app3:node1         |  |  |  |  |
         '- Online IBM.Application:app3:node2          |  |  |  | DO
      '- Online IBM.Application:fs3                    |  |  |  | <'
         |- Offline IBM.Application:fs3:node1          |  |  |  |
         '- Online IBM.Application:fs3:node2           |  |  |  |
   '- Online IBM.ResourceGroup:rg4 Nominal=Offline     |  |  |  |
      |- Online IBM.ResourceGroup:rg5 Nominal=Offline  |  |  |  |
         |- Online IBM.AgFileSystem:fs5 MNT=/fs5       |  |  |  |    <. -.
            |- Offline IBM.AgFileSystem:fs5:node1      |  |  |  |     |  |
            '- Online IBM.AgFileSystem:fs5:node2      AA AA SA  |    SA SO
         '- Online IBM.Application:app5               -' <' <'  | -. -' <'
            |- Offline IBM.Application:app5:node1               |  |
            '- Online IBM.Application:app5:node2                |  |
      '- Online IBM.ResourceGroup:rg6 Nominal=Offline          DO SA
         |- Online IBM.Application:app6               -.       -' <'
            |- Offline IBM.Application:app6:node1      |
            '- Online IBM.Application:app6:node2      SA
         '- Online IBM.ServiceIP:ip1 IP=10.1.1.23     <' -.
            |- Offline IBM.ServiceIP:ip1:node1            |
            '- Online IBM.ServiceIP:ip1:node2            DO
Online IBM.Equivalency:niequ                             <'
   |- Online IBM.NetworkInterface:eth0:node1
   '- Online IBM.NetworkInterface:eth0:node2
lsrg –Ab –V –g <resource-group-name>

The command lsrg –Ab shows all information about the resource groups that are defined in the automation policy. Add the -V option to also display the most important automation details, including the DesiredState, ObservedState and BindingState. When analyzing the output, make sure to check if the value of the ConfigValidity attribute indicates a configuration problem. For more information about the ConfigValidity attribute, refer to Dynamically verifying resources.

This example shows the information that is displayed for a resource group that is Offline: 
node1:~ # lsrg -Ab -V -g rg1
Starting to list resource group information.

Displaying Resource Group information:
All Attributes
For Resource Group "rg1".

Resource Group 1:
        Name                             = rg1
        MemberLocation                   = Collocated
        Priority                         = 0
        AllowedNode                      = ALL
        NominalState                     = Offline
        ExcludedList                     = {}
        Subscription                     = {}
        Owner                            =
        Description                      =
        Instruction                      =
        ActivePeerDomain                 = domain1
        OpState                          = Offline
        TopGroup                         = rg1
        MoveStatus                       = [None]
        ConfigValidity                   =
        AutomationDetails[CompoundState] = Satisfactory
                          [DesiredState] = Offline
                         [ObservedState] = Offline
                          [BindingState] = Unbound
                       [AutomationState] = Internal
                        [StartableState] = Yes
                           [HealthState] = Not Applicable
This example shows the information that is displayed for a resource group that is Online: 
node1:~ # lsrg -Ab -V -g rg1
Starting to list resource group information.

Displaying Resource Group information:
All Attributes
For Resource Group "rg1".

Resource Group 1:
        Name                             = rg1
        MemberLocation                   = Collocated
        Priority                         = 0
        AllowedNode                      = ALL
        NominalState                     = Online
        ExcludedList                     = {}
        Subscription                     = {}
        Owner                            =
        Description                      =
        Instruction                      =
        ActivePeerDomain                 = domain1
        OpState                          = Online
        TopGroup                         = rg1
        MoveStatus                       = [None]
        ConfigValidity                   =
        AutomationDetails[CompoundState] = Satisfactory
                          [DesiredState] = Online
                         [ObservedState] = Online
                          [BindingState] = Bound
                       [AutomationState] = Internal
                        [StartableState] = Yes
                           [HealthState] = Not Applicable
lsrg –m

The command displays the operational state of all managed resources and shows mandatory versus non-mandatory resource members.

Example:

node1:~ # lsrg –m

Displaying Member Resource information:
Class:Resource:Node[ManagedResource]  Mandatory  MemberOf  OpState  WinSource  Location
IBM.ServiceIP:ip1                     True       rg1       Online   Nominal	   node1
IBM.Application:app1                  True       rg1       Online   Nominal	   node1
IBM.Application:app2                  True       rg2       Offline
lssamctrl

The command lssamctrl displays the global automation parameters. For example whether the automation is currently enabled, the list of excluded nodes, and the RetryCount. The RetryCount specifies the maximum number of retries for the StartCommand when the resource does not start at the first attempt.

Example: 
node1:~ # lssamctrl
Displaying SAM Control information:

SAMControl:
TimeOut = 60
RetryCount = 3
Automation = Auto
ExcludedNodes = {}
ResourceRestartTimeOut = 5
ActiveVersion = [4.1.0.0,Thu Apr 24 11:10:58 METDST 2014]
EnablePublisher = Disabled
TraceLevel = 31
ActivePolicy = []

The command lsrgreq –L lists all requests that were issued against resource groups. Such requests are either issued directly from the command line using the command rgreq, or implicitly by the System Automation Application Manager end-to-en automation manager. In the latter case, the Source of the request is ‘Automation’.

Example: 
node1:~ # lsrgreq –L

Displaying Resource Group request information:
All request information
ResourceGroup Priority Action Source   NodeList  Active  UserID  MoveStatus
rg1           low      Start  Operator {}        Active          None

If the command is executed with the -m option, all requests against resource group members are displayed.

Example: 
node1:~ # lsrgreq -L –m

Displaying Member Resource request information:
All request information

Member Resource 1:
        Class:Resource:Node[ManagedResource] = IBM.Application:app1
        Priority                             = low
        Action                               = Start
        Source                               = Operator
        ActiveStatus                         = Active
        UserID                               =

The command lsequ –Ab is used to display all resources of type "equivalency" that are defined in the automation policy. Typically, equivalencies are used to define the network interfaces that can be used by resources of type IBM.ServiceIP.

When analyzing the output of the command, make sure to check if the value of the ConfigValidity attribute indicates a configuration problem (for more information about the ConfigValidity attribute, refer to Dynamically verifying resources).

Example: 
node1:~ # lsequ –Ab
Displaying Equivalency information:
All Attributes

Equivalency 1:
        Name                                = eq1
        MemberClass                         = IBM.NetworkInterface
        Resource:Node[Membership]           = {eth0:node1,eth0:node2}
        SelectString                        = ""
        SelectFromPolicy                    = ANY
        MinimumNecessary                    = 1
        Subscription[Consumer,…]            = {[EEZ,All,None]}
        ActivePeerDomain                    = domain1
        Resource:Node[ValidSelectResources] = {eth0:node1,eth0:node2}
        Resource:Node[InvalidResources]     = {}
        ConfigValidity                      =
        AutomationDetails[CompoundState]    = Undefined
The attribute Resource:Node[ValidSelectResources] must contain resources, especially if a dynamic SelectString is used. Subsequently, the OpState of the valid resources must be checked: 
# lsrsrc IBM.<MemberClass-attribute-value> Name NodeNameList OpState
lsrgreq -L {-m}
lsrel –Ab

The command lsrel –Ab is used to display all relationships that are defined in the automation policy.

Example: 
node1:~ # lsrel –Ab
Displaying Managed Relationship Information:
All Attributes

Managed Relationship 1:
        Class:Resource:Node[Source] = IBM.Application:app1
        Class:Resource:Node[Target] = {IBM.Application:app2}
        Relationship                = StartAfter
        Conditional                 = NoCondition
        Name                        = app1_StartAfter_app2
        ActivePeerDomain            = domain1
        ConfigValidity              =

When analyzing the output, check the relationships for completeness (for example, both the Source and the Target of a relationship must be defined), and check if the value of the ConfigValidity attribute indicates a configuration problem (for more information about the ConfigValidity attribute, refer to Dynamically verifying resources).

This command lists the public attribute of the CHARMControl class, which represents the automation engine itself. The command currently only returns the value of the Automating attribute, which is usually 1. In rare cases, the configuration of resource groups and relationships may become damaged, preventing automation from working properly. When this happens, the value of the Automating attribute changes to 0, indicating that automation has been stopped.

Stopping automation instead of recycling the RecoveryRM (as was the case in System Automation for Multiplatforms releases prior to 2.3) allows you to collect diagnostic information and to even fix the configuration. A relationship with a non-existing target, for example, is considered a damaged configuration. Once such relationship is corrected or deleted, automation can be resumed by stopping and restarting the current master RecoveryRM with the commands stopsrc and startsrc.

lsrsrc -A d -c IBM.CHARMControl

The command samdiag is used to display detailed state information for an individual resource. The command can also be used to externalize all internal variables of a resource from the automation manager. The command is very helpful in problem analysis, but it is not intended for daily use because it generates a large amount of information.

Example: 
node1:~ # samdiag -g rg1

Displaying information for the following:
 Resource Group "rg1":

Diagnosis::Resource: rg1/ResGroup/IBM.ResourceGroup
    type: CHARM Resource Group
   Status -
       Observed: Online          - Available
        Desired: Online          - Requested Online
       (Nominal: Online          - Nominal State: Online)
     Automation: Idle            - CharmBase trigger linked
      Startable: Yes             - Resource is startable
        Binding: Bound           - Bound
       Compound: Satisfactory    - Satisfactory

 Resource Based Quorum: Not Supported   - CharmBase trigger linked
   Members and Memberships:
       +---HasMember                       ---> app1/Fixed/IBM.Application/node1
       +---HasMember                       ---> ip1/Fixed/IBM.ServiceIP/node1
       +---bind/HasMember                  ---> app1/Float/IBM.Application
       +---bind/HasMember                  ---> ip1/Float/IBM.ServiceIP
  Group Constraint: Collocated
   Binding Constraints:
   Flags:
     None
   Orders:
     Outstanding Order: None            - Resource is Available
   Dependencies:
     Start: Satisfied
       +---InCluster                       ---> Cluster
     Stop: Satisfied
   Binding exceptions:
     None
   Static Relationships:
       +---InCluster                       ---> Cluster
   Dynamic Relationships:
       +---bind/HasMember                  ---> app1/Float/IBM.Application
       +---bind/HasMember                  ---> ip1/Float/IBM.ServiceIP"       
samdiag IBM.<resource-class-name>:<resource-name>:<node-name>

To query the details for a specific resource, the command has to be executed as follows:

For example, for the resource ‘app1’ on node ‘node1’: 
node1:~ # samdiag IBM.Application:app1:node1
Displaying information for the following:
 Resource "IBM.Application:app1:node1":

Diagnosis::Resource: app1/Fixed/IBM.Application/node1
    type: Fixed Resource
   Status -
       Reported: Online          - Online
       Observed: Online          - Online
        Desired: Online          - Requested Online
       (Nominal: Offline         - Defaulted: offline)
     Automation: Idle            - Idle - Online completed
      Startable: Yes             - Resource is startable
        Binding: Bound           - Bound
       Compound: Satisfactory    - Satisfactory

Resource Based Quorum: Not Supported   - CharmBase trigger linked
   Groups and Aggregates:
       <---HasMember                  ---- rg1/ResGroup/IBM.ResourceGroup
       <---bind/HasMember             ---- rg1/ResGroup/IBM.ResourceGroup
   Binding Constraints:
   Flags:
     None
   Orders:
     Outstanding Order: None       - Idle - Online completed
  Dependencies:
     Start: Satisfied
       +---RunsOn                     ---> node1/Node/IBM.PeerNode
     Stop: Satisfied
       <---HasMember                  ---- rg1/ResGroup/IBM.ResourceGroup
   Static Relationships:
       +---RunsOn                     ---> node1/Node/IBM.PeerNode
   Dynamic Relationships:
       <---bind/HasMember             ---- rg1/ResGroup/IBM.ResourceGroup"
samdiag
For deep analysis.