Fabric-specific problems

You can use this information to troubleshoot fabric-specific problems.

Fabric discovery

You might encounter the following of problems when you discover the fabrics in a storage environment:
  • An error in the logs. Typically the first error you see is the most important.
  • Information in the GUI that is incorrect or unexpected.
For information about error messages, see Message types. Follow the explanations, user responses, and administrator responses.
For problems related to the discovery of fabrics, follow these steps:
  1. Run the service tool. See Service tool overview.
  2. Run the repocopy command to capture the IBM Spectrum Control database. Follow these steps:
    1. Go to the following directory:
      Windows™ operating systems:
      installation_directory\data\server\tools
      Linux® or AIX® operating systems:
      installation_directory/data/server/tools
    2. Run the following command:
      Windows operating systems:
      repocopy.bat
      Linux or AIX operating systems:
      repocopy.sh
    3. Select Export data from repository tables.
    4. Select the directory for the export file location.
    5. Use the default values for delimiter and double quotation marks.
    6. Use the default values for the connection properties.
    7. Compress all the files in the output directory and send to the IBM Software Support.
  3. Get more trace information. To turn on or change the tracing for the component servers and Device server services, follow these steps:
    1. Go to Home > System Management.
    2. Click Component Servers in the Components section. In the Component Servers pane, you can view performance information for each component server. The current trace level for each component server and the Device server services is highlighted.
    3. Adjust the trace level for a component server or a Device server service by clicking Low, Medium, or High.
  4. Correlate the time stamps in the job log.
  5. Check the health of the Device server.
  6. Check these files:
    • installation_directory\device\log\traceTPCDeviceServer.log
    • installation_directory\device\log\msgTPCDeviceServer.log
To check on incorrect information, check these items:
  • Did IBM Spectrum Control receive an event? Look for the following items:
    • Events in the alerts.
    • ALR4100I: Received an SNMP trap notification from source.
  • The health of the probe:
    • Look in the log file: installation_directory\device\log\msg.probeFabricAgents.x.x.log
    • Look for exceptions or error messages.
  • If the population of the results into DB2® failed:
    • Check the following log file: installation_directory\device\log\msg.probeFabricAgents.x.x.log
    • Look for exceptions or error messages.
The following is a list of some of the known problems:
  • Some unsupported devices like multi-protocol routers can cause failures during the discovery process.
  • AIX HBA information is limited and provides only name, manufacturer, firmware, and model.
  • HBA hardware and BIOS versions are not shown in the IBM Spectrum Control GUI.

HBAs not discovered by a Storage Resource agent

The Storage Resource agents do not use RNID and SCSI inquiries for remote HBA and resource identification. You need a Storage Resource agent on each system for which HBA information is needed. You need a native interface storage system or SIM-S Provider to identify the storage devices.

Check for any messages in the probe log for information about what might be the problem. You can also check the following items:
  • If the supported HBA driver and API are installed. In some cases, the HBA driver includes the API. In other cases, the HBA driver and API must be installed separately. Contact your HBA vendor for information about the driver and API.
  • If the HBA and API are at a level that is supported by IBM Spectrum Control. See External link iconhttps://www.ibm.com/support/pages/node/388393.
  • There are some operating system restrictions for Storage Resource agent functions.

    For more information about the requirements for the Storage Resource agent, seeExternal link iconhttps://www.ibm.com/support/pages/node/6249361#Agents.

    .
  • Check to see whether the HBA is configured appropriately. For information about how to configure the HBA, contact your HBA vendor.

The HBA to port relationship is a logical concept. Sometimes one physical card might have more than one HBA port. Multiple ports in the same HBA share node WWN.

Probe failures

If the probe of a fabric fails, check the following items:
  • Look at the probe job log and Device server message and trace log files. If the error message is in processing before the agent is called, the issue is with the agent assignment.
  • Check for communication errors when the agent is called, any error response from the agent, or no response from the agent.
  • Check the agent log files for error messages.

Fabric configuration

Check the following items for fabric configuration problems:

  • Look at the GUI or CLI error code. See IBM Spectrum Control Messages for the error codes. The error codes have two parts: command failure code and native failure code.
    Command failure code
    The command failure code specifies which operation failed, such as an error that occurred while a zone is created.
    Native failure code
    The native failure code gives the reason for the error, such as Already exists.
  • Whether the agent that is connected to the fabric is configured appropriately for zoning.
Check the following logs:
  • Web server log files.
  • AuditTrace.log in the Data Server log directory.
  • TPCZoneControl.log in the Device server log directory.
  • msg.control.123.123.log in the Device server log directory
  • traceTPCDeviceServer.log in the Device server log directory
How to adjust log and trace levels
You can adjust the following trace levels in the DataStore.properties and nativelog.properties files in the Device server conf directory.
Tip: Make a backup copy of the DataStore.properties file before you modify this file. A corrupt DataStore.properties file can cause IBM Spectrum Control to not start. 
san.ControlTraceLogger.level=ALL
san.SanZoneControlTraceLogger.level=ALL
san.SanZoneControlAgentTraceLogger.level=ALL
Commands and tools for troubleshooting
  • On the seed switch of the fabric, check for a transaction lock. For example, for the Brocade CLI, use the cfgTransShow and cfgTransAbort commands.
  • Look for IBM Spectrum Control GUI or CLI instances that can be holding a transaction lock. It might be difficult to abandon the transaction lock on older switch models without rebooting the switch.

Fabric events

Some of the common reasons for not getting notification of external fabric events or not having fabric probes run automatically in response to events are as follows:
  • Switches are not configured to send SNMP traps to the server.
  • The fabric for which an event was received or fabrics that are discovered by the Storage Resource agent are not included in any probes that are defined by the user. An automatic probe of fabrics is done only for fabrics that are included in some probe definition.
  • The switch is configured for a trap level that results in the filtering of the trap and not sent from the switch to the server. The trap that is sent from the switch to the server is filtered by the server because of default filter rules or filtering rules, or both, configured by the user.

To configure tracing to check that the traps are filtered, set the trace level parameter dsn.eventFactoryTrace.level = INFO by using the CLI command to get detailed trace information. Ensure that the created FabricAlert object is sent to the Data Server successfully.

The detailed trace information is saved in the following directory: installation_directory\data\log\traceTPCDeviceServer.log.

If you are not getting a fabric alert when you expected one, check if the information shown in IBM Spectrum Control reflects the change. Check that the appropriate types of alerts are defined for the fabric.

Check the following logs for error messages:
  • installation_directory/data/log/server_xxxx.log
  • installation_directory/data/log/TPCD_xxxx.log

CIM indications

Here is a list of some things to check when you are having issues with CIM indications from Network Advisor:
CIM indication not received
  • Verify that SNMP registration is successful by the Network Advisor discovery dialog
  • Verify if the configured SNMP port is blocked by a firewall
  • Verify if the Management Information Base (MIB) capability for trap forwarding and tracking changes are configured properly
To get the Network Advisor logs, follow these steps:
  1. Open the Network Advisor Server Management Console.
  2. Go to the Technical Support Information tab.
  3. Click Browse and select the log destination. Click capture.

Fabric data retention

Check for problems that are related to the removal of missing fabric resources and data in the following logs:
  • server_xxxxx.log
  • TPCD_xxxxx.log
These logs are in the following directory: installation_directory/data/log/