IBM Support

No Connection / Cannot Add a Managed System to the HMC

Troubleshooting


Problem

Check list for problems adding a managed system to a HMC or when a system drops into a No Connection state after some runtime. These steps are specific to FSP based systems. ,

Symptom

No Connection is shown in the lssysconn -r all output or in the lssyscfg -r sys -Fname:state output.

Cause

There are multiple causes that can contribute to No Connection state.  This document covers troubleshooting techniques for all No Connection state issues for problem isolation and identification for FSP based systems.

Environment

Any HMC Version or Release
Any HMC 7042 or 7063 hardware platform
Any Virtual HMC platform, out of scope is problem isolation of the virtualization layer that can be the cause

Diagnosing The Problem

Anytime the GUI, lssysconn -r all or lssyscfg -r sys -Fname:state display No Connection for the managed system(s).

Resolving The Problem

This document is intended for the following scenarios:

  • A new managed system is powered on and connected to the HMC but does not appear in the list of managed systems. 
  • The managed system appears but is in a "No Connection" state.
  • An existing system that was working and in Connected state but now shows in "No Connection" state.

Failures usually fall into 3 general categories:

  1. The first is network connectivity:  Assigning an IP address to the managed system then establishing a network connection between the HMC and FSP. 
  2. The second is application connectivity: Establishing the managed system TCP connection between the HMC hardware server task and the FSP daemon (port 30000).
  3. The third is contributing hardware issues, cabling, switch port, fsp settings, conflicts in the network.

A) Gather Basic Information

Step A1: Record the HMC configuration.

At the HMC, open a restricted shell window or ssh into the HMC remotely. Collect a pedbg or at a minimum run the following commands to gather basic information on the version and the network configuration.

The key fields to record are underlined and highlighted:

hscpe@hmchost:~> lshmc -V
"version= Version: 10
Release: 3
Service Pack: 1061
HMC Build level 2411282125
MF71710 - HMC V10R3 M1061
","base_version=V10R3
" 


hscpe@:hmchost~> lshmc -n
hostname=hmchost,domain=rchland.ibm.com,"ipaddr=192.168.0.1,1.1.1.25,0.0.0.0,0.0.0.0","networkmask=255.255.0.0,255.255.255.0",gateway=1.1.1.1,"nameserver=9.10.244.200,9.10.244.100","domainsuffix=rchland.ibm.com,ibm.com",ipaddrlpar=1.1.1.25,networkmasklpar=255.255.255.0,"clients=192.168.255.254,192.168.254.254"

hscpe@hmchost:~> netstat -r
Destination Gateway Genmask Flags MSS Window irtt Iface
1.1.1.0 * 255.255.255.0 U 40 0 0 eth1
10.253.0.0 * 255.255.0.0 U 40 0 0 sl0
192.168.0.0 192.168.0.1 255.255.0.0 UG 40 0 0 eth0
default vlan734.rchland 0.0.0.0 UG 40 0 0 eth1

hscpe@hmchost:~> lssysconn -r all
resource_type=sys,type_model_serial_num=9080-M9S*10ABCDE,sp=primary,ipaddr=192.168.255.254,state=No Connection

Step A2: Record the Managed System FSP HMC1 and HMC2 Addresses

Control panel 30 can be used to view the current FSP IP address settings.

Complete network information and also the firmware level can be obtained using ASMI.  Cable a PC into the FSP HMC2 port and make an ASM connection to the default address ( http://www.ibm.com/support/docview.wss?uid=nas8N1018623. Record the firmware levels. Check the network settings and address on eth0.

Step A3: Record the Cabling

Clarify how the HMC is cabled to the server:

  • HMC port and FSP port used
  • Direct cabled or through a switch
  • Type of network topology: Intent to use open network, static IP addresses or private network, dhcp addressing.

Step A4: Record Managed System Control Panel Status

Record the control panel value  ("HMC=" and firmware side of "P" or "T" ) and power indicator light status.

Sample control panel states:

01 A N V=F  
         T  		 01 A N V=F  
HMC=0    T  		 01 A N V=F  
HMC=1    T  

  • No power indicator light indicates an AC power failure.  Correct the AC Power issue.
  • Scrolling dots or a quickly flashing power indicator light indicate the FSP is initializing.  This is normal immediately after apply of AC power or FSP reset/reload.  If the condition does not clear after several minutes it normally indicates a FSP hardware issue. 
  • A slow blink of the power indicator indicates FSP standby (server power off state).  The FSP should be fully functional.
  • A missing "HMC=x" status indicator on the panel implies that the HMC has never connected at the managed system layer (but a lease may of been assigned). "HMC=0" implies that a HMC has connected at one time but is not currently connected.  If a SRC appears on the control panel and is not changing then the managed system may have a hardware or firmware failure. Perform debug on the SRC before continuing. 
 
Step A5: Server Firmware level

This can be found from the ASM logon panel or from a partition running on the server:

IBM i: DSPFMWSTS
VIOS: lsfware
AIX: lsmcode -c


B) Verify Network Connectivity

If the HMC command lssysconn -r all shows the system, attempt to ping the IP address. If the ping succeeds continue to Step C. Otherwise, verify each of the settings below until the FSP HMC1 port obtains an address and the address can be pinged from the HMC.

Successful attempt example:

hscpe@hmchost:~> ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=2.46 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.294 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.315 ms
64 bytes from 10.0.0.1: icmp_seq=4 ttl=64 time=0.305 ms
64 bytes from 10.0.0.1: icmp_seq=5 ttl=64 time=0.289 ms
^C
--- 10.0.0.1 ping statistics ---
6 packets transmitted, 5 received, 0% packet loss, time 5001ms

Failure attempt example:

hscpe@hmchost:~> ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
^C
--- 192.168.0.1 ping statistics ---
26 packets transmitted, 0 received, 100% packet loss, time 25001ms

Step B1a: (Static) Verify the static ipaddress routing and netmask are correct for the network.

OR

Step B1b: (DHCP): Has a DHCP lease been given out?

The "clients=" field from the lshmc -n (above) lists the DHCP leases that have been given out. If this field contains an IP address, then a physical connection should exist; attempt to ping the address. If the IP address responds to pings but does not appear in the system management list (lssysconn -r all) continue to step C1 (add the IP address to the system manager list).

If a lease has not been given out perform the following steps until a lease has been assigned. After correcting any problem perform step B5 "Attempt to obtain a DHCP lease", then rerun the lshmc command to verify that the DHCP lease has been given out.

Step B1c: Verify the HMC dhcp server is running

netstat -lnu | grep :67 should show port 67 (the dhcp daemon) is active

hscroot@hmchost:~> netstat -lnu |grep :67
udp 0 0 0.0.0.0:67 0.0.0.0:*

If dhcp is not running first reboot the HMC and recheck. If the dhcp server still will not start debug the HMC for DHCP issues.


Step B2: Verify cabling.

See Support Document N1015834 .

  • Check link status. In Version 7.3.3 and later use HMC Management, Test Network Connectivity, Ethernet Settings tab to check link status.
  • Check the correct HMC port is cabled to the server:
  • Unplug the FSP cable from the HMC port.  
  • Use the Test Network Connectivity, Ethernet Settings tab to check link status is "Link detected: no".
  • Plug the cable back in.  Click "Refresh".   Verify link status changed to "Link detected: yes"
  •  If a hub or switch is used, verify that only one of the two FSP ethernet ports is cabled to the HMC, not both.

Step B3: Verify Routing Information

Using the information recorded in the lshmc -n and netstat -r (step A3), check for conflicting IP address ranges or routes.

Check the ipaddr and networkmask fields. If there are two ore more IP addresses for the HMC,  verify they are on different, non-overlapping, subnets.

Verify there is no conflict with the network used internally for sl0.

In the typical configuration, the address range assigned to eth0 has a direct route pointing to the IP address assigned to eth0.  For example if lshmc -n has "ipaddr=192.168.0.1,192.168.24.23,subnet=255.255.0.0,255.255.255.0" conflicts will occur. Depending on the routes, one of the two interfaces will not respond. Correct the problem.

Step B4: Verify media speed

  • The HMC default is autodetect (1G/100/10 card, 10Gb for some model 7063 adapters).
  • FSP port is autodetect 100/10 card (Power7 and earlier), 1G/100Mb autodetect (Power8, Power9, Power10).

Use the HMC Management > Test Network Connectivity, Ethernet Settings tab to check link settings and current speed/duplex. The switch setting for the FSP must be autodetect (it cannot be hard coded).

The switch setting for the HMC port must match the HMC settings (for example both autodetect or both 100/full).

A current setting of "Half-duplex" indicates a likely mismatch in settings.

Step B5: (DHCP only) Force a DHCP Discover.

Pull the ethernet cable attached to the FSP HMCx port, wait 2 minutes, plug it back in. The link deactivation/activation should cause the FSP to attempt an immediate DHCP lease acquire. This is concurrent and can be done while partitions are running.

Monitor dhcp lease activity on the HMC:

As user hscroot use "tail -F /var/log/messages | grep DHCP".  

The command "lshmc -n -F clients" shows any leases given out.

Note: In some rare cases, the link deactivate fails to force a new lease request because the dhcp client on the FSP has failed. In these instances, you can force a lease renewal via ASMI.

Obtain ASMI access (may require a laptop or other PC), expand Network Services, Network Configuration. Scroll down to the bottom and click the "Reset Network Connection" button, click to confirm the reset. 

If the FSP is in an out of memory (OOM) condition, a FSP reset/reload may be needed.

Step B6: Replace the ethernet cable. Remove any hub/switch.

Isolate any customer switch/hub issue by direct cabling from the HMC to the FSP HMCx port. 

Check link status (HMC Management > Test Network Connectivity, Ethernet Settings tab) and replace cable if needed.

Repeat step B5.

Step B7: Verify the FSP settings

If not performed earlier, perform step A2 now to verify the FSP HMC1 and HMC2 port settings are correct. Cable a laptop to the FSP HMC2 port and make an ASM connection to the default address.


  • If no connection can be made to HMC2 then attempt a ASM connection to HMC1 port.
  • If no connection can be made to either port, verify the IP addresses using control panel 30 again and that the laptop is configured for a compatible range.
  • If that fails as well, see FSP reset below.

Step B7a: ASM: Check the network settings on eth0 and eth1.

For POWER5 and P6 EM310, check for overlapping/conflicting IP address/subnets. If a conflict is found, see "FSP reset".
For all server models, verify the HMCx ports have the desired settings.

(DHCP) If  a port is incorrectly set to static, set to dynamic and repeat step B5.

(Static) If  a port is incorrectly set to dynamic, set to static (and reenter the configuration as needed).  

Verify unused ports are the default address. If the port that is cabled to the HMC is correctly set to dynamic but still at the default address then verify the port is working by re-connecting to ASMI using the HMC port and its default IP address.

Step B7b: (ASM) Check Error/Event Log

Expand "System Service Aids", "Error/Event logs" and record any related SRCs that may indicate a FSP problem.

Step B8: (DHCP only) Debug HMC DHCP

If a lease has still not been obtained (clients= is empty), then verify that the DHCP server lease file matches the "clients=" field by running the command cat /var/lib/dhcp/dhcpd.leases. Next, configure a PC as a DHCP client.

Disconnect the ethernet cable from the FSP and attach to the PC. Use the same cable. If an address is not assigned, debug the DHCP setup and configuration.

Test the HMC eth0 port by assigning a static address or running PC diagnostics:

  • Wiggle the connection.
  • If eth0 is suspect, eth0 can be configured for a nonconflicting static address, and eth1 configured as the dhcp server.
  • Do not continue until the HMC DHCP server is working and assigning addresses.
  • After resolving the problem, reconnect the HMC to the FSP and repeat step B5.

Step B9: (POWER5 only) FSP reset for two  IP addresses on same subnet

POWER5 and P6 EM310 only
If an ASMI connection cannot be made to one or both of the HMC ports, one possibility is duplicate address/subnet on HMC1 and HMC2 ports causing a routing conflict on the FSP. Control panel 30, if available, can be used to verify this. An example of this problem would be HMC1 at 192.168.254.255 and HMC2 at 192.168.255.254. Depending on the FSP routing table both addresses might be pingable from one port but not the other, or neither pingable unless both ports are plugged in.

One resolution is to:

  • Repair cabling (one FSP port per HMC)
  • Change the HMC eth0 to a new, different private DHCP range
  • Reboot the HMC
  • Remove FSP HMCx cable for > 1 minute, the plug back in.
This link drop should force one port to obtain a new (non-conflicting IP).

Alternate resolution:

Obtain ASMI access to reset the network settings. Since the routing may be unpredictable attempt the following until you can ping one of the addresses:
 

  • Direct connect to HMC1 port.
  • Direct connect to HMC 2 port
  • Insert a hub, cable both HMC 1 and HMC2 ports and the laptop/HMC into the same hub. (This works if the routes direct replies from HMC1 port to HMC2 port and vice-versa).
Once connected use ASMI Network Services, Network Configuration, to assign a static ip in another subnet to the unused HMC port. Save. Remove the one cable. Then reset the unused port back to defaults use Network Services, Network Configuration, Reset Network Configuration button.

Another resolution is to power off, remove the FSP and perform a FSP flip switch reset to reset the network settings to their "temporary" defaults. This will also clear the FSP passwords.

Ping the FSP address:

Network connectivity is established when the FSP address can be pinged from the HMC.
If the ping works continue to Verify Application Connectivity.


C) Verify Application Connectivity

A "no connection" state may imply no network connectivity or it may imply that a problem occurred connecting to the FSP/HMC server daemon on port 30000. If the ping test from step B completes successfully, a "no connection" indicates problems with the FSP/HMC connection on port 30000. This could be an external firewall (when using an open network) or a firmware issue.  One of the more common causes is HMC or FSP memory/resource leaks associated with long uptimes and firmware bugs.

Step C1: Verify the HMC version supports the model server

The HMC can typically connect to any server family with any server firmware level that is "N to N-2".  For example V10 HMC can manage POWER10, POWER9 and POWER8.  Always check the HMC and firmware release notes for exceptions.
Resources:


The IBM FLRT tool and "Supported Combinations"  

Note: The "supported combinations" and FLRT tool describe the levels supported for managing the server in a production environment; not the minimum levels required to connect to the server.    

  • HMC NIST Security mode requires a minimum level for POWER6 and POWER7 servers.  A circumvention is to toggle the HMC security mode back to "legacy" until the firmware is updated.
  • HMC V9R2 has a minimum required firmware level for POWER7 servers.  See the HMC V9R2 release notes in fix central.

Step C2: Verify no external firewall blocks the connection

This is typically seen only in wide area network (open) topologies however ASM also has a firewall setting (Network services, Network access). Any external firewall must allow TCP ports 443 (ASM), 30000 (HMC connection), and 30001 (vterm/5250 console).

Step C3: Check the Operator Panel Value for the no connection state

The HMC will automatically add any IP address given out by its DHCP server unless the address was previously removed (without running mksysconn -o auto). If the IP/system does not appear in the server management list, use the "Add Managed System" function and specify the assigned IP address. Type in the HMC access password as prompted.


image-20220422074603-1

Use the no connection operator panel value to determine the type of network failure.

Examples:

  • x00000068 (connection reset by peer) is usually FSP not listening on port or severe error in negotiate of connection. Collect pedbg data. Attempt a FSP dump from ASMI. Check for error logs in ASMI. Attempt pinhole reset.
  • x00000071 indicates no route. Usually a problem with cabling or the HMC network config or routing table. Ping usually fails. Review previous steps for error.


Step C4: Collect firmware levels and Error logs

Always collect FW level and any error logs. If the network connection is functional (Step B above) then ASMI should be available on the HMC even though the state is no connection. If ASMI fails to launch, verify network connectivity using the ping command. Repeat step B above.

If ping works but ASMI cannot launch on the local HMC, plug a pc into the FSP HMC2 port and make an ASM connection to the default address.

Expand "System Service Aids", "Error/Event logs" and record any related SRCs. Investigate any possible hardware issue with hardware support.

When both the ASM and HMC connections are failing (and firewall has been ruled out) a likely cause is a FSP resource leak.

Step C5: Reboot HMC

Check uptime on the HMC with the who -b command. A reboot is recommended to rule out resource leaks, especially for extended uptimes.

Step C6: Reset FSP

FSP errors such as the daemon refusing the connection or some HMC "already connected" errors can be cleared by a FSP reset/reload.

For older server models and firmware this is disruptive.  For POWER7 and later it can be done concurrently under the direction of IBM hardware support using celogin (or with even newer servers) ASM menus.  Note that it must be a reset/reload.  After the reboot use ASMI to check the error logs for any errors related to the FSP.

POWER8 and later:

Soft reset of the service processor

During soft reset of the service process, the host partitions are not powered down.

To complete this operation, you must have one of the following authority levels:

  • Administrator
  • Authorized service provider

To reboot your service processor, complete the following steps:

  1. On the ASMI Welcome pane, specify your user ID and password, and click Log In.
  2. In the navigation area, expand System Service Aids.
  3. Click Soft Reset Service Processor.
  4. Click Continue to complete the soft reset.
     


Miscellaneous Connection Errors

1) The system appears in the "Server and Partition: Server Management" panel with an error state other than "No Connection" (Recovery, Incomplete, Firmware Mismatch, Version Mismatch).
See information center and the support knowledge base for debug information.

2) MTU mismatch
This issue is typically only seen when managing servers over a wide area network.
Depending on the version of HMC and power firmware, the error may surface as incomplete state, a no connection or a connection cycling between no connection/connected/connecting. The issue may be the TCP do-not-fragment (DNF) flag. HMC Version 6 and later sets the dnf bit. Customers need to ensure the network MTU is large enough or that the network passes ICMP destination unreachable as defined in RFC 1191 to allow a smaller MTU to be negotiated. The HMC ping command can be used to test for this condition. Use the -M switch to set do not fragment and -s to increase the size. See if it will pass the 1412 MTU. For example:
ping -M do -s 1412

3) HMC already connected
The most common cause is an incorrect configuration where the HMC has two connections to the same FSP. Either a) a private/static or static/static topology where the customer added both IPs to the HMC or b) a private topology where the switch is incorrectly configured/cabled allowing the HMC to connect to both FSP HMCx ports. Remove the additional ip and/or fix the switch cabling/configuration.

The issue can occur for a single connection when network or firmware issues cause the connection to cycle between connected and no connection too rapidly. Debug this as an application layer type of issue.

Another cause is "cloning" a vHMC instance which results in two hmcs having the same UUID and virtual MTMS.  "cloning" is not supported.  To resolve the issue, delete the clone and deploy a fresh instance.

Must Gather
Before escalating always gather:
- The information from section A) Gather Basic Information
- pedbg -c -q 3 from HMC
- Results of ping test and any additional debug done from step B.
- tcpdump (when requested)

[{"Product":{"code":"SSB6AA","label":"Power System Hardware Management Console Physical Appliance"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":"HMC","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}},{"Type":"MASTER","Line of Business":{"code":"LOB68","label":"Power HW"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"7063CR1","label":"Hardware Management Console (7063-CR1)"},"ARM Category":[{"code":"a8m3p000000F985AAC","label":"System States (No Connection, Incomplete, Recovery, Version Mismatch, etc.)"}],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB68","label":"Power HW"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"7063CR2","label":"Hardware Management Console (7063-CR2)"},"ARM Category":[{"code":"a8m3p000000F985AAC","label":"System States (No Connection, Incomplete, Recovery, Version Mismatch, etc.)"}],"Platform":[{"code":"PF025","label":"Platform Independent"}]},{"Type":"MASTER","Line of Business":{"code":"LOB66","label":"Technology Lifecycle Services"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"SSRUTI","label":"HMC FOR POWER"},"ARM Category":[{"code":"a8m0z0000001jEzAAI","label":"HMC Enhanced UI"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Historical Number

355412482

Document Information

More support for:
Power System Hardware Management Console Physical Appliance

Software version:
Version Independent

Document number:
643955

Modified date:
19 March 2025

UID

nas8N1019176

Page Feedback