LPM validation errors: HSCLA27C HSCL400A HSCLA29A with OS RC 82 for AIX NPIV client

Troubleshooting

Problem

LPM validation fails for virtual FC client adapter ID(s) indicating we are unable to get the physical device for those adapter IDs from the source VIOS(s).

Symptom

Errors:
HSCLA27C The operation to get the physical device location for adapter <ID> on the virtual I/O server partition <VIOS_partition_name> has failed.
The partition command is: 
migmgr -f get_adapter  -t vscsi -s U9117.MMC.10XYZ12-V2-C470 -w 13857705817975095820 -W 13857705817975095821 -d 1
The partition standard error is:
null
Details:
HSCL400A There was a problem running the VIOS command. HSCLA29A The RMC command issued to partition <VIOS_partition_name> failed.
The partition command is: 
migmgr -f get_adapter  -t vscsi -s U9117.MMC.10XYZ12-V2-C470 -w 13857705817975095820 -W 13857705817975095821 -d 1
The RMC return code is:
0
The OS command return code is:
82
The OS standard out is:
Running method '/usr/lib/methods/mig_vscsi'
82
VIOS_DETAILED_ERROR
...snip...

Cause

The probable cause may vary depending on the details included in the VIOS_DETAILED_ERROR section.

Environment

PowerVM VIOS 3.1 and 4.1 managed by an HMC.

Resolving The Problem

A more granular level of details about the failure is available under the "VIOS_DETAILED_ERROR" section in the HMC Error tab.

Carefully assess each of the following scenarios and follow the recommendation, if applicable.

Scenario #1 Either Inactive needs to be preceeded by at least 1 successful live migration or login through vfchostN adapter, for WWPN propagation

Errors:
HSCLA27C The operation to get the physical device location for adapter <ID> on the virtual I/O server partition <VIOS_partition_name> has failed.
The partition command is: 
migmgr -f get_adapter  -t vscsi -s U9117.MMC.10XYZ12-V2-C470 -w 13857705817975095820 -W 13857705817975095821 -d 1
The partition standard error is:
null
Details:
HSCL400A There was a problem running the VIOS command. HSCLA29A The RMC command issued to partition <VIOS_partition_name> failed.
The partition command is: 
migmgr -f get_adapter  -t vscsi -s U9117.MMC.10XYZ12-V2-C470 -w 13857705817975095820 -W 13857705817975095821 -d 1
The RMC return code is:
0
The OS command return code is:
82
The OS standard out is:
Running method '/usr/lib/methods/mig_vscsi'
82
VIOS_DETAILED_ERROR
Executed get_adapter on VIOS '<VIOS_partition_name>' (hostname: <VIOS_hostname>)
VFC adapter name: vfchost68 proto name: fscsi2, rc=0
domain_id for fscsi2 is: 7
VFC adapter vfchost68 is on partition 11, count= 0
VFC adapter name: vfchost69 proto name: fscsi6, rc=0
domain_id for fscsi6 is: 8
VFC adapter vfchost69 is on partition 11, count= 1
VFC adapter name: vfchost65 proto name: fscsi7, rc=0
domain_id for fscsi7 is: 8
VFC adapter vfchost65 is on partition 11, count= 2
Comparing SCSI targets for 3 VFC adapters
Comparing SCSI targets for vfchost68 seen through fscsi2 to those of vfchost55
Either Inactive needs to be preceeded by at least 1 successful live migration or login through vfchost68 adapter, for WWPN propagation
rc = 82 MIG_NOT_READY_TO_MIGRATE
Please collect HMC logs and VIOS snap data for further analysis by IBM support.
End Detailed Message.
The OS standard err is:

The key error here is "Either Inactive needs to be preceeded by at least 1 successful live migration or login through vfchost68 adapter, for WWPN propagation".

RECOMMENDATION

By LPM design, inactive mobility must be preceeded by at least 1 successful live migration for wwpn propagation.
Typically, this error is working as designed when trying to migrate an NPIV client that has never been successfully migrated before to the target system. In such cases, as the error message says, the mobile partition must first be migrated live successfully for WWPN propagation.

Scenario #2 NPIV LUN or disk level validation is enabled and LPM validation fails with "LU validation SCSI Command to get the REPORT_LUNS data failed."

If NPIV LUN or disk level validation is enabled on the Mover Service Partition, the VIOS_DETAILED_ERROR data (as shown in the sample below) may be due to an issue on the SAN (storage zoning or LUN masking issue):

HSCL400A There was a problem running the VIOS command. HSCLA29A The RMC command issued to partition <VIOS_partition_name> failed.
The partition command is:
migmgr -f get_adapter -t vscsi -s U9117.MMD.21846D7-V3-C465 -w 13857705828798562460 -W 13857705828798562461 -c RPA -a ACTIVE_LPM -d 0
The RMC return code is:
0
The OS command return code is:
82
The OS standard out is:
82

VIOS_DETAILED_ERROR
Executed get_adapter on VIOS '<VIOS_partition_name>' (hostname: <VIOS_hostname>)
get_adapter called with NPIV LU level validation
domain_id for fscsi0 is: 143
Found target WWPN=500507680c216ab9 with SCSI_ID=8f1580
Found target WWPN=500507680c216ab6 with SCSI_ID=8f15e0
Found target WWPN=500507680c216db5 with SCSI_ID=8f1780
Found target WWPN=500507680c216dac with SCSI_ID=8f17a0
VFC adapter name: vfchost10 proto name: fscsi2, rc=0
domain_id for fscsi2 is: 144
VFC adapter vfchost10 is on partition 34, count= 0
Comparing SCSI targets for 1 VFC adapters
Comparing SCSI targets for vfchost10 seen through fscsi2 to those of vfchost11
Found target WWPN=500507680c526ab6 with SCSI_ID=d80860
Found target WWPN=500507680c526ab9 with SCSI_ID=d808a0
Found target WWPN=500507680c526dac with SCSI_ID=d808e0
Found target WWPN=500507680c526db5 with SCSI_ID=d80920
Match 0 targets, source has 4 targets
num_paths = 1 num_phys_adap_used = 1
domain_id for fscsi0 is: 143
LU validation SCSI Command to get the REPORT_LUNS data failed.
SCSI command timed out and was cancelled by VIOS.

Failed to get the Fibre Channel devices information provisioned to the client partition, rc = 82.
Operation could be retried after turning off NPIV Logical Unit validation.
rc = 82 MIG_NOT_READY_TO_MIGRATE
Please collect HMC logs and VIOS snap data for further analysis by IBM support.
End Detailed Message.
The OS standard err is:

The purpose of NPIV LUN or disk level validation is to determine if the SAN disks mapped to the client partition on the source system are properly mapped to the destination system. If they are not, the LPM validation phase will fail.
LPM requires that the client's storage be zoned to the inactive WWPN for each virtual FCA on the mobile partition. The destination Virtual I/O Servers must contain an NPIV-capable physical FC adapter that is connected to the NPIV-enabled port on the switch that has connectivity to a port on a SAN device that has access to the same targets as the client is using on the source system. In general, this is the requirement for a successful Live Partition Mobility.

RECOMMENDATION
1. Using lsmap command, determine the vfchost# corresponding to the adapter ID location code on the <VIOS_partition_name> for which the error was logged against. In this example, the adapter ID location code is U9117.MMD.21846D7-V3-C465).
$ lsmap -all -npiv|grep -i adapter_id_location_code

2. Identify the client’s virtual FC adapter (fcs#) associated with the vfchost in step 1. Then, log in to the mobile partition and verify ALL the paths are enabled for the virtual FC adapter.

3. If all paths are Enabled, contact your SAN administrator to validate switch zoning and storage provisioning is in place for the inactive WWPN of all virtual fibre channel client adapters.

Scenario #3 LPM Validation fails with HSCLA27C HSCL400A HSCLA29A with OS command return code 82 when Target Port being used for LU validation is busy when there are unsupported devices (Tape or Lun ID is 0 in devscan) zoned.

HSCL400A There was a problem running the VIOS command. HSCLA29A The RMC command issued to partition <VIOS_partition_name> failed. The partition command is: migmgr -f get_adapter -t vscsi -s U9119.MME.21D4D67-V2-C5 -w 13857705828798365952 –W 13857705828798365953 -c RPA -a ACTIVE_LPM -d 0 The RMC return code is: 0 The OS command return code is: 82 The OS standard out is: VIOS_DETAILED_ERROR Executed get_adapter on VIOS <VIOS_partition_name> (hostname: <VIOS_partition_name>) get_adapter called with NPIV LU level validation domain_id for fscsi0 is: 3 VFC adapter name: vfchost23 proto name: fscsi1, rc=0 domain_id for fscsi1 is: 2 VFC adapter vfchost23 is on partition 8, count= 0 Comparing SCSI targets for 1 VFC adapters Comparing SCSI targets for vfchost23 seen through fscsi1 to those of vfchost22 Match 0 targets, source has 8 targets num_paths = 1 num_phys_adap_used = 1 domain_id for fscsi0 is: 3 LU validation system call (START_TARGET_PORT) failed for storage target port with WWPN = 0x500507680c216abc Target Port being used for LU validation is busy. Reason : Migrating client partition is performing certain operations on the target port, interfering with LU validation process. Failed to get the Fibre Channel devices information provisioned to the client partition, rc = 82. Operation could be retried after turning off NPIV Logical Unit validation. rc = 82 MIG_NOT_READY_TO_MIGRATE Please collect HMC logs and VIOS snap data for further analysis by IBM support. End Detailed Message. The OS standard err is:

The key message says "Target Port being used for LU validation is busy."

During LPM validation on source frame, there will be a check to validate if the configuration can be migrated to destination frame. This information is collected in an XML file, which is transferred to the destination frame for validation. In above error it failed during LPM validation on source frame.
In LPM validation src_lun_val script is used and it verifies storage luns on the source frame which includes making a list of storage luns, and running validation of the storage luns on destination frame. This verification of lun validation in the above example is failing.

The lun validation are only valid for SAN disks and supported devices. In the devscan output, we see some extra SAN devices (Lun_Id 0 in devscan) being mapped/masked.
In most cases, common practice of SAN admin is to mask a lun id 0 device or it would fail with LUN Validation as above mentioned error.

This text is revealed when the collapsible item is expanded.

RECOMMENDATION

Devscan of the LPAR is required for debugging the issue.

The purpose of devscan is to make debugging storage problems faster and easier. Devscan does this by rapidly gathering a great deal of information about the Storage Area Network (SAN) and displaying it in an easy to understand manner. Devscan can be run from any AIX host, including VIO clients, or from a VIOS.

1. Devscan tool can be downloaded using below link

https://www.ibm.com/support/pages/devscan-tool

2. unzip the devcan tool using below command,

# uncompress devscan.tar.Z
# tar -xvf devscan.tar

3. Run below command to get the devscan:

# /usr/local/bin/devscan -FcX

After running the output should be the similar output as following, however the identifiers will be unique to your system

For example : target device 500507680c216xyz having LUN ID 0 in fcs0. Need to identify similar target devices having LUN ID 0

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Processing FC device:   
Adapter driver: fcs0 
Protocol driver: fscsi0 
Connection type: fabric 
Link State: up 
Local SCSI ID: 0x032262 
Local WWPN: 0xc050760627510abc 
Local WWNN: 0xc050760627510def 
Device ID: Virtual FC Adapter 
SCSI ID                          'LUN ID'              WWPN   WWNN 
-----------------------------------------------------------------------------------
032700                        '0000000000000000'    500507680c216xyz 
Vendor ID: IBM          Device ID: 2145     Rev: 0000 NACA: yes 
PDQ: Not connected      PDT: Unknown 
Name:     No ODM match  VG:       None found 
Dynamic Tracking Enabled 
TUR SCSI status: 
Check Condition (sense key: (0x05) ILLEGAL_REQUEST; ASCQ: (0x250 
0) LOGICAL UNIT NOT SUPPORTED) 
ALUA-capable device

Extra targets (500507680c216xyz) on which we see LUN_ID:0 (unsupported devices) being seen which causes the issue. SAN team need to remove zoning of extra targets.

By Kamal Raj M and Deepak Menezes

Scenario #4 HSCLA27C rc = 82

HSCLA27C The operation to get the physical device location for adapter U9080.HEX.78XXXF8-V2-C53 on the virtual I/O server partition VIOS has failed. The partition command is: migmgr -f get_adapter -t vscsi -s U9080.HEX.78XXXF8-V2-C53 -w 138577########101182 -W 138577########101183 -c RPA -a ACTIVE_LPM -M 0x1 -L MOBILE_LPAR -l 19 -i 137188########794939 -e 1-2 -d 1 The partition standard error is: Running method '/usr/lib/methods/mig_vscsi' VIOS_DETAILED_ERROR Executed get_adapter on VIOS 'VIOS' (hostname: VIOS_hostname) get_adapter called with NPIV LU level validation Comparing targets for 1 VFC adapters Comparing targets for vfchost15 seen through fcs1 to those of vfchost14 num_paths = 1 num_phys_adap_used = 1 LU validation system call (START_TARGET_PORT) failed for storage target port with WWPN = 0x50050768101565f0. Target Port being used for LU validation is busy. Reason : Migrating client partition is performing certain operations on the target port, interfering with LU validation process. Failed to get the Fibre Channel devices information provisioned to the client partition, rc = 82. Operation could be retried after turning off NPIV Logical Unit validation. rc = 82 MIG_NOT_READY_TO_MIGRATE Please collect HMC logs and VIOS snap data for further analysis by IBM support. End Detailed Message.

RECOMMENDATION

Change the sw_prli_rjt attribute of the physical Cavium or QLogic adapter to "yes" on the source VIOS. This will turn on PRLI reject to disable device scanning on the SAN switch.

To change the fiber channel adapter attribute while the adapter is in use, run:

$ chdev -dev fcs# -attr sw_prli_rjt=yes -perm

fcs# changed

where fcs# is the fibre channel adapter on the source VIOS bridging NPIV traffic for the mobile partition in question.
Reboot the VIOS for the change to take effect.

For more details, see Certain Fibre Adapters in IBM Power Systems are Incorrectly Registered as Targets instead of Initiators in the SAN fabric.

OTHER PROBABLE CAUSES

1. Verify the mobile partition has all paths enabled thru all the virtual FC client adapter IDs for which the errors were generated. If any paths show failed, correct that before retrying LPM.

2. If the mobile partition has any virtual FC adapters with no storage configured on them, they should be removed before retrying LPM.

Related Information

Partition Mobility Testcase Collection for HMC Managed Systems

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSPHKW","label":"PowerVM Virtual I\/O Server"},"ARM Category":[{"code":"a8m50000000L0K3AAK","label":"LPM"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"All Versions"}]

Tips