Solving common problems

Edit online

When you receive errors for any ksysmgr commands, the command output shows the error and the suggested resolution. However, if you are not able to determine the issue, review the following approaches to diagnose the issue.

The discovery operation failed.
If a problem occurs during the discovery steps, you must analyze the ksysmgr.log file for any failures. After the discovery is complete, you can query the IBM.VMR_LPAR resource class to confirm the successful completion of discovery operation:
lsrsrc IBM.VMR_LPAR
The output might be similar to the following sample:
        Name                = "xxx"
        LparUuid            = "59C8CFxx-4Bxx-43E2-A0CE-F028AEB5Fxxx"
        LparIPList          = {}
        SiteCleanupTastList = {}
        ActiveSiteID        = 80708xxxx
        LCB                 = {  }
        BootDiskList        = {}
        CecUuid             = "6ce366c5-f05d-3a12-94f8-94a3fdfcxxxx"
        ErrMsg              = ""
        Phase               = "READY"
        PhaseDetail         = 4194305
        Memory              = "4352"
        Processors          = "0.1"
        ActivePeerDomain    = "vmdr"
In case of any errors in the discovery operation, the Phase field is set as VERIFY and the ErrMsg field indicates the error details. The Phase field is set as READY after a successful discovery operation.
The discovery operation failed with the getlcb error.
The cause for this error might be that the virtual machine's Fibre Channel port in the storage area network (SAN) fabric is zoned with a storage port that does not provide any logical unit numbers (LUNs) to the virtual machine. You can resolve this error by completing one of the following steps:
  • Ensure that the virtual machine is zoned only with those storage ports that provide LUNs to the virtual machines.
  • Run the cfgmgr command in the virtual machine where the getlcb failure occurred and then run the discovery operation again.
The discovery operation failed indicating that the storage disk was already a part of an existing composite group.
If any of the storage disks in the VM Recovery Manager DR solution are already a part of an existing composite group, the discovery operation cannot complete successfully. A storage disk in the VM Recovery Manager DR solution must be associated with a single composite group, which is asynchronously consistent. Remove the older composite groups, and run the discovery operation again.
The discovery operation failed indicating that the disk group is created in only one site.
Review the /var/ksys/log/ksys_srdf.log file for any consistency-enabled issue. Ensure all the disks that belong to a Remote Data Facility (RDF) group are also a part of the composite group.
The discovery operation failed indicating that the target port being used for LU validation is busy.
This error occurs when the identified target storage port does not respond to the commands. If this error persists and the storage device at the identified target port does not support the VM Recovery Manager DR solution, ensure that the virtual machine is zoned with storage ports of the storage device that are supported by the VM Recovery Manager DR solution.
The verification phase failed.
After the validation is complete, you can query the IBM.VMR_LPAR resource class to ensure that the virtual machines are ready to be moved during a disaster situation:
lsrsrc IBM.VMR_LPAR
The output might be similar to the following sample:
        Name                = "xxx"
        LparUuid            = "59C8CFxx-4Bxx-43E2-A0CE-F028AEB5Fxxx"
        LparIPList          = {}
        SiteCleanupTastList = {}
        ActiveSiteID        = 80708xxxx
        LCB                 = {  }
        BootDiskList        = {}
        CecUuid             = "6ce366c5-f05d-3a12-94f8-94a3fdfcxxxx"
        ErrMsg              = ""
        Phase               = "READY_TO_MOVE"
        PhaseDetail         = 4194305
        Memory              = "4352"
        Processors          = "0.1"
        ActivePeerDomain    = "vmdr"
In case of any errors in configuration validation, review the error details in the ErrMsg field. The Phase field is set as READY_TO_MOVE after a successful verification operation.
The test-discovery step in the DR failover rehearsal operation is failing with the error message: Tertiary storage copy is missing.

This error occurs when one or more of the third copy of disks required for cloning the backup storage data is missing. For each backup (2nd copy) disk, a corresponding tertiary (third copy) disk must exist in the backup site. Check the availability and accessibility of the tertiary disks in the storage subsystem. You can also check the status of cloning relationship by using commands that are provided by the specific storage vendor.

The test-discovery step in the DR failover rehearsal operation is failing with the error message: Storage agent is not accessible.
This error occurs because of a problem in communication between the KSYS subsystem and storage subsystem. Check for any hardware issues. For example, ensure proper connectivity between all the subsystems. Also, identify the issue by analyzing resource manager trace log files.
The HMC interface indicates that the LPAR has no Resource Monitoring Control (RMC) connection or the RMC is inactive.
Check whether the LPAR properties also indicate an RMC issue between the HMC and VIOS. The RMC connectivity issue can occur because of the security mode that is set in the LPAR. The security mode for both the HMC and LPAR must be set to the same value.
For example, list the security mode for LPAR by running the following command:
/usr/sbin/rsct/bin/lssecmode
The output might look like the following sample:
Current Security Mode Configuration
Compliance Mode : nist_sp800_131a
Asymmetric Key Type : rsa2048_sha256
Symmetric Key Type : default
Similarly, list the security mode for the HMC by running the following command:
/usr/sbin/rsct/bin/lssecmode
The output might look like the following sample:
Current Security Mode Configuration
Compliance Mode : none
Asymmetric Key Type : rsa512
Symmetric Key Type : default
In this case, the LPAR has the nist_xxx security mode enabled, but the HMC has no security mode. This mismatch can occur if another HMC was connected or a security mode was set before any reset operation was started.
Errors occur when you register an EMC storage agent.
Check the /var/symapi/config/netcnfg file to determine whether the configuration contains at least two EMC subsystems.
You want to view all the EMC SRDF storage disks in the active site and the backup site.
Run the following command to list all the EMC SRDF storage disks:
# symcfg list
The output might be similar to the following sample:
                                S Y M M E T R I X                         

                                       Mcode    Cache      Num Phys  Num Symm
    SymmID       Attachment  Model     Version  Size (MB)  Devices   Devices 

    000196800508 Local       VMAX100K  5977      217088        65      9964
    000194901326 Remote      VMAX-1SE  5876       28672         0       904
    000196800573 Remote      VMAX100K  5977      217088         0      7275
    000198701861 Remote      VMAX10K   5876       59392         0       255
The output of ksysmgr query command is not updated for hosts, HMCs, and VIOS even when the entities are updated.
Sometimes, the ksysmgr query command displays static information even when the hosts or HMCs are modified. To update the ksysmgr query command output dynamically, complete the following steps:
  1. Unpair the hosts across the sites by using the following command:
    ksysmgr pair host host_name pair=none
  2. Remove the hosts from the site by using the following command:
    ksysmgr remove host host_name
  3. Add the new HMC to the site by using the following command:
    ksysmgr add hmc hmc_name login=login_name password=login_password 
       ip=ip_address  site=site_name
  4. Add the corresponding managed hosts to the site by using the following command:
    ksysmgr add host host_name
The disaster recovery move operation failed.
When a disaster occurs and you initiate a move operation, the KSYS subsystem coordinates the start of the virtual machines on the backup site. During this process, LPAR profiles are created through HMCs on the backup site hosts. During the LPAR profile creation, if any errors occur such that the KSYS cannot communicate with HMCs, the move operation might fail. At that time, any virtual machines that are partially created on HMC require manual restart. For the rest of the virtual machines, you can use the ksysmgr recover command to recover and start the virtual machines.
An unplanned move operation from the active site to the backup site is successfully completed and the specified flexible capacity policy is followed. Later, another unplanned move operation from the backup site to the source site fails. When the virtual machines are recovered to the source site, the virtual machines are started but without any change in the processor and memory values, (that is, without following the flexible capacity policy).
This situation can happen when the active hosts and backup hosts are connected to the same HMC. You must connect the source hosts and target hosts to different HMCs to continue unplanned move operations in case of source HMC failures.
LPM verification operation fails with error: HSCLB401 The maximum number of partition migration commands allowed are already in progress. Try again later.
In the scalability environment where the VMs are spread across the hosts of a host group, and the LPM verification operation is run on the host group, based on the type of configuration, at some point of time, many requests might go to one host and if the number of requests are more than the maximum requests that the host can handle, the verification might fail with the error: HSCLB401 The maximum number of partition migration commands allowed are already in progress. To resolve this error, re-run the verify operation on the host group.
Error message that the failed VMs are not part of the workgroup
When a workgroup move operation fails, run the recovery command or the move command ,based on the failure state, on the failed workgroup before running a move operation on the host group that includes the workgroup.
Database access lost after VIOS upgrade
The database access might be lost after upgrading the VIOS. In this case, stop the VIOS daemon (vio_daemon) node by node and verify whether the VIOS daemon selected a database node (DBN). If the database node (DBN) is not selected, restart the VIOS daemon on all nodes that you stopped. If the VIOS daemon still does not select a database node (DBN), restart one of the nodes and wait until the VIOS daemon selects the database node (DBN) for the restarted node. To check the status, whether the database node (DBN) is selected, run the following command:
cluster status –verbose
VM failed during the Live Partition Mobility (LPM) process
If a VM fails during LPM, run the LPM recovery operation to recover the failed VM. If the VM is not recovered after the LPM recovery operation, run the restart operation on the VM.

You cannot log in to the VM Recovery Manager DR GUI

Problem
You cannot log in to the VM Recovery Manager DR GUI.
Solution
  1. Check for issues in the /opt/IBM/ksys/ui/server/logs/uiserver.log file.
  2. If you received an error message, Permission missing on Smuiauth: login will not be done, verify that the smuiauth command is installed correctly. Also, verify that the smuiauth command has the correct permissions by running the ls -l command from the /opt/IBM/ksys/ui/server/lib/auth/smuiauth directory. An example output follows: 
    (0) root @ gui-ksys01: /opt/IBM/ksys/ui/server/lib/auth 
# ls -l smuiauth 
-r-x------ 1 root system 71782 Jan 28 00:13 smuiauth
  3. Verify that you can run the smuiauth command successfully by running the command along with the -h flag.
  4. Verify that the pluggable authentication module (PAM) framework is configured correctly by locating the following lines in the /etc/pam.conf file:
    smuiauth        auth       required     pam_aix
smuiauth        account    required     pam_aix
    The PAM is configured when you install the ksys.ui.server fileset.

You cannot register a KSYS node in the VM Recovery Manager DR GUI server

Problem
You cannot register a KSYS node in the VM Recovery Manager DR GUI server.
Solution
  1. Check for issues in the /opt/IBM/ksys/ui/server/logs/uiserver.log file by performing the following steps:
    1. If SSH File Transfer Protocol (SFTP)-related signatures exist in the log file, such as Received exit code 127 while establishing SFTP session, a problem exists with the SSH communication between the VM Recovery Manager DR GUI server and the KSYS node that you are trying to add.
    2. From the command line, verify that you can connect to the target system by using SFTP. If you cannot connect, verify that the daemon is running on the GUI server and the target node by running the ps -ef | grep -w sshd | grep -v grep command.
    3. Check the SFTP subsystem configuration in the /etc/ssh/sshd_config file and verify that following path is correct.
      Subsystem       sftp    /usr/sbin/sftp-server
      If the path is not correct, you must enter the correct path in the /etc/ssh/sshd_config file, and then restart the sshd subsystem.
  2. Check for issues in the /opt/IBM/ksys/ui/agent/logs/agent_deploy.log file on the target cluster.

Node server down: GUI fails to start

Problem
The VM Recovery Manager DR GUI server is not working correctly.
Solution
If the applications are not running correctly, the node server status might be causing the issue. Run the ps -ef | grep node command to check the status and run the startsrc -s vmruiserver command to start the node server.

Unplanned system reboot causes fallover attempt to start GUI

Problem
You cannot access the VM Recovery Manager DR GUI because of an unplanned GUI server or GUI agent node reboot operation.
Solution
During the system node reboot operation, you cannot access the GUI. Run the lssrc -s vmruiserver command to check the status of the vmruiserver subsystem. 
#lssrc -s vmruiserver
Subsystem       Group     PID     Status
vmruiserver     vmrui             inoperative
If the status of the vmruiserver subsystem is displayed as inoperative, run the startsrc -s vmruiserver command to restart the UI server node from the command line. You can then access the GUI and register the agent nodes again.

Unsuccessful Deployment: Dependency file missing during installation

Problem
A dependency file is missing during the installation of the GUI server and the GUI agent filesets.
Solution
Determine the missing file from the log files that you received by using the installp -e flag and install that dependency file from a certified host.

You cannot stop or start the GUI server and GUI agent processes

Problem
You cannot stop or start the GUI server and agent processes.
Solution
  • GUI server: Stop the GUI server by running the following command: stopsrc -s vmruiserver.

    Restart the GUI server by running the following command: startsrc -s vmruiserver. If you are starting the GUI server for the first time after installing GUI server, run the vmruiinst.ksh command. For information about running this command, see Installing GUI filesets.

  • GUI agent: Stop the GUI agent process by running the following command in the guest VM: stopsrc -s vmruiagent. This command unregisters the KSYS node from the GUI server and the KSYS node will no longer be accessible from the GUI server.

    Restart the GUI agent by running the following command: startsrc -s vmruiagent. This command registers the KSYS node again.

    If you don’t have the GUI server installed on the same machine, run the vmragentinst.ksh command. For information about running this command, see Installing GUI filesets.

VM does not boot after move operation

Problem
After the move operation, virtual machines do not boot in the target site.
Solution
Restart the virtual machines by using the sample script that is provided at the following location: /opt/IBM/ksys/storages/utils. For more information about the usage of the script, run the hg_vm_status -h command.
You can run this script from a KSYS node or a non-KSYS node. While running the script from a non KSYS node, you can use the -ksys option to provide the KSYS node details. The passwordless SSH communication should be established across KSYS node, non-KSYS node, and HMCs:
  • To enable passwordless SSH communication between a KSYS node and a non-KSYS node, run the following commands:
    1. Run the following cd command:
      cd /.ssh
    2. Run the following scp command:
      scp root@ksysnode_name:.ssh/authorized_keys /.ssh/tmp_authorized_keys
      An output that is similar to the following example is displayed:
      (0) root @ ksys204: /.ssh
# scp root@ksys205gui:.ssh/authorized_keys /.ssh/tmp_authorized_keys
root@ksys205gui's password:
authorized_keys                               100% 7445   122.4KB/s   00:00
    3. Run the following cat command:
      cat ~/.ssh/id_rsa.pub >> ~/.ssh/tmp_authorized_keys
      An output that is similar to the following example is displayed:
      (0) root @ ksys204: /.ssh
# cat ~/.ssh/id_rsa.pub >> ~/.ssh/tmp_authorized_keys
    4. Run the following scp command:
      scp ~/.ssh/tmp_authorized_keys root@ksysnode_name:.ssh/authorized_keys
      An output that is similar to the following example is displayed:
      (0) root @ ksys204: /.ssh
# scp ~/.ssh/tmp_authorized_keys root@ksys205gui:.ssh/authorized_keys
root@ksys205gui's password:
tmp_authorized_keys                           100% 7862     5.9MB/s   00:00
  • To enable passwordless SSH communication between a KSYS node or a non-KSYS node and HMCs, run the following commands:
    1. Run the following cd command:
      cd /.ssh ; mykey=`cat id_rsa.pub`
      An output that is similar to the following example is displayed:
      (130) root @ ksys204: /.ssh
# cd /.ssh ; mykey=`cat id_rsa.pub`
    2. Run the following ssh command:
      ssh <hmc_user>@<hmc_name> mkauthkeys -a \"$mykey\"
      An output that is similar to the following example is displayed:
      (0) root @ ksys204: /.ssh
# ssh hscroot@e17vhmc2.aus.stglabs.ibm.com mkauthkeys -a \"$mykey\"
Password:

This script helps you to debug and reboot a virtual machine that is not in the running state. You must specify the host group name or workgroup name if the workgroup is configured. Also, you must provide the KSYS node details if you are running the script from a non-KSYS node. To restart a virtual machine, run the script with -r or -reboot option.

For example:
  • To check a virtual machine status, run the following command: 
    hg_vm_status -hg HG1
    An output that is similar to the following example is displayed:
    ################################################################################
 VMDR Tools to check lpar status
################################################################################

S.NO  VM Name           Host Name                         Status

1    ziti036           ragu-9119-MME-SN106CDC7           Running
2    ziti004           ragu-9119-MME-SN106CDC7           Running
3    ziti043           ragu-9119-MME-SN106CDC7           Running
4    ziti010           ragu-9119-MME-SN106CDC7           AA06000D
################################################################################
End
  • To restart a virtual machine with a reference code, run the following command: 
    hg_vm_status -hg HG1 -c AA06000D -r
    An output that is similar to the following example is displayed:
    ################################################################################
 VMDR Tools to check lpar status
################################################################################

S.NO  VM Name           Host Name                         Status

1    ziti036           ragu-9119-MME-SN106CDC7           Running
2    ziti004           ragu-9119-MME-SN106CDC7           Running
3    ziti043           ragu-9119-MME-SN106CDC7           Running
4    ziti010           ragu-9119-MME-SN106CDC7           AA06000D
Rebooting LPAR ziti010
ssh hscroot@<hmc_ip> chsysstate -m ragu-9119-MME-SN106CDC7 -r lpar -n ziti010 -o shutdown   --immed --restart
Rebooted LPAR ziti010
################################################################################
End
  • To check the status of a virtual machine that is from a non-KSYS node that has workgroup configured, run the following command:
    hg_vm_status -ksys ksys913 -hg Default_WG_2
    An output that is similar to the following example is displayed:
    ################################################################################
 VMDR Tools to check lpar status
################################################################################

S.NO  VM Name              Host Name         Status

1    e10josh007           e10josh           Running
2    ziti049              e10josh           Running
3    e10josh004           e10josh           Running
4    e10josh009           e10josh           AA00E158
LPAR require to reboot : ['e10josh009']

 Run with -reboot option for rebooting
################################################################################

VM is slow to boot after move operation

Problem
After the move operation, the virtual machines are slow to boot in the target site.
Solution
The boot operation of the VMs that are running on the Red Hat Enterprise Linux (RHEL) VM agent might take a longer time than expected. Because, multiple storage disks that are associated to the VM open during the boot operation. Consequently, the move operation on a host group might complete with errors, and the host group goes into the recovery state. To resolve the error, run the ksysmgr recover command. You can prevent this error by setting 2 in the ofdisk_retries grub variable in your RHEL environment. To set the ofdisk_retries grub variable to 2 in your RHEL environment, run the following command:
grub2-editenv /boot/grub2/grubenv set ofdisk_retries=2
To set the disk_retries variable, RHEL version 8.7, or a hotfix of RHEL is required.

Target ports are not reachable at remote site (zoning issue)

Problem
When a storage area network (SAN) zoning error occurs at the switch level, it indicates that the storage ports are corrupt or are missing from the zone, or the World Wide Port Number (WWPN) of the LPAR is missing or is wrong. In this scenario, WWPN of the LPAR cannot access the storage system. The ksysmgr displays the following error message: 
"VIOS <vios1> has returned error for VM <vm_name>:
Following error is returned for VM <vm_name>: This physical port can not access storage for the client wwpn '<vm_wwpn>'
Following error is returned for VM <vm_name>: No target ports found through 'fscsi3'.
Following error is returned for VM <vm_name>: This physical port can not access storage for the client wwpn '<vm_wwpn>'
Following error is returned for VM <vm_name>: No target ports found through 'fscsi2'.
Following error is returned for VM <vm_name>: Please correct the LU masking from Storage and retry.
Following error is returned for VM <vm_name>: rc = 83 MIG_RESOURCE_NOT_FOUND"
Solution
Ensure that the WWPN of the VM and the SAN are zoned correctly at the target switch such that the VIOS can access the replicated disk by using the WWPN of the VM.

One VIOS does not have access to SAN in a dual-path configuration

Problem
In a dual-path configuration, replicated disks are expected to be available from two Virtual I/O Servers on the target site. Both Virtual I/O Servers must be able to access the storage system. If one of the configuration paths has a problem, the KSYS subsystem indicates an error with the initiator details that you can use to debug the problem. The ksysmgr displays the following error message: 
"Following error is returned for VM <vm_name>: Executed find_devices on VIOS 'vios1' (hostname: vios1.ausprv.stglabs.ibm.com)
Following error is returned for VM <vm_name>: domain_id for fscsi0 is: 12
Following error is returned for VM <vm_name>: Start initiator failed. errno=69
Following error is returned for VM <vm_name>: scsi_sciolst error info: adap_flags: 0x0, failure_type: 0, retry_count=0,
Following error is returned for VM <vm_name>: fail_reason_code: 0, fail_reason_exp: 0, einval_arg: 0
Following error is returned for VM <vm_name>: Start initiator failed for wwpn <vm_wwpn> on /dev/fscsi0
Following error is returned for VM <vm_name>: There are no adapters available that are capable of supporting a virtual fibre channel adapter
Following error is returned for VM <vm_name>: rc = 69 MIG_LACK_RESOURCE"
Solution
Ensure that all Virtual I/O Servers on the target site have access to the replicated disk from the target switch.

VIOS shows additional disks on the target site

Problem
Hosts might be mapped to extra disks on the target storage system or the switch zoning might show additional disks and the Virtual I/O Server is also showing these additional disks. The error indicates the disk count mismatch at the source and destination sites and indicates the additional disks with the disk ID details. You can use the disk IDs to remove the disks from the host. The ksysmgr displays the following error message: 
VIOS vios1 has returned error for VM <vm_name>:
Following error is returned for VM <vm_name>: List of Logical Units found extra on the destination VIOS:
Following error is returned for VM <vm_name>: Logical Unit 1 : Descriptor VOL_ID = <vol_id>, VOL_NAME=<vol_name>, managed_disk=0
Following error is returned for VM <vm_name>: Logical Units discovered through 'fscsi3' for the client's WWPN = <vm_wwpn> on
Following error is returned for VM <vm_name>: destination did not match with the logical units collected on source.
Following error is returned for VM <vm_name>: List of Logical Units found extra on the destination VIOS:
Following error is returned for VM <vm_name>: Please correct the LU masking from Storage and retry.
Following error is returned for VM <vm_name>: rc = 83 MIG_RESOURCE_NOT_FOUND
Solution
Additional disks must not be masked with the existing host in the storage system. If the disks are masked, remove the additional disk from the target host in the storage system.

LUN masking is not available from target site or LUN masking is incorrect

Note: The following solutions are applicable to only EMC SRDF storage subsystem, SAN Volume Controller system, and Hitachi storage systems.
Problem
The World Wide Port Name (WWPN) is not added to any host in the target storage system. The ksysmgr commands displays the following error message during the discovery operation:
SM ERROR:99:Disk Pair, Unable to find the WWPN <wwpn> for disk <diskID> on the storage <storage_name>
Solution
Map the World Wide Port Name (WWPN) of the storage system to the host.
Problem
The logical unit number (LUN) is not added to any host in the storage system or the LUN is added to the incorrect host in the storage system. The ksysmgr command displays the following error message during the discovery operation:
SM ERROR:96:Disk Pair, Host-Lun mapping missing for <host>-><replicated disk> on the storage <storage_name>
Solution
Map the replicated disk in the storage system to the host.

Some Fibre Channel adapters cannot access disks

Problem
Two Fibre Channel (FC) adapters are connected to a source site and only good FC adapter is available on the target site. Also, enough FC adapters are not available to host the new virtual machines. The ksysmgr displays the following error message:
0 18874832 49807673 03/02/21-08:21:39.861 dr_util.c 1.39 153] :49807673 --  violibDB.c           modifyDB             1.
132.2.548 1955   Statement = UPDATE viosdr.vios_partition SET failed=1 WHERE viosdr.vios_partition.vios_part_num='2' AND
viosdr.vios_partition.client='5079073E7ABF4724893E8C8831BD8DBC'
[0 18874832 49807673 03/02/21-08:21:39.865 npiv_set_adapter.c 1.19.2.107 3213] npiv_dest_adapter: rc = 5.
[0 18874832 49807673 03/02/21-08:21:39.865 vsmig_set.c 1.18.2.51 315] rc = 5 MIG_NPIV_LOST_PATH
[0 18874832 49807673 03/02/21-08:21:39.872 mig_vscsi.c 1.11.2.46 1019] Leaving mig_vscsi fn=find_devices, rc=5
[ERROR 18874832 49807673 03/02/21-08:21:39.872 mig_vscsi.c 1.11.2.46 1034] Executed find_devices on VIOS 'vios1' (hostname: vios1.ausprv.stglabs.ibm.com)
VIOS_DETAILED_ERROR
Source paths = 2 Destination paths = 1
Error 106 detected on vios1.
There are not enough physical fibre channel ports on the destination
 VIOS to allow this client to maintain path redundancy.  The migration
 will result in the loss of one or more redundant paths leaving the
 client vulnerable to hardware/cable failure
rc = 5 MIG_NPIV_LOST_PATH
End Detailed Message.
Solution
Ensure that all the required configuration paths are available from one Virtual I/O Server (VIOS) such that all configuration paths have access to the replicated disks on the target site.

One FC adapter from VIOS is mapped to more than one virtual Fibre Channel of the VM

Problem
As an example if the Fibre Channel fcs0 is mapped to the virtual Fibre Channels vfchost0 and vfchost1 of the same virtual machines, the discovery operation will fail. The ksysmgr displays the following error message: 
VIOS vios1 has returned error for VM <vm_name>:
Following error is returned for VM <vm_name>: Executed get_adapter on VIOS 'vios1' (hostname: vios1.ausprv.stglabs.ibm.com)
Following error is returned for VM <vm_name>: domain_id for fscsi3 is: 16
Following error is returned for VM <vm_name>: VFC adapter name: vfchost9 proto name: fscsi3, rc=0
Following error is returned for VM <vm_name>: Client LPAR with ID 104 mapped to fscsi3 more than once via vfchost4 vfchost9. Cannot migrate.
Following error is returned for VM <vm_name>: rc = 80 MIG_NOT_MIGRATABLE
Following error is returned for VM <vm_name>: Please collect HMC logs and VIOS snap data for further analysis by IBM support.
Please review errors.  The "ksysmgr query system status" command may provide additional details.
You cannot stop or start the GUI server and GUI agent processes
Solution
Ensure that the virtual Fibre Channels of a physical adapter is mapped to the virtual FC host of the same virtual machine. You can remove the mapping from one of the virtual Fibre Channels by using the following command: 
unmap fcs " vfcmap -vadapter vfchost -fcp"

vLAN or vSwitch on the source site VIOS and the target site VIOS are different (at the system-level)

Problem
The disaster recovery operation fails with the HSCLA251 error. The management console cannot suggest a VIOS on the destination managed system to host the LPAR’s virtual Ethernet client adapter that is configured to use vLAN2 and virtual switch ETHERNET0. The ksysmgr displays the following error message:
The Disaster Recovery operation failed with the following errors HSCLA251 The management console is not able to suggest a Virtual I/O Server (VIOS) on the 
destination managed system to host the partition's virtual Ethernet client adapter that is configured to use VLAN 2 and virtual switch ETHERNET0.  Verify that the 
VIOSs on the source and destination managed systems are set up correctly, then try this operation again.
Solution
Verify that the Virtual I/O Servers on the source and the backup sites are set up correctly. If the vLAN IDs are different at the source and the target sites, update the KSYS subsystem by using the following command: 
ksysmgr modify system network=vlanmap sites=<src_site_name>,<bkp_site_name> src_site_name=<id> bkp_site_name=<id>

vLAN or vSwitch values are different on the source and backup site for a specific host group

Problem
The disaster recovery operation fails with the HSCLA251 error. The management console cannot suggest a VIOS on the destination managed system to host the LPAR’s virtual Ethernet client adapter that is configured to use vLAN2 and virtual switch ETHERNET0. The ksysmgr displays the following error message:
The Disaster Recovery operation failed with the following errors HSCLA251 The management console is not able to suggest a Virtual I/O Server (VIOS) on the 
destination managed system to host the partition's virtual Ethernet client adapter that is configured to use VLAN 2 and virtual switch ETHERNET0.  Verify that the 
VIOSs on the source and destination managed systems are set up correctly, then try this operation again.
Solution
Verify that the Virtual I/O Servers on the source and the backup site are set up correctly. If the vLAN IDs at the source and the target site are different, update the KSYS subsystem by using the following command: 
ksysmgr modify host_group <hg_name> network=vlanmap sites=<src_site_name>,<bkp_site_name> src_site_name=<id> bkp_site_name=<id>

Copy type mismatch error

Problem
The replication type of the disk-pair in the storage system and in the KSYS subsystem is different. The ksysmgr displays the following error message:
ERROR: KSYS has encountered storage subsystem related error.
SM ERROR:73:Disk Group: <ksys_Dg_group>, GROUP copy type mismatch. Type=metro
SM ERROR:67:Disk Group, Verification failed, run discovery.
Please check storage log file /var/ksys/log/ksys_svc.log for further investigation.
Please review errors.  The "ksysmgr query system status" command may provide additional details.
Solution
Change the replication type of the storage system by using the following command: 
ksysmgr modify system replication_type=<sync/async> sites=<site1, site2>

Mirror-pair missing or Disk-pair missing

Problem
During the discovery operation, the mirror-pair or disk-pair is missing. The ksysmgr displays the following error message: 
ERROR: KSYS has encountered storage subsystem related error.
SM ERROR:82:Disk ID: <source_storage_disk_id>, Mirror pair missing.
Please check storage log file /var/ksys/log/ksys_svc.log for further investigation.
Following error is returned for VM vm_name:
Please review errors.  The "ksysmgr query system status" command may provide additional details.
Solution
Identify the disk with an incorrect pairing and rectify it, then attempt the discovery operation again. Alternatively, create a replication or relationship between the source disk and the target disk.

Only for SVC storage : Disk role error

Problem
The discovery operation fails because the virtual machine are on the source site and the disk type at IBM SAN Volume Controller (SVC) storage system is auxiliary disks. The ksysmgr displays the following error message:
ERROR: KSYS has encountered storage subsystem related error.
SM ERROR:68:Disk Group: <KSYS_DG_Name_to_be_created>, Disk in AUX role, instead of Master role <disk_id>
Please check storage log file /var/ksys/log/ksys_svc.log for further investigation.
Please review errors.  The "ksysmgr query system status" command may provide additional details.
Solution
Remove the replication and create the storage disk at the source site using the primary-master role.

Disks part of non-KSYS group

Problem
During the discovery operation, if the KSYS subsystem finds a disk-pair that is already part of another group in the storage, the discovery operation will fail, and the disk is not added to the new group that the KSYS creates. The ksysmgr displays the following error: 
ERROR: KSYS has encountered storage subsystem related error.
SM ERROR:60:Disk Group: <ksys_dg_name_src>, Disks belong to a non-KSYS group.
SM ERROR:60:Disk Group: <ksys_dg_name_target>, Disks belong to a non-KSYS group.
Please check storage log file /var/ksys/log/ksys_svc.log for further investigation.
Please review errors.  The "ksysmgr query system status" command may provide additional details.
Solution
If the disks that are part of the KSYS subsystem belongs to another group, remove the disk-pair from the non-KSYS group. Alternatively, you can also modify the workgroup name by using the customcgname attribute to be same custom group name as the storage before performing the discovery operation.

FlashCopy mapping is missing

Problem
During the test-discovery operation, FlashCopy mapping or clone copy is not available. The ksysmgr displays the following error: 
ERROR: Discovery has encountered a DrTest Disk Pair error for Host_group HG, please investigate
04:09:13 Dr_test Discovery has finished for Site1
0 out of 3 managed VMs have been successfully discovered
ERROR: KSYS has encountered storage subsystem related error.
Disk Pair: Clone disks, Failed getting FC MAP for <disk1_id>
Disk Pair: Clone disks, Failed getting FC MAP for <disk2_id>
Disk Pair: Clone disks, Failed getting FC MAP for <disk3_id>

Please check storage log file /var/ksys/log/ksys_svc.log for further investigation.
Solution
Create FlashCopy mapping or clone copy in the target storage.

EMC unity consistency group goes to lost communication state

Problem
If jittering or any other issue occurs in the replication network of consistency group disks, the consistency group of the EMC Unity storage system goes to lost communication state.
Solution
Rectify the environment issues in the replication network of storage systems and move the consistency group to consistent or synchronous state.

Target ports are not reachable at remote site (zoning issue)

Problem
There is issue in zoning at the switch level. The issue can be storage ports are bad or missing from the zone, the WWPN of LPAR are missing or it's incorrect. When you verify, the error message will show the WWPN which are not able to access the storage. Take the corrective action on the target switch and verify whether zoning for that particular VM and storage are correct. For this issue the following error is displayed:
"VIOS <vios1> has returned error for VM <vm_name>:
Following error is returned for VM <vm_name>: This physical port can not access storage for the client wwpn '<vm_wwpn>'
Following error is returned for VM <vm_name>: No target ports found through 'fscsi3'.
Following error is returned for VM <vm_name>: This physical port can not access storage for the client wwpn '<vm_wwpn>'
Following error is returned for VM <vm_name>: No target ports found through 'fscsi2'.
Following error is returned for VM <vm_name>: Please correct the LU masking from Storage and retry.
Following error is returned for VM <vm_name>: This physical port can not access storage for the client wwpn '<vm_wwpn>'
Following error is returned for VM <vm_name>: No target ports found through 'fscsi3'.
Following error is returned for VM <vm_name>: This physical port can not access storage for the client wwpn '<vm_wwpn>'
Following error is returned for VM <vm_name>: No target ports found through 'fscsi2'.
Following error is returned for VM <vm_name>: Please correct the LU masking from Storage and retry.
Following error is returned for VM <vm_name>: rc = 83 MIG_RESOURCE_NOT_FOUND"
Solution
Remove target SAN ports from the target zone:
  1. Get switch details with lscfg -vpl fcs command from views.
  2. Check the zone details by using zoneshow command.
  3. Check the aliases of the VMs by using alishow command.
  4. Remove the storage ports from the zone using the zoneremove command.
  5. Save the configuration changes in switch by using the cfgsave command.
  6. Enable switch stk by using the cfgenable <stk_cfg> command.
Ensure that the VM WWPN and SAN WWPN are zoned properly at target the SAN switch, so that the target VIOS can see the replicated disk using the VM WWPN.

When VM need paths from both the VIOS and one VIOS don't have access to SAN

Problem
When there is dual path configuration, that means the disks are given from 2 VIOSes. The same is expected on the target site. Both VIOSes on the target site must have accessibility to the storage. If one path is missed, you get the error from KSYS that gives the initiator details. And, user can try the debug steps mentioned in the following paragraph. In case, you do not want to resolve it, you can still try the move and verify operations with lose VIOS redundancy option set to true in KSYS. For this issue the following error is displayed:
"Following error is returned for VM <vm_name>: Executed find_devices on VIOS 'vios1' (hostname: vios1.ausprv.stglabs.ibm.com)
Following error is returned for VM <vm_name>: domain_id for fscsi0 is: 12
Following error is returned for VM <vm_name>: Start initiator failed. errno=69
Following error is returned for VM <vm_name>: scsi_sciolst error info: adap_flags: 0x0, failure_type: 0, retry_count=0,
Following error is returned for VM <vm_name>: fail_reason_code: 0, fail_reason_exp: 0, einval_arg: 0
Following error is returned for VM <vm_name>: Start initiator failed for wwpn <vm_wwpn> on /dev/fscsi0
Following error is returned for VM <vm_name>: domain_id for fscsi1 is: 12
Following error is returned for VM <vm_name>: Start initiator failed. errno=69
Following error is returned for VM <vm_name>: scsi_sciolst error info: adap_flags: 0x0, failure_type: 0, retry_count=0,
Following error is returned for VM <vm_name>: fail_reason_code: 0, fail_reason_exp: 0, einval_arg: 0
Following error is returned for VM <vm_name>: Start initiator failed for wwpn <vm_wwpn> on /dev/fscsi1
Following error is returned for VM <vm_name>: domain_id for fscsi4 is: 1
Following error is returned for VM <vm_name>: Start initiator failed. errno=69
Following error is returned for VM <vm_name>: scsi_sciolst error info: adap_flags: 0x0, failure_type: 0, retry_count=0,
Following error is returned for VM <vm_name>: fail_reason_code: 0, fail_reason_exp: 0, einval_arg: 0
Following error is returned for VM <vm_name>: Start initiator failed for wwpn <vm_wwpn> on /dev/fscsi4
Following error is returned for VM <vm_name>: domain_id for fscsi5 is: 20
Following error is returned for VM <vm_name>: Start initiator failed. errno=69
Following error is returned for VM <vm_name>: scsi_sciolst error info: adap_flags: 0x0, failure_type: 0, retry_count=0,
Following error is returned for VM <vm_name>: fail_reason_code: 0, fail_reason_exp: 0, einval_arg: 0
Following error is returned for VM <vm_name>: Start initiator failed for wwpn <vm_wwpn> on /dev/fscsi5
Following error is returned for VM <vm_name>: There are no adapters available that are capable of supporting a virtual fibre channel adapter
Following error is returned for VM <vm_name>: rc = 69 MIG_LACK_RESOURCE"""
Solution
Find out the port details for all VIOSes and the switches connected to each VIOS. Disable the port by using the portdisable command. Check the status of the port by using the switchshow command. Port status will be offline.
Ensure from that target site that all VIOSes are seeing replicated disk from the target switch.

Additional Disk visible to VIOS on target site

Problem
There are chances that on the target storage the hosts are mapped with extra disks. Or, the zoning on the switch might see those disk and the VIOS too is seeing those disks. In this case, you get the error mentioning the count mismtach on the destination and source host. So the error will mention the extra disk. You can determine the disk from the disk Id and remove the disk from the host. The debug process in this sheet that shows command for SVC storage. Similarly, you can debug other storages too. For this issue the following error is displayed:
"VIOS vios1 has returned error for VM <vm_name>:
Following error is returned for VM <vm_name>: List of Logical Units found extra on the destination VIOS:
Following error is returned for VM <vm_name>: Logical Unit 1 : Descriptor VOL_ID = <vol_id>, VOL_NAME=<vol_name>, managed_disk=0
Following error is returned for VM <vm_name>: Logical Units discovered through 'fscsi3' for the client's WWPN = <vm_wwpn> on
Following error is returned for VM <vm_name>: destination did not match with the logical units collected on source.
Following error is returned for VM <vm_name>: List of Logical Units found extra on the destination VIOS:
Following error is returned for VM <vm_name>: Logical Unit 1 : Descriptor VOL_ID = <vol_id>, VOL_NAME=<vol_name>, managed_disk=0Following error is returned for VM <vm_name>: destination did not match with the logical units collected on source.
Following error is returned for VM <vm_name>: Please correct the LU masking from Storage and retry.
Following error is returned for VM <vm_name>: rc = 83 MIG_RESOURCE_NOT_FOUND"
Solution
Map extra LUN to the host from the target SAN. The following is the SVC command: storagesvctask mkvdiskhostmap -force -host <host_name> <vdisk_name>.
There must not be any extra disk masked with the existing host at the storage system. If there are any extra disks, remove the extra disk from target host at the storage system.

LUN masking missing from target site or LUN Masking done is wrong

Problem
The LUN that is newly created are not added to any host at the storage system. Or, those LUNs are added to the wrong host at the storage system. The following is displayed:
"Following error is returned for VM <vm_name>: destination did not match with the logical units collected on source.
Following error is returned for VM <vm_name>: Please correct the LU masking from Storage and retry.
Following error is returned for VM <vm_name>: List of Logical Units without mirror partner(s) on the destination VIOS:
Following error is returned for VM <vm_name>: Logical Unit 1 : Descriptor VOL_ID = <vol_id>, VOL_NAME=<vm_name>, managed_disk=0
Following error is returned for VM <vm_name>: Logical Units discovered through 'fscsi3' for the client's WWPN = <vm_wwpn> on
Following error is returned for VM <vm_name>: destination did not match with the logical units collected on source.
Following error is returned for VM <vm_name>: List of Logical Units without mirror partner(s) on the destination VIOS:
Following error is returned for VM <vm_name>: Logical Unit 1 : Descriptor VOL_ID = <vol_id>, VOL_NAME=<vm_name>, managed_disk=0
Following error is returned for VM <vm_name>: Logical Units discovered through 'fscsi2' for the client's WWPN = <vm_wwpn> on
Following error is returned for VM <vm_name>: destination did not match with the logical units collected on source.
Following error is returned for VM <vm_name>: Please correct the LU masking from Storage and retry.
Following error is returned for VM <vm_name>: rc = 83 MIG_RESOURCE_NOT_FOUND"
Solution
Unmap the replicated disk from the target SAN. For example, for SVC storage system, run the ssh admin@storwize svctask rmvdiskhostmap -host <host_name> < disk_name> command.
Map the replicated disk on the target storage system to the host.