IBM Support

IBM Spectrum Protect™ Plus on IBM® Cloud: Recovery procedure when vCenter or other management VMs are not accessible - Version2

Troubleshooting


Problem

Note: The information in this document applies only for VMware vCenter Server (VCS) environments. Use the steps in this procedure to complete the recovery operation when a management virtual machine (VM) is not accessible.

Attention:
Use this procedure for IBM Spectrum Protect™ Plus on IBM® Cloud V10.1.1patch1 and later versions.
For IBM Spectrum Protect™ Plus on IBM® Cloud V10.1.0-V10.1.1 follow the procedure described in technote: http://www.ibm.com/support/docview.wss?uid=swg22014655

Symptom

This procedure applies if any of the following management services are partially or fully lost in the IBM Cloud development environment:

  • vCenter

  • Platform Services Controller (PSC)

  • Software-Defined Data Center (SDDC)

  • Active Directory (AD)/Domain Name System (DNS) servers
     

Environment

IBM Spectrum Protect Plus provides an option to automatically restore by using ESXi hosts if the vCenter is not accessible. This option restores the vCenter VM or VMs that the vCenter is dependent on.

image-20180801155957-1

Additional manual steps are required to complete a restore operation if the original ESXi host is not available and you are recovering to a different host.
 

Resolving The Problem

Recovering to a different ESXi host could result in the loss of access to VMs. This is caused because of a misconfiguration of the virtual switch. Complete the following steps on the affected VMs to recover from this state and to complete a recovery operation:

  1. Connect to the destination ESXi UI host and create a new standard virtual switch. At the beginning, there are no port groups or uplinks for the switch.

    image-20180801160152-2
     

  2. Use SSH to connect to the ESXi server. Identify and select the physical NIC and the port group of the existing distributed virtual switch SDDC-Dswitch-Private. This sample used the vnic vmnic0, which is part of the port ID 64. You can list the Distributed Virtual Switch (DVS) information by issuing this command:
    #esxcli network vswitch dvs vmware list

    image-20180801160247-3
     

  3. Based on the previous information, remove the NIC and port ID (port group) from the SDDC-Dswitch-Private DVS. Use the port ID from step 2.
    #esxcfg-vswitch -Q <physical_vnic> -V <port_group> SDDC-Dswitch-Private

    image-20180802122351-1
     

  4. Add the NIC and port group into the standard switch created in step 1:
    #esxcli network vswitch standard uplink add –-uplink-name=<physical_vnic> --vswitch-name=<standard_vswitch>

    image-20180802122429-2
     

  5. In the ESXi UI, add a port group and select the standard virtual switch:

    image-20180803093847-1

    At this point, the virtual switch should have one uplink and one port group.
     

  6. Run a restore operation by using the ESX host if vCenter is down option in the IBM Spectrum Protect Plus UI (see the first screenshot).
     

  7. Click Options and make sure you choose the new networking switch that you created in step 1 under Networking.

    image-20180803094047-2
     

  8. Using the destination ESXI UI, power on the recovered VM.
     

  9. Once the VMs are reachable, log in into the vCenter UI and start the migration of the management VMs from the temporary port group created in step 5 to the original distributed port group, SDDC-DPortGroup-Mgmt.

    Start a migration from the Networking tab by selecting a datacenter and then selecting Migrate VMs to Another Network from the Actions menu. Select the source network (the temporary switch created in step 5 ) and the destination network (management switch).

    image-20180803094654-3

    image-20180803094718-4

  10. Once all VMs are migrated into the original port group, reincorporate the physical NIC and the port group into the original distributed virtual switch:

    1. Remove the network cards (known as vmnics) from a standard vSwitch that was re-assigned previously by using the following command:
      #esxcli network vswitch standard uplink remove --uplink-name=vmnic --vswitch-name=vSwitch

      For example: #esxcli network vswitch standard uplink remove --uplink-name=vmnic0 --vswitch-name=vered_recovery

    2. Add network cards to a vNetwork Distributed Switch (vDS) by using the following command:
      #esxcfg-vswitch -P vmnic -V unused_dvPort_ID dvSwitch # add a vDS uplink

      For example: #esxcfg-vswitch -P vmnic0 -V 64 SDDC-Dswitch-Private
       

  11. Delete the temporary port group and the standard vSwitch from the ESXi host UI.
     

  12. After the VMs are migrated and accessible, use the ESXI host UI to unregister (do not delete) the old VMs (if the original host is reachable). This avoids having duplicated information such as names, Media Access Control (MAC) addresses, OS level IDs, and VM Universal Unique Identifiers (UUIDs). This step must to be performed even if you are using a new datastore.

    Note: In some vSphere or ESXi versions, the unregister operation can be performed by using the Remove from inventory option. This unregisters a VM from the vCenter catalog but leaves vmdk files on the datastore consuming storage space. Once you have fully recovered the VM and the environment is successfully running, you can regain the space by manually removing these files from the datastore.

    image-20180803095025-5
     

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSNQFQ","label":"IBM Spectrum Protect Plus"},"Component":"Not Applicable","Platform":[{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"10.1.1patch1 and later versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
07 August 2018

UID

ibm10719679

Page Feedback