MustGather: Master Document for IBM i Storage

Resolving The Problem

• Native Fibre Channel (Direct Attach)

  IBM i connected directly to external storage via a natively owned fibre channel adapter such as the IBM Spectrum Virtualize Family (SVC/Storwize - 2145), DS8000 (2107), IBM Flash System (D840), and EMC (D910).

  • End to End Diagram

    

    • Must Gather Instructions

      1)  System Snapshot (SYSSNAP): https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap

      For the SYSSNAP be sure to include LIC LOGs, QHST, and PALs. Change the date range to include the date range of the problem.  The PEDBG can be included as part of the SYSSNAP or separately in item 2 below.  

      2) HMC (for HMC-managed systems only): pedbg
              pedbg -c -q 4 
              https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc

      3) External Storage:
              Storwize and Flash Systems: Support package > Option 4 - Standard logs plus new statesaves
              https://www.ibm.com/support/pages/what-data-should-you-collect-problem-spectrum-virtualize-systems
              https://www.ibm.com/support/pages/data-collection-recommendations-v9000    
              DS8K: 
              state save, PE package, and/or ODD

• Native Fibre Channel (SAN Switch)

  IBM i connected via a natively owned fiber channel adapter through a SAN switch. The switch is typically Brocade or Cisco. The storage can include the IBM Spectrum Virtualize Family (SVC/Storwize - 2145), DS8000 (2107), IBM Flash System (D840), and EMC (D910).

  • End to End Diagram

    

    • MustGather Instructions

      1) System Snapshot (SYSSNAP): https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap

      For the SYSSNAP be sure to include LIC LOGs, QHST, and PALs. Change the date range to include the date range of the problem.  The PEDBG can be included as part of the SYSSNAP or separately in item 2 below.  

      2) HMC (for HMC-managed systems only): pedbg
              pedbg -c -q 4 
              https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc

      3) Switch logs:
              Brocade: Supportsave
              https://www.ibm.com/support/pages/how-collect-supportsave-brocade-switch
              Cisco Switch: Show Tech and Logging onboard: https://www.ibm.com/support/pages/how-collect-logs-cisco-mds-9000-series-switches
      
              
       4) External Storage:
              Storwize and Flash Systems: Support package > Option 4 - Standard logs plus new statesaves
              https://www.ibm.com/support/pages/what-data-should-you-collect-problem-spectrum-virtualize-systems
              https://www.ibm.com/support/pages/data-collection-recommendations-v9000    
              DS8K: 
              state save, PE package, and/or ODD

• Virtual Fibre Channel (NPIV)

  IBM i connected via NPIV / Virtual Fiber Channel to external storage. NPIV/VFC requires VIOS and a switch, so there will always be both in this environment. Typically there will be two or more VIOS partitions and two or more switches. The switch is typically Brocade or Cisco. The storage can include the IBM Spectrum Virtualize Family (SVC/Storwize - 2145), DS8000 (2107), IBM Flash System (D840), and EMC (D910).

  • End to End Diagram

    

    • MustGather Instructions

      1) System Snapshot (SYSSNAP): https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap

      For the SYSSNAP be sure to include LIC LOGs, QHST, and PALs. Change the date range to include the date range of the problem.  The PEDBG and VIOS Snaps can be included as part of the SYSSNAP or separately in items 2 and 3 below.  

      2) VIOS snap from both VIO Servers: 
              http://www-01.ibm.com/support/docview.wss?uid=isg3T1013064

      3) HMC (for HMC-managed systems only): pedbg
              pedbg -c -q 4 
              https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc

      4) Switch logs:
              Brocade: Supportsave
              https://www.ibm.com/support/pages/how-collect-supportsave-brocade-switch
              Cisco Switch: Show Tech and Logging onboard 
              https://www.ibm.com/support/pages/how-collect-logs-cisco-mds-9000-series-switches
              
      5) External Storage:
              Storwize and Flash Systems: Support package > Option 4 - Standard logs plus new statesaves
              https://www.ibm.com/support/pages/what-data-should-you-collect-problem-spectrum-virtualize-systems
              https://www.ibm.com/support/pages/data-collection-recommendations-v9000    
              DS8K:  State save, PE package, and/or ODD

• VIOS VSCSI to External Storage

  IBM i connected via VIOS Virtual SCSI (VSCSI). The IOA on the IBM i will be type 290A and the device type will be 6B22. The storage backing the hdisks on the VIOS will be fiber channel connected external storage. The storage can include the IBM Spectrum Virtualize Family (SVC/Storwize), DS8000, IBM Flash System, IBM XIV, IBM A9000, and EMC.

  • End to End Diagram

    

    • MustGather Instructions

      1) System Snapshot (SYSSNAP): https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap

      For the SYSSNAP be sure to include LIC LOGs, QHST, and PALs. Change the date range to include the date range of the problem.  The PEDBG and VIOS Snaps can be included as part of the SYSSNAP or separately in items 2 and 3 below.  

      2) VIOS snap from both VIO Servers: 
              http://www-01.ibm.com/support/docview.wss?uid=isg3T1013064

      3) HMC (for HMC-managed systems only): pedbg
              pedbg -c -q 4 
              https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc

      4) Switch logs:
              Brocade: Supportsave
              https://www.ibm.com/support/pages/how-collect-supportsave-brocade-switch
              Cisco Switch: Show Tech and Logging onboard 
              https://www.ibm.com/support/pages/how-collect-logs-cisco-mds-9000-series-switches
              
      5) External Storage:
              Storwize and Flash Systems: Support package > Option 4 - Standard logs plus new statesaves
              https://www.ibm.com/support/pages/what-data-should-you-collect-problem-spectrum-virtualize-systems
              https://www.ibm.com/support/pages/data-collection-recommendations-v9000    
              DS8K:  State save, PE package, and/or ODD

• VIOS VSCSI to Internal Storage

  IBM i connected via VIOS Virtual SCSI (VSCSI). The IOA on the IBM i will be type 290A and the device type will be 6Bxx. The storage backing the hdisks on the VIOS will be a RAID set or file backed logical volumes. 

  • End to End Diagram

    

    • MustGather Instructions

      1) System Snapshot (SYSSNAP): https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap

      For the SYSSNAP be sure to include LIC LOGs, QHST, and PALs. Change the date range to include the date range of the problem.  The PEDBG and VIOS Snaps can be included as part of the SYSSNAP or separately in items 2 and 3 below.  

      2) VIOS snap from both VIO Servers: 
              http://www-01.ibm.com/support/docview.wss?uid=isg3T1013064

      3) HMC (for HMC-managed systems only): pedbg
              pedbg -c -q 4 
              https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc

• IBM i Internal Storage

  IBM i with it's own disk units. The IOA on the IBM i will be various types and may be set up with various types of protection (RAID or mirroring).   The device type will depend on the type of HDD or SSD disk being used.   

  • End to End Diagram

    

    • MustGather Instructions

      1) System Snapshot (SYSSNAP): https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap For the SYSSNAP be sure to include LIC LOGs, QHST, and PALs.  Change the date range to include the date range of the problem.  The PEDBG and can be included as part of the SYSSNAP or separately in item 2 below.  

      2) HMC (for HMC-managed systems only): pedbg
              pedbg -c -q 4 
              https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc

• i Hosting i to Internal Storage

  IBM i Guest Partition with it's virtual disks hosted by an IBM i Hosting partition.   The IOA on the IBM i Guest will be type 290A and the device type will be 6Bxx. The IOA and disks on the IBM i Host can be any supported IOA and Internal HDD/SSDs.   

  • End to End Diagram

    

    • MustGather Instructions

      1) System Snapshot (SYSSNAP): https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap from both the Guest and Host partitions.  

      For the SYSSNAP be sure to include LIC LOGs, QHST, and PALs.  Change the date range to include the date range of the problem.  The PEDBG can be included as part of the SYSSNAP or separately in item 2  below.   The PEDBG only needs to be collected one time as the Guest and Host partitions will be on the same physical system.  

      2) HMC (for HMC-managed systems only): pedbg
              pedbg -c -q 4 
              https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc

• i Hosting i to External Storage

  IBM i Guest Partition with it's virtual disks hosted by an IBM i Hosting partition.   The IOA on the IBM i Guest will be type 290A and the device type will be 6Bxx. The IOA and disks on the IBM i Host can be any supported fibre channel IOA and External Storage.   The storage can include the IBM Spectrum Virtualize Family (SVC/Storwize), DS8000, IBM Flash System, and EMC.   The End to End diagram shows native fibre channel with a SAN switch, but this could be any supported IBM i external storage configuration.  

  • End to End Diagram

    

    • MustGather Instructions

      1) System Snapshot (SYSSNAP): https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap from both the Guest and Host partitions.  

      For the SYSSNAP be sure to include LIC LOGs, QHST, and PALs.  Change the date range to include the date range of the problem.  The PEDBG can be included as part of the SYSSNAP or separately in item 2 below.   The PEDBG only needs to be collected one time as the Guest and Host partitions will be on the same physical system. 

      2) HMC (for HMC-managed systems only): pedbg
              pedbg -c -q 4 
              https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc

      3) Switch logs:
              Brocade: Supportsave
              https://www.ibm.com/support/pages/how-collect-supportsave-brocade-switch
              Cisco Switch: Show Tech and Logging onboard 
              https://www.ibm.com/support/pages/how-collect-logs-cisco-mds-9000-series-switches
              
      4) External Storage:
              Storwize and Flash Systems: Support package > Option 4 - Standard logs plus new statesaves
              https://www.ibm.com/support/pages/what-data-should-you-collect-problem-spectrum-virtualize-systems
              https://www.ibm.com/support/pages/data-collection-recommendations-v9000    
              DS8K:  State save, PE package, and/or ODD

• NVMe Information and Documentation

  The following links provide NVMe Concepts, information and documentation:

  NMVe Concepts: https://www.ibm.com/docs/en/ssw_ibm_i_75/rzaly/rzalynvmeconcepts.htm

  Creating NVMe namespaces and adding them to an ASP - This document describes how to create your NVMe namespaces and add them to an ASP.

  Managing NVMe - This topic describes how to manage your NVMe devices and namespaces.

  Display NVMe namespaces - Select this function to show all NVMe devices and namespaces that exist on the IBM® i partition.

  Display NVMe devices - Select this function to show all NVMe devices that exist on the IBM i partition.

  Create NVMe namespaces - Select this function to allocate storage and create namespaces on a NVMe device.

  Create NVMe namespaces to match another NVMe device - Select this function to create namespaces on a NVMe device that match the capacity of namespaces on another NVMe device.

  Create NVMe namespaces to pair with Active mirror protected NVMe units - Select this function to create namespaces on a NVMe device to pair with active mirror protected NVMe units.

  Delete existing NVMe namespaces - Select this function to destroy the IBM i disk unit associated with the NVMe namespace and return the namespace storage to the available pool in the NVMe device.

  Sanitize/Erase NVMe device - Select this function to remove and destroy all data from the NVMe media and the cache.

  Format NVMe device to prepare device for IBM i - Select this function to change the block size of the NVMe device to a format that can be used on the IBM i partition.

  Display NVMe health - Select this function to show the NVMe device health data associated with the NVMe devices that exist on the IBM i partition.

  NVMe password protection - Select this function to use the features of the NVMe password protection policy.

  Other NVMe concepts:

  Disk management checklist - This topic contains checklists for performing configuration procedures. Use this information to determine which checklist to use for your situation.

  Configuring disks on a new system - This checklist shows the sequence of tasks that you use to configure disks on a new system. Whether you need to perform all the tasks depends on the disk protection that you want on your system.

  Adding disk units without disk protection -This checklist shows the sequence of tasks that you use to configure disks on a new system. Whether you need to perform all the tasks depends on the disk protection that you want on your system.

  Adding disk units to an existing IOA - This checklist shows the sequence of tasks that you use to add one or more disks to an existing IOA that has built-in device parity protection.

  Adding a new IOA - This checklist shows the sequence of tasks that you use to add a new IOA and new disk units to your system.

  Deleting a disk pool - This checklist shows the sequence of tasks that you use to delete a basic disk pool or an independent disk pool.

  Removing disk units without disk protection - This checklist shows the sequence of tasks that you use to remove one or more disk units from your system when the disk units do not have device parity protection.

  Removing disk units that have device parity protection from a disk pool without mirrored protection - This checklist shows the sequence of tasks that you use to remove one or more disk units from an IOA with built-in device parity protection.

  Removing disk units that have device parity protection from a disk pool with mirrored protection - This checklist shows the sequence of tasks that you use to remove one or more disk units from an IOA that is capable of device parity protection.

  Upgrading the load source disk unit with device parity protection - This checklist shows the sequence of tasks to upgrade your load source disk unit (unit 1) with a disk unit that has at least a 70-GB capacity while keeping device parity active. The disk units that are replaced in this procedure are discarded.

  Upgrading the load source disk unit with local mirroring using non-configured disk units - This checklist shows the sequence of tasks to upgrade your load source disk unit (unit 1) and the mirrored load source with disk units that have equal capacity, are at least 70-GB, and are mirror protected. The disk units that are replaced in this procedure are discarded.

  Upgrading the load source disk unit with local mirroring using newly installed disk units - This checklist shows the sequence of tasks to upgrade your load source disk unit (unit 1) and the mirrored load source with disk units that have equal capacity, are at least 70-GB, and are mirror protected. The disk units that are replaced in this procedure are discarded.

  Upgrading the load source disk unit without disk protection -This checklist shows the sequence of tasks to upgrade your load source disk unit (unit 1) with a disk unit that has at least a 70-GB capacity. The disk units that are replaced in this procedure are discarded.

  Installing a new partition using a NVMe as the load source disk unit - To use a namespace on a NVMe device as a load source disk unit, follow this procedure.

  Removing an operational NVMe device from a partition - To remove an operational NVMe device from an IBM® i partition, follow this procedure.

  Upgrading the system ASP to NVMe - mirror protected - To upgrade the system ASP which contains all SAS or SAN LUNs to use all NVMe namespaces, where the current system ASP has mirror protection, follow this procedure.

  Upgrading the system ASP to NVMe - not mirror protected - To upgrade the system ASP which contains all SAS or SAN LUNs to use all NVMe namespaces, where the current system ASP is not mirror protected, follow this procedure.

  Upgrading a user ASP to NVMe - To upgrade a user ASP which contains all SAS or SAN LUNs to use all NVMe namespaces, follow this procedure.

  Secure Data Erase for the IBM i - Use this procedure to complete the secure data deletion for the SAS attached disk drives, SAS attached solid-state drives, and PCIe attached NVMe devices connected to an IBM i partition.

  Must Gather Instructions

  1)  QMGTOOLS - Install IBM i Support MustGather QMGTOOLS (of if already installed, unsure it is at the most current level) see the following document for install and update instructions: https://www.ibm.com/support/pages/node/645929

  2) Capture a System Snapshot (SYSSNAP) and upload to ecurep from these instructions: https://www.ibm.com/support/pages/qmgtools-system-snapshot-syssnap Note: Include LIC LOGs, QHST, and PALs.  Change the date range to include the date range of the issue. 

  3) If needed, from QMGTOOLS, use the Auxiliary Storage Summary report to gather storage related information (disk, ASP, temporary storage, balancing, etc.) all in one place. Refer to these instructions to run QMGTOOLS/DSPASPSUM: https://www.ibm.com/support/pages/qmgtools-auxiliary-storage-summary It will provide the NVMe summary amongst other auxiliary storage information.