IBM Support

VH950_119 / FW950.70 Release Notes

Fix Readme


Abstract

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.

This package provides firmware for Power Systems E980 (9080-M9S) servers only.

Release notes for VH950_119 / FW950.70.

Read the following important information prior to installing this Service Pack.

Service Pack Summary: Concurrent Service Pack.

This service pack addresses a HIPER (High Impact/Pervasive) issues. Please see the Description file for details.

Concurrent updates of certain SR-IOV adapters to the firmware level provided by this service pack require an AIX/VIOS fix to prevent VF failures for these partitions.
For additional details, see Section 2.0 of the description file.

The Minimum HMC level supports the following HMC models: x86 - KVM, XEN, VMWare ESXi (6.0/6.5) ppc64le - 7063-CR1,vHMC on PowerVM (POWER8 and POWER9) systems The 7042-CR9 is the ONLY Machine Type HMC appliances for x86 supported for the Minimum HMC level. See the Description file for details.

NovaLink levels earlier than "NovaLink 1.0.0.16 Feb 2020 release" with partitions running certain SR-IOV capable adapters is NOT supported at FW930 and FW940. Upgrading systems in this configuration is supported only if NovaLink is first updated to "NovaLink 1.0.0.16 Feb 2020 release" or later. For additional details, see Section 2.0 of the description file.

Update all adapters which are boot adapters, or which may be used as boot adapters in the future, to the latest microcode from IBM Fix Central. The latest microcode will ensure the adapters support the new Firmware Secure Boot feature of Power Systems. This requirement applies when updating system firmware from a level prior to FW940 to levels FW940 and later. For additional details, see Section 2.0 of the description file.

Content

Minimum HMC Code Level

This section is intended to describe the "Minimum HMC Code Level" required by the System Firmware to complete the firmware installation process. When installing the System Firmware, the HMC level must be equal to or higher than the "Minimum HMC Code Level" before starting the system firmware update.  If the HMC managing the server targeted for the System Firmware update is running a code level lower than the "Minimum HMC Code Level" the firmware update will not proceed.

The Minimum HMC Code levels for this firmware for HMC x86,  ppc64 or ppc64le are listed below.

x86 -  This term is used to reference the legacy HMC that runs on x86/Intel/AMD hardware for the Virtual HMC that can run on the Intel hypervisors (KVM, XEN, VMWare ESXi).

  • The Minimum HMC Code level for this firmware is:  HMC V9R2M950 (PTF MH01869).
    Note:     The 7042-CR9 is the ONLY Machine Type HMC appliances for x86 supported for the Minimum HMC level.
  • Although the Minimum HMC Code level for this firmware is listed above, V9R2, HMC V9R2M951.2 (PTF MH01892) or higher is recommended to avoid an issue that can cause the HMC to lose connections to all servers for a brief time with service events E2FF1409 and E23D040A being reported. This will cause all running server tasks such as server firmware upgrade to fail.
ppc64 or ppc64le - describes the Linux code that is compiled to run on Power-based servers or LPARS (Logical Partitions)
  • The Minimum HMC Code level for this firmware is:  HMC V9R2M950 (PTF MH01870).
  • Although the Minimum HMC Code level for this firmware is listed above, V9R2, HMC V9R2M951.2 (PTF MH01893) or higher is recommended to avoid an issue that can cause the HMC to lose connections to all servers for a brief time with service events E2FF1409 and E23D040A being reported. This will cause all running server tasks such as server firmware upgrade to fail.

The Minimum HMC level supports the following HMC models:
x86 - KVM, XEN, VMWare ESXi (6.0/6.5)
ppc64le - 7063-CR1,vHMC on PowerVM (POWER8 and POWER9 systems

For information concerning HMC releases and the latest PTFs,  go to the following URL to access Fix Central:
http://www-933.ibm.com/support/fixcentral/

For specific fix level information on key components of IBM Power Systems running the AIX, IBM i and Linux operating systems, we suggest using the Fix Level Recommendation Tool (FLRT):
http://www14.software.ibm.com/webapp/set2/flrt/home

NOTES:
                -You must be logged in as hscroot in order for the firmware installation to complete correctly.
                - Systems Director Management Console (SDMC) does not support this System Firmware level.

Important Information

Concurrent firmware update of certain SR-IOV adapters needs AIX/VIOS fix
If the  adapter firmware level in this service pack  is concurrently applied, AIX and VIOS VFs may become failed. To prevent the VF failure, the VIOS and AIX partitions must have the fix for IJ44288 (or a sibling APAR) applied prior to concurrently updating SR-IOV adapter firmware. AIX/VIOS SPs Spring 2023 will ship this fix.  Until then, interim fixes (ifixes) are available from https://aix.software.ibm.com/aix/efixes/ij44288/ or by calling IBM support if an ifix is required for a different level. A re-IPL of the system instead of concurrently updating the SR-IOV adapter firmware would also work to prevent a VF failure.   The following SR-IOV adapter Feature Codes and CCINs are affected : #EC2R/EC2S with CCIN 58FA; #EC2T/EC2U with CCIN 58FB; and #EC66/EC67 with CCIN 2CF3.

NovaLink levels earlier than "NovaLink 1.0.0.16 Feb 2020 release" with partitions running certain SR-IOV capable adapters is NOT supported at this firmware release
NovaLink levels earlier than "NovaLink 1.0.0.16 Feb 2020 release" do not support IO adapter FCs EC2R/EC2S, EC2T/EC2U, EC3L/EC3M, EC66/EC67 with FW930 and later.   If the adapter was already in use with FW910/920 at an older NovaLink level, upgrading to FW930/940 will result in errors in NovaLink and PowerVC which causes the loss of any management operation via NovaLink / PowerVC combination.  Upgrading systems in this configuration is not supported at the older NovaLink levels.  If the system is required to be at FW930/940 or was shipped with FW930/940, NovaLink must first be updated to "NovaLink 1.0.0.16 Feb 2020 release"  or later.

Boot adapter microcode requirement
Update all adapters which are boot adapters, or which may be used as boot adapters in the future, to the latest microcode from IBM Fix Central.  The latest microcode will ensure the adapters support the new Firmware Secure Boot feature of Power Systems. This requirement applies when updating system firmware from a level prior to FW940 to levels FW940 and later.
The latest adapter microcode levels include signed boot driver code. If a boot-capable PCI adapter is not installed with the latest level of adapter microcode, the partition which owns the adapter will boot, but error logs with SRCs BA5400A5 or BA5400A6 will be posted.  Once the adapter(s) are updated, the error logs will no longer be posted.

Downgrading firmware from any given release level to an earlier release level is not recommended
Firmware downgrade warnings:

  1. Adapter feature codes (#EC2R/#EC2S/#EC2T/#EC2U and #EC3L/#EC3M and #EC66/EC67) when configured in SR-IOV shared mode in FW930 or later, even if originally configured in shared mode in a pre-FW930 release, may not function properly if the system is downgraded to a pre-FW930 release. The adapter should be configured in dedicated mode first (i.e. take the adapter out of SR-IOV shared mode) before downgrading to a pre-FW930 release.
  2. If partitions have been run in POWER9 compatibility mode in FW940, a downgrade to an earlier release (pre-FW940) may cause a problem with the partitions starting.  To prevent this problem, the "server firmware" settings must be reset by rebooting partitions in "Power9_base" before doing the downgrade.

If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.

IPv6 Support and Limitations

IPv6 (Internet Protocol version 6) is supported in the System Management Services (SMS) in this level of system firmware. There are several limitations that should be considered.

When configuring a network interface card (NIC) for remote IPL, only the most recently configured protocol (IPv4 or IPv6) is retained. For example, if the network interface card was previously configured with IPv4 information and is now being configured with IPv6 information, the IPv4 configuration information is discarded.

A single network interface card may only be chosen once for the boot device list. In other words, the interface cannot be configured for the IPv6 protocol and for the IPv4 protocol at the same time.

Concurrent Firmware Updates
Concurrent system firmware update is supported on HMC Managed Systems only.

Ensure that there are no RMC connections issues for any system partitions prior to applying the firmware update.  If there is a RMC connection failure to a partition during the firmware update, the RMC connection will need to be restored and additional recovery actions for that partition will be required to complete partition firmware update

Memory Considerations for Firmware Upgrades

Firmware Release Level upgrades and Service Pack updates may consume additional system memory.
Server firmware requires memory to support the logical partitions on the server. The amount of memory required by the server firmware varies according to several factors.
Factors influencing server firmware memory requirements include the following:

  •     Number of logical partitions
  •     Partition environments of the logical partitions
  •     Number of physical and virtual I/O devices used by the logical partitions
  •     Maximum memory values given to the logical partitions

Generally, you can estimate the amount of memory required by server firmware to be approximately 8% of the system installed memory. The actual amount required will generally be less than 8%. However, there are some server models that require an absolute minimum amount of memory for server firmware, regardless of the previously mentioned considerations.

Additional information can be found at:
https://www.ibm.com/support/knowledgecenter/9080-M9S/p9hat/p9hat_lparmemory.htm

SBE Updates

Power 9 servers contain SBEs (Self Boot Engines) and are used to boot the system.  SBE is internal to each of the Power 9 chips and used to "self boot" the chip.  The SBE image is persistent and is only reloaded if there is a system firmware update that contains a SBE change.  If there is a SBE change and system firmware update is concurrent, then the SBE update is delayed to the next IPL of the CEC which will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  If there is a SBE change and the system firmware update is disruptive, then SBE update will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  During the SBE update process, the HMC or op-panel will display service processor code C1C3C213 for each of the SBEs being updated.  This is a normal progress code and system boot should be not be terminated by the user. Additional time estimate can be between 12-20 minutes per drawer or up to 48-80 minutes for maximum configuration.

The SBE image is only updated with this service pack if the starting firmware level is less than FW950.10.

Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For systems that are not managed by an HMC, the installation of system firmware is always disruptive.

Note: The concurrent levels of system firmware may, on occasion, contain fixes that are known as Deferred and/or Partition-Deferred. Deferred fixes can be installed concurrently, but will not be activated until the next IPL. Partition-Deferred fixes can be installed concurrently, but will not be activated until a partition reactivate is performed. Deferred and/or Partition-Deferred fixes, if any, will be identified in the "Firmware Update Descriptions" table of this document. For these types of fixes (Deferred and/or Partition-Deferred) within a service pack, only the fixes in the service pack which cannot be concurrently activated are deferred.

Note: The file names and service pack levels used in the following examples are for clarification only, and are not necessarily levels that have been, or will be released.

System firmware file naming convention:

01VHxxx_yyy_zzz

  • xxx is the release level
  • yyy is the service pack level
  • zzz is the last disruptive service pack level
NOTE: Values of service pack and last disruptive service pack level (yyy and zzz) are only unique within a release level (xxx). For example, 01VH900_040_040 and 01VH910_040_045 are different service packs.

An installation is disruptive if:

  • The release levels (xxx) are different.     
            Example: Currently installed release is 01VH900_040_040, new release is 01VH910_050_050.
 
  • The service pack level (yyy) and the last disruptive service pack level (zzz) are the same.     
            Example: VH910_040_040 is disruptive, no matter what level of VH910 is currently installed on the system.
 
  • The service pack level (yyy) currently installed on the system is lower than the last disruptive service pack level (zzz) of the service pack to be installed.
            Example: Currently installed service pack is VH910_040_040 and new service pack is VH910_050_045.
 

An installation is concurrent if:

The release level (xxx) is the same, and
The service pack level (yyy) currently installed on the system is the same or higher than the last disruptive service pack level (zzz) of the service pack to be installed.

Example: Currently installed service pack is VH910_040_040, new service pack is VH910_041_040.
 

Firmware Information and Description

Filename Size Checksum md5sum
01VH950_119_045.rpm 176902125 
6DCB257C
 
594693e48aca95e084bc0442978c1d31

Note: The Checksum can be found by running the AIX sum command against the rpm file (only the first 5 digits are listed).
ie: sum 01VH950_119_045.rpm
 
VH950
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
https://www.ibm.com/support/pages/node/6555136
 
VH950_119_045 / FW950.70

02/15/23		 Impact: Data   Severity:  HIPER

System firmware changes that affect all systems
  • HIPER/Pervasive:  If a partition running in Power9 compatibility mode encounters memory errors and a Live Partition Mobility (LPM) operation is subsequently initiated for that partition, undetected data corruption within GZIP operations (via hardware acceleration) may occur within that specific partition.
  • HIPER/Pervasive:  If a partition running in Power9 mode encounters an uncorrectable memory error during a Dynamic Platform Optimization (DPO), memory guard, or memory mirroring defragmentation operation, undetected data corruption may occur in any partition(s) within the system or the system may terminate with SRC B700F105.
  • A problem was fixed for performance slowdowns that can occur during the Live Partition Mobility (LPM) migration of a partition in POWER9, POWER10, or default processor compatibility modes. For this to happen to a partition in default processor compatibility mode, it must have booted on a Power10 system.  If this problem occurs, the performance will return to normal after the partition migration completes.  As a workaround, the partition to be migrated can be put into POWER9_base processor compatibility mode or older.
  • A problem was fixed for a processor core not being called out and guarded if a recoverable core error recovery fails and triggers a system checkstop.  This happens only if core error recovery fails with a core unit checkstop.
  • A problem was fixed for an errant concurrent firmware update that results in a deconfigured FSP.  This is a rare error that can occur if an FSP runs out of memory during the code update while a firmware file is being updated on it.  If this problem occurs, contact your next level of support.
  • For a system with I/O Enlarged Capacity enabled, and greater than 8 TB of memory, and having an adapter in SR-IOV shared mode, a problem was fixed for partition or system termination for a failed memory page relocation.  This can occur if the SR-IOV adapter is assigned to a VIOS and virtualized to a client partition and then does an I/O DMA on a section of memory greater than 2 GB in size.  This problem can be avoided by not enabling "I/O Enlarged Capacity".
  • A problem was fixed for an SR-IOV adapter showing up as "n/a" on the HMC's Hardware Virtualized I/O menu.  This is an infrequent error that can occur if an I/O drawer is moved to a different parent slot.  As a workaround, the PowerVM Hypervisor NVRAM can be cleared or the I/O drawer can be moved back to the original parent slot to clean up the configuration.
  • A problem was fixed for too frequent callouts for repair action for recoverable errors for Predictive Error (PE) SRCs B7006A72, B7006A74, and B7006A75.  These SRCs for PCIe correctable error events called for a repair action but the threshold for the events was too low for a recoverable error that does not impact the system.  The threshold for triggering the PE SRCs has been increased for all PLX and non-PLX switch correctable errors.
  • A problem was fixed for not being able to reduce partition memory when the PowerVM hypervisor has insufficient memory for normal operations.  With the fix, a partition configuration change to reduce memory is allowed when the hypervisor has insufficient memory.  A possible workaround for this error is to free up system memory by deleting a partition.
  • A problem was fixed for an incorrect capacity displayed for a Fibre Channel device using SMS option "I/O Device Information".  This happens every time for a device that has a capacity greater than 2 TB.  For this case, the capacity value displayed may be significantly less than 2 TB.   For example, a 2 TB device would be shown as having a capacity of 485 GB.
  • A problem was fixed for a partition firmware data storage error with SRC BA210003 logged or for a failure to locate NVMe target namespaces when attempting to access NVMe devices over Fibre Channel (FC-NVME) SANs connected to third-party vendor storage systems.  This error condition, if it occurs, prevents firmware from accessing NVMe namespaces over FC as described in the following scenarios:
     1) Boot attempts from an NVMe namespace over FC using the current SMS bootlist could fail.
     2) From SMS menus via option 3 - I/O Device Information - no devices can be found when attempting to view NVMe over FC devices.
     3) From SMS menus via option 5 - Select Boot Options - no bootable devices can be found when attempting to view and select an NVMe over FC bootable device for the purpose of boot, viewing the current device order, or modifying the boot device order.
    The trigger for the problem is the attempted access of NVMe namespaces over Fibre Channel SANs connected to storage systems via one of the scenarios listed above.  The frequency of this problem can be high for some of the vendor storage systems.
  • A problem was fixed for a failed NIM download/install of OS images that are greater than 32M.  This only happens when using the default TFTP block size of 512 bytes.  The latest versions of AIX are greater than 32M in size and can have this problem.  As a workaround, in the SMS menu, change "TFTP blocksize" from 512 to 1024. To do this, go to the SMS "Advanced Setup: BOOTP" menu option when setting up NIM install parameters.  This will allow a NIM download of an image up to 64M.
  • A problem was fixed for a security scan with NSFOCUS reporting the following low-priority vulnerabilities:
    1. Low. Web server enabled "options"
    2. Low. Response no "Referrer-Policy" header
    3. Low. Response no "X-Permitted-Cross-Domain-Policies" header
    4. Low. Response no "X-Download-Options" header
    5. Low. Response no "Content-Security-Policy" header
    There is no impact to the system from these as the FSP service processor does not provide any features which can be exploited by the five vulnerabilities.
  • A problem was fixed for a security scan with NSFOCUS reporting a medium-level vulnerability for a slow HTTPS request denial of service attack against ASMI.  This occurs whenever NSFOCUS scans are run.
  • Support for using a Redfish (REST) API to gather power usage for all nodes in watts and the ambient temperature for the system.
    Redfish sample response is as shown below:
    ==>> GET redfish/v1/Systems/<>
    ...
        "Oem": {
            "IBMEnterpriseComputerSystem": {
                ...
                ...
                "PowerInputWatts" : <> ( number in watts),  <<<<============
                "AmbientTemp" : <> (number in Celsius) <<<<============
            }
        },
    ...
System firmware changes that affect all systems
  • For a system with an IBM i partition, a problem was fixed for the IBMi  60-day "Trial 5250" function not working. The "Trial 5250" is only needed for the case of an incomplete system order that results in the IBM i 100% 5250 feature being missing.  Since the "Trial 5250" is temporary anyway and valid for only 60 days, an order for the permanent 5250 feature is needed to fully resolve the problem.

How to Determine The Currently Installed Firmware Level

You can view the server's current firmware level on the Advanced System Management Interface (ASMI) Welcome pane. It appears in the top right corner. Example: VH920_123.

Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

Note: If your HMC is not internet-connected you will need to download the new firmware level to a USB flash memory device or ftp server.

Installing the Firmware

The method used to install new firmware will depend on the release level of firmware which is currently installed on your server. The release level can be determined by the prefix of the new firmware's filename.

Example: VHxxx_yyy_zzz

Where xxx = release level

  • If the release level will stay the same (Example: Level VH920_040_040 is currently installed and you are attempting to install level VH920_041_040) this is considered an update.
  • If the release level will change (Example: Level VH900_040_040 is currently installed and you are attempting to install level VH920_050_050) this is considered an upgrade.
Instructions for installing firmware updates and upgrades can be found at https://www.ibm.com/support/knowledgecenter/9080-M9S/p9eh6/p9eh6_updates_sys.htm

IBM i Systems:
For information concerning IBM i Systems, go to the following URL to access Fix Central: 
http://www-933.ibm.com/support/fixcentral/

Choose "Select product", under Product Group specify "System i", under Product specify "IBM i", then Continue and specify the desired firmware PTF accordingly.

HMC and NovaLink Co-Managed Systems (Disruptive firmware updates only):
A co-managed system is managed by HMC and NovaLink, with one of the interfaces in the co-management master mode.
Instructions for installing firmware updates and upgrades on systems co-managed by an HMC and Novalink is the same as above for a HMC managed systems since the firmware update must be done by the HMC in the co-management master mode.  Before the firmware update is attempted, one must be sure that HMC is set in the master mode using the steps at the following IBM KnowledgeCenter link for NovaLink co-managed systems:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eig/p9eig_kickoff.htm

Then the firmware updates can proceed with the same steps as for the HMC managed systems except the system must be powered off because only a disruptive update is allowed.   If a concurrent update is attempted, the following error will occur: " HSCF0180E Operation failed for <system name> (<system mtms>).  The operation failed.  E302F861 is the error code:"
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eh6/p9eh6_updates_sys.htm

Firmware History

The complete Firmware Fix History (including HIPER descriptions)  for this Release level can be reviewed at the following url:
https://www.ibm.com/support/pages/node/6955585

[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"TI0007I","label":"Power System E980 Server (9080-M9S)"},"ARM Category":[{"code":"a8m0z000000bpKLAAY","label":"Firmware"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
23 June 2023

UID

ibm16955853

Page Feedback