Troubleshooting
Problem
VIOS login attempts or padmin commands fail with fork function errors, i.e.
$ oem_setup_env rksh: 0403-031 The fork function failed. There is not enough memory available
Symptom
Error may be encountered when trying to login to the VIOS, when trying to run padmin commands, or it may be seen from the VIOS console.
Cause
VIOS is running out of memory.
Environment
PowerVM VIOS 3.1 and 4.1
Diagnosing The Problem
Refer to the Virtual I/O Server release notes for your VIOS release and ensure the minimum memory requirement is met.
Memory requirements
The minimum memory requirement for the VIOS Version 4.1.0.10 varies based on the configuration.
The minimum memory requirement for the VIOS Version 4.1.0.10 is 4 GB. A minimum current memory requirement might support a configuration with a minimum number of devices or a small maximum memory configuration. However, to support shared storage pools, the minimum memory requirement is 4 GB.
The VIOS Version 4.1.0.10 requires the minimum current memory requirement to increase when either the maximum memory configuration or the number of devices increases, or when they both increase. Larger maximum memory configurations or more devices increase the minimum current memory requirement. If the minimum memory requirements do not increase along with the maximum memory configuration, the partition stops during the initial program load (IPL).
Memory requirements
The minimum memory requirement for the VIOS Version 3.1 varies based on the configuration.
A general rule for the minimum memory requirement for the VIOS Version 3.1 is 4 GB. A minimum current memory requirement might support a configuration with a minimum number of devices or a small maximum memory configuration. However, to support shared storage pools, the minimum memory requirement is 4 GB.
The VIOS Version 3.1 requires the minimum current memory requirement to increase when either the maximum memory configuration or the number of devices increases, or when they both increase. Larger maximum memory configurations or more devices increase the minimum current memory requirement. If the minimum memory requirement isn’t increased along with the maximum memory configuration, the partition stops during the initial program load (IPL).
Resolving The Problem
IMPORTANT
The above memory requirements is for a bear minimum VIOS installation.
Once the PowerVM virtual environment is built and client partitions have been created, it is best practice to evaluate the VIOS resources by using the Virtual I/O Server Performance Advisor. The Virtual I/O Server Performance Advisor report can then be reviewed for recommendations.
A Recommended Value may be generated based on the VIOS workload going on at the time the performance data was collected. Therefore, it is best to capture the data when VIOS workload is at its peak. For how-to details, refer to MUSTGATHER data to diagnose Performance Issues on PowerVM VIOS.
If a Recommended Value is generated, increase the memory to such value, then try running the VIOS command again.
If a Recommended Value is generated, increase the memory to such value, then try running the VIOS command again.
For an HMC-managed system, you can increase the memory on the VIOS partition using Dynamic Logical Partitioning (DLPAR). Then try the operation again. If this resolves the problem, save the DLPAR changes to the VIOS profile.
Note 1:
DLPAR operations require network connectivity and an active RMC connection on the VIOS. If the VIOS partition does not have network access or an active RMC connection, increase the memory on the VIOS partition's profile; then, reactive (Shutdown; Activate) the VIOS partition for the profile change to take effect.
DLPAR operations require network connectivity and an active RMC connection on the VIOS. If the VIOS partition does not have network access or an active RMC connection, increase the memory on the VIOS partition's profile; then, reactive (Shutdown; Activate) the VIOS partition for the profile change to take effect.
Note 2:
For single VIOS environments where DLPAR operation is not possible (due to lack of network or inactive RMC connection), a maintenance window will be required to shutdown all the clients prior to reactivating the VIOS partition.
For dual VIOS environments, ensure the clients' storage and network are fully redundant through a second VIOS prior reactivating the VIOS in question.
For single VIOS environments where DLPAR operation is not possible (due to lack of network or inactive RMC connection), a maintenance window will be required to shutdown all the clients prior to reactivating the VIOS partition.
For dual VIOS environments, ensure the clients' storage and network are fully redundant through a second VIOS prior reactivating the VIOS in question.
Note 3:
If the errors have caused the VIOS LPAR to become unresponsive, the errors can be cleared by rebooting the VIOS.
However, if Root Cause Analysis is needed, force an OS system dump on the VIOS LPAR while it remains in the problematic state for investigation. For how-to details, refer to MustGather Testcase for VIOS Crash or VIOS Hang Condition > VIOS Hang Condition
[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSPHKW","label":"PowerVM Virtual I\/O Server"},"ARM Category":[{"code":"a8m50000000L0MiAAK","label":"PowerVM VIOS-\u003EMemory"}],"ARM Case Number":"TS015030910","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
25 April 2025
UID
ibm10732948