The concurrency level attribute

The concurrency level attribute was introduced with Virtual I/O Server (VIOS) version 2.2.2.0 and is used to control the amount, and configuration, of resources that are allocated to a partition mobility operation by the mover service partition (MSP). The actual resources that are associated with a specific concurrency level value might change when new VIOS versions are released, but lower concurrency level values always equate to more resources allocated and in general, lower migration times.

From the VIOS versions 2.2.2.0 to 2.2.3.x, the concurrency level attributes controlled the amount of memory that is allocated for partition mobility operations. Starting with the version 2.2.4.0, the concurrency level also controls the number of threads that are used to send and receive the memory pages of the mobile partition. More threads require more processor and network bandwidth to be fully utilized, a strict limit on the number of partition mobility threads that are running is imposed to prevent the VIOS partition from being over loaded. This limit results in a lower number of allowed concurrent operations when concurrency level values less than 4 are used. It is recommended that the default value be used in most cases. The table provides use cases and recommendations for changing the concurrency level either for all migrations or for a specific partition mobility operation.

If you are using VIOS version 3.1.2.0, or later and HMC version 9.2.950, or later, it is not recommended to use the concurrency level attribute to control the amount of resources that are allocated to a partition mobility operation. The HMC determines the optimal amount of required resources based on the characteristics of the logical partition that is being migrated. If you change the concurrency_lvl attribute from its default value of 4, the values determined by the HMC is overridden. You must not change the concurrency_lvl attribute so that the HMC can determine the optimal resources to use, unless otherwise directed by IBM® Support. If you are using VIOS version 3.1.2.0, or later and HMC version 9.2.950, or later, and Power® hypervisor version 950, or later, automatic selection of the optimal resources might increase the maximum number of threads that are utilized by a single partition mobility operation from 4 threads to 8 threads. When a concurrent operation is run with 8 threads, only a single partition mobility operation is allowed to run at a time.
Table 1. Setting the concurrency level
VIOS Version Recommended usage
Concurrency level Usage
2.2.2.0 - 2.2.3.x 5 Recommended concurrency level if a previous partition mobility operation failed because of insufficient memory.
4 Not a recommended concurrency level.
3 The default value, and is the recommended concurrency level for most situations, including but not limited to the following scenarios:
  • Running concurrent LPM operations.
  • System evacuations.
Note: The default concurrency_lvl value that is changed to 4 from a value of 3 in VIOS version 2.2.4.0.
2 Not a recommended concurrency level.
1 Not a recommended concurrency level.
2.2.4.0 - 3.1.1.x 5 Recommended concurrency level when any of the following scenarios are true:
  • If a previous partition mobility operation fails because of insufficient memory.
  • If the partition mobility operation runs on a low speed network (less than 10 GB) and the migrating partition had failed previously, or rebooted because an application that is running on the partition had a heartbeat timer or Dead Man Switch (DMS) trigger.
  • When you are migrating from a MSP with a high-speed network to an MSP with a low speed network.
Note: Migrating a partition from a high-speed network to a low speed network is not recommended. However, if this situation cannot be avoided, using a concurrency level of 5 provides a higher probability of success.
4 The default value, and is the recommended concurrency level for most situations, including but not limited to the following scenarios:
  • Running concurrent LPM operations.
  • System evacuations.
Note: The default concurrency_lvl value changed to 4 from a value of 3 in VIOS version 2.2.4.0.
3 The recommended concurrency level only when all the following scenarios are true:
  • At least 20 Gb (gigabits) of network bandwidth is available for the MSP for each concurrent operation that is planned.
  • Both source and destination MSPs are assigned with at least two processors assigned.
  • Client logical partitions are configured with at least 50 GB of memory.
  • Both the source and destination hypervisors are at version 8.4.0, or later.
  • Both source and target MSPs are at VIOS version 2.2.4.0, or later.
Note: A maximum of four concurrent partition mobility operations can be run for each MSP pair at this concurrency level.
2.2.4.0 - 3.1.1.x 2 The recommended concurrency level only when all the following scenarios are true:
  • At least 28 Gb (gigabits) of network bandwidth is available for the MSP for each concurrent operation that is planned.
  • Both source and destination MSPs are assigned with at least 2.5 processors.
  • Client logical partitions are configured with at least 50 GB of memory.
  • Both the source and destination hypervisors are at version 8.4.0, or later.
  • Both the source and target MSPs are at VIOS version 2.2.4.0, or later.
Note: A maximum of three concurrent partition mobility operations can be run at this concurrency level. The limit is two if the operations are run with the strict flag.
1 The recommended concurrency level only when all the following scenarios are true:
  • Greater than 30 Gb (gigabits) of network bandwidth is available to the MSP for each concurrent operation that is planned.
  • Both the source and destination MSPs are assigned with at least three processors.
  • Client logical partitions are configured with at least 100 GB of memory.
  • Both the source and destination system hypervisors are at version 840, or later.
  • Both the source and destination MSPs are at version 2.2.4.0, or later.
Note: A maximum of two concurrent partition mobility operations can be run for each MSP pair at this concurrency level.
3.1.2.0, or later 4 The recommended concurrency level to enable the HMC to select appropriate resources, for all scenarios.

If the concurrency level values on the source and destination MSP different or if the MSPs are at different VIOS versions, the source and destination MSPs negotiate to a common set of resources. This generally results in either the source or destination MSPs negotiating to match the resources of the other. For migrations where you do not want the resources to be negotiated, the Hardware Management Console (HMC) version 8.4.0 and the VIOS 2.2.4.0 introduced the option of strict requirements. By specifying that the concurrency level value of strict requirements, the partition mobility validation fails if the requested resources cannot be satisfied by both the source and destination MSPs.

If you determine that the default concurrency level is not ideal for a specific partition mobility operation or for all partition mobility operations that use a specific VIOS as a MSP, you can perform one of the following actions:
  • Change the concurrency level value for all of the partition mobility operations that use a specific VIOS. The value can be set by using either the VIOS chdev command, or the HMC migrlpar command. For more information about changing the value of the concurrency level, see Live Partition Mobility pseudodevice.
  • To change the concurrency level value for a single partition mobility operation, the VIOS must be at version 2.2.4.0, or later, and the HMC must be at version 8.4.0, or later. The HMC command line provides a concurrency level override option. For a single migration operation, run the following command: 
    migrlpar -o v -m <srcCecName> -t <srcCecName> -p <lparName> -i 
"concurr_migration_perf_level=<overrideValue>"
    where the valid override values are 1, 2, 3, 4, 5, 1r, 2r, 3r, 4r, and 5r.
    For multiple migration operations, run the following command: 
    migrlpar -o v -m <srcCecName> -t <srcCecName> -p <lparName> -i  
multiple_concurr_migration_perf_levels="<lparName_1>/<lparID_1>/<perfLvl_1>,  
<lparName_2>/<lparID_2>/<perfLvl_2>,...<lparName_n>/<lparID_n>/<perfLvl_n>"
    Where the values 1 - 5 indicate the concurrency level and the values 1r - 5r indicate that the concurrency level must be strictly enforced and the migration validation fails if the resources requested by the concurrency level value cannot be fully satisfied.

    If either the source or destination MSP is at VIOS version 2.2.2.0, or earlier, the concurrence level value is ignored and the migration runs with a predefined buffer configuration and uses a single thread for sending data. This is applicable only if you select the concurrency level values in the range 1 - 5. If you select the concurrency level values in the range 1r - 5r, validation fails because the MSPs do not support multi-threading.