Multi-node on the IBM z16

On IBM® z16® hardware, you have different options with regard to a multi-node deployment. Each option uses 4 Central Processing Complexes (CPCs) or drawers for a minimal configuration. This is the recommended setup, which requires four or five accelerator LPARs.

Cross-drawer head node

An advantage of this setup is that it uses Processor Resource/Systems Manager (PR/SM) capabilities. If demand for the head node resources is low, the data nodes can fully exploit the CPU resources. If all you want to do is accelerate queries, this setup allows you to fully devote 200 IFLs to this task for a longer period of time.

Figure 1. Cross-drawer head-node setup on IBM z16
Cross-drawer head node on IBM z16

The term MLN refers to a database limit that defines the maximum number of "multiple logical nodes" per processing node. In this case, you have one logical node for the catalog in the cross-drawer head node LPAR, and four logical nodes per data LPAR.

Attention: Due to significant performance advantages of a confined head-node architecture, use of the cross-drawer head node architecture is deprecated. In a future version of the product, confined head node will be the only supported architecture for multi-node implementations. This has the following advantages:
  • Improved load and operational performance because cross-drawer traffic is avoided
  • No PR/SM interaction

Confined head node

This is a deployment option for high-end installations because a mixed workload of table load processes, IBM Integrated Synchronization updates, and analytical queries claims the entire resources of an IBM z16 system. The recommended setup depends on the number of drawers.

Important:
  • You can run shared IFL workloads on the data LPARs of a confined head node setup. However, this is not recommended for performance reasons. If you do run shared workloads, make sure these are rather small workloads. Ideally, an accelerator cluster takes up the entire resources of an IBM z16 system.
  • The distribution of MLNs across drawers and LPARs is determined by PR/SM. It is not possible to interfere manually in this process. In the example figures in this section, you find optimal distributions, which might not be achieved in reality.
  • LPAR, CPU, and memory placement play an important role in the performance of a multi-drawer setup on the IBM z16 . Any LPAR, CPU or memory that needs to work with another LPAR, CPU or memory not in the same drawer might show a certain degree of performance degradation due to cross-drawer communication. This extent of this degradation depends on the workload mix.
  • Each node, that is, the head node and each data node, requires the same amount of memory (RAM).

For 2 drawers:

For IBM z16 systems with two drawers, 4 LPARs are recommended. This translates to two LPARs per drawer. See Figure 2.

If you want to increase the system capacity, you can add IFLs to the existing drawers until you reach 50 IFLs per drawer. This leads to a very good scalability of the performance. The same is true for adding two additional drawers.

Figure 2. Confined head-node setup on IBM z16 with two drawers
Confined head node setup on IBM z16 with two drawers

For 3 drawers:

For an IBM z16 or higher with three drawers, install 3 LPARs, that is, one LPAR per drawer. See Figure 3.

If you ran four LPARs on a system with only three physical drawers, or used a system with an unbalanced number of IFLs and an unbalanced amount of memory, the overhead caused by the resulting cross-drawer memory access could be so significant that a three-drawer system would be barely faster than a two-drawer system. This is why three LPARs are recommended for a three-drawer system (one LPAR per drawer).

Figure 3. Confined head-node setup on IBM z16 with three drawers
Confined head node setup on IBM z16 with 3 drawers

For 4 drawers:

For IBM z16 systems with four drawers, four LPARs are recommended. That means one LPARs per drawer. See Figure 4:

Figure 4. Confined head-node setup on IBM z16 with four drawers
Confined head node setup on IBM z16 with 4 drawers

The term MLN refers to a database limit that defines the maximum number of "multiple logical nodes" per processing node. In this case, you have one logical node for the catalog and up to three logical nodes for data processing in the head node LPAR, plus two or four logical nodes for data processing in each data LPAR.

The IFLs on the head node should be dedicated IFLs. The IFLs of the data nodes, on the other hand, can be shared IFLs. Usually, the sharing has no great impact on the performance. This allows you, for example, to share the IFLs of a production data node with less important LPARs, such as another single-node accelerator for development and testing purposes.

Background: IBM Z® hardware distinguishes between dedicated and shared IFLs. If an IFL is dedicated to an LPAR, it belongs to that single LPAR and the PR/SM Hypervisor does not treat it as a potential resource any longer. If an IFL is not dedicated, it is shared, and thus belongs to the shared processor pool. The PR/SM Hypervisor dynamically re-assigns IFLs out of the shared pool to LPARs depending on the LPAR weights (priorities) and the resource requirements.

It is important that every LPAR runs in a single drawer only, so that cross-drawer traffic can be avoided and the available machine resources are used as much as possible.

The disadvantage of this setup is that the query processing capacity is slightly reduced if the system does not process all kinds of workloads all the time. This is because the head node has between one and three data MLNs only, whereas the data nodes have more data MLNs (two or four respectively). The database engine of the accelerator assigns the same amount of data to each data MLN.

Important:
  • In general, to change the number of LPARs, a complete reload of the accelerator and a fresh installation is required. For example, if you start with a three-drawer system with one LPAR each (as recommended), and then want to add a fourth drawer, create a new setup to reinstall the cluster with 4 drawers.
  • For two-drawer systems and four-drawer systems, 4 LPARs are recommended. That is, you either have 2 LPARs per drawer (two-drawer system), or 1 LPAR per drawer (four-drawer-system). If you want to extend a two-drawer system to a four-drawer system, you can keep the existing 4 LPARs. You need not reinstall the cluster.
  • To migrate from a setup on the IBM z15®, you do not have to change the number of accelerator LPARs. You can continue with a setup of three, four, or five LPARs in the cluster (depending on your number of drawers; use or continue with as many LPARs as drawers.). However, the performance of any such setup is not as good as a setup with three or four nodes on the IBM z16 or the IBM z17.
  • For confined head node installations, the recommendation is to use dedicated IFLs only. However, if you want to use a confined head node setup for shared IFL workloads, make sure that the LPARs that provide the shared workloads have a significantly lower priority than the LPARs of the confined head node cluster. This way, the slowing impact on the performance is kept at a minimum.