Steps

What are the different communication channel used during LPM?

There are 3 different server types involved in an LPM operation:

• The Hardware Management Console (HMC): this is the orchestrator of the operation. The LPM is issued by the HMC, and managed by the HMC. It is responsible of making sure all operation are successfully done in the correct order, and interacts with any other components to synchronize the whole migration process.

• The Virtual IO Server (VIOS): multiple VIOS will be involved in the LPM, and they could act differently:

  • Either « hosting » VIOS: they will have the virtual scsi, virtual fc and/or network resources for the client LPAR to be moved.

  • Either « mover » VIOS: they will be responsible of the memory transfer between the 2 physical systems (CEC). While it is not mandatory, usually the mover VIOS is one of the hosting VIOS. If you want to make sure the LPM does not interfere with other VIOS activity (network bridging, VFC activity…), you may want to configure some dedicated VIOS which does not have any virtual resources but will just act as MSP VIOS.

  • Or it could be a « mover and hosting » VIOS.

• The client LPAR: this is the LPAR that will move from one CEC to the other.

The different communication channels involved in LPM are:

• RMC connection (Port 657) : this is the communication channel used by the HMC to exchange with any other actor of the LPM (VIOS and LPAR).
  It is used to:

  • Check client LPAR readiness for LPM

  • Check MSP VIOS capabilities

  • Check hosting VIOS that required resources are present

  • Unconfigure / configure all required resources on the hosting VIOS

  • …

  • Basically this is the communication path used for orchestrating the LPM.

• ICMP connection: this is used to check the 2 MSP VIOS are well capable to communicate to each other, and it has enough bandwidth to handle an LPM operation.

• LPM connection (Ephemeral ports in range 32768-65535): this connection is required to transfer all the data from the source MSP VIOS to the destination MSP VIOS.
  Actually there are 2 connections with ephemeral port for each LPM operation :

  • One is the "control channel",  used by the MSP to make sure the LPM progress as expected. There are very few data transferred through this path.

  • The other is "data channel", for actual data transfer. This is the channel through which all the LPAR’s memory will be exchanged.

Here is an overview of those communication:

Note: You may also choose to use the Dual MSP feature for LPM resiliency and improved performance, in that case it is required to perform the following procedure on all involved VIOS.
For more information about Dual MSP LPM, please refer to the following technote : Live Partition Mobility Performance

How to setup the firewall on VIOS?

During the process, there should be no loss of connection to the VIOS (as long as you are connected using SSH), but it might be better to perform the procedure while connected on the VIOS console, and during a maintenance window.

Note: If your VIOS is part of an SSP cluster, you will also need to configure the firewall for SSP related communication.
Please refer to the following technote : How to setup VIOS firewall in an SSP Cluster environment.
To determine if your VIOS is part of an SSP cluster, you can run the following :
$ cluster -list
CLUSTER_NAME: galaxy_cluster
CLUSTER_ID: 871b1460190111e6800200215e89cd29
If it shows « Cluster does not exists. », then there’s no SSP cluster on your VIOS.

1. The best starting point for configuring the firewall on VIOS is to activate it with the default settings :

$ viosecure -firewall on -reload

This will enable the firewall, and start with the predefined filter rules. By default the firewall blocks all communications except a short range of port number including « RMC (657), SSH (22), ftp (21)... ». As you can the RMC communication will be enabled by default (which is good because it’s no only used for LPM but also for DLPAR, VIOS inventory for HMC, …).
You can check this default configuration with :

$ viosecure -firewall view

2. Now that firewall is enabled we’ll have to allow the ICMP request and reply. Unfortunately there are no « padmin » commands to do so, hence we have to go OEM :

$ oem_setup_env
# /usr/sbin/genfilt -v 4 -n 16 -a P -s 0.0.0.0 -m 0.0.0.0 -d 0.0.0.0 -M 0.0.0.0 -g n -c icmp -o eq -p 0 -O any -P 0 -r L -w I -l N -t 0 -i all -D echo_reply
# /usr/sbin/genfilt -v 4 -n 16 -a P -s 0.0.0.0 -m 0.0.0.0 -d 0.0.0.0 -M 0.0.0.0 -g n -c icmp -o eq -p 8 -O any -P 0 -r L -w I -l N -t 0 -i all -D echo_request
# exit

3. We need to setup the LPM related ports, first of all we have to restrict the range of ephemeral port used by LPM, this can be done using the vioslpm0 pseudo-device:

$ chdev -dev vioslpm0 -attr tcp_port_high=40010
$ chdev -dev vioslpm0 -attr tcp_port_low=40001

Note: As the LPM requires 2 channel between each MSP pair, with the above settings the number of concurrent LPM operation possible on the MSP will be 5. You should set your ephemeral port range according to the number of concurrent LPM operation you want the MSP to handle.

With above settings the LPM operation will now use a set of port number between 40001 and 40010, so we have to open those port in our firewall :

$ viosecure -firewall allow -port 40001
$ viosecure -firewall allow -port 40001 -remote
$ viosecure -firewall allow -port 40002
$ viosecure -firewall allow -port 40002 -remote
…

You may also script this with the following command :

$ for i in 01 02 03 04 05 06 07 08 09 10
do
viosecure -firewall allow -port 400$i
viosecure -firewall allow -port 400$i -remote
done

Those last 2 steps (2 & 3) are mandatory on all VIOS you want to use as MSP VIOS.

If you have an external firewall, you may use the same set of port range and communications channel to setup this firewall.

Once you completed this setup, the LPM operation should work just like if there’s no firewall at all.

Quick setup on VIOS 3.1.4.10.

Starting with VIOS 3.1.4.10, some new option were added to viosecure command to help setting up the firewall in an LPM capable environment.

It's a two steps configuration :

1. As in older version, first of all we have to restrict the range of ephemeral port used by LPM, this can be done using the vioslpm0 pseudo-device:
   $ chdev -dev vioslpm0 -attr tcp_port_high=40010
   $ chdev -dev vioslpm0 -attr tcp_port_low=40001
   Note: As the LPM requires 2 channel between each MSP pair, with the above settings the number of concurrent LPM operation possible on the MSP will be 5. You should set your ephemeral port range according to the number of concurrent LPM operation you want the MSP to handle.

2. Then, a single viosecure command enables ping (with "-icmp" option) and open all required port for MSP work (option "-lpm") :

How to disable the firewall?

If anything goes wrong, or you simply decide to not use the firewall anymore, you can just stop the firewall with the following command :

$ viosecure -firewall off

If you are facing any issue with the LPM, collect the following testcase :

• on all VIOS involved: « snap » (just run snap command from padmin without any option)

• on all VIOS the output of: « viosecure -firewall view »

• on HMC: a « pedbg -cq 4 » (please refer to https://www.ibm.com/support/pages/hmc-enhanced-view-collecting-pedbg-hmc )

And open a case with PowerVM VIOS support.