Topic
  • 8 replies
  • Latest Post - ‏2013-01-16T22:03:19Z by SystemAdmin
SystemAdmin
SystemAdmin
1743 Posts

Pinned topic Live Partition Mobility and concurrent migrations.

‏2012-05-18T13:20:13Z |
Hi,

We are experiencing problems with concurent LPM migration, it seems that migrations are ALWAYS sequentials and not parallels. (we are trying to migrate two lpars at the same time)

  • Names are replaced for this example.

  • source system : P795 (PSERIE1)
  • destination system : P795 (PSERIE2)
  • PSERIE1 : 16 Vio Servers, and 2 MSP (vios1 and vios2).
  • PSERIE2 : 16 Vio Servers, and 2 MSP (vios3 and vios4).
  • VIO Server version (2.2.1.3) :

# ioslevel 2.2.1.3


On HMC we can check that we can migrate 8 lpars in the same time :

# lslparmigr -r sys -m PSERIE1 inactive_lpar_mobility_capable=1,num_inactive_migrations_supported=4,num_inactive_migrations_in_progress=0,active_lpar_mobility_capable=1,*num_active_migrations_supported=8*,num_active_migrations_in_progress=0,inactive_prof_policy=config # lslparmigr -r sys -m PSERIE2 inactive_lpar_mobility_capable=1,num_inactive_migrations_supported=4,num_inactive_migrations_in_progress=0,active_lpar_mobility_capable=1,*num_active_migrations_supported=8*,num_active_migrations_in_progress=0,inactive_prof_policy=config


We are running in the same time these commands on HMC :
  • migrating lpar1 :

# migrlpar -o m -m PSERIE1 -t PSERIE2 -p lpar1 -w 1 -i 
'virtual_fc_mappings="4/vios3/1,5/vios4/2",source_msp_name=vios1,dest_msp_name=vios2,shared_proc_pool_name=shp1'

  • migrating lpar2 :

# migrlpar -o m -m PSERIE1 -t PSERIE2 -p lpar2 -w 1 -i 
'virtual_fc_mappings="4/vios3/1,5/vios4/2",source_msp_name=vios1,dest_msp_name=vios2,shared_proc_pool_name=shp1'


While migrating we are checking vasi0 status on vios1 :
  • Migrating lpar1 :

# vasistat -d vasi0 |grep -wp ACTIVE   Operation #0 (ACTIVE): ---------------------- Operation Type: Migration (Source) Stream ID: CDF16F16BD2BD31A TOP/BOTTOM: 00000000/00001000 Elapsed Time: 0 days 0 hours 3 minutes 20 seconds Flags: <RUNNABLE> Operation State: Operational Stream State: Resumed Total Bytes to Transfer: 17721458688 Bytes Left to Transfer: 16547840 # vasistat -d vasi0 |grep -wp ACTIVE Operation #0 (ACTIVE): ---------------------- Operation Type: Migration (Source) Stream ID: CDF16F16BD2BD31A TOP/BOTTOM: 00000000/00001000 Elapsed Time: 0 days 0 hours 3 minutes 25 seconds Flags: <RUNNABLE> Operation State: Operational Stream State: Completed Total Bytes to Transfer: 17721458688 Bytes Left to Transfer: 0

  • waiting :

# vasistat -d vasi0 |grep -wp ACTIVE # vasistat -d vasi0 |grep -wp ACTIVE # vasistat -d vasi0 |grep -wp ACTIVE

  • then lpar 2 is finally migrating :

# vasistat -d vasi0 |grep -wp ACTIVE # date && vasistat -d vasi0 |grep -wp ACTIVE Fri May 11 11:33:24 DFT 2012 Operation #0 (ACTIVE): ---------------------- Operation Type: Migration (Source) Stream ID: 99FE86696005168A TOP/BOTTOM: 00000000/00001000 Elapsed Time: 0 days 0 hours 0 minutes 15 seconds Flags: <RUNNABLE> Operation State: Operational Stream State: Enabled Total Bytes to Transfer: 8863088640 Bytes Left to Transfer: 7526625280


It seems that migration are not running at the same time. On HMC GUI we have the same problem.

Has anyone succeded in concurrent migration, or am I missing some configuration ?

Hope you can help me.

Benoît.
Updated on 2013-01-16T22:03:19Z at 2013-01-16T22:03:19Z by SystemAdmin
  • pchen@us.ibm.com
    pchen@us.ibm.com
    2 Posts

    Re: Live Partition Mobility and concurrent migrations.

    ‏2012-05-18T21:32:09Z  
    LPM operation has several steps. They are:

    1. Validation,
    2. Create and configure the partition on the target side
    3. Start moving memory data
    4. Clean up after finishing the migration

    When we do first two steps, we have to lock the target machine to make sure no one is doing any update to it.
    When we start step 3 (which is what you checked for VASI activities), the other migration thread can come in to do the first two steps
    But if the partition's memory size is not large enough, you will see the first migration thread already finished step 3 and the second migration thread have not start the step 3 yet.

    That is why you see the migration always happened sequentially. So just checking the VASI activities will not give you the complete picture of a LPM
  • SystemAdmin
    SystemAdmin
    1743 Posts

    Re: Live Partition Mobility and concurrent migrations.

    ‏2012-05-19T13:31:00Z  
    First of all thanks a lot for your quick answer.

    If I understand what you mean :
    • step 1 and step 2 cannot be running in the same time for two different lpar ?
    • When step 1 and step 2 are finished for one lpar, another can begin step 1 and step 2, while another is already on step 3 ?

    On our two P795 we already have ~30 lpars on each side. Migrating all theses lpars to the other side takes ~4 hours. We are migrating ~10 lpars/weeks on theses two 795, so lpar count is growing fast and it's not acceptable for us to wait an half day for hardware maintenance (what will happen when we'll have 100 lpars on each side, 2 days for migration ? :-) ).

    We have setup LPM on 2 P720 and one migration take ~2min, on P795 it take ~15min per lpar. The only difference between p720 and p795 is the number of VIO Server running on each side !
    • on p720 : two VIO Server per Pserie.
    • on p795 : 16 VIO server per Pserie (our customers want to have their own VIO Server, we know it's not the best choice, but we don't have any choice left if we want to consolidate our AIX Servers).

    It seems that step 1 (validition) check each possibility on each VIO Server. Is it possible that migration takes a lot of time on P795 because we have a lot VIO Server, and validition checks are running for each one ? Is it possible to force check on 2 destination VIO (ignoring the others ?).

    Thanks in advance for your answer.

    Best.

    Benoît.
  • seroyer
    seroyer
    352 Posts

    Re: Live Partition Mobility and concurrent migrations.

    ‏2012-05-19T17:52:29Z  
    First of all thanks a lot for your quick answer.

    If I understand what you mean :
    • step 1 and step 2 cannot be running in the same time for two different lpar ?
    • When step 1 and step 2 are finished for one lpar, another can begin step 1 and step 2, while another is already on step 3 ?

    On our two P795 we already have ~30 lpars on each side. Migrating all theses lpars to the other side takes ~4 hours. We are migrating ~10 lpars/weeks on theses two 795, so lpar count is growing fast and it's not acceptable for us to wait an half day for hardware maintenance (what will happen when we'll have 100 lpars on each side, 2 days for migration ? :-) ).

    We have setup LPM on 2 P720 and one migration take ~2min, on P795 it take ~15min per lpar. The only difference between p720 and p795 is the number of VIO Server running on each side !
    • on p720 : two VIO Server per Pserie.
    • on p795 : 16 VIO server per Pserie (our customers want to have their own VIO Server, we know it's not the best choice, but we don't have any choice left if we want to consolidate our AIX Servers).

    It seems that step 1 (validition) check each possibility on each VIO Server. Is it possible that migration takes a lot of time on P795 because we have a lot VIO Server, and validition checks are running for each one ? Is it possible to force check on 2 destination VIO (ignoring the others ?).

    Thanks in advance for your answer.

    Best.

    Benoît.
    Yes, you can tell the HMC which VIOS(s) to consider. That can significantly reduce the time required for the validation/setup steps if you have many VIOSs on your systems. See the help or man page for migrlpar. You'll want to specify the virtual_scsi_mappings/virtual_fc_mappings as appropriate for your lpars, as well as the source_msp_/dest_msp_ attributes.

    Steve
  • SystemAdmin
    SystemAdmin
    1743 Posts

    Re: Live Partition Mobility and concurrent migrations.

    ‏2012-05-19T19:45:31Z  
    Hi Steve,

    Thanks you too for your answer.

    As you can see on my first post I already set source and destination MSP, and virtual_fc_mapping. In my opinion source and destination MSP are not necessarily source and destination VIOS (used for vscsi,vfcs,and vethernet), they are just used to transfert memory between source and destination systems (for example in our P795 we just have two MSP per Pserie, and we are using it for all migrations, no matter if vscsi,vfcs and vethernet as to be on another non MSP vios).

    Adding -v to migrlpar command get me the information if i can migrate my adapter to vios. And as I can see it today, LPM check that adapter can be hosted on each VIOs on the pserie, even if virtual_fc_mapping. Have you already made this test.

    Benoît.
  • pchen@us.ibm.com
    pchen@us.ibm.com
    2 Posts

    Re: Live Partition Mobility and concurrent migrations.

    ‏2012-05-21T21:18:45Z  
    Would you tell me what is your HMC level. You can get it by running command "lshmc -V"
  • SystemAdmin
    SystemAdmin
    1743 Posts

    Re: Live Partition Mobility and concurrent migrations.

    ‏2012-05-22T08:10:54Z  
    Would you tell me what is your HMC level. You can get it by running command "lshmc -V"
    Hi,

    Here is my HMC Level :
    
    # lshmc -V 
    "version= Version: 7 Release: 7.4.0 Service Pack: 1 HMC Build level 20111216.1 
    ","base_version=V7R7.3.0 
    "
    


    Thanks a lot for your answer.
  • SystemAdmin
    SystemAdmin
    1743 Posts

    Re: Live Partition Mobility and concurrent migrations.

    ‏2013-01-16T17:15:40Z  
    Hi,
    I was just going through these posts and noticed this interesting discussion. We have a similar environment with 2 Power 795s but we also have two HMCs managing them.
    so If I use 2 HMCs I can start the migration of LPARs in parallel and steps 1 & 2 can be performed at the same time on two different LPARs moving from say 795-1 to 795-2.
    I did not see any issues LPMing them but if you think this is not supported please let me know.

    • Thanks in advance for your reply.
  • SystemAdmin
    SystemAdmin
    1743 Posts

    Re: Live Partition Mobility and concurrent migrations.

    ‏2013-01-16T22:03:19Z  
    Hi,
    I was just going through these posts and noticed this interesting discussion. We have a similar environment with 2 Power 795s but we also have two HMCs managing them.
    so If I use 2 HMCs I can start the migration of LPARs in parallel and steps 1 & 2 can be performed at the same time on two different LPARs moving from say 795-1 to 795-2.
    I did not see any issues LPMing them but if you think this is not supported please let me know.

    • Thanks in advance for your reply.
    Hi Rkhaleel,

    My problem was solved with the latest HMC version and Virtual I/O Server version and a workaround. Our main problem was the max_virtual_slot value (higher than 1000 on our Virtul I/O Servers). Changing the max_virtual_slot value to 256 on Virtual I/O Server profile (and reboot) has corrected the problem. Starting two parallels migrartions from differents HMCs will work, but will not correct any performance issues due to max_virtual_slot value. After reboot of the virtual I/O Server, 8 lpars can be moved at the same time.

    You can check this APAR : http://www-01.ibm.com/support/docview.wss?uid=isg1IV20409

    Hope this can help.

    Benoit.

    http://chmod666.org