AIX6 Workload Partitions (WPAR) & Application Mobility (AM)

Workload Partitions improved administrative efficiency by reducing the number of AIX images to maintain by allowing you to run mini copies of AIX on a global level of AIX which owns the hardware resources. This also avoid the work involved in creating a separate Logical Partition (LPAR) for each copy of AIX.

Briefly:

• WPAR comes as part of standard AIX 6 and installed as default (no extras to install)
• AIX is effectively partitioned in software terms rather than in hardware like LPARs
• Each Workload Partition (WPAR)
  • Obtains a regulated share of system resources
  • May have unique network & filesystems
  • Separate administrative & security domain each WPAR has a unique root user and regular users and passwords

The only prerequisites is AIX 6 which means any POWER4, POWER5 or POWER6 based pSeries or System p machine or a LPAR on it.

There are two types of Workload Partitions

• System Partitions
  • Comprehensive like a miniature AIX systems
  • Complete virtualised OS environment running multiple services & applications
  • Need to created as it builds its filesystems
  • Removed only when requested.
  • Like another AIX system
    • Own root user, users, and groups
    • Own services like inetd, cron, syslog
    • Can be stopped and restarted.
  • Does not share writable file systems with other workload partitions or the global system.
  • Integrated with RBAC - granular security controls
• Application Partitions
  • Isolate an individual application
  • Light-weight, one process
  • Can start further processes
  • Created & started in seconds
  • Starts when created
  • Automatically removed when Application stops
  • Shares global file system
  • Good for HPC
  • Looks like a good candidate for High Performance Computing (HPC) with long running (as in days) application code and the checkpoint/restart could be very useful.
    Once the application stops the Application WPAR is dismantled.

Below we are concentrating on System WPARs.

WPARs share the global system resources like:

• Operating System
• underlying file systems
• Real or virtual disk adapters
• Processor and
• Memory
• Paging space
• Real or virtual network card

Also I have used the term global AIX - this is just a regular copy of AIX6 which hosts one or more workload partitions - there is nothing special here.

Moving a WPAR Between Machines 

Now the real trick - with the extra Workload Partition Manager the WPARs can be moved in a few second to a global copy of AIX on different machine. the diagram below tries to express this:

On the left the AIX copy is over worked so a small WPAR is relocated to a different machine (possibly a new machine to ease the performance), this makes room for the more important WPAR on the original machine. On the right is a machine that needs to be taken down for some repair, upgrade or maintenance - here the WPARs can be relocated temporarily to other machines. Then the global AIX shutdown along with the machine. In the future we may be able to move a WPAR from a POWER6 machine to a follow on new range of machines which will be much simpler than mksysb to tape or NIM server moving to a new machine.

Note the Workload manager is a separate piece of software and needs to be purchased. It provides:

• Enablement for Live Application Mobility (the checkpoint/restart function)
• Central Manager with underlying DB2 database and agent on each global AIX machine so you don't need to log into dozens of AIX LPARs to work with WPARs
• Cross System Management for Workload Partitions - excellent if you have hundreds of WPARs
• Automated, Policy-based Application Mobility to reduce your work

Three Names

I have found this call different things in various places:

• by marketing "Live Application Mobility" or LAM for short
• but performed by the checkpoint (chkptwpar) and restart (restartwpar) commands on AIX
• and on the Graphical Manager browser interface "Relocate"

They all refer to the same thing

Mobility How?

How is this movement actually done?

In this diagram we show th work flow:

• First we create a WPAR but place the read/write filesystem on an NFS server
• Note the read-only file systems can be shared with the global AIX to reduce disk space
• Then the WPAR can be started.
• To relocate the WPAR, the processes of the WPAR are frozen and memory+state of the WPAR and its processes are written to State Files in the NFS server
• The WPAR and NFS mount points are recreated on the new AIX6 machine
• The memory+state of the WPAR is read from the NFS files and the WPAR recreated
• the WPAR is allowed to run
• If successful, the frozen WPAR on the original machine is killed and removed
• If unsuccessful, the frozen WPAR is allowed to run - thus minimizing the restart time

What is captured in the State Files?

• Memory context:
  • resources, process hierarchy, CPU states
  • IPC
  • pipes, semaphores, shared memory segments, File descriptors, ...
  • Network
• TCP & UDP stacks, preserves connections
• No modification on WPAR side

How does WPAR Manager fit in?

A WPAR Agent is installed on each global AIX and a command run to connect it to the WPAR Manager LPAR.
The WPAR Manager with a supporting copy of DB2 runs a website on a further AIX6 machine or LPAR.

Actually this could also be one of the machines running WPARs.

You connect to this WPAR Manager from a web browser on your workstation or PC and it presents you with a screen like this:

Lets Get Practical

Below is a typical setup of machines.

Note: The NFS server does not have to be an AIX6 machine.

Creating a WPAR

You can create a WPAR in may ways:

• Command mkwpar
• AIX Smitty interface - the ever popular curses based AIX system admin tool
• The new pconsole a replacement for the older WebSM
• WPAR Manager GUI

Below we are going to use the command line because it is simple and straightforward.
If you use one of the other tools you have to support all the same information but it takes longer to type it all in and you can't cut'n'paste the next WPAR that you want to create. The command line is quite simple but there are options here too depending on what you want of your WPAR.

1 - Make the simplest possible WPAR

To make a simple WPAR with its file systems on the global AIX disks, just type the following:

mkwpar -n myfirstwpar

All we have supplied is a WPAR name and this WPAR is not even on the network. We have to use a special command (clogin) to gain access to the WPAR from the global area. while very secure this is not very useful but shows that all the defaults can be taken. This mkwpar command takes a few minutes and does the following:

• Creates the filesystems for the WPAR
• Mounts the filesystems
• copies over some files from the global AIX
• Performance an install of AIX packages (much like what we see while installing AIX) to create the files in the WPAR private filesystems (/, /var, /home and /tmp) we don't need to rebuild the read-only filesystems).
• The WPAR is prepared and made ready to run.

This takes about 3 to 4 minutes on my System p p550Q where my global LPAR is a Virtual I/O Server (VIOS) client so the disks are virtualized and actually on the VIOS. It also seems to be roughly the same regardless of local disk or NFS filesystems.

2 - Make local disk WPAR with a network.

We might as well put the WPAR straight on the network to save time. Simply add the -N options as you might expect. You will need to have a IP address and hostname allocated. We also need to supply the hostname with the -h option. Note: I name the WPAR after the hostname (i.e. wp13) - this makes life simpler but its not mandatory. Finally, the -r option means use the DNS setting of the global AIX for the WPAR - this saves time. In this example, we have assumed the global AIX is on just the one default network. If there was more than one we would need to specify which. We will explore how the WPAR gets network access later on.

mkwpar -n wp13 -h wp13 -N netmask=255.255.255.0 address=8.9.10.11 -r

Note: Of course, replace the network IP addresses to suite your network - I just made these number up.

3 - Make NFS disk WPAR with a network - ready for Mobility/Relocation

If we want to move the WPAR between global AIX copies i.e. between LPARs or machines then we need to NFS mount the WPAR private filesystems. Assuming here that we have the NFS running and have made the NFS mount points available to this global AIX and to the WPAR hostnames (in this example wp13) then we use the following command.

mkwpar \
-n wp13 \
-h wp13 \
-N netmask=255.255.255.0 address=3.4.5.6 \
-r \
-c \
-M directory=/     vfs=nfs host=blue.ibm.com dev=/scratch/wp13root \
-M directory=/home vfs=nfs host=blue.ibm.com dev=/scratch/wp13home \
-M directory=/tmp  vfs=nfs host=blue.ibm.com dev=/scratch/wp13tmp  \
-M directory=/var  vfs=nfs host=blue.ibm.com dev=/scratch/wp13var

Note the extra options in this command:

• -c means this WPAR is "checkpointable" which is used to in Live Application Mobility = Relocate to a different AIX.
• -M there is one of these for each of the four private filesystems for the WPAR. Hopefully, it is clear in the example that we have the filesystem name (directory=), the connection(vfs=nfs) and there may be other ways of supporting the file system in the future, and the hostname and mount point of the NFS server.

This gives you a fully functioning relocatable WPAR and is the command I use to create new WPARs. I simply take an existing script containing this command and edit it to suit the next WPAR. You could make a more generic single script if you have time and things like the mount points are similar for each WPAR.

4 - As above but via a Spec File

An alternative way to create a WPAR is using a "spec file" which includes the options you want. The option you specify on the command like will over ride the file contents. You can ask the mkwpar command to save a spec file with the -o option the first time you run it or you can find the master copies in the /wpars directory. You can take a spec file of a current WPAR and change it for WPAR specific details and use that to create a new WPAR.

mkwpar -f my.spec

the spec file looks like the one below:

general:
        name = "wp13"
        preserve = "no"
        hostname = "wp13"
        checkpointable = "yes"
        directory = "/wpars/wp13"
        privateusr = "no"
        devices = "/etc/wpars/devexports"

network:
        broadcast = "9.69.44.255"
        interface = "en0"
        netmask = "255.255.255.0"
        address = "9.69.44.123"

mount:
        dev = "/scratch/wp13root"
        mountopts = "bg,intr"
        directory = "/"
        vfs = "nfs"
        host = "blue.ibm.com"

mount:
        dev = "/scratch/wp13home"
        mountopts = "bg,intr"
        directory = "/home"
        vfs = "nfs"
        host = "blue.ibm.com"

mount:
        dev = "/scratch/wp13tmp"
        mountopts = "bg,intr"
        directory = "/tmp"
        vfs = "nfs"
        host = "blue.ibm.com"

mount:
        dev = "/scratch/wp13var"
        mountopts = "bg,intr"
        directory = "/var"
        vfs = "nfs"
        host = "blue.ibm.com"

mount:
        dev = "/usr"
        directory = "/usr"
        vfs = "namefs"
        mountopts = "ro"

mount:
        dev = "/proc"
        directory = "/proc"
        vfs = "namefs"
        mountopts = "rw"

mount:
        dev = "/opt"
        directory = "/opt"
        vfs = "namefs"
        mountopts = "ro"

security:
        secfile = "/etc/wpars/secattrs"

5 - Creating with smitty

The "smitty wpar" command will take you to the smitty panels to control and create a WPAR.
If you have understood the above information requirements you should have no problem at all using smitty but I find filling in all the NFS mount points etc. a little tedious.

6 - Creating with WPAR Manager

The WPAR Manager allows two ways to create a WPAR:

1. The full Guided Activity - Create Workload Partition (take the link on the left had sidebar) which will take you through a series of panels for the various aspects of the WPAR and then create a WPAR for you
2. The simple quick Resource Views - Workload Partitions (the panel which shows the WPARs) and click on "New". this asks the bare minimum of info and creates a WPAR to which you can later add.

We will look at WPAR Manager further down this article.

Starting a WPAR

This could not be simpler:

• At the command line type "startwpar wp13"
• At the WPAR Manager, select the WPAR in the defined state then Actions and start

Starting a WPAR takes only a few seconds.

Lets take a look around a WPAR and compare it to the global AIX

What can we find out from the global AIX about the WPARs we already have?

List the WPARs with lswpar

WPARs called wp03 and wp05 are active system Workload Partiotion but wp09 is only defined it can be made Active by running the "startwpar wp09" command.

Key:

• State A=active
• State D=defined
• State B=broken
• State T=transitory i.e. changing state

1. Type A=Application WPAR
2. Type S=System WPAR

In this example all my WPAR are have simple names starting with "wp" and then two digits. This is, or course, not mandatory.

What is in /wpars ?

This is a new directory specifically for WPAR use in this we find:

One sub-directory for each WPAR and in this sub-directory we find:

If you have used AIX much you will recognize this as the regular files and directories of AIX because that is pretty much what you get with an WPAR.

When I check the disk space use of a WPAR on the NFS server I find a basic just installed WPAR is using:

I make that around 25MB in size - this is a very small overhead. Remember by default the big filesystems like /usr and /opt are shared with the global AIX. If you really need a writable /usr, for example, this is possible but then you need to find the extra disk space too. If you have an application that is normally, installed in these filesystems (generally regarded as a poor application) then you can install it in the global AIX and every sharing WPAR can access the files - this can reduce the workload of installing applications a great deal.

List the mount points

The global AIX has its own mount point and the mount points for the WPARs it is hosting:

Here we see the regular mount points of AIX: /, /usr, /var, /tmp, /home, /opt
and then for each of the active WPAR there are additional mounts as below

• NFS mount for the read/write private filesystem: /, /home. /tmp and /var
• remounts to make them read-only of: /opt and /user

More WPAR details with lswpar -L

This "lswpar -L <wparname>" list all the attributes of the WPAR

Sections

• General section - has the basic information and where the WPAR data can be found
• Network section - has the IP address etc.
• Filesystems - note some local read-only filesystems nd some read-write and private over NFS in this case
• Resources like Workload Manager (WLM) which has been part of AIX for years and years - see http://www.ibm.com/developerworks/aix/library/au-Practical_WLM.html for details of how these are used as the WPAR resource control is built on top of WLM. When a WPAR is started a WLM Class is automatically created.
• Operation - this WPAR has not been moved
• Security based on Role Based Access Control (RBAC) - not covered here see the AIX Security Redbook for more details
• Export devices are various items that the WPAR has which are distinct form the global AIX for security and performance

Network

In the global AIX we can find the following out about the network:

See the network aliases (9.69.44.115 and 113) that are used to "piggy back" the real network card in the global AIX with IP address 9.69.44.236. The top line with the IP Address is the IP Address that the global AIX uses the lines below are the network aliases used by the WPAR - one each. This also highlights that the global AIX and WPAR needs to be in the same sub-net.

Logging into a WPAR

clogin

• From the global area
• WARNING any process started from a clogin will NOT survive a checkpoint and restart. This is regardless of any attempts to disconnect the processes created from the terminal session like UNIX daemon processes try to do.
• Recommend - only ever use clogin to repair a broken network otherwise you risk forgetting and your important services halting on you first Relocate. Been there - it was mystifying, painful and wasted half a day.
  See the example below:
  

telnet

• Just as you would access a regular copy of AIX you ca
• Personally, I use the excellent and freely available tool called PuTTY from my Windows XP based PC. Most of the screen captures here are taken from PuTTY.

VNC

• Again just like a regular copy of AIX, VNC can be installed and used to gain graphical X Windows access to your machine.

I have not tried it personally, but I assume SSL and ssh will work fine too once set up.

WPAR views of the filesystems

Here you can see the WPAR can only list the filesystems that it has - this is good for security. We can also see that the private filesystems (/, /home, /var and /tmp) are read/write but the file systems shared from the global AIX are read-only. This makes it very simple to make a tool or command available to all WPARs on the system - i.e. put the file in the global filesystem that is shared like /usr/local/bin and it becomes available to all WPARs.

WPAR view of the network

Here you can see the the WPAR is only aware of this network connection and so it can't get access to the global network or the network of other WPAR. This is good for security.

Compare global and WPAR view of disks and paging space

Global AIX	Inside a WPAR

Here we see that the global AIX as real disks and paging space but the WPAR has neither.
This can confuse some tools - how can a system run with no disks??? Well the WPAR does have filesystems but no direct access to the disks - this means a WPAR systems administrator can't create logical volumes nor filesystems. this is a two edged sword

• The disk management must be done at the global AIX level and then a new filesystem added to a WPAR = OK we can live with that as the global AIX is in charge of real resources.
• This stops WPAR system administrators from messing up the machine configuration.

Compare global and WPAR view of processes

Below we use "ps -ef" but not the global AIX listing is truncked as thee are lots more processes

Global AIX	Inside a WPAR

The global AIX sees all the processes on the machine and at the WPAR level only the processes in that WPAR. This means that performance management is probably best performed at the global AIX level although using resource controls the global AIX systems administrator and assign compute resources using the WLM like features and then the WPAR can manager these resources at WPAR level.

We can get a WPAR view from the global AIX in a number of clever ways:

• Add the WPAR name as the first column to the ps output: ps -ef -@

• List just the processes of a particular WPAR with: ps -ef -@ wp03

This output exactly matches the "ps -ef" output from within the WPAR itself.

Topas AIX performance monitoring tool for the global and WPAR levels

At the WPAR level topas output looks like this:

From the global AIX level it looks like this:

And you have a new WPAR view as follows with the command topas -@

Or from the global area you can view the same topas screen as the WPAR systems administrator with: topas -@ wp03

Managing lots of WPARs with WPAR Manager

Above we have been looking at a single WPAR in great detail but if you have a number of POWER based machines, with a number of Logical Partitions (LPAR) on each and then some of the LPARs are in turn running Workload Partitions (WPAR) then you will find the number of WPARs will be quite high at least a few hundred. The aim of Workload Manager is to make this very simple and to also enable The WPAR Mobility/Relocation from a graphical user interface.

Workload Manager is a graphical application and that makes if hard work to describe and explain in a web article like this but there are three main panels that do the bulk of the work and it is a good user interface and easy to pick up from scratch (once you understand a little about WPARs and hopefully the above has helped).

I will assume you have Workload manager installed - it takes me around 20 minutes to do this but it is mostly taken up with transferring the files to the LPAR on which I want to run it and then just a few commands to do the install.

First you have to log in and you can create different users etc. but we will skip this side of things an concentrate of working with the WPARs themselves.

The Workload Partitions Panel

This is the view of the most important panel.:

On the left hand side is the links to the main panels including the link to the Workload partitions panel shown in this screen capture. In the centre we have the Workload Partitions listing and various informations about them. They are all System WPARs and some Active (running) and some Defined (could be started). Let us start one of the Defined WPAR - say wp09, select the tick button next to the WPAR then select "Start" from the Actions list, then OK.

Hit OK to confirm

The WPAR goes to Transitional State

and a couple of seconds later goes Active - it is now running.

Stopping a WPAR is much the same but you get a choice of:

1. Normal stop - the users are given a 5 minute warning
2. Hard stop - the WPAR is stopped without a warning
3. Force workload partitions to stop - this should not normally be used

The Monitoring Task Activity Panel

The Workload Manager tends to only tell you that the task has been successfully started. If you want to find out what happened or even watch progress for tasks that take time you need to look at the Task Activity panel as below:

For move information on one task select the underlined Description - here is the details of the very simple Start WPAR task.

If a task fails you can also find here:

1. The command ran to action the task
2. The output from the command
3. The errors from the command
4. The name of the log file on the global AIX system that should include more information.

The Managed Systems Panel

The last important panel to cover is the Managed Systems as below:

Here we have a list of all the global AIX machines that have the WPAR Manager agent installed and have been configured (one simple command) to communicate with this copy of Workload Manager.

Conclusions

• Workload partitions are a new technology for AIX6 and are both simple you use but are a powerful tools to use to reduce Systems Administration man-power by reducing the number of copies of AIX that need to be installed, setup, secured, monitored, updated and managed.
• Well that's all folks I hope this has been useful.
• Feel free to fix up my errors and typo's as that is after all what a Wiki is all about or leave a comment below.