My team and I have recently
been trying to stream line our AIX disaster recovery process. We’ve been
looking for ways to reduce our overall recovery time. Several ideas were tossed
around such as a) using a standby DR LPAR with AIX already installed and using
rsync/scp to keep the Prod & DR LPARs in sync and b) using alt_disk_copy
(with the –O flag for a device reset) to clone rootvg to an alternate disk
which is then replicated to DR. These methods may work but are cumbersome to
administer and (in the case of alt_disk_copy) require additional (permanent) resources
on every production system. With over 120 production instances of AIX, the disk
space requirements start to add up.
So far we’ve concluded that
the best way to achieve our goal is by using SAN replicated rootvg volumes at
our DR site.
Our current DR process relies
on recovery of AIX systems from mksysb images from a NIM master. All our data
(non-rootvg) LUNs are already replicated to our DR site. The aim was to change the
process and ‘recover’ our AIX images using replicated rootvg LUNs. This will
reduce our overall recovery time at DR (which is crucial if we are to meet the proposed
recovery time objectives set by our business). Based on current IBM
documentation we were relatively comfortable with the proposed approach. The
following IBM developerWorks article (originally
published in 2009 and updated in late 2010) describes “scenarios in which remapping, copying, and reuse of SAN disks is
allowed and supported. More easily switch AIX environments from one system to
another and help achieve higher availability and reduced down time. These
scenarios also allow for fast deployment of new systems using cloning.”
The document focuses on fully virtualised environments
that utilise shared processors and VIO servers. One area where this document is
currently lacking in information is the use of NPIV and virtual fibre channel
adapters in a DR scenario. We reached out to our contacts in the AIX
development space and asked the following question:
“Hoping you can help us find some statements regarding support
for a replicated rootvg environment using NPIV/Virtual Fibre Channel adapters?
The following IBM developerWorks article discusses VSCSI and we are looking for
something similar for NPIV.
http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html
My guess is that restrictions similar to those for physical FC adapters will
apply here? But I'm hoping that given the adapters are virtual the limitations
may be relaxed.
Are you aware of any statement regarding support (or not) for booting from
another system using a disk subsystem image of rootvg replicated to another
disk subsystem when using NPIV? And what, if any, additional requirements/restrictions
may apply when using NPIV?”
We received the following
responses:
“There are some
additional considerations when using NPIV for booting from a replicated rootvg.
With NPIV the client partitions has virtual Fibre Channel adapter ports,
but has physical access to the actual (physical) disk devices. There may
be an increased chance of needing to update the boot list via the Open Firmware
SMS menu. Since the clients have access to the actual disks, you have the
possibility of running multipathing software besides AIX MPIO. If you
are using a multipathing software to manage the NPIV attached disks besides AIX
MPIO, then you should contact the vendor that provided the software to check
their support statement,
Since one
or more of the physical devices will change when booting from an NPIV
replicated rootvg, it is recommend to set the ghostdev attribute. The
ghostdev attribute will trigger when it detects the AIX image is booting from
either a different partition or server. Ghostdev attribute should not
trigger during LPM operations (Live Partition Mobility). Once triggered,
ghostdev will clear the customized ODM database. This will cause detected
devices to be discovered as new devices (with default settings), and avoid the
issue with missing/stale device entries in ODM. Since ghostdev does clear
the entire customized ODM database, this will require you import your data
(non-rootvg) volume groups again, and perform any (device) attribute
customization. To set ghostdev, run "chdev -l sys0 -a ghostdev=1". Ghostdev
must be set before the rootvg is replicated.
Similar
with virtual devices, the client partition is booting an existing rootvg where
the hardware may be different. It's possible some applications have
dependency on tracking the actual physical devices (instead of data on the
disks). For example PowerHA, may keep track of a disk for a cluster
health checks. If you do have applications that have a dependency
on tracking physical devices, then additional setup (of those applications) may
be required after the first boot from the replicated rootvg.
We do have
multiple customers using NPIV for such scenarios. I believe most of them
worked with IBM Lab Based Services to assist with implementing such a
configuration, and some of the customers required some custom scripts to
further customize their system after booting from the replicated rootvg.
Those customers set the ghostdev attribute, and had custom scripts to
import their data (non-rootvg) volume groups, and update PowerHA to point to
the new health check disk.
You should
get support for such an NPIV setup with IBM as long as you follow the
considerations listed in the WhitePaper.”
“Development has approved using
NPIV to do this for one customer. Below are more detailed requirements for this
DR strategy using NPIV. If a Disaster Recovery (DR) environment not using
PowerHA Enterprise Edition is used, then
we believe the white paper located http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html provides the guidelines in regards to setup, prerequisites, and
as well as the limitations of such a DR deployment. Deployments as detailed in the white paper
are supported by IBM. However note that
such a deployment has many manually instituted responsibilities on the customer
to setup and maintain such an environment. IBM expects that the customer
carefully manage these manual steps without any mistakes. The White paper does not currently cover
using NPIV in such a DR scenario.
We have the following guidelines
regarding the configuration, which includes NPIV as an option: All of the
system configuration should be virtualized, with the possible exception of disk
devices when using NPIV. If NPIV is
used then AIX MPIO must be used as the multi-pathing solution. If multi-pathing software is used besides AIX
MPIO, then the vendor of that software must be contacted regarding a support
statement. Install AIX (at least minimum required TLs/SPs for desired AIX
version) and software stack (Middleware and applications) on primary systems,
which is compatible with systems at both sites. Primary and secondary sites
should be using systems with similar hardware, same microcode levels, and same
VIOS levels.
Many manual steps are needed to
setup the virtual and physical devices accurately on secondary site VIOS. If Virtual SCSI Disks are being used, then
discover the unique identification on the primary site and map the disks to the
corresponding replication disks. Map the same appropriately on VIOS on
secondary site. Level of VIOS should support attribute to open the secondary devices
passively. This setting needs to be setup correctly on the VIOS on secondary
site. Operating environment should not have subnet dependencies. Manage the
replication relationships accurately. Manually may need to switch the secondary
disks to primary node. Raw disk usage may cause problems. Some middle ware
products may bypass Operating system and use the disk directly. They might have
their own restrictions for this environment.
(e.g. anything that is device location code or storage LUN unique ID dependant
may have issues when the cloned image is restarted on the secondary system with
replicated storage).
Set the "ghostdev"
attribute using chdev command (must be done on the primary). This attribute can
be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev
attribute will delete customized ODM database on rootvg when AIX detects it has
booted from a different LPAR or system. If the "ghostdev" attribute
is not set, then the booting from the alternate site will result in devices in
ODM showing up in "Defined" or "Missing" state.
After a failover to secondary
site, may need to reset the boot device list for each LPAR before the boot of
the LPAR using SMS menus of the firmware Note that this is not an exhaustive
list of issues. Refer to the white paper and study the same as it applies to
the environment. So as long as they are using MPIO, you're OK. If not using MPIO and some OEM storage, then
the storage vendor must also support it.”
While these responses indicate that this form of
recovery is supported by IBM, we were still looking to IBM for clarity on the
support position. It has been noted that other IBM customers have had mixed
responses when contacting AIX support for feedback and assistance with this
type of DR procedure. And it’s not hard to see why when you read statements
like this from the “Supported Methods of Duplicating an AIX
System” document:
“Unsupported
Methods
1. Using a bitwise copy of a rootvg disk
to another disk.
This bitwise copy can be a one-time snapshot copy such as flashcopy, from one
disk to another, or a continuously-updating copy method, such as Metro Mirror.
While these methods will give you an exact duplicate of the installed AIX
operating system, the copy of the OS may not be bootable. A typical scenario
where this is tried is when one system is a production host and there is a
desire to create a duplicate system at a disaster recovery site in a remote
location.
2. Removing the rootvg disks from one
system and inserting into another.
This also applies to re-zoning SAN disks that contain the rootvg so another
host can see them and attempt to boot from them.
Why don't these methods work?
The reason for this is there are many objects in an AIX system that are unique
to it; Hardware location codes, World-Wide Port Names, partition identifiers,
and Vital Product Data (VPD) to name a few. Most of these objects or
identifiers are stored in the ODM and used by AIX commands.
If a disk containing the AIX rootvg in one system is copied bit-for-bit (or
removed), then inserted in another system, the firmware in the second system
will describe an entirely different device tree than the AIX ODM expects to
find, because it is operating on different hardware. Devices that were
previously seen will show missing or removed, and usually the system will
typically fail to boot with LED 554
(unknown boot disk).”
So, as a secondary objective,
we have been working closely with our local IBM representatives to obtain some
surety from IBM that our proposed DR strategy for AIX is fully supported by
both the AIX development and support teams.
With that in mind I’ll
provide an overview of our new DR approach and hope that it offers others insight
to alternative method for recovery and also to assist IBM in further
understanding what some of the “larger” AIX customers are looking for in terms
of simplified AIX disaster recovery.
What follows is a detailed
description of our IBM AIX, PowerVM/Power Systems environment, the proposed
recovery steps and other items for consideration.
Our Environment:
· -
AIX 5.3*, 6.1
& 7.1 - 5300-12-04-1119, 6100-06-05-1115 and 7100-01-04-1216.
·
- VIOS 2.2.1.3 –
both production and DR.
· -
VIOS physical FC
adapters – both production and DR - 8Gb PCI Express Dual Port FC Adapter (5735)
– Firmware level: 2.00X7.
·
- All client LPARs
utilise Virtual FC adapters (NPIV) for disk storage.
·
- Production:
POWER7 795 (9119-FHB). Firmware level: AH730_078.
·
- DR: POWER6
595 (9119-FHA). Firmware level: EH350_120.
·
- HDS VSP storage –
at both production and DR sites.
·
- AIX MPIO only –
HDS ODM driver is installed (devices.fcp.disk.Hitachi.array.mpio.rte) – HDLM is
not installed.
·
- Production:
Dual VIOS.
·
- DR: Single
VIOS.
·
- Production:
LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru
via dual VIOS. MPIO is in use i.e. one path per VIOS/Physical FC.
·
- DR: LUNs are
mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via a
single VIOS. i.e. single path only.
·
- Please refer to the following table for a summary of
the environment details.
Our Recovery Procedure:
1.
- Change the sys0 ghostdev attribute value to 1 on the source production AIX system.
Set the "ghostdev" attribute using chdev command (must be done on the
primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The
ghostdev attribute will delete customized ODM database on rootvg when AIX
detects it has booted from a different LPAR or system.
2.
- Take note of the
source systems rootvg hdisk PVID.
3.
- Select the source
production rootvg LUN for replication on the HDS VSP.
4. - Replicate the LUN
from the production site to the DR HDS VSP.
5.
- In a DR test, suspend HDS replication from production to DR.
6.
- Assign the
replicated LUN to the target LPAR on the DR POWER6 595 i.e. map the LUN to the
WWPN of the virtual FC adapter on the DR LPAR.
7.
- Attempt to boot
the DR LPAR using the replicated rootvg LUN. If necessary, enter SMS menu to
update the boot list i.e. select the correct boot disk, check for the same PVID
as the source host.
8.
- Once the LPAR has
successfully booted, the AIX administrator would configure the necessary
devices i.e. import data volume groups, configure network interfaces, etc. This
may also be scripted for execution during the first boot process.
9. - Please refer
to the following diagrams for a visual representation of the proposed process.
Some Notes/Caveats:
The following is a list of
items that we understand are possible limitations and issues with our new DR
process.
·
- Booting from
replicated rootvg disks may fail for several reasons, such as, a) there is
unexpected corruption in the replicated LUN image due to rootvg not being
quiesced during replication or b) there is a unidentified issue with the AIX
system that is only apparent the next time the system is booted; this could be
mis-configuration by the administrator or some other unforseen problem.
·
- In the event that
an LPAR fails to boot via a replicated rootvg LUN, a backup method is available
for recovery. Switching back to manual NIM mksysb restore provides a sufficient
backup should the replicated rootvg be unusable.
·
- If the
"ghostdev" attribute is not set, then booting from the DR site will
result in devices in the ODM showing up in a "Defined" or
"Missing" state.
· -
Once a DR test is
completed, the DR LPAR should be de-activated immediately so that SAN disk
replication can be restarted between production and DR. Failure to perform this
step may result in the DR LPAR failing as a result of file system corruption.
·
- At present we are
using AIX MPIO only. There is discussion of using HDLM in the future. We will
contact HDS for a support statement regarding booting from replicated rootvg
LUNs with HDLM installed.
·
- The ghostdev
attribute is not implemented in AIX 5.3. AIX 5.3 is no longer supported*.
So far all of our testing has
been successful. We verified that we could replicate an SOE rootvg image of AIX
6.1 and 7.1 to DR and successfully boot an LPAR using the replicated disk.
Based on these tests there doesn’t appear to be anything stopping us from using
this method for DR purposes. The following table outlines the different
versions of AIX we tested and the results.
Once the system was booted we
needed to perform some post boot configuration tasks. These tasks were handled
by two scripts that were called from /etc/inittab. On the source system we
installed the new scripts (in /etc) and added new entries to the /etc/inittab
file. These scripts only run if the systemid matches that of the DR systemid. Note: Only partial contents of each script
are shown below…but you get the idea.
# mkitab
-i srcmstr "AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console
2>&1 #Reconfigure the system for DR on boot."
#
mkitab -i srcmstr "AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh >
/dev/console 2>&1 #Reconfigure the system for DR on boot."
#
lsitab -a | grep DR
AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh
> /dev/console 2>&1 #Reconfigure the system for DR on boot.
AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh
> /dev/console 2>&1 #Reconfigure the system for DR on boot.
AIX_DRconfig.ksh:
#!/bin/ksh
#set
-xv
#################################################################
#
#
This script will configure the AIX environment for DR purposes.
#
This script will only run if the systemid matches the systemid of the DR
system.
#
aixdr : / # lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print
$2}'
#
# 0211D11C1
= DR POWER System
#
#################################################################
MYNAME=$(basename
$0)
DR_SYSTEMID=0211D11C1
LSATTR_SYSTEMID_DR=`lsattr
-EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'`
if [
"$LSATTR_SYSTEMID_DR" = "$DR_SYSTEMID" ]
then
#Set the DR hostname, IP address, netmask
and gateway.
#mktcpip -h <hostname> -a <IP to
use in DR> -m <network mask> -i en0 -g <gateway IP>
#en0:10.1.5.99:255.255.252.0:10.1.1.10,255.255.255.0
10.2.2.10,255.255.255.0
dr_hostname=`cat /usr/local/dr/dr_hostname.txt`
dr_defgw=`cat /usr/local/dr/dr_defgw.txt`
dr_intf=`cat /usr/local/dr/dr_en0.txt | awk
-F: 'print $1}'`
dr_ip=`cat /usr/local/dr/dr_en0.txt | awk
-F: 'print $2}'`
dr_netm=`cat /usr/local/dr/dr_en0.txt | awk
-F: 'print $3}'`
echo "$MYNAME: Setting DR hostname and
IP address."
mktcpip -h $dr_hostname -a $dr_ip -m
$dr_netm -i $dr_intf -g $dr_defgw
#Configure IP aliases
#chdev -l en0 -a
alias4=10.1.1.10,255.255.255.0
#Remove IP aliases
#chdev -l en0 -a
delalias4=10.1.1.10,255.255.255.0
for i in `cat /usr/local/dr/dr_en0.txt | awk
-F: '{print $4}'`
do
chdev -l $dr_intf -a alias4=$i
done
for inet in `cat dr_inet.txt | grep -v en0`
do
for i in `cat /usr/local/dr/dr_$inet.txt
| awk -F: '{print $4}'`
do
chdev -l $inet -a alias4=$i
done
done
#Configure the AIX environment for DR.
echo "$MYNAME: Configure the AIX
environment for DR."
echo
#Delete the MDC DNS entries
echo "$MYNAME: Configuring DNS for
DR."
namerslv -d -i 10.1.6.21
namerslv -d -i 10.1.4.21
namerslv -d -i 10.1.5.21
#Add the DR DNS entry
namerslv -a -i 10.1.7.38
echo
echo "$MYNAME: Configuring bootlist for
DR."
bootlist -m normal -o
bootlist -m normal hdisk0
bootlist -m normal -o
echo
echo "$MYNAME: Configuring /etc/inittab
for DR."
lsitab -a | grep nim
chitab
"nimclient:2:off:/usr/sbin/nimclient -S running > /dev/console
2>&1 # inform nim we're
running"
lsitab -a | grep nim
echo
else
MYNAME=$(basename $0)
echo "$MYNAME: The systemid
$LSATTR_SYSTEMID_DR does not match the expected DR systemid of
$DR_SYSTEMID."
echo "$MYNAME: This script should only
be executed at DR."
echo "$MYNAME: If you are not booting
the system at the DR site, then you can ignore this message."
echo "$MYNAME: No changes have been
perfomed. Script is exiting."
fi
The ghostdev attribute essentially provides us with a clean ODM
and allows the system to discover new devices and build the ODM from scratch.
If you attempt to boot from a replicated rootvg disk without first setting the ghostdev attribute, your system may fail to boot (hang at LED 554)
because of a new device tree and/or missing devices. You might be able to recover from this
situation (without restoring from mksysb) by performing the steps outlined on
pages 16-20 of the following document (thanks to Dominic Lancaster at IBM for
the presentation).
https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/IBMSTGEducationSymposium_AX61_POWERVM_P2V.pdf
13
