Introduction

IBM IDS 9.4 has made few major enhancements on ontape utility as follows:

• Rename option in restore. This is a revolutionary enhancement. IDS 9.4 has added a new option in ontape which allows you to change chunk path and chunk offsets when restore:

Previous IDS versions are very strict about chunk path and chunk offsets in ontape restore; if the chunk path or chunk offsets in restore is different from that in backup, the restore would fail. With rename option, this is no longer a restriction in restore which gives ontape utility a variety of flexibility. You can actually use ontape to backup a database in one location and restore it in another without worrying about the differences in chunk path and chunk offsets. The only requirement is that the devices or chunks being restored must have correct ownership and group permissions.

ontape
usage:
{ -a  | 
  -c  | 
  -l  | 
  -p  | 
  -r [-rename {-f filename |-p old path -o old offset -n new path -o new offset...}]
     [-D DBspace_list] | 
  -s [-L archive_level] [-A database_list] [-B database_list]
     [-N database_list] [-U database_list]  }
 
-a Automatic backup of logical logs
-c Continuous backup of logical logs
-l Logical restore
-p Physical restore for HDR
-r Full restore DBspaces/BLOBspaces as listed
-s Archive full system
-A set the following database(s) to ansi logging
-B set the following database(s) to buffered logging
-N set the following database(s) to no logging
-U set the following database(s) to unbuffered logging
-rename rename chunks during cold restore
  with -rename options :
    -f filename pathname of file containing list of mapped
                  chunk pathnames and offsets
    -p old pathname of chunk
    -o old offset of chunk
    -p new pathname of chunk
    -o new offset of chunk

With this rename option, IDS 9.40 even allows you to restore an Informix instance to non-existing devices. This is very useful if you want to perform database restore before you install new devices. But you need to remember to do hot restore after you installed new devices. For more details about this new feature, refer to IBM Informix Backup and Restore Guide.

• To use all available spaces on backup media or devices. If you set TAPESIZE parameter in Informix configuration file to 0, IDS 9.40 will allow you to use all available spaces on the media or devices for backup. This feature allows more flexibility in backup. With previous versions, you would need to set this parameter to the exact available spaces on the media or devices whether it is 1 G or 1.29346 G. This is difficult to estimate especially for a highly dynamic system. If the backup exceeded the value of this parameter, the server will hang and the backup would fail. Now with this new feature, you do not have to worry about all those details; Informix server is smart enough to figure out how much space is left on the device by itself.
• Much better compression. IDS 9.40 ontape utility provided better compression of the backup data than previous versions. Numerous repeated tests, found a database that used to take about 6 G disk spaces to store backup files in IDS 9.21 now it takes only 5 G in IDS 9.40; this saves 1 G of disk space with new IDS. This is a big help especially for small systems where spaces are expensive.
• Much faster backup and restore. The backup and restore speed of IDS 9.40 is also a lot faster than that of previous versions. The following table shows the comparison:

Table 1: Backup and restore comparison between old and new IDS

Time recorded in the table is the average of 5 repeated tests and the test is performed on an Informix instance that has 20 G of data.

Those new features on ontape are very helpful; we can use this new feature in our cutover process to reduce chances of Informix errors and to reduce system or Informix downtime. This will eventually improve customer satisfaction if it is based on system downtime. Cutover is a process of upgrading a products or rather software from release to release. Now let us have a close look at our system configuration and the current cutover process to see where we can make use of IDS 9.4 improved ontape utility to improve the process.

Overview of the system

The system is built on Sun Enterprise 3500 box with 3 G of Ram and 6 CPUs configured as follows:

System Configuration:  Sun Microsystems  sun4u 5-slot Sun Enterprise E3500
System clock frequency: 80 MHz
Memory size: 3072Mb
 
========================= CPUs =========================
 
                    Run   Ecache   CPU    CPU
Brd  CPU   Module   MHz     MB    Impl.   Mask
---  ---  -------  -----  ------  ------  ----
 5    10     0      400     8.0   US-II    10.0
 5    11     1      400     8.0   US-II    10.0
 7    14     0      400     8.0   US-II    10.0
 7    15     1      400     8.0   US-II    10.0
 9    18     0      400     8.0   US-II    10.0
 9    19     1      400     8.0   US-II    10.0
 
 
========================= Memory =========================
 
                                              Intrlv.  Intrlv.
Brd   Bank   MB    Status   Condition  Speed   Factor   With
---  -----  ----  -------  ----------  -----  -------  -------
 5     0    1024   Active      OK       60ns    2-way     A
 7     0    1024   Active      OK       60ns    2-way     A
 9     0    1024   Active      OK       60ns    1-way  

There are 8 disks as follows:

AVAILABLE DISK SELECTIONS:
       0. c0t0d0 SUN18G cyl 7506 alt 2 hd 19 sec 248
          /sbus@2,0/SUNW,socal@d,10000/sf@0,0/ssd@w2100002037c33d93,0
       1. c0t1d0 SUN18G cyl 7506 alt 2 hd 19 sec 248
          /sbus@2,0/SUNW,socal@d,10000/sf@0,0/ssd@w21000020376ec916,0
       2. c0t2d0 SUN18G cyl 7506 alt 2 hd 19 sec 248
          /sbus@2,0/SUNW,socal@d,10000/sf@0,0/ssd@w2100002037c340b4,0
       3. c0t3d0 SUN36G cyl 24620 alt 2 hd 27 sec 107
          /sbus@2,0/SUNW,socal@d,10000/sf@0,0/ssd@w21000004cf62336f,0
       4. c2t4d0 SUN18G cyl 7506 alt 2 hd 19 sec 248
          /sbus@6,0/SUNW,socal@d,10000/sf@0,0/ssd@w2100002037386b2d,0
       5. c2t5d0 SUN18G cyl 7506 alt 2 hd 19 sec 248
          /sbus@6,0/SUNW,socal@d,10000/sf@0,0/ssd@w2100002037605d59,0
       6. c2t6d0 SUN18G cyl 7506 alt 2 hd 19 sec 248
          /sbus@6,0/SUNW,socal@d,10000/sf@0,0/ssd@w210000203770dc2d,0
       7. c2t7d0 SUN36G cyl 24620 alt 2 hd 27 sec 107
          /sbus@6,0/SUNW,socal@d,10000/sf@0,0/ssd@w21000004cf623520,0

The configuration consists of 4 mirroring pairs on these 8 disks; all c2 devices above are mirrored to its corresponding c0 devices. Also configured, are 20 Informix links for the databases to use with 2 G for each dblink. Currently there is one Informix instance running with four databases, namely cell1, cell2 , cell31 and cell32. Of all four databases, cell1 is the major and also the largest database; its size is different depending on the customer, but on average it has more than 520 tables, 2000 indexes and 1500 stored procedures and has over 20 G of data.

Back to top

Overview of cutover process

As mentioned before, the cutover is the process of upgrading software from release to release. This software including two parts, the first are system files that include Unix system files and esqlc executables, and the other is Informix databases. Thus the ultimate goal of cutover is to upgrade system files and Informix databases without losing any current customer data. This is achieved by following four stages, namely hot platform, hot cutover, cold cutover and cold platform. At each stage different Unix and Informix database actions are performed.Below is the detailed description of each of these cutover stages:

• Hot Platform:
• Break disk mirrors into active and offline sides.
• Upgrade Unix file systems on offline side.
• Upgrade Informix server and client SDK versions and new esqlc executables on offline side.

At this stage, what we are doing is basically breaking the mirrors into two parts, the active side and the offline side. This way the customer can still do their work on the active side while the upgrade is being done to Unix and Informix on the offline side. This includes installing a new Unix operating system version and Informix server and client SDK versions as new esqlc executables.

• Hot Cutover:
• Export database cell1 for upgrade.
• Upgrade schema for cell1, including upgrading tables extent sizes and modifying old table structures.
• Import upgraded database cell1 back to Informix server, but rename it locked_cell1_db for further upgrade.
• Add new tables and stored procedures into the locked_cell1_db.

At this stage, we are upgrading our databases. The method used is exporting and importing the database, since there is only one Informix instance. Although you are only performing export and import operations on the cell1 database and the customer can still access other three databases, there is a big drawback. The export and import database took a long time because cell1 is the largest database and during exporting and importing, the customer can not access the database. This would add to the system downtime.

• Cold Cutover:
• Export other three databases for upgrade.
• Upgrade schema and add new tables and stored procedures.

At this stage, we are shutting down the system completely; customers are not allowed to access any databases. And since you used export, it took a long time to upgrade all three databases. This will certainly add to system downtime.

• Cold Platform:
• Boot machines to the offline side.
• Set up new Informix dbspace layout.
• Import all upgraded databases back into the Informix server.
• Resynchronize the mirrors.

It is obvious that at this stage, the customer can not access the system and Informix databases. This will also add to system downtime.
This process has two problems:

• Long system and database downtime. As you see there are a few database exports and imports during the whole cutover process. Those operations are lengthy but more importantly, the customer can not access the databases during the database export and import. Time used for export and import is different depending on the sizes of the database; it could take as long as 5 hours. When a database is being exported or imported, Informix will block the database exclusively for the operation, in other words, no users are permitted to use the database. In addition, in cold platform, you need to re-layout Informix or re-set all Informix chunks, and this would take 30 minutes to finish. So total system and database downtime during cutover could be as long as 6 hours.
• More chances for the Informix engine to crash or malfunctioning. Since there is only one Informix instance, all database upgrade operations need to be performed on this instance, such as export, import, adding tables and indexes, adding stored procedures and update statistics after tables and indexes being modified. All those heavy database operations will certainly put extra loads on this already heavily loaded instance and would easily cause some performance problems. Additionally, you do not have the freedom to bounce the instance if any critical problem such as long transaction or integrity violations occurs during the upgrade.

What can we do to improve this? Configure more Informix instances when upgrading. The idea is to perform all heavy database upgrades on another Informix instance while making the old Informix instance available to users. This would obviously reduce the loads on the old Informix instance and hence avoid some performance issues. In addition, with multiple Informix instances, you are able to perform binary backup and restore. This would reduce long system and database downtime caused by exporting and importing databases and with the new features on ontape utility in IDS 9.4, you should be able to speed up the whole cutover process. The new architecture of the cutover process after such improvements is as follows:

• Hot Platform:
• Break mirrors into active and offline sides.
• Upgrade Unix file systems on offline side.
• Upgrade Informix server and client SDK versions and new esqlc executables on offline side.
• Configure 2nd Informix instance.
• Do binary backup of old or original Informix instance and restore it on the 2nd Informix instance.

There is a step at this stage; to configure the second Informix instance and use it as a clone of the old Informix instance. You can only do this using the rename feature of IDS 9.4 ontape utility; otherwise, you can not use ontape to backup the database at one location and restore it at another.

• Hot and Cold Cutover:
• Export all databases on the 2nd Informix instance.
• Upgrade schemas of all databases.
• Upgrade dbspace layout for 2nd Informix instance.
• Import all upgraded databases back to the 2nd Informix instance with upgraded dbspace layout.
• Import newly changed data from corresponding database on the old Informix instance.
• Do binary backup for the 2nd Informix instance.

There are a lot of changes here. Hot Cutover and Cold Cutover are combined. Since there is a clone of the old Informix instance, all upgrades on the 2nd Informix instance can be done without even touching the old Informix instance. The customers can still use the old Informix instance to do their work. The 2nd Informix instance can be shut down and upgraded dbspace layout without affecting our customer. This reduces system and database downtime. After finishing the upgrade, pull out customer newly changed data dynamical from the corresponding database on the old Informix instance and finally make a ontape backup of what you did, so that during the Cold Platform, you can restore all database upgrades.

• Cold Platform:
• Reboot to offline side.
• Restore Informix from binary backup taken from Hot and Cold Cutover.

The major change at this stage is to use ontape restore rather than database import. This greatly reduces the system and database downtime. Tests will show that downtime is reduced from 6 hours to 15 minutes. This is major progress and will surely improve customer satisfaction.

But the prerequisite of this architecture change is IDS 9.40; it would be totally impossible with previous versions of IDS. Previous versions of IDS are all very strict about ontape utility in making backup and restore; all chunk path and chunk offsets must be exactly the same in backup and restore in other words, you can not make a backup at one physical location and restore it at another. This restriction is however lifted in IDS 9.40; with IDS 9.40.

Back to top

Conclusion

This new cutover process is suspected to significantly cut down on total system and database downtime, but there are two concerns:

• Are there enough system resources to configure new Informix instances?
• Could the new Informix instances produce any ill effects on the original instance in terms of performance?

Carefully evaluated these concerns and heavily test this new cutover process. As a result, you could cut down on total downtime (from 5 hours to 1 hour) without a reverse affect on performance resulting in improved customer satisfaction. Again, if your customer satisfaction is mainly based on system downtime. Without the IDS 9.40 new rename restore feature, you may not able to accomplish this architecture change and reduce your total downtime in a cutover process.

Here are some scripts that can be used for Informix backup and restore:

#!/usr/bin/ksh

. /hot_usr/informix/setinfx3env

#generate restore file for use in restore

sql='create table fan (name char(128), oset int);
insert into fan
select fname, offset*2
from syschunks;
unload to /hot_usr/informix/restorelist delimiter " "
select name, oset, name[1,1]||name[6,128],oset
from fan;
drop table fan;'

echo $sql > /var/tmp/f.sql

dbaccess sysmaster /var/tmp/f.sql 

#do ontape backup

echo '\n\n' | ontape -s -L 0

#!/usr/bin/ksh

oldon=/usr/informix/etc/onconfig

cp $oldon $oldon.cut

cat $oldon.cut | sed -e 's/^ROOTPATH.*$/ROOTPATH  
\/hot_usr\/informix\/dblink/' -e 's/^TAPEDEV.*$/TAPEDEV  
\/usr\/gsm\/ne_data\/infxback/' -e 
's/^TAPEBLK.*$/TAPEBLK  2048/' -e 's/^TAPESIZE.*$/TAPESIZE  
0/' > $oldon

echo "\ny\nn\nn\n" | ontape -r -rename -f /usr/informix/restorelist

#check informix state and change to multiple user mode

ok=`/usr/informix/bin/onstat - | grep Quiescent |wc -l`

while [ ${ok} -ne 1 ]
do
  sleep 10
  ok=`/usr/informix/bin/onstat - | grep Quiescent |wc -l`
done

onmode -m

Readers may wonder why I did not discuss onbar. Well, according to Informix, onbar has a better functionality than ontape; it can perform parallel backup and restore and that would be beneficial to overall performance. But onbar also requires a rather complicated setup and as one of the prerequisites it requires a volume manager or Informix storage manager to be setup. Compared with onbar, ontape is much simpler in setting up and much easier to use; ontape does not require storage volume manager and could either use tape or disk devices for backup and restore. With the new features in IDS 9.40, ontape becomes even more powerful.

In order to make full use of IDS 9.40 ontape rename new feature, we need to remember few things:

• There are two ways you can rename chunks; you can rename chunks on command line or you can rename it using a file. To rename chunks on command line, use following commands:

ontape -r -rename -p /chunk1_path -o 200000 -n /newchunk1 -o 100000
          -rename -p /chunk2_path -o 200000 -n /newchunk2 -o 100000

Where p and first o specifies the old chunk path and offset and n and second o specifies the new chunk path and offset. If you want to rename multiple chunks, it would be easier to use a file. Here is an example:

ontape -r -rename -f /usr/informix/restorelist

And here is the sample /usr/Informix/restorelist file:

/usr/informix/dblink 10000 /informix/dblink 10000
/usr/informix/dblink 140000 /informix/dblink 140000
/usr/informix/dblink 290000 /informix/dblink 290000
/usr/informix/dblink 1200000 /informix/dblink 1200000
/usr/informix/dblink1 0 /informix/dblink1 0
/usr/informix/dblink2 0 /informix/dblink2 0
/usr/informix/dblink2 1100000 /informix/dblink2 1100000
/usr/informix/dblink2 1600000 /informix/dblink2 1600000
/usr/informix/dblink3 0 /informix/dblink3 0
/usr/informix/dblink4 0 /informix/dblink4 0
/usr/informix/dblink20 0 /informix/dblink20 0
/usr/informix/dblink5 0 /informix/dblink5 0
/usr/informix/dblink5 500000 /informix/dblink5 500000
/usr/informix/dblink18 0 /informix/dblink18 0
/usr/informix/dblink18 400000 /informix/dblink18 400000
/usr/informix/dblink6 0 /informix/dblink6 0
/usr/informix/dblink19 0 /informix/dblink19 0
/usr/informix/dblink7 0 /informix/dblink7 0
/usr/informix/dblink8 0 /informix/dblink8 0
/usr/informix/dblink9 0 /informix/dblink9 0
/usr/informix/dblink9 1100000 /informix/dblink9 1100000
/usr/informix/dblink9 1600000 /informix/dblink9 1600000
/usr/informix/dblink9 1900000 /informix/dblink9 1900000
/usr/informix/dblink10 0 /informix/dblink10 0
/usr/informix/dblink11 0 /informix/dblink11 0
/usr/informix/dblink12 0 /informix/dblink12 0
/usr/informix/dblink15 0 /informix/dblink15 0
/usr/informix/dblink13 0 /informix/dblink13 0
/usr/informix/dblink13 100000 /informix/dblink13 100000
/usr/informix/dblink14 0 /informix/dblink14 0
/usr/informix/dblink17 0 /informix/dblink17 0
/usr/informix/dblink17 1100000 /informix/dblink17 1100000
/usr/informix/dblink17 1600000 /informix/dblink17 1600000

If you use this option, be sure to specify the whole path of the file after f so that IDS will not be confused about the file location.

• Make sure the new chunk path and old chunk path are not overlapping each other. If they are overlapped, then there would be data integrity and memory problems and new Informix instance may fail.
• Make sure to set ROOTPATH and ROOTOFFSET parameters in new Informix configuration file to the original ROOTPATH and ROOTOFFSET where the backup was taken. When you restore using rename option, IDS will first check and validate the old ROOTPATH and ROOTOFFSET and then change them to the new settings. If you forget to change those two parameters, IDS would not let you continue with restore.
• If you use rename option to restore your Informix instance to non-existing devices, you need to do warm or hot restore after you installed new devices, otherwise, you will suffer from data integrity problems and can not use the new Informix instance.

Resources

• IBM Informix Dynamic Server Administrator's Guide, Version 9.4.
  
  
• IBM Informix Dynamic Server Getting Started Guide, Version 9.4.
  
  
• IBM Informix Dynamic Server Performance's Guide, Version 9.4.
  
  
• IBM Informix Dynamic Server Administrator's Reference, Version 9.4.
  
  
• IBM Informix Dynamic Server Backup and Restore Guide, Version 9.4.
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Introduction
• Overview of cutover process
• Conclusion
• Resources
• About the author
• Comments

Next steps from IBM

• Try: Informix Ultimate Edition
• Buy: Informix Ultimate Edition

Dig deeper into Information Management on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers