DB2 Advanced Copy Services: The scripted interface for DB2 Advanced Copy Services, Part 2

Implement the scripted interface for DB2 ACS using Linux LVM

The DB2® Advanced Copy Services supports taking snapshots for backup purposes in DB2 for Linux®, UNIX® and Windows® databases. Customers can use the DB2 ACS API either through libraries implemented by their storage hardware vendors or implement this API on their own. Additionally, it requires a high degree of effort for customers to implement. This changes with IBM DB2 10.5.

Share:

Joern Klauke (joern.klauke@de.ibm.com), Staff Development Engineer, IBM Deutschland Research and Development GmbH

Joern KlaukeJoern Klauke works as a software developer for SAP on DB2 for Linux, UNIX, and Windows at the IBM Research and Development Lab in Boeblingen (Germany). He has five years of experience with IBM and DB2 LUW assisting customers with best practices, problem analysis, and troubleshooting. He holds a degree in computer science from the Martin-Luther-University of Halle (Germany).



Martin Jungfer (martin.jungfer@de.ibm.com), Software Engineer, IBM

Martin                 Jungfer photoMartin Jungfer has many years of experience and deep knowledge of administrating DB2 for Linux, UNIX, and Windows in SAP environments.



10 October 2013

DB2 10.5 introduces a new feature called scripted interface for DB2 Advanced Copy Services (DB2 ACS). This makes it possible for customers to implement shell scripts instead of C-libraries. These scripts can use the tools provided by the storage vendors to run the snapshot operations. You can use the scripted interface independent from your storage hardware. Additionally, DB2 supports storage hardware as soon as it is available in the market.

The feature supports all three architectures of DB2: enterprise server, multipartitioned databases using the database partitioning feature (DPF), and databases using pureScale. It is supported on all UNIX and Linux platforms where DB2 is certified on.

This series provides an introduction to this feature and presents some real-life examples in the coming articles. This second article in the series demonstrates the scripted interface with the use of Linux LVM.

LVM is the Logical Volume Manager for the Linux operating system. Among other features, it supports snapshots of logical volumes. This article walks through snapshot backups of DB2 databases with the help of the scripted interface for DB2 ACS.

Introduction to the Linux LVM

The Linux LVM is a Linux kernel extension that introduces an abstraction layer over storage hardware. This increases the flexibility of storage management; for example, it eases the resizing of partitions. Additionally, the LVM allows taking snapshots of partitions in LVM volume groups.

This article uses the layout shown in Figure 1:

Figure 1. Logical volume groups
Logical volume groups

In Figure 1, you can see the physical volume /dev/sdb. The LVM volume group data takes up the complete physical volume. The volume group data contains the logical volumes joern, data1, data2, and logs. There is still free space in this volume group that will be used to take the snapshots of the logical volumes. The logical volumes are formatted with the ext4 file system.

The logical volumes are mounted as shown in Table 1:

Table 1. Logical volumes that are mounted
Logical volumeMount pointUsed for
/dev/data/joern /db2/joern Database directory
/dev/data/data1 /db2/data1 Automatic storage path 1
/dev/data/data2 /db2/data2 Automatic storage path 2
/dev/data/logs /db2/logs Database logs

Table 2 provides a short overview of the commands for the LVM utilities that are mentioned in this article, and it indicates what the commands are used for:

Table 2. Commands overview
CommandUse
lvdisplay Display information on logical volumes. In particular, the option –c is used to get a colon-separated list of information, including size. For an overview of which columns are included, take a look at the man page.
lvcreate Create logical volumes in a volume group. This command is also used to create snapshot volumes for the logical volumes containing the storage paths and the log paths.
lvconvert Merge the snapshots back into the logical volumes that contain storage and log paths; that is, it can be used for restoring the snapshots.
lvchange Change attributes of a logical volume. In certain cases, it is necessary to deactivate and activate logical volumes to complete merges successfully.
lvremove Remove logical volumes, used during the deletion of images.
vgdisplay Query information of volume groups. In particular, the option –c is used to get a colon-separated list of information, including size. For an overview of which columns are included, take a look at the man page.

The LVM snapshots are created using the lvcreate command and the option –s. LVM works with copy on write; that is, when a block is first changed on the original volume, an image of the original contents of the block is written to the snapshot. At the end, the snapshot contains only the blocks that were changed in the original volume but with the contents that these blocks had during the point in time the snapshot was taken. This can be a small amount of data. But because we also want to be able to restore dropped databases, we recommend keeping the size of the snapshot logical volume equal to the original volumes. If an LVM snapshot volume runs out of space, Linux stops writing to it, which in turn corrupts the snapshot backup image.

Getting the right privileges

A customer script that is invoked by the scripted interface runs with the privileges of the instance owner. Typically, this user is not allowed to run commands that change the LVM configuration.

In this article, the privileges to run the commands were granted using sudo. The commands were added to the sudo configuration with the use of the command visudo adding lines like the following for each used command:

jklauke ALL=NOPASSWD: /sbin/lvdisplay

With this, the user no longer needs to enter the root password when the command is used.

For other platforms, there might be other solutions; for example, adding the instance owner to groups that are allowed to run the appropriate commands. We discourage using scripts with the setuid bit set.

Using LVM for snapshot backups using the scripted interface

This section provides a sample implementation of a customer script using the abilities of Linux LVM to create, use, and delete snapshots.

The following loop is needed frequently in the customer script:

110    for i in `grep "^DATAPATH" $config | \
111              awk -F= '{print $2}' | \
112              xargs -I\{\} df \{\} | \
113              grep '^/dev' | \
114              awk '{print $1;}' | \
115              uniq `
116    do

This gives you the set of logical volumes that need to be backed up during the snapshot. The command performs the following steps:

  1. Takes the list of the paths from the protocol file. In this example, the data path that is returned as a result from line 110 is:
    DATAPATH=/db2/joern
  2. Separates the path names from the option names (take from DATAPATH=path only the part after the equal sign (line 111):
    /db2/joern
  3. Translates the path to the logical volume on which the path resides (line 112):
    Filesystem 1K-blocks Used Available
    Use% Mounted on/dev/mapper/data-joern 2064208 268980 1690372 14% /db2/joern
  4. Takes only the lines that contain the logical volume names (line 113):
    /dev/mapper/data-joern 2064208 268980 1690372
    14% /db2/joern
  5. Takes only the name of the logical volume (line 114):
    /dev/mapper/data-joern
  6. Takes each logical volume only once (line 115):
    /dev/mapper/data-joern
    (In this case, this is trivial because there is only one logical volume.)

Backup

The rest of this article explains the operations backup, restore, and delete in detail. The actions of the operations were already introduced in the first part of this series of articles. In the case of backup, the actions are: prepare, snapshot, verify, and, depending on the result of the verify call, storemetadata or rollback.

Again, we recommend sizing the snapshot volumes using the same size as the original volume.

Prepare

The goal of the prepare action in the LVM script is to determine whether there is enough free space in each volume group, so that every logical volume can be backed up in its volume group. For this purpose, the function doPrepare() was implemented, shown in Listing 1:

Listing 1. doPrepare() function code
65 doPrepare() {
66 #
67 # P R E P A R E
68 #
69 # --------------------------------------
70    getSetting "OPERATION"
71    operation=$_setting
72    if [ $operation = "SNAPSHOT" ]
73    then
74       for group in `sudo vgdisplay -c | awk -F: '{print $1}'`
75       do
76          freespace=`sudo vgdisplay -c $group | \
77                     awk -F":" '{print $16}'`
78          cmd="egrep '^DATAPATH|^LOGPATH' $config | \
79               awk -F= '{print \$2}' | \
80               xargs -I\{\} df \{\} | \
81               grep '^/dev' | \
82               awk '{print \$1}' | \
83               uniq | \
84               xargs -I\{\} sudo lvdisplay -c \{\} | \
85               awk -F: -v c=$group '\$2==c { SUM += \$8 } END {print SUM}'"
86
87          echo "# cmd: "$cmd
88          neededspace=`eval $cmd`
89
90          if [ $neededspace -gt $freespace ]
91          then
92             echo "# Goup: " $group
93             echo "# Freespace: " $freespace
94             echo "# Needed Space: " $neededspace
95             RC=$RC_NOT_ENOUGH_SPACE
96             break
97          fi
98       done
99    fi
100 # --------------------------------------
101 }

The function doPrepare() performs the following steps:

  1. Determines whether the snapshot operation is currently run. Therefore, the option OPERATION is read from the protocol file, and it is checked whether the current operation is the snapshot operation (lines 70 – 73).
  2. Checks each volume group; for every volume group in vgdisplay, takes the name of the volume group and separates the name from the given string (line 74).
  3. Determines the free space for the current volume group: takes the vgdisplay output and parses only the 16th column (line 76 and 77).
  4. Looks for all paths of the database that reside on this volume group. At first, the logical volumes are determined as shown in the previous section (lines 78 – 83).
  5. For each of the logical volumes, takes the information of lvdisplay (line 84).
  6. Sums up the needed space for all logical volumes that reside in the current volume group (line 85).
  7. Writes the command to the protocol file for debugging purposes and runs the command (lines 87 and 88).
  8. Checks if the needed space for the snapshot in this volume group is smaller than the free space. If not, returns an error (lines 90 to 97).

Snapshot

Now that we are sure that we have enough space, we take a snapshot using the function doSnapshot(), shown in Listing 2:

Listing 2. doSnapshot() function code
103 doSnapshot() {
104 #
105 # S N A P S H O T
106 #
107 # --------------------------------------
108    getSetting "TIMESTAMP"
109    timestamp=$_setting
110    for i in `grep "^DATAPATH" $config | \
111              awk -F= '{print $2}' | \
112              xargs -I\{\} df \{\} | \
113              grep '^/dev' | \
114              awk '{print $1;}' | \
115              uniq `
116    do
117       vol=`sudo lvdisplay -c $i | awk -F: '{print $1;}'| tr -d ' '`
118       echo "USER_VOLUME_DATA="$vol
119       snapName=`basename $vol`"_snap_"$timestamp
120       sudo lvcreate -s -n $snapName -l100%ORIGIN $vol
121    done
122    getSetting "DB2BACKUP_LOGS"
123    includeLogs=$_setting
124    if [ $includeLogs = "INCLUDE" -a $RC -eq 0 ]
125    then
126       for i in `grep "^LOGPATH" $config | \
127                 awk -F= '{print $2}' | \
128                 xargs -I\{\} df \{\} | \
129                 grep '^/dev' | \
130                 awk '{print $1;}' | \
131                 uniq `
132       do
133          vol=`sudo lvdisplay -c $i | awk -F: '{print $1;}'| tr -d ' '`
134          echo "USER_VOLUME_LOG="$vol
135          snapName=`basename $vol`"_snap_"$timestamp
136          sudo lvcreate -s -n $snapName -l100%ORIGIN $vol
137       done
138    fi
139 # --------------------------------------
140 }
  1. Takes every logical volume that stores data in the first loop that follows the standard command (lines 110 to 116), after having determined the timestamp of the image (lines 108 and 109).
  2. Translates the device mapper to the real name of the logical volume (line 117).
  3. For an easier restore, writes the logical volume that will be the snapshot to the protocol file (line 118).
  4. Generates the name of the new snapshot by appending the timestamp to the name of the original logical volume (line 119).
  5. Runs the snapshot (line 120).

The options of this command are the following:

  • -s: makes a snapshot of the logical volume $vol
  • -n: provides the name of the new logical volume
  • -l100%ORIGIN: sizes the new logical volume with the same size as the original volume.

In the previous steps, the snapshots of the data logical volumes were taken. Now we need to take the snapshots of the log volumes. First determine whether this is necessary (lines 122 to 125).

The following lines do the same for log volumes as was done for the data volumes: determines the logical volumes, stores the logical volumes with the prefix USER_VOLUME_LOG for restore purposes, and runs the snapshot (lines 126 to 137).

Verify

Listing 3. doVerify() function code
238 doVerify() {
239 #
240 # V E R I F Y
241 #
242 # --------------------------------------
243    mkdir /tmp/verify
244    getSetting "TIMESTAMP" "" $oldConfig
245    timestamp=$_setting
246    for i in `grep "^USER_VOLUME_DATA" $config | \
247              awk -F= '{print $2}'`
248    do
249       vol=$i"_snap_"$timestamp
250       sudo mount $vol /tmp/verify
251       $RC=$?
252       sudo umount /tmp/verify
253       if [ $RC -neq 0 ]
254       then
255          echo "# Mounting of $vol failed"
256          break
257       fi
258       echo "# Volume $i checked"
259    done
260    getSetting "DB2BACKUP_LOGS"
261    includeLogs=$_setting
262    if [ $includeLogs = "INCLUDE" -a $RC -eq 0 ]
263    then
264       for i in `grep "^USER_VOLUME_LOG" $config | \
265                 awk -F= '{print $2}'`
266       do
267          vol=$i"_snap_"$timestamp
268          sudo mount $vol /tmp/verify
269          $RC=$?
270          sudo umount /tmp/verify
271          if [ $RC -neq 0 ]
272          then
273             echo "# Mounting of $vol failed"
274             break
275          fi
276          echo "# Volume $i checked"
277       done
278    fi
279    rmdir /tmp/verify
280 # --------------------------------------
281 }

To find out whether the snapshot was taken successfully, the script tries to mount the logical volume of every snapshot, checks if this was successful, and unmounts it again. In detail, the script runs the following steps:

  1. Creates a temporary directory that works as the mount point in the directory structure of the system (line 243).
  2. Reads the timestamp needed to generate the snapshot name from the protocol file (lines 244 and 254).
  3. For every data volume in the protocol file (lines 246 to 248), generates the name of the snapshot volume (line 249) and tries to mount it to the temporary directory (line 250).
  4. Checks if the mounting was successful (line 251).
  5. Unmounts the volume (line 252).
  6. If mounting the volume failed, returns an error message (lines 253 to 257).

If log files had to be included, the script has to check whether the snapshot of these volumes was successful as follows:

  1. Reads the option for including logs from the protocol file (lines 260 and 261).
  2. Checks if logs had to be included and if the mounting of the data volumes was successful (line 262).
  3. For every log volume in the protocol file (lines 264 and 265), generates the name of the snapshot volume (line 267)
  4. Tries to mount the volume (line 268)
  5. Tracks the success of mounting the volume (line 269) and unmounts it (line 270).
  6. Checks the result of mounting and returns an error message if it failed (lines 271 to 275).

At the end, the temporary directory is removed (line 280).

Rollback

If an error occurred during the verify action, that is, if at least one of the snapshot volumes could not be mounted, the next action to be called is rollback. This is implemented as shown in Listing 4:

Listing 4. doRollback() function code
295 doRollback() {
296 #
297 #  R O L L B A C K
298 #
299 # --------------------------------------
300    getSetting "OPERATION"
301    operation=$_setting
302    if [ $operation = "SNAPSHOT" ]
303    then
304       for i in `grep "^VOLUME_DATA" $config | \
305                 awk -F= '{print $2}'`
306       do
307          sudo lvremove $i_snap_$timestamp
308       done
309       getSetting "DB2BACKUP_LOGS"
310       includeLogs=$_setting
311       if [ $includeLogs = "INCLUDE" -a $RC -eq 0 ]
312       then
313          for i in `grep "^VOLUME_LOG" $config | \
314                    awk -F= '{print $2}'`
315          do
316             sudo lvremove $i_snap_$timestamp
317          done
318       fi
319    fi
320 # --------------------------------------
321 }

Essentially, the rollback deletes all snapshot volumes that should exist by running the following steps:

  1. Checks whether this is a rollback of a snapshot operation (lines 300 to 303).
  2. Loops across all data volumes given in the protocol file (lines 304 to 306).
  3. Removes the logical volume (line 307) with lvremove. This lvremove command takes only the volume name as a parameter.
  4. Checks whether there are snapshot volumes of the logs (lines 309 to 312).
  5. Loops across the log volumes and removes them (line 316).

Storemetadata

Storemetadata does not run any commands. Storemetadata could save the protocol file to any backup infrastructure. As already stated in the first article, the protocol file is essential to be able to restore the image. At the point when storemetadata is called, no further required information is added to the protocol file anymore; that is, it can be used in this state.

Restore

The purpose of the restore is obvious: It merges the snapshot volumes back to the original volumes; in other words, in the original volume, it replaces every changed block with the original block from the snapshot. Listing 5 shows the implementation:

Listing 5. doRestore() function code
142 doRestore() {
143 #
144 # R E S T O R E
145 #
146 # --------------------------------------
147
148    getSetting "OBJ_ID"
149    id=$_setting
150    # Construct key to search for in currenct protocol file
151    key="RESULT_"$id"_FILE"
152    getSetting $key
153    oldConfig=$_setting
154
155    getSetting "TIMESTAMP" "" $oldConfig
156    timestamp=$_setting
157    for i in `grep "^USER_VOLUME_DATA" $oldConfig | \
158              awk -F= '{print $2}'`
159    do
160       vol=$i"_snap_"$timestamp
161       echo "# Unmounting volume $vol"
162       sudo umount -f $i
163       echo "# Merging volume $vol"
164       sudo lvconvert --merge --background $vol
165       if [ $? -neq 0 ]
166       then
167          echo "# Deactivating volume $vol"
168          sudo lvchange -an $i
169          echo "# Activating volume $vol"
170          sudo lvchange -ay $i
171       fi
172       echo "# Mounting volume $vol"
173       sudo mount $i
174       echo "# Take the backup of volume $vol again"
175       sudo lvcreate -s -n $vol -l100%ORIGIN $i
176    done
177    # if logs included
178    getSetting "ACTION"
179    readAction=$_setting
180    if [ $readAction = "DB2ACS_ACTION_READ_BY_OBJECT" ]
181    then
182       for i in `grep "^USER_VOLUME_LOG" $oldConfig | \
183                 awk -F= '{print $2}'`
184       do
185          vol=$i"_snap_"${timestamp}
186          echo "# Umounting volume $vol"
187          sudo umount -f $i
188          echo "# Merging volume $vol"
189          sudo lvconvert --merge --background $vol
190          if [ $? -neq 0 ]
191          then
192             echo "# Deactivating volume $vol"
193             sudo lvchange -an $i
194             echo "# Activating volume $vol"
195             sudo lvchange -ay $i
196          fi
197          echo "# Mounting volume $vol"
198          sudo mount $i
199          echo "# Take the backup of volume $vol again"
200          sudo lvcreate -s -n $vol -l100%ORIGIN $i
201       done
202    fi
203 # --------------------------------------
204 }

During the restore, there are two protocol files open: the protocol file of the current restore operation that is still changed and the protocol file of the snapshot backup operation, opened read-only. The second protocol file must be opened first. To do this, the object ID is read from the restore protocol file (lines 148 and 149), the option name is generated (line 151), and the name of the backup protocol file is read (lines 152 and 153). Now, the restore from this protocol file and the corresponding image is performed as follows:

  1. Reads the timestamp needed to generate logical volume names from the protocol file (lines 155 and 156).
  2. Loops across every data volume that can be found in the protocol file (lines 157 to 176).
  3. Generates the snapshot volume name (line 160).
  4. Unmounts the original logical volume (line 162).
  5. Merges the snapshot volume (line 164).
  6. If this was not successful, deactivates and activates the volume again (lines 165 to 171).
  7. Mounts the merged original volume (line 173).
  8. Because merging removes the snapshot volume, it is necessary to recreate the snapshot volume immediately (line 175).

The log volumes are handled as follows:

  1. Reads the action option from the restore protocol file (line 178 and 179). Depending on this option, the log volumes may have to be restored. If its value is DB2ACS_ACTION_READ_BY_OBJECT, the restore is needed. If its value is DB2ACS_ACTION_READ_BY_GROUP, it is not needed.
  2. In the following steps, the log volumes are restored using the same steps as for restoring the data volumes (lines 182 to 201).

Delete

The action delete, shown in Listing 6, removes the snapshot volumes from the corresponding volume groups by running the following steps:

Listing 6. doDelete() function code
206 doDelete() {
207 #
208 # D E L E T E
209 #
210 # --------------------------------------
211    getSetting "RESULT_"${objectId}"_FILE"
212    oldConfig=$_setting
213    getSetting "TIMESTAMP" "" $oldConfig
214    timestamp=$_setting
215    for i in `grep "^USER_VOLUME_DATA" $oldConfig | \
216              awk -F= '{print $2}'`
217    do
218       vol=$i"_snap_"${timestamp}
219       echo "# Volume $vol"
220       echo "# "`sudo lvremove -f $vol`
221    done
222    getSetting "DB2BACKUP_LOGS" "" $oldConfig
223    includeLogs=$_setting
224    if [ $includeLogs = "INCLUDE" ]
225    then
226       for i in `grep "^USER_VOLUME_LOG" $oldConfig | \
227                 awk -F= '{print $2}'`
228       do
229          vol=$i"_snap_"${timestamp}
230          echo "# Volume $vol"
231          echo "# "`sudo lvremove -f $vol`
232       done
233    fi
235 # --------------------------------------
236 }
  1. Retrieves the object ID from the command given by –o.
  2. Generates the name of the option with this object ID and retrieves the name of the backup protocol file (line 211).
  3. Reads this file (line 212).
  4. Now, the timestamp can be read that is needed to generate the names of the logical volumes of the snapshots (lines 213 and 214).
  5. Loops over all data volumes (lines 215 to 221), generates the name of the snapshot volume (line 218), and removes the volume (line 220).
  6. Checks if the backup image contains log files; that is, checks if the value of DB2BACKUP_LOGS is set to INCLUDE in the backup protocol file.
  7. If so, removes these snapshot volumes by generating their names (line 229) and by removing them (line 231) in the loop (lines 226 to 232).

Conclusion

This article demonstrates the use of LVM to create snapshot images of DB2 databases. It explains all actions of all operations in detail. It should be clear that the implementation of customer scripts using the scripted interface is easy to handle.


Download

DescriptionNameSize
Script for downloaddm-1310acs_lvm.sh---

Resources

Learn

Discuss

  • Get involved in the My developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Information management on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Information Management
ArticleID=947822
ArticleTitle=DB2 Advanced Copy Services: The scripted interface for DB2 Advanced Copy Services, Part 2
publish-date=10102013