Replacing a source container from backup in a replicated VOB

Answer

THESE INSTRUCTIONS ARE ONLY FOR REPLACING A CONTAINER IN A REPLICATED VOB FROM BACKUP.

• If you are replacing the container from a sibling replica use technote 1221900.
  
  
• If you are replacing the container from backup in a non-replicated VOB review technote 1226698.
  

Note: These procedures may NOT work in a UCM environment as they may result in metadata skew. This would be the case, for instance, if the container being used as the replacement is missing versions that were associated with a UCM activity or baseline. If that is the case, then you may need to restore the entire UCM component, or just remove the corrupt container as detailed in technote 1222072. The procedure for restoring a component from backup is documented in the IBM Rational ClearCase Administrator's Guide, under the section Restoring Critical ClearCase Data > Restoring One or More Members of a Set of Related Databases.

---

1. Dump the object (for example cleartool dump foo.c), and look for the source container and cleartext container for the version selected by the view.

Note: The source container will always be the same for all versions; however, the cleartext container may be different for each version (depending on the type manager used).

Example 1:

%>cleartool dump foo.c
foo.c (41fbe138.ed0511d2.b552.00:01:80:86:79:de)
/tmp/tag1/foo.c@@/main/2
oid=41fbe138.ed0511d2.b552.00:01:80:86:79:de dbid=36 (0x24)
... SNIP ...
cont dbid=536870958 container="16/2e/0-41fbe138ed0511d2b5520001808679de-k5"
source cont="/net/host1/tmp/tag1.vbs/s/sdft/16/2e/0-41fbe138ed0511d2b5520001808679de-k5"
clrtxt cont="/net/host1/tmp/tag1.vbs/c/cdft/2f/2b/41fbe138ed0511d2b5520001808679de"


Example 2:

%>cleartool dump bar.c
bar.c (7bdbe150.ed0711d2.b552.00:01:80:86:79:de)
/tmp/tag1/bar.c@@/main/2
oid=7bdbe150.ed0711d2.b552.00:01:80:86:79:de dbid=41 (0x29)
... SNIP ...
cont dbid=536870982 container="9/1b/2-7bdbe150ed0711d2b5520001808679de-d0"
source cont="/net/host1/tmp/tag1.vbs/s/sdft/9/1b/2-7bdbe150ed0711d2b5520001808679de-d0"
clrtxt cont="/net/host1/tmp/tag1.vbs/s/sdft/9/1b/2-7bdbe150ed0711d2b5520001808679de-d0"



2. Check that the source container is accessible. Navigate to the location indicated in the dump output and check to ensure there is the file in that location.

• Is the file there? If so, try opening it using a text editor, such as Notepad or vi, and verify the contents are appropriate; does the file contain text or binary characters? If it is truly a text file, then it should not contain any non-ascii characters.
• Is the source container empty and have zero length? If so go to step 4.
• Is the source container missing? If so go to step 5.

3. Run checkvob on the element. You will probably see 0 missing, 0 misprotected.

</tmp/tag1>cleartool checkvob foo.c
>The session's log directory is 'checkvob.12-Apr-99.11:45:04'.
>=================================================================
>Processing element "/tmp/tag1/foo.c@@".
>Checking status of 1 referenced containers in pool "s/sdft"...
>Initial container status: 0 missing, 0 misprotected.
>=================================================================



4. If the container is present, but corrupted, move the file to a new location, then rename the source container in the production VOB's storage.

Note: You will want to make note of the file name as you may need to rename the container that you restore from the backup VOB.



5. Run checkvob on the element, and make the output confirms the container is missing:

</tmp/tag1>cleartool checkvob foo.c
>The session's log directory is 'checkvob.12-Apr-99.11:45:45'.
>=================================================================
>Processing element "/tmp/tag1/foo.c@@".
>Checking status of 1 referenced containers in pool "s/sdft"...
> 16/2e/0-cafbe174f0f211d2b5520001808679de-c3 (missing)
>Initial container status: 1 missing, 0 misprotected.
>=================================================================



6. Determine which event was more recent:

1. A backup of the VOB
   or
2. MultiSite synchronization (multitool syncreplica -export)

You will take the replacement container from whichever is most recent and has the most data. It is safer to go with getting the container from the "most-recently-updated" replica since getting it from backup might cause "ndata" versions that other replicas do not consider "ndata".

Note: If you need to use a version from a different replica, STOP HERE. Refer to technote 1221900 for instructions.


Since you will be using a source container from a backup of the VOB storage, any versions created after the backup will be lost. If no versions were created since the backup, then you will be able to restore the entire element (all versions) without losing data.

Be aware that the backup's copy of the source container may also be corrupt. Use a backup that was made before the problem started to maximize the chance of finding a healthy source container.

The replacement source container must come from a “live” copy of the backup, because each the pathname of the source container changes with each new version added and you will need to use cleartool commands to determine which source container to use.

The “backup” VOB needs to be run without disturbing the live copy of the “production” VOB. This means it must be run in a separate ClearCase region from the “production” VOB. The only alternative is to unregister and un-tag the "production" VOB without removing the VOB itself.



7. Create or set a new region using cleartool mkregion <new_region_name>


WINDOWS INSTRUCTIONS:

1. Locate a system you can use to restore the backup VOB.
2. On that system, go to Start > Settings > Control Panel and double-click the ClearCase (Control Panel)
3. Go to the Registry tab.
4. Using the drop-down selection list for Windows NT Region, select an the new region.
5. If no other regions exist, then:
• Cancel out of the control panel
• On the command line, run cleartool mkregion -tag <newname>
• Repeat steps a-d and select the new region
6. Click OK to apply the changes
7. Use the ClearCase Control Panel or cleartool hostinfo –long on the command line to confirm the region change

UNIX & LINUX INSTRUCTIONS:

1. Locate a system you can use to restore the backup VOB
2. Modify the contents of /var/adm/atria/rgy/rgy_region.conf to reflect the alternate region name.
3. Stop and start ClearCase on the machine using /usr/atria/etc/atria_start –stop and /usr/atria/etc/atria_start –start, respectively
4. Confirm region change by running cleartool hostinfo –long

8. Follow the standard procedures for restoring a VOB from backup; see the note at the end of this technote. With your host set to the new ClearCase region you created, register and tag the VOB (using either the GUI or the commands cleartool register and cleartool mktag; for help run cleartool man <command>).



9. Mount the restored/backup VOB. On UNIX/Linux, you may need to create the mount-over directory to mount the newly restored VOB.



10. In the mounted copy of the restored/backup VOB, find the element that is missing a container in the production VOB.



11. Run cleartool dump <path to element> on the version of the element that has the corrupt source container in the production VOB and look at the output to determine the source container location for this element in the restored VOB. (The production VOB may have versions that the backup one lacks. In that case, use the latest version that the restored VOB has.)



12. Follow steps 2 through 5 to confirm that in the restored VOB the source container file exists, is of the appropriate size, and is not corrupted.

Because of the time difference between the creation of the backup and the incident which caused the problem, there is no way to recover any missing versions' contents.

For example, if the production VOB is looking for the container for foo.c and it expects there to be versions up to foo.c@@/main/3. However, this container is corrupt, and needs to be replaced with a version from the restored VOB, which only knows about versions up to foo.c@@/main/2. Because of the discrepancy, the contents for version foo.c@@/main/3 are lost unless there is another backup that may contain the later version.



13. Take the container from the restored VOB's storage and put it in the production VOB’s storage, with the same name as the container that you renamed in step 4, with a .1 at the end.

For example, if the corrupt local container were

16/2e/0-cafbe174f0f211d2b5520001808679de-c3

you would take the 0-41fbe138ed0511d2b5520001808679de-k5 container from the restored VOB's storage and put it in the production VOB's storage (directory 16/2e) with the name 16/2e/0-cafbe174f0f211d2b5520001808679de-c3.1  

Renaming the restored VOB's container to the expected production VOB's container (+ .1 suffix) causes the checkvob tool to recognize the restored VOB's container as a candidate for fixing the missing newer container. Without this step, checkvob may see that there is a missing container, but not find any candidates it considers useful in repairing the problem.



14. Run cleartool checkvob -fix <filename>

The checkvob utility should find what it calls an alternate related container, report which versions are found in the alternate (or which it cannot find), and ask whether it should update the VOB database. The missing versions (/main/3 in the above example) should be marked as ndata meaning that the version containers will become empty place holders.

Note: You should only allow checkvob to mark a container as ndata if the alternate related container is no longer available. THIS WILL PERMANENTLY REMOVE THESE VERSIONS except as a place holder, so only do this if the alternate source container would not include them, such as mentioned above. Verify if the versions in question are truly missing at the remote site where you got the alternate related container, or if they are accessible at the remote site, as the only versions you should be setting 'ndata' are ones that do not exist at the remote site at all.

</tmp/tag1>cleartool checkvob -fix foo.c
>The session's log directory is 'checkvob.12-Apr-99.12:18:23'.
>======================================
>Processing element "/tmp/tag1/foo.c@@".
>The element is now locked.
>Checking status of 1 referenced containers in pool "s/sdft"...
>    16/2e/0-cafbe174f0f211d2b5520001808679de-c3 (missing)
>Initial container status: 1 missing, 0 misprotected.
>Fixing element with 1 missing containers (out of 1)...
>Found 1 alternate related containers.
>Constructing new containers...
>.
>
>Unable to find the data (cleartext) of 1 versions (out of 4):
>    12-Apr-99.12:07:23 /main/3
>0 of these versions are "interesting" (i.e. have labels,
>sub-branches, attributes or hyperlinks).
>
>Do users want to continue and update the VOB database
>(including irreversibly making 1 versions 'ndata')? [no] y
>
>OK. Updating the VOB database...
>Cleaning up unused containers...
>Moving unreferenced containers to the pool's lost and found area...
>.
>The VOB database has been updated with the fixes.
>Final container status: 0 missing, 0 misprotected.
>1 new 'ndata' versions created.
>The element is now unlocked.
>======================================

---

If checkvob -fix fails with the following error:

cleartool: Error: Type manager "text_file_delta" failed get_cont_info operation.
cleartool: Error: Unable to process container

Review technote 1259721 for the corresponding explaination and workaround to the problem.

---

At this point, foo.c@@/main/3 is ndata, but now the production VOB has most of the versions restored.

Note: There is no way to populate ndata versions. Also, in a case like this, where foo.c@@/main/3 is the LATEST version on a branch, an attempt to cleartool rmver /main/3 and then check out and check in to recreate it will result in the creation of /main/4 since removed versions cannot be recreated.

The procedure for restoring a single element from backup is documented in the IBM Rational ClearCase Administrator's Guide, under the section Restoring Critical ClearCase Data.