Image Services: Improvements to the SCAVENGE option in CSM_tool

Symptom

The CSM_tool Scavenge option

In the previous version of CSM_tool the System Administrator was able to run Scavenge while users were logged on to the FileNet Image Services software resulting in lower performance. Scavenge cannot be interrupted once it is started and must run to completion. If Scavenge is aborted while it is running, cache becomes corrupted.

Now, when Scavenge is run with the FileNet Image Services software fully operational, the following messages are displayed on the screen.

<CSM_tool>SCavenge
IS software should be stopped before running scavenge.
Bring IS down, and then bring up in maintenance mode.

The new procedure to run Scavenge is to fully stop the FileNet Images Services software, restart the databases and then to run CSM_tool.

initfnsw -y stop
killfnsw -ADy
fn_util startdb
CSM_tool
CSM_tool> allowupdates
CSM_tool> Enter allowupdates password
CSM_tool> SCavenge

Before Scavenge starts, a warning message is now displayed.

<CSM_tool>SCavenge
***WARNING***
Cache Access will not be allowed until scavenge has completed.
If scavenge finds cache corruption, a transient database
initialization will be required.

There was no record that the Scavenge command was run. Scavenge now writes a message to the FileNet Image Services error log at the time it starts and another message at the time it stops.

2013/04/09 09:08:59.696 77,1,68 <fnsw> CSM_tool (8305.1.9 0x2071.1) ...
CSMl(8305): Cache Scavenge Starting.

2013/04/09 09:11:47.982 77,1,68 <fnsw> CSM_tool (8305.1.9 0x2071.1) ...
CSMl(8305): Scavenge finished successfully.

In the past when Scavenge found an overlap condition in cache it would report the error and immediately stop processing. An overlap condition occurs when the csm_used_space and cms_free_space tables contain the same cache address. The FileNet Image Services software could not be restarted until the overlap issue was resolved by initializing the transient database.

The Scavenge option in CSM_tool has been enhanced to continue processing and report the all of cache addresses with an overlap condition. The bad cache addresses are displayed on the screen and written in the FileNet Image Services error log.

CSM_tool
CSM_tool> allowupdates
<CSM_tool>SCavenge
***WARNING***
Cache Access will not be allowed until scavenge has completed.
If scavenge finds cache corruption, a transient database
initialization will be required.

Do you want to continue (y/n)?y
Deleting csm_free_space. Number of rows deleted:
56
Refreshing cache used space. Number of objects refreshed:
721544 (of 721824 or 99.97%) done.
Overlapping objects found, ssn=10312008, obj_id=913579, page=1
Cache start: partition, offset: 1, 966062 (0xebdae) size: 3
721825 (of 721824 or 100.01%) done.
Current stats shows a total of 721824 objects in cache.
Scanning all cache used space. Number of objects scanned:
721825

New stats determined from offline scan:
cache inuse_objs inuse_secs locked_objs locked_secs
----- ---------- ---------- ----------- -----------
1 721825 1118405 110 300848
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
Scavenge completed with 1 Overlap object(s) found.

Image Services Scavenge error log entries:

2013/04/09 10:41:12.549 77,1,68 <fnsw> CSM_tool (10667.1.21 0x29ab.1)
CSMl(10667): Cache Scavenge Starting.

2013/04/09 10:42:37.888 77,1,69 <fnsw> CSM_tool (10667.1.21 0x29ab.1)
CSMl(10667): Overlapping objects found, ssn=10322008, obj_id=913579, page=1
Cache start: partition, offset: 1, 966062 (0xebdae) size: 3

2013/04/09 10:42:56.347 77,1,69 <fnsw> CSM_tool (10667.1.21 0x29ab.1)
CSMl(10667): Scavenge completed with 1 Overlap object(s) found.

After Scavenge completes, the FileNet Image Services software can be restarted.

initfnsw -y stop
killfnsw -ADy
initfnsw start

The CSM_tool FINdused option

A new option has been added to CSM_tool that allows the System Administrator to locate all of the documents that could be impacted when an overlap condition is reported. The new option is named FINdused and is able to be accessed after entering the allowupdates password in CSM_tool.

The parameters entered are the partition (cache id), offset and block count. All three items are displayed in the CSM_tool Scavenge output and written to the FileNet Image Services system error log. The System Administrator should run the FINdused command on every error reported by Scavenge.

CSM_tool
CSM_tool> allowupdates
CSM_tool> Enter allowupdates password
<CSM_tool>FINdused 1 966062 3

Scanning cache used space. Number of objects scanned:
204828 (of 721825 or 28.38%) done.
Overlapping objects found, ssn=10312008, obj_id=910823, page=0
Cache start: partition, offset: 1, 966062 (0xebdae) size: 1

Overlapping objects found, ssn=10312008, obj_id=910823, page=1
Cache start: partition, offset: 1, 966063 (0xebdaf) size: 1
721544 (of 721825 or 99.97%) done.
Overlapping objects found, ssn=10322008, obj_id=913579, page=1
Cache start: partition, offset: 1, 966062 (0xebdae) size: 3
721825 (of 721825 or 100.00%) done.
Find Duplicate completed with 3 Overlap object(s) found.

Image Services Finused error log entries:

2013/04/09 10:44:32.918 77,1,68 <fnsw> CSM_tool (10667.1.21 0x29ab.1)
CSMl(10667): Cache Duplicate Find Starting.

2013/04/09 10:44:37.705 77,1,69 <fnsw> CSM_tool (10667.1.21 0x29ab.1)
CSMl(10667): Overlapping objects found, ssn=10312008, obj_id=910823, page=0
Cache start: partition, offset: 1, 966062 (0xebdae) size: 1

2013/04/09 10:44:37.705 77,1,69 <fnsw> CSM_tool (10667.1.21 0x29ab.1)
CSMl(10667): Overlapping objects found, ssn=10312008, obj_id=910823, page=1
Cache start: partition, offset: 1, 966063 (0xebdaf) size: 1

2013/04/09 10:44:49.777 77,1,69 <fnsw> CSM_tool (10667.1.21 0x29ab.1)
CSMl(10667): Overlapping objects found, ssn=10312008, obj_id=913579, page=1
Cache start: partition, offset: 1, 966062 (0xebdae) size: 3

2013/04/09 10:44:49.778 77,1,69 <fnsw> CSM_tool (10667.1.21 0x29ab.1)
CSMl(10667): Find Duplicate completed with 3 Overlap object(s) found.

The System Administrator can then analyze the documents that are reported to see if they became corrupted due to the cache overlap condition by attempting to view or retrieve them. If they are found to be corrupt, the System Administrator can delete the objects from cache by using the MKF_tool command. The following is an example of deleting objects from the csm_used_space table.

MKF_tool
MKF_tool> allowupdated
MKF_tool> Enter allowupdates password
MKF_tool> delete csm_used_space object=1:10312008:910823:0
MKF_tool> delete csm_used_space object=1:10312008:910823:1
MKF_tool> delete csm_used_space object=1:10312008:913579:0
MKF_tool> quit

If there are numerous overlap errors reported in the csm_used_space table, it is recommended to initialize the transient database to recover.