Moving Tealeaf Canister Data and Indexes

Answer

Before you start

The simplest scenario is to move both data and indexes from one drive to another. Alternatively, data and indexes can each be moved to a different drive or directory. The spool directory location can also be modified. You cannot span canister data, indexes, or the spool across multiple drives or folders. All canister data must reside in a single location, and the same applies to indexes and spool data.

Note: changing file locations is not the only way to expand storage space for session data. A disk imaging or copying tool could be used to transfer all data to a larger drive which could then be assigned the same drive letter as the original within Windows. This type of approach does not require any configurations changes in Tealeaf, as the paths do not change at the OS level.

Space considerations

Canister data typically consumes about twice the storage index data does. Average daily storage requirements for each can be estimated from the system status storage report in the Tealeaf Portal. The spool directory is where incoming hits will accumulate in the event the canister is unavailable (stopped for maintenance, crashed, or overloaded). Hits accumulating in the spool are uncompressed, so consume more space that the same data does once it is archived in the canister data directories. Ideally there should be enough spool space to store at least 3 days of traffic, to cover a weekend outage event.

Disk I/O considerations

The canister is disk I/O intensive: it generates a lot of reads and writes to disk. If spooling is occurring during canister operations (e.g. during a traffic spike) the problem is compounded: hits are being written to the spool, read from the spool, and then written to disk again by the canister. To avoid a bottleneck in disk I/O bandwidth, consider putting the spool on a different physical disk to the canister data and indexes. You can also put the canister data and indexes on two separate disks, further spreading the load. Using separate physical disks is not a requirement. It is just one way to increase available disk I/O bandwidth.

Timing

There will be spooling during the operation to move canister or index data as the canister services must be stopped whilst the data is being moved. Scheduling the change during a period of low traffic is recommended.

Create new directories

Create directories for the Canister data, indexes, FilesToIndex, and spool (optional) on the destination volumes. Make a note of the full paths to these directories. The new Indexes and FilesToIndex directories are usually created alongside each other on the same volume, although this is not a requirement. If the canister data and indexes are going to be located on the same volume it is conventional to create the Indexes and FilesToIndex directories as subdirectories within the "Canister" folder, alongside the "CANISTER.dbs" folder. This is not a requirement.

1. Edit current configuration: canister data and indexes

Warning! It is crucial not to apply these changes prematurely. Make sure to use the "Add to Current Job" option in TMS as explained below. Warning! Do not attempt to modify the spool directory location at the same time as changing the canister data and index locations. Modify the spool location as a separate operation afterwards (see below).

Canister data

• Log into the Tealeaf Portal as an administrator
• Tealeaf > TMS > WorldView tab
• Expand the canister server being modified
• Canister Server > Canister Server configuration > View/Edit button
• Expand "Default" section
• Edit "LOCAL_DIRECTORY" parameter and enter the new path for canister data. Be sure to include a trailing \ character.
• Save, and ?Add to Current Job?. Do not ?Add Tasks and Submit? at this time, as you we not want the change to take effect until we have copied the data to the new locations.

Canister indexes

• Log into the Tealeaf Portal as an administrator
• Tealeaf > TMS > WorldView tab
• Expand the canister server being modified
• Session Indexer > Index Service configuration > View/Edit button
• Select "Scheduling/Diagnostic" tab
• Edit "Index Directory" parameter in the "Index Lib" area and enter the new path for the Indexes directory. Do not add a trailing \ character.
• Select "Indexing Options" tab
• Edit "Path for Temp XML Files" and enter the new path for the Indexes directory, as in the previous step. Do not add a trailing \ character.
• Save, and ?Add to Current Job?. Do not ?Add Tasks and Submit? at this time, as you we not want the change to take effect until we have copied the data to the new locations.

Canister FilesToIndex

• Log into the Tealeaf Portal as an administrator
• Tealeaf > TMS > WorldView tab
• Expand the canister server being modified
• Canister > Canister configuration > View/Edit button
• Select "Canister Services" tab
• Edit "Location of Files to be Indexed" in the "Free-Text Indexing" area and enter the new path for the FilesToIndex directory. This directory is typically located alongside the new Indexes directory. Do not add a trailing \ character.
• Save, and ?Add to Current Job?. Do not ?Add Tasks and Submit? at this time, as you we not want the change to take effect until we have copied the data to the new locations.

2. Stop canister services and migrate data

Stop canister services from TMS

Warning! Stopping services from TMS occasionally fails. Always ensure you have access to the canister server via Remote Desktop as a backup so that you can stop the services directly.
Warning! In some environments Windows services that are meant to run continuously, such as the canister, are configured to automatically restart when stopped. To guard against this the services can be temporarily set to ?Disabled?. This requires direct access to the canister server.
Warning! On a busy canister server with a large amount of RAM the Canister Manager service can take more than 10 minutes to stop. This is normal.

• Log into the Tealeaf Portal as an administrator
• Tealeaf > TMS > WorldView tab
• Expand the canister server being modified
• Select "Canister" in the list of components > Stop button
• Select "Canister Manager" in the list of components > Stop button
• Select "Session Indexer" in the list of components > Stop button
• Wait for all three services to stop. Use the Refresh button to check their status. Because the canister is now stopped, incoming hits will be saved to the spool directory. Confirm this is occurring in the Pipeline Status tab in TMS.

Copy over data and indexes

If the destination is on another volume, copy the data and indexes rather than moving them. A copy takes no more time and allows for rolling back if there are problems. If the destination is on the same volume as the source, move the data rather than copying it.

The canister data consists of the entire contents of the top level folder (typically called "Canister") including the "CANISTER.dbs" subfolder, and all other files and subfolders except for the "Indexes" and "FilesToIndex". The exception is the "Canister.spacer" file, which does not need to be copied as it will be automatically recreated when the services restart.

The index data consists of the entire contents of the "Indexes" folder. The FilesToIndex folder is usually empty. If it does contain anything, copy it to the destination.

3. Apply configuration changes and start canister

Once the data has been moved to the new locations the configuration changes entered earlier can be applied and the services restarted.

• Log into the Tealeaf Portal as an administrator
• Tealeaf > TMS > Jobs tab
• Review the pending job in the "Tasks" pane on the left. It should consist of the three configuration changes we made earlier, together with a service restart task for each.
• Click "Submit" to apply the changes and start the services. The three configuration changes made earlier in TMS are pushed to the canister, and the three canister services that were stopped above are restarted.
• Tealeaf > TMS > WorldView tab
• Expand the canister server being modified
• Select "Search Server" in the list of components > Restart button. This ensures the Search Server picks up the new configuration.
• The canister will start clearing the spool. Confirm this is occurring in the Pipeline Status tab in TMS.

Moving the spool directory (optional)

The spool directory location can be changed at any time and is not dependant on the canister data or indexes locations. You do not need to change the spool location as part of any migration of canister data.

Warning! Do not change the spool directory location as part of the canister data/index file location change procedure described above. Perform it once those changes have been applied successfully.
Warning! Do not change the spool directory location whilst the system is spooling. Confirm that the current directory is empty and that no spooling is visible in the Pipeline Status tab in TMS first.

Change spool file location in TMS

• Log into the Tealeaf Portal as an administrator
• Tealeaf > TMS > WorldView tab
• Expand the canister server being modified
• Transport Service > Transport Service configuration > View/Edit button
• Select "DecoupleEx" session agent in the left pane > Edit button
• Expand "Spooling" section
• Edit "Spool Directory" parameter and enter the new path for spool directory. Do not add a trailing \ character.
• OK, Save, and "Add Tasks and Submit". The Transport Service will restart and the change will take effect immediately.

Next steps

Validate system is working

• Confirm all Tealeaf services are running
• Confirm that active sessions are present
• Confirm that the spool has been cleared (this may take some time)
• Confirm that searches work as expected (active and completed)
• Confirm that searches work for past days (e.g. one week ago)
• Replay a session
• Confirm that data is no longer being written to the old data and index locations

Apply exclusions

Set up exclusions for any virus scanning software or backup agents so that they do not scan the canister data, index, and spool locations when the canister is running. This type of scanning can cause data corruption and degraded performance. The following locations must be excluded:

• Canister data directory
• Canister index directory
• Canister FilesToIndex directory
• Spool directory

Any exclusions that apply to the old locations can be removed.

Remove old data

Delete old canister data and indexes from the source locations once you are confident the system is working as expected.

Appendix: Troubleshooting

If the canister does not start up correctly after applying the configuration changes in section (3) above, spooling will continue. If the Canister and Canister Manager services display as running in TMS (green icons) attempt to resolve the issue by restarting the Transport Service. This may cause the spool to begin to clear.

Note that clearing a large spool will result in a "stop-go" pattern where the spool clears for some time before the canister pauses to process the data it currently has in memory. After clearing that it will begin to accept more hits from the spool again. The spool is therefore cleared in a series of "steps".

If the Canister and Canister Manager services display as stopped, and cannot be successfully started (or start, but stop again within a minute) then consider using the CanRebuild utility. This requires Remote Desktop access to the server and cannot be performed from the Portal.

CanRebuild

On the canister server, check the <Tealeaf Installation Directory>\Logs\TLTMaint.log file. The last line in this log should be "TLTMaint completed with no errors in x seconds". If that is not the case, perform a CanRebuild.

• Run <Tealeaf Installation Directory>\CanRebuild.exe as an administrator
• Make sure that "Rebuild Full Canister" and "Preserve Session Data" are checked
• Make sure that "Delete Spool Files" is not checked
• Click "Rebuild"
• Click "Yes" when the warning is displayed. You will not lose data, despite the warning message.

The CanRebuild utility now stops the Tealeaf services and rebuilds the canister data structures. The process can take several minutes to complete. During this time incoming traffic spools so no data is lost. Once the rebuild process completes, use the "Start Tealeaf Services" shortcut in the Windows Start menu again, as the CanRebuild utility does not restart the services automatically. If this does not help, and the canister services still cannot be started, contact Tealeaf Support and attach the <Tealeaf Installation Directory>\Logs\TLTMaint.log file.
?