FTO HUBS need to be kept up to date with each other.

Body

FTO HUBS need to be kept up to date with each other.

There have been a number of Cases recently when there has been issues with a failover to the second TEMS in a FTO pair.

Now there are some known issues with the synchronization of these machines, but there are fixes available; and  I would recommend that you are at 6.3 FP06 to get all the fixes that are out there at the moment for FTO pairs. Note, that most up to date version of ITM at the moment (30 March 20202) is 6.3 FP07 SP3.

However in more than one case the issues were more fundamental than that.

The FTO pair had not been kept up to date with each other.

This was usually due to the secondary HUB not normally being up and running and being kept as just a backup machine.
This is not the way a FTO pair should be kept, they should both be running at all times, as they then synchronize changes in information and so it does not matter which of the two machines is the primary and there is a smooth transfer if one server fails or needs to be stopped.

Some of the issues that are seen when the machines are not kept in step are:

1) Application support; either not applied to one of the machines or it has not been updated to the same level on both machines.
This does not result in records being lost from the TEMS database, but it does mean that attributes are not recognized and situations not being handled correctly from one of the servers.
Queries from the TEPS to the TEMS may not be recognized as the attribute is not known.
This can be checked by getting a cinfo list from both machines and checking that the application support is to the same level on both machines, and all application support is on both machines.

2) If it is used, the xml file created by tedgen (output.xml) for the sending of events to an Event Server is not the same on both machines, if separate TEPS are used for each TEMS.
This can result in events not being able to be sent to the Event Server when the machine missing some definitions is the primary server, as the definitions are not recognized.
The location of the file can be found by checking for the attribute KFW_MCS_XML_FILES=<tedgen_output_file> in the TEPS config file.
A quick check of the size and date of the file, should show if the files are the same.  
Again changes in application support effect this file, and it should be updated when new application support added, and then the file populated to any other TEPS.

3) Configuration changes being applied to one but not both machines.

Have seen issues like
 a) KMS_SDA=N  on one machine and  KMS_SDA=Y on the other.

If the self-describing agent function is set on at one HUB it should be set as on for both HUBS.
 

b) Missing lines such as:
 KT1_TEMS_SECURE=YES
 KT1_RPC_THREADS=32

c) Make sure any tuning parameters (such as MAX_CTIRA_RECURSIVE_LOCKS) are also on both servers, if they are needed on one  machine they will be needed on the other machine.

4) Both HUB FTO's should be kept at the same level of ITM and this Tech Note gives the best steps for upgrading a FTO pair.

http://www-01.ibm.com/support/docview.wss?uid=swg21499940

5) Even with a FTO pair,  backups should be done regularly and this blog gives good details of this:

https://itm-sitworld.blog/2013/07/12/sitworld-best-practice-tems-database-backup-and-recovery/