IC74123: AFTER UPGRADE SERVER MAY EXPERIENCE LONG DELAYS DURING STARTUP DUE TO TABLE SCHEMA RECONCILIATION

APAR status

• Closed as program error.

Error description

• The Tivoli Storage Manager  Server database schema is sometimes
  updated in order to implement certain fixes/enhancements. As a
  result the schema needs to be reconciled before the server can
  become operational again. The time needed to complete this
  process will vary depending on the size of the database table(s)
  that require the schema reconciliation.  During this long start
  up, the administrator is unable to connect with a TSM admin
  command line client.  The Tivoli Storage Manager should post
  messages providing the status of the running schema
  reconciliations.
  
  Customer/L2 Diagnostics
  Example of a start up log taking over 7 hours:
  Tivoli Storage Manager
  Version 6, Release 2, Level 2.0
  Licensed Materials - Property of IBM
  (C) Copyright IBM Corporation 1990, 2010.
  All rights reserved.
  U.S. Government Users Restricted Rights - Use, duplication or
  disclosure
  restricted by GSA ADP Schedule Contract with IBM Corporation.
  
  23:47:36 ANR7801I Subsystem process ID is xx.
  23:47:36 ANR0900I Processing options file
  /tsm_servers/dsmserv.opt.
  23:47:36 ANR7814I Using instance directory /tsm_servers.
  23:47:37 ANR4726I The ICC support module has been loaded.
  23:47:37 ANR7859W Error reading from standard input.
  23:47:37 ANR0990I Server restart-recovery in progress.
  23:47:50 ANR0152I Database manager successfully started.
  23:50:14 ANR1380I The buffer pool monitor switch is enabled.
  23:50:15 ANR1383I Memory usage by the database is limited to
  32219MB.
  00:01:33 ANR1635I The server machine GUID, xx
  00:01:33 ANR1635I has initialized.
  00:01:53 ANR2100I Activity log process has started.
  00:01:54 ANR4726I The NAS-NDMP support module has been loaded.
  00:01:54 ANR1811W HBAAPI returned zero FC adapters from the
  system.
  00:02:01 ANR0275I Detected schema change for table
  BF.Aggregated.Bitfiles, attempting to
  00:02:01 ANR0275I reconcile.
  02:50:05 ANR0276I Schema reconciliation for table
  BF.Aggregated.Bitfiles succeeded.
  05:33:34 ANR0275I Detected schema change for table
  BF.Bitfile.Extents, attempting to
  05:33:34 ANR0275I reconcile.
  05:50:08 ANR0276I Schema reconciliation for table
  BF.Bitfile.Extents succeeded.
  07:47:44 ANR2803I License manager started.
  07:47:44 ANR8200I TCP/IP Version 4 driver ready for connection
  with clients on port
  07:47:44 ANR8200I 1600.
  
  Work around:
  The following DB2 select commands can be used, during the server
  start up, to find out how much of the  reconciliation has
  completed:
  
  For the table name listed in the schema reconciliation message,
  use the following SQL to obtain the "pct" of completion:
  
  db2 "select (count(*) * 100)/( select count(*) from <table_name>
  where srvid=0 with ur ) from <table_name_bad> where srvid=0 for
  read only with ur"
  I.E.
  ANR0275I Detected schema change for table
  BF.Aggregated.Bitfiles, attempting to reconcile.
  Login as the instance owner account
  db2 connect to tsmdb1
  db2 set schema tsmdb1
  db2 "select (count(*) * 100)/( select count(*) from
  bf_aggregated_bitfiles where srvid=0 with ur ) from
  bf_aggregated_bitfiles_bad where srvid=0 for read only with ur"
  1
  -----------
           11
  
  This indicates the upgrade for table bf_aggregated_bitfiles is
  11% complete. This can be done for any table involved in the
  schema reconciliation process.
  
  Initial Impact: Medium
  
  Additional Keywords: TSM dedup ZZ62
  

Local fix

• Allow DB2 to complete the needed schema reconciliation (do not
  stop the server).
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be         *
  *                 fixed in level 6.2.3. Note that this         *
  *                 is subject to change at the discretion       *
  *                 of IBM.                                      *
  ****************************************************************
  *
  

Problem conclusion

• This problem was fixed.
  Affected platforms: AIX, HP-UX, Sun Solaris, Linux and Windows.
  
  A new message ANR0335I will be issued every 5 minutes to
  report the progress of the running schema reconciliations.
  
  ANR0335I: Schema reconciliation for table table name in
  progress; X% completed.
  
  Explanation:
  The schema for this table is reconciling.
  
  System action:
  Schema reconciliation will continue for this table.
  
  User response:
  None.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP