DB2 : Remove old and out-of-sequence LOG file from failarchpath

Body

If you see an old log file S0XXXXXX.LOG which is out of sequence, in failarchpath,
and have no db2diag.log to show if the file was archived properly or if it was retrieved again etc., there is no need to panic, we can fix it.

You will see that db2 continuously is logging messages as below or similar in the db2diag.log :
------------------------------------------------------------------------------------------------------------------------------------

2016-01-04-17.39.49.971346+000 E2137A532 LEVEL: Warning
PID : 16384028 TID : 4371 PROC : db2sysc 0
INSTANCE: db2inst NODE : 000 DB : dbname
HOSTNAME: host
EDUID : 4371 EDUNAME: db2logmgr (dbname) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile,
probe:3150
MESSAGE : ADM1848W Failed to archive log file "S0XXXXXX.LOG" to "TSM
chain 0" from "/<path>/log_archive/<instance>/<dbname>/NODE0000/LOGSTREAM0000/C0000000/".

2016-01-04-17.39.49.972939+000 E2670A643 LEVEL: Error
PID : 16384028 TID : 4371 PROC : db2sysc 0
INSTANCE: db2inst NODE : 000 DB : dbname
HOSTNAME: host
EDUID : 4371 EDUNAME: db2logmgr (dbname) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile,
probe:3160
MESSAGE : ZRC=0x8610000D=-2045771763=SQLP_BADLOG "Log File cannot be
used"           DIA8414C Logging can not continue due to an error.
DATA #1 : <preformatted>
Failed to archive log file S0XXXXXX.LOG to TSM chain 0 from
/<path>/log_archive/<instance>/<dbname>/NODE0000/LOGSTREAM0000/C0000000/.

2016-01-04-17.39.49.973869+000 E3314A543 LEVEL: Warning
PID : 16384028 TID : 4371 PROC : db2sysc 0
INSTANCE: db2inst NODE : 000 DB : dbname
HOSTNAME: host
EDUID : 4371 EDUNAME: db2logmgr (dbname) 0
FUNCTION: DB2 UDB, data protection services, sqlpgCleanFailArchPath,
probe:4940
DATA #1 : <preformatted>
Unable to archive log file XXXXXX from failarchpath
/<path>/log_archive/<instance>/<dbname>/NODE0000/LOGSTREAM0000/C0000000/ for
LOGARCHMETH1 due to rc -2045771763.
------------------------------------------------------------------------------------------------------------------------------------------------

RC -2045771763 translates to SQL0980C which indicates a possible disk error where the LOG file S0XXXXXX.LOG is located.

Check your system for any errors, which is present, must be rectified first. This shall most probably resolve the problem and the file could be successfully
archived.

If even after checking and/or rectifying the same messages are seen, then we can use the solution stated below.

Solution :

- Check the size of log file. If it is not the same as LOGFILESIZ then there is something wrong with the file. In most cases, we have seen
the file to be 0 bytes. We term it as corruption.

Irrespective of the root cause of log file corruption, the best and safest solution for you to resolve this problem is :

I) If you already have a latest online backup and current log files are achieving fine, you may try to remove* the unresolved log
file in the FAILARCHPATH directory without stopping DB2 database/instance.

If I gives any error, then :

II) Next approach :
1> stop DB2 (command : db2stop)
2> Backup current database (command : db2 backup db <dbname> or the command you use regularly for backups)
3> Remove* unresolved log file in the FAILARCHPATH.

*When we say "remove the log file" we mean copy it to some other unrelated location so we retain it until db starts working properly.

(Note: Taking a backup now, will ensure that this log file is not needed for any future recoveries of database should it shutdown)