Rollforward database command failed with SQL1042C

Body

This blog explains a scenario that caused rollforward failure.

The following is the rollforward command that failed:

$ db2 "rollforward database DB1 to end of backup ON ALL DBPARTITIONNUMS and
complete tablespace (TS1) online"

SQL1042C An unexpected system error occurred. SQLSTATE=58004

To diagnose the problem, verify db2daig.log:

<timestamp> LEVEL: Warning
PID : 12345678 TID : 123456 PROC : db2sysc 0
INSTANCE: db2ins1 NODE : 000 DB : DB1
APPHDL : 0-1234 APPID: *N0.db2ins1.12345678
AUTHID : DB2INS1 HOSTNAME: host11
EDUID : 123456 EDUNAME: db2agent (DB1) 0
FUNCTION: DB2 UDB, recovery manager, sqlpRcvRfwReplies, probe:49
MESSAGE : Error at Node 1 with rc = -2045837302

2016-12-04-18.58.30.938756-300 E667518352A891 LEVEL: Error (OS)
PID : 12345678 TID : 123456 PROC : db2sysc 1
INSTANCE: db2ins1 NODE : 001 DB : DB1
APPHDL : 0-14313 APPID: *N0.udbprod.170412102111
AUTHID : DB2INS1 HOSTNAME: host11
EDUID : 67898 EDUNAME: db2agntp (DB1) 1
FUNCTION: DB2 UDB, oper system services, sqlomkdirp, probe:10
MESSAGE : ZRC=0x860F000A=-2045837302=SQLO_FNEX "File not found."
DIA8411C A file "" could not be found.
CALLED : OS, -, mkdir
OSERR : ENOENT (2) "No such file or directory"
DATA #1 : File name, 21 bytes
/db2log/NODE0001/
DATA #2 : signed integer, 4 bytes
488
DATA #3 : signed integer, 4 bytes
488
DATA #4 : String, 105 bytes
Search for ossError*Analysis probe point after this log entry for
further
self-diagnosis of this problem.

2016-12-04-18.58.30.943636-300 I667519244A579 LEVEL: Error
PID : 12345678 TID : 123456 PROC : db2sysc 1
INSTANCE: db2ins1 NODE : 001 DB : DB1
APPHDL : 0-14313 APPID: *N0.db2ins1.170412102111
AUTHID : DB2INS1 HOSTNAME: host11
EDUID : 67898 EDUNAME: db2agntp (DB1) 1
FUNCTION: DB2 UDB, data protection services,
sqlpgCreateMemberSpecificLogPath, probe:100
MESSAGE : ZRC=0x860F000A=-2045837302=SQLO_FNEX "File not found."
DIA8411C A file "" could not be found.

The above error means, on node 1, DB2 was not able to create directory
"/db2log/NODE0001/"
Please ensure directory /db2log/NODE0001/ exists.

Once this is checked, please re-issue the rollforward command.