Troubleshooting
Problem
BACKUP DB fails with ANR2993E and sqlcode: -2428
Symptom
BACKUP DB fails with ANR2993 and the db2diag.log file reports DISK FULL.
Cause
Not enough space available in the specified ACTIVELOGDIRECTORY
Environment
The dsmserv.opt has the following settings:
ACTIVELOGSize 131072
ACTIVELOGDirectory /home/tsminst1/log
ARCHLOGDirectory /home/tsminst1/archlog
Diagnosing The Problem
When the BACKUP DB process fails the following messages will be seen in the Tivoli Storage Manager server's activity log:
- ANR2993E The database backup terminated with a log problem. DB2 sqlcode: -2428. DB2 sqlerrmc: .
ANR0985I Process ## for Database Backup running in the FOREGROUND completed with completion state FAILURE at 05:55:52 AM.
ANR1893E Process ## for Database Backup completed with a completion state of FAILURE.
Since the message is vague, you need to take the DB2 sqlcode found in the ANR2993E message, -2428, and search within the db2diag.log file for the code. Once the code is found, verify the timestamp is fairly close to the timestamp in the activity log for the ANR2993E. If the messages are within seconds of one another then you have found the entry in the db2diag.log file. Once you find the sqlcode, review the messages prior to the sqlcode for an indication as to why the BACKUP DB failed.
2014-06-18-05.19.03.317791-240 E104494E1180 LEVEL: Error (OS)
PID : 13431 TID : 46912975660800 PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
HOSTNAME: server1
EDUID : 255 EDUNAME: db2logmgr (TSMDB1) 0
FUNCTION: DB2 UDB, oper system services, sqlowrite, probe:70
MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
DIA8312C Disk was full.
CALLED : OS, -, write OSERR: ENOSPC (28)
DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes
File Handle = 441
File System Block Size = 0 bytes
File System Type = UNKNOWN
File Handle Flags :
Require Sector Align = No
DIO/CIO Mode = No
Raw Block Device = No
Reserved Handle = No
Flush On Close = Yes
Thread-Level Lock = Yes
Write-through Mode = No
File Not Tracked = No
DATA #2 : unsigned integer, 8 bytes
131072
DATA #3 : signed integer, 8 bytes
114688
DATA #4 : String, 105 bytes
Search for ossError*Analysis probe point after this log entry for further
self-diagnosis of this problem.
2014-06-18-05.19.03.463002-240 I105675E3229 LEVEL: Error (OS)
PID : 13431 TID : 46912975660800 PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
HOSTNAME: server1
EDUID : 255 EDUNAME: db2logmgr (TSMDB1) 0
FUNCTION: DB2 Common, OSSe, ossErrorIOAnalysis, probe:100
CALLED : OS, -, write OSERR: ENOSPC (28)
DATA #1 : String, 147 bytes
A total of 5 analysis will be performed :
- User info
- ulimit info
- Target file info
- I/O attempt
- File system
Target file handle = 441
DATA #2 : String, 184 bytes
Real user ID of current process = 501
Effective user ID of current process = 501
Real group ID of current process = 501
Effective group ID of current process = 501
DATA #3 : String, 371 bytes
Current process limits (unit in bytes except for nofiles) :
mem (S/H) = unlimited / unlimited
core (S/H) = unlimited / unlimited
cpu (S/H) = unlimited / unlimited
data (S/H) = unlimited / unlimited
fsize (S/H) = unlimited / unlimited
nofiles (S/H) = 65534 / 65534
stack (S/H) = unlimited / unlimited
rss (S/H) = unlimited / unlimited
DATA #4 : String, 41 bytes
current sbrk(0) value: 0x0000000000695000
DATA #5 : String, 262 bytes
Target File Information :
Size = 507232256
Link = No
Reference path = N/A
Type = 0x8000
Permissions = rw-------
UID = 501
GID = 501
Last modified time = 1403083143
DATA #6 : String, 33 bytes
I/O attempt not implemented yet.
DATA #7 : String, 407 bytes
File System Information of the target file :
Block size = 4096 bytes
Total size = 149022154752 bytes
Free size = 0 bytes
Total # of inodes = 9240576
FS name = /dev/tsminst1_log_vg
Mount point = /home/tsminst1/log
FSID = 64822
FS type name = ext4
DIO/CIO mount opt = None
Device type = N/A
FS type = 0xf
CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol)
[0] 0x00002AAAB709723B /home/tsminst1/sqllib/lib64/libdb2osse.so.1 + 0x21723B
[1] 0x00002AAAB70987C3 ossLogSysRC + 0xB3
[2] 0x00002AAAB708A934 /home/tsminst1/sqllib/lib64/libdb2osse.so.1 + 0x20A934
[3] 0x00002AAAB7087BFD ossErrorAnalysis + 0x2D
[4] 0x00002AAAB181A36B sqloSystemErrorHandler + 0x71B
[5] 0x00002AAAAF4A7F2A sqlowrite + 0x44A
[6] 0x00002AAAB1A0ADBE _Z22sqlpgRetrieveLogToTempP9SQLP_DBCBP12SQLO_FHANDLEP15SQLPG_VENDOR_CBPcP28SQLPG_RETRIEVE_ARRAY_ELEMENTPm + 0x122E
[7] 0x00002AAAB188A7A3 _Z20sqlpgRetrieveLogDiskP9SQLP_DBCBPKcS2_jjtb + 0x223
[8] 0x00002AAAB188FF03 _Z20sqlpgRetrieveLogFileP9SQLP_DBCB + 0x2D73
[9] 0x00002AAAB1892D83 _Z21sqlpgHandleLogMgrPostP12sqpLogMgrEduP9SQLP_DBCBj + 0xCA3
[10] 0x00002AAAB189528F _Z15sqlpgLogManagerP12sqpLogMgrEdu + 0x25F
[11] 0x00002AAAB19A0647 _ZN12sqpLogMgrEdu6RunEDUEv + 0x27
[12] 0x00002AAAAFC4C124 _ZN9sqzEDUObj9EDUDriverEv + 0xF4
[13] 0x00002AAAAF4B4217 sqloEDUEntry + 0x2F7
[14] 0x00000031C8A07851 /lib64/libpthread.so.0 + 0x7851
[15] 0x00000031C82E890D clone + 0x6D
...
2014-06-18-05.19.03.550330-240 I108905E467 LEVEL: Error
PID : 13431 TID : 46912975660800 PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
HOSTNAME: server1
EDUID : 255 EDUNAME: db2logmgr (TSMDB1) 0
FUNCTION: DB2 UDB, data protection services, sqlpgRetrieveLogToTemp, probe:7600
MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
DIA8312C Disk was full.
...
2014-06-18-05.19.52.866605-240 I187272E571 LEVEL: Error
PID : 13431 TID : 47018911196928 PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-55839 APPID: *LOCAL.tsminst1.140618134115
AUTHID : TSMINST1 HOSTNAME: server1
EDUID : 1208 EDUNAME: db2bm.1022.4 (TSMDB1) 0
FUNCTION: DB2 UDB, data protection services, sqlpOpenLogForBackup, probe:20
RETCODE : ZRC=0x82100016=-2112880618=SQLPLFR_RC_RETRIEVE_FAILED
"Log could not be retrieved"
2014-06-18-05.19.52.898048-240 E187844E621 LEVEL: Severe
PID : 13431 TID : 47018911196928 PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-55839 APPID: *LOCAL.tsminst1.140618134115
AUTHID : TSMINST1 HOSTNAME: server1
EDUID : 1208 EDUNAME: db2bm.1022.4 (TSMDB1) 0
FUNCTION: DB2 UDB, database utilities, sqlubProcessLogExtent, probe:2406
MESSAGE : ADM8010E Backup was unable to copy requested log file "S0044241.LOG"
for inclusion in the backup image. The backup has been aborted.
2014-06-18-05.19.52.913017-240 E189205E612 LEVEL: Severe
PID : 13431 TID : 47018982500096 PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-55839 APPID: *LOCAL.tsminst1.140618134115
AUTHID : TSMINST1 HOSTNAME: server1
EDUID : 1218 EDUNAME: db2bm.1022.0 (TSMDB1) 0
FUNCTION: DB2 UDB, database utilities, sqlubProcessLogExtent, probe:2530
DATA #1 : Sqlcode, PD_TYPE_SQLCODE, 4 bytes
-2428 <----- HERE -----
DATA #2 : Hexdump, 4 bytes
0x00002AC08195AA18 : 84F6 FFFF ....
For this particular case above, a DISK FULL condition has been encountered for the ACTIVELOGDIRECTORY /home/tsminst1/log. Reviewing the operating system output from command df -m it appears there is space available in the specified ACTIVELOGDIRECTORY:
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/tsminst1_log_vg 142119 131775 3127 98% /home/tsminst1/log
/dev/tsminst1_archlog_vg 2889706 1957142 785876 72% /home/tsminst1/archlog
The output above shows the /home/tsminst1/log directory is approximately 138 GB and has about 3 GB of available space. The server's activity log is consuming 128 GB of the active log directory.The remaining space is being consumed by log files being moved to and from the archive log directory (ARCHLOGDIRECTORY) during the BACKUP DB process.
Resolving The Problem
The Tivoli Storage Manger publication states you should provision an extra 20% of free space for the ACTIVELOGDIRECTORY that can be used if needed. In the example above, one of two items needs to occur:
- Increase the available free space in the ACTIVELOGDIRECTORY to be at least 20% greater than the ACTIVELOGSIZE (Formula: (ACTIVELOGSIZE * .20) + ACTIVELOGSIZE ). For the example above the ACTIVELOGDIRECTORY should be increased to approximately 153 GB of space.
- Decrease the ACTIVELOGSIZE to allow the current filesystem to be 20% greater than the ACTIVELOGSIZE specified. Since the above example has an ACTIVELOGDIRECTORY specified as 138 GB, the ACTIVELOGSIZE would need to be set to a value that is 20% of the 138 GB (Formula: ACTIVELOGDIRECTORY_size - (ACTIVELOGDIRECTORY_size * .20)). In this particular example, the ACTIVELOGSIZE would need to be adjusted down to 112640 and would require a restart of the Tivoli Storage Manager server. Here is the example formula:
138 - (138 * 0.20 ) = 110.4 This means the ACTIVELOGSIZE needs to be set to 110 GB, which is specified as 112640 MB in the options file. The .4 was drop because the value of the ACTIVELOGSIZE should be divisible by 512, which 512 goes into 112640, 220 times.
Related Information
[{"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Server","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Supported Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]
Product Synonym
TSM ADSM IBM Spectrum Protect
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21677011