Troubleshooting
Problem
The Tivoli Storage Manager server seems to be unstable and there are a lot of DB2 errors reported in activity log. Those errors may include SQL code SQL0290N or SQL1585N , translating to DB2 having problem to access it's tablespaces.
Symptom
A lot of errors like the ones below are reported in the activity log :
ANR0162W Supplemental database diagnostic information:
-1:54048:-1585 ([IBM][CLI Driver][DB2/LINUXX8664]
SQL1585N A system temporary table space with sufficient
page size does not exist. SQLSTATE=54048). (PROCESS: 67)
ANR9999D_3831306406 ReportSQLDiagInfo(dbieval.c:1406)
Thread<54>: Missing sqlState=55039, sqlCode=-290 from
table. Returning rc = 2158.
ANR9999D Thread<54> issued message 9999 from:
ANR9999D Thread<54> 0x00000000c7c203 OutDiagToCons
ANR9999D Thread<54> 0x00000000c7ee88 outDiagfExt
ANR9999D Thread<54> 0x00000000762035 DbiEvalSQLOutcomeX
ANR9999D Thread<54> 0x0000000079fcd8 DoTheReorg
ANR9999D Thread<54> 0x000000007a0bfd RdbReorg
ANR9999D Thread<54> 0x0000000078a17c
RdbMonitorStatsThread
ANR9999D Thread<54> 0x00000000cf305b StartThread
ANR9999D Thread<54> 0x00003f97e077f1 *UNKNOWN*
ANR9999D Thread<54> 0x00003f976e570d *UNKNOWN*
ANR0162W Supplemental database diagnostic information:
-1:55039:-290 ([IBM][CLI Driver][DB2/LINUXX8664] SQL0290N
Table space access is not allowed. SQLSTATE=55039).
In db2diag.log, there are records showing that the tablespaces are offline :
2014-02-10-10.02.14.793133+120 I18260718E523 LEVEL: Error
PID : 47238 TID : 140109005448960PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-26914 APPID: *LOCAL.tsminst1.140210080216
AUTHID : TSMINST1
EDUID : 396 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbCreateObject, probe:20
MESSAGE : ZRC=0x8002014E=-2147352242=SQLB_NOT_ALLOWED_OFFLINE
"Access not allowed. Tblspc Offline."
Earlier in db2diag.log, a problem can be seen when accessing directory "/tmp/SQLDIR.LK0" :
2014-02-05-09.03.34.689729+120 E4606943E898 LEVEL: Error (OS)
PID : 47238 TID : 140113480771328PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-102 APPID: *LOCAL.tsminst1.140205070449
AUTHID : TSMINST1
EDUID : 193 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, oper system services, sqlomkdirp, probe:10
MESSAGE : ZRC=0x840F0001=-2079391743=SQLO_ACCD "Access Denied"
DIA8701C Access denied for resource "", operating system return code
was "".
CALLED : OS, -, mkdir OSERR: EACCES (13)
DATA #1 : File name, 15 bytes
/tmp/SQLDIR.LK0
DATA #2 : signed integer, 4 bytes
509
DATA #3 : signed integer, 4 bytes
509
DATA #4 : String, 105 bytes
Search for ossError*Analysis probe point after this log entry for further
self-diagnosis of this problem.
2014-02-05-09.03.34.689897+120 I4607842E1875 LEVEL: Error (OS)
PID : 47238 TID : 140113480771328PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-102 APPID: *LOCAL.tsminst1.140205070449
AUTHID : TSMINST1
EDUID : 193 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 Common, OSSe, ossErrorIOAnalysis, probe:100
CALLED : OS, -, mkdir OSERR: EACCES (13)
DATA #1 : String, 113 bytes
A total of 2 analysis will be performed :
- User info
- Path access permission
Target file = /tmp/SQLDIR.LK0
DATA #2 : String, 188 bytes
Real user ID of current process = 7004
Effective user ID of current process = 7004
Real group ID of current process = 1143
Effective group ID of current process = 1143
DATA #3 : String, 41 bytes
current sbrk(0) value: 0x00000000018f4000
DATA #4 : String, 226 bytes
Information of each subdirectory leading up to the first inaccessible one is shown in the format below :
<UID>:<GID>:<permissions> (subdirectories)
0:0:555 (tmp)
Cannot descend any further than the above subdirectory.
CALLSTCK:
[0] 0x00007F749B3C43BA pdOSSeLoggingCallback + 0x20C
[1] 0x00007F7499FDE6D6 /home/tsminst1/sqllib/lib64/libdb2osse.so.1 + 0x1CD6D6
[2] 0x00007F7499FDE5CD ossLogSysRC + 0xBF
[3] 0x00007F7499FD36F0 /home/tsminst1/sqllib/lib64/libdb2osse.so.1 + 0x1C26F0
[4] 0x00007F7499FD09CD ossErrorAnalysis + 0x2D
[5] 0x00007F749CE658DB sqloSystemErrorHandler + 0x6D7
[6] 0x00007F749B8C3794 sqlomkdirp + 0x26C
[7] 0x00007F749B41C6A7 _Z15sqlbLockAllDirsP16SQLO_NLCK_HANDLEP12SQLB_GLOBALS + 0xA3
[8] 0x00007F749B41CF98 _Z17sqlbLockParentDirPKcP16SQLO_NLCK_HANDLEtP12SQLB_GLOBALS + 0x98
[9] 0x00007F749B4277EB _Z20sqlbSMSDoContainerOpiPK9SQLP_LSN8iP16SQLB_POOLCONT_CB12SQLB_CONT_OPjP12SQLB_GLOBALS + 0xD9
2014-02-05-09.03.34.692363+120 I4609718E629 LEVEL: Error
PID : 47238 TID : 140113480771328PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-102 APPID: *LOCAL.tsminst1.140205070449
AUTHID : TSMINST1
EDUID : 193 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbLockAllDirs, probe:905
MESSAGE : ZRC=0x840F0001=-2079391743=SQLO_ACCD "Access Denied"
DIA8701C Access denied for resource "", operating system return code
was "".
DATA #1 : <preformatted>
Cannot create /tmp/SQLDIR.LK0
2014-02-05-09.03.34.692495+120 I4610348E576 LEVEL: Error
PID : 47238 TID : 140113480771328PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-102 APPID: *LOCAL.tsminst1.140205070449
AUTHID : TSMINST1
EDUID : 193 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbLockParentDir, probe:910
MESSAGE : ZRC=0x840F0001=-2079391743=SQLO_ACCD "Access Denied"
DIA8701C Access denied for resource "", operating system return code
was "".
Cause
Symptom is caused by a permission problem with directory "/tmp" or "/tmp/SQLDIR.LK0" .
Root cause for the permission issue is unknown.
Environment
Tivoli Storage Manager server on UNIX platforms
Diagnosing The Problem
Verify the db2diag.log for the tablespace offline message and/or an access problem for /tmp/SQLDIR.LK0.
Resolving The Problem
Check the permissions for directory "/tmp" and "/tmp/SQLDIR.LK0" .
"/tmp" should have permissions set to 1777 rwxrwxrwt. and "/tmp/SQLDIR.LK0" should have permissions set to 777 rwxrwxrwx.
If this is already the case, then open a ticket with IBM support so the symptom can get investigated. For this, please have the output for
db2support -d tsmdb1 -c -s -g
and
query system
available.
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21664588