APAR status
Closed as fixed if next.
Error description
After a topology change (adding/removing worker nodes), FODC folders are being created every 10 minutes and filling up the /var/ibm/bigsql/diag/DIAG0000/ directory, even when nothing appears to be running on the cluster. In addition, SQL statements or commands may fail with sqlcode -901. The db2diag.log may contain messages similar to the following: 2018-11-27-13.40.00.243584+000 E54044987E599 LEVEL: Severe PID : 906 TID : 140180333782784 PROC : db2sysc 0 INSTANCE: bigsql NODE : 000 DB : BIGSQL APPHDL : 0-48496 APPID: *N0.bigsql.xxxxxxxxxxxx AUTHID : BIGSQL HOSTNAME: xxxxxxxxxxxxxxxxxxx EDUID : 41056 EDUNAME: db2agent (BIGSQL) 0 FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:200 MESSAGE : ADM0001C A severe error has occurred. Examine the administration notification log and contact IBM Support if necessary. 2018-11-27-13.40.00.253612+000 I54045587E1063 LEVEL: Severe PID : 906 TID : 140180333782784 PROC : db2sysc 0 INSTANCE: bigsql NODE : 000 DB : BIGSQL APPHDL : 0-48496 APPID: *N0.bigsql.xxxxxxxxxxxx AUTHID : BIGSQL HOSTNAME: xxxxxxxxxxxxxxxxxxx EDUID : 41056 EDUNAME: db2agent (BIGSQL) 0 FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:250 MESSAGE : ZRC=0x80310004=-2144272380=SQLNN_E_NFOUND "Could not find an internal object required to process the command" DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes sqlcaid : SQLCA sqlcabc: 136 sqlcode: -901 sqlerrml: 58 sqlerrmc: sqlno_node_set_add [100]:rc( 0) sqlerrp : SQLNN067 sqlerrd : (1) 0x80310004 (2) 0x00000004 (3) 0x00000000 (4) 0x00000000 (5) 0xFFFFFDA8 (6) 0x00000000 sqlwarn : (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) sqlstate: 2018-11-27-13.40.04.733998+000 E54116889E2338 LEVEL: Info (Origin) PID : 906 TID : 140180333782784 PROC : db2sysc 0 INSTANCE: bigsql NODE : 000 DB : BIGSQL APPHDL : 0-48496 APPID: *N0.bigsql.xxxxxxxxxxxx AUTHID : BIGSQL HOSTNAME: xxxxxxxxxxxxxxxxxxx EDUID : 41056 EDUNAME: db2agent (BIGSQL) 0 FUNCTION: DB2 UDB, SW- common services, sqlnn_cmpl, probe:650 MESSAGE : ZRC=0x80310004=-2144272380=SQLNN_E_NFOUND "Could not find an internal object required to process the command" DATA #1 : String, 62 bytes An unexpected error was detected during statement compilation. DATA #2 : Boolean, 1 bytes false DATA #3 : Boolean, 1 bytes false DATA #4 : Boolean, 1 bytes true DATA #5 : Boolean, 1 bytes false DATA #6 : Hex integer, 4 bytes 0x00000000 DATA #7 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes sqlcaid : SQLCA sqlcabc: 136 sqlcode: -901 sqlerrml: 58 sqlerrmc: sqlno_node_set_add [100]:rc( 0) sqlerrp : SQLNN067 sqlerrd : (1) 0x80310004 (2) 0x00000004 (3) 0x00000000 (4) 0x00000000 (5) 0xFFFFFDA8 (6) 0x00000000 sqlwarn : (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) sqlstate: DATA #8 : Hex integer, 4 bytes 0x00000040 DATA #9 : String with size, 419 bytes UPDATE SYSTOOLS.ADMINTASKSTATUS S SET S.STATUS='UNKNOWN', S.END_TIME = CURRENT TIMESTAMP WHERE S.STATUS='RUNNING' AND (S.BEGIN_TIME < (SELECT D.DB2START_TIME FROM SYSIBMADM.SNAPDBM D WHERE D.DBPARTITIONNUM=0) OR (TIMESTAMPDIFF(8, CHAR(CURRENT TIMESTAMP - S.BEGIN_TIME) ) > 12 AND S.TASKID IN (SELECT T.TASKID FROM SYSTOOLS.ADMINTASKS T WHERE T.PROCEDURE_SCHEMA='SYSHADOOP' AND T.PROCEDURE_NAME='BIGSQL_RUN_ANALYZE') ) ) DATA #10: String, 481 bytes Compiler error stack for rc = -2144272380: sqlnn_cmpl[380] sqlno_exe[500] sqlno_prep_phase[600] sqlno_int_part_produced[200] sqlno_int_part_produced_qt[100] sqlno_int_part_produced_qu[100] sqlno_int_part_produced_qt[100] sqlno_int_part_produced_qu[100] sqlno_int_part_produced_qt[100] sqlno_int_part_produced_qu[100] sqlno_int_part_produced_qt[100] sqlno_int_part_produced_qu[700] sqlno_build_base_part[120] sqlno_part_init[40] sqlno_part_from_base_ftb[65] sqlno_node_set_add[100] Inside the FODC_AppErr directories, the failing EDU stack will look similar to: 0x00007F94D0F0A1F7 _Z13sqlzeDumpFFDCP8sqeAgentjP5sqlcai + 0x00a7 (/home/bigsql/sqllib/lib64/libdb2e.so.1) 0x00007F94D0F098C0 _Z11sqlzeMapZrcP8sqeAgentjmjP5sqlcaiPK12sqlzeContextb + 0x0940 (/home/bigsql/sqllib/lib64/libdb2e.so.1) 0x00007F94CE3A1E20 _Z10sqlnn_erdmiiiiPK12sqlzeContext + 0x00f0 (/home/bigsql/sqllib/lib64/libdb2e.so.1) 0x00007F94CE3ACA98 _Z10sqlnn_cmplP8sqeAgentP11sqlrrstrings17sqlnn_compileModesP14sq lrr_cmpl_enviiPP9sqlnq_qur + 0x82f8 (/home/bigsql/sqllib/lib64/libdb2e.so.1) 0x00007F94CE3A4791 _Z10sqlnn_cmplP8sqeAgentP11sqlrrstrings17sqlnn_compileModesP14sq lrr_cmpl_env + 0x0021 (/home/bigsql/sqllib/lib64/libdb2e.so.1) 0x00007F94CFDB059B _Z17sqlra_compile_varP8sqlrr_cbP14sqlra_cmpl_envPhitiiiiiP14SQLP _LOCK_INFOP16sqlra_cached_varPiS5_Pb + 0x10eb (/home/bigsql/sqllib/lib64/libdb2e.so.1) 0x00007F94CFDCCD9E _Z14sqlra_find_varP8sqlrr_cbP17sqlra_cached_stmt13sqlra_stmt_idj jPhjthP14sqlra_cmpl_env15sqlra_fill_modePiiS8_iiiS8_P14SQLP_LOCK _INFOPP16sqlra_cached_varS8_bSA_Pb + 0x096e (/home/bigsql/sqllib/lib64/libdb2e.so.1) 0x00007F94CFE6E1F1 _Z13sqlra_get_varP8sqlrr_cbiibPbS1_ + 0x0f11 (/home/bigsql/sqllib/lib64/libdb2e.so.1) 0x00007F94CFB24BE6 _Z23sqlrr_execute_immediateP8sqlrr_cbi + 0x15c6 (/home/bigsql/sqllib/lib64/libdb2e.so.1)
Local fix
Restart the Big SQL service after a topology change
Problem summary
Please see problem description.
Problem conclusion
Temporary fix
Comments
APAR Information
APAR number
PH06249
Reported component name
IBM BIG SQL
Reported component ID
5737E7400
Reported release
502
Status
CLOSED FIN
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2018-12-06
Closed date
2020-09-09
Last modified date
2020-09-09
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Applicable component levels
[{"Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSCRJT","label":"IBM Db2 Big SQL"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"502"}]
Document Information
Modified date:
10 September 2020