IBM Support

On RHEL 7.2, Db2 may generate trap files which points gsignal in /lib64/libc.so.6

Troubleshooting


Problem

Why does Db2 generate trap files which points gsignal in /lib64/libc.so.6 on RHEL 7.2?

Symptom

Following is example stack trace of trap files which can be found under
~/sqllib/db2dump/FODC_Trap_xxxxx directories.

Example #1)
-----------
<POFDisassembly>
gsignal + 0x0037 (/lib64/libc.so.6)

0x00007FCFC6AB45F7 : 483D00F0FFFF771E
</POFDisassembly>
<StackTrace>
-----FUNC-ADDR---- ------FUNCTION + OFFSET------
0x00007FCFC7B5BF16 _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm + 0x0356
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007FCFC7B5BB5B ossDumpStackTraceV98 + 0x002b
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007FCFC7B57D07 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm + 0x00d7
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007FCFCECD4F17 sqlo_trce + 0x03c7
(/home/db2inst1/sqllib/lib64/libdb2e.so.1)
0x00007FCFCEE39E19 sqloEDUCodeTrapHandler + 0x03a9
(/home/db2inst1/sqllib/lib64/libdb2e.so.1)
0x00007FCFD5F83100 address: 0x00007FCFD5F83100 ; dladdress: 0x00007FCFD5F74000 ; offset in lib: 0x000000000000F100 ;
(/lib64/libpthread.so.0)
0x00007FCFC6AB45F7 gsignal + 0x0037
(/lib64/libc.so.6)
0x00007FCFC6AB5CE8 abort + 0x0148
(/lib64/libc.so.6)
0x00007FCFC6AF4317 address: 0x00007FCFC6AF4317 ; dladdress: 0x00007FCFC6A7F000 ; offset in lib: 0x0000000000075317 ;
(/lib64/libc.so.6)
0x00007FCFC6AFBFE1 address: 0x00007FCFC6AFBFE1 ; dladdress: 0x00007FCFC6A7F000 ; offset in lib: 0x000000000007CFE1 ;
(/lib64/libc.so.6)
0x00007FCFCECDF7A9 sqloGetGroupAttribById + 0x0169
(/home/db2inst1/sqllib/lib64/libdb2e.so.1)
-----------
.
Example #2)
-----------
<POFDisassembly>
gsignal + 0x0037 (/lib64/libc.so.6)

0x00007F9D3A1205F7 : 483D00F0FFFF771E
</POFDisassembly>
<StackTrace>
-----FUNC-ADDR---- ------FUNCTION + OFFSET------
0x00007F9D3B1C7F16 _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm + 0x0356
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007F9D3B1C7B5B ossDumpStackTraceV98 + 0x002b
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007F9D3B1C3D07 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm + 0x00d7
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007F9D42340F17 sqlo_trce + 0x03c7
(/home/db2inst1/sqllib/lib64/libdb2e.so.1)
0x00007F9D424A5E19 sqloEDUCodeTrapHandler + 0x03a9
(/home/db2inst1/sqllib/lib64/libdb2e.so.1)
0x00007F9D495EF100 address: 0x00007F9D495EF100 ; dladdress: 0x00007F9D495E0000 ; offset in lib: 0x000000000000F100 ;
(/lib64/libpthread.so.0)
0x00007F9D3A1205F7 gsignal + 0x0037
(/lib64/libc.so.6)
0x00007F9D3A121CE8 abort + 0x0148
(/lib64/libc.so.6)
0x00007F9D3A160317 address: 0x00007F9D3A160317 ; dladdress: 0x00007F9D3A0EB000 ; offset in lib: 0x0000000000075317 ;
(/lib64/libc.so.6)
0x00007F9D3A167FE1 address: 0x00007F9D3A167FE1 ; dladdress: 0x00007F9D3A0EB000 ; offset in lib: 0x000000000007CFE1 ;
(/lib64/libc.so.6)
0x00007F9C81FB0DFD sqloCleanupEnvironment + 0x047d
(/home/db2inst1/sqllib/lib64/libdb2.so.1)
-----
.
Example #3)
-----------
<POFDisassembly>
gsignal + 0x0037 (/lib64/libc.so.6)

0x00007F293BB865F7 : 483D00F0FFFF771E
</POFDisassembly>
<StackTrace>
-----FUNC-ADDR---- ------FUNCTION + OFFSET------
0x00007F293CC2DF16 _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm + 0x0356
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007F293CC2DB5B ossDumpStackTraceV98 + 0x002b
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007F293CC29D07 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm + 0x00d7
(/home/db2inst1/sqllib/lib64/libdb2osse.so.1)
0x00007F2943DA6F17 sqlo_trce + 0x03c7
(/home/db2inst1/sqllib/lib64/libdb2e.so.1)
0x00007F2943F0BE19 sqloEDUCodeTrapHandler + 0x03a9
(/home/db2inst1/sqllib/lib64/libdb2e.so.1)
0x00007F294B055100 address: 0x00007F294B055100 ; dladdress: 0x00007F294B046000 ; offset in lib: 0x000000000000F100 ;
(/lib64/libpthread.so.0)
0x00007F293BB865F7 gsignal + 0x0037
(/lib64/libc.so.6)
0x00007F293BB87CE8 abort + 0x0148
(/lib64/libc.so.6)
0x00007F293BBC6317 address: 0x00007F293BBC6317 ; dladdress: 0x00007F293BB51000 ; offset in lib: 0x0000000000075317 ;
(/lib64/libc.so.6)
0x00007F293BBCF7FD address: 0x00007F293BBCF7FD ; dladdress: 0x00007F293BB51000 ; offset in lib: 0x000000000007E7FD ;
(/lib64/libc.so.6)
0x00007F293BBD087C __libc_malloc + 0x004c
(/lib64/libc.so.6)
0x00007F293BC0A381 address: 0x00007F293BC0A381 ; dladdress: 0x00007F293BB51000 ; offset in lib: 0x00000000000B9381 ;
(/lib64/libc.so.6)
0x00007F2943E16B42 sqlofindf2 + 0x0342
(/home/db2inst1/sqllib/lib64/libdb2e.so.1)
-----------

Cause

It is an known behavior when using DB2 on RHEL 7.2.

Environment

DB2 : V11.1 FP1
OS : Red Hat Enterprise Linux (RHEL) Server release 7.2
gligc : glibc-2.17-105.el7.x86_64
glibc-common-2.17-105.el7.x86_64

Diagnosing The Problem

From trap files, <POFDisassembly> points the same function gsignal() which is OS one.
And it always seemed that abort() calls gsignal() and then DB2 got a trap.

From Example #1, here is the calling chain to gsignal:
sqloGetGroupAttribById -> first OS function -> second OS function -> abort -> gsignal
The first OS function offset is 0x000000000007CFE1
The second OS function offset is 0x0000000000075317

From Example #2,
sqloCleanupEnvironment -> first OS function -> second OS function -> abort -> gsignal
The first OS function offset is 0x000000000007CFE1
The second OS function offset is 0x0000000000075317

From Example #3,
sqlofindf2 -> __libc_malloc -> first OS function -> second OS function -> abort -> gsignal
The first OS function offset is 0x000000000007E7FD
The second OS function offset is 0x0000000000075317

And from a debugger, it can be read as:
A DB2 function -> _int_free() -> _libc_message() -> abort() -> gsignal

Resolving The Problem

The symptom has been gone after gligc was upgraded to below level:
glibc-2.17-196.el7.x86_64
glibc-common-2.17-196.el7.x86_64

Or it can be solved by upgrading RHEL 7.4 which has above glibc level.
Current glibc version can be confirmed by below command:
$ rpm -qa | grep glibc

Red Hat Enterprise Linux release date and kernel version list:
https://access.redhat.com/ja/node/16476#RHEL7
...
RHEL 7.4 2017-07-31 2017-07-31 RHBA-2017:1850 3.10.0-693
RHEL 7.3 2016-11-03 2016-11-03 RHEA-2016-2544 3.10.0-514
RHEL 7.2 2015-11-19 2015-11-19 RHEA-2015:2461 3.10.0-327
RHEL 7.1 2015-03-05 2015-03-05 RHEA-2015:0524 3.10.0-229
...

Here is gligc version list for each RHEL version.
...
RHEL 7.4 2017-07-31 glibc-2.17-196.el7.x86_64
RHEL 7.3 2016-11-03 glibc-2.17-157.el7.x86_64
RHEL 7.2 2015-11-19 glibc-2.17-105.el7.x86_64
RHEL 7.1 2015-03-05 glibc-2.17-78.el7.x86_64
...

[{"Product":{"code":"SSEPGG","label":"Db2 for Linux, UNIX and Windows"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Operating System \/ Hardware - OS Configuration","Platform":[{"code":"PF016","label":"Linux"}],"Version":"9.7;10.1;10.5;11.1","Edition":"Advanced Enterprise Server;Advanced Workgroup Server;Enterprise Server;Workgroup Server","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
16 June 2018

UID

swg22015540

Page Feedback