IBM Support

Semaphore leak with Information Server on AIX due to DataDirect ODBC Driver Manager

Flashes (Alerts)


Abstract

Users may experience Semaphore leaks on the system, that can result in the server running out of free semaphores and cause issues related to the execution of jobs. Issues seen may include: job hangs, failures during project creation or failures in the creation of new JVMs (required for some of the operations).

Content

The following issues have been observed with the exhaustion of the semaphores:

  • The error, “Cannot create the Java virtual machine”, is received
  • Jobs randomly hang
  • High semaphore counts and semaphores not releasing

Environment: This issue is seen only on the AIX platform. Versions of Information Server affected are: IS v8.5, IS v8.7, IS v9.1, IS v11.3, IS v11.5 and  IS v11.7. Users must also be using DataDirect (DD) v6.1, or greater, drivers. The problem is manifested in the DD Driver Manager above the following build level:

Load of odbc.so successful, qehandle is 0x3
File version: 06.10.0058 (U0044)

Analysis: The issue has been identified to be with the DataDirect Driver Manager (odbc.so under $DSHOME/../branded_odbc/lib). There are two issues:

  1. Semaphore leak on every execution.
  2. Semaphore leak in parallel jobs only with ODBC related stages or any tools such as Information Analyzer using the ODBC Connector.

Steps to check if you are experiencing the issue:

  1. Check the number of semaphores in use on the server using the command "ipcs -rsa | wc -l"
  2. Change directory to $DSHOME, e.g. cd $DSHOME or cd /opt/IBM/InformationServer/Server/DSEngine
  3. Source the dsenv in a telnet or putty session, e.g. ". ./dsenv".
  4. Execute uvsh as follows:  bin/uvsh ""
  5. Check again the number of semaphores in use on the server using the command provided above in step 1.

  6. Here is an example that shows the leak:

    $ ipcs -rsa | wc -l
    188
    $ cd /opt/IBM/InformationServer/Server/DSEngine
    $ . ./dsenv
    $ bin/uvsh ""
    $ ipcs -rsa | wc -l
    192

If there is a increase of 4 semaphores, then you are running into the semaphore leak of Type (1) mentioned above. If there is no increase in semaphores seen with the steps above and yet you see a high number of semaphores in use on the server and you are using ODBC related stages on the parallel canvas, then follow the steps below to check if you observe a semaphore leak.

  1. Check the number of semaphores in use on the server using the command "ipcs -rsa | wc -l"
  2. Identify a job using the ODBC Connector or the ODBC Enterprise stage.
  3. Execute the job.
  4. Check again the number of semaphores in use on the server using the command provided above in step 1.

If you are seeing an increase in the number of semaphores, then you are running into the Type (2) semaphore leak.

Note : Ensure that no other jobs are running when you are carrying out the tests above.

Additional leak: Another minor semaphore leak can be observed when tracing is turned on (VMtrc00.so) or with 3rd party ODBC drivers, like Netezza. The leak is resolved with latest 7.1.6 DataDirect ODBC drivers.

Versions of Information Server affected are: IS v9.1, IS 11.3 , IS 11.5 and IS v11.7. You must also be using DataDirect (DD) v7.1, or greater, drivers. The problem is manifested in the VMtrc00.so driver at the following build level:

Load of odbc.so successful, qehandle is 0x3
File version: 07.10.0011

It is fixed at version: 07.16.0207 or higher.

Applying the solution

Resolution: Apply the appropriate version of the latest DataDirect ODBC drivers.
DataDirect (OEM) 6.1 ODBC Drivers, for InfoSphere Information Server for IS v8.5 and v8.7
DataDirect (OEM) 7.1 ODBC Drivers, for InfoSphere Information Server for IS v9.1 and later

Important Information All Releases: If you are utilizing the ODBC Connector or ODBC Enterprise stages in your job designs, ensure that:

  • Environment variable: DD_USE_SHARED_SEM=0 is set and exported in dsenv
  • The DataStage Engine and ASBNodeAgents are restarted after the change in the dsenv
  • ../branded_odbc/lib appears first in your library path.
  • Permissions are the same as the original branded_odbc.

InfoSphere Information Server, Version 8.5

ODBC Connector: Apply Fix Pack 3
Fix Pack 3 includes fixes that have JR42587
The environment variable CC_ODBC_USE_NEW_DRIVER_MANAGER should be added and set to 1 when the ODBC Connector is being used. More details about this variable and the fix for JR42587 can be found in this technote ODBC Connector jobs fail, hang, or create a core dump when using the DataDirect ODBC driver manager

ODBC Enterprise: Apply Fix Pack 2 or Fix Pack 3
JR39854 is included in both Fix Pack 2 and Fix Pack 3
JR39854 - Includes a fix to address a known issue with the ODBC Enterprise stage while using it with the latest DD drivers.

InfoSphere Information Server, Version 8.7

ODBC Connector: Apply Fix Pack 2
Fix Pack 2 includes fixes that have JR42587
The environment variable CC_ODBC_USE_NEW_DRIVER_MANAGER should be added and set to 1 when the ODBC Connector is being used. More details about this variable and the fix for JR42587 can be found in this technote: ODBC Connector jobs fail, hang, or create a core dump when using the DataDirect ODBC driver manager

ODBC Enterprise: No additional fix is needed. The fix is included in the IS 8.7 GA and all its fixpacks.

InfoSphere Information Server, Version 9.1, 11.3, 11.5 and 11.7:

Set DD_USE_SHARED_SEM=0, no additional fixes or environment variables are needed at IS 9.1 and later.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"Component":"--","Platform":[{"code":"PF002","label":"AIX"}],"Version":"9.1;8.7;8.5;11.5;11.3;11.7","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
25 September 2022

UID

swg21654008

Page Feedback