IBM Support

IT25264: Agents abend with ArrayIndexOutOfBoundsException or NullPointerException after upgrading to IBM MQ V8.0.0.6 or later

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • After upgrade to IBM MQ v8.0.0.6 (or later) Managed File
    Transfer, agents running under heavy load abend. When the abend
    occurs, the following messages are written to the agent's event
    log (output0.log)
    
    000002de StdErr        E
    java.lang.ArrayIndexOutOfBoundsException: Array index out of
    range: 72
    000002de StdErr        E    at
    com.ibm.crypto.fips.provider.af.a(Unknown Source)
    000002de StdErr        E    at
    com.ibm.crypto.fips.provider.af.decrypt(Unknown Source)
    000002de StdErr        E    at
    com.ibm.crypto.fips.provider.W.decrypt(Unknown Source)
    000002de StdErr        E    at
    com.ibm.crypto.fips.provider.P.b(Unknown Source)
    000002de StdErr        E    at
    com.ibm.crypto.fips.provider.at.a(Unknown Source)
    000002de StdErr        E    at
    com.ibm.crypto.fips.provider.at.a(Unknown Source)
    000002de StdErr        E    at
    com.ibm.crypto.fips.provider.at.b(Unknown Source)
    000002de StdErr        E    at
    com.ibm.crypto.fips.provider.at.engineDoFinal(Unknown Source)
    000002de StdErr        E    at
    javax.crypto.Cipher.doFinal(Unknown Source)
    000002de StdErr        E    at
    com.ibm.wmqfte.utils.CredentialsFileEncoder.decodeUnwrapped(Cred
    entialsFileEncoder.java:161)
    000002de StdErr        E    at
    com.ibm.wmqfte.utils.CredentialsFileHelper.getFilePassword(Crede
    ntialsFileHelper.java:388)
    000002de StdErr        E    at
    com.ibm.wmqfte.transfer.TransferConnectionDetailsFactory.getConn
    ectionDetails(TransferConnectionDetailsFactory.java:87)
    000002de StdErr        E    at
    com.ibm.wmqfte.transfer.impl.TransferReceiverRunnable.run(Transf
    erReceiverRunnable.java:398)
    000002de StdErr        E    at
    java.lang.Thread.run(Thread.java:809)
    000002de StdErr        E    at
    com.ibm.wmqfte.thread.FTEThread.run(FTEThread.java:70)
    000002de ABENDImpl     E   BFGUT0003E: An internal error has
    occurred. An unexpected condition has occurred that has
    resulted in the abnormal end of this process.  Failure data was
    captured in file "ABEND.FTE.YYYYMMDDHHMMSSxxx.xxxxxxx.log".
    
    
    The ABEND.FTE.YYYYMMDDHHMMSSxxx.xxxxxxx.log file shows:
    
    Class:      com.ibm.wmqfte.thread.FTEThread
    ethod:     uncaughtException
    Probe:      ABEND_001
    Cause:      java.lang.NullPointerException
    java.lang.NullPointerException
     at com.ibm.crypto.fips.provider.af.a(Unknown Source)
    
    The NullPointerException has the same stack as shown in the
    event log (output0.log) for the
    agent.
    
    The errors do not happen when IBM MQ Managed File Transfer agent
    trace is enabled.
    
    
    Additional Symptom(s) Search Keyword(s): MFT transfer
     java.lang.NullPointerException
    java.lang.ArrayIndexOutOfBoundsException
    com.ibm.crypto.fips.provider.af.a
    

Local fix

  • restart agent
    

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of:
    
    - IBM MQ V8.0.0.6 (and later) Managed File Transfer.
    - IBM MQ V9 Managed File Transfer.
    
    who have agents that have been configured to use an IBM MQ
    Managed File Transfer credentials file, where the sensitive data
    in the credentials file has been encrypted using the
    fteObfuscate command.
    
    
    Platforms affected:
    MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    If the queue manager for an IBM MQ Managed File Transfer (MFT)
    agent has been configured with security enabled, and requires a
    user identifier and password to be flowed when the agent
    connects, then the user identifier and password can be stored in
    an IBM MQ MFT credentials file (MQMFTCredentials.xml). By
    default, agents will try to load the IBM MQ MFT credentials file
    from the home directory of the user that the agent is running
    as. To specify a different location, the property
    agentQMgrAuthenticationCredentialsFile needs to be set in the
    agent.properties file for the agent.
    
    The password stored in the IBM MQ MFT credentials file can be
    encrypted using the fteObfuscate command.
    
    All of the threads within an agent use the same internal object
    whenever they need to decrypt a password contained within the
    IBM MQ MFT credentials file.  This internal object makes use of
    some functionality provided by the underlying Java Runtime
    Environment (JRE) in order to perform the decryption.
    
    Now, if an agent was performing lots of managed transfers
    simultaneously, then it was possible for multiple threads within
    an agent to be decrypting passwords at the same time. When using
    IBM MQ V8.0.0.5 (or earlier) MFT, the functionality provided by
    the Java Runtime Environment allowed this behaviour. However,
    the Java Runtime Environments shipped with IBM MQ V8.0.0.6 (and
    later), and IBM MQ V9, would throw
    ArrayIndexOutOfBoundsExceptions or NullPointerExceptions if two
    or more threads were performing decryption operations
    simultaneously. Because the threads within the agent were not
    expecting these exceptions to be thrown, they generated an abend
    file (called "ABEND.FTE.<date and
    timestamp><number>.<number>.log") and then shut the agent down.
    Some examples of the abend files that were generated are shown
    below:
    
    ABEND File 1:
    -----------------
    Class:      com.ibm.wmqfte.thread.FTEThread
    Method:     uncaughtException
    Probe:      ABEND_001
    Cause:      java.lang.NullPointerException
    java.lang.NullPointerException
        at com.ibm.crypto.fips.provider.af.a(Unknown Source)
        at com.ibm.crypto.fips.provider.af.decrypt(Unknown Source)
        at com.ibm.crypto.fips.provider.W.decrypt(Unknown Source)
        at com.ibm.crypto.fips.provider.P.b(Unknown Source)
        at com.ibm.crypto.fips.provider.at.a(Unknown Source)
        at com.ibm.crypto.fips.provider.at.a(Unknown Source)
        at com.ibm.crypto.fips.provider.at.b(Unknown Source)
        at com.ibm.crypto.fips.provider.at.engineDoFinal(Unknown
    Source)
        at javax.crypto.Cipher.doFinal(Unknown Source)
        at
    com.ibm.wmqfte.utils.CredentialsFileEncoder.decodeUnwrapped(Cred
    entialsFileEncoder.java:161)
        at
    com.ibm.wmqfte.utils.CredentialsFileHelper.getFilePassword(Crede
    ntialsFileHelper.java:388)
        ..............
    
    ABEND File 2:
    -----------------
    Class:      com.ibm.wmqfte.thread.FTEThread
    Method:     uncaughtException
    Probe:      ABEND_001
    Cause:      java.lang.ArrayIndexOutOfBoundsException
    java.lang.ArrayIndexOutOfBoundsException
    &#09;at com.ibm.crypto.fips.provider.at.a(Unknown Source)
    &#09;at com.ibm.crypto.fips.provider.at.engineUpdate(Unknown
    Source)
    &#09;at com.ibm.crypto.fips.provider.at.b(Unknown Source)
    &#09;at com.ibm.crypto.fips.provider.at.engineDoFinal(Unknown
    Source)
    &#09;at javax.crypto.Cipher.doFinal(Unknown Source)
    &#09;at
    com.ibm.wmqfte.utils.CredentialsFileEncoder.decodeUnwrapped(Cred
    entialsFileEncoder.java:161)
    &#09;at
    com.ibm.wmqfte.utils.CredentialsFileHelper.getFilePassword(Crede
    ntialsFileHelper.java:388)
        ..............
    

Problem conclusion

  • In order to resolve this issue, locking has been added to the
    internal object used by IBM MQ MFT agents to ensure that only
    one thread is decrypting a password stored within an IBM MQ MFT
    credentials file at any one time. This prevents the
    ArrayIndexOutOfBoundsException or NullPointerException being
    thrown from the Java Runtime Environment, and means that the
    agent will no longer abend.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v8.0       8.0.0.11
    v9.0 LTS   9.0.0.5
    v9.x CD    9.1.0.0
    
    The latest available MQ maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT25264

  • Reported component name

    IBM MQ MFT V8.0

  • Reported component ID

    5724H7252

  • Reported release

    800

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2018-06-04

  • Closed date

    2018-06-08

  • Last modified date

    2020-11-16

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    IBM MQ MFT V8.0

  • Fixed component ID

    5724H7252

Applicable component levels

[{"Line of Business":{"code":"LOB36","label":"IBM Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0"}]

Document Information

Modified date:
18 November 2020