IT30454: AFTER UPGRADING FROM 5.2.6.3 TO 6.0.1 USING IIM MEMORY LEAKS ARE CAUSED BY JGROUPS IN NOAPP JVM

APAR status

• Closed as program error.

Error description

• Various symptoms can constitute a memory leak.
  The JVM Heap
  size increases progressively. The dashboard user interface may
  become very slow or not respond.
  The load balancing may seem to
  stop functioning.
  When one of the nodes in the cluster is
  stopped, it releases the memory on another node.
  

Local fix

• B2BISFG-50627
  

Problem summary

• Users Affected:
  All
  
  
  Problem Description:
  
  After upgrading from 5.2.6.3 to 6.0.1 using IIM,
  memory leaks are caused by jgroups in noapp JVM.
  This impacts the Workflow load distribution and stability.
  Restart IBM Sterling B2B Integrator before OOM occurs
  A memory leak in noapp JVM is related to jgroups which is
  related to Workflow distrubuiotn on UNICAST3 message exchange
  acklowlegements between two cluster node communication.
  
  It was primarily due to below String in CUSTOM_PROPERTY table
  which does workflow load distribution from Groups.
  
  <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="urn:org:jgroups"   xsi:schemaLocation="urn:org:jgroups
  http://www.jgroups.org/schema/jgroups.xsd">
    <TCP bind_addr="&HOST_ADDR;"
  bind_port="&MULTICAST_NODE_PORT2;" />
    <TCPPING async_discovery="true"
  initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0"
  send_cache_on_join="true" />
    <MERGE3  min_interval="3000" max_interval="5000" />
    <FD_ALL timeout="20000" interval="15000" />
    <FD_SOCK/> <FD timeout="5000" max_tries="48" />
    <VERIFY_SUSPECT timeout="1500" />
    <BARRIER />
    <pbcast.NAKACK2 use_mcast_xmit="false"
  discard_delivered_msgs="true" />
    <UNICAST3 /> <pbcast.STABLE desired_avg_gossip="20000"
  max_bytes="0" stability_delay="1000"/> <pbcast.GMS
  print_local_addr="true" join_timeout="15000" />
    </config>
  
  
  Platforms Affected:
  All
  

Problem conclusion

•       Resolution Summary
  
  A code fix is provided.
  
  Removed the BARRIER and MERGE3 along with
  UNICAST 3 for the Property String. Changes are applicable for
  IIM and Docker instances.
  
  1: Removed use_mcast_xmit="false" from NACACK2 protocol.
  
  2:Removed UNICAST3 since heap memory growth was due to these
  messages. This is actually not required for Workflow load
  distribution.
  
  3: Added FD_ALL parameter.
  
  4: Removed BARRIER.
  
  New Change:
  
  <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups
  http://www.jgroups.org/schema/jgroups.xsd"> <TCP
  bind_port="&MULTICAST_NODE_PORT1;" /> <TCPPING
  async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;"
  port_range="0" send_cache_on_join="true" /> <MERGE3
  min_interval="3000" max_interval="5000"/><FD_ALL timeout="20000"
  interval="15000" /> <FD_SOCK/> <FD timeout="5000" max_tries="48"
  /> <VERIFY_SUSPECT timeout="1500" /> <pbcast.NAKACK2
  discard_delivered_msgs="true"/> <pbcast.STABLE
  desired_avg_gossip="20000" max_bytes="0" stability_delay="1000"
  /> <pbcast.GMS print_local_addr="true"
  join_timeout="15000"/></config>
  
  Delivered In:
  
  6000300
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  STR B2B INTEGRA

• Fixed component ID

  5725D0600

Applicable component levels