IT15772: JGROUPS ALLOCATE 1650814064 BYTES OF MEMORY

si_52_build_5020500_interimfix_14
si_52_build_5020500_interimfix_14
si_52_build_5020500_interimfix_14
Media_IM_5020601_7
Media_IM_5020601_7
Media_IM_5020601_7
Media_IM_5020601_7

APAR status

• Closed as program error.

Error description

• Jgroups allocates 1650814064 bytes of memory (
  1.527 GB) and the noapp heap is set to maximum heap
  of  -Xmx1280m (1 GB)so this will always causes the OOM :-
  
  3XEHSTTYPE     07:33:24:346183000 GMT j9mm.126 -   at
  0000000050F8CD60 java/lang/Thread.run()V, jit 00007FCF323EA580,
  pc 00007FCF489E0A36
  3XEHSTTYPE     07:33:24:346179000 GMT j9mm.126 -   at
  0000000053644748
  org/jgroups/blocks/TCPConnectionMap$TCPConnection$Receiver.run()
  V, jit 0000000000000000, pc 00007FCF3354D334
  3XEHSTTYPE     07:33:24:346175000 GMT j9mm.101 -
  J9AllocateIndexableObject() returning NULL! 1650814064 bytes
  requested for object of class 0000000050F79700 from memory
  space 'Generational' id=00007FCF440427C0
  
  The heapdumps and javacore show minimal heap
  usage until jgroups trys to request this large amount of memory.
  

Local fix

• STRTC 490911
  
  AC/AC
  
  Workaround - Increase heap to 2GB
  

Problem summary

• Users Affected:
  All
  
  Problem Description:
  JGroups is generating OOM on each node as it was requesting for
  more memory even though JGroups did not need that much memory
  for working properly historically.
  A ticket was raised with JGroups. Further with JGroup suggestion
  we were able to resolve the issue by setting the port_range = 0
  ( We Ship port_range =1 ) in jgroups_cluster.property_string.
  Sample property string after making changes is as below. Notice
  that we have "port_range=0"
  jgroups_cluster.property_string=TCP(bind_addr=x.x.x.x;bind_port=
  60061;enable_diagnostics=false;level=ERROR):TCPPING(initial_host
  s=x.x.x.x[60061],y.y.y.y[60061];port_range=0;timeout=5000;num_in
  itial_members=2):MERGE2(min_interval=3000;max_interval=5000):FD
  ALL(interval=5000;timeout=20000):FD(timeout=5000;max_tries=48;le
  vel=ERROR):VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(retransmit
  _timeout=100,200,300,600,1200,2400,4800;discard_delivered_msgs=t
  rue):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=20000
  ;max_bytes=0):pbcast.GMS(print_local_addr=true;join_timeout=5000
  )
  jgroups_cluster.distribution_property_string=TCP(bind_port=60060
  ;thread_pool_rejection_policy=run;enable_diagnostics=false):TCPP
  ING(initial_hosts=lnvfr99720042[60060];port_range=0;timeout=5000
  ;num_initial_members=2):MERGE2(min_interval=3000;max_interval=50
  00):FD_SOCK:FD(timeout=5000;max_tries=48;level=ERROR):VERIFY_SUS
  PECT(timeout=1500):pbcast.NAKACK(retransmit_timeout=3000;discard
  _delivered_msgs=true):pbcast.STABLE(stability_delay=1000;desired
  _avg_gossip=20000;max_bytes=0):pbcast.GMS(join_timeout=5000;prin
  t_local_addr=true)
  
  Platforms Affected:
  All
  

Problem conclusion

• Resolution Summary:
  A code fix is provided.
  
  Delivered In:
  5020500_14
  5020603
  5020601_7
  5020602_4
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  STR B2B INTEGRA

• Fixed component ID

  5725D0600

Applicable component levels