Fixes are available
8.5.0.2: WebSphere Application Server V8.5 Fix Pack 2
8.0.0.6: WebSphere Application Server V8.0 Fix Pack 6
7.0.0.29: WebSphere Application Server V7.0 Fix Pack 29
8.0.0.7: WebSphere Application Server V8.0 Fix Pack 7
8.0.0.8: WebSphere Application Server V8.0 Fix Pack 8
7.0.0.31: WebSphere Application Server V7.0 Fix Pack 31
7.0.0.33: WebSphere Application Server V7.0 Fix Pack 33
8.0.0.9: WebSphere Application Server V8.0 Fix Pack 9
7.0.0.35: WebSphere Application Server V7.0 Fix Pack 35
8.0.0.10: WebSphere Application Server V8.0 Fix Pack 10
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.29: Java SDK 1.6 SR13 FP2 Cumulative Fix for WebSphere Application Server
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.31: Java SDK 1.6 SR15 Cumulative Fix for WebSphere Application Server
7.0.0.35: Java SDK 1.6 SR16 FP1 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
Obtain the fix for this APAR.
APAR status
Closed as program error.
Error description
A load test of a Portal front-end application is executed. The application in the Portal front-end invokes EJBs over RMI/IIOP hosted by a clustered backend WebSphere for z/OS 7.0.0.21 Application server. During the test, a back-end server controller region experiences java/lang/OutOfMemoryError due to huge allocation request of around 2GB of char[]. - In addition, several variations of java.rmi.MarshalException: CORBA MARSHAL exceptions are reported by the application code running in the Portal front-end servant regions. The back-end WAS servers ASR1/ASR2 are running in 31 bit mode. ===== BACKEND ===== In the back-end controller SYSOUT we see this allocation: <af type="tenured" id="1" timestamp="Jun 21 20:39:55 2012" intervalms="0.000"> <minimum requested_bytes="2038957072"/> <time exclusiveaccessms="0.038" meanexclusiveaccessms="0.038" threads="0" lastthreadtid="0x00000008090AE700" /> <refs soft="23882" weak="29930" phantom="31" . . </af> When this occurs there is a headpdump, javacore and snap. JVMDUMP006I Processing dump event "allocation", detail "2038957072 bytes, type char[] - please wait. JVMDUMP032I JVM requested Heap dump using '/SYS/T/Websenv/home/Y/heapdump.20120940.703887.33333.0001.phd' in response to an event. The javacore for the controller shows call stack of the thread hitting OutOfMemoryError: "WebSphere:ORB.thread.pool t=00bc1cf0" J9VMThread: 0x0000000809079700, j9thread_t:0x00000008090785A0, java/lang/Thread: 0x0000000837279C50, state:R, prio=5 (native thread ID:0x1D00F500, native priority: 0x5, native policy:UNKNOWN) Java callstack: at com/ibm/rmi/iiop/CDRReader.readBytesForString(CDRReader. java:2296(Compiled Code))at com/ibm/rmi/iiop/CDRReader. readStringOrIndirection(CDRReader.java(Compiled Code)) at com/ibm/rmi/iiop/CDRReader.read_codebase_URL(CDRReader. java:2890) at com/ibm/rmi/iiop/CDRReader.fast_read_value(CDRReader. java:1910(Compiled Code)) at com/ibm/rmi/io/IIOPInputStream.readValueType(IIOPInputStream. java:2242(Compiled Code)) at com/ibm/rmi/io/IIOPInputStream.readValueArray( IIOPInputStream.java:1252(Compiled Code)) at com/ibm/rmi/io/IIOPInputStream.inputArray(IIOPInputStream. java:1187(Compiled Code)) at com/ibm/rmi/io/IIOPInputStream.continueSimpleReadObject( IIOPInputStream.java:585(Compiled Code)) at com/ibm/rmi/io/IIOPInputStream.simpleReadObjectLoop( IIOPInputStream.java:537(Compiled Code)) at com/ibm/rmi/io/IIOPInputStream.simpleReadObject( IIOPInputStream.java:475(Compiled Code)) at com/ibm/rmi/io/ValueHandlerImpl.readValue(ValueHandlerImpl. java:210(Compiled Code)) at com/ibm/rmi/iiop/CDRReader.read_value (CDRReader.java:2028(Compiled Code)) at com/bc/comput/vdd/_VDDsomethingZZ_Stub. getTransactions(_VDDsomethingZZ_Stub.java:529 (Compiled Code)) at com/bc/tunerset/vddsomethingZZportlet/util/ vddDestiny.getRangeTransactions(vddDestiny.java:102) at com/bc/tunerset/vddsomethingZZportlet/actions/ vddSumDo.loadRangeTransactions(vddSumDo.java:138) at com/bc/tunerset/vddsomethingZZportlet/actions/vddSumDo. doExecute(vddSumDo.java:113(Compiled Code)) at com/bc/tunerset/vddsomethingZZportlet/actions/BeginTo. execute(BeginTo.java:64(Compiled Code)) at com/ibm/portal/struts/portlet/WpRequestProcessor. processActionPerform(WpRequestProcessor.java:400 (Compiled Code)) at org/apache/struts/action/RequestProcessor.process( RequestProcessor.java:274(Compiled Code)) The java.lang.OutOfMemory error occurs in method com/ibm/rmi/iiop/CDRReader.readBytesForString while the ORB is demarshalling the supplied parameters. The data stream being parsed by the ORB appears to have been corrupted, and a very huge value is being interpreted as the current parameter's length. The ORB trace from the front-end portal server that originally marshalled this parameter shows the length being entered into the message was a 'reasonable' value. A TCPIP Packet Trace on this front-end portal server clearly shows that the length value in this IIOP message has been changed to a very huge number. This corruption must have happened between the time the message was built and handed to TCPIP for transmission to the back end server, and the time TCPIP actually sent the packets to the back-end server. In every case, the IIOP message being sent was 300 kilobytes or larger. This kind of damage was never noticed for messages with length less than TCPIP's default send buffer size. (Look in TCPIP's TCPCONFIG file to find the setting for TCPSENDBFRSIZE. Analysis of the corruption suggests that perhaps the "send" buffer containing the corrupted message was concurrently being used for a "receive" operation as well. =====FRONT END===== java.lang.NullPointerException at java.util.Date.getMillisOf(Date.java:950) at java.util.Date.compareTo(Date.java:970) ... java.lang.NullPointerException at com.user.name.type.sort.stuff.compare( stuff.java:74) at org.apache.commons.collections.comparators.ComparatorChain. compare(ComparatorChain.java:310) at java.util.Arrays.mergeSort(Arrays.java:2874) ... java.rmi.MarshalException: CORBA MARSHAL 0x4942f89a No; nested exception is: org.omg.CORBA.MARSHAL: Unable to read value from underlying bridge : null vmcid: IBM minor code: 89A completed: No at com.ibm.CORBA.iiop.UtilDelegateImpl.mapSystemException( UtilDelegateImpl.java:277) at javax.rmi.CORBA.Util.mapSystemException(Util.java:84) ... Caused by: org.omg.CORBA.MARSHAL: Unable to read value from underlying bridge : null vmcid: IBM minor code: 89A completed: No at com.ibm.rmi.iiop.CDRReader.read_value(CDRReader.java:2046) ... Caused by: java.lang.NegativeArraySizeException at com.ibm.rmi.iiop.CDRReader.readBytesForString( CDRReader.java:2296) at com.ibm.rmi.iiop.CDRReader.readRepIdOrIndirection( CDRReader.java:1019) -or- Caused by: com.ibm.rmi.io.FastIndirectionException: vmcid: 0x0 minor code: 0 completed: Maybe -or- Caused by: org.omg.CORBA.MARSHAL: No available data: Read beyond end of input stream vmcid: OMG minor code: 8 completed: Maybe at com.ibm.rmi.iiop.CDRReader.availableData(CDRReader.java:905) -or- Caused by: org.omg.CORBA.MARSHAL: RepID indirection @ ###### vmcid: IBM minor code: 8A3 completed: Maybe -or- Caused by: org.omg.CORBA.MARSHAL: No available data: Read beyond end of input stream vmcid: OMG minor code: 8 completed: Maybe at com.ibm.rmi.iiop.CDRReader.availableData(CDRReader.java:905) -or- Caused by: java.lang.NegativeArraySizeException at com.ibm.rmi.iiop.CDRReader.readBytesForString( CDRReader.java:2296) ... -or- Caused by: java.io.IOException: The RepId is not of an RMI value type (Expected RepId = IDL:omg.org/CORBA/WStringValue:1.0; Received RepId = RMI:org.omg.customRMI.java.util.Date:AC117E28FE36587A:686A81014B 597419) at com.ibm.rmi.util.RepositoryId.useFullValueDescription( RepositoryId.java:734) ... -or- Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1072) ... -or- Caused by: java.lang.ArrayIndexOutOfBoundsException at com.ibm.rmi.iiop.CDRInputStream.read_octet_array( CDRInputStream.java:408) -or- Caused by: org.omg.CORBA.MARSHAL: Undefined type information value in value_tag vmcid: IBM minor code: 899 completed: No at com.ibm.rmi.iiop.CDRReader.fast_read_value( CDRReader.java:1930) ... -or- Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: 0xc9c26000 minor code: A3B completed: Maybe at java.lang.J9VMInternals.newInstanceImpl(Native Method) ... -or- Caused by: org.omg.CORBA.MARSHAL: Requested stream format not supported vmcid: OMG minor code: 7 completed: Maybe at com.ibm.rmi.io.IIOPInputStream.setFormatVersion(IIOPInputStream. java:312)
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server V7.0, V8.0 and V8.5 * **************************************************************** * PROBLEM DESCRIPTION: An outbound IIOP request sent from an * * application running in a WebSphere * * Application Server for z/OS Servant * * becomes corrupted while being sent * * over TCP/IP. * **************************************************************** * RECOMMENDATION: * **************************************************************** In certain situations, the storage used for sending an outbound IIOP request can be released before TCP/IP has a chance to read all of the data. This means that TCP/IP may send a portion of the outbound request, the storage may then be reused and overwritten, and when TCP/IP reads the rest of the request the storage no longer contains the request data. The external symptom of this problem can vary depending on how the target server handles reading the corrupt data. In general, the target server will be unable to parse the inbound IIOP message and will respond to the client with an error message.
Problem conclusion
Code was fixed to prevent the early release of storage. APAR PM73621 is currently targeted for inclusion in WebSphere Application Server Fix Packs 7.0.0.29, 8.0.0.6, and 8.5.0.2. Please refer to URL: //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970 for Fix Pack availability.
Temporary fix
Comments
APAR Information
APAR number
PM73621
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2012-09-25
Closed date
2012-11-21
Last modified date
2013-07-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R700 PSY UK94926
UP13/06/20 P F306
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
Document Information
Modified date:
29 October 2021