Spark dirver reported OutOfMemoryError

Troubleshooting

Problem

Spark reported the below error in the driver log, executor in WAIT status.

19/07/11 18:53:43 WARN DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1565110817-10.161.67.203-1521280878051:blk_1440726264_367066900
java.io.EOFException: Premature EOF: no length prefix available
     at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2282)
     at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:733)
19/07/11 18:53:43 WARN DFSClient: Error Recovery for block BP-1565110817-10.161.67.203-1521280878051:blk_1440726264_367066900 in pipeline DatanodeInfoWithStorage[10.161.67.103:50010,DS-4ab9c888-29b3-4b5e-840e-e2253d930c2c,DISK], DatanodeInfoWithStorage[10.161.67.124:50010,DS-3c5fa367-befc-474d-94c5-7f382e024de6,DISK], DatanodeInfoWithStorage[10.161.67.128:50010,DS-feabd3ed-5141-46b2-a8bf-dacbfb6ca593,DISK]: bad datanode DatanodeInfoWithStorage[10.161.67.103:50010,DS-4ab9c888-29b3-4b5e-840e-e2253d930c2c,DISK]
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2019/07/11 19:48:30 - please wait.
JVMDUMP032I JVM requested System dump using '/home/tianyan/redis_forecast/core.20190711.194830.31121.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/tianyan/redis_forecast/core.20190711.194830.31121.0001.dmp
JVMDUMP032I JVM requested Heap dump using '/home/tianyan/redis_forecast/heapdump.20190711.194830.31121.0002.phd' in response to an event
JVMDUMP010I Heap dump written to

Document Location

Worldwide

[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SS4H63","label":"IBM Spectrum Conductor"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB77","label":"Automation Platform"}}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Was this topic helpful?

Document Information

More support for:
IBM Spectrum Conductor

Software version:
All Versions

Operating system(s):
Linux

Document number:
966849

Modified date:
22 August 2019

UID

ibm10966849

IBM Support

Tips