IBM Support

Transformer job fails with message: Operator Pivot_03 failed due to org.apache.spark.SparkException

Troubleshooting


Problem

By checking the transformer.log, you can find the following stack trace:

ERROR DataTransformerLauncher - Error stack trace:
com.streamsets.datatransformer.dag.error.OperatorFailureException: Operator Pivot_03 failed due to org.apache.spark.SparkException, check Driver Logs for further information
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner.com$streamsets$datatransformer$dag$BaseBatchDAGRunner$$generateOperatorFailure(BaseBatchDAGRunner.scala:713)
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner$$anon$3.call(BaseBatchDAGRunner.scala:773)
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner$$anon$3.call(BaseBatchDAGRunner.scala:749)
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner$$anonfun$materialize$2$$anon$4.call(BaseBatchDAGRunner.scala:790)
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner$$anonfun$materialize$2$$anon$4.call(BaseBatchDAGRunner.scala:786)
	at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:44)
	at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:25)
	at com.streamsets.datatransformer.dag.BaseBatchDAGRunner$$anon$2.call(BaseBatchDAGRunner.scala:742)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 29.0 failed 4 times, most recent failure: Lost task 1.3 in stage 29.0 (TID 61, some.server.net, executor 10): ExecutorLostFailure (executor 10 exited caused by one of the running tasks) Reason: Container from a bad node: container_f075_1702234099237_607753_01_000011 on host: some.server.net. Exit status: 143. Diagnostics: [2024-02-06 07:52:15.958]Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143. 
Killed by external signal

The exit code 143 is usually related to Memory/GC issues.

In YARN clusters, you may find in the driver logs a message similar to the following:

INFO org.apache.flink.yarn.YarnResourceManager - Closing TaskExecutor connection container_f075_1702234099237_607753_01_000011 because: Container [pid=10876,containerID=container_f075_1702234099237_607753_01_000011] is running beyond physical memory limits. Current usage: 2.1 GB of 2.1 GB physical memory used; 3.0 GB of 4.9 GB virtual memory used. Killing container. Dump of the process-tree for container_f075_1702234099237_607753_01_000011

Symptom

Your job running on StreamSets Transformer for Spark fails with the message: Operator Pivot_03 failed due to org.apache.spark.SparkException, check Driver Logs for further information. The job can run successfully in other executions but still shows the error message in a few runs.

Resolving The Problem

Review the driver logs of your application for the reason for the failure.

In case of exceeding the memory limits, you can increase the memory of the application.

You can define the properties spark.driver.memoryOverhead and spark.executor.memoryOverhead in your Extra Spark Configuration. If they are already defined, then increase the values. It is suggested to set those values as 10% of the value for spark.executor.memory.

Document Location

Worldwide

[{"Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSZUHT0","label":"IBM StreamSets Transformer for Spark"},"ARM Category":[{"code":"","label":""}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]

Document Information

Modified date:
15 March 2025

UID

ibm17186105

Page Feedback