Question & Answer
Question
We are currently using Modeler to access Apache Hadoop via Apache Hive ODBC native driver.
It was determined the resources required for the query request have increased to the point the query is failing due to a conflict of resources with other running jobs for example:
Error: Connected to server: nlvmad425.nwie.net:28056, session: 34698-52077 Stream execution started Stream optimisation complete (20 rules applied) HY000[35] [Hortonworks][Hardy] (35) Error from server: error code: '1' error message:
'Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask'.
Stream execution complete, Elapsed=65.0 sec, CPU=0.31 sec
Execution was interrupted HY000[35] [Hortonworks][Hardy] (35) Error from server: error code: '1' error message: 'Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask'. com.spss.psapi.session.SessionException: HY000[35] [Hortonworks][Hardy] (35)
Error from server: error code: '1' error message: 'Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask'.
at com.spss.clementine.psapi.task.TaskManager.execute(Unknown Source) at com.spss.clementine.psapi.task.SessionTaskManager.execute(Unknown Source)
at com.spss.clementine.session.DefaultSession.run(Unknown Source) at com.spss.psapi.ra.internal.PSAPIManagedConnection.run(PSAPIManagedConnection.java:919)
at com.spss.psapi.ra.internal.SessionImpl.run(SessionImpl.java:377)
It was determined and tested via another application that these parameter settings for the query will limit the resources and allow the job to complete.
hive.tez.container.size=16384
hive.tez.java.opts=Xmx13107m
hive.task.resource.memory.mb=8096
hive.am.resource.memory.mb=8096
hive.tez.java.opts=Xmx13107m
hive.task.resource.memory.mb=8096
hive.am.resource.memory.mb=8096
Is there a way to implement these parameters within the Modeler stream so that the resource limiting parameters will be honored on the Hadoop side?
[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SS3RA7","label":"IBM SPSS Modeler"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB76","label":"Data Platform"}}]
Log InLog in to view more of this document
This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.
Was this topic helpful?
Document Information
Modified date:
01 May 2019
UID
ibm10883214