IBM Support

Resolving an exceeded max jobconf size error in Hive

Troubleshooting


Problem

You run a query in Hive to join two large tables, and you receive an error: IOException: Exceeded max jobconf size.

Symptom

You receive the following error message:

select count(*) from rq_feature_detail A join vclaim_tab B where A.
country=B.country and A.year=B.year;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.
IOException: Exceeded max jobconf size: 7374812 limit: 5242880
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3944)
at org.apache.hadoop.mapred.EnhancedJobTracker.submitJob
(EnhancedJobTracker.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(AccessController.java:
310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at org.apache.hadoop.security.UserGroupInformation.doAs
(UserGroupInformation.java:1144)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
Caused by: java.io.IOException: Exceeded max jobconf size: 7374812
limit: 5242880
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:
408)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3942)
... 12 more

[{"Product":{"code":"SSCRJT","label":"IBM Db2 Big SQL"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Component":"Open Source Tools","Platform":[{"code":"PF016","label":"Linux"}],"Version":"2.0.0;2.1.0;1.4.0;1.3.0","Edition":"Enterprise Edition;Basic Edition","Line of Business":{"code":"LOB76","label":"Data Platform"}}]

Log InLog in to view more of this document

This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.

Document Information

Modified date:
08 April 2021

UID

swg21642231