Topic
No replies
Karim34
Karim34
1 Post
ACCEPTED ANSWER

Pinned topic Java Heap Size

‏2013-06-26T14:04:24Z |

Hi! I've some problems about my hadoop configuration when I try to run my App on big text files.

I've already modify :

- mapred.child.java.opts to 2GB

-  io.sort.mb to 1GB

- mapred.job.reuse.jvm.num.tasks to 1

 

but I still have the following java heap size error:

Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.JaqlMain], main() threw exception, 
Java heap space
2013-06-26 15:14:48,047  WARN JaqlActionExecutor:542 - USER[biadmin] GROUP[-] TOKEN[] 
APP[Ticket_Problem_Analyzer_V4] JOB[0000000-130626151301207-oozie-biad-W] 
ACTION[0000000-130626151301207-oozie-biad-W@action1] 
Launcher exception: Java heap spacejava.lang.OutOfMemoryError: Java heap space        
java.util.Arrays.copyOfRange(Arrays.java:4174)        
at java.util.Arrays.copyOf(Arrays.java:3893)        
at com.ibm.jaql.util.FastPrintBuffer.overflow(FastPrintBuffer.java:54)        
at com.ibm.jaql.util.FastPrinter.write(FastPrinter.java:145)        
at com.ibm.jaql.util.FastPrinter.print(FastPrinter.java:238)        
at com.ibm.jaql.json.util.JsonUtil.printQuoted(JsonUtil.java:158)       
 at com.ibm.jaql.json.util.JsonUtil.printQuoted(JsonUtil.java:178)        
at com.ibm.jaql.io.serialization.text.def.StringSerializer.write(StringSerializer.java:54)        
at com.ibm.jaql.io.serialization.text.def.StringSerializer.write(StringSerializer.java:25)        
at com.ibm.jaql.io.serialization.text.EncodingMapFullSerializer.write(EncodingMapFullSerializer.java:63)        
at com.ibm.jaql.io.serialization.text.def.RecordSerializer.write(RecordSerializer.java:58)        
at com.ibm.jaql.io.serialization.text.def.RecordSerializer.write(RecordSerializer.java:29)        
at com.ibm.jaql.io.serialization.text.EncodingMapFullSerializer.write(EncodingMapFullSerializer.java:63)        
at com.ibm.jaql.io.serialization.text.TextFullSerializer.write(TextFullSerializer.java:41)        
at com.ibm.jaql.json.type.JsonUtil.print(JsonUtil.java:28)        
at com.ibm.jaql.json.type.JsonUtil.printToString(JsonUtil.java:122)        
at com.ibm.jaql.lang.ThresholdExceptionHandler.handleException(ThresholdExceptionHandler.java:39)        
at com.ibm.jaql.lang.expr.core.TransformExpr$1.moveNextRaw(TransformExpr.java:222)        
at com.ibm.jaql.lang.expr.core.Expr$JsonIteratorFromExpr.moveNext(Expr.java:915)        
at com.ibm.jaql.json.util.JsonIterator.hasNext(JsonIterator.java:157)        
at com.ibm.jaql.json.type.SpilledJsonArray.setCopy(SpilledJsonArray.java:505)        
at com.ibm.jaql.lang.expr.core.IterExpr.asArray(IterExpr.java:114)        
at com.ibm.jaql.lang.expr.core.IterExpr.evalRaw(IterExpr.java:102)        
at com.ibm.jaql.lang.expr.core.Expr.eval(Expr.java:810)        
at com.ibm.jaql.lang.core.Var.setEval(Var.java:411)        
at com.ibm.jaql.lang.expr.core.BindingExpr.evalRaw(BindingExpr.java:222)        
at com.ibm.jaql.lang.expr.core.Expr.eval(Expr.java:810)        
at com.ibm.jaql.lang.expr.core.DoExpr.evalRaw(DoExpr.java:145)        
at com.ibm.jaql.lang.expr.core.Expr.eval(Expr.java:810)        
at com.ibm.jaql.lang.expr.top.QueryExpr.evalRaw(QueryExpr.java:94)        
at com.ibm.jaql.lang.expr.core.Expr.eval(Expr.java:810)        
at com.ibm.jaql.lang.StreamPrinter.print(StreamPrinter.java:63)        
at com.ibm.jaql.lang.Jaql.run(Jaql.java:800)        
at com.ibm.jaql.lang.Jaql.run(Jaql.java:159)        
at com.ibm.jaql.util.shell.AbstractJaqlShell.run(AbstractJaqlShell.java:55)        
at com.ibm.jaql.util.shell.AbstractJaqlShell.main(AbstractJaqlShell.java:93)        
at com.ibm.jaql.util.shell.JaqlShell.main(JaqlShell.java:65)        
at org.apache.oozie.action.hadoop.JaqlMain.runJaqlJob(JaqlMain.java:235)        
at org.apache.oozie.action.hadoop.JaqlMain.run(JaqlMain.java:177)        
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)        
at org.apache.oozie.action.hadoop.JaqlMain.main(JaqlMain.java:35)        
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)        
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)        
at java.lang.reflect.Method.invoke(Method.java:611)        
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)        
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)        
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)        
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)        
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)        
at java.security.AccessController.doPrivileged(AccessController.java:310)        
at javax.security.auth.Subject.doAs(Subject.java:573)        
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1144)        
at org.apache.hadoop.mapred.Child.main(Child.java:249)

Any body have an idea? Thanks for your help!

Updated on 2013-06-26T14:10:50Z at 2013-06-26T14:10:50Z by Karim34