PI12718: HCATALOG PERIODICALLY STOPS WORKING DUE TO SERVER CRASHES AFTER A CERTAIN NUMBER OF CONNECTIONS IS ESTABLISHED.

APAR status

• Closed as program error.

Error description

• Hcatalog periodically stops working due to server crashes after
  a certain number of connections is established.
  
  Client exception:
  
  14/02/13 09:57:02 ERROR hive.log: Got exception:
  org.apache.thrift.transport.TTransportException null
  14/02/13 09:57:02 ERROR hive.log:
  org.apache.thrift.transport.TTransportException
          at
  org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTra
  nsport.java:132)
          at
  org.apache.thrift.transport.TTransport.readAll(TTransport.java:8
  4)
          at
  org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtoc
  ol.java:378)
          at
  org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtoc
  ol.java:297)
          at
  org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBin
  aryProtocol.java:204)
          at
  org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java
  :69)
          at
  org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.
  recv_get_all_databases(ThriftHiveMetastore.java:442)
          at
  org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.
  get_all_databases(ThriftHiveMetastore.java:430)
          at
  org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatab
  ases(HiveMetaStoreClient.java:613)
          at TestHcat.doit(TestHcat.java:24)
          at TestHcat.main(TestHcat.java:15)
  
  Exception in thread "main" MetaException(message:Got exception:
  org.apache.thrift.transport.TTransportException null)
          at
  org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaE
  xception(MetaStoreUtils.java:785)
          at
  org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatab
  ases(HiveMetaStoreClient.java:615)
          at TestHcat.doit(TestHcat.java:24)
          at TestHcat.main(TestHcat.java:15)
  
  Server exception (from hive.log)
  
  2014-02-13 08:28:27,866 INFO  metastore.HiveMetaStore
  (HiveMetaStore.java:logInfo(385)) - 47: get_all_databases
  2014-02-13 08:28:27,870 ERROR server.TThreadPoolServer
  (TThreadPoolServer.java:run(182)) - Error occurred during
  processing of message.
  javax.jdo.JDOFatalUserException: Persistence Manager has been
  closed
          at
  org.datanucleus.jdo.JDOPersistenceManager.assertIsOpen(JDOPersis
  tenceManager.java:2088)
          at
  org.datanucleus.jdo.JDOPersistenceManager.currentTransaction(JDO
  PersistenceManager.java:305)
          at
  org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(Obj
  ectStore.java:294)
          at
  org.apache.hadoop.hive.metastore.ObjectStore.getDatabases(Object
  Store.java:488)
          at
  org.apache.hadoop.hive.metastore.ObjectStore.getAllDatabases(Obj
  ectStore.java:522)
          at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
  Source)
          at
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
  AccessorImpl.java:37)
          at java.lang.reflect.Method.invoke(Method.java:611)
          at
  org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(Retryin
  gRawStore.java:111)
          at $Proxy1.getAllDatabases(Unknown Source)
          at
  org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_al
  l_databases(HiveMetaStore.java:660)
          at
  org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Process
  or$get_all_databases.getResult(ThriftHiveMetastore.java:4749)
          at
  org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Process
  or$get_all_databases.getResult(ThriftHiveMetastore.java:4737)
          at
  org.apache.thrift.ProcessFunction.process(ProcessFunction.java:3
  2)
          at
  org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
          at
  org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(
  TSetIpAddressProcessor.java:48)
          at
  org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TTh
  readPoolServer.java:176)
          at
  java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoo
  lExecutor.java:897)
          at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:919)
          at java.lang.Thread.run(Thread.java:738)
  
  RTC defect 61244 has created to track this issue.
  

Local fix

• Once it failes, restart HCatalog server using
  $BIGINSIGHTS_HOME/hcatalog/sbin/hcat_server.sh.
  

Problem summary

• See problem description
  

Problem conclusion

• This problem is fixed in Version 2.1.2 and later fix packs.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  INFO BIGINSIGHT

• Fixed component ID

  5725C0900

Applicable component levels

• R212 PSY

     UP