PH19658: UNABLE TO READ FROM BIGSQL HADOOP TABLES WITH CUSTOM STORAGE HANDLER

APAR status

• Closed as fixed if next.

Error description

• Customer is connecting their Kafka Message Queue to Big SQL
  Hadoop Table. As we allow custom external storage handler in
  stored by clause,this looks like it should work.
  Creating the hadoop table is ok, however reading from it fails
  as Big SQL is unable to get the metadata for the table, due to a
  null pointer exception
  
  
  bigsql-sched.log shows
  
  
  2019-11-26 16:30:39,718 ERROR
  com.ibm.biginsights.bigsql.scheduler.server.cache.DescriptorTabl
  eCache [pool-1-thread-3] : [SCL-0-6a85603e6] Incomplete-table:
  Front-End-Exception. schemaName=bigsql tableName=big_topic1.
  com.thirdparty.cimp.catalog.TableLoadingException: Failed to
  load metadata for table: big_topic1
  at
  com.thirdparty.cimp.catalog.HdfsTable.load(HdfsTable.java:1427)
  at
  com.thirdparty.cimp.catalog.TableLoader.load(TableLoader.java:85
  )
  at
  com.thirdparty.cimp.catalog.TableLoadingMgr$3.call(TableLoadingM
  gr.java:296)
  at
  com.thirdparty.cimp.catalog.TableLoadingMgr$3.call(TableLoadingM
  gr.java:267)
  at java.util.concurrent.FutureTask.run(FutureTask.java:277)
  at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec
  utor.java:1160)
  at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:635)
  at java.lang.Thread.run(Thread.java:812)
  Caused by: java.lang.NullPointerException
  at
  com.google.common.base.Preconditions.checkNotNull(Preconditions.
  java:191)
  at
  com.thirdparty.cimp.catalog.HdfsFileFormat.fromJavaClassName(Hdf
  sFileFormat.java:108)
  at
  com.thirdparty.cimp.catalog.HdfsTable.markNativeStatus(HdfsTable
  .java:1620)
  at
  com.thirdparty.cimp.catalog.HdfsTable.load(HdfsTable.java:1291)
  ... 7 more
  2019-11-26 16:30:39,808 WARN com.thirdparty.cimp.catalog.Table
  [pool-7-thread-2] : Unknown type of table found during
  Table.fromMetastoreTable(). Trying hive-custom-file-format
  2019-11-26 16:30:39,808 ERROR
  com.thirdparty.cimp.catalog.HdfsTable [pool-7-thread-2] :
  Exception during HdfsTable.load()
  java.lang.NullPointerException
  at
  com.google.common.base.Preconditions.checkNotNull(Preconditions.
  java:191)
  at
  com.thirdparty.cimp.catalog.HdfsFileFormat.fromJavaClassName(Hdf
  sFileFormat.java:108)
  at
  com.thirdparty.cimp.catalog.HdfsTable.markNativeStatus(HdfsTable
  .java:1620)
  at
  com.thirdparty.cimp.catalog.HdfsTable.load(HdfsTable.java:1291)
  at
  com.thirdparty.cimp.catalog.TableLoader.load(TableLoader.java:85
  )
  at
  com.thirdparty.cimp.catalog.TableLoadingMgr$3.call(TableLoadingM
  gr.java:296)
  at
  com.thirdparty.cimp.catalog.TableLoadingMgr$3.call(TableLoadingM
  gr.java:267)
  at java.util.concurrent.FutureTask.run(FutureTask.java:277)
  at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec
  utor.java:1160)
  at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:635)
  at java.lang.Thread.run(Thread.java:812)
  2019-11-26 16:30:40,025 WARN com.thirdparty.cimp.catalog.Table
  [pool-7-thread-3] : Unknown type of table found during
  Table.fromMetastoreTable(). Trying hive-custom-file-format
  2019-11-26 16:30:40,026 ERROR
  com.thirdparty.cimp.catalog.HdfsTable [pool-7-thread-3] :
  Exception during HdfsTable.load()
  java.lang.NullPointerException
  at
  com.google.common.base.Preconditions.checkNotNull(Preconditions.
  java:191)
  at
  com.thirdparty.cimp.catalog.HdfsFileFormat.fromJavaClassName(Hdf
  sFileFormat.java:108)
  at
  com.thirdparty.cimp.catalog.HdfsTable.markNativeStatus(HdfsTable
  .java:1620)
  at
  com.thirdparty.cimp.catalog.HdfsTable.load(HdfsTable.java:1291)
  at
  com.thirdparty.cimp.catalog.TableLoader.load(TableLoader.java:85
  )
  at
  com.thirdparty.cimp.catalog.TableLoadingMgr$3.call(TableLoadingM
  gr.java:296)
  at
  com.thirdparty.cimp.catalog.TableLoadingMgr$3.call(TableLoadingM
  gr.java:267)
  at java.util.concurrent.FutureTask.run(FutureTask.java:277)
  at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec
  utor.java:1160)
  at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:635)
  at java.lang.Thread.run(Thread.java:812)
  2019-11-26 16:30:40,098 WARN com.thirdparty.cimp.catalog.Table
  [pool-7-thread-4] : Unknown type of table found during
  Table.fromMetastoreTable(). Trying hive-custom-file-format
  2019-11-26 16:30:40,099 ERROR
  com.thirdparty.cimp.catalog.HdfsTable [pool-7-thread-4] :
  Exception during HdfsTable.load()
  java.lang.NullPointerException
  at
  com.google.common.base.Preconditions.checkNotNull(Preconditions.
  java:191)
  at
  com.thirdparty.cimp.catalog.HdfsFileFormat.fromJavaClassName(Hdf
  sFileFormat.java:108)
  at
  com.thirdparty.cimp.catalog.HdfsTable.markNativeStatus(HdfsTable
  .java:1620)
  at
  com.thirdparty.cimp.catalog.HdfsTable.load(HdfsTable.java:1291)
  at
  com.thirdparty.cimp.catalog.TableLoader.load(TableLoader.java:85
  )
  at
  com.thirdparty.cimp.catalog.TableLoadingMgr$3.call(TableLoadingM
  gr.java:296)
  at
  com.thirdparty.cimp.catalog.TableLoadingMgr$3.call(TableLoadingM
  gr.java:267)
  at java.util.concurrent.FutureTask.run(FutureTask.java:277)
  at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec
  utor.java:1160)
  at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe
  cutor.java:635)
  at java.lang.Thread.run(Thread.java:812)
  

Local fix

• ----------------------------------------------------------
  Workaround #1: Use ROW FORMAT SERDE with KafkaSerDe class
  ----------------------------------------------------------
  CREATE EXTERNAL HADOOP TABLE kafka_tbl2
  ( packageId string, account string, accountExternalId string,
  userId string , partnerId string , partnerCode string,
  transactions string )
  ROW FORMAT SERDE 'org.apache.hadoop.hive.kafka.KafkaSerDe'
  TBLPROPERTIES
  (
  'kafka.topic' = 'topic.v1',
  'kafka.bootstrap.servers'='serverserverserver',
  'kafka.consumer.security.protocol'='SSL',
  'kafka.consumer.ssl.truststore.location'='/cert/truststore.jks',
  'kafka.consumer.ssl.truststore.password'='x',
  'kafka.consumer.ssl.keystore.type' = 'PKCS12',
  'kafka.consumer.ssl.keystore.location' = '/cert/client.p12',
  'kafka.consumer.ssl.keystore.password' = 'x'
  );
  
  ----------------------------------------------------------------
  ----------------
  Workaround #2: Use STORED AS with KafkaInputFormat and
  KafkaOutputFormat classes
  ----------------------------------------------------------------
  ----------------
  CREATE EXTERNAL HADOOP TABLE kafka_tbl3
  ( packageId string, account string, accountExternalId string,
  userId string , partnerId string , partnerCode string,
  transactions string )
  STORED AS
  INPUTFORMAT
          'org.apache.hadoop.hive.kafka.KafkaInputFormat'
      OUTPUTFORMAT
          'org.apache.hadoop.hive.kafka.KafkaOutputFormat'
  TBLPROPERTIES
  (
  'kafka.topic' = 'topic.v1',
  'kafka.bootstrap.servers'='serverserverserver',
  'kafka.consumer.security.protocol'='SSL',
  'kafka.consumer.ssl.truststore.location'='/cert/truststore.jks',
  'kafka.consumer.ssl.truststore.password'='x',
  'kafka.consumer.ssl.keystore.type' = 'PKCS12',
  'kafka.consumer.ssl.keystore.location' = '/cert/client.p12',
  'kafka.consumer.ssl.keystore.password' = 'x'
  );
  

Problem summary

• Please see problem discription.
  

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels