PI94314: BIGSQL SCHEDULER DOES NOT RETRY LOADING METADATA WHEN IT FINDS A CATALOG EXCEPTION AFTER AN INSERT

APAR status

• Closed as fixed if next.

Error description

• You are executing a series of INSERT statements into an hadoop
  table ( myschema.tab1 for example) on a SpectrumScale based
  BigInsights environment.
  INSERTs work fine until, at some point, they start failing. At
  the same time in the BigSQL scheduler log ( bigsql-sched.log)
  exceptions similar to the following are logged:
  
  Caused by: java.io.FileNotFoundException: No such file or
  directory:
  /bigpfs/apps/hive/warehouse/myschema.db/tab1/TEMP_1516620150285_
  1875897054_201801220332965
  at
  org.apache.hadoop.hdfs.server.namenode.GPFSNative.getFileStatusN
  ative(Native Method)
  at
  org.apache.hadoop.hdfs.server.namenode.GPFSNative.getFileStatus(
  GPFSNative.java:239)
  at
  org.apache.hadoop.hdfs.server.namenode.GPFSNamesystemV0.getListi
  ng(GPFSNamesystemV0.java:1162)
  at
  org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getList
  ing(NameNodeRpcServer.java:1008)
  at
  org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi
  deTranslatorPB.getListing(ClientNamenodeProtocolServerSideTransl
  atorPB.java:634)
  (...some calls skipped...)
  
  at
  sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
  Method)
  at
  sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCons
  tructorAccessorImpl.java:88)
  at
  sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delega
  tingConstructorAccessorImpl.java:57)
  at
  java.lang.reflect.Constructor.newInstance(Constructor.java:437)
  at
  org.apache.hadoop.ipc.RemoteException.instantiateException(Remot
  eException.java:106)
  at
  org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(Remo
  teException.java:73)
  at
  org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2088)
  (...some calls skipped...)
  at
  org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Distribu
  tedFileSystem.java:860)
  at
  com.thirdparty.cimp.catalog.HdfsTable.createPartition(HdfsTable.
  java:892)
  ... 9 more
  
  From this point an on Scheduler cannot load metadata for the
  offending table anymore. This problem can happen if you use
  SpectrumScale and the HDFS transparency connector. It won't
  happen on a HDFS based system.
  

Local fix

• Restart the BigSQL scheduler issuing the commands
  
  bigsql stop -scheduler
  bigsql start -scheduler
  
  from the BigSQL head node, as user "bigsql". Note that while
  the scheduler is restarting running statements may fail.
  

Problem summary

• See error description
  The problem is fixed in Version 5.0.3 and later fix packs.
  

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• Unknown
  

Fix information

Applicable component levels

• R425 PSY

     UP