PH04201: SQL WITH A VERY LONG IN-LIST CLAUSE AGAINST PARQUET TABLES MAY CAUSE JAVA.LANG.STACKOVERFLOWERROR IN WORKER NODE

APAR status

• Closed as program error.

Error description

• If you run a query against a parquet table and the query
  contains a very long IN ( ) clause, like for example
  SELECT  * FROM my_parquet_table
  WHERE   mycolumn NOT IN ( 'Val1','Val2', ... list with many ,
  e.g. 3000+ values ...)
  The statement may fail with the error
  SQL0973N  Not enough storage is available in the "BigSQL IO"
  heap or stack to process the statement.  SQLSTATE=57011
  and in the bigsql.log file in the worker node affected you will
  notice a java.lang.StackOverflowError exception associated to a
  very deep stack as in:
  2018-10-15 07:32:46,594 ERROR
  com.ibm.biginsights.bigsql.dfsrw.reader.DfsBaseReader
  [Master-4-S:5.1001.1.0.0.1433] : [BSL-4-67824c101] Exception
  raised by Reader at node: 4 Scan ID: S:5.1001.1.0.0.1433 Table:
  w.w1 Spark: false VORC: false VPQ: true VAVRO: false VTEXT:
  false VRCFILE: false VANALYZE: false
  Exception Label: UNMAPPED(java.lang.StackOverflowError)
  java.lang.StackOverflowError
  at
  java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.jav
  a:947)
  at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:356)
  at
  java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
  1160)
  at
  java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream
  .java:1615)
  at
  java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.ja
  va:1576)
  at
  java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStrea
  m.java:1499)
       ... about 10000 Lines of calls to writeSerialData() and
  writeOrdinaryObject()
        ...
  at
  parquet.hadoop.util.SerializationUtil.writeObjectToConfAsBase64(
  SerializationUtil.java:64)
  at
  parquet.hadoop.ParquetInputFormat.setFilterPredicate(ParquetInpu
  tFormat.java:186)
  (...)
  

Local fix

• Try to rewrite the query using a smaller IN clause or avoiding
  or using alternative clauses (for example using a support table
  as in   IN (SELECT ... FROM )
  

Problem summary

• See error description
  

Problem conclusion

• The problem is fixed in Version 5.0.4 and later fix packs.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• Unknown
  

Fix information

• Fixed component name

  IBM BIG SQL

• Fixed component ID

  5737E7400

Applicable component levels