PI86943: JAVA.LANG.INDEXOUTOFBOUNDSEXCEPTION RECEIVED AFTER ADDING A COLUMN ON ORC DATA

APAR status

• Closed as program error.

Error description

• SELECT statement will fail with
  java.lang.IndexOutOfBoundsException after adding additional
  column on ORC data.
  
  Steps to recreate:
  create external hadoop table bigsql.t1 ( col1 varchar(10) )
  stored as orc location '/user/bigsql/t1'
  DB20000I  The SQL command completed successfully.
  
  insert into bigsql.t1 values ('abc')
  DB20000I  The SQL command completed successfully.
  
  alter table bigsql.t1 add col2 varchar(10)
  DB20000I  The SQL command completed successfully.
  
  db2 "select * from t1"
  COL1       COL2
  ---------- ----------
  SQL5105N  The statement failed because a Big SQL component
  encountered an
  error. Component receiving the error: "BigSQL IO". Component
  returning the
  error: "UNKNOWN". Log entry identifier: "[BSL-3-62f91a21c]".
  SQLSTATE=58040
  
  2017-08-29 12:57:06,973 ERROR
  com.ibm.biginsights.bigsql.dfsrw.reader.DfsBaseReader [Master
  S:19.1001.1.0.0.316] : [BSL-3-62f91a21c] Exception raised\
  by Reader at node: 3 Scan ID: S:19.1001.1.0.0.316 Table:
  bigsql.t1 Spark: false VORC: true
  Exception Label: UNMAPPED(java.lang.IndexOutOfBoundsException:
  Index: 1, Size: 1)
  java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
          at java.util.ArrayList.rangeCheck(ArrayList.java:664)
          at java.util.ArrayList.get(ArrayList.java:440)
          at
  java.util.Collections$UnmodifiableList.get(Collections.java:1320
  )
          at
  org.apache.orc.mapred.OrcInputFormat.parseInclude(OrcInputFormat
  .java:69)
          at
  com.ibm.biginsights.bigsql.dfsrw.orc.DfsOrcReader.getRecordReade
  r(DfsOrcReader.java:412)
          at
  com.ibm.biginsights.bigsql.dfsrw.orc.DfsOrcReader$ReaderWorker.r
  un(DfsOrcReader.java:274)
          at java.lang.Thread.run(Thread.java:785)
  

Local fix

• - create a dummy table that has the ddl of the old table before
  it was altered. Make sure the location keyword refers to a
  new/different path.
  - hdfs dfs -mv <bad orc file> <new dummy table hdfs location>
  - insert into current_table select * from dummy_table, null,
  null
  - drop dummy_table
  By doing this, you can access all the data from the "bad" files
  and reload them into the current table by adding nulls for the
  missing columns.
  

Problem summary

• See error description
  

Problem conclusion

• The problem is fixed in Version 5.0.2.0 and later fix packs
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• Unknown
  

Fix information

• Fixed component name

  BIG SQL 4 BI

• Fixed component ID

  5725C09BQ

Applicable component levels

• R425 PSY

     UP