LI72659: CASTING LITERAL CONSTANT TO A LARGE SIZE VARCHAR CORRUPTS MEMOR

DB2 Version 9.5 Fix Pack 3b for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 1  for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 2a for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 2  for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 4  for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 3a for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 3 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 4a for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 5 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 6a for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 7 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 8 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 9 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 10 for Linux, UNIX, and Windows

APAR status

• Closed as program error.

Error description

• Casting literal constant to a large size varchar corrupts
  memory.
  Following test query corrupts DB2 process stack memory and may
  crash the instance:
  CREATE TABLE PS_SET_CNTRL_REC
     (
     SETCNTRLVALUE CHAR(20) NOT NULL ,
     REC_GROUP_ID  CHAR(10) NOT NULL ,
     RECNAME       CHAR(15) NOT NULL ,
     SETID         CHAR(5) NOT NULL
     );
  
  insert into PS_SET_CNTRL_REC values ( '111', '222', '333', '444'
  ) ;
  
  select
     SETID
  from
     ps_set_cntrl_rec
  where
     substr(cast(('R0551') as varchar(1000)),1,1)='' ;
  
  This query may look unusual, but it is a part of Peoplesoft
  application the customer has and it does
  crash their engine consistently on v8 FP14. On Viper FP2, FP3 we
  do not see the crash, but memory
  is still being corrupted - so different memory layout may be
  coming into play.
  
  The corruption happens here, when we try to cast R0551 value
  into varchar(1000):
  
  LINT SQLZ_DYNLINK sqlnq_encap_doit(
  ..............................................
  
      case SQLNQ_ENCAP_STRING_COMPARE:
      {
        sqlzCollID collID;
  //@w035360sab
        UCHAR *pOldConstant;
  //@d80961kmw
        UCHAR  NewConstantBuf[SQL_MAXCHAR];
  //@d80961kmw
        LINT   OldConstantLen;
  //@d80961kmw   //@d120628spr
        int    NewConstantLen;
  //@d80961kmw
  .................................................
  
          // make a new constant from the old constant
          // and blank-pad it (if needed) to the other operand's
  length
          //
          sqlomcpy( NewConstantBuf, pOldConstant, OldConstantLen
  );
          sqlomset( NewConstantBuf + OldConstantLen,
                    SQLZ_ASCII_BLANK,
                    NewConstantLen - OldConstantLen );
  
  The problem is that size of the NewConstantBuf is 254
  characters, but NewConstantLen is actually
  1002, so sqlomset() overwrites the memory beyond NewConstantBuf.
  
  We have tried this on Viper II and we get database crashing with
  -901. This is due to the fact that we
  do only have 'n' builds for Viper II and on 'n' builds we have a
  following check:
  
          #ifdef DB2_DEBUG
          // sanity check that the new constant will
          // not be longer than the size of our new constant
  buffer
          //
          if ( NewConstantLen > sizeof(NewConstantBuf) )
          {
            rc = SQLNN_E_BADNEWS;
            sqlnn_tra_push( rc, func_name, 1503,
                            "NewConstantBuf too small" );
            sqlnn_erds( SQLT_SQLNQ, SQLNQ_ENCAP_DOIT, SQL_RC_E901,
  1503, 1,
                        sizeof("NewConstantBuf too small")-1,
                               "NewConstantBuf too small" );
            goto exit;
          }
          #endif
  
  So, the problem does exist on Viper II as well.
  
  ClearQuest defect is  wsdbu00191949 for this issue.
  

Local fix

• Limit casting literal string to varchar(254) or less.
  

Problem summary

• see problem description
  

Problem conclusion

• First fixed in DB2 UDB Version 9.5, FixPak 1
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  LI72468

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  DB2 UDB WSE LIN

• Fixed component ID

  5765F3504

Applicable component levels

• R910 PSY

     UP

• R950 PSY

     UP