JR33566: USING ODBC ENTERPRISE TO READ COLUMNIZE UTF-8 SIMPLIFIED CHINESE DATA FROM DB2 BACK INTO DATASTAGE BUT ARE CORRUPTED.

APAR status

• Closed as program error.

Error description

• Information Server 8.1 on AIX
  
  In DataStage, the customer is able to insert columnize UTF-8
  simplified Chinese data into a VARCHAR column in a UTF-8 DB2
  data base using an ODBC Enterprise stage.   Using the ODBC
  Enterprise stage attempt to read the data back into DataStage
  but are observing data corruption.  The customer has updated
  Data Direct drivers. Tried reading the data in as a VARCHAR and
  using a string_to_ustring conversion in a transfortmer without
  success.
  

Local fix

• As a workaround the customer can read the data in as vargraphic
  e.g
  
  select ADMIN_SYS_TP_CD, CAST(ADMIN_CLIENT_ID as vargraphic(20))
  ADMIN_CLIENT_ID from IS_ADMINCLIENT
  
  In this case the ODBC stage should have ADMIN_CLIENT_ID as
  nvarchar .
  
  This fix is included in 8.0.1 fix pack 3
  

Problem summary

• ****************************************************************
  USERS AFFECTED:
  All platforms , jobs using ODBC EE to read UTF-8 characters
  ****************************************************************
  PROBLEM DESCRIPTION:
  Using the ODBC Enterprise stage to read columnize UTF-8
  simplified Chinese data from DB2 back into DataStage but are
  observing data truncation.
  ****************************************************************
  RECOMMENDATION:
  Install patch for JR33566
  The fix has been included in the IS 8.1 FP1. Apply the FixPack1
  for IS 8.1.
  ****************************************************************
  

Problem conclusion

• When UTF-8 data is stored in char/varchar columns, ODBC EE is
  unable to read the data correctly as it fetches the data as
  SQL_C_CHAR type.
  
  Implemented option Use Ustrings, to read all char/varchar
  columns as unicode strings.
  When this option is selected , char/varchar columns are
  retrieved as SQL_C_WCHAR type.
  
  Note: Since we are attempting to read varchar/char columns as
  nvarchar/nchar , hence the output will be inflated
  nvarchar/nchar columns.
  
  e.g if the user is writing nvarchar(20) to a column of
  varchar(80), then it will be read back as nvarchar(80) . This
  is because the application cannot make a guess about the number
  of utf8 characters in the column. It could be from 20
  charactersto 80 characters.
    If the user changes the column definition in the GUI to
   nvarchar(20), they will see a warning message like
  
         "Implicit conversion from source type
  "ustring[max=80]" to     result type "ustring[max=20]":
  Possible truncation of variable  length string."
  
      To get the correct column
  definition, the customer should load  the imported table
  definition and check the box that says      "Ensure all char
  columns use unicode", whenever the option  Use UStrings is
  set to true.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• ODBCEE
  

Fix information

• Fixed component name

  WIS DATASTAGE

• Fixed component ID

  5724Q36DS

Applicable component levels

• R801 PSN

     UP