IT35412: WRONG ESCAPING OF CHARACTERS IN BSON FIELDS GENERATED BY GENBSONFUNCTION

APAR status

• Closed as duplicate of another APAR.

Error description

• The handling of escape sequences within the BSON functions has
  had issues (see APAR IT28448 and APAR IT33799).
  There still seems to be a problem with the GenBSON function.
  
  If we try to do:
  
  Short example to reproduce the problem:
  
  echo "UNLOAD TO 'test.unl' SELECT data, data::JSON as json,
  bson_value_lvarchar(data, 'test') as bvstring, teststring as
  output_wished FROM (SELECT test as teststring,
  genbson(ROW(test),1,1) as data FROM (SELECT 'abc/def\ghi\"jkl'
  as test FROM systables where tabid = 1))" | dbaccess -e
  sysmaster
  
  This shows that the GenBSON function returns the following byte
  representation:
  
  22000000027465737400130000006162635c2f6465665c5c6768695c226a6b6c
  0000
  
  This sequence is wrong.  The correct sequence is:
  
  1f000000027465737400100000006162632f6465665c676869226a6b6c0000
  
  Note that there are 3 additional backslash bytes (0x5C) inside
  the BSON byte representation (despite the length changes from
  0x00000022 to 0x0000001f and 0x00000013 to 0x00000010). Those
  additional backslash bytes seem to have been added as encoding
  the JSON string in a 1:1 manner - but these are really only
  needed to create a valid JSON representation. So these bytes
  should under no circumstances be stored inside the BSON field
  BUT need to be added if handling the field with the ::JSON cast
  (as well as that they should be discarded if a JSON string is
  presented to the database and written into the BSON object).
  
  If we "omit" the GenBSON function for this test and present the
  JSON (obviously with the correct needed escaping) and use this
  command:
  
  echo "UNLOAD TO 'test.unl' SELECT data, data::JSON as json,
  bson_value_lvarchar(data, 'test') as bvstring FROM (SELECT
  '{\"test\":\"abc\\/def\\\\ghi\\\"jkl\"}'::JSON::BSON as data
  FROM systables where tabid = 1)" | dbaccess -e sysmaster
  
  We still see the same problem:
  
  Database selected.
  
  UNLOAD TO 'test.unl' SELECT data, data::JSON as json,
  bson_value_lvarchar(data, 'test') as bvstring FROM (SELECT
  '{"test":"abc\/def\\ghi\"jkl"}'::JSON::BSON as data FROM
  systables where tabid = 1)
  1 row(s) unloaded.
  Database closed.
  
  Result in test.unl:
  22000000027465737400130000006162635c2f6465665c5c6768695c226a6b6c
  0000|{"test":"abc\\/def\\\\ghi\\"jkl"}|abc\\/def\\\\ghi\\"jkl|
  
  So the problem seems not to be related to the GenBSON function
  but to the internal JSON<->BSON cast.
  

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

• DUPLICATE OF IT33799
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels