Hebrew Characters Reversing Order When Converting to Unicode

Resolving The Problem

Problem

When converting from the Hebrew CCSID 424 to a Unicode CCSID, such as 1200 or 1208, the characters in the Unicode string may be in a different order than they were in CCSID 424.

Resolution

The reordering is due to the bidirectional nature of the Hebrew language. Bidirectional text is text that can consist of text that has one directionality, such as Arabic text written from right to left, and text that has an opposite directionality, such as English or numbers written from left to right.

Bidirectional text may be stored in either its logical order or in its physical order. On the IBM i and z, EBCDIC bidirectional text is almost always stored in physical order. On other platforms, bidirectional text is almost always stored in logical order. Logical order is used for Unicode.

Integration of bidirectional text from mainframes and other environments requires transformation of the bidi text into a layout where all text has the same order. On the IBM i, when converting between two bidi CCSIDs, the conversion also does bidirectional processing. The result is that some of the text may be reversed during processing.

The bidirectional processing depends on the string type of the CCSID. Each bidi CCSID is tagged with an ST (String Type) that tells the conversion code what action to take.

For instance:
CCSID 424 is tagged as ST 4 (Visual)
CCSID 1200 (UTF-16) and 1208 (UTF-8) are tagged as ST 10

On a conversion from ST 4 to ST 10, if the data is divided into parts and some parts happen to start or end with a Hebrew character, the conversion to Unicode will inverse the relative order of the directional runs in that part.

To resolve this, you should use the CCSID 62211 in place of 424. CCSID 62211 is ST 5. The conversion from 62211 to 1200 or 1208 (ST 5 to ST 10) results in no reordering.

For more information on bidirectional languages, you should refer to the following link:
http://www-01.ibm.com/software/globalization/topics/bidi/bidirectionality.html

For more information on string types, you should refer to the following link:
http://www-01.ibm.com/software/globalization/ccsid/string_types.html