com.ibm.lpex.core
Class LpexNls

java.lang.Object
  com.ibm.lpex.core.LpexNls

public final class LpexNls
extends Object

This class provides several multicultural support functions. These are mostly functions related to DBCS/MBCS and bidirectional (bidi) editing, and may not work correctly for other character encodings. There is one instance of this class for each document view.

These functions attempt to ease the handling by users of files originating on and/or targeted for a remote system, in a manner similar to their handling in that particular environment (for example, IBM System z members being edited with a local editor such as ISPF): emulation of SO/SI control characters, awareness of the text's actual positions, columns, length, and sequence numbers (all defined in terms of bytes), etc.

The current implementation, while attempting to be generic, focuses on Windows as the workstation, and z/OS (IBM System z, zSeries, S/390) and IBM i (iSeries, AS/400) as the remote system. These remote systems use EBCDIC character encodings.

DBCS source encodings

In the EBCDIC character encodings used by these remote systems a DBCS character uses two bytes, and a DBCS string is delimited by SO/SI controls. Also, for these systems each byte in the character encoding takes up one display column on the screen (1-byte SBCS character = 1 display column, 2-byte DBCS character = 2-column display width, the SO and SI control characters = 1 display column each).

Related editor parameters: sourceCcsid, sourceEncoding, useSourceColumns. Related methods: addSourceSosi(String), addSourceSosi(String,char,char), displayingSosi(), isSourceMbcs(), isSourceSosi(), usingSourceColumns().

Bidi source encodings

Bidi support is not available on all platforms. See isBidi().

LPEX is a programming editor and, as such, its bidirectional support is limited. It is based on IBM's "Guidelines of a Logical User Interface for Editing Bidirectional Text", Guidelines for Programming Editors. These guidelines define a programming editor as a "textual application used to write a formal language (like a programming language, or HTML) where most syntactic elements of the language (keywords, identifiers) are in English, while character values and comments may be in any language. [...] Each line of text is considered an independent paragraph. The paragraph embedding level is always 0.".

In the EBCDIC character encodings used by the remote systems mentioned above, certain character combinations (Arabic lam + alef ligatures) may translate into a single-byte character (a visual lamalef code point).

LPEX handles logical layout bidi documents. As such, text locations (such as the column parameters of its commands) refer to positions in the logical text. It supports smart-logical (SL) documents, where bidi marks are added not merely for presentation purposes, but as metadata in order to ensure the correct conversion to visual layouts (solving round-trip conversion issues and preserving the logical structure of the document). Parser support can be provided to add context-sensitive, inter-token bidi marks for this purpose.

LPEX bidi enablement includes logical cursor movements, logical stream and character selections, visual rectangle selections, setting the keyboard language based on the context upon mouse clicks inside the document, maximum record-length calculations for correct saves to the remote (ensuring that the number of Unicode characters in each line does not exceed the save.textLimit setting of the document), support for ignorable bidi marks, and a visual edit line feature. Most document parsers shipped with the editor are capable of maintaining inter-token bidi marks for SL documents, and API methods are provided for editor extensions to provide the same support.

Related editor parameters: maintainBidiMarks, sourceCcsid, sourceEncoding, useSourceColumns, visualEditLineDialog. Related editor actions: insertLtrMark, insertRtlMark, visualEditLine. Related methods: isBidi(), isBidiMark(char), isIgnoringBidiMarks(), isLigature(java.lang.String, int), isSourceBidi(), isShapedLigatures(), isSmartLogical(), isSyntaxSmartLogical(), setTextStyleBidiMarks(int, java.lang.String), usingSourceColumns().

See also Bidi support for document-parser development considerations.

Other encodings

For EUC (Extended Unix Code) character encodings (UNIX, AIX, Linux), LpexNls may not provide adequate emulation of their source edit environments. The byte-length of characters differs from their display-column width, so you may have to use native code (JNI), for example the *mb* C library functions, for display-width calculations (as only byte-length information is currently obtainable in Java); these calculations affect the (column-based) tabs expansion in LPEX.

Terminology used in here

Unicode	LPEX, like any Java program, uses Unicode for its internal representation of characters. More specifically, this is the UTF-16 encoding, which encodes the basic multilingual plane of Unicode version 1 directly, and uses surrogate pairs as the escape mechanism to encode the next 16 planes of Unicode version 3
encoding	a Java-supported character encoding, for example, "Cp1252" (Windows Latin-1)
native encoding	this is the default character encoding of the platform (host operating system) that LPEX runs on, according to the default locale. This is usually an ASCII character encoding on a workstation (Windows, Linux, etc.). This encoding is normally determined from the "file.encoding" Java system property
file encoding	this is the character encoding of an LPEX document's underlying file. The file encoding is normally the native encoding, as files are usually stored in an encoding that is same as the default encoding of the host operating system (for example, on the Japanese Linux, files are typically stored in EUC-JP). In a heterogeneous platform environment, the encoding of the host operating system may be different from the encoding of a file we want to load into the editor. In such a case, one must explicitly specify the encoding of the file, or let the editor attempt to detect it (see LpexView.LpexView(String,String,boolean)); the editor will then perform the character code conversion on loading the file in, and similarly when saving the document. Internet character encodings are not supported. For these encodings, the surrounding escape sequences are not currently handled in the code. Their processing should be in a manner quite similar to the handling of SO/SI controls in EBCDIC DBCS. An application using the LPEX text widget may also read and write files in any encoding on its own, and use the editor API, for example, to set the document text from a Reader and save it through a Writer. As an example, an Eclipse technology LpexAbstractTextEditor-based editor may extend the FileDocumentProvider to read and save local UTF-8 files, while the LPEX Editor plug-in loads and saves its contents, all in Unicode, from and to the Eclipse IDocument provided.
source encoding	the source file's character encoding: the file being edited may originate from and/or be targeted for a remote system (i.e., different from the platform that LPEX runs on). Setting the source encoding information in the editor allows LPEX to emulate features of the file's original editing environment (for example, display emulated SO/SI controls), correctly establish the sequence numbers in effect, calculate the length limit of text lines for save operations, etc.
DBCS	Asian character set/encoding that contains double-byte characters
MBCS	Asian character set/encoding that contains multi-byte characters
SO, SI	Shift-out and Shift-in control characters. Only EBCDIC DBCS encodings use SO/SI escape characters. Balanced SO/SI characters enclose sequences of DBCS character bytes. LPEX can display emulation SO/SI characters in order to present the user an image of the file similar to the one seen in its source natural habitat (for example, an IBM System z member being edited with a local editor such as ISPF).

Here is a scenario for editing a remote file with LPEX:

 
 -----------------------------                   ----------------------------
 | Windows XP workstation    |                   | IBM System z mainframe   |
 | IBM Java 2 SDK 1.5.0      |                   |                          |
 |                           |                   |                          |
 | Native encoding: MS 932   |                   | Source encoding: CP 939  |
 |  (PC DBCS)                |                   |  (EBCDIC DBCS + SO/SIs)  |
 |                           |                   |                          |
 |                           |                   | 1.remote file XXX        |
 |                           |  2.file-transfer  |                          |
 |                           |    utility:       |                          |
 |                           |    System z ->    |                          |
 |                           |      workstation  |                          |
 |                           |<===================                          |
 |  3.local file xxx         |                   |                          |
 |                           |                   |                          |
 |  4.LPEX loads file        |                   |                          |
 |    from workstation:      |                   |                          |
 |    "MS932" -> Unicode     |                   |                          |
 |                           |                   |                          |
 |   [5.set sourceEncoding   |                   |                          |
 |      to "Cp939" for       |                   |                          |
 |      emulation purposes   |                   |                          |
 |    6.set sourceCcsid      |                   |                          |
 |    7.set sequenceNumbers] |                   |                          |
 |                           |                   |                          |
 |    8.LPEX handles         |                   |                          |
 |      the editing of       |                   |                          |
 |      document xxx         |                   |                          |
 |      (all in Unicode)     |                   |                          |
 |                           |                   |                          |
 |  9.LPEX saves file        |                   |                          |
 |    to workstation:        |                   |                          |
 |    Unicode -> "MS932"     |                   |                          |
 |                           |                   |                          |
 | 10.local file xxx'        |                   |                          |
 |                           | 11.file-transfer  |                          |
 |                           |    utility:       |                          |
 |                           |    workstation -> |                          |
 |                           |      System z     |                          |
 |                           ===================>|                          |
 |                           |                   | 12.remote file XXX'      |
 |                           |                   |                          |
 -----------------------------                   ----------------------------

Notes

• when the remote file has sequence numbers, the sequenceNumbers editor parameter should be set only after the sourceEncoding (and, when applicable, the sourceCcsid) parameter(s) were set; sequence numbers are defined in terms of byte columns in the source encoding
• the Java character-encoding converters remove SO/SIs during the EBCDIC DBCS to Unicode conversion, and insert SO/SIs in the Unicode to EBCDIC DBCS conversion. If the EBCDIC DBCS text contains (malformed) adjacent SO/SI-delimited sequences (such as <SO>D1D2<SI><SO>D1D2<SI>), this information is lost in the conversion
• if the Java Unicode String includes SO/SIs (values U+000E and U+000F), these will be kept (and balanced) in the converted EBCDIC by the Java character-encoding converters
• as of Java 2 SDK v1.4, there is new support for encoding sets in the java.nio package. The canonical names currently used in here for character encodings are the ones used in the java.io and java.lang APIs. See also Java Supported Encodings.

Method Summary

String	addSourceSosi(String s) Adds emulation SO/SIs to a Java Unicode string originating from, or targeted for, an EBCDIC DBCS source character encoding.
String	addSourceSosi(String s, char shiftOut, char shiftIn) Adds emulation SO/SIs to a Java Unicode string originating from, or targeted for, an EBCDIC DBCS source character encoding.
boolean	displayingSosi() Returns whether the view is displaying emulation SO/SI controls.
int	encodingCharIndex(String s, int index) Returns the character index into the encoded string (i.e., as converted from s using the source character encoding and CCSID), which corresponds to the index into the Java Unicode text string s.
String	getFileEncoding() Returns the character encoding of the document's underlying file.
static String	getNativeEncoding() Returns the native (platform's default) character encoding.
String	getSourceEncoding() Returns the source character encoding of the document.
static int	indexFromEncodingIndex(String s, int index, String encoding, int ccsid) Returns the index into a Java Unicode text string which corresponds to the specified index into its encoded string (i.e., as converted using the specified character encoding and CCSID).
static boolean	isBidi() Returns whether the editor is running in a bidirectional environment which it can handle.
static boolean	isBidiEncoding(String encoding) Returns whether a character encoding is bidirectional.
static boolean	isBidiMark(char c) Returns whether the specified character is an implicit directional mark.
boolean	isIgnoringBidiMarks() Returns whether the document might contain ignorable bidi marks.
static boolean	isLigature(String s, int offset) Returns whether the characters in the given logical text string are Arabic lam + alef.
static boolean	isMbcsEncoding(String encoding) Returns whether a character encoding is DBCS/MBCS.
boolean	isShapedLigatures() Returns whether the document source encoding and CCSID identify an encoding scheme with shaped Arabic ligatures.
boolean	isSmartLogical() Returns whether the format of this view's document is smart logical.
static boolean	isSosiEncoding(String encoding) Returns whether a character encoding uses SO/SI control characters (is an EBCDIC DBCS character encoding).
boolean	isSourceBidi() Returns whether the source character encoding of the document is bidi (Arabic/Hebrew).
boolean	isSourceMbcs() Returns whether the source character encoding of the document is DBCS/MBCS.
boolean	isSourceSosi() Returns whether the source character encoding of the document is EBCDIC DBCS.
boolean	isSyntaxSmartLogical() Returns whether this view's document is smart-logical format with parser support.
static boolean	isValidEncoding(String encoding) Validates a character encoding.
void	setStyleBidiMarks(int element, String style) Convenience utility to set the style string for an element that has bidi marks.
void	setTextStyleBidiMarks(int element, String style) Convenience utility to maintain inter-token bidi marks for the given parsed element in a smart-logical document.
int	sourceLength(char c) Returns the byte-length for a string consisting of one Java Unicode char converted to the source character encoding.
int	sourceTruncate(String s, int textLimit) Truncates a Java Unicode string so that it fits within the specified number of bytes when converted to its source encoding.
static String	stripBidiMarks(String text) Convenience utility to strip bidi marks from the given text string.
String	stripStyleBidiMarks(int element) Convenience utility to strip bidi marks from the style of the given element.
boolean	usingSourceColumns() Returns whether the view's document is effectively using source columns.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

isValidEncoding

public static boolean isValidEncoding(String encoding)

Validates a character encoding.

Parameters:

encoding - character encoding to validate

Returns:

true = the character encoding is valid and supported by the active Java run environment, or
false = the encoding is null or not supported

isMbcsEncoding

public static boolean isMbcsEncoding(String encoding)

Returns whether a character encoding is DBCS/MBCS. The given encoding is checked against internal tables of DBCS/MBCS character encodings.

You must also use isValidEncoding(java.lang.String) to ensure that this character encoding is supported by the active Java run environment.

Parameters:

encoding - the canonical name (java.io, java.lang) of a character encoding

isSosiEncoding

public static boolean isSosiEncoding(String encoding)

Returns whether a character encoding uses SO/SI control characters (is an EBCDIC DBCS character encoding). The given encoding is checked against an internal table of EBCDIC DBCS character encodings.

You must also use isValidEncoding(java.lang.String) to ensure that this character encoding is supported by the active Java run environment.

Parameters:

encoding - the canonical name (java.io, java.lang) of a character encoding

isBidiEncoding

public static boolean isBidiEncoding(String encoding)

Returns whether a character encoding is bidirectional. The given encoding is checked against internal tables of bidi (Arabic and Hebrew) character encodings.

You must also use isValidEncoding(java.lang.String) to ensure that this character encoding is supported by the active Java run environment.

Parameters:

encoding - the canonical name (java.io, java.lang) of a character encoding

See Also:

isBidi()

getNativeEncoding

public static String getNativeEncoding()

Returns the native (platform's default) character encoding. The native encoding is normally determined from the "file.encoding" Java system property.

Returns:

the canonical name (java.io, java.lang) of the native encoding

getFileEncoding

public String getFileEncoding()

Returns the character encoding of the document's underlying file. This is usually the native (platform's default) character encoding.

Returns:

the canonical name (java.io, java.lang) of the file encoding

getSourceEncoding

public String getSourceEncoding()

Returns the source character encoding of the document. This is the character encoding of the document's underlying file on its origin / target platform.

The source character encoding either defaults to the file character encoding (usually the platform's default encoding), or has been explicitly set with the set sourceEncoding command for files brought over from a remote system.

Returns:

the canonical name (java.io, java.lang) of the source encoding

isSourceMbcs

public boolean isSourceMbcs()

Returns whether the source character encoding of the document is DBCS/MBCS.

isSourceSosi

public boolean isSourceSosi()

Returns whether the source character encoding of the document is EBCDIC DBCS.

isSourceBidi

public boolean isSourceBidi()

Returns whether the source character encoding of the document is bidi (Arabic/Hebrew).

isShapedLigatures

public boolean isShapedLigatures()

Returns whether the document source encoding and CCSID identify an encoding scheme with shaped Arabic ligatures. In this case, the source encoding uses a one-byte visual (shaped) lamalef code point for lam + alef character ligatures. Note that the Java Unicode-to-Cp420 converter preserves lam + alef combinations as two distinct bytes.

This method returns true when the document source encoding is Cp420 and its source CCSID is 420 or 16804. The state of isBidi() is not considered.

See Also:

isLigature(java.lang.String, int)

isIgnoringBidiMarks

public boolean isIgnoringBidiMarks()

Returns whether the document might contain ignorable bidi marks.

Bidirectional marks LRM and RLM may be found in files brought over from a remote system as a result of the conversion from the visual-order EBCDIC encoding in the remote file to a logical-order UTF-8 / Unicode encoding in the workstation file.

LPEX should ignore these marks for most intents and purposes in this scenario (visual Arabic and Hebrew CCSIDs), as they are removed when the file is converted back to the remote.

This method returns true when the document's source encoding is bidirectional, and the source CCSID defines a visual encoding. The state of isBidi() is not considered.

isSmartLogical

public boolean isSmartLogical()

Returns whether the format of this view's document is smart logical. A smart logical (SL) document is a bidirectional logical document with extra bidi marks added as metadata for the correct conversion to visual layouts.

See Also:

isSyntaxSmartLogical()

isSyntaxSmartLogical

public boolean isSyntaxSmartLogical()

Returns whether this view's document is smart-logical format with parser support. This describes a smart logical (SL) document with syntax-sensitive, inter-token bidirectional marks added by the parser for the correct conversion to visual layouts. If no document parser is associated with the bidi document, or its maintainBidiMarks property is not on, this method returns false.

See Also:

isSmartLogical()

isBidiMark

public static boolean isBidiMark(char c)

Returns whether the specified character is an implicit directional mark.

This method returns true if the given character is LRM (left-to-right mark U+200E) or RLM (right-to-left mark U+200F).

Unicode Standard Annex #9, The Bidirectional Algorithm: The directional formatting codes are used only to influence the display ordering of text. In all other respects they should be ignored - they have no effect on the comparison of text, nor on word breaks, parsing, or numerical analysis.

isLigature

public static boolean isLigature(String s,
                                 int offset)

Returns whether the characters in the given logical text string are Arabic lam + alef.

Parameters:

s - Java Unicode text string

offset - ZERO-based offset in s

Returns:

true when the given string contains lam at offset and an alef at offset+1

See Also:

isShapedLigatures()

usingSourceColumns

public boolean usingSourceColumns()

Returns whether the view's document is effectively using source columns.

This method returns true when the current.useSourceColumns setting for the document is on and any one of these conditions applies:

• the document source encoding is DBCS (as set in the current.sourceEncoding parameter)
• the document source encoding is MBCS
• the document source encoding and CCSID (as set in the sourceCcsid parameter) identify an encoding scheme with shaped Arabic ligatures
• the document may contain ignorable bidirectional marks which are not converted back into the source encoding (visual encoding CCSIDs)
• the document source encoding is UTF8.

In these cases document column positions, text lengths, etc., as normally calculated by the internal editor's Unicode text processing may differ from the underlying file's byte-oriented text processing in its source encoding.

displayingSosi

public boolean displayingSosi()

Returns whether the view is displaying emulation SO/SI controls. The method returns true when the source encoding of the document is EBCDIC DBCS and the current.showSosi parameter is on.

sourceLength

public int sourceLength(char c)

Returns the byte-length for a string consisting of one Java Unicode char converted to the source character encoding.

For an EBCDIC DBCS character, this method returns 2 (i.e., the length of the two-byte character itself, without the SO/SI controls). For other character encodings, the length returned may include control bytes. This method assumes that c is not a surrogate character, part of a cluster, or an ignorable bidi mark.

Parameters:

c - regular Java Unicode char

Returns:

1 if c converts to a single-byte character;
2 if the source character is double-byte;
n if the source character is multi-byte

encodingCharIndex

public int encodingCharIndex(String s,
                             int index)

Returns the character index into the encoded string (i.e., as converted from s using the source character encoding and CCSID), which corresponds to the index into the Java Unicode text string s.

If the encoding is EBCDIC DBCS, the index returned is positioned away from a SO/SI control character.

Parameters:

s - Java Unicode text string

index - ZERO-based index into s

Returns:

ZERO-based index into encoded string

indexFromEncodingIndex

public static int indexFromEncodingIndex(String s,
                                         int index,
                                         String encoding,
                                         int ccsid)

Returns the index into a Java Unicode text string which corresponds to the specified index into its encoded string (i.e., as converted using the specified character encoding and CCSID). The ccsid parameter is used to determine whether the specified Java Unicode text string may contain ignorable bidi marks (see isIgnoringBidiMarks()) or ligatures (see isShapedLigatures().

For surrogate pairs in the Java Unicode text, the index of the high surrogate char is returned. The returned index may be the offset of an ignorable bidi mark.

Parameters:

s - Java Unicode String

index - ZERO-based index into the encoding string of s

encoding - encoded string's character encoding

ccsid - encoded string's CCSID (specify 0 if unknown or not relevant)

Returns:

ZERO-based index into s

sourceTruncate

public int sourceTruncate(String s,
                          int textLimit)

Truncates a Java Unicode string so that it fits within the specified number of bytes when converted to its source encoding. Character clusters may also be truncated as a result.

Used in the editor for text-limit operations, only when it is effectively using source columns for this document. It may not work correctly in the case of other character encodings or CCSIDs.

Parameters:

s - Java Unicode String

textLimit - maximum number of bytes in the source string

Returns:

length of the text String s, optionally truncated so that it fits textLimit bytes in its source encoding

See Also:

usingSourceColumns()

addSourceSosi

public String addSourceSosi(String s)

Adds emulation SO/SIs to a Java Unicode string originating from, or targeted for, an EBCDIC DBCS source character encoding. If the source encoding is not EBCDIC DBCS, the original string is returned unchanged.

The SO/SI characters added are those defined by the current.shiftOutCharacter and current.shiftInCharacter editor parameters in this view.

Parameters:

s - Java Unicode String

See Also:

addSourceSosi(String,char,char)

addSourceSosi

public String addSourceSosi(String s,
                            char shiftOut,
                            char shiftIn)

Adds emulation SO/SIs to a Java Unicode string originating from, or targeted for, an EBCDIC DBCS source character encoding. If the source encoding is not EBCDIC DBCS, the original string is returned unchanged.

Parameters:

s - Java Unicode String

shiftOut - SO control character to use

shiftIn - SI control character to use

See Also:

addSourceSosi(String)

isBidi

public static boolean isBidi()

Returns whether the editor is running in a bidirectional environment which it can handle. This method checks whether the native character encoding is bidirectional (Arabic, Hebrew), and whether LPEX supports it.

Currently, only SWT LPEX running on MS Windows platforms (Arabic and Hebrew) provides adequate bidirectional functionality, making use of the underlying Eclipse technology support.

stripBidiMarks

public static String stripBidiMarks(String text)

Convenience utility to strip bidi marks from the given text string. This method returns the effective logical text for the given string. This differs from the original string if it contains bidi marks, which are removed by this method.

See Also:

stripStyleBidiMarks(int)

stripStyleBidiMarks

public String stripStyleBidiMarks(int element)

Convenience utility to strip bidi marks from the style of the given element. This method returns the effective style of an element in the current document section. This differs from the actual element style if the element text contains bidi marks, whose style characters are removed by this method.

Parameters:

element - an element in the document section currently loaded in the editor

See Also:

stripBidiMarks(java.lang.String), setStyleBidiMarks(int, java.lang.String)

setStyleBidiMarks

public void setStyleBidiMarks(int element,
                              String style)

Convenience utility to set the style string for an element that has bidi marks. This method sets the style for an element in the current document section. The specified style string is first adjusted for any bidi marks in the element text.

Parameters:

element - a parsed element in the document section currently loaded in the editor

style - the effective style string to set

See Also:

stripStyleBidiMarks(int), setTextStyleBidiMarks(int, java.lang.String)

setTextStyleBidiMarks

public void setTextStyleBidiMarks(int element,
                                  String style)

Convenience utility to maintain inter-token bidi marks for the given parsed element in a smart-logical document. This method may be called during incremental parsing by the primary parser of a document view, when the document source encoding is bidi and the parser property maintainBidiMarks is on, to set the parsed style and also adjust, if needed, the text of the given element. This method may insert LRMs between the tokens of this element in order to permit a correct conversion of the document to its visual source encoding.

Bidi marks encountered inside the current tokens are assumed to be round-trip marks (bidi marks set by the flat-text, i.e. non-syntax aware, visual to logical conversion, and needed for converting back to visual), and are left 'as is'.

Parameters:

element - a parsed element in the document section currently loaded in the editor

style - the effective style string to set

See Also:

setStyleBidiMarks(int, java.lang.String), LpexView.parsePending(int)