From the earliest conception of Unicode, a character representation standard designed to cover the vast array of human writing systems, it was clear that it would be essential to provide a convenient way for computers to query information about characters. Part of each version of the Unicode standard has been a corresponding version of The Unicode Character Database (UCD). The Unicode Character Database (UCD) home page describes the Unicode Character Database as follows:

The Unicode Character Database (UCD) consists of a number of data files listing character properties and related data along with a documentation file that explains the organization of the database and the format and meaning of the data in the files.

Character properties are information needed to understand and use a character. Some character properties are independent of the context in which the character is used, for example, whether or not a character is customarily used as a numerical digit. Some character properties depend on its role in a sequence of characters, such as directionality (some writing systems proceed from left to right on a page, some from right to left, and some in further variations).

Navigating the Unicode database

The third edition of XML 1.0 incorporates by reference The Unicode Standard, Version 3.2, so this is the version of UCD XML developers will be most concerned with. The UCD includes a couple dozen data files and about a half dozen documentation files. In this article I'll focus on the main file, UnicodeData-3.2.0.txt. You can find a link to the 3.2 UCD directory in Resources. If you open this file, you'll find a line for each character. Each line is a set of fields delimited by semicolons. As an example the following is the line for the uppercase A character.

0041;LATIN CAPITAL LETTER A;Lu;0;L;;;;;N;;;;0061;

The first field is the code point, 0041. Obtain the conventional identifier for this character by prepending "U+" (In this case it is U+0041). You can represent the character in XML regardless of encoding using the entity A. Beware that code points can use up to six characters, although five characters is the limit you'll find in UnicodeData-3.2.0.txt. The second field LATIN CAPITAL LETTER A is the character name, which is very important in discussion of the character. The third field, Lu is the general category, which is the most important key in Unicode's system for organizing characters. The value Lu is an abbreviation for "Letter, Uppercase". Examples of other catagories are Nd ("Number, Decimal Digit"), Pd ("Punctuation, Dash"), Sc ("Symbol, Currency") and Zs ("Separator, Space"). There are several more fields in each UnicodeData-3.2.0.txt line, but the ones I've mentioned are the most widely used, and give you a flavor of the sort of information you can find in the UCD.

Finding characters for fun and profit

Since the UCD files are simple text you can use all sorts of generic tools to process them. You can often find a character of interest by loading UnicodeData-3.2.0.txt in a text editor and searching for a key word in the character name. You can also use command line tools such as grep in UNIX. In the following example, I look for the dagger characters commonly used to mark notes.

$ grep -i "dagger" UnicodeData-3.2.0.txt
2020;DAGGER;Po;0;ON;;;;;N;;;;;
2021;DOUBLE DAGGER;Po;0;ON;;;;;N;;;;;

The -i option makes the search case insensitive. You can also find fun characters by name. Some that I have come across are pencil, skull-and-crossbones and snowman.

$ grep -i skull UnicodeData-3.2.0.txt 2620;SKULL AND CROSSBONES;So;0;ON;;;;;N;;;;;
$ grep -i pencil UnicodeData-3.2.0.txt 
270E;LOWER RIGHT PENCIL;So;0;ON;;;;;N;;;;;
270F;PENCIL;So;0;ON;;;;;N;;;;;
2710;UPPER RIGHT PENCIL;So;0;ON;;;;;N;;;;;
$ grep -i snowman UnicodeData-3.2.0.txt
2603;SNOWMAN;So;0;ON;;;;;N;;;;;

Help from the library

Your programming language of choice might provide convenient tools for accessing the Unicode database, so that you do not need to parse it yourself. The unicodedata module in the Python standard library provides a thin layer on the UnicodeData-3.2.0.txt file from Unicode 3.2. The following interactive Python session demonstrates a few queries.

>>> import unicodedata
>>> print unicodedata.name(u'5')
DIGIT FIVE
>>> print unicodedata.lookup('DIGIT FIVE') #The lookup is case-insensitive
5
>>> print unicodedata.digit(u'5')
5
>>> #There are many ways in the world for writing the digit five
>>> print unicodedata.digit(unicodedata.lookup('ARABIC-INDIC DIGIT FIVE'))
5
>>> #Get the code point in decimal and hex ...
>>> print ord(unicodedata.lookup('ARABIC-INDIC DIGIT FIVE'))
1637
>>> print hex(ord(unicodedata.lookup('ARABIC-INDIC DIGIT FIVE')))
0x665
>>>

In Python you indicate a unicode string by prepending a u, as you can see in the second line of the session.

Back to top

Wrap up

Once you are familiar with the UCD, you can use it in all sorts of advanced ways: sorting values in XML files in internationally sound ways, normalizing data in XML files for easier comparison and digital signing, and much more. I use the database a lot because I'm not good at rememebering the code points for some uncommon characters I use in XML files, such as special bullet points and international symbols. With the UCD, the wealth of the world's writing systems is right at your fingertips.

Resources

Learn

• The skew.org XML Tutorial by Mike Brown: Learn more about the character model for XML in ,this tutorial, subtitled "A reintroduction to XML with an emphasis on character encoding".
  
  
• The unicodedata module: Check out Python's wrapping of the UCD.
  
  
• developerWorks XML zone: Find more XML resources here, including articles, tutorials, tips, and standards. For a complete list of XML tips to date, check out the tips summary page.
  
  
• IBM Certified Solution Developer -- XML and related technologies: Learn how to get certified.
  
  

Get products and technologies

• The Unicode home page: Look for all the Unicode resources you need. Download the UCD files from the 3.2 UCD directory. Remember to bookmark the Unicode FAQ.
  
  

About the author

Uche Ogbuji is a consultant and co-founder of Fourthought Inc., a software vendor and consultancy specializing in XML solutions for enterprise knowledge management. Fourthought develops 4Suite, an open source platform for XML, RDF, and knowledge-management applications. Mr. Ogbuji is also a lead developer of the Versa RDF query language. He is a computer engineer and writer born in Nigeria, living and working in Boulder, Colorado, USA. You can find more about Mr. Ogbuji at his Weblog Copia or contact him at uche@ogbuji.net.

Rate this article

Comments

Back to top

Table of contents

• Navigating the Unicode database
• Finding characters for fun and profit
• Wrap up
• Resources
• About the author
• Comments