I covered the WordNet project in a previous article, focusing on its use in RDF form. Much has changed since then: The WordNet project has gone from version 1.7 to 2.0, bringing about additional nuances in how words are classified; the RDF representations I covered earlier are still stuck with WordNet 1.6; meanwhile, some of the 2.0 changes have introduced backwards incompatabilities with all previous versions.

Another important change is that Semantic Web technologies have begun to creep into the mainstream as more developers start looking for ways to address the semantic transparency problems that are at the heart of the entire Thinking XML column. More and more of the discussion in the trenches is shifting from whether these sorts of technologies are practical to considerations of best practice and application interface. Some interested parties, including me, continue to seek ways to take advantage of Semantic Web technologies without having to encode everything explicitly into an RDF serialization. WordNet is a very large project based on data whose basic underpinnings are very rigorously defined. It is also very broad in its application. (I showed in my earlier article how WordNet can be used to add much more intelligence to even application-specific searches.) As such, it is a perfect focal point for the discussion of Semantic Web technologies in practice.

This article is the first in a series that will take an updated look at WordNet 2.0 and how it can be used with XML and RDF technology.

Setting up the WordNet infrastructure

There are many WordNet projects languishing with older versions of the data, in many cases because they haven't gotten around to the database format conversion. I think this illustrates the usefulness of discussing reusable means of creating formats, such as XML WordNet files (possibly including XML/RDF). One good way to do this is to write code that takes advantage of the many projects for accessing the WordNet database from your favorite programming language. In my case, this means Python. There are a couple of projects for WordNet in Python. One, PyWN, is also based on an older version, so I'll work with PyWordNet (see Resources). I already had Python 2.4 installed, and I downloaded PyWordNet 2.0.1, installing with the usual python setup.py install.

PyWordNet does not come with the WordNet database files. These you must download separately. You do not have to actually build the WordNet package; it's enough to just unpack the downloaded source to a suitable location. The files PyWordNet needs are in the dict subdirectory. PyWordNet looks for the WordNet files in a location given by the WNHOME environment variable, or in /usr/local/wordnet2.0 by default on UNIX if this variable is not set. Once you have completed all of this setup, you can make sure that everything is in order by running python -c "from wntools import *". In my case, I saw four instances of the following, apparently harmless error message:

Exception exceptions.AttributeError:
 "DbfilenameShelf instance has no attribute 'writeback'" in  ignored
  

I think this error message is caused by a well-known bit of awkwardness in the Python shelve module, which reads the sort of hash database files that are used for WordNet. I was able to proceed without problems despite these messages.

Back to top

From database to data

PyWordNet exposes WordNet data as a collection of Python data structures with some specialized utility methods thrown in. It's documented fairly well within the code, though not in any stand-alone resource. Therefore, you'll want to be liberal with your use of the Python built-in function help while navigating and experimenting, starting with the examples on the PyWordNet home page. My task is now to use this to create a tool for serializing WordNet into XML upon request. This could be used as is to satisfy semantic queries with a minimum of overhead, or if you prefer, as the meat of a tool to generate complete stand-alone WordNet data files (more on that choice in upcoming articles). There are many possible serializations of WordNet, but I'll start with the simplest one: plain old XML with a minimum of namespaces. Listing 1 is the module. This code requires Python 2.2 or more recent, 4Suite 1.0a3 or more recent, and PyWordNet 2.0.2. (See Resources for download details.)

import sys
import cStringIO
#Import PyWordNet libraries
from wntools import *
#Import 4Suite XML generation libraries
from Ft.Xml.Xslt.XmlWriter import XmlWriter
from Ft.Xml.Xslt.OutputParameters import OutputParameters
from xml.dom import EMPTY_NAMESPACE, XML_NAMESPACE


def serialize_synset(synset):
    '''
    Return an XML serialization of a synset as a byte string
    '''
    #Buffer the output into a byte string
    stream = cStringIO.StringIO()
    #Set up the XML output parameters
    oparams = OutputParameters()
    oparams.indent = 'yes'
    #Create an XML writer instance
    writer = XmlWriter(oparams, stream)

    writer.startDocument()
    writer.startElement(unicode(synset.pos))
    #use the offset as the ID
    writer.attribute(u'xml:id', unicode(synset.offset), XML_NAMESPACE)
    writer.startElement(u'gloss')
    writer.text(unicode(synset.gloss))
    writer.endElement(u'gloss')
    #Write out the lexical word forms
    for sense in synset:
        writer.startElement(u'word-form')
        writer.text(unicode(sense.form))
        writer.endElement(u'word-form')
    #Write out all the "pointers" (relationships between words, or
    #to be more precise, between synsets)
    for ptr in synset.pointers():
        writer.startElement(unicode(ptr.type))
        writer.attribute(u'part-of-speech', unicode(ptr.pos))
        writer.attribute(u'target', unicode(ptr.targetOffset))
        writer.endElement(unicode(ptr.type))

    writer.endElement(unicode(synset.pos))
    writer.endDocument()
    #Extract the text from the buffer
    return stream.getvalue()


def serialized_synsets_for_word(word_form):
    '''
    Return a list of all serializations of synsets associated
    with a word form
    '''
    synsets = []
    for d in Dictionaries:
        part = d.pos
        try:
            word = getWord(word_form, part)
            #Get all synonym sets associated with a word form
            #In this part of speech
            synsets.extend([ w.synset for w in word ])
        except:
            pass
    #Iterate over each synset and return the list of serializations
    return [ serialize_synset(ss) for ss in synsets ]
  

PyWordNet is organized into dictionaries, one for each of the four parts of speech WordNet 2.0 covers -- noun, verb, adjective, and adverb. The lexical word forms are the main keys, and each entry manages word senses, synonym sets, glosses, and pointers. The terms in bold are all common parlance in WordNet.

• Synonym set (synset): A collection of synonyms -- concepts that roughly share meaning.
• Sense: The specific meaning of a specific word; the shade of meaning within a synonym set given by a specific word form.
• Gloss: A brief definition for a synonym set.
• Pointer: A relationship between synonym sets.

I think that most applications of WordNet in semantic transparency involve working with pointers. In my earlier article on WordNet, I did a lot of processing of hypernym/hyponym relationships. According to this expected emphasis, I've chosen the granularity of the query results to center around serializing synsets. If you invoke Listing 1 with code such as serialized_synsets_for_word_form('selection'), the result is a list of XML documents. The first one is Listing 2 and the third is Listing 3. (I chose these arbitrarily from the five resulting synsets.)

<?xml version="1.0" encoding="UTF-8"?>
<noun xml:id="152253">
  <gloss>the act of choosing or selecting; "your choice of colors was
unfortunate"; "you can take your pick"</gloss>
  <word-form>choice</word-form>
  <word-form>selection</word-form>
  <word-form>option</word-form>
  <word-form>pick</word-form>
  <hypernym part-of-speech="noun" target="32816"/>
  <frames part-of-speech="verb" target="653781"/>
  <frames part-of-speech="verb" target="656613"/>
  <frames part-of-speech="verb" target="652154"/>
  <hyponym part-of-speech="noun" target="152613"/>
  <hyponym part-of-speech="noun" target="152749"/>
  <hyponym part-of-speech="noun" target="152898"/>
  <hyponym part-of-speech="noun" target="153642"/>
  <hyponym part-of-speech="noun" target="154057"/>
  <hyponym part-of-speech="noun" target="170871"/>
  <hyponym part-of-speech="noun" target="173378"/>
</noun>
  

<?xml version="1.0" encoding="UTF-8"?>
<noun xml:id="5455460">
  <gloss>the person or thing chosen or selected; "he was my pick
for mayor"</gloss>
  <word-form>choice</word-form>
  <word-form>pick</word-form>
  <word-form>selection</word-form>
  <hypernym part-of-speech="noun" target="5453619"/>
  <frames part-of-speech="verb" target="653781"/>
  <hyponym part-of-speech="noun" target="5455670"/>
  <hyponym part-of-speech="noun" target="5455968"/>
  <hyponym part-of-speech="noun" target="5456920"/>
</noun>
  

I have edited these listings to break the gloss into a couple of lines for better formatting. These XML results contain pretty much all of the noteworthy information for each synset.

Getting the point

The pointers are in the form:

   <hypernym part-of-speech="noun" target="32816"/> 

The attributes part-of-speech and target provide all the information you need to uniquely identify a synset in the entire WordNet corpus. The value of target is an offset into the original database. I'm not sure whether it's safe to use across WordNet versions, but you shouldn't usually need to. Within a particular version, you can resolve a pointer in the above form using PyWordNet's getSynset function, to which you pass in the part of speech and the offset from the pointer. Notice that I also use these offsets as unique XML identifiers in the synset output, using xml:id (see Resources).

Back to top

Wrap-up

I can't say enough about the value of WordNet as a resource. Semantic transparency is always going to be an exceedingly hard problem, but it's a great help to have a project that provides such a computer-readable framework for analyzing relationships between words. In future articles, I plan to further explore how applications can pragmatically use WordNet, building on the XML query basis in this article. Some will involve RDF and some plain XML, so don't worry that I won't touch on your own preference in such matters. If you have any thoughts on WordNet usage, or experience of your own, please share by posting on the Thinking XML discussion forum.

Resources

• Participate in the discussion forum.
  
  
• Explore WordNet, the English lexical database project. Among the related projects are database interfaces for many languages and platforms, most of which require that you separately install the WordNet 2.0 database package. If you've already used WordNet in some way, take a look at the changes in version 2.0.
  
  
• Check out the W3C xml:id Working Draft, a proposal for a system for providing attributes that are treated as unique identifiers, without requiring DTDs.
  
  
• Try out PyWordNet, a library for accessing WordNet data as very flexible Python data structures.
  
  
• Download the required software for the code in this article. You need Python, version 2.2 or more recent, 4Suite, version 1.0a3 or more recent, and PyWordNet 2.0.2 or more recent (see above).
  
  
• Philip McCarthy covers WordNet and RDF in his article "Introduction to Jena" (developerWorks, June 2004).
  
  
• Browse for books on these and other technical topics.
  
  
• Learn how you can become an IBM Certified Developer in XML and related technologies.
  
  

About the author

Uche Ogbuji is a consultant and co-founder of Fourthought Inc., a software vendor and consultancy specializing in XML solutions for enterprise knowledge management. Fourthought develops 4Suite, an open source platform for XML, RDF, and knowledge-management applications. Mr. Ogbuji is also a lead developer of the Versa RDF query language. He is a computer engineer and writer born in Nigeria, living and working in Boulder, Colorado, USA. You can contact Mr. Ogbuji at uche@ogbuji.net.

Rate this article

Comments

Back to top

Table of contents

• Setting up the WordNet infrastructure
• From database to data
• Wrap-up
• Resources
• About the author
• Comments