Introduction

WebSphere® Everyplace^TM Access (hereafter called Everyplace Access) lets developers and administrators build wireless applications quickly and effectively. Transcoding Technology, a component of WebSphere Everyplace Access, enables one to make information on the Web available to multiple types of devices (for example, PDAs, cell phones, etc.). Simply, Transcoding Technology converts content from one form to another. It includes conversions from one markup language to another and content annotation of HTML. It is also pluggable and includes a powerful architecture for developing specialized transformations.

This article discusses the Transcoding Technology feature, and specifically focuses on what this tool provides and how developers can use it effectively in a pervasive portal. An overview of the Transcoding Technology architecture is also provided, along with a description of the various tasks that developers can accomplish with Transcoding Technology in Everyplace Access. This article assumes some knowledge of how an enterprise portal works. For introductory material, please refer to the Related resources section.

A stylesheet example is also provided with this article. See the Download section below.

Back to top

What is transcoding?

Transcoding is the process of dynamically mutating and customizing a document based on the characteristics of the requesting device and on the user's preferences. Transcoding Technology is the component of WebSphere Everyplace Access that provides portlet developers and portal administrators with transcoding capability. The Transcoding Technology component provides two essential functions: deck fragmentation and annotation, which the Web clipping portlet uses. This article will discuss both deck fragmentation and annotation, but will not discuss the Web clipping portlet. For details on the Web clipping portlet, refer to Scott DeWitt's article, Basic Web Clipping Using WebSphere Portal Version 4.1, which was published in June 2002 on the WebSphere Developer Domain.

Many Transcoding Technology functions will be discussed in this article.

Back to top

Architecture

To get the most out of this article, I recommend that you become familiar with the basic WebSphere Portal (hereafter called Portal) architecture by referencing the various items listed in the Related resources section below.

WebSphere Portal invokes Transcoding Technology at two different levels: the individual portlet level and the full page level. The flow of a request is illustrated in Figure 1:

The above diagram can be understood as follows:

1. The client device makes a request to the portal, which the Portal Filter receives.
2. The Portal Filter calls WebSphere Portal to receive its contents, which then calls the Aggregator to aggregate the page.
3. The Aggregator selects the portlets based on a combination of values. If a portlet is configured to be transcoded, and it provides a markup that Transcoding Technology can use, then it will be selected. For example, if the portlet provides HTML and the client needs WML, then the Aggregator will select the portlet because it is transcodable. If a portlet is configured to use Transcoding Technology, then the Portlet Filter will be placed in front of the portlet.
4. The Portlet Filter calls the portlet to receive its contents.
5. The portlet returns its contents.
6. The Portlet Filter sends the portlet's contents to Transcoding Technology for processing, associating the portlet's contents with a default URI unless otherwise specified (see the Portlet-Level Transcoding section). Note that the portlet thinks it is talking directly to the Aggregator, so no special programming style is needed to write a portlet that is filtered.
7. Transcoding Technology returns the processed contents to the Portlet Filter.
8. The Portlet Filter returns the processed contents as if it were a proper portlet returning contents normally.
9. The Aggregator, having finished aggregating the page, returns the page.
10. The Portal Filter sends the portal's aggregated contents to Transcoding Technology for processing.
11. Transcoding Technology returns the processed contents.
12. The Portal Filter returns the processed contents to the client.

Back to top

General transcoding concepts

Certain aspects of Transcoding Technology apply to both portlet-level transcoding and portal-level transcoding, so we will cover these aspects first:

Device profiles

WebSphere Portal has a simple and effective method for determining device characteristics. A regular expression, meant to match a particular User-Agent string, is mapped to a client, which has an associated aggregator and set of capabilities. This associated aggregator specifies a particular markup. For example, the HTML aggregator specifies text/html as its markup. Transcoding Technology has a separate device profile system with a separate set of device profiles that is tied into the portal as follows:

• The specific client is determined by the User-Agent field in the HTTP request headers. This client has a manufacturer property, and may also have model and version properties. An example of properties for Microsoft® Internet Explorer^TM would be Microsoft, Internet Explorer, and 6.0.
• The manufacturer, model, and version are combined to form a unique device profile name, which is passed into Transcoding Technology. For example, Microsoft, Internet Explorer, and 6.0 would be combined to form Microsoft!Internet+Explorer!6.0.
• Transcoding Technology matches this unique name with a predefined device profile.

When clients are created and modified in Portal, they are automatically propagated into Transcoding Technology's device profiles; this means that the user does not have to worry about keeping them synchronized. However, the Transcoding Technology profiles have some properties that aren't shared by Portal, since they are transcoding-specific. Some of the more important ones include:

• WMLMaximumDeckSize The maximum deck size for a WML document. Fragments will be this size or smaller.
• HTMLMaximumPageSize The maximum page size for an HTML document. Note that this is for i-mode or cHTML devices.
• TextLinksPreferredToImages This instructs Transcoding Technology to convert an embedded image into a link that points to that image in order to reduce the amount of valuable space that an image takes up on a device.
• supportedImages A list of image types that the device supports.
• imageScalingValue A fraction that specifies how much to scale images.
• colorSupported Indicates whether the device supports color.
• colorDesired Indicates whether the device wants color.
• screenCapability Indicates a screen resolution of high, medium, or low.
• colorDepth The bit depth of the number of colors that are supported by the device.

To change one of these settings, you must use the XML configuration tools.

XML configuration

The Transcoding Technology XML configuration tools included with Everyplace Access is very similar to the XMLConfig utility that ships with WebSphere Application Server. There are two main batch files, ExportResources.bat and ImportResources.bat; both are located in the Transcoding Technology installation directory (on Windows, this would be c:\Program Files\IBMTrans, on Linux, /opt/IBMTrans, and on AIX, /usr/IBMTrans).

To make any changes, you must first run ExportResources.bat. The default options (i.e., no parameters) will cause all of the configuration to be dumped into the WTPResources.xml file. Because the file is in XML format, you can use any text editor to edit it. For example, let's say that you have an Alcatel^TM WAP phone, and want to increase the WMLMaximumDeckSize value to 5K (please note that here, I am simply making up a limitation for illustration purposes). According to the Manage Clients administration portlet in Everyplace Access, the manufacturer value for this client is Alcatel. However, this client does not have a model or version property, so the Device Profile that Transcoding Technology will use is simply named Alcatel.

Therefore, after you open the WTPResources.xml file and search for Alcatel, you will find a Device element with a SelectorName child that contains Alcatel. It should look something like this:

<Device> 
<SelectorName>Alcatel</SelectorName> 
<UserAgent>(User_Agent=*Alcatel*)</UserAgent> 
<DesiredMimeType order="0">text/vnd.wap.wml<DesiredMimeType> 
<SupportsJavaApplets>false</SupportsJavaApplets> 
<SupportsJavaScripts>false</SupportsJavaScripts> 
<SupportsFrames>false</SupportsFrames> 
<CreateCHTML>false</CreateCHTML> 
<IsPortalClient>true</IsPortalClient> 
<PortalOrdinal>40</PortalClient> 
<PortalParent>WML-Device</PortalParent> 
<Enable>true</Enable> 
</Device>

This Device element is the entry that corresponds to the Alcatel client in Portal. Unfortunately, we can't find a WMLMaximumDeckSize entry in this device profile, so we must look for the PortalParent child element. This is the device from the Alcatel entry. We see that it is WML-Device, and after looking at the WML-Device entry, we see that the default WMLMaximumDeckSize is 2880 bytes. To change this for Alcatel phones, we need to add the following to the Alcatel Device element:

<Keys> 
<Key Name="WMLMaximumDeckSize">5120</Key> 
</Keys>

Then, save the file and run ImportResources.bat, which defaults to using the WTPResources.xml file. The decks will now be fragmented to 5K instead of to 2880 bytes for Alcatel phones.

Transcoders

Transcoding Technology's main function is built around a set of plugins called transcoders. This includes annotation, markup language conversion, and so forth. It is designed to let others write plugins for Transcoding Technology. I will briefly cover how to use the XML Configuration tool to register a transcoder that you've already built. For more details on writing transcoders, refer to the Related resources section below.

Let's say that you have a transcoder contained in myUsefulTranscoder.jar. You will need to create an XML file to register it:

<Resources> 
   <Plugin> 
      <Location>c:/mydirectory/myUsefulTranscoder.jar</Location> 
      <Force>true</Force><!-- optional parameter --> 
      <Enable>true</Enable><!-- optional parameter --> 
   <Plugin> 
</Resources>

Now, run ImportResources using the following command: ImportResources -File myFile.xml. The Location element in the XML file will tell the XML Configuration tool where to find the transcoder.

Back to top

Portlet-level transcoding

Portlet-level transcoding is used for content transformations at the portlet level. This includes markup conversions, annotations, stylesheet applications, and any other conversion that affects an individual portlet. As was mentioned in step 6 of the Architecture section above, the portlet's contents are associated with a URI when passed through Transcoding Technology. By default, this URI is http://<hostname>/%%WPS/Portlet. However, the URI can be modified by setting the PortletURI parameter in the portlet configuration. This URI can be used for transcoders, and along with other resources that use it.

Configuring a portlet to use transcoding

In the most basic case, to configure a portlet to use Transcoding Technology, you can do one of the following:

• Add a parameter to the portlet descriptor, portlet.xml, as a <config-param> in the concrete portlet definition, or
  
  
• Add a parameter via the manage portlets administration portlet using the Modify Parameters option.

In general, you will most likely have to decide between two options:

• You want the portlet to be available in all markups. To configure this, add the parameter FilterChain=Transcoding to the portlet, or
  
  
• You only want the portlet to be transcoded to some markups. For example, if the portlet provides only HTML, and you want it to be transcoded to WML, but not cHTML, then do the following:

  Open the file <AppServer>/lib/app/config/services/PortletFilterService.properties in a text editor. You will see the following entry:

  filtername1=Transcoding Transcoding.classname = com.ibm.transform.wps.portletfilter.TranscodingPortletFilter Transcoding.transcodeMarkup.1 = html->wml Transcoding.transcodeMarkup.2 = html->chtml Transcoding.transcodeMarkup.3 = html->vxml

  
  

  This is the default Portlet Filter. It is configured to support conversions from HTML to WML, HTML to cHTML (compact HTML for I-mode phones in Japan), and HTML to VXML (Voice XML for voice servers).

  Add the following entry:

  filtername2 = wmlTranscoding wmlTranscoding.classname = com.ibm.transform.wps.portletfilter.TranscodingPortletFilter wmlTranscoding.transcodeMarkup.1 = html->wml

  
  

  This creates a Portlet Filter that will only transcode from HTML to WML and HTML to HTML. When you configure your portlet, use FilterChain=wmlTranscoding instead of FilterChain=Transcoding. The portlet will only be selected if the client device wants HTML or WML, assuming that the portlet provides HTML.

Use this same technique every time you want to limit when a portlet gets selected. For more complicated cases, such as when you want it to be selected for WML and cHTML, but not for VXML, then you will have to create a wmlChtmlTranscoding filter with two transcodeMarkup entries, one each for WML and cHTML.

Back to top

Annotation

The Transcoding Technology Annotation feature lets you modify the structure of an HTML document via an XML-based Annotation Language. The following is a quick overview of how it works and what you can do with it. See the Related resources section for links to information on annotators and how to build them.

How annotation works

Annotation works on a Document Object Model (DOM) version of a parsed HTML document by applying annotations to DOM elements that are selected by XPath statements. For example, the statement:

<description take-effect="before" target="/descendant::TABLE[2]"> 
   <remove /> 
</description>

will apply the remove annotation immediately before the second table in the document. The annotation engine augments the DOM by adding annotation nodes where the XPath statements match.

After the DOM is augmented with annotation statements, the Annotation Engine makes another pass through the DOM, and processes the annotations. In the <keep/> and <remove/> cases, the annotation engine turns element removal off and on, respectively, meaning that the engine will throw out all elements that it finds after a <remove/> annotation until it hits a <keep/> annotation.

Other annotations include markup replacement, markup insertion, and splitpoints.

Here's a quick summary of the more useful annotations:

• <remove> Turn on the remove state.
• <keep> Turn off the remove state.
• <table> Use for annotating tables.
• <column> Use inside the table annotator to select an entire column.
• <row> Use to select an entire row.
• <field> Use to modify fields in a form.
• <option> Used inside the field annotator to specify a selectable option.
• <insertattribute> Insert an attribute in an HTML tag.
• <inserthtml> Insert an attribute in an HTML tag.
• <replace> Use to replace the selected element in your document with the specified content.
• <replacewithhtml> Use to replace the selected element with HTML markup.
• <setpreference> Use to set a preference that Transcoding Technology will use when transcoding.
• <splitpoint> Specify in an HTML document a fragmentation hint for fragmentation.

The various uses of annotation

Often, it would seem easier and faster to write the portlet source using a JSP file and conditional statements, instead of writing an annotation that needs to be applied. Therefore, understanding where and when annotation can be used is very important in using annotation effectively.

• Annotation is useful if the person generating the markup for the portlet is not comfortable with writing Java code to create or modify the structure of a document.
• Annotation is useful for processing content that is not directly generated by the portlet itself, or content that you do not have control over.
• Annotation is useful for adding markup by using XPath statements.
• Annotation is useful for device-specific annotations.

Configuring a portlet to use annotation

To apply annotation to a portlet, you must first specify that the portlet is to be transcoded (see the Portlet-level transcoding section above). Then, you have the following options:

• Specify the annotation file via a URL in the portlet's configuration. You can do this by either adding a config-param to the portlet.xml file, or by using the manage portlets administration portlet to add the parameter AnnotationFile=<url>, where <url> is the URL of the annotator. This can be any URL that is accessible by the server machine. It is important to note that in the case of a file URL (for example, file:c:/path/file.ann), it must be on the server machine itself.
  
  
• Register the annotation file into Transcoding Technology itself, using the XML Configuration tools. Now, in order for this particular scenario to work, you will need to use the PortletURI parameter mentioned above so that the registered annotator will know which portlet to annotate, so that it doesn't annotate all of the portlets indiscriminately.
  
  
• Use internal annotation. This lets you embed annotation statements such as the following into your document:

  <p>Here is some HTML content that might have been removed by a prior <remove/> tag</p> <!-- <?xml version="1.0" encoding="ISO-8859-1"?> <annot version="1.0"> <keep/> </annot> --> <p>Here is some more HTML that will get kept, since we found a <keep/> tag</p>

  
  

Notice the lack of XPath entries. When you embed the statements directly into your HTML source, the annotation engine already knows where to annotate, so they become unnecessary.

Annotation for multi-page portlets

If your portlet has multiple pages and you want to apply annotation to them, then you are limited to either using internal annotation, or ensuring that each page has the same structure (this way, you can apply the same annotator to each page). It is important to keep the same structure because the XPath statements that make up an annotator rely heavily on the structure of the document; if the structure is not the same, then the annotators will not be applied as expected.

Back to top

Stylesheets

One popular way to convert from XML to any markup language is by using XSL stylesheets. Tools such as Xalan use stylesheets to process the XML content according to the included instructions in the stylesheets. Transcoding Technology lets you apply stylesheets to portlets that generate XML content.

There are some qualities of Transcoding Technology's XSL handling that make it very useful for multi-device scenarios. For example, it provides device properties to the XSLT engine that let you parameterize the stylesheet based on device characteristics, such as screen resolution, whether color is supported, and so forth. This lets you customize your transformation, which becomes especially useful if you want to create only one stylesheet to generate HTML from your XML, but you would also like to support WML. In this case, you could parameterize your stylesheet to generate a smaller HTML document for a WML device, which Transcoding Technology would then transcode to WML (see the article by Alan Booth and Kathryn Britton, Spinning your XML for screens of all sizes, published on IBM developerWorks).

Configuring WebSphere Portal to support the use of stylesheets

Before configuring the portlet, you must configure the portal. However, you only need to do this once. Complete the following:

1. Open the file <AppServer>/lib/app/config/services/PortletFilterService.properties in a text editor.
2. Find the following entries:

   Transcoding.transcodeMarkup.1 = html->wml Transcoding.transcodeMarkup.2 = html->chtml Transcoding.transcodeMarkup.3 = html->vxml

   
   
3. Add the following entries:

   Transcoding.transcodeMarkup.4 = xml->html Transcoding.transcodeMarkup.5 = xml->wml

   
   
4. Save and close the file.
5. Log on to your portal as the Administrator, and select Portal Administration => Portal Settings => Manage Markups.
6. Click Add new markup.
7. Enter xml, text/xml, and UTF-8 for the markup name, MIME type, and default character set, respectively.
8. Restart the portal.

Configuring a portlet to use a stylesheet

1. When creating the portlet.xml for your XML portlet, specify the following in the <supports> element:
   
   

   <supports> <markup name="xml"> <view output="fragment"/> </markup> </supports>

   
   
2. After installing your portlet, select Portal Administration => Portlets => Manage Portlets.
3. Select your portlet from the list, then click Modify Parameters.
4. Add FilterChain=Transcoding and StylesheetFile=<url_to_file>. Make sure that the URL you specify for StylesheetFile is accessible by WebSphere Portal itself, since Portal will be prompting for the file. If the file is on the same machine that Portal is on, then you can use a file URL, for example, file:c:/mydirectory/myfile.xsl.

A quick example

RSS (Rich Site Summary) is a common format for syndicated news feeds, Web logs, etc. It is XML, and therefore also a prime target for an example in this kind of paper. I won't include the XSL and XML details here; see the Related resources section for the files I used.

I used ServletInvokerPortlet that is included in Everyplace Access, and packaged it as above, specifying that it provides XML. Then, I created a JSP file to get the contents of a URL and spit them out. This URL, http://www.xmlhack.com/rsscat.php, is an RSS feed for XMLHack; it is a popular news site for XML information. I set up the portlet to use this JSP source. As a result, I now have a portlet that spits out RSS.

Then, I wrote an XSL stylesheet to convert the RSS into meaningful HTML. I used the parameterization feature to key off the textLinksPreferredToImages device property. This way, if a device doesn't prefer images, it will get text for the title instead of the image.

Because this XSL can be applied to more than just my specific XML page, I decided to register it with Transcoding Technology instead of using the StylesheetFile property. When a stylesheet is registered with Transcoding Technology, it can be applied to every XML that specifies a certain DTD, or that matches a particular URL pattern. To do this, I first called the ExportResources batch file to export all of the preferences. I did this to obtain the ServerModel value for this installation of Transcoding Technology, as well as for the Version. Then, I stripped away unnecessary items until I got the following XML file:

<?xml version="1.0" encoding="UTF-8"?> 
<Resources ServerModel="testOne" Version="magic411srv"> 
</Resources>

Then, I added information for my stylesheet, and saved the file as myStylesheetResource.xml:

<?xml version="1.0 encoding="UTF-8"?> 
<Resources ServerModel="testOne" Version="magic411srv"> 
<Stylesheet> 
<SelectorName>RSS to HTML</SelectorName> 
<Name>RSS to HTML</Name> 
<Description>Converts an RSS feed into HTML</Description> 
<OutputMimeType Enable="true" order="0">text/vnd.wap.wml</OutputMimeType> 
<OutputMimeType Enable="true" order="1">text/html</OutputMimeType> 
<Location>file:/C:/rss2html.xsl</Location> 
<InputDTD>http://my.netscape.com/publish/formats/rss-0.91.dtd</InputDTD> 
<Enable>true</Enable> 
</Stylesheet> 
</Resources>

Please note that what is specified in the Location element is on the server machine, where Transcoding Technology is installed.

I then ran the ImportResources.bat -File myStylesheetResource.xml command to register it. After restarting WebSphere Portal, the Web browser displays the following:

Back to top

Web clipping portlets

Web clipping portlets let a portal administrator create a portlet that consumes existing content from a Web page and keeps only what is necessary. The process to create a Web clipper is very simple. To see documentation on Web clipping, see the WebSphere Portal InfoCenter's Web clipping page. Also refer to Scott DeWitt's article, Basic Web Clipping Using WebSphere Portal Version 4.1.

Back to top

Portal-level transcoding

Portal-level transcoding is used modify an entire page. By design, the entire page is usually in the correct output markup language by this point, so no markup conversions should take place at this level. The main purpose for portal-level transcoding is to provide deck fragmentation.

Deck fragmentation

Many wireless devices, especially cell phones, can support only a limited page (or deck, as it is called in WML) size, and if they receive a page that is too large, they will give an error, or possibly even crash. After the portal page has been assembled, Transcoding Technology will split the page into the correct size fragments and serve the first fragment, caching the rest to be served when the device requests them. This brings up the question, what is the correct size for fragments? Transcoding Technology lets you to modify these values to best suit your devices. See the example in the Device profiles section.

Fragmentation can sometimes lead to strange splits in the document. To address this, Transcoding Technology lets you insert fragmentation hints, so that you can indicate the appropriate places to fragment the document. If you are generating the WML or cHTML source yourself, then insert the following comment in your document as a fragmentation hint:

 
<!-- wtp-frag-point -->

Otherwise, if you are editing HTML that may become transcoded into WML, then you can use the <splitpoint/> annotation, either externally using XPath statements or internally. See the Annotation section for details on how to carry out either task. Make sure to avoid whitespace inside the comment when you add this to your document.

Back to top

Conclusion

Transcoding Technology is an excellent tool for developers or administrators interested in making a pervasive portal even more pervasive. Now that you have a basic understanding of how Transcoding Technology works and how to develop your own transcoders, you can take advantage of this function to build more powerful portals.

Back to top

Download

Name	Size	Download method
stylesheetExample.zip	2 KB	FTP|HTTP

Information about download methods

Resources

• Guide to WebSphere Portal.
  
  
• WebSphere Portal Architecture
  
  
• WebSphere Transcoding Publisher Developer's Guide.
  
  
• Document clipping with annotation: How to keep the good stuff and throw out the rest
  
  
• Spinning your XML for screens of all sizes: Using HTML as an intermediate markup language
  
  
• Basic Web Clipping Using WebSphere Portal Version 4.1
  
  

About the author

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• What is transcoding?
• Architecture
• General transcoding concepts
• Portlet-level transcoding
• Annotation
• Stylesheets
• Web clipping portlets
• Portal-level transcoding
• Conclusion
• Download
• Resources
• About the author
• Comments

Next steps from IBM

With the no-charge DB2 Express-C, pureXML, Zend Core PHP, and Java you can develope advanced Web 2.0 applications quickly that matter.

---

• Try: Download a free trial version of DB2 Express-C, which includes Zend Core PHP and pureXML technology that is ideally suited for managing both XML and relational data.
• Article: See how DB2 pureXML schema capabilities can evolve schemas without making any changes to your applications.
• Tutorial: This tutorial shows you how to use DB2's pureXML to expose the native XML capabilities of DB2.
• Buy: DB2 9 for Linux, UNIX and Windows

My developerWorks community

Dig deeper into WebSphere on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers