Testing the Converters (Examples)

At this point, we highly recommend that you use the testing functionality of the converters to familiarize yourself with the process. You can test the conversion process that is associated with a search collection by opening that collection in the Watson Explorer Engine administration tool, selecting the Configuration tab, and selecting the Converting sub-tab. On this page, you can enter any URL in the test box and it will download the URL and run the conversion process. Normally, the Watson Explorer Engine administration tool saves a copy of the data that it downloads to avoid repeatedly downloading the same information. To avoid using the saved data, check the Force a recrawl box. If you are developing a parser (XSL stylesheet, for example), you may want to check the Enable debug box which will turn on debugging and add a link to the debugging session on the testing page. Finally, the test uses your exact crawling configuration, which may cause URLs to not be allowed. To test these URLs, check the Force allow box (and don't forget to force a recrawl if you need it to be recrawled!).

To see how this works, go to the Converters sub-tab in the Configuration tab of the default Search Collection. For example, enter http://vivisimo.com in the URL box, check the force allow box and click Test It. In this display, a series of sequential steps are run:

• The attribute filetypes is added to the crawled data with the value html.
• Data is converted from type text/html to type application/vxml-unnormalized by running an XSL stylesheet. Click on the type-out link for application/vxml-unnormalized to see the XML that it produced.
• The XML output by this conversion step is normalized by running another XSL stylesheet. By convention, the converters generate the application/vxml-unnormalized type and not application/vxml to make it very simple to add final normalization steps to the data. In this case, we are truncating long titles and adding the last-modified date to the documents.
• The final step changes the type to application/vxml without running any actions.

In the testing display, each of the types is a link. Clicking on the link will show you the corresponding data. That is, the type-in link for a rule shows the input to the action and the type-out shows the output. If the output produces multiple outputs, for example when testing ZIP files, the type-out link will not exist. For each rule there is also an edit link that will let you view/edit the specific rule.

Immediately after this table showing the conversion steps, there is a table that summarizes the cost of the different converters. A discussion of the subtleties and exact interpretation of these numbers is in the Statistics section. In short, it displays information about every conversion step that is taken and reports the total running time as well as the average running time per step and the amount of data being processed. Furthermore, it separates out the successful from the unsuccessful conversions and the cost of testing any conditions that you place on the converters.

Next on the page, there is a table with information about the URL that you crawled and its header. After that is a table that lists the actual crawling options that were used to crawl this page. Finally, there is a log of the crawler states for the request.