Wildcard Dictionary

About this task

As an alternative to creating a domain list, a wildcard dictionary can be used instead. If you do not already have a wildcard dictionary created from your domain collection, you must create one which will be used as concepts for the data-driven conceptual search algorithm. The wildcard dictionary that is created will contain a list of words found in the domain collection and the number of times that the word appears in the collection. Default output options for dictionaries create spelling, wildcard, and stemming dictionaries. For data-driven conceptual search, only a wildcard dictionary is needed. To save time and space, you can remove the other dictionaries from the outputs tab.

Note: It is important to know that words that appear very often (more than half the number of documents in the collection), in the domain collection used to derive a data-driven ontolection will be discarded by the conceptual search algorithm, since they are not considered to be descriptive of the domain.

Compared to a domain list, using a wildcard dictionary as a seed will increase crawling time and will generate a less compact CCS collection.

To create a wildcard dictionary, you must complete the following steps:

Procedure

  1. Click the Add icon () beside the Dictionaries listing in the left-hand Watson™ Explorer Engine navigation menu.

    For this tutorial, enter francis as the name, and click Add.

  2. Select the Inputs tab, and click the Add Input Source button.
  3. Select Search Collection from the pop-up dialog that displays, as shown in Figure 1
  4. Click Add.
    Figure 1. Adding an Input Source to the Dictionary
  5. In the Collection Name field, enter the name of the search collection for the domain of interest. For this tutorial, enter binning-collection. Click the OK to save the changes.
  6. Optionally, if you wish to remove the additional dictionaries select the Outputs tab
  7. Click the remove button in the headers for the Spelling Dictionary and both entries for Stemming Dictionary.

    Only Wildcard Dictionary should remain as shown in Figure 2

    Figure 2. Only Dictionary Required for Data-Drive Conceptual Search

    You will want to consolidate words that appear in the dictionary in both their singular and plural form. For example, both horse and horses are the same concept and should not be regarded as separate concepts. To do this

  8. Select the Overview tab of your dictionary
  9. Scroll down to the Meta Information section
  10. Click edit. In the Input Stemmer field, type case+depluralize

    This is shown in Figure 3.

    Note: We will not do this for the collection used in this tutorial because the collection is so small. If we depluralize this small collection, it will result in some words being removed as too common when we create the conceptual search ontolection. Words are considered too common if the frequency that the word appears is more than half the size of the domain collection.
    Figure 3. Input Stemmer Setting
  11. On the Overview tab's Live Status section header, click the build button

    This is shown in Figure 4.

    Figure 4. Building a Dictionary
    Note: The creation of the dictionary may take some time depending on the size of the search collection. If you have a large domain collection, you may want to remove outputs other than wildcard to reduce the time used in building the dictionary.

Results

Once the wildcard dictionary is built, you will see a screen like the one shown in Figure 5.

Figure 5. Dictionary is Built
Note: If you make any significant changes to the domain collection that introduce new topics or subtopics, you may wish to update the domain list or to rebuild the wildcard dictionary. Also, if you make any changes to the dictionary settings after you have built the dictionary, you will need to rebuild the dictionary. To rebuild the dictionary, click build on the Overview tab.

To proceed with this tutorial, click Creating a Conceptual Context Signature (CCS) Collection.