Configuring Form-Based Authentication

Once you have added the Form-Based Authentication crawler condition to the search collection that is associated with the resource that you want to crawl, enter the following information in the configuration for that crawler condition:

• Form-Based Authentication Root URL - the URL the form handler will visit to try and acquire cookies before starting the crawl. This value is usually the same as the seed URL for your search collection.
• Debug Mode Enabled - a Boolean that identifies whether debugging information is collected during an attempt to authenticate using form-based authentication. When true, the form-handler creates this debugging file in the tmp sub-directory of the crawl with which it is associated. The crawl directory for a collection is located on the system where Watson Explorer Engine is installed. The full pathname of this directory has a name of the following form:

  INSTALL-DIR/data/search-collections/*/COLLECTION-NAME/crawlNUMBER/tmp

  The debugging file in this directory has the prefix viv_cvo.

  Note: You do not need to enable debugging when you are initially configuring form-based authentication support in a search collection. Enabling debugging is usually only necessary if more than one form must be navigated to authenticate using form-based authentication. See Enabling HTTP Debugging for more information.
• Form-Based Authentication Steps - defines how to process the forms necessary to authenticate Watson Explorer to a remote resource in order to crawl that resource. This section is generally referred to as a form handler. See the information after this list for detailed information about specifying forms and parameters in this field.

Watson Explorer Engine will automatically handle HTTP redirects of various sorts, but requires explicit information about how to provide the data that is required to log in on a remote resource. You must provide this information in the Form-Based Authentication Steps field for the crawler condition that you are defining.

In order to provide this information in the Form-Based Authentication Steps field, click the xml button. An edit dialog displays in which you can enter form definitions in the following format:

<form xpath="XPATH">
  <parameter name="NAME" value="VALUE" />
  ...
  <parameter name="NAME" value="VALUE" />
</form>

The elements and associated attributes in a form definition have the following meaning and requirements:

• The form element's xpath argument is optional, and is only necessary if the default xpath value xpath="//form" is not sufficient to uniquely identify the form that you want to pass parameters to, or if multiple forms are required during the authentication process and you want to uniquely identify a single one.
• The parameter elements are only necessary if the form requires that you specify explicit name/value pairs as parameters in order to authenticate or progress from one form to another.

In order to provide an initial value for the contents of the Form-Based Authentication Steps field, examine the transcript that you captured in Capturing HTTP Transactions. Beginning at the end of the transcript, search backward for the credentials that you had to provide in order to authenticate. You should find these values inside POST data. Enter an equivalent <form>, with any mandatory parameters, in the Form-Based Authentication Steps field, and click OK to save your changes and close the crawler condition that you have just defined.

As an example, suppose that the final form in your transcript has a unique ID of openid_login and that the value that you entered when authenticating are being passed as fields named username and password. The form element that you would specify in the Form-Based Authentication Steps field would look like this:

<form xpath="//form[@id='openid_login']" />
  <parameter name="username" value="joe-user" />
  <parameter name="password" value="joes-password" />
</form>

The next section explains how to test the form-handler that you just defined.