Capturing HTTP Transactions

The easiest way to identify the form(s) to which you need to submit authentication information for a remote resource is to capture a complete transcript of all exchanges between your browser and that resource when successfully authenticating. Different browsers and browser versions provide different ways of capturing such transcripts, and different developers may prefer different solutions. Table 1 provides a list of tools that you may find useful in capturing such interactions.

Table 1. Mechanisms for Capturing Browser/Resource Interaction
Browser/Version Name/URL HOWTO
Mozilla Firefox Firebug, Plugin Select Tools > Firebug > Show Firebug (F12). In the Firebug frame, select the Net menu's Enable item to enable tracing. Click the All secondary menu, and click Persist.
Google Chrome Developer Tools, Builtin Select Chrome Menu > Tools > Developer Tools. Select Preserve Log upon Navigation button in bottom bar.
Internet Explorer 9 and Greater Developer Tools, Builtin Press F12 to display the Internet Explorer Developer Tools. Use the Network tab.

After installing any necessary plugins and configuring your browser to record all communications between your browser and the remote resource that you want to crawl, open a new window or tab in which to do your experimentation. The session associated with that tab or window must not already have cookies that could authenticate it to the remote resource, which is usually done by opening an incognito browsing tab or window. This is done in different ways in different browsers and browser versions:

  • Internet Explorer 8 and earlier - Select the Safety menu, located in the upper right hand corner of a browser window. Select InPrivate Browsing from the drop-down menu that displays.
  • Internet Explorer 9 and later - Select the gear icon, located in the upper right hand corner of a browser window. Select the Safety sub-menu. Select the option labeled InPrivate Browsing from the menu that displays.
  • Firefox 3.6 and later - Enter Ctrl+Shift+P or select the Tools menu, and select Start Private Browsing from the drop-down menu that displays.
  • Google Chrome - Press Ctrl+Shift+N or select Chrome's main menu button (three horizontal lines), and select New Incognito window from the drop-down menu that displays.

In the incognito or private browsing tab or window, enable HTTP transaction logging, and connect and authenticate to the remote resource. Once you have authenticated, do not close the window that displays the log - you will need that information in Configuring Form-Based Authentication.

The next section explains how to add the component that enables Watson™ Explorer to create a search collection based on the content of a resource that uses form-based authentication.