IBM Support

Release Notes - IBM Watson Explorer Foundational Components, Version 11.0

Release Notes


Abstract

This document describes new features, known issues and workarounds, and installation information for the foundational components of IBM Watson Explorer Version 11.0.

Content

Overview

IBM Watson Explorer Version 11.0 helps search, analyze, and interpret vast amounts of structured and unstructured information across many enterprise systems and data repositories. It also provides a cost-effective and efficient entry point for exploring the value of big data technologies through a powerful framework for developing applications that use existing enterprise data. IBM Watson Content Analytics is now part of Watson Explorer. Watson Content Analytics brings its powerful text analytics to Watson Explorer applications along with its advanced content mining capabilities. IBM Watson Explorer Version 11.0 also delivers enhancements to the Application Builder and Watson Explorer Engine modules of the Watson Explorer suite along with many other new and improved capabilities.


The IBM Watson Explorer Version 11.0 announcement is available here. If prompted to do so, select the appropriate values for the Home Country and Site Language fields and click Continue. You can then enter Watson Explorer in the search box and click Search.

See the announcement for the following information:
  • Detailed product description
  • Product-positioning statement
  • Packaging and ordering details


Installation

For step-by-step installation instructions, see Installing and Integrating IBM Watson Explorer Foundational Components and Watson Explorer Upgrade Options. For installation and upgrade information, see Upgrading to Watson Explorer Foundational Components Version 11.0.0.

For information about installing the Annotation Administration Console, see: Installing Annotation Administration Console.



System Requirements

For information about the requirements for installing any Watson Explorer component, see Watson Explorer Version 11.0 system requirements.



Release Notes for Version 11.0.0.3


What's New in 11.0.0.3

IBM Watson Explorer Version 11.0.0.3 introduces new features and provides enhancements to the existing product capabilities.

Installation

Watson Explorer Engine

The following feature has been introduced to Watson Explorer Engine in Version 11.0.0.3:

  • The following additional command-line tools are introduced in Watson Explorer Engine Version 11.0.0.3: engine-start, engine-shutdown, engine-cmd, engine-export, engine-import, engine-status, generate-key, embedded-webserver, and embedded-webserver-config. These tools will eventually replace velocity-startup, velocity-shutdown, velocity-webserver, velocity-cmd, and velocity-config and cannot be used interchangeably with the velocity commands. See Additional Command-Line Tools for details.

The following updates and improvements have been made to Watson Explorer Engine in Version 11.0.0.3:

  • The following admin service argument flags have been updated to make the list of running Watson Explorer Engine processes more descriptive: "--go" was updated to "--daemonize", "--start" was updated to "--subordinate", "--rerun" was updated to "--supervisor". If you have automation, scripts, monitoring, or maintenance tools that parse command line arguments of running Watson Explorer Engine processes, they will need to be updated to reflect these changes.
  • Resolved an issue in collections not using light crawler mode where the cache.sqlt would grow indefinitely.
  • Improved ingestion performance in Lexical Analysis streams for very large documents.
  • A topic that explains how valid URLs are tied to standards in Watson Explorer Engine has been added to the Knowledge Center. This information is useful when configuring a project. See URL RFC Standards.
  • Resolved an issue where a Peer Repository client was not synchronizing its repository to the Peer Repository master.

Watson Explorer Engine Connectivity

The following updates and improvements have been made to Watson Explorer Engine Connectors in Version 11.0.0.3:

  • The converter framework has been enhanced by replacing several Java-based converters with converters that are based on C++. Additionally, updates have been made to the underlying converter libraries. These changes result in improved converter stability and better performance. The new converters are installed by default. The old Java-based converters can still used but must be added to the converter chain manually.

Watson Explorer Application Builder

The following new features have been introduced to Watson Explorer Application Builder in Version 11.0.0.3:

  • Notifications

    End users can choose to subscribe to items they follow if the new notifications feature is enabled. When users subscribe to items that they follow, they receive email notifications when activities are generated for those items. For information about configuring email notifications, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/t_de-ab-devapp-email-notifications.html

  • Reports

    Reports for the end-user application were added to the Application Builder administration tool on the Reports tab of the Settings page (the cog icon in the navigation bar). Some examples of the available reports are: top queries, top zero-document queries, and top IP addresses. For more information about reports, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/t_de-ab-devapp-reports.html

  • Monitoring Saved Searches

    When enabled and monitored by users, activities are posted to workspaces that contain saved searches when the results change. If notifications are also enabled, users will receive email notifications about those activities. For more information about saved search monitoring, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/r_de-ab-devapp-global-options.html

  • The following updates and improvements have been made to Watson Explorer Application Builder in Version 11.0.0.3:

    • Query API and Faceting API Documentation

      A significant amount of documentation was added to IBM Knowledge Center for the query API. The documentation is divided into two major sections:

    • Refinement Widget Time Display

      An option was added to the refinement widget for configuring the date format on the tooltip. Now, you can set the format that the date is shown as in the time chart tooltip. For information about configuring the tooltip date format, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/r_de-ab-devapp-pages-widget-refinement.html

    • Save and Create Buttons

      In the administration tool, the Save button and the Create button were updated. Now, the buttons are in a fixed position at the bottom of the screen, which prevents you from having to scroll to click them. Relatedly, when the new Save button is available, you can now use the keyboard shortcut, Ctrl+S.

    • Field Names in the Entity Configuration

      A change was made to allow you to enter names for fields in the filters, user options, display options, and associations. This change is helpful when you want to enter field names before the entity is saved and the drop-down list is populated by the data source.

    • Workspace Descriptions

      Now, end users can add descriptions to workspaces.

    • Creating Workspaces

      An additional way to create a workspace was added. A Create a workspace button was added to the workspaces list page.

    • Workspace Names

      Fixed the issue where users surrounding workspace names with greater than and less than symbols, such as <workspace1>, caused the alert messages in the application to not show the workspace name when users add items to the workspace.

    • Autocomplete

      Fixed the issue where users entering the name of a saved search into the search field of the application and selecting it as the search term by using the autocomplete feature performed a search on the name of the saved search. That behavior was counterintuitive. Now, in the saved-search entity, the title field is not autocompleted by default.

    • Container Widget

      Fixed an issue where a container widget on a page for an entity that was created in v11.0.0.1 didn't load the widgets within the container widget if you minimized the widget, reloaded the page, and then restored the widget. Now, you don't have to reload the page after you restore the widget.

      Fixed an issue where the container widget (formerly known as the flip widget) display name would show in the drop-down list, even though the widget in the container was hidden because of the display conditions. That behavior caused the drop-down list and widgets to get out of sync.

    • iWidget Widget

      This widget was removed. Instead, use the iFrame widget.

    • Useful Links Widget

      This widget was removed. A similar widget exists on the IBM Watson Explorer Application Builder page on GitHub at https://github.com/IBM-Watson/wex-appbuilder-samples/tree/master/save_search_page Users might also use the saved search feature, which is documented in the application in the end user help in the Saving searches section.

    • Added a field to exported results. The new exported_by_title field contains the title of the user who exported the results, in addition to the user ID.
    • Fixed an issue that created duplicate fields for the space entity when a workspace was exported.

    Watson Explorer BigIndex

    The following updates and improvements have been made to Watson Explorer BigIndex in Version 11.0.0.3: There are no updates to report for this version.

    Watson Explorer Results Module

    The following updates and improvements have been made to Watson Explorer Results Module in Version 11.0.0.3: There are no updates to report for this version.

    Watson Explorer Annotation Administration Console

    The following updates and improvements have been made to Watson Explorer Annotation Administration Console in Version 11.0.0.3: There are no updates to report for this version.



Known Issues in 11.0.0.3

At the time of publication, the following issues are known to be unresolved in the Watson Explorer foundational components:

Watson Explorer Engine

  • Linux users importing a peer repository may see an "Unhandled signal 11" message after clicking View or Merge on a repository element with conflicting versions. This error can be safely ignored. To import the remote element, delete the local version manually first.
  • The Scheduler will enter a persistent, illegal state if it is asked to schedule any event more than 24 days in the future. The workaround is:
    1. Delete the Scheduled task.
    2. Restart the Scheduler service (If it won't restart, confirm that you deleted the correct task.)
    3. Recreate the Scheduled task, making sure that it will be run at least once every 24 days.

    See Scheduler for more information about scheduling tasks in Watson Explorer Engine.

  • If you set up remote-push in a version of Watson Explorer earlier than version 11.0.0.1, and did not have user names and passwords specified in your remote-push configuration, remote-push and mirror configurations will stop working when you install Watson Explorer version 11.0.0.3. In addition, if your remote-push configuration includes unencrypted passwords, remote-push will stop working when you install Watson Explorer version 11.0.0.3. To fix this issue, see High Availability and Scalability Through Distributed Indices.
  • Beginning with Watson Explorer version 11.0.0.1, all Watson Explorer Engine user passwords are secured using the scrypt algorithm, which generates one-way cryptographic hashes. Password hashing may increase your ingestion or query times. If this affects your performance significantly, you may disable it. To learn how to do so, see Disabling Password Hashing. In addition, if you are using the Native Document Export feature, password hashing must be disabled for the ICS user(s). See Adding Native Document Export to a Display for more information about native document export and ICS users.
  • When configuring Watson Explorer Engine Search Engine Crawling Options for Java converters, it is strongly recommended that you do not create additional named JVMs using the Additional Java converter JVMs field. Using the default JVMs is recommended.
  • In the Watson Explorer Engine Administration Tool, the following error can occur when attempting to update an existing scheduled job: "Can't get element with elt-id="<nnn>". Existing scheduled tasks cannot be updated as a result. To solve this problem, create a new scheduled task, enter the required fields, and save it. All previously existing tasks can now be edited. If desired, delete the new task after making your edits.
  • The velocity-shutdown command may display the error message "os: process already finished" and return a non-zero exit code when shutting down Watson Explorer Engine. To ensure Watson Explorer Engine has shutdown completely, re-run velocity-shutdown until it no longer displays the error message.
    • When upgrading from Watson Explorer 9.0.0.3 or earlier, if you applied the 9.0.0.3-WS-WatsonExplorer-Windows-HF001 hotfix posted on Fix Central prior to March 23, 2015, you will need to perform an additional step during your installation. This is because the MD5 checksum values that were documented in that hotfix were incorrect, and as a result the files listed in the hotfix will not be deleted during the uninstallation of Watson Explorer 9.0.0.x. To work around this issue, do one of the following before starting your installation of 11.0.0.3:
      • Manually delete these files from your system after uninstalling the 9.0.0.x software.
      • Before uninstalling 9.0.0.x, go to Fix Central and reapply the hotfix. The MD5 values listed in the version posted on March 23, 2015 will delete the files correctly.
    • Installation Manager might crash when run on Red Hat Linux 6.5 because of an issue with out-of-date graphics libraries. To work around this issue, either modify the IM/IMLinux/install.ini file in the offering directory (see IBM Installation Manager can not start from GUI mode for instructions) or upgrade the Installation Manager libraries (see IBM Installation Manager and IBM Packaging Utility GUI crashes while advancing through wizard screens on recent updates of RHEL 6). Alternatively, you can use the console installer to completely bypass this issue.
    • Only an Administrator should start IBM Knowledge Center. If a non-Administrator starts an instance of the service, that instance might not be detected during an uninstall.
    • On Windows, there can be an issue deleting data on a search collection if the file path is longer than 260 characters. To work around this issue, keep collection names as short as possible, but descriptive (for example, "docs" instead of "crawled-documents") or shorten the path to the installation directory (or the collection's data directories if located elsewhere). Contact IBN Software Support if you see errors (for example, in system reporting) regarding an inability to delete files and those file paths are >= 260 characters.
    • If the Maximum JVM Idle Time is set to 0 and you enqueue a large number of documents, such as 50 or more, in quick succession, many of the enqueues are rejected with the following error: [JAVA_CONVERTERS_REMOTE_JVM_COMMUNICATION_FAILED] There was a problem connecting or communicating with the JVM used for running Java converters. Last system error was: Connection refused. To work around the issue, increase the maximum JVM idle time to a value greater than 5.
    • The template Java converter that is described in the tutorial for the Java converter might not compile with some versions of Apache Ant. The Java converter tutorial was tested and successfully compiled with Version 1.7.1, which is available from the Apache Ant archive page.
    • If a collection is configured with invalid parameters (for example, specifying a stemmer that does not exist), an issue might arise where the indexer service enters a continuous restart state. This functionality is documented in the Indexing section. Parameters that can contain invalid values are:
      • The stemmer of a custom language stream.
      • The path name for the PEAR file for a custom content analytics stream.
      • In the Advanced section of the global settings for the indexer, the Lexical analysis logging configuration field. This field has no administrative use other than potentially as a tool for complex debugging.
      • Also in the Advanced section, the Lexical analysis JVM arguments field.
      If you encounter this issue, stop the crawler, correct the configuration, and restart the indexer.
    • (Linux only) If you use the PDF to HTML converter for PDFs that use fonts of double-byte character sets, some text might not be extracted. Text that is not extracted is not included in the converted document. To work around the issue, you can use the PDF to text converter instead. Because the two converters work differently, you might see slight differences in the output of the converted documents.
    • When installing Watson Explorer Engine in non-admin mode on a Microsoft Windows system, the redistributable Microsoft Visual C++ runtime components that are required by Watson Explorer Engine are not installed correctly. This issue can prevent any of the Watson Explorer Engine Windows executables from running correctly. This problem does not occur if Watson Explorer Engine is installed by a user with administrative privileges. To resolve this problem, remove the non-admin installation and reinstall as a user with administrative privileges.
    • When performing a silent installation of Watson Explorer, if any nodes have been removed from a response file, the value for these nodes will be set to the default value.
    • When using continuous update mode for the Oracle WCC connector, filters are not supported if the universal query format is used. If filters are required, use the database query format.
    • When saving a refined query to a folder using the Watson Explorer Engine display, the less than symbol is not supported in the query label field. If the less than symbol is used, the label will be truncated to the content before the less than symbol.
    • Enqueueing a vse-key delete node on a distributed indexing server might result in one corresponding audit log entry on the client when two are expected.
    • When using the Web Parts for Watson Explorer, removing the HTTP authentication user name and password will fail if Remember passwords for sites is enabled in your browser.
    • When using the reporting feature, if the Split System Reporting Database option is set to something other than the default value of monthly, Watson Explorer Engine might incorrectly log some error messages to the monthly reporting database. Similarly, the System Reporting Database Maximum option is ignored, so it might grow in size until hitting the default maximum limit of 2GB.
    • When adding a new seed to a collection, the list of seed types to choose from includes vse-crawler-seed-TEMPLATE. This seed is intended for IBM internal development and should not be used.
    • If you remove a connector plug-in, a search collection will continue to refresh without generating an error message that the connector plug-in was removed.
    • 
      	#search-primary table.form tr.search-inputs {
      	display:block;
      	z-index:1000;
      	position:relative;
      	}
      	
      	#search-primary table.form tr.below {
      	display:block;
      	position:relative;
      	z-index:0; }
      	
    • The Watson Explorer Engine internal web server could be vulnerable to the denial-of-service attack (DoS) that is described on the Apache wiki. If you are using the Watson Explorer Engine internal web server as an Internet facing portal, follow the mitigation steps listed at that link.
    • The vsw-form erroneously states that XML is acceptable for the document query condition field. This text is incorrect. XML should not be used for the document query condition field.
    • Bulk collaboration features (express tagging and renaming, adding, or deleting folders) have crawl-url priority of 1, which might result in a delay for users during normal refresh operations.
    • Links in the crawl graph might not be consistent if crawl URLs are enqueued with the parent-url attribute set. This behavior can cause a potential issue when switching a client to a server in distributed indexing if you are using the IO SharePoint Connector.
    • The use of non-ASCII characters in a Watson Explorer Engine installation path is not supported.
    • Calling the search-collection-read-only-all API function on a server with a large number (thousands) of collections results in increased CPU and memory usage, and can make the server appear to be unresponsive while the search collections on that server are being processed.
    • When using distributed indexing, if all data is deleted from the master server and a crawl is initiated, the client servers never attempt to synchronize, nor do they complain about the data mismatch.
    • Database Connector 5.0.1: In this version of the Database Connector, the document URLs will include the fetch size. This should not have any consequences, unless the fetch size is changed while continuous update is enabled.
    • IO SharePoint connector: When building access control lists (ACLs), the connector does not check the web application policy. When security is enabled for the search collection, the SharePoint user for the crawling account cannot see search results if the account was granted only Full Read permissions at the web application policy level. To work around the issue, grant the SharePoint user for the crawling account Full Read permissions at the site level and document level.
    • In some cases, large documents (for example, large Microsoft Excel files) will fail during conversion, with an error message that references the ISYS_XML_MEMORY_LIMIT environment variable. To workaround this issue, set the environment variable ISYS_XML_MEMORY_LIMIT high enough to allow the converter to process the document.
    • Nested document nodes can cause the query-meta application to exit abnormally.
    • The Watson Explorer autocomplete feature does not collect reporting information.
    • Watson Explorer might fail to open files with a path length of over 255 characters on Microsoft Windows systems.
    • When the Watson Explorer Engine Search Web Parts repair installer is used, the repaired features are left inactive and must be manually activated in each site collection.
    • When using the body toggle option in the custom document section of the Watson Explorer Engine display, the toggle option will not appear in the search results if the body toggle option does not specify any non-annotation, non-zero length content elements. For example, if rating (annotation) and URL (document attribute) were specified as body toggle fields, the body toggle option would not appear in the search results. If a content element (such as snippet) was also specified, the body toggle option would be present in results that have a non-zero length snippet content element. This situation is avoided by always specifying a non-zero length content element as a body toggle field.
    • By default, Log4J logging is set to debug. If performance for that connector is an issue, you can turn off logging in the Watson Explorer Engine administration tool by navigating to the Configuration tab for your search collection seed and adding the following logging settings in the Advanced - Logging section:<root><priority value='OFF'/></root> Note that disabling logging will make it difficult to trace future issues.

    • When crawling an SMB file system that is configured with NETBIOS to limit which workstations the crawling account can log on with, the user not allowed to log on to this computer error message can occur if the computer VIV_CRAWLER is not present in the list of allowed workstations.

    Watson Explorer Results Module

    • If you query a Results Module collection and subsequently edit the query service port, you will no longer be able to see query expansions or spotlights from Results Module collections. This behavior can be resolved by editing the environment-get-config node in Watson Explorer Engine to refer to the correct query service port.
    • Multiple identical spotlights can be created within the same spotlight collection.
    • An ActiveRecord::RecordNotFound in termsController#update error is returned if a user attempts to save a term or spotlight that was deleted by another user.
    • This version of Results Module supports the following maximum number of contents:
      • Maximum number of terms: Approximately 5000
      • Maximum number of spotlights: Approximately 5000
      • Number of spotlight images: At least 50
    • Results Module calls to the Watson Explorer Engine API send user and password information in the clear. Configuring the Results Module to use SSL will prevent these parameters from being displayed. See the Watson Explorer Results Module documentation for further details.
    • An exception might occur if an environment becomes invalid due to an external reason. Possible reasons for an external environment becoming invalid include:
      • Deleting the API user defined for the environment
      • Changing the password of the API user defined for the environment
      • Changing the URL of the installation specified in the environment
      • Removing the Watson Explorer Engine installation specified in the environment
      • Changing the HTTPS security settings of the installation specified in the environment

      The exception can be avoided if all affected environments are updated before a user attempts to access any of the components.

    • If a long running action (for example an import) times out, other requests (such as login) might occasionally time out.
    Spotlight Manager
    • If you use Internet Explorer 11 to access Results Module, you cannot select an image to add to a spotlight in the Spotlight Manager. To work around the issue, add the fully qualified domain name of Results Module to the list of sites that use Internet Explorer Compatibility View. For more information, see Spotlight Manager: Cannot add images when using Internet Explorer 11.
    • If your lowest environment is not correctly pointing to your Watson Explorer Engine installation, saving a spotlight (and possibly terms and collections as well) will fail with no error message.
    • When you delete a spotlight, it will not be immediately deleted in Watson Explorer Engine, but is guaranteed to be enqueued for deletion and will be deleted eventually. This behavior means that if you perform a search that would have returned the spotlight you just deleted, you might still see the spotlight.
    • It is possible to create spotlights whose names are comprised of spaces. This feature does not cause a functional issue, but links to those spotlights in the Results Module might be difficult to use.
    Terminology Manager
    • Importing terms that contain commas in their entries causes inconsistent behavior: in the main term, the comma is retained; in the associated terms for an entry, it is treated as a delimiter.
    • It is possible to import a term collection directly to an environment that is not the lowest environment in the promotion chain. This action should not be done, as the environments and promotion chain will become out of sync.
    • Adding multiple relationships between two terms causes a Watson Explorer Engine exception error. For example, this exception occurs if you add colour as a spelling variation of color, and then attempt to add colour as a synonym of color.

    Watson Explorer Application Builder

    • For the new reporting feature in the administration tool, reporting appears to be enabled by default, but it is not. To work around this issue, on the Reports tab of the Settings page (the cog icon in the navigation bar), click the Save button. For information about reports, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/t_de-ab-devapp-reports.html
    • Configuring a nested facet to be displayed causes the search page of the application to return 0 results.
    • iFrame Widget
      • The iFrame widget contains an iFrame, so it has any limitations that an iFrame has for a given browser.
      • In some browsers, when you move an iFrame widget between columns on the home page, the widget might not resize as accurately as other widgets.

    Watson Explorer Annotation Administration Console

    • Error FFQG0024E indicates that a NullPointerException occurred after you exported an annotator from Content Analytics Studio to Annotation Administration Console. To avoid this problem, ensure that the following check boxes are clear in the Export to Content Analytics window before you export an annotator:
      • Restart the collection's index service
      • Re-index existing collection documents
      These options are not needed because collections in Annotation Administration Console do not include an index service or documents. Annotation Administration Console, you must enable the exported annotator in the target collection by deploying analytic resources and restarting the text analytics session.
    • You might see misleading messages when you attempt to start the Annotation Administration Console after you run the 'esadmin system startall' command, such as "Context root not found" or "Unable to read the configuration data for collection .". You can ignore these messages. Services required to run the administration console are still being started. Wait a couple of minutes and refresh your web browser before attempting to access the administration console again.



Release Notes for Version 11.0.0.2


IBM Watson Explorer Version 11.0.0.2 is a fix pack that addresses security issues. See the following Security Bulletin for more information: http://www.ibm.com/support/docview.wss?uid=swg21974808.

For new features and known issues of this release, see Release Notes for Version 11.0.0.1


Release Notes for Version 11.0.0.1


What's New in 11.0.0.1

IBM Watson Explorer Version 11.0.0.1 introduces new features and provides enhancements to the existing product capabilities.

Installation

  • The Watson Explorer Foundational Components version 11.0.0.1 features a new installer. Installation Manager is no longer used to install Watson Explorer beginning with Watson Explorer version 11.0.0.1. See Watson Explorer Upgrade Options for more information and installation instructions.

Watson Explorer Engine

The following features have been introduced to Watson Explorer Engine in Version 11.0.0.1:

  • This release introduces a new encryption method for passwords and process-to-process communication in Watson Explorer Engine. Passwords and process-to-process communication in Watson Explorer Engine are secured with the AES-128-GCM encryption algorithm, which uses 128 bit keys, and OpenSSL v1.0.1m. A default encryption key is included when you install Watson Explorer Engine. IBM recommends that you generate a new, unique key before starting Watson Explorer Engine for maximum security. See Generating a New Encryption Key with the Encryption Key Utility for details.
  • Beginning with Watson Explorer version 11.0.0.1, all Watson Explorer Engine user passwords are secured using the scrypt algorithm, which generates one-way cryptographic hashes.

The following updates and improvements have been made to Watson Explorer Engine in Version 11.0.0.1:

  • Added a new optional crawl condition named Recognize File Extensions to Watson Explorer Engine that will recognize common file extensions and set the file type based on those extensions. See Conditional Settings for details.
  • Added an option to both the micro-test utility and the collection-service that checks two conditions relevant to the ability to create core dumps. See Checking for the Ability to Create Core Dumps for instructions to run the checks, and how to interpret them. Note: This option is only available for Linux systems.
  • There are new guidelines for the number of open files and processes in Watson Explorer Engine. See Increasing Open File and Process Settings on Linux Systems.
  • There is a change to the default behavior of the Watson Explorer Engine environment variable VIVISIMO_SOURCE_TEST_RESULTS_FILE. The default value for VIVISIMO_SOURCE_TEST_RESULTS_FILE is now Engine/data/source-test. See vivisimo.conf and Watson Explorer Engine Environment Variables for details.
  • Added a new converter, Content Type by File Extension, to Watson Explorer Engine that recognizes common extensions and sets the content type based on the file extension in the URL.
  • Enhanced the converter framework with the ability to convert PDF files into paragraph content. The feature was present in Watson Explorer Engine 10.0.0.0, however, due to changes introduced in the converter framework in Watson Explorer 11.0.0.0, it was temporarily removed for that version.
  • Resolved an issue where the converter framework was not properly displaying the Excel to CSV converter in the converter list.
  • Fixed an issue where the converter framework was incorrectly converting date data in some cells.
  • Collection Broker is not supported on Linux in Watson Explorer version 11.0.0.1 and above.

Watson Explorer Engine Connectivity

Associated with this release of Watson Explorer Engine are the following new connectors and connector deliverables that are available on IBM Fix Central (posted December 17, 2015):

The following updates and improvements have been made to Watson Explorer Engine Connectors in Version 11.0.0.1:

  • Enhanced the IBM Connections connector to support IBM Connections 5.0 and added support for Watson Explorer Application Builder security. IBM Connections 4.5 is still supported. Additionally, the connector supports Watson Explorer Engine 11.0.0.1 encryption and has been incremented to version 5.0.1.
  • Improved the MDM SE connector to use Application Builder security to retrieve group names from MDM repositories. Previously this could only be done through LDAP. Additionally, the connector has been updated to support Watson Explorer Engine 11.0.0.1 encryption and has been incremented to version 5.0.3.
  • Added FileNet P8 connector support for FileNet P8 versions 5.2. Additionally, the FileNet P8 connector uses a new converter, FileNet Content Navigator, to generate URLs. Security for this connector can now be configured in Watson Explorer Application Builder. Additionally, the FileNet P8 connector has been updated to support Watson Explorer Engine 11.0.0.1 encryption and has been incremented to version 5.0.0.
  • Improved WCM Seedlist connector by providing security configuration ability in Watson Explorer Application Builder.
  • Improved the Salesforce connector by upgrading the Apache Commons Collection library to version 3.2.2. This version of the Apache Commons Collection resolves a security vulnerability and minor crawl issues with data and user crawls that were caused by earlier versions of the apache Commons Collections.
  • Upgraded connector support libraries for Watson Explorer Engine 11.0.0.1 encryption compatibility.
  • Fixed an issue in the IO SharePoint connector where a description was missing from the Sharepoint Simple Claims Based Windows CBA Rights Form Component.
  • Resolved an issue with the IO Sharepoint connector where no alert message was being generated when a user neglected to supply a security token service endpoint value when configuring Claims Based Authentication.
  • Corrected an issue where connector PDF documentation files were being generated and installed in the SharePoint connector installation directory.

Watson Explorer Application Builder

The following new features have been introduced to Watson Explorer Application Builder in Version 11.0.0.1:

The following updates and improvements have been made to Watson Explorer Application Builder in Version 11.0.0.1:

Watson Explorer BigIndex

The following updates and improvements have been made to Watson Explorer BigIndex in Version 11.0.0.1: There are no updates to report for this version.

Watson Explorer Results Module

The following updates and improvements have been made to Watson Explorer Results Module in Version 11.0.0.1: There are no updates to report for this version.

Watson Explorer Annotation Administration Console

The following updates and improvements have been made to Watson Explorer Annotation Administration Console in Version 11.0.0.1:

  • The Annotation Administration Console provides support for text analytics in Slovak. In addition, the Named Entity Recognition annotator provides support for Hebrew.


Known Issues in 11.0.0.1

At the time of publication, the following issues are known to be unresolved in the Watson Explorer foundational components:

Watson Explorer Engine

  • If you set up remote-push in a version of Watson Explorer earlier than version 11.0.0.1, and did not have user names and passwords specified in your remote-push configuration, remote-push and mirror configurations will stop working when you install Watson Explorer version 11.0.0.1. In addition, if your remote-push configuration includes unencrypted passwords, remote-push will stop working when you install Watson Explorer version 11.0.0.1. To fix this issue, see High Availability and Scalability Through Distributed Indices.
  • Beginning with Watson Explorer version 11.0.0.1, all Watson Explorer Engine user passwords are secured using the scrypt algorithm, which generates one-way cryptographic hashes. Password hashing may increase your ingestion or query times. If this affects your performance significantly, you may disable it. To learn how to do so, see Disabling Password Hashing. In addition, if you are using the Native Document Export feature, password hashing must be disabled for the ICS user(s). See Adding Native Document Export to a Display for more information about native document export and ICS users.
  • When configuring Watson Explorer Engine Search Engine Crawling Options for Java converters, it is strongly recommended that you do not create additional named JVMs using the Additional Java converter JVMs field. Using the default JVMs is recommended.
  • In the Watson Explorer Engine Administration Tool, the following error can occur when attempting to update an existing scheduled job: "Can't get element with elt-id="<nnn>". Existing scheduled tasks cannot be updated as a result. To solve this problem, create a new scheduled task, enter the required fields, and save it. All previously existing tasks can now be edited. If desired, delete the new task after making your edits.
  • The velocity-shutdown command may display the error message "os: process already finished" and return a non-zero exit code when shutting down Watson Explorer Engine. To ensure Watson Explorer Engine has shutdown completely, re-run velocity-shutdown until it no longer displays the error message.
  • When upgrading from Watson Explorer 9.0.0.3 or earlier, if you applied the 9.0.0.3-WS-WatsonExplorer-Windows-HF001 hotfix posted on Fix Central prior to March 23, 2015, you will need to perform an additional step during your 11.0.0 installation. This is because the MD5 checksum values that were documented in that hotfix were incorrect, and as a result the files listed in the hotfix will not be deleted during the uninstallation of Watson Explorer 9.0.0.x. To work around this issue, do one of the following before starting your installation of 11.0.0:
    • Manually delete these files from your system after uninstalling the 9.0.0.x software.
    • Before uninstalling 9.0.0.x, go to Fix Central and reapply the hotfix. The MD5 values listed in the version posted on March 23, 2015 will delete the files correctly.
  • Installation Manager might crash when run on Red Hat Linux 6.5 because of an issue with out-of-date graphics libraries. To work around this issue, either modify the IM/IMLinux/install.ini file in the offering directory (see IBM Installation Manager can not start from GUI mode for instructions) or upgrade the Installation Manager libraries (see IBM Installation Manager and IBM Packaging Utility GUI crashes while advancing through wizard screens on recent updates of RHEL 6). Alternatively, you can use the console installer to completely bypass this issue.
  • Only an Administrator should start IBM Knowledge Center. If a non-Administrator starts an instance of the service, that instance might not be detected during an uninstall.
  • On Windows, there can be an issue deleting data on a search collection if the file path is longer than 260 characters. To work around this issue, keep collection names as short as possible, but descriptive (for example, "docs" instead of "crawled-documents") or shorten the path to the installation directory (or the collection's data directories if located elsewhere). Contact IBN Software Support if you see errors (for example, in system reporting) regarding an inability to delete files and those file paths are >= 260 characters.
  • If the Maximum JVM Idle Time is set to 0 and you enqueue a large number of documents, such as 50 or more, in quick succession, many of the enqueues are rejected with the following error: [JAVA_CONVERTERS_REMOTE_JVM_COMMUNICATION_FAILED] There was a problem connecting or communicating with the JVM used for running Java converters. Last system error was: Connection refused. To work around the issue, increase the maximum JVM idle time to a value greater than 5.
  • The template Java converter that is described in the tutorial for the Java converter might not compile with some versions of Apache Ant. The Java converter tutorial was tested and successfully compiled with Version 1.7.1, which is available from the Apache Ant archive page.
  • If a collection is configured with invalid parameters (for example, specifying a stemmer that does not exist), an issue might arise where the indexer service enters a continuous restart state. This functionality is documented in the Indexing section. Parameters that can contain invalid values are:
    • The stemmer of a custom language stream.
    • The path name for the PEAR file for a custom content analytics stream.
    • In the Advanced section of the global settings for the indexer, the Lexical analysis logging configuration field. This field has no administrative use other than potentially as a tool for complex debugging.
    • Also in the Advanced section, the Lexical analysis JVM arguments field.
    If you encounter this issue, stop the crawler, correct the configuration, and restart the indexer.
  • (Linux only) If you use the PDF to HTML converter for PDFs that use fonts of double-byte character sets, some text might not be extracted. Text that is not extracted is not included in the converted document. To work around the issue, you can use the PDF to text converter instead. Because the two converters work differently, you might see slight differences in the output of the converted documents.
  • When installing Watson Explorer Engine in non-admin mode on a Microsoft Windows system, the redistributable Microsoft Visual C++ runtime components that are required by Watson Explorer Engine are not installed correctly. This issue can prevent any of the Watson Explorer Engine Windows executables from running correctly. This problem does not occur if Watson Explorer Engine is installed by a user with administrative privileges. To resolve this problem, remove the non-admin installation and reinstall as a user with administrative privileges.
  • When performing a silent installation of Watson Explorer, if any nodes have been removed from a response file, the value for these nodes will be set to the default value.
  • When using continuous update mode for the Oracle WCC connector, filters are not supported if the universal query format is used. If filters are required, use the database query format.
  • When saving a refined query to a folder using the Watson Explorer Engine display, the less than symbol is not supported in the query label field. If the less than symbol is used, the label will be truncated to the content before the less than symbol.
  • Enqueueing a vse-key delete node on a distributed indexing server might result in one corresponding audit log entry on the client when two are expected.
  • When using the Web Parts for Watson Explorer, removing the HTTP authentication user name and password will fail if Remember passwords for sites is enabled in your browser.
  • When using the reporting feature, if the Split System Reporting Database option is set to something other than the default value of monthly, Watson Explorer Engine might incorrectly log some error messages to the monthly reporting database. Similarly, the System Reporting Database Maximum option is ignored, so it might grow in size until hitting the default maximum limit of 2GB.
  • When adding a new seed to a collection, the list of seed types to choose from includes vse-crawler-seed-TEMPLATE. This seed is intended for IBM internal development and should not be used.
  • If you remove a connector plug-in, a search collection will continue to refresh without generating an error message that the connector plug-in was removed.
  • 
    	#search-primary table.form tr.search-inputs {
    	display:block;
    	z-index:1000;
    	position:relative;
    	}
    	
    	#search-primary table.form tr.below {
    	display:block;
    	position:relative;
    	z-index:0; }
    	
  • The Watson Explorer Engine internal web server could be vulnerable to the denial-of-service attack (DoS) that is described on the Apache wiki. If you are using the Watson Explorer Engine internal web server as an Internet facing portal, follow the mitigation steps listed at that link.
  • The vsw-form erroneously states that XML is acceptable for the document query condition field. This text is incorrect. XML should not be used for the document query condition field.
  • Bulk collaboration features (express tagging and renaming, adding, or deleting folders) have crawl-url priority of 1, which might result in a delay for users during normal refresh operations.
  • Links in the crawl graph might not be consistent if crawl URLs are enqueued with the parent-url attribute set. This behavior can cause a potential issue when switching a client to a server in distributed indexing if you are using the IO SharePoint Connector.
  • The use of non-ASCII characters in a Watson Explorer Engine installation path is not supported.
  • Calling the search-collection-read-only-all API function on a server with a large number (thousands) of collections results in increased CPU and memory usage, and can make the server appear to be unresponsive while the search collections on that server are being processed.
  • When using distributed indexing, if all data is deleted from the master server and a crawl is initiated, the client servers never attempt to synchronize, nor do they complain about the data mismatch.
  • Database Connector 5.0.1: In this version of the Database Connector, the document URLs will include the fetch size. This should not have any consequences, unless the fetch size is changed while continuous update is enabled. The next version of the connector will be updated to be consistent with earlier versions. That version will be available in the next release of Watson Explorer, and on FixCentral.
  • IO SharePoint connector: When building access control lists (ACLs), the connector does not check the web application policy. When security is enabled for the search collection, the SharePoint user for the crawling account cannot see search results if the account was granted only Full Read permissions at the web application policy level. To work around the issue, grant the SharePoint user for the crawling account Full Read permissions at the site level and document level.
  • In some cases, large documents (for example, large Microsoft Excel files) will fail during conversion, with an error message that references the ISYS_XML_MEMORY_LIMIT environment variable. To workaround this issue, set the environment variable ISYS_XML_MEMORY_LIMIT high enough to allow the converter to process the document.
  • Nested document nodes can cause the query-meta application to exit abnormally.
  • The Watson Explorer autocomplete feature does not collect reporting information.
  • Watson Explorer might fail to open files with a path length of over 255 characters on Microsoft Windows systems.
  • When the Watson Explorer Engine Search Web Parts repair installer is used, the repaired features are left inactive and must be manually activated in each site collection.
  • When using the body toggle option in the custom document section of the Watson Explorer Engine display, the toggle option will not appear in the search results if the body toggle option does not specify any non-annotation, non-zero length content elements. For example, if rating (annotation) and URL (document attribute) were specified as body toggle fields, the body toggle option would not appear in the search results. If a content element (such as snippet) was also specified, the body toggle option would be present in results that have a non-zero length snippet content element. This situation is avoided by always specifying a non-zero length content element as a body toggle field.
  • By default, Log4J logging is set to debug. If performance for that connector is an issue, you can turn off logging in the Watson Explorer Engine administration tool by navigating to the Configuration tab for your search collection seed and adding the following logging settings in the Advanced - Logging section:<root><priority value='OFF'/></root> Note that disabling logging will make it difficult to trace future issues.

  • When crawling an SMB file system that is configured with NETBIOS to limit which workstations the crawling account can log on with, the user not allowed to log on to this computer error message can occur if the computer VIV_CRAWLER is not present in the list of allowed workstations.

Watson Explorer Results Module

  • If you query a Results Module collection and subsequently edit the query service port, you will no longer be able to see query expansions or spotlights from Results Module collections. This behavior can be resolved by editing the environment-get-config node in Watson Explorer Engine to refer to the correct query service port.
  • Multiple identical spotlights can be created within the same spotlight collection.
  • An ActiveRecord::RecordNotFound in termsController#update error is returned if a user attempts to save a term or spotlight that was deleted by another user.
  • This version of Results Module supports the following maximum number of contents:
    • Maximum number of terms: Approximately 5000
    • Maximum number of spotlights: Approximately 5000
    • Number of spotlight images: At least 50
  • Results Module calls to the Watson Explorer Engine API send user and password information in the clear. Configuring the Results Module to use SSL will prevent these parameters from being displayed. See the Watson Explorer Results Module documentation for further details.
  • An exception might occur if an environment becomes invalid due to an external reason. Possible reasons for an external environment becoming invalid include:
    • Deleting the API user defined for the environment
    • Changing the password of the API user defined for the environment
    • Changing the URL of the installation specified in the environment
    • Removing the Watson Explorer Engine installation specified in the environment
    • Changing the HTTPS security settings of the installation specified in the environment

    The exception can be avoided if all affected environments are updated before a user attempts to access any of the components.

  • If a long running action (for example an import) times out, other requests (such as login) might occasionally time out.
Spotlight Manager
  • If you use Internet Explorer 11 to access Results Module, you cannot select an image to add to a spotlight in the Spotlight Manager. To work around the issue, add the fully qualified domain name of Results Module to the list of sites that use Internet Explorer Compatibility View. For more information, see Spotlight Manager: Cannot add images when using Internet Explorer 11.
  • If your lowest environment is not correctly pointing to your Watson Explorer Engine installation, saving a spotlight (and possibly terms and collections as well) will fail with no error message.
  • When you delete a spotlight, it will not be immediately deleted in Watson Explorer Engine, but is guaranteed to be enqueued for deletion and will be deleted eventually. This behavior means that if you perform a search that would have returned the spotlight you just deleted, you might still see the spotlight.
  • It is possible to create spotlights whose names are comprised of spaces. This feature does not cause a functional issue, but links to those spotlights in the Results Module might be difficult to use.
Terminology Manager
  • Importing terms that contain commas in their entries causes inconsistent behavior: in the main term, the comma is retained; in the associated terms for an entry, it is treated as a delimiter.
  • It is possible to import a term collection directly to an environment that is not the lowest environment in the promotion chain. This action should not be done, as the environments and promotion chain will become out of sync.
  • Adding multiple relationships between two terms causes a Watson Explorer Engine exception error. For example, this exception occurs if you add colour as a spelling variation of color, and then attempt to add colour as a synonym of color.

Watson Explorer Application Builder

  • If you surround workspace names with greater than and less than symbols, such as <workspace1>, alert messages in the application do not show the workspace name when users add items to the workspace.
  • For saved searches, if users enter the name of a saved search into the search field of the application and select it as the search term by using the autocomplete feature, the search is performed on the name of the saved search. That behavior is counterintuitive, and should be disabled by editing the saved-search entity. To disable the behavior, in the saved-search entity, for the Fields to autocomplete on field, remove title and save the entity.
  • For the container widget (formerly known as the flip widget), if display conditions prevent a widget from showing in the container widget, the display name of the widget shows in the container widget drop-down list, but the widget itself is hidden. The potential end result to users is that the drop-list and visible widgets can become out of sync. To work around this issue, instead of using display conditions to hide a widget within a container widget, remove the widget from the container widget.
  • If a container widget is on an page for an entity that was created in v11.0.0.1, the widgets within the container widget do not load if you minimize the widget, reload the page, and then restore the widget. To work around this issue, reload the page after you restore the widget.
  • Configuring a nested facet to be displayed causes the search page of the application to return 0 results.
  • The application detail page fails to load when an iWidget is created without values in one or more fields in the Type-specific configuration section. Also, Runtime URL must begin with the scheme :// otherwise it fails to load the application detail page.
  • iFrame Widget
    • The iFrame widget contains an iFrame, so it has any limitations that an iFrame has for a given browser.
    • In some browsers, when you move an iFrame widget between columns on the home page, the widget might not resize as accurately as other widgets.

Watson Explorer Annotation Administration Console

  • Error FFQG0024E indicates that a NullPointerException occurred after you exported an annotator from Content Analytics Studio to Annotation Administration Console. To avoid this problem, ensure that the following check boxes are clear in the Export to Content Analytics window before you export an annotator:
    • Restart the collection's index service
    • Re-index existing collection documents
    These options are not needed because collections in Annotation Administration Console do not include an index service or documents. Annotation Administration Console, you must enable the exported annotator in the target collection by deploying analytic resources and restarting the text analytics session.
  • You might see misleading messages when you attempt to start the Annotation Administration Console after you run the 'esadmin system startall' command, such as "Context root not found" or "Unable to read the configuration data for collection .". You can ignore these messages. Services required to run the administration console are still being started. Wait a couple of minutes and refresh your web browser before attempting to access the administration console again.


Release Notes for Version 11.0.0


What's New in 11.0.0

IBM Watson Explorer Version 11.0.0 introduces new features and provides enhancements to the existing product capabilities.

Watson Explorer Engine

The following features have been introduced to Watson Explorer Engine in Version 11.0.0:

  • Lexical Analysis language support includes the following new features and capabilities:
    • Improved language support extends Lexical Analysis capabilities to Arabic, Chinese, Czech, Danish, Dutch, French, German, Greek, Hebrew, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, and Turkish.
    • Enhanced detection of failed states prevents the indexer from restarting with the same configuration that caused a failed state until that configuration is changed or modified.
    • Added support for custom PEAR file creation.
    • Improved support for Unicode characters, better handling of invalid HTML input, and general improvements to JVM out-of-memory conditions.
    • Improved Lexical Analysis administration with the ability to enable and configure a knowledge base when using a lexical analysis stream.
    • Added the ability for application developers to enable and configure clustering when using a lexical analysis stream.
    • Enhanced Lexical Analysis streams to support stop words, term expansion dictionaries, knowledge bases, wildcards, and clustering.
  • Added Alchemy API integration, which leverages sophisticated natural language processing techniques to analyze content and add high-level semantic information and delivers insights from unstructured data.
  • Enhanced converter framework by upgrading Perceptive document filters to version 11.2. The upgraded document filters alleviate the need to, under rare circumstances, rename older Microsoft Excel documents to more recent Excel extensions.
  • Added custom web service converter to enable Watson Explorer Engine integration with REST-based web services to accelerate cognitive computing initiatives by quickly integrating and analyzing data from Representative State Transfer-based (RESTful) web services, as well as services from Watson Developer Cloud. Integration with these services now takes minutes rather than hours.
  • Upgraded the bundled version of Java provided in Watson Explorer Engine to Java 8.
  • Improved Watson Explorer Engine connector installation process. All connectors are now installed by default.
  • Added connectivity support for integrating data from the cloud-based Salesforce CRM to leverage user engagement and collaboration data. The Salesforce connector crawls a broad range of Salesforce content including records, documents, and social data. Features of the Salesforce connector include support for Watson Explorer Application Builder security, crawling in Continuous Update mode.
  • Enhanced the IO SharePoint connector to support the cloud-based SharePoint Online to drive user engagement and collaboration. SharePoint Online content that can be crawled includes:
    • Site Collections
    • Sites
    • Document Libraries
    • Attachments
    • Personal Sites
    • Profile Information
    • Social Content
    • Lists
    • List Items
  • Added a timeout option to enhance the cache generation of rich previews.
  • XML Schema Documentation — New elements were added to the XML schema reference documentation, which is available in IBM Knowledge Center at http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.engine.schema.doc/c_schema-wrapper.html
  • Debugging for ACLs—A parameter called force-output-acls was added to the query-search function that enables you, at search time, to force the output of ACLs that exist on documents and contents. The force-output-acls parameter outputs the ACLs even when the Collection > Configuration > Searching > Debugging > Output ACLs option is not enabled. To enable the option in debugging mode of the application, replace the variables in the following URL example: http://<host>/cgi-bin/<project_name>?v:sources=<source_name>&v:project=<project_name>&query=&v:debug=1&v:force-output-acls=1
  • Added Watson Explorer Application Builder security integration to the following connectors:
    • Alfresco connector
    • Database connector
    • Documentum connector
    • IBM Domino connector
    • Imap connector
    • IO SharePoint connector
    • Samba connector
    • SharePoint connector (On Premise versions)
    • WebCenter Content connector

  • The following updates and improvements have been made to Watson Explorer Engine in Version 11.0.0:

    • Fixed an issue where in some circumstances content was missing after a collection refresh or resume. The issue occurred because the Watson Explorer Engine crawler sometimes deleted a single URL during initialization. This problem was often invisible because the deleted URL would be found again later on during the refresh.
    • Fixed an issue where the indexer continued to index documents when the Lexical Analysis JVM was generating out of memory exceptions.
    • Fixed an issue where Microsoft Office documents were not indexed when the value of -1 was configured in the converter elapsed time setting.
    • Fixed an issue where navigational tabs were not displaying in Watson Explorer Engine results display forms.
    • Fixed an issue where titles were not generated for attachments in rich preview for emails that including PST files.
    • Resolved an issue that caused document export display messages to appear scrambled under certain conditions.
    • Fixed an issue where the query-search API did not forward desirable clustering information to the query-cluster under certain circumstances.
    • Fixed an issue where ANSI encoded text files were not being indexed as attachments of PST files by the PST converter.
    • Resolved an issue where the knowledge bases for clustering were incorrectly displaying extra characters.
    • Fixed an issue when toggling the output-acls collection search option did not expire the cache.
    • Fixed an issue that caused the API function query-cluster to return empty responses or corrupt XML on Windows if a Lexical Analysis stream was specified.
    • Resolved an issue where delete references were not being purged properly for empty documents during merging.
    • Updated the BBC News federated source example so that it works with the current version of the BBC News website.
    • Addressed an issue that was causing the indexer to fail when using the japanese_files1.tst test file in a regression suite.
    • Resolved an issue that was causing Watson Explorer Engine to not release swap memory resulting in poor performance.
    • Fixed an issue in Lexical Analysis streams that was causing incomplete HTML tags to generate poorly formatted search results.
    • Resolved an issue that was causing the rating label to overlap with a rating star when using a 3 column display.
    • Fixed an issue when adding annotations to search results using a 3-column display was generating errors.
    • Fixed an issue when configuring the heap size values for the database connector or the keyed database connector, invalid heap size messages were incorrectly displaying.
    • Resolved an issue when stopping Watson Explorer Engine on Linux using the velocity-shutdown script caused a fatal error message to be generated.
    • Removed deprecated elapsed option in the Java converter framework.

    Watson Explorer Application Builder

    The following new features have been introduced to Watson Explorer Application Builder in Version 11.0.0:

  • Endpoints

    Custom functions that produce data for use by widgets and external systems. Endpoints can access entity data and external web services. For more information, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/c_de-ab-devapp-endpoints.html.

  • Custom Pages

    Pages that you create that are not associated with specific entities. You can display information from one or more entities on custom pages. Also, you can add a URL to a custom page from a widget i n another page. For more information about configuring pages, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/t_de-ab-devapp-pages-configure.html.

  • Modular Chrome

    Enables more control of the header chrome of the application. Elements of the application header are now widgets that you can configure, such as a search box widget and a mobile navigation widget. For more information, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/t_de-ab-devapp-header-populate.html.

  • Custom Head Tags

    Enables adding custom content such as meta tags within the head tag of the application pages. For more information, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/r_de-ab-devapp-global-display.html.

  • Contextual Access to the Administration Tool

    From the end-user application, edit links on widgets and pages take you directly to the location in the administration tool where you can modify the widget or page. Links for front-end access to the header layout and global configuration options are located in the new administrator user menu. For more information, seehttp://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/r_de-ab-devapp-global-display.html

  • Code Suggestion for the Query API

    Suggests methods for the query API in fields that you can enter Ruby code or erb (embedded Ruby) code, such as display options for entities and adding content for custom widgets. Methods are suggested automatically as you type. For example, entering entity_types("entity_name"). shows a list of query API methods and their descriptions that are valid in that context.

  • The following updates and improvements have been made to Watson Explorer Application Builder in Version 11.0.0:

  • Select Fields Per Entity Type

    Support was added to enable querying for specific fields of an entity type. For example, for a custom widget, to show only the contents of field1 and field2 for the first result where the Title field contains "thisText", you could enter the following code in the View field: entity_types("entity1").where(field("title").contains("thisText")).selecting_fields(["field1","field2"]).requesting(1).first

  • Authentication and Authorization

    LDAP configuration for Application Builder and IBM WebSphere Liberty profile server.xml file was removed from IBM Installation Manager. You can modify the default authentication configuration after installation. For information about configuring the server.xml file of the Liberty profile and the user entity for Application Builder, see the enhanced system administration documentation at http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/c_de-ab-system.html.

  • Query API Documentation

    The query API topic in IBM Knowledge Center contains additional documentation for several methods, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/c_de-ab-devapp-search-queryapi.html.

  • Facet API

    Calculation methods for standard deviation and average were added to the facet API. For information, see http://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.swg.im.infosphere.dataexpl.appbuilder.doc/c_de-ab-devapp-search-facets.html.

  • Themes

    The theme section of the global display options was removed. To control the color and other style characteristics of applications, enter custom CSS in the global display options.

  • Pages & Widgets Tab

    Renamed the tab to Layouts to accommodate the broader scope of configuration options that are available. Changes include the ability to create a custom page, subtabs for pages and the header, filtering on page names, and the reorganization of the pages into default pages, entity detail pages, custom pages, and entity list pages.

  • Comments on Secured Entities in the Activity Feed

    Fixed an issue where comments on secured entities were not listed in the activity feed for users who have access to the entity. Note: If security is enabled for entities that were previously not secured, activities that were generated before the entity was secured are not removed from activity feeds of non-privileged users.

  • User Entity Title Field

    Fixed an issue where the user entity, when connected to a directory data store, returns 0 results if the Title field in the "Display options" section is not configured to match a field that exists in the directory store. Now, the Title field defaults to sAMAAccountName if no value is set.

  • Rich Preview

    Fixed an issue where using Firefox to preview a document might repeatedly show a message that says an unresponsive script is running. The issue occurred only when previewing a long document that has many matches for the search term. A fix was made to give users the opportunity to continue waiting to find all the matches in the document, or to change the search term.

  • Watson Explorer BigIndex

    The following updates and improvements have been made to Watson Explorer BigIndex in Version 11.0.0: There are no updates to report for this version.

    Watson Explorer Results Module

    The following updates and improvements have been made to Watson Explorer Results Module in Version 11.0.0:

    • Upgraded WebSphere Application Server Liberty Profile to version 8.5.5.4


Known Issues in 11.0.0

At the time of publication, the following issues are known to be unresolved in the Watson Explorer foundational components:


Watson Explorer Engine

  • When upgrading to Watson Explorer Engine 11.0.0 from a version prior to 10.0.0.2, it is recommended that you perform a parallel installation.

1. Install Watson Explorer Engine 11.0.0 in a different directory than your current installation. See Installing Watson Explorer Product Modules for instructions.

2. Copy your search collection data, any modified files (modified files in your current installation that were originally installed by the Watson Explorer installer), and all manually created files from your old installation directory to the new installation directory. Most modified and manually created files will be stored in (but not limited to) the following directories:

  • install_directory/Engine/data/axl-pools/*
  • install_directory/Engine/data/dictionaries/*
  • install_directory/Engine/data/fonts/*
  • install_directory/Engine/data/key-match/*
  • install_directory/Engine/data/pears/*
  • install_directory/Engine/data/reporting/*
  • install_directory/Engine/data/repository-supplements/*
  • install_directory/Engine/data/sessions/*
  • install_directory/Engine/data/static/*
  • install_directory/Engine/data/su-tokens/*

These files may have been modified also:

  • install_directory/Engine/data/repository-internal.package
  • install_directory/Engine/data/ISYS-license

Search collection data is stored in the directory listed below by default:

  • install_directory/Engine/data/search-collections/*

Changing the location of this directory is common. For information on how to change the location of a search collection directory, see Relocating Collections. Note: It is recommended that you perform regular backups of Watson Explorer Engine. See Backing Up Watson Explorer Engine.

3. Verify that the new installation is working.

4. Uninstall the old version of Watson Explorer Engine and delete the search collection data. See Uninstalling Watson Explorer Modules for uninstall instructions.

  • In the Watson Explorer Engine Administration Tool, the following error can occur when attempting to update an existing scheduled job: "Can't get element with elt-id="<nnn>". Existing scheduled tasks cannot be updated as a result. To solve this problem, create a new scheduled task, enter the required fields, and save it. All previously existing tasks can now be edited. If desired, delete the new task after making your edits.
  • The velocity-shutdown command may display the error message "os: process already finished" and return a non-zero exit code when shutting down Watson Explorer Engine. To ensure Watson Explorer Engine has shutdown completely, re-run velocity-shutdown until it no longer displays the error message.
  • When upgrading from Watson Explorer 9.0.0.3 or earlier, if you applied the 9.0.0.3-WS-WatsonExplorer-Windows-HF001 hotfix posted on Fix Central prior to March 23, 2015, you will need to perform an additional step during your 11.0.0 installation. This is because the MD5 checksum values that were documented in that hotfix were incorrect, and as a result the files listed in the hotfix will not be deleted during the uninstallation of Watson Explorer 9.0.0.x. To work around this issue, do one of the following before starting your installation of 11.0.0:
    • Manually delete these files from your system after uninstalling the 9.0.0.x software.
    • Before uninstalling 9.0.0.x, go to Fix Central and reapply the hotfix. The MD5 values listed in the version posted on March 23, 2015 will delete the files correctly.
  • Installation Manager might crash when run on Red Hat Linux 6.5 because of an issue with out-of-date graphics libraries. To work around this issue, either modify the IM/IMLinux/install.ini file in the offering directory (see IBM Installation Manager can not start from GUI mode for instructions) or upgrade the Installation Manager libraries (see IBM Installation Manager and IBM Packaging Utility GUI crashes while advancing through wizard screens on recent updates of RHEL 6). Alternatively, you can use the console installer to completely bypass this issue.
  • Only an Administrator should start IBM Knowledge Center. If a non-Administrator starts an instance of the service, that instance might not be detected during an uninstall.
  • On Windows, there can be an issue deleting data on a search collection if the file path is longer than 260 characters. To work around this issue, keep collection names as short as possible, but descriptive (for example, "docs" instead of "crawled-documents") or shorten the path to the installation directory (or the collection's data directories if located elsewhere). Contact IBN Software Support if you see errors (for example, in system reporting) regarding an inability to delete files and those file paths are >= 260 characters.
  • If the Maximum JVM Idle Time is set to 0 and you enqueue a large number of documents, such as 50 or more, in quick succession, many of the enqueues are rejected with the following error: [JAVA_CONVERTERS_REMOTE_JVM_COMMUNICATION_FAILED] There was a problem connecting or communicating with the JVM used for running Java converters. Last system error was: Connection refused. To work around the issue, increase the maximum JVM idle time to a value greater than 5.
  • The template Java converter that is described in the tutorial for the Java converter might not compile with some versions of Apache Ant. The Java converter tutorial was tested and successfully compiled with Version 1.7.1, which is available from the Apache Ant archive page.
  • If a collection is configured with invalid parameters (for example, specifying a stemmer that does not exist), an issue might arise where the indexer service enters a continuous restart state. This functionality is documented in the Indexing section. Parameters that can contain invalid values are:
    • The stemmer of a custom language stream.
    • The path name for the PEAR file for a custom content analytics stream.
    • In the Advanced section of the global settings for the indexer, the Lexical analysis logging configuration field. This field has no administrative use other than potentially as a tool for complex debugging.
    • Also in the Advanced section, the Lexical analysis JVM arguments field.
    If you encounter this issue, stop the crawler, correct the configuration, and restart the indexer.
  • (Linux only) If you use the PDF to HTML converter for PDFs that use fonts of double-byte character sets, some text might not be extracted. Text that is not extracted is not included in the converted document. To work around the issue, you can use the PDF to text converter instead. Because the two converters work differently, you might see slight differences in the output of the converted documents.
  • When installing Watson Explorer Engine in non-admin mode on a Microsoft Windows system, the redistributable Microsoft Visual C++ runtime components that are required by Watson Explorer Engine are not installed correctly. This issue can prevent any of the Watson Explorer Engine Windows executables from running correctly. This problem does not occur if Watson Explorer Engine is installed by a user with administrative privileges. To resolve this problem, remove the non-admin installation and reinstall as a user with administrative privileges.
  • When performing a silent installation of Watson Explorer, if any nodes have been removed from a response file, the value for these nodes will be set to the default value.
  • When using continuous update mode for the Oracle WCC connector, filters are not supported if the universal query format is used. If filters are required, use the database query format.
  • When saving a refined query to a folder using the Watson Explorer Engine display, the less than symbol is not supported in the query label field. If the less than symbol is used, the label will be truncated to the content before the less than symbol.
  • Enqueueing a vse-key delete node on a distributed indexing server might result in one corresponding audit log entry on the client when two are expected.
  • When using the Web Parts for Watson Explorer, removing the HTTP authentication user name and password will fail if Remember passwords for sites is enabled in your browser.
  • When using the reporting feature, if the Split System Reporting Database option is set to something other than the default value of monthly, Watson Explorer Engine might incorrectly log some error messages to the monthly reporting database. Similarly, the System Reporting Database Maximum option is ignored, so it might grow in size until hitting the default maximum limit of 2GB.
  • When adding a new seed to a collection, the list of seed types to choose from includes vse-crawler-seed-TEMPLATE. This seed is intended for IBM internal development and should not be used.
  • If you remove a connector plug-in, a search collection will continue to refresh without generating an error message that the connector plug-in was removed.
  • 
    	#search-primary table.form tr.search-inputs {
    	display:block;
    	z-index:1000;
    	position:relative;
    	}
    	
    	#search-primary table.form tr.below {
    	display:block;
    	position:relative;
    	z-index:0; }
    	
  • The Watson Explorer Engine internal web server could be vulnerable to the denial-of-service attack (DoS) that is described on the Apache wiki. If you are using the Watson Explorer Engine internal web server as an Internet facing portal, follow the mitigation steps listed at that link.
  • The vsw-form erroneously states that XML is acceptable for the document query condition field. This text is incorrect. XML should not be used for the document query condition field.
  • Bulk collaboration features (express tagging and renaming, adding, or deleting folders) have crawl-url priority of 1, which might result in a delay for users during normal refresh operations.
  • Links in the crawl graph might not be consistent if crawl URLs are enqueued with the parent-url attribute set. This behavior can cause a potential issue when switching a client to a server in distributed indexing if you are using the IO SharePoint Connector.
  • The use of non-ASCII characters in a Watson Explorer Engine installation path is not supported.
  • Calling the search-collection-read-only-all API function on a server with a large number (thousands) of collections results in increased CPU and memory usage, and can make the server appear to be unresponsive while the search collections on that server are being processed.
  • When using distributed indexing, if all data is deleted from the master server and a crawl is initiated, the client servers never attempt to synchronize, nor do they complain about the data mismatch.
  • Database Connector 5.0.1: In this version of the Database Connector, the document URLs will include the fetch size. This should not have any consequences, unless the fetch size is changed while continuous update is enabled. The next version of the connector will be updated to be consistent with earlier versions. That version will be available in the next release of Watson Explorer, and on FixCentral.
  • IO SharePoint connector: When building access control lists (ACLs), the connector does not check the web application policy. When security is enabled for the search collection, the SharePoint user for the crawling account cannot see search results if the account was granted only Full Read permissions at the web application policy level. To work around the issue, grant the SharePoint user for the crawling account Full Read permissions at the site level and document level.
  • IO SharePoint connector: In some cases, this connector may not return all results. To work around this issue, you need to make an XML configuration change to "sharepoint-simple-windows-cba-rights" function node, which can be found by clicking on XML in the Administration Tool and navigating to "sharepoint-simple-windows-cba-rights" in the All XML list. (Or enter "sharepoint-simple-windows-cba-rights" in the quick jump box.) Click the Override button, then the Edit button, and replace this:

    <xsl:variable name="ldap-key">
            <xsl:choose>
                <xsl:when test="viv:vivisimo-version('major') &gt;= 10">tokenGroups;binary::</xsl:when>
                <xsl:otherwise>tokenGroups;binary:</xsl:otherwise>
            </xsl:choose>
        </xsl:variable>

    With this, and then click OK:

    <xsl:variable name="ldap-key">
            <xsl:choose>
                <xsl:when test="viv:vivisimo-version-is-newer-than('10')">tokenGroups;binary::</xsl:when>
                <xsl:otherwise>tokenGroups;binary:</xsl:otherwise>
            </xsl:choose>
        </xsl:variable>
  • In some cases, large documents (for example, large Microsoft Excel files) will fail during conversion, with an error message that references the ISYS_XML_MEMORY_LIMIT environment variable. To workaround this issue, set the environment variable ISYS_XML_MEMORY_LIMIT high enough to allow the converter to process the document.
  • Nested document nodes can cause the query-meta application to exit abnormally.
  • The Watson Explorer autocomplete feature does not collect reporting information.
  • Watson Explorer might fail to open files with a path length of over 255 characters on Microsoft Windows systems.
  • When the Watson Explorer Engine Search Web Parts repair installer is used, the repaired features are left inactive and must be manually activated in each site collection.
  • When using the body toggle option in the custom document section of the Watson Explorer Engine display, the toggle option will not appear in the search results if the body toggle option does not specify any non-annotation, non-zero length content elements. For example, if rating (annotation) and URL (document attribute) were specified as body toggle fields, the body toggle option would not appear in the search results. If a content element (such as snippet) was also specified, the body toggle option would be present in results that have a non-zero length snippet content element. This situation is avoided by always specifying a non-zero length content element as a body toggle field.
  • By default, Log4J logging is set to debug. If performance for that connector is an issue, you can turn off logging in the Watson Explorer Engine administration tool by navigating to the Configuration tab for your search collection seed and adding the following logging settings in the Advanced - Logging section:<root><priority value='OFF'/></root> Note that disabling logging will make it difficult to trace future issues.

  • When crawling an SMB file system that is configured with NETBIOS to limit which workstations the crawling account can log on with, the user not allowed to log on to this computer error message can occur if the computer VIV_CRAWLER is not present in the list of allowed workstations.

Watson Explorer Results Module

  • If you query a Results Module collection and subsequently edit the query service port, you will no longer be able to see query expansions or spotlights from Results Module collections. This behavior can be resolved by editing the environment-get-config node in Watson Explorer Engine to refer to the correct query service port.
  • Multiple identical spotlights can be created within the same spotlight collection.
  • An ActiveRecord::RecordNotFound in termsController#update error is returned if a user attempts to save a term or spotlight that was deleted by another user.
  • This version of Results Module supports the following maximum number of contents:
    • Maximum number of terms: Approximately 5000
    • Maximum number of spotlights: Approximately 5000
    • Number of spotlight images: At least 50
  • Results Module calls to the Watson Explorer Engine API send user and password information in the clear. Configuring the Results Module to use SSL will prevent these parameters from being displayed. See the Watson Explorer Results Module documentation for further details.
  • An exception might occur if an environment becomes invalid due to an external reason. Possible reasons for an external environment becoming invalid include:
    • Deleting the API user defined for the environment
    • Changing the password of the API user defined for the environment
    • Changing the URL of the installation specified in the environment
    • Removing the Watson Explorer Engine installation specified in the environment
    • Changing the HTTPS security settings of the installation specified in the environment

    The exception can be avoided if all affected environments are updated before a user attempts to access any of the components.

  • If a long running action (for example an import) times out, other requests (such as login) might occasionally time out.
Spotlight Manager
  • If you use Internet Explorer 11 to access Results Module, you cannot select an image to add to a spotlight in the Spotlight Manager. To work around the issue, add the fully qualified domain name of Results Module to the list of sites that use Internet Explorer Compatibility View. For more information, see Spotlight Manager: Cannot add images when using Internet Explorer 11.
  • If your lowest environment is not correctly pointing to your Watson Explorer Engine installation, saving a spotlight (and possibly terms and collections as well) will fail with no error message.
  • When you delete a spotlight, it will not be immediately deleted in Watson Explorer Engine, but is guaranteed to be enqueued for deletion and will be deleted eventually. This behavior means that if you perform a search that would have returned the spotlight you just deleted, you might still see the spotlight.
  • It is possible to create spotlights whose names are comprised of spaces. This feature does not cause a functional issue, but links to those spotlights in the Results Module might be difficult to use.
Terminology Manager
  • Importing terms that contain commas in their entries causes inconsistent behavior: in the main term, the comma is retained; in the associated terms for an entry, it is treated as a delimiter.
  • It is possible to import a term collection directly to an environment that is not the lowest environment in the promotion chain. This action should not be done, as the environments and promotion chain will become out of sync.
  • Adding multiple relationships between two terms causes a Watson Explorer Engine exception error. For example, this exception occurs if you add colour as a spelling variation of color, and then attempt to add colour as a synonym of color.

Watson Explorer Application Builder

  • Anonymous LDAP binding for the user entity directory store is not supported.
  • For large LDAP directory stores, if virtual paging is not supported, an error can occur because of a lack of memory.
  • Configuring a nested facet to be displayed causes the search page of the application to return 0 results.
  • In Application Builder, when you use the Refine by type option for a refinement widget, the refinements are displayed as the value of the entity name. This entity name is different from the entity name that is displayed in the header of the application if the display name field is populated in the entity configuration.
  • iWidget Widget

    The application detail page fails to load when an iWidget is created without values in one or more fields in the Type-specific configuration section.

  • Iframe Widget

    • The iframe widget contains an iframe, so it has any limitations that an iframe has for a given browser.
    • In some browsers, when you move an iframe widget between columns on the home page, the widget might not resize as accurately as other widgets.
  • Flip Widget

    • Some widgets behave incorrectly if multiple copies of them exist on the page, for example multiple chart widgets. The flip widget contains multiple nested widgets. If a widget is nested within the flip widget, you can still add a copy of that widget to the page. This behavior might result in the end user seeing one stand-alone version of the widget and one nested version of the widget in the flip widget.
    • When you add a flip widget to the page from the widget bar on the home page, the remove icon will not be visible in the header of the flip widget. This behavior means that you cannot immediately remove the widget again. You can work around this issue by reloading the page or closing and re-opening the widget bar.

Watson Explorer Annotation Administration Console

  • Error FFQG0024E indicates that a NullPointerException occurred after you exported an annotator from Content Analytics Studio to Annotation Administration Console. To avoid this problem, ensure that the following check boxes are clear in the Export to Content Analytics window before you export an annotator:
    • Restart the collection's index service
    • Re-index existing collection documents
    These options are not needed because collections in Annotation Administration Console do not include an index service or documents. Annotation Administration Console, you must enable the exported annotator in the target collection by deploying analytic resources and restarting the text analytics session.
  • You might see misleading messages when you attempt to start the Annotation Administration Console after you run the 'esadmin system startall' command, such as "Context root not found" or "Unable to read the configuration data for collection .". You can ignore these messages. Services required to run the administration console are still being started. Wait a couple of minutes and refresh your web browser before attempting to access the administration console again.

IBM Software Support

As problems are discovered and resolved, the IBM Support team updates the IBM Support Portal. Search the IBM Support Portal to find workarounds or solutions to problems.

[{"Product":{"code":"SS8NLW","label":"IBM Watson Explorer"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"--","Platform":[{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"11.0.0;11.0.0.1;11.0.0.2;11.0.0.3","Edition":"All Editions","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
17 June 2018

UID

swg27045729