Enabling IBM Text Search for Db2 for z/OS with large objects

When you work with text search indexes, there are indexing limits for objects, such as the maximum size for a document or the maximum number of characters in a document. However, optimizing settings within Db2 for z/OS and IBM® Content Manager can help you avoid text indexing failures on large objects.

About this task

The maximum document size is 60 MB. IBM Text Search for Db2 for z/OS limits the number of Unicode characters that can be indexed for each text document. Sometimes this character limit results in the truncation of large text documents in the text search index.

The default value for the number of Unicode characters that are allowed for each text document is 10 million. For a rich text document, this limit is applied after the document is transformed to plain text. Therefore, you can successfully index large file sizes, for example, 60 MB, that have a maximum of 10 million Unicode characters of text.

If a text document is truncated during the parsing stage, you receive a warning that some documents were not processed correctly or completely, and the document is partially indexed. Details about the warning are written to the event table that is created for the text search index. Text that is in the document after the limit is reached is not indexed and cannot be searched.

Procedure

To enable text search indexing of large objects:

  1. When you define the item type, use the following table as a guideline for planning your text indexing strategy.
    Use the following Update Frequency and Commit Count parameters in the Text Search window for item type definitions.
    Table 1. Parameters that you can use for item type definitions
    File size File type Commit count (Number of files per index update) Update frequency
    50 MB PDF, Word, Excel, 20% (estimated) text 5 Every hour or less frequently
    10 million Unicode characters of text text, 100% text 5 Every hour or less frequently
  2. Depending on the size, number, or frequency, provide adequate hardware resources to keep the system from becoming degraded.
    Recommendation: Create and run stress and endurance tests with these large objects to determine an optimal set of resource requirements before you move into the production phase.