OA14068: TEXT SEARCH ALTERNATIVE USING TSXS

Obtain the fix for this APAR.

APAR status

• Closed as new function.

Error description

• Currently, InfoMan supports freeform text searching using
  the Text Search Engine (TSE) shipped with z/OS 1.6 and lower.
  This APAR will ship new TSXs that can be used in place of
  TSE to cognize and search freeform text.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: This is a new function APAR.  Users          *
  *                 who wish to take advantage of the            *
  *                 new cognize FFT process described            *
  *                 below should apply the PTF.                  *
  ****************************************************************
  * PROBLEM DESCRIPTION: A TSX alternative process for cognizing *
  *                      FFT is provided by this APAR.  This     *
  *                      process may be used instead of the      *
  *                      Text Search Engine (TSE).               *
  *                      *SPARKLER*                              *
  ****************************************************************
  * RECOMMENDATION: Apply the PTF, run TSX BLHRCDSL with         *
  *                 record list BLHL5BA5, and run BLGUT6.        *
  ****************************************************************
  Several new TSXs will be provided by this APAR to form a
  process that will take the FFT data in an InfoMan record and
  cognize it by storing the unique and desired words from the FFT
  text into a cognized List Processor (LP) field in the record.
  The Program Administrator (PA) will determine the record types
  that they want to cognize FFT in and then the specific FFT text
  entry types (e.g., s-word index for description text) that they
  want to cognize in those records.  The PA will also set up a
  list of words that they don't want cognized (to ignore) even
  when they exist in the text.
  
  The cognize FFT process will be run in batch.  The batch job
  will do a search to generate a Search Results List (SRL) of the
  desired records to be processed and will then run a TSX to
  initiate the cognizing.  Subsequent runs of the batch job will
  use a modified search that typically will find all the records
  that should be processed that have had FFT modified since the
  last cognize FFT process run.
  
  In addition, a TSX tool is provided to help the PA understand
  the potential words to be cognized so that they can determine
  what words to add to the list of words that should not be
  cognized.
  
  There are 4 new TSXs:
    BLGTPCMD - Process a command string
    BLGTCOGF - Cognize freeform text
    BLGTCNTT - Count freeform text words
    BLGTNCOG - Process the list of words that should not be
               cognized, called internally, contains user
               modifiable variables
  There are 2 new validation records:
    BLGVNCOG - Contains the list of words that should not be
               cognized
    BLGVFFTS - Contains the s-word indexes associated with
               the freeform text to process
  There are 9 new panels:
    BLH0BU00 - Default problem summary panel
    BLH1A111 - Default problem file time control panel
    BLH0B100 - Selection panel for cognize word list or sort
    BLHLWORD - List Processor (LP) panel for cognize word list
    BLH2WORD - Help for panel BLHLWORD
    BLH6WORD - Assisted-Entry Panel (AEP) for BLHLWORD
    BLHLVLST - LP panel for cognize sort
    BLH2VLST - Help panel for BLH2VLST
    BLH6VDES - AEP for BLHLVLST
  
  Post Installation Set-up Work:
  - Determine the desired record types for FFT cognizing.
  - Determine the desired text types within those records for
    FFT cognizing.
  - Create or choose a privilege class that has the necessary
    authorities for the FFT cognize process.  The class will need
    display and update authority for all the record types
    that will be processed and DBADMIN 0AA1 authority.
  - Create a batch job for TSX BLGTCNTT and TSX BLGTPCMD/BLGTCOGF.
  - Optionally, update the COMMAND record with the command alias
    and string for TSX BLGTPCMD. It may be necessary for the PA
    to first create the COMMAND record.  The PA needs to add an
    entry similar to this one in their COMMAND record:
      %FFT   ;RUN BLGTCOGF CURRENTONE
  - Optionally, update the ALIAS record with entries for the TSXs
    involved in the process to maximize performance.  It may be
    necessary to first create the ALIAS record.
  - The PA should verify that the BLGVFFTS and BLGVCOGN records
    were loaded by TSX BLHRCDSL with APAR record list BLHL5BA5.
  - Review the user modifiable parameters section in the TSXs
    and make changes as necessary.  If InfoMan commands have been
    remapped in the COMMAND record, the PA may need to update
    some of the user modifiable variables.  TSX BLGTNCOG also
    has additional processing variables which should be reviewed.
  - Optionally, run TSX BLGTCNTT in batch to determine potential
    additional words to add to the list of words that should not
    be cognized in record BLGVNCOG.  The IRC should look something
    like:
      ISPSTART PGM(BLGINIT) PARM(SESS(xx) CLASS(class)
      IRC(6,1,SE + DATE/2004/01/01 -12/31,;RUN BLGTCNTT,;QUIT))
    The search argument provided should generate a SRL with a
    good representative number of records to be cognized on a
    subsequent run of TSX BLGTCOGF.  In this example, all the
    problem records created in 2004 are reviewed.  TSX BLGTCNTT
    will create a list of all of the words found in the records
    and how many times each word exists in the records.  Words
    already in the BLGVNCOG record will not be included in the
    lists.  The words will be sorted in three different ways:
    most to fewest, fewest to most, and alphabetical order.
  - Optionally, update record BLGVNCOG.  Note:  No method is
    provided to only exclude words by record type.
  - Update record BLGVFFTS to list the FFT types to cognize in the
    validation entry column.  The default entry is S0E01 for
    description FFT.  The entry description column can be used to
    exclude a given FFT s-word to one record type.  Specify the
    record type s-word (e.g., S0032 to limit to problem records).
  - Hook up the cognize FFT panels to each record type that will
    be processed.
    * Create a new VSAM panel data set.
    * Create a new session-parameters member that puts the new
      VSAM panel data set first in the read panel concatenation.
    * Place a default file time control panel for each record
      type that FFT will be cognized for in the new VSAM panel
      data set.  To do this, the PA will determine the file time
      control panel name for each record type that FFT will be
      cognized for and then the PA will copy panel BLH1A111 and
      call it each of those names.  the PA may also have to
      update the newly copied panels and change the name of the
      record access panel to the appropriate record access panel
      for each record type.  BLH1A111 as shipped contains
      BLG1A115 as the record access panel which is for problem
      records.  For example, if change records will have FFT
      cognized, BLH1A111 will be copied to BLG1A121, and then
      BLG1A115 will be changed to BLG1A125.  The Web Access 1.2
      book on page 217 contains instructions on how to determine
      the record access panel (SC32-1427-00).
    * Place a default summary panel for each record type that
      FFT will be cognized for in the new VSAM panel data set
      (i.e., copies of BLH0BU00).  For out-of-the-box problem
      records, BLH0BU00 will be copied to BLG0BU00.  For
      out-of-the-box change records, BLH0BU00 will be copied
      to BLG0CU00.  The PA may also have to update the control
      information behind the 9. File record. selection and
      change the file time control panel name.  As shipped,
      BLH0BU00 specifies BLH1A111 for problem records.
  - Optionally, the PA should consider whether or not it is
    desirable for end users to be able to display or update
    the cognize word lists.  If so, panel updates will need
    to be done on the end user session parameters member.
  - Verify that there is enough space in the SDLDS for the
    extra record update activity that the cognize FFT process will
    generate.  The size of the SDLDS can be increased, the SDLDS
    can be offloaded using BLGUT4 more often, or this process can
    be run without an SDLDS.
  - Verify that there is enough space in the SDIDS for the
    new cognize word bitlists.  There will be an immediate need
    for extra SDIDS space, but the amount of new space will drop
    off dramatically after the first few cognize FFT process runs.
  - Monitor the size of the average SDDS record.  The average size
    is going to increase because of the new list processor words.
    More space may be required for the SDDS.  The PA should plan
    to run BLGUT20 before implementing the cognize FFT process
    and again several months after implementing it to compare the
    average SDDS record size and determine if the CISIZE and LSR
    buffers are still efficient.
  
  Once the set-up work is complete, submit TSX BLGTPCMD/BLGTCOGF
  in batch with the newly created session-parameters member and
  appropriate class.  The IRC should look something like:
    ISPSTART PGM(BLGINIT) PARM(SESS(xx) CLASS(class)
    IRC(6,1,SEARCH + DATM/2005/08/18 ¬INDX/COGFFT,
        ;RUN BLGTPCMD %FFT,;QUIT))
  In this example, all the problem records modified on 08/18/2005
  will be processed.
  
  The PA will need to determine what the requirement is for
  cognizing FFT at their site.  Some choices might be hourly,
  daily, or weekly. Below is a sample that includes time modified:
    ISPSTART PGM(BLGINIT) PARM(SESS(xx) CLASS(class)
    IRC(6,1,SEARCH + DATM/2005/08/18 TIMM/01:00-59
        ¬INDX/COGFFT,;RUN BLGTPCMD %FFT,;QUIT))
  
  Input requirements:
  BLGTPCMD
  - TSX BLGTPCMD must be run from a SRL with hits.
  - TSX BLGTPCMD accepts an input parameter of a command string
    to be processed in update mode against each record in the SRL.
  BLGTCNTT
  - TSX BLGTCNTT must be run from a SRL with hits.
  - TSX BLGTCNTT does not accept any input parameters.
  BLGTCOGF
  TSX BLGTCOGF requires an RNID as an input parameter.  That
  parm may be passed as a literal or obtained from the record
  that is currently being updated (if CURRENTONE is used).  All
  the other parms are optional.
  
  RUN BLGTCOGF { rnid | CURRENTONE }   REDO     NOCHECK
                 LLlll    DEBUG
  
  rnid - The ID of a specific record to be processed.  Required
         unless CURRENTONE is specified.
  CURRENTONE - Process the current record.  Either the update
               command has been entered against the current
               record, or TSX BLGTPCMD is looping through the
               records in a SRL.  When CURRENTONE is entered, the
               record update summary panel must be the current
               panel. Required unless a specific RNID is passed.
  REDO - Discard all existing cognized text data in the record
         and rebuild it completely.  REDO may be faster than
         NOCHECK since it starts from scratch and does not have
         to process a delta between what's currently in the
         record and what is to be added.  Optional.
  RELOAD - Call BLGTNCOG to create the word and FFT stems.
           Note: If these do not exist, then BLGTNCOG will be
           called to create them.  Use RELOAD if you change
           validation record BLGVNCOG or BLGVFFTS.  Optional.
  NOCHECK - Process text even if the INDX/COGFFT value exists
            in the record indicating that the text has not been
            updated since it was cognized.  Optional.
  LLlll - Maximum length of data to use in the cognize word
          column.  The value can be 32 to 200.  A larger value
          helps BLGTCOGF process faster.  Since you can only see
          72 columns on BLHLWORD, you will have to use the View
          Internals command to see any data beyond column 72 if
          values > 72 are used.  Example, LL45.  Optional.
  DEBUG - Turns on debugging which causes extra information
          to be output by the EXEC in the form of SAY statements.
          Optional.
  
  Messages:
  BLG20442I TSX &1:  BLGTNCOG failed with RC = &2.
  BLG20443I TSX &1:  &2 variable pool &3 failed with RC = &4
            and Text = &5.
  BLG20446I TSX &1 requires at least one parameter.
  BLG20448I TSX &1 is unable to flow between summary panel &2 and
            the cognize freeform text process panels for RNID &3.
  BLG20449I TSX &1:  Unable to return to summary panel &2
            from panel &3 for RNID &4.
  BLG20450I TSX &1:  There is no list of words that should not be
            cognized in the BLGVNCOG record.
  BLG20451I TSX &1:  No freeform text s-word indexes were located
            in the BLGVFFTS record.
  BLG20452I TSX &1:  S-word index &2 found in the BLGVFFTS record
            is invalid and is ignored.
  BLG20453I TSX &1:  None of the freeform text s-word indexes
            specified in the BLGVFFTS record are valid.
  BLG20454I TSX &1 must be run from a Search Results List that
            has hits.
  BLG20455I TSX &1 will attempt to process &2 records using
            command string &3.
  BLG20456I TSX &1 was unable to process RNID &2.
  BLG20457I TSX &1 processed &2 records using command string &3.
  BLG20458I TSX &1 ended in error with message:  &2.
  
  BLGTNCOG user modifiable processing parameters:
  - COGFFT.DEFAULT.IRC_Summary_Panel_2_Sort_Panel = '10,2'
    The IRC to go from the default summary panel to the sort
    panel, BLHLVLST.
  - COGFFT.DEFAULT.IRC_Sort_Panel_2_Old_Words_Panel =
    cancel_command',1'
    The IRC to go from the sort panel back to the default summary
    panel.
  - COGFFT.xxxxxxxx.IRC_Summary_Panel_2_Sort_Panel = '10,2'
    The IRC to go from a customized summary panel to the sort
    panel, BLHLVLST.
  - COGFFT.xxxxxxxx.IRC_Sort_Panel_2_Old_Words_Panel =
    cancel_command',1'
    The IRC to go from the sort panel back to the cognized words
    panel through the customized summary panel.
  - COGFFT.LP_S_Word_For_Cognizing = 'S0451'
    The s-word index for the s-word that will be used to cognize
    the FFT words as part of a list.  If the PA decides to change
    this, the PA must update panels BLHLWORD and BLH6WORD to use
    the desired s-word instead of S0451.
  - COGFFT.special_characters = ' &#172;%<>:_+-;,."?*=/()|!'||"'"
    Characters identified as special characters.  They get
    translated to blanks in TSX BLGTCOGF.  InfoMan has processing
    to turn special characters to blanks during cognizing.  By
    translating the special characters to blanks before entering
    them on panel BLHLWORD, the words displayed on BLHLWORD will
    exactly match what is cognized.
  - COGFFT.default_line_length = 72
    The maximum string length of the words entered onto panel
    BLHLWORD.  The AEP, BLH6WORD, accepts up to 200 characters,
    but only the first 72 are visible without scrolling.  Most
    PAs will probably want to stick with 72.  The LL input
    parameter to TSX BLGTCOGF can be used to pass the string
    length.  This value is used if LL is not passed.
  
  Additional information about TSX BLGTPCMD:
  The purpose of TSX BLGTPCMD is to take an input command string
  and process it against every record in the current Search
  Results List (SRL).  Although a command string may be specified
  directly as an input parameter to TSX BLGTPCMD, that method is
  not practical since command strings typically include commas
  and a comma is the default command delimiter.  For example, if
  TSX BLGTPCMD is invoked with:  RUN BLGTPCMD 1,14,closed,,;end
  only the value '1' will be passed to TSX BLGTPCMD.  The first
  comma in the command string 'breaks up' the string.  The best
  method for providing TSX BLGTPCMD with a command string is
  to set up a command alias in the COMMAND record.
  
  Note:  Since the command alias in the COMMAND record must start
  with an actual command, often the command alias will be just to
  run a TSX.  Other ways around this are to enter an unnecessary
  command on the start of the 'real' command
  (e.g., ;help,;end,14,closed,,;end).
  
  The command string should file any changes to the record that
  should be saved.
  The PA can add additional words to the list of words that should
  not be cognized in validation record BLGVNCOG at any time.
  However, the additional words will only be excluded from text
  that is processed after the addition.
  
  The PA can limit text processing to specific record types using
  record BLGVFFTS.  In the following example, FFT text s-word
  index S0E01 (description text) will be processed for all record
  types.  FFT s-word index S0E03 (resolution text) will only be
  processed for problem records (S0032):
  
   BLGLVLST   VALIDATION DATA ENTRY     LINE 1 OF 10
   USE....Enter a list of validation data.
   FORM...Valid response, maximum of 30 characters for validation
          entry.                  RECORD:  BLGVFFTS
       Prefix  Prefix Validation    Validation          Entry
   ''''  ____  ______  _________  S0E01___________  _____________
   ''''  ____  ______  _________  S0E03___________  S0032________
  
  TSX BLGTCOGF adds the unique and desired words using S0451.
    0451  XISFFTCG  IS=FF TEXT COG NO FREEFORM TEXT COGNIZE LIST
  If more than one type of text is cognized, all of the cognizable
  words from each type of text are put into the one list.
  
  Some conditions may cause records to fail processing (e.g.,
  record in update).  BLGTCOGF keeps a list of records that fail
  and retries each record once.  Records that fail on the retry
  are marked as failing.
  
  At the end of TSX BLGTCOGF processing, a FFT index marker is
  also added to the record.  This marker looks like:
    BLGTXADD C/2 00 B/U 0D/00/00 S12E3 BC IM0ICINDX INDX/COGFFT
  and can be seen using the View Internals command.  This marker
  is used by TSX BLGTCOGF to make logic decisions on subsequent
  runs against the same record.  The index marker is removed by
  any subsequent text updates to the record, so if BLGTCOGF runs
  and the index marker is missing, BLGTCOGF will cognize the text.
  

Problem conclusion

Temporary fix

Comments

• Parts are modified or added as needed.  V71MNEW
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• BLGTCNTT BLGTCOGF BLGTMSGS BLGTNCOG BLGTPCMD
  BLGVFFTS BLGVNCOG BLG4T442 BLG4T443 BLG4T446 BLG4T448 BLG4T449
  BLG4T450 BLG4T451 BLG4T452 BLG4T453 BLG4T454 BLG4T455 BLG4T456
  BLG4T457 BLG4T458 BLHLVLST BLHLWORD BLHL5BAS BLHL5BA5 BLH0BU00
  BLH0B100 BLH1A111 BLH2VLST BLH2WORD BLH6VDES BLH6WORD
  

Fix information

• Fixed component name

  INFO MAN Z/OS 7

• Fixed component ID

  5697SD900

Applicable component levels

• R103 PSY UA22856

     UP05/12/14 P F512

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.