PH24872: LOGSTREAM DATASETS DELETE PENDING

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• Directory Full Occurred due to LogStream (LogStream)
  DataSets Delete Pending
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All IBM Z Decision Support (IZDS), IBM Z     *
  *                 Decision Support for Capacity Planning       *
  *                 (IZDS for CP) and IBM Z Performance and      *
  *                 Capacity Analytics (IZPCA) who use the       *
  *                 following functionality:                     *
  *                   Automated Data Gathering for SMF data.     *
  *                   Streaming curated data off platform.       *
  *                   Forecasting (IZDS for CP and IZPCA only)   *
  *                   Collator (IZPCA only)                      *
  ****************************************************************
  * PROBLEM DESCRIPTION: 1. Storage for the log stream data sets *
  *                      is exhausted or higher than expected.   *
  *                                                              *
  *                      One or more data sets show as being     *
  *                      in a DELETE PENDING status when the     *
  *                      log stream is queried.                  *
  *                                                              *
  *                                                              *
  *                      2. The Data Mover stops processing      *
  *                      data but does not abend and the         *
  *                      joblog displays the following           *
  *                      messages:                               *
  *                      Exception in thread                     *
  *                      "Spoke(1)_Output_0"                     *
  *                      java.lang.NullPointerException at       *
  *                      com.twentyfirstcsw.Data Mover.          *
  *                      output.OutputTCPIP.clearBuffers         *
  *                      (OutputTCPIP.java:704)                  *
  *                      at com.twentyfirstcsw.Data Mover.       *
  *                      output.OutputTCPIP.run                  *
  *                      (OutputTCPIP.java:407)                  *
  *                      at java.lang.Thread.run                 *
  *                      (Thread.java:818)                       *
  *                      There is no preceding DRLJ message.     *
  *                                                              *
  *                      3. The Data Mover stops processing      *
  *                      data but does not abend and the         *
  *                      joblog displays the following           *
  *                      message:                                *
  *                      DRLJ0274I Exception received trying     *
  *                      to RELEASE Enqueue null                 *
  *                                                              *
  *                      4. The Data Mover fails to complete the *
  *                      shutdown request and the joblog         *
  *                      displays the following message:         *
  *                      DRLJ0110I Hub(1)                        *
  *                      TCPIP-XXX.XXX.XXX.XXX:XXXXX Input       *
  *                      generated 251161 packets                *
  *                      Exception in thread "Stopper"           *
  *                      java.util                               *
  *                      .ConcurrentModificationException        *
  *                      .at java.util.LinkedList$ListItr        *
  *                      .checkForComodification                 *
  *                      (LinkedList.java:977)                   *
  *                                                              *
  *                      5. A shutdown was issued and the Data   *
  *                      Mover stops processing                  *
  *                      but does not shutdown. The joblog       *
  *                      displays the following message:         *
  *                      DRLJ0022I Engine stopped; Running       *
  *                      Cleanup                                 *
  *                                                              *
  *                      6. The Data Mover initialises but does  *
  *                      not process any data when configured    *
  *                      with a Split and Join stages.           *
  *                                                              *
  *                      7. The Data Mover initialises but stops *
  *                      processing data.The job log may display *
  *                      a JAVA exception.                       *
  *                                                              *
  *                      8. The build timestamp message isn't    *
  *                      sufficiently granular to identify the   *
  *                      precise build.  Current message is:     *
  *                      DRLJ0089I Level: 19_PH24872             *
  *                      Build: MikG3-61                         *
  *                      Built: 2020 June 18 16:00:50            *
  *                                                              *
  *                      9. The JSON and SQL conversion code     *
  *                      will fail when making forecasts with    *
  *                      the following errors:                   *
  *                      java.time.format.DateTimeParseException *
  *                      Text '2020-06-09-' could not be parsed  *
  *                      at index 11                             *
  *                      at java.time.format.DateTimeFormatter.  *
  *                      parseResolved0                          *
  *                      (DateTimeFormatter.java:1960)           *
  *                                                              *
  *                      10. Packed decimal values are much      *
  *                      larger than expected when reading       *
  *                      via JDBC.                               *
  *                                                              *
  *                      11. If a job is running with multiple   *
  *                      routes, when the first route closes     *
  *                      the whole Data Mover terminates, even   *
  *                      if the other routes are not yet         *
  *                      complete.                               *
  ****************************************************************
  * RECOMMENDATION: Apply the provided PTFs.                     *
  ****************************************************************
  1. If the Data Mover is restarted and a significant backlog of
  records has built up in the log stream, it will start reading
  from an offloaded data set.  This leaves an ENQUEUE in place
  that later prevents the system logger from deleting the
  offloaded data set.  Ending the log browse session releases the
  ENQUEUE and permits deletion to occur.
  
  2. If the TCPIP sessions disconnects during catch up the Output
  TCPIP stage can abend, leaving the stage non-functional.  The
  Data Mover is not detecting this situation and shutting itself
  down as it should do.
  
  3. Following a problem writing to a log stream a null pointer
  exception may occur trying to free an enqueue called 'null'.
  
  The previous error also resulted in the suppression of the
  details of the log stream write error that triggered it.
  
  4. While a controlled shutdown of the Data Mover is ongoing a
  concurrent modification exception can be received from the
  TCPIP input queue.  This causes the shutdown process to
  abort, resulting in a hung Data Mover.
  
  5. If an uncaught exception occurs on a Data Mover stage
  thread the result is that the Data Mover stops processing
  data but doesn't shutdown.  It's not immediately obvious
  to the user that it isn't working.
  
  6. If the Data Mover is asked to run an invalid configuration,
  where a SPLIT stage is set to duplicate data to a channel
  that is not defined by a JOIN stage the Data Mover hangs
  during initialization.
  
  7. Uncaught exceptions occurring during general stage
  processing are not being trapped and used to trigger
  Data Mover shutdowns even though they prevent it from
  processing data.
  
  8. The text of the DRLJ0089I message has been enhanced
  to further identify the build and APAR level.
  
  9. When streaming data as JSON or CSV, if the table was
  defined with a column of type DATE, TIME or TIMESTAMP
  in its data section, this would confuse the timestamp
  extraction logic leading to either a parsing exception
  or an incorrect timestamp.  The fix is to make the
  timestamp extraction logic ignore DATE, TIME and
  TIMESTAMP fields that are a not a part of the primary
  key.  The primary key is expected to contained one of
  the following
  A TIMESTAMP column
  A DATE column
  Or both a DATE and a TIME column.
  
  10. When any of our Java products
  reads a table via JDBC, if this table
  has Packed Decimals these will be
  read incorrectly.
  
  11. If a job is running with multiple routes,
  when the first route closes the whole Data Mover
  terminates, even if the other routes are not yet
  complete.
  

Problem conclusion

• 1........*.........*.........*.........*.........*.........*..64
  1. The Data Mover logic that reads the log stream is enhanced
  to close and re-establish the log browse session once an hour.
  This should prevent any long term issues with ENQUEUEs on
  offload datasets.  Customers are advised to have enough log
  stream storage available to hold several hours worth of heavy
  traffic.
  
  2. The Output TCPIP stage has been enhanced to avoid the null
  value that was triggering the failure and to catch future
  uncaught failures.
  
  3. Following a log stream error, the Data Mover may encounter an
  unexpected java exception which results in the
  Data Mover no longer processing data.  The cause of
  the exception has been fixed, logic has been added to
  capture similar exceptions and properly handle them and
  the details of the original log stream error that started
  the sequence are now output before the bit of code that had
  the error is executed.
  
  4. When a Data Mover on a Hub with multiple inbound connections
  is shutdown it may occasionally throw a concurrent
  modification exception during the shutdown process.
  This results in the shutdown hanging and the stop command
  being non-functional.  Logic has been added to trap the
  exception and immediately terminate the Data Mover.
  
  5. If an unexpected exception occurs during the shutdown
  process, it will be trapped and an immediate exit will be
  performed.
  
  6. If a Data Mover is told to run a configuration that has a
  SPLIT stage with an undefined channel, it hangs.  Logic has
  been modified to issue message DRLJ0237E and shutdown.
  
  7. If an unexpected exception occurs on a Data Mover stage
  thread it will be caught, reported and the Data Mover will then
  shutdown.
  
  8. Improved resolution of build identification message
  DRLJ0089I, adding APAR number and branch identifier.
  
  9. The JSON and CSV conversion logic will now ignore DATE, TIME,
  and TIMESTAMP fields outside of the primary key. The primary
  key is expected to contained one of the following:
  A TIMESTAMP column
  A DATE column
  Or both a DATE and a TIME column.
  
  10. The parsing problem has been corrected.
  
  11. Added a check to see if all routes have COMPLETED
  before termination.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI71415 UI71414 UI71416 UI71417

Modules/Macros

• DRLPJCO  DRLPJDM  DRLPJFC
  

Fix information

• Fixed component name

  TDS FOR Z/OS

• Fixed component ID

  569510100

Applicable component levels

• R901 PSY UI71416

     UP20/09/12 P F009

• R9A8 PSY UI71417

     UP20/09/12 P F009

• RC01 PSY UI71414

     UP20/09/12 P F009

• RCA8 PSY UI71415

     UP20/09/12 P F009

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.