Topic
33 replies Latest Post - ‏2012-09-04T21:09:11Z by SystemAdmin
SystemAdmin
SystemAdmin
849 Posts
ACCEPTED ANSWER

Pinned topic Postprocessor Issues

‏2012-04-09T08:51:11Z |
When we follow the steps of inventory collection and Finish it, the Postprocessor Job hangs sometimes at the "Identify file language" step. Please suggest a solution around this issue.

When it hangs, how do I stop the postprocessor? I was not able to stop the job too. What should be the next course of action when the postprocessing is stuck? Will restarting WAS is the only workaround for this issue? Please advice.
Updated on 2012-09-04T21:09:11Z at 2012-09-04T21:09:11Z by SystemAdmin
  • boyerpl
    boyerpl
    190 Posts
    ACCEPTED ANSWER

    Re: Postprocessor Issues

    ‏2012-04-09T17:49:43Z  in response to SystemAdmin
    I need to know what state the queue is really in to know what you should do next.

    1. Is this RAA installed on Windows or z/OS? What version/fixpack is it?
    2. Have you successfully scanned other files previously, or is this a new installation?
    3. If this is a new installation, were there any problems creating the database? (In particular, I'm wondering if there were failures with the BIND commands.)
    4. Have you made any updates to Common.cfg to increase classificationScanBatchCount so that more file are classified as a single batch?
    5. If you go to the Database -> Work queue page, is there an entry with the name AnalysisQueueProcessor? If so, what state is it in? Are there any exceptions listed?
    6. Can you post your RaaRestWar*.log files here? You can view/download them using http://hostname:port/dmh/LogView
    • SystemAdmin
      SystemAdmin
      849 Posts
      ACCEPTED ANSWER

      Re: Postprocessor Issues

      ‏2012-04-10T11:19:12Z  in response to boyerpl
      I have answered your questions.

      1. Is this RAA installed on Windows or z/OS? What version/fixpack is it?
      This was installed on Windows. The version is 6.0.0.6

      2. Have you successfully scanned other files previously, or is this a new installation?
      Yes. We loaded and been successfully scanning the inventory.

      3. If this is a new installation, were there any problems creating the database? (In particular, I'm wondering if there were failures with the BIND commands.)
      No. It was working fine and we got stuck suddenly.

      4. Have you made any updates to Common.cfg to increase classificationScanBatchCount so that more file are classified as a single batch?
      No.

      5. If you go to the Database -> Work queue page, is there an entry with the name AnalysisQueueProcessor? If so, what state is it in? Are there any exceptions listed?
      It remains in Running Status at the step "Identify file language".

      6. Can you post your RaaRestWar*.log files here? You can view/download them using http://hostname:port/dmh/LogView
      Attached the log file with this post.
      • boyerpl
        boyerpl
        190 Posts
        ACCEPTED ANSWER

        Re: Postprocessor Issues

        ‏2012-04-10T18:45:12Z  in response to SystemAdmin
        Thanks for providing the details and the log file.

        It looks like you're running into a deadlock issue that was reported in v6.0.0.6 and corrected in fixpack v6.0.0.7.

        
        SQLCODE=-911: Deadlock with Plans 
        
        for DMH0731/DMH6000 (33411)
        


        You'll want to upgrade to the latest fixpack level (v6.0.0.8) of RAA and then start the postprocessor again. Before you do that, you need to stop the processor.

        To stop the postprocessor:
        1. Press Ctrl+Shift+Esc to bring up the Windows Task Manager.
        2. On the Processes tab, you can click the first column heading to sort by name.
        3. Right-click on the dmh0731.exe entry and choose End Process.
        4. If the dmh6000.exe entry is still there, right-click on it and choose End Process as well.
        5. If you go to the Analysis queue page, does it still have a DMH5349W message at the top that says the postprocessor is still running? (It shouldn't.) It's ok if there is still work listed in the various steps. That will be completed the next time the postprocessor is started.
        6. If you go to the Work queue page, is the AnalysisQueueProcessor* work item now completed?

        If the AnalysisQueueProcessor* work item is no longer running, but you still have that DMH5349W message, you can reset an attribute in the database to get rid of it.

        To reset the postprocessor status in the database:
        1. Choose Start -> Run... and type: db2cmd.exe
        2. In the DB2 Command Window that opens, type:
        db2 connect to dmhdb user <userid> using <password>
        db2 update dmh.dmh_system_attr set sys_int_attr = 3 where sys_attr_id = 110
        db2 connect reset

        Once the postprocessor is no longer running, you can install fixpack v6.0.0.8 and then start the postprocessor.
        • SystemAdmin
          SystemAdmin
          849 Posts
          ACCEPTED ANSWER

          Re: Postprocessor Issues

          ‏2012-04-11T05:03:15Z  in response to boyerpl
          Thanks boyerpl. We will update to v6.0.0.8 and I will update the status here in this thread.
          Thanks again.
          • SystemAdmin
            SystemAdmin
            849 Posts
            ACCEPTED ANSWER

            Re: Postprocessor Issues

            ‏2012-04-18T09:55:02Z  in response to SystemAdmin
            We have now upgraded the RAA by installing the Fixpack 8. But, again we got the scanning issue today. The scanning process is in Running status for very long. I am attaching the RaaRestWar log. Please provide a solution/workaround to resolve this.

            Thank you.
            • boyerpl
              boyerpl
              190 Posts
              ACCEPTED ANSWER

              Re: Postprocessor Issues

              ‏2012-04-18T15:04:26Z  in response to SystemAdmin
              I'm sorry that didn't solve your problem. It sounded like the deadlock issue.

              When the scan runs for very long, if you look in the Windows Task Manager, do you see dmh6000.exe running?

              Can you post your C:\PROGRA~2\IBM\RATION~1\log\DMH6000.log file here?
            • SystemAdmin
              SystemAdmin
              849 Posts
              ACCEPTED ANSWER

              Re: Postprocessor Issues

              ‏2012-04-19T04:54:01Z  in response to SystemAdmin
              Are you using long names for the folders (space in-between the folder names) where you have stored the application assets?
              • SystemAdmin
                SystemAdmin
                849 Posts
                ACCEPTED ANSWER

                Re: Postprocessor Issues

                ‏2012-04-19T06:09:43Z  in response to SystemAdmin
                Attaching the dmh6000.log file.
                The folder name we used had 10 characters with periods in between, not any spaces. Ex:- RAA.TEST.COB

                Attachments

                • SystemAdmin
                  SystemAdmin
                  849 Posts
                  ACCEPTED ANSWER

                  Re: Postprocessor Issues

                  ‏2012-04-19T06:17:20Z  in response to SystemAdmin
                  Not sure about period, but I had a similar issue while scanning the assets. Resolved by removing the spaces in-between the folder name.
                • SystemAdmin
                  SystemAdmin
                  849 Posts
                  ACCEPTED ANSWER

                  Re: Postprocessor Issues

                  ‏2012-04-19T06:24:40Z  in response to SystemAdmin
                  Also, now that the Postprocessor itself is not started at all. Its always in Not Running mode now. Attaching the RaaRestWar.log as well. I saw one SQL error in this log file which is SQLCODE=-302, SQLSTATE=22001. Is this something to do with the DB2 for RAA? Do we need to do any DB2 update as well along with this Fixpack 8?
                  • jcdelmo
                    jcdelmo
                    343 Posts
                    ACCEPTED ANSWER

                    Re: Postprocessor Issues

                    ‏2012-04-19T16:40:34Z  in response to SystemAdmin
                    Regardling the SQLCODE=-302, the internal character limit for storing the memberFilter(s) of a scan request is 255. One string in the log file indicates there are 470 bytes in the concatenated list of member names that are in the scan request.

                    For now, I would suggest breaking the scan request into (at least) two requests splitting the member name list .

                    Longer term, a PMR could be opened to request our addressing this limitation.
                  • boyerpl
                    boyerpl
                    190 Posts
                    ACCEPTED ANSWER

                    Re: Postprocessor Issues

                    ‏2012-04-19T17:23:37Z  in response to SystemAdmin
                    Someone else is going to address the -302 error you saw. I'd like to focus on how the postprocessor appears to "hang" for a while.

                    I was looking at both the RaaRestWar*.log files you've posted. The first log file ends at 2012/04/18 05:26:11:282 and the new one picks up at 2012/04/18 13:44:39:765. Do you have the RaaRestWar*.log files for the period in between? I want to see the activity from after it completed the type 4 processing that started at 2012/04/18 04:36:50:010.

                    Did the postprocessor just complete on its own the first time? When it was running, did you happen to check the Analysis queue page, and see that it was progressing? If you refresh the page while the postprocessor is running, you should see the numbers for each step change as work moves through the queue.

                    How many programs have been scanned? If it's just that the analysis is taking a while, the slowness could be related to the time it takes to load results into the database. If the size of the tables changes substantially, then statistics need to be updated and BINDs need to be performed (to take advantage of the updated statistics). To perform both, you can open a command window and execute the following (customizing it if necessary):
                    
                    cd /d 
                    "C:\Program Files\IBM\Rational Asset Analyzer\bin" rexx dmhRunstats.rexx DMHDB DMH
                    
                    • Harshix
                      Harshix
                      40 Posts
                      ACCEPTED ANSWER

                      Re: Postprocessor Issues

                      ‏2012-04-20T16:23:42Z  in response to boyerpl
                      The issue with the postprocessor seems to be settled right now. We tried loading the components from a folder with a shorter name and it worked. The postprocessor is running without any manual intereference now.

                      Answering question about the state of postprocessor:
                      When the Postprocessor hung, it was completely idle. The was no change in the numbers for each step even when left for days. The postprocessor had to be manually stopped.

                      Unfortunately, I am unable to find the log of the events between 2012/04/18 05:26:11:282 and 2012/04/18 13:44:39:765 in any of the available versions of RaaRestWar*.log
                    • Harshix
                      Harshix
                      40 Posts
                      ACCEPTED ANSWER

                      Re: Postprocessor Issues

                      ‏2012-04-26T09:35:35Z  in response to boyerpl
                      The issues with Postprocessor surfaced again. As an attempt to load all the COBOL components into RAA, we tried loading components application by application. Few programs passed through smoothly, but as we tried to load about 100 components at a time, the postprocessor went into "Not Running" state. Even when tried to start the postprocessor manually, the processor won't start. Things in log file look good until the time-stamp 2012/04/26 02:27:59:26 but after 2012/04/26 02:27:59:260, they all went wrong.

                      The log file has been shared. Could this mean anything?
                      
                      Caused by: DB2 SQL Error: SQLCODE=-302, SQLSTATE=22001, SQLERRMC=null, DRIVER=3.63.75
                      
                      • Harshix
                        Harshix
                        40 Posts
                        ACCEPTED ANSWER

                        Re: Postprocessor Issues

                        ‏2012-04-26T09:38:32Z  in response to Harshix
                        Attaching Log file
                      • jcdelmo
                        jcdelmo
                        343 Posts
                        ACCEPTED ANSWER

                        Re: Postprocessor Issues

                        ‏2012-04-26T19:16:58Z  in response to Harshix
                        The issues is related to my prior post above. It has not (yet) been addressed in an RAA fixpack.

                        The limit is related to the number of member filters associated with a container and is cumulative for all same named containers!!!

                        So what I believe you need to do is delete all but 19 (or so) of the same container scan requests. Then, run post-processing. Then re-queue the additional member filters with wildcards or smaller batches of members for the given container(s).

                        Note: there currently is a hard limit (data base column size) of 255 bytes for the list of member filters for a given container queued at the same time.
                        • Harshix
                          Harshix
                          40 Posts
                          ACCEPTED ANSWER

                          Re: Postprocessor Issues

                          ‏2012-04-27T12:27:59Z  in response to jcdelmo
                          1) Please correct my understanding. So you mean to say that if a filter is being used during Collection of Inventory, the number of elements being given in list to filter should be ~20 (less than 255 bytes), right?

                          2) Is it ok to give a wildcard(*) to collect all the components in the inventory? We tried it and the processor failed.

                          3) We have more than 6000 components to be loaded from a inventory. What is the best approach to do that?
                          • jcdelmo
                            jcdelmo
                            343 Posts
                            ACCEPTED ANSWER

                            Re: Postprocessor Issues

                            ‏2012-04-27T15:33:16Z  in response to Harshix
                            1) Yes, until that limit is addressed in a future fixpack (RAA work item 35071 has been opened to track this limitation)

                            2a) Yes, the inventory wizard is designed to accept member filter(s) with both the '*' and '?' wildcard characters.
                            2b) You state the 'processor failed' - could you provide details of the failure, as wildcard support has been enabled for years without failures?

                            3) The best-practices approach is (for a new data base) to load ~100 assets than execute runstats/rebinds (to allow DB2 to optimize accesspaths, followed by loading the remainding assets using '*' wildcard for each container. The most efficient order to load assets (assuming Batch COBOL application) is COPYBOOKs, PROGRAMs, JCL PROCs followed by JCL JOBs.
                            • Harshix
                              Harshix
                              40 Posts
                              ACCEPTED ANSWER

                              Re: Postprocessor Issues

                              ‏2012-04-29T13:31:51Z  in response to jcdelmo
                              Thank you for your support.

                              RE: 2b) We asked the admin to load all the components with wildcard(*) and it went well. It was when we tried to load those components from a user account, we had that processer hanging problem. Right now, it is resolved as all the components are in place.
                            • Harshix
                              Harshix
                              40 Posts
                              ACCEPTED ANSWER

                              Re: Postprocessor Issues

                              ‏2012-05-04T16:18:39Z  in response to jcdelmo
                              Another issue with the postprocessor.

                              We tried to load the PSBs into RAA. The processor keeps running continuously with no progress.
                              We attempted to load all the PSBs using the wildcard(*). Please find the ZIP containing the log file and a document with screenshots within a gap of 4-5 hours. Observe that there is no change in the number of requests the postprocessor is processing.

                              I can see progress in the logfile every minute but the log seems to be in loop i.e. keeps repeting the following
                              
                              [2012/05/04 10:34:58:148] RaaRestWar INFO: Manually checking status on [1] scans. [2012/05/04 10:34:58:148] RaaRestWar INFO: Done manually checking status.
                              


                              As of this moment, the processor is still running. Is this normal? Or something is wrong?
                              • jcdelmo
                                jcdelmo
                                343 Posts
                                ACCEPTED ANSWER

                                Re: Postprocessor Issues

                                ‏2012-05-04T17:11:58Z  in response to Harshix
                                I assure you this is not normal, something is wrong as a container scan (at least the one queued request) should run in less than a minute.

                                There are a number of things we can check.

                                Can you go to Database->Work queue menu and show the work in-progess?

                                Than enter http://CIWAPPXD0061.silver.com:8088/scans in a web broswer and let me know the list of scans in-progress that show up?

                                Can you zip up and other log files that are in <installDir/logs and attach them?
                                • Harshix
                                  Harshix
                                  40 Posts
                                  ACCEPTED ANSWER

                                  Re: Postprocessor Issues

                                  ‏2012-05-06T10:13:17Z  in response to jcdelmo
                                  I have attached all the log files and screenshots of the analysis and work queue pages.

                                  I was unable to open the URL http://CIWAPPXD0061.silver.com:8088/scans. Please find the error in the screenshot. Kindly let me know if I can get the information you are looking for in some other way.

                                  The number of outstanding requests is changed to 2. This is because we tried to queue a program for analysis which din't go through.
                                  • jcdelmo
                                    jcdelmo
                                    343 Posts
                                    ACCEPTED ANSWER

                                    Re: Postprocessor Issues

                                    ‏2012-05-07T18:35:19Z  in response to Harshix
                                    The logs show a number of things: The RaaRestWar shows a SQLCODE=-551, which indicates no authority to update the DMH_SYSTEM_ATTR table, the Scan REST server is shutdown and the post-processor is still running. So...

                                    Can you, using the same ID that start WAS, from a DB2 command line see if you can do: SELECT * from $(SCHEMA).DMH_SYSTEM_ATTR ?

                                    If that fails, we have a DB2 authorization issue.

                                    Can you, from the <installDir>\scan directory, restart the Scan REST server with: dmhScan.bat
                                    After which, the 8088 port URL should work. I do not know why the Scan REST server was shutdown, the logs show it was requested to.

                                    Can you, stop and restart WAS (or the wsaa application in WAS)?

                                    After which, it would be helpful to know the value of SYS_INT_ATTR from: SELECT SYS_INT_ATTR FROM $(SCHEMA).DMH_SYSTEM_ATTR WHERE SYS_ATTR_ID = 110

                                    If the value is 3, restart the post-processor - if it is 1, set it to 3 and restart the post-processor.
                                    • Harshix
                                      Harshix
                                      40 Posts
                                      ACCEPTED ANSWER

                                      Re: Postprocessor Issues

                                      ‏2012-05-14T15:25:44Z  in response to jcdelmo
                                      As you asked, we performed the SELECT SYS_INT_ATTR FROM $(SCHEMA).DMH_SYSTEM_ATTR WHERE SYS_ATTR_ID = 110 and found the value of SYS_INT_ATTR to be 3.

                                      We have tried loading 1 or 2 PSB components at a time and it worked. But whenever we load more than 20 components, the processor gets stucks. We tried giving the wildcard(*) in filter for loading all PSBs at a time, which too din't help. Of latest, as we tried to load 4 components, the processor got stuck again.

                                      We restart the WAS after every failed attempt. Please let me know if we are missing something.

                                      PFA containing the screenshot and the log file.
                                    • Harshix
                                      Harshix
                                      40 Posts
                                      ACCEPTED ANSWER

                                      Re: Postprocessor Issues

                                      ‏2012-05-16T13:29:26Z  in response to jcdelmo
                                      Loading PSB is giving lot of issues. Whenever I try loading PSB (JCL too) even as low as 4 components at a time, the processor hangs.
                                      The log keeps updating with loops of the following
                                      
                                      [2012/05/04 10:34:58:148] RaaRestWar INFO: Manually checking status on [1] scans. [2012/05/04 10:34:58:148] RaaRestWar INFO: Done manually checking status.
                                      


                                      1)Is it because of the fact that all Copybook, DCLGEN, DBD and COBOL are already loaded?
                                      2)Are PSBs supposed to be loaded before COBOL components?

                                      In light of the prolonged issues, we are considering a full DB refresh and loading all components once again.
                                      Is it feasible? We have the following in our environment, numbers mentioned in the brackets.
                                      COBOL(6K), Copybook(3K), DCLGEN(1.5K), JCL(3K), PSB(3K), DBD(1K), EZT(for purpose of symbol scanning).

                                      3)Can you please suggest the right order of loading the above?
                                      • jcdelmo
                                        jcdelmo
                                        343 Posts
                                        ACCEPTED ANSWER

                                        Re: Postprocessor Issues

                                        ‏2012-05-17T16:48:41Z  in response to Harshix
                                        Your issue surprises me. Have not seen this before. Would you post the results from this query:

                                        
                                        select aq.analysis_type_cd, count(*) as rowCount from dmh.dmh_analysis_queue aq group by aq.analysis_type_cd order by aq.analysis_type_cd
                                        


                                        1) What is already loaded in the data base should have no impact on a current PSB source file load, so I do not think that is the issue.
                                        2) PSBs can be loaded in any order with the other assets. PSB resolution has its own queue request type and is processed after run unit building.
                                        3) You can clear the data base and start over (if you'd like to) by running the <installDir>\bin\dmhClearDB.rexx script -- I would not do that, since reloading takes time
                                        4) The load order does not really matter, but best practice order (to reduce rescans for missing pieces) is: CopyBook, JCL Proc, COBOL, JCL jobs, DBD, PSB
                                        • Harshix
                                          Harshix
                                          40 Posts
                                          ACCEPTED ANSWER

                                          Re: Postprocessor Issues

                                          ‏2012-06-06T15:27:41Z  in response to jcdelmo
                                          The query gave 0 results.

                                          Lately, I was able to load few components successfully. So I tried to load all the PROCs and JCL and that worked very well. But as I tried to load PSB the nex day, I started facing the same error again.

                                          I compared the successful events and the failed events in the log files and found that the issue is because some Signal-complete message is not being recieved, which suggests some kind of server unavailablity. I have attached the Log file, comparison files and the screenshots.

                                          1) How can i know if the server responsible for Signal-complete message is up and running?
                                          2) How can i ensure that the server is always on?
                                          • Harshix
                                            Harshix
                                            40 Posts
                                            ACCEPTED ANSWER

                                            Re: Postprocessor Issues

                                            ‏2012-06-06T15:47:31Z  in response to Harshix
                                            Attaching the Log file, comparison files and the screenshots.
                                          • jcdelmo
                                            jcdelmo
                                            343 Posts
                                            ACCEPTED ANSWER

                                            Re: Postprocessor Issues

                                            ‏2012-06-06T18:31:20Z  in response to Harshix
                                            The word document shows 3 containers are queued up for scanning, correct? How many files are in the three containers? Since you said you are trying to scan PSB are the files in the directories PSB source or PSB binaries?

                                            You can check if the server is running by opening a browser to the URL http://CIWAPPXD0061.silver.com:8088/ping --> which should return OK
                                            • Harshix
                                              Harshix
                                              40 Posts
                                              ACCEPTED ANSWER

                                              Re: Postprocessor Issues

                                              ‏2012-06-08T19:27:43Z  in response to jcdelmo
                                              I am sorry, only one container was loaded with 3 files in the filter. The 3 components were PSB source.

                                              I tried the url but it gave me a time out error. But when i asked my colegue in another location to try opening this url, he recieved OK.

                                              What server is reaponsible for the OK message?
                                              Why is this server unavailable for me?
                                              How can i make that server available for me?

                                              My location is India, my colegue's location is US and the RAA install location is another city in US.
                                              • jcdelmo
                                                jcdelmo
                                                343 Posts
                                                ACCEPTED ANSWER

                                                Re: Postprocessor Issues

                                                ‏2012-06-12T14:28:54Z  in response to Harshix
                                                The SCAN rest server is responsible. I do not know why you are not getting a response, if your colleague is, than I only have to assume it is a network issue (not an RAA issue). Is the port 8088 used on your machine for anything else? Can your colleague scan the 3 files correctly?
                                                • Harshix
                                                  Harshix
                                                  40 Posts
                                                  ACCEPTED ANSWER

                                                  Re: Postprocessor Issues

                                                  ‏2012-08-27T14:31:25Z  in response to jcdelmo
                                                  Hi,

                                                  I am unable to load PSB components into RAA like usual. But I used to get help from my collegue in another location who used to be successful in loading components.

                                                  Of lately, my collegue is unable to load components too. He is now facing the same problem as me. The postprocessor freezes at analyzing container step. The log keeps repeating with the following two steps

                                                  
                                                  [2012/05/04 10:34:58:148] RaaRestWar INFO: Manually checking status on [1] scans. [2012/05/04 10:34:58:148] RaaRestWar INFO: Done manually checking status.
                                                  


                                                  One of the earlier responses in the forum suggests this is a connectivity issue. I have checked with the Network Support team in this regard but they could not help. We have restarted WAS several times but in vain.

                                                  This is causing a serious delay in RAA activities as we are unable to load any new components.
                                                  Could you exactly point out what is the reason of this problem? What can be done to resolve this issue parmanently? How can I ensure the server in available all times (if due to server unavailability)?

                                                  Running the following ULR gives a time-out page for me. But my collegue still gets an "OK" inspite of being unable to load.
                                                  http://CIWAPPXD0061.silver.com:8088

                                                  Thank you.
  • SystemAdmin
    SystemAdmin
    849 Posts
    ACCEPTED ANSWER

    Re: Postprocessor Issues

    ‏2012-09-04T21:09:11Z  in response to SystemAdmin
    Moved to a new thread: -302 SQL code errors - Post Processor

    Edited by: Leshek_admin on Sep 4, 2012 5:04 PM
    Updated on 2012-09-04T21:09:11Z at 2012-09-04T21:09:11Z by SystemAdmin