Topic
7 replies Latest Post - ‏2013-01-16T17:37:24Z by jcdelmo
SystemAdmin
SystemAdmin
849 Posts
ACCEPTED ANSWER

Pinned topic Loop when starting postprocessor in batch

‏2013-01-15T13:15:39Z |
There seems to be a problem with the process initiated by startqp.bat.
GUI initiation does not cause any problems. Running 6.0.0.11. When started like
call %WSAA_HOME%\bin\dmhstartqp.bat -s 17 -q 17 -d 0 -w 86400
returns immediatly. But when started as
rem call %WSAA_HOME%\bin\dmhstartqp.bat -s 10 -q 4 -d 0 -w 86400
then RAA seems to get into a tight loop that, using FINER, overruns the three RestWar logs in very short time with:

2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?

It sounds like a run away "lets check progression" or looking for work. When running the tests, the queue may be empty or just with some import files with should not be processing. The queue never gets to rebuild run units.
Updated on 2013-01-16T17:37:24Z at 2013-01-16T17:37:24Z by jcdelmo
  • jcdelmo
    jcdelmo
    341 Posts
    ACCEPTED ANSWER

    Re: Loop when starting postprocessor in batch

    ‏2013-01-15T15:44:34Z  in response to SystemAdmin
    What requests are in the queue when you run dmhstartqp.bat ?
    • SystemAdmin
      SystemAdmin
      849 Posts
      ACCEPTED ANSWER

      Re: Loop when starting postprocessor in batch

      ‏2013-01-15T18:09:26Z  in response to jcdelmo
      I think most of the times it may have had a few import files and the rest empty, so there where no entries in the selected range.
      • jcdelmo
        jcdelmo
        341 Posts
        ACCEPTED ANSWER

        Re: Loop when starting postprocessor in batch

        ‏2013-01-15T21:16:47Z  in response to SystemAdmin
        Hmm... It looks like you have stumbled on an interesting logic error.

        The queue requests are run in the following order: 17-18-10-12-13-8-11-4-2-7-14-1-5-16-15-21-22

        The queue processing range you selected for the second execution was -s10 -q4 ... the subset 10-12-13-8-11-4

        Type 10 (container scans) normally are not be run if there are any type 18 (Load import file) requests queued. If there are any 18s when the 10 requests are tried (even if there are not any 10s) the queue order starts over. This is an attempt to get the 17s and 18s out of the queue first.

        Since your range does not include type 18s they are not run, and remain in the queue. The type 10s can not run until they are complete - and your start/quit parameters do not allow the 18s to run.

        Sorry for the technical description (number, not name of queue requests)

        Would you try your 2nd request with -s18 -q4 ?
        • SystemAdmin
          SystemAdmin
          849 Posts
          ACCEPTED ANSWER

          Re: Loop when starting postprocessor in batch

          ‏2013-01-15T21:55:23Z  in response to jcdelmo
          Yes, if I start from 18 it does not loop. Now that you pointed to the technicality, I can also recreate it with the GUI.

          Now, here is the problem. What is the technical reason to mandate that the import files are processed before the source? I think it is backwards because if a source artifact related to the import is scanned after the import is processed, it may strip the artifact of its complete status. This is the case of the assembler source we discussed weeks back, when NTFS remapping of library locations has been made inside the import file. I know we are stretching the envelope, but I think this is a valid situation to do so.

          What, negative impact would have to allow to select the 10 to 4 sequence and run later the 18-22 for complete resolution ?

          Why is not the process of imports located right after file analysis ?

          If this is not allowed, the GUI should be changed to prevent the loop.
          • SystemAdmin
            SystemAdmin
            849 Posts
            ACCEPTED ANSWER

            Re: Loop when starting postprocessor in batch

            ‏2013-01-15T22:35:31Z  in response to SystemAdmin
            Obviously, I could always manage to generate the requests for source scan and let the post processor take it, and then go for the imports...
            • SystemAdmin
              SystemAdmin
              849 Posts
              ACCEPTED ANSWER

              Re: Loop when starting postprocessor in batch

              ‏2013-01-16T17:20:26Z  in response to SystemAdmin
              John,

              knowing how to get around it, I will implement quick circumvention, so the logic may be left as is. It would probably make sense to prevent the GUI from choosing the unsupported ranges.

              Thank you for all your help
              • jcdelmo
                jcdelmo
                341 Posts
                ACCEPTED ANSWER

                Re: Loop when starting postprocessor in batch

                ‏2013-01-16T17:37:24Z  in response to SystemAdmin
                Pretty sure the UI will stay as is. I'll open a work item to deal with the issue (and the scenario questions you raised)