Topic
  • 7 replies
  • Latest Post - ‏2013-01-16T17:37:24Z by jcdelmo
SystemAdmin
SystemAdmin
849 Posts

Pinned topic Loop when starting postprocessor in batch

‏2013-01-15T13:15:39Z |
There seems to be a problem with the process initiated by startqp.bat.
GUI initiation does not cause any problems. Running 6.0.0.11. When started like
call %WSAA_HOME%\bin\dmhstartqp.bat -s 17 -q 17 -d 0 -w 86400
returns immediatly. But when started as
rem call %WSAA_HOME%\bin\dmhstartqp.bat -s 10 -q 4 -d 0 -w 86400
then RAA seems to get into a tight loop that, using FINER, overruns the three RestWar logs in very short time with:

2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select ANALYSIS_QUEUE_ID from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?
2013/01/15 07:00:22:200 RaaRestWar FINER: Running prepared statement: select count(*) from $(SCHEMA).DMH_ANALYSIS_QUEUE where ANALYSIS_TYPE_CD = ?

It sounds like a run away "lets check progression" or looking for work. When running the tests, the queue may be empty or just with some import files with should not be processing. The queue never gets to rebuild run units.
Updated on 2013-01-16T17:37:24Z at 2013-01-16T17:37:24Z by jcdelmo
  • jcdelmo
    jcdelmo
    347 Posts

    Re: Loop when starting postprocessor in batch

    ‏2013-01-15T15:44:34Z  
    What requests are in the queue when you run dmhstartqp.bat ?
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Loop when starting postprocessor in batch

    ‏2013-01-15T18:09:26Z  
    • jcdelmo
    • ‏2013-01-15T15:44:34Z
    What requests are in the queue when you run dmhstartqp.bat ?
    I think most of the times it may have had a few import files and the rest empty, so there where no entries in the selected range.
  • jcdelmo
    jcdelmo
    347 Posts

    Re: Loop when starting postprocessor in batch

    ‏2013-01-15T21:16:47Z  
    I think most of the times it may have had a few import files and the rest empty, so there where no entries in the selected range.
    Hmm... It looks like you have stumbled on an interesting logic error.

    The queue requests are run in the following order: 17-18-10-12-13-8-11-4-2-7-14-1-5-16-15-21-22

    The queue processing range you selected for the second execution was -s10 -q4 ... the subset 10-12-13-8-11-4

    Type 10 (container scans) normally are not be run if there are any type 18 (Load import file) requests queued. If there are any 18s when the 10 requests are tried (even if there are not any 10s) the queue order starts over. This is an attempt to get the 17s and 18s out of the queue first.

    Since your range does not include type 18s they are not run, and remain in the queue. The type 10s can not run until they are complete - and your start/quit parameters do not allow the 18s to run.

    Sorry for the technical description (number, not name of queue requests)

    Would you try your 2nd request with -s18 -q4 ?
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Loop when starting postprocessor in batch

    ‏2013-01-15T21:55:23Z  
    • jcdelmo
    • ‏2013-01-15T21:16:47Z
    Hmm... It looks like you have stumbled on an interesting logic error.

    The queue requests are run in the following order: 17-18-10-12-13-8-11-4-2-7-14-1-5-16-15-21-22

    The queue processing range you selected for the second execution was -s10 -q4 ... the subset 10-12-13-8-11-4

    Type 10 (container scans) normally are not be run if there are any type 18 (Load import file) requests queued. If there are any 18s when the 10 requests are tried (even if there are not any 10s) the queue order starts over. This is an attempt to get the 17s and 18s out of the queue first.

    Since your range does not include type 18s they are not run, and remain in the queue. The type 10s can not run until they are complete - and your start/quit parameters do not allow the 18s to run.

    Sorry for the technical description (number, not name of queue requests)

    Would you try your 2nd request with -s18 -q4 ?
    Yes, if I start from 18 it does not loop. Now that you pointed to the technicality, I can also recreate it with the GUI.

    Now, here is the problem. What is the technical reason to mandate that the import files are processed before the source? I think it is backwards because if a source artifact related to the import is scanned after the import is processed, it may strip the artifact of its complete status. This is the case of the assembler source we discussed weeks back, when NTFS remapping of library locations has been made inside the import file. I know we are stretching the envelope, but I think this is a valid situation to do so.

    What, negative impact would have to allow to select the 10 to 4 sequence and run later the 18-22 for complete resolution ?

    Why is not the process of imports located right after file analysis ?

    If this is not allowed, the GUI should be changed to prevent the loop.
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Loop when starting postprocessor in batch

    ‏2013-01-15T22:35:31Z  
    Yes, if I start from 18 it does not loop. Now that you pointed to the technicality, I can also recreate it with the GUI.

    Now, here is the problem. What is the technical reason to mandate that the import files are processed before the source? I think it is backwards because if a source artifact related to the import is scanned after the import is processed, it may strip the artifact of its complete status. This is the case of the assembler source we discussed weeks back, when NTFS remapping of library locations has been made inside the import file. I know we are stretching the envelope, but I think this is a valid situation to do so.

    What, negative impact would have to allow to select the 10 to 4 sequence and run later the 18-22 for complete resolution ?

    Why is not the process of imports located right after file analysis ?

    If this is not allowed, the GUI should be changed to prevent the loop.
    Obviously, I could always manage to generate the requests for source scan and let the post processor take it, and then go for the imports...
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Loop when starting postprocessor in batch

    ‏2013-01-16T17:20:26Z  
    Obviously, I could always manage to generate the requests for source scan and let the post processor take it, and then go for the imports...
    John,

    knowing how to get around it, I will implement quick circumvention, so the logic may be left as is. It would probably make sense to prevent the GUI from choosing the unsupported ranges.

    Thank you for all your help
  • jcdelmo
    jcdelmo
    347 Posts

    Re: Loop when starting postprocessor in batch

    ‏2013-01-16T17:37:24Z  
    John,

    knowing how to get around it, I will implement quick circumvention, so the logic may be left as is. It would probably make sense to prevent the GUI from choosing the unsupported ranges.

    Thank you for all your help
    Pretty sure the UI will stay as is. I'll open a work item to deal with the issue (and the scenario questions you raised)