Topic
  • 15 replies
  • Latest Post - ‏2012-04-20T17:43:58Z by SystemAdmin
E12590
E12590
23 Posts

Pinned topic Regarding RAA and distributed scanner - Automating scanning

‏2012-04-04T12:33:58Z |
We are running RAA on the host(zos), and are now in the process of
getting RAA running on the distributed platform as well.

We are at the moment running RAA v6.0.0.7, and are planning to go to
v.6.0.0.8 later in April 2012.

The questions I have is regarding automating the scanning on the distributed
platform:
1) What should the bat script or bat command look like to add source scanning requests
to RAA on the Distributed platform?
2) What should the bat script or bat command look like to request analysis of the distributed assets?
3) Which user is the bat scripts or bat commands using, with regards to DmhScan.cfg
LocalAuthentication and RemoteAuthentication?
And what if both of these are set to False?
Updated on 2012-04-20T17:43:58Z at 2012-04-20T17:43:58Z by SystemAdmin
  • jcdelmo
    jcdelmo
    347 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-09T17:21:23Z  
    What should the bat script or bat command look like to add source scanning requests to RAA on the Distributed platform?
    What should the bat script or bat command look like to request analysis of the distributed assets?

    I'm not sure which scenario you mean...
    • You have installed the remote-components on Windows connecting to RAA on z/OS.
    • You have installed a full-blown instance of RAA on Windows. z/OS is not involved.

    Since I don't know which it is, I'll describe both.

    If you have installed the remote-components on Windows connecting to RAA on z/OS:
    Running batch files is not required to load distributed assets. Distributed container scan requests can be queued up like any other container scan request using whatever mechanisms are available for z/OS containers up on z/OS. As long as the site and resource manager on the queue entry is set to the distributed site and NTFS, the Java Queue Processor will forward the container scan request to the remote scanner daemon and download and load the results.

    It's all transparent from the user's perspective, and fundamentally not different from z/OS containers. They really never need to touch the remote machine after it's been installed. It's just a dummy service listening for container scan requests from the z/OS machine. The user does everything from the z/OS UI or DMH0700-style batch inserts.

    If you have installed a full-blown instance of RAA on Windows. z/OS is not involved:
    There is not a distinct command for submitting container scan requests from a command line in Windows.

    The closest we have would be dmhrest.bat, where you would post DMH0700-style import files to /raarest/import
    dmhrest.bat POST /raarest/import -user me -pw mypw -i importFile.txt

    ...followed by starting the queue processor...
    dmhstartqp.bat -user me -pw mypw

    But again, you're not touching the scanner at all. The user is doing everything through the RAA REST interface. The scanner is just a dummy service waiting for scan requests from the queue processor.
    Which user is the bat scripts or bat commands using, with regards to DmhScan.cfg: LocalAuthentication and RemoteAuthentication? And what if both of these are set to False?

    The LocalAuthentication settings are no longer used. This setting was the first attempt at securing the scanner daemon's REST interface. It simply set a user/pw on the interface. We stopped using it because that user/pw would have to be stored in the database in order to access the scanner daemon.

    The RemoteAuthentication setting simply authenticates against the RAA REST interface (/raarest/admin/authenticateUser). So the user/pw to access the scanner daemon is the same as the user/pw to access the RAA REST API. It's a preferred solution since you don't need to store the user/pw in DmhScan.cfg or the database. The RAA REST engine can simply use it's own credentials to authenticate to the scanner daemon REST interface. The only downside to this approach is that you cannot connect to the same scanner daemon from two different instances of RAA (unless they're using the exact same credentials).

    If both are set to false, then there is no security enabled on the scanner daemon, and anyone with a browser can look at all the files on your file system. So it's important that security be enabled.
  • E12590
    E12590
    23 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-10T09:28:00Z  
    • jcdelmo
    • ‏2012-04-09T17:21:23Z
    What should the bat script or bat command look like to add source scanning requests to RAA on the Distributed platform?
    What should the bat script or bat command look like to request analysis of the distributed assets?

    I'm not sure which scenario you mean...
    • You have installed the remote-components on Windows connecting to RAA on z/OS.
    • You have installed a full-blown instance of RAA on Windows. z/OS is not involved.

    Since I don't know which it is, I'll describe both.

    If you have installed the remote-components on Windows connecting to RAA on z/OS:
    Running batch files is not required to load distributed assets. Distributed container scan requests can be queued up like any other container scan request using whatever mechanisms are available for z/OS containers up on z/OS. As long as the site and resource manager on the queue entry is set to the distributed site and NTFS, the Java Queue Processor will forward the container scan request to the remote scanner daemon and download and load the results.

    It's all transparent from the user's perspective, and fundamentally not different from z/OS containers. They really never need to touch the remote machine after it's been installed. It's just a dummy service listening for container scan requests from the z/OS machine. The user does everything from the z/OS UI or DMH0700-style batch inserts.

    If you have installed a full-blown instance of RAA on Windows. z/OS is not involved:
    There is not a distinct command for submitting container scan requests from a command line in Windows.

    The closest we have would be dmhrest.bat, where you would post DMH0700-style import files to /raarest/import
    dmhrest.bat POST /raarest/import -user me -pw mypw -i importFile.txt

    ...followed by starting the queue processor...
    dmhstartqp.bat -user me -pw mypw

    But again, you're not touching the scanner at all. The user is doing everything through the RAA REST interface. The scanner is just a dummy service waiting for scan requests from the queue processor.
    Which user is the bat scripts or bat commands using, with regards to DmhScan.cfg: LocalAuthentication and RemoteAuthentication? And what if both of these are set to False?

    The LocalAuthentication settings are no longer used. This setting was the first attempt at securing the scanner daemon's REST interface. It simply set a user/pw on the interface. We stopped using it because that user/pw would have to be stored in the database in order to access the scanner daemon.

    The RemoteAuthentication setting simply authenticates against the RAA REST interface (/raarest/admin/authenticateUser). So the user/pw to access the scanner daemon is the same as the user/pw to access the RAA REST API. It's a preferred solution since you don't need to store the user/pw in DmhScan.cfg or the database. The RAA REST engine can simply use it's own credentials to authenticate to the scanner daemon REST interface. The only downside to this approach is that you cannot connect to the same scanner daemon from two different instances of RAA (unless they're using the exact same credentials).

    If both are set to false, then there is no security enabled on the scanner daemon, and anyone with a browser can look at all the files on your file system. So it's important that security be enabled.
    We have installed the remote-components on Windows connecting to RAA on zOS.
  • E12590
    E12590
    23 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-10T10:47:50Z  
    • E12590
    • ‏2012-04-10T09:28:00Z
    We have installed the remote-components on Windows connecting to RAA on zOS.
    I haven't been able to find a description of a distributed scan request, I have found out that the type is DIR but what about the other parameters you specify when you request a scanning via the WEB interface?

    We have 3 containers (distributed), but they are still lying in the ANALYSIS queue and look like they aren't procssed even though we have run a analysis scan (from Q17 to Q14).
  • E12590
    E12590
    23 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-11T08:15:47Z  
    • jcdelmo
    • ‏2012-04-09T17:21:23Z
    What should the bat script or bat command look like to add source scanning requests to RAA on the Distributed platform?
    What should the bat script or bat command look like to request analysis of the distributed assets?

    I'm not sure which scenario you mean...
    • You have installed the remote-components on Windows connecting to RAA on z/OS.
    • You have installed a full-blown instance of RAA on Windows. z/OS is not involved.

    Since I don't know which it is, I'll describe both.

    If you have installed the remote-components on Windows connecting to RAA on z/OS:
    Running batch files is not required to load distributed assets. Distributed container scan requests can be queued up like any other container scan request using whatever mechanisms are available for z/OS containers up on z/OS. As long as the site and resource manager on the queue entry is set to the distributed site and NTFS, the Java Queue Processor will forward the container scan request to the remote scanner daemon and download and load the results.

    It's all transparent from the user's perspective, and fundamentally not different from z/OS containers. They really never need to touch the remote machine after it's been installed. It's just a dummy service listening for container scan requests from the z/OS machine. The user does everything from the z/OS UI or DMH0700-style batch inserts.

    If you have installed a full-blown instance of RAA on Windows. z/OS is not involved:
    There is not a distinct command for submitting container scan requests from a command line in Windows.

    The closest we have would be dmhrest.bat, where you would post DMH0700-style import files to /raarest/import
    dmhrest.bat POST /raarest/import -user me -pw mypw -i importFile.txt

    ...followed by starting the queue processor...
    dmhstartqp.bat -user me -pw mypw

    But again, you're not touching the scanner at all. The user is doing everything through the RAA REST interface. The scanner is just a dummy service waiting for scan requests from the queue processor.
    Which user is the bat scripts or bat commands using, with regards to DmhScan.cfg: LocalAuthentication and RemoteAuthentication? And what if both of these are set to False?

    The LocalAuthentication settings are no longer used. This setting was the first attempt at securing the scanner daemon's REST interface. It simply set a user/pw on the interface. We stopped using it because that user/pw would have to be stored in the database in order to access the scanner daemon.

    The RemoteAuthentication setting simply authenticates against the RAA REST interface (/raarest/admin/authenticateUser). So the user/pw to access the scanner daemon is the same as the user/pw to access the RAA REST API. It's a preferred solution since you don't need to store the user/pw in DmhScan.cfg or the database. The RAA REST engine can simply use it's own credentials to authenticate to the scanner daemon REST interface. The only downside to this approach is that you cannot connect to the same scanner daemon from two different instances of RAA (unless they're using the exact same credentials).

    If both are set to false, then there is no security enabled on the scanner daemon, and anyone with a browser can look at all the files on your file system. So it's important that security be enabled.
    Thank you for your answer.

    We have installed the remote-components on Windows connecting to RAA on zOS.

    In response to your answer I haven't been able to find a description of the input format, for requesting an analysis of a distributed asset scan request - when adding items to the zOS analysis queue (as a batch job on zOS).

    I have found out that the type is DIR, but what about the other parameters you specify when you request a scanning via the WEB interface?
    • How should the request input string look like on zOS for distributed assets?
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-11T14:19:34Z  
    • E12590
    • ‏2012-04-11T08:15:47Z
    Thank you for your answer.

    We have installed the remote-components on Windows connecting to RAA on zOS.

    In response to your answer I haven't been able to find a description of the input format, for requesting an analysis of a distributed asset scan request - when adding items to the zOS analysis queue (as a batch job on zOS).

    I have found out that the type is DIR, but what about the other parameters you specify when you request a scanning via the WEB interface?
    • How should the request input string look like on zOS for distributed assets?
    This is what I would suggest, which has been used successfully at some installations I have been involved

    1) For a given JAVA application, create some kind of script that when the application is promoted, it creates a ZIP file containing the byte code archive (EAR, WAR..) and a directory structure with the source used to build the part. Name the ZIP file with the application name, no spaces, lets say "myJavaAppl.zip". Same script should drop the file in what I call the distributed root directory (2) and create the scan request card (3)
    2) Create the distributed scan root in the scanning server. Make it as short as possible, no spaces, lets say D:\JVA_ASSETS. The ZIP file from (1) should drop in it.
    3) Scan request. The format is, which applies to z/OS and distributed assets is

    C rmType siteName containerName memberFilter [parentApp,]app N) N) N)

    For JAVA file schema in (1) it would be

    C NTFS siteName D:\JVA_ASSETS\myJavaAppl.zip * MYJAVAAPPL SUBDIR=Y UTSCAN=Y SCANZIPS=Y

    That should result in scanning all the files in container "D:\JVA_ASSETS\myJavaAppl.zip", opening ZIPs and recursing directories, with symbol scanning.

    Make sure you have good network bandwidth. Make sure you perform progressive RUNSTATS on z/OS a few times as you scan a few applications. A single JAVA application can insert many hundred thousands or even millions.
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-11T14:32:03Z  
    This is what I would suggest, which has been used successfully at some installations I have been involved

    1) For a given JAVA application, create some kind of script that when the application is promoted, it creates a ZIP file containing the byte code archive (EAR, WAR..) and a directory structure with the source used to build the part. Name the ZIP file with the application name, no spaces, lets say "myJavaAppl.zip". Same script should drop the file in what I call the distributed root directory (2) and create the scan request card (3)
    2) Create the distributed scan root in the scanning server. Make it as short as possible, no spaces, lets say D:\JVA_ASSETS. The ZIP file from (1) should drop in it.
    3) Scan request. The format is, which applies to z/OS and distributed assets is

    C rmType siteName containerName memberFilter [parentApp,]app N) N) N)

    For JAVA file schema in (1) it would be

    C NTFS siteName D:\JVA_ASSETS\myJavaAppl.zip * MYJAVAAPPL SUBDIR=Y UTSCAN=Y SCANZIPS=Y

    That should result in scanning all the files in container "D:\JVA_ASSETS\myJavaAppl.zip", opening ZIPs and recursing directories, with symbol scanning.

    Make sure you have good network bandwidth. Make sure you perform progressive RUNSTATS on z/OS a few times as you scan a few applications. A single JAVA application can insert many hundred thousands or even millions.
    Sorry, the format got mangled, I will try again

    "C rmType siteName containerName memberFilter [ parentApp] ,app N) N) N) "
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-11T14:38:07Z  
    Sorry, the format got mangled, I will try again

    "C rmType siteName containerName memberFilter [ parentApp] ,app N) N) N) "
    Goody !

    C rmType siteName containerName memberFilter appls subDirOpt symScan zipScan

    where appls, subDirOpt symScan and zipScan are optional and like

    parentApp,app (parentAppl optional, must exist already. Child application will be created if not exists
    SUBDIR=Y|N
    UTSCAN=Y|N
    SCANZIPS=Y|N
  • E12590
    E12590
    23 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-16T15:38:32Z  
    Goody !

    C rmType siteName containerName memberFilter appls subDirOpt symScan zipScan

    where appls, subDirOpt symScan and zipScan are optional and like

    parentApp,app (parentAppl optional, must exist already. Child application will be created if not exists
    SUBDIR=Y|N
    UTSCAN=Y|N
    SCANZIPS=Y|N
    Thank you for your answer.

    We have tried it in RAA v6.0.0.8 but it seems that it doesn't recognize then SUBDIR, UTSCAN and SCANZIPS parameters, but it doesn't give any error messages. We have tried with SUBDIR=Y and also just an Y but with the same result.
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-16T19:43:07Z  
    • E12590
    • ‏2012-04-16T15:38:32Z
    Thank you for your answer.

    We have tried it in RAA v6.0.0.8 but it seems that it doesn't recognize then SUBDIR, UTSCAN and SCANZIPS parameters, but it doesn't give any error messages. We have tried with SUBDIR=Y and also just an Y but with the same result.
    What are you using to submit the request and from where? The format I have provided is supposed to be consumed via the REST API. I am not totally familar with the parts that ship with the remote component for the z/Series version. Look for a dmhload.bat file. It is a REST interface for submission.
  • E12590
    E12590
    23 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-17T15:49:48Z  
    What are you using to submit the request and from where? The format I have provided is supposed to be consumed via the REST API. I am not totally familar with the parts that ship with the remote component for the z/Series version. Look for a dmhload.bat file. It is a REST interface for submission.
    I have tried to submit it from zOS using the DMH0700 program and analysing it with the DMH6000 program.
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-17T15:55:17Z  
    • E12590
    • ‏2012-04-17T15:49:48Z
    I have tried to submit it from zOS using the DMH0700 program and analysing it with the DMH6000 program.
    As far as I know 700 only supports z/OS assets. You have to use the REST API path.
  • rogern
    rogern
    5 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-17T16:55:00Z  
    As far as I know 700 only supports z/OS assets. You have to use the REST API path.
    I spoke to the former developer/owner of the distributed scanner support. He mentioned there is a dmhstartqp.sh script that could be executed in a USS process that would signal the servlet to start
    the queue processor. We have never tested such a scenario in-house so I'm checking with a couple of
    my colleagues to see if they have sample JCL we can model for running USS batch scripts.

    I'm not sure if RAA will officially support this solution, but I will at least see what I can
    dig up
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-19T16:00:50Z  
    • rogern
    • ‏2012-04-17T16:55:00Z
    I spoke to the former developer/owner of the distributed scanner support. He mentioned there is a dmhstartqp.sh script that could be executed in a USS process that would signal the servlet to start
    the queue processor. We have never tested such a scenario in-house so I'm checking with a couple of
    my colleagues to see if they have sample JCL we can model for running USS batch scripts.

    I'm not sure if RAA will officially support this solution, but I will at least see what I can
    dig up
    What are you trying to scan on distributed side?
    Are those development side artifacts, or the deployed artifacts?
  • E12590
    E12590
    23 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-20T09:13:32Z  
    What are you trying to scan on distributed side?
    Are those development side artifacts, or the deployed artifacts?
    It is .EAR files that are ready for deployment.
  • SystemAdmin
    SystemAdmin
    849 Posts

    Re: Regarding RAA and distributed scanner - Automating scanning

    ‏2012-04-20T17:43:58Z  
    • E12590
    • ‏2012-04-20T09:13:32Z
    It is .EAR files that are ready for deployment.
    Make sure that the requests for scanning have the EAR file name as part of the container with members '*', and not a base container with a member name of earFilename.EAR. For RAA an archive file is really a container.