I'm working with AppScan Enterprise. Currently, I'm dealing with an issue where the scan log is reporting "Scan task taking more time than expected". To source the problem, I've turned on verbose logging and traffic logging. I've been going through the logs for the better part of two days, but have been unable to optimize the scan.
In the Item Log, I've located messages that look like...
"A worker thread is taking more than expected to process a task" "WorkerThread Monitor"
" Thread ID: 1234" "WorkerThread Monitor" ""
Processed: 139 task(s)." "WorkerThread Monitor" ""
" CurTask is taking: 00:40:14.1250136" "WorkerThread Monitor" ""
"Task name: Security Entity: ParamName | SOME_PARAMETER_NAME |
By subtracting the duration from the timestamp, I'm able to trace that back to an area in the Item Log where I believe the task starts. That looks like...
information "02/04/2013 15:14:29.066" AgentHost.exe 1515 1515 2020 "Scan Engine" 0x0 "XRule 'probeXSS1' on 'Security Entity: ParamName | SOME_PARAMETER_NAME | SOME_URL ' - rule is running" "XsrExecutionEnvironment.RunRule" ""
What's odd, is although probeXSS1 may say that it's "running" at 15:14:29, I don't actually see it reflected in the traffic logs until 10+ minutes later. I'm assuming at this point... the message "Scan task taking more time than expected" is likely referring to a huge class of tests and not just one specific, long running, test.
Further, from the traffic log, it's clear that tests are still being executed.
Is there any better way to isolate the source(s) of this problem? At this point, I figured I would just write a script to analyze the time it takes for the application to respond to each request and look for the tests that are taking significant amounts of time to run. Do any utilities (or reports within AppScan Enterprise) already exist for this purpose? Something that could provide the following information would be very useful:
(1) Average request/response time.
(2) Longest request/response wait times.
(3) Security test execution time.
Any information would be very very helpful.
This topic has been locked.
Pinned topic Long scan times and tools for analyzing traffic logs?
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Re: Long scan times and tools for analyzing traffic logs?2013-02-06T04:48:52ZThis is the accepted answer. This is the accepted answer.For anyone else having trouble with this, the most effective approach I've found is setting the number of allowed threads/connections to 1 and searching the Item Log file for "Executing". This should provide an ordered list of rules that are executed. By looking through the timestamps, it should be fairly clear where the majority of time is being spent. If you start seeing areas where it's taking more than 5 seconds to go from one rule to the next (in my case, I was seeing delays of over 20 seconds), it would be worth investigating the traffic log for those rules and see what's going on.
If anyone else has a more effective approach, I would be interested in hearing it.
Somewhat related, if anyone knows if it's possible to disable logging the response body (when you logging traffic), I would be excited to know how. Often, when troubleshooting through the traffic log -- only the request header, request body, and response header are important (IMO). In my situation, this would reduce an 800MB log file by probably 90% and make it much more manageable (and reduce the potential of logging sensitive application information).
Re: Long scan times and tools for analyzing traffic logs?2013-02-09T01:24:52ZThis is the accepted answer. This is the accepted answer.That notice may not be a problem if the thread does complete.
Setting your threads to 1 will make it easier to see what is happening in the logs and may make the notice go away.
This might be due to your scan being too large. There are a number of knowledge articles on this topic.
Re: Long scan times and tools for analyzing traffic logs?2013-02-11T20:54:36ZThis is the accepted answer. This is the accepted answer.
- SystemAdmin 110000D4XK
> That notice may not be a problem if the thread does complete.
> Setting your threads to 1 will make it easier to see what is happening in the logs and may make the notice go away.
> This might be due to your scan being too large. There are a number of knowledge articles on this topic.
Again, this is a manually recorded scan that I am performing (it does not use auto-spider).
Re: Long scan times and tools for analyzing traffic logs?2013-02-11T21:20:41ZThis is the accepted answer. This is the accepted answer.
- SystemAdmin 110000D4XK
FWIW, I did go through the articles you listed in a separate thread and can confirm that my situation does not match any of the scenarios that were listed. The site is very (very) small and the only rules that are currently being tested are the 'Vital Few'.
warrenm1 270001F39C224 Posts
Re: Long scan times and tools for analyzing traffic logs?2013-02-13T19:51:03ZThis is the accepted answer. This is the accepted answer.
- SystemAdmin 110000D4XK
Other possible reasons for multiple requests in between tests is if there is something wrong in your in-session settings generating frequent logins, for that type of analysis you'd need to log a pmr with Appscan Support so the log/settings can be analyzed in detail.
Hope this helps,