Introduction to AppScan Policies
Select the best AppScan policies for your web application security testing
IBM Security Appscan (AppScan) is an automated dynamic security testing tool. It runs various tests that probe for known vulnerabilities and weaknesses and formats the results into reports to help you understand the vulnerabilities in your applications. These reports also provide more information to assist you when you implement the necessary protections.
AppScan comes in several editions to address the web application security testing needs throughout the Software Development Life Cycle (SDLC). In this article, we refer to AppScan Standard and Enterprise Editions, typically used by developers, testers, and security teams.
AppScan contains many tests, almost 2000, with new tests being added with each update. Naturally, not all tests are necessary for all applications. To make things easier, AppScan groups tests into test policies. By selecting a policy, you customize the type of tests that are run against your application, allowing you to target the right tests for your application and environment.
This article explains in detail what tests and policies are, examines the "out-of-the-box" available policies, and discusses how to pick the policy that best fits your needs.
Understanding policies, tests, and variants
At the highest level, a test policy in AppScan is a collection of security tests that are used to evaluate the security of a web application. Different policies contain different sets of tests to achieve specific testing goals. A test probes the application and attempts to detect a specific type of vulnerability. Tests might look for general vulnerabilities, such as "Cross-Site Scripting" or they might look for more specific issues.
Tests are composed of one or more test variants. A test variant is a specific procedure that is intended to detect a specific vulnerability. The number of test variants included in a test range from one to over 100. The more general tests have more variants.
As an example, AppScan has a test that checks for "Cross-Site Scripting"
errors, which includes 105 different variants. One such variant would be
to modify a user input to a value like
>'"<script>alert(1)</script>. Another variant
might be to insert a value of
--><script>alert(1)</script> into a page
parameter. Both of these examples would be useful for detecting
"Cross-Site Scripting" in a web application given different scenarios.
When you configure a test policy, you can select the tests and test variants that are right for your application and organizational goals.
Understanding test categories
AppScan classifies tests according to whether they are invasive, their type, their severity, and their Web Application Security Consortium (WASC) classification.
A test can be categorized as invasive or non-invasive. Invasive tests probe for vulnerabilities such as buffer overflow, command execution, and denial of service. Invasive tests are usually bugs in the hosting middleware or application libraries rather than defects introduced by the application team. Because invasive tests can corrupt the database or manipulate the server's configuration, they are usually run in an isolated test environment. Running an invasive test can also cause server instability.
Tests can be one of these types:
- Application: These tests check for vulnerabilities that are usually associated with non-secure coding practices in the application.
- Infrastructure: These tests look at the configuration and known weaknesses of middleware (for example, web server or database). They test for both known product vulnerabilities and insecure configurations.
- Third-party web components: Similar to application tests, these tests look for non-secure coding practices, but target specific known issues that are found in libraries (for example, Dojo or MyFaces), and applications (for example, WordPress or Joomla).
Tests are ranked as high, medium, low, or informational based on their CVSS or user-defined severity score.
- High Severity tests look for vulnerabilities that are expected to have the greatest impact and easiest execution. Examples include cross-site scripting, SQL injection, or path traversal.
- Medium Severity tests look for vulnerabilities that are expected to have less impact or require a higher degree of difficulty during execution. Some examples include cross-site request forgery, HTTP response splitting, or brute force attacks.
- Low Severity tests look for vulnerabilities that enable the gathering of information about your site or leak information, such as path disclosures, directory listings, or credit card patterns.
- Informational severity tests report various conditions that might not be optimal, but do not rise to the level of a low severity finding. Some examples include default installation pages or email address patterns that are found in the HTML source.
Managing test policies
Test policies are managed differently between AppScan Standard Edition and AppScan Enterprise Edition and are described in the following sections.
Managing test policies with AppScan Standard Edition
Polices are accessible from the AppScan tool bar. Select Scan configuration from the tool bar, as shown in Figure 1.
To manage test policies, select Test > Test Policy from the Scan Configuration window, as shown in Figure 2.
Scan Configuration window
AppScan includes nine preconfigured policies that can be selected or edited as needed. To assist you during this process, AppScan Standard groups the policy settings according to the policy categories described in Managing test policies. You can also use the "Use CVSS" filter to view tests that have a user-specified severity level. The "Enabled/Disabled" grouping in the filter lets you view which tests are enabled for a particular test policy.
AppScan Standard edition lets you create advanced test policies that contain the exact set of tests you choose to run during your testing. These advanced test policies can be exported from AppScan Standard Edition and imported into other instances of AppScan Standard edition or AppScan Enterprise edition.
Managing test policies with AppScan Enterprise Edition
In AppScan Enterprise Edition, only users with administrative access can manage test policies. To manage test policies, open the Security Test Policies page in the Administrative console, as shown in Figure 3.
Security test policies window
From the Security Test Policies page, you can create a new simple policy, edit an existing policy, delete an existing policy, or import a new Advanced test policy. You cannot create an advanced test policy in AppScan Enterprise Edition, but you can create an advanced test policy in AppScan Standard edition, export it to a file, and import it into AppScan Enterprise edition.
Click the name of the policy to open the Policy Details page. The Policy Details page shows all tests that are defined within a particular policy. It also lists some configuration settings, test type, and invasive testing options. You can also adjust the severity of individual tests through the Issue Types page.
Preconfigured test policies
AppScan comes with nine predefined policies. Table 1 summarizes the types of tests that are included in each predefined policy.
Test policy categories
|Policy name||Includes application tests?||Includes third-party tests?||Includes infrastructure tests?||Includes invasive tests?||Notes|
|Complete||Yes||Yes||Yes||Yes||This policy includes all tests. It can take a long time to complete, and invasive tests can cause server instability|
|Default||Yes||Yes||Yes||No||Similar to the Complete test, but without invasive tests that could cause the application or web server to crash.|
|Application Only||Yes||No||No||No||A good policy to run at any time in the development cycle. This test checks for common web application vulnerabilities such as sql injection, unencrypted credentials, proper security configuration (for example, ssl, session, or cookies), or cross-site scripting.|
|Infrastructure Only||No||No||Yes||No||Helps evaluate proper configuration and security of the operating system, servers, and middleware. The test is targeted for system administrators and ideal to test production servers. It can yield false positives when you are testing more laxly configured development environments.|
|Third Party Only||No||Yes||No||No||You should use this test if you are deploying third-party applications or libraries. To improve performance, you can customize and only select tests for the third-party applications you use.|
|Developer Essentials||Partial||No||No||No||The smallest policy available (26 tests). It has an assortment of application test types with high success chances.|
|The Vital Few||Partial||Partial||Partial||No||A small policy (229 tests) with selected high success rate tests. This policy is good for initial triage to get a sense of how much attention was given to different security areas. Because it is not comprehensive, the policy does not guarantee complete security coverage.|
|Invasive||Partial||Partial||Partial||Yes||This policy includes all invasive tests. Most invasive tests fall in two categories: denial of service and buffer overflows. They exploit known vulnerabilities in applications or third-party middleware and can crash or affect the server stability.|
|Web Services||Partial||Partial||Partial||No||This policy includes tests that are related to common aspects of services implementation such as parameter validation and spoofing, session and authentication security, encryption, and configuration.|
Selecting a test policy
You should consider several important factors when you are selecting a test policy. You need to consider the stage of the software development life cycle when the testing is taking place; the role, responsibilities, and goals of the tester; the type of application being tested; and the maturity of the organization that is performing the test. Each of these factors is discussed in the following sections.
Considering the stage of development
It is generally accepted that the earlier in the Software Development Life Cycle (SDLC) you discover defects, the cheaper it is to correct them. With that in mind, many organizations are changing security testing from being a gate at the end of the SDLC to earlier stages.
In the development stages of the SDLC, it is most useful to focus on application code vulnerabilities. The specific policy choices can differ based on the type of development that is taking place. In a pure custom development scenario, good choices are Developer Essentials and the full Application Only test policies. If you are customizing third-party code and libraries, it can be very important to include testing of those components as well, and you might want to include specific, relevant third-party tests in your test policy. If your application includes web services, it is similarly important to test those separately using the web services test policy.
As an application moves into the integration stages of the SDLC, you must ensure that the application functions with the production configuration settings and determines the security of the various components as they interact with one another. To support these goals, the development stage test policy is often comprised of extended infrastructure tests to validate system and third-party component configurations or patch levels.
The preproduction stage of the SDLC is the best place for in-depth testing, and it is common to test with the more extensive test policies at this stage, including invasive tests. Preproduction environments should be configured to be identical to the production servers, and these configurations should be validated through testing. This is also often the stage where compliance testing occurs and a policy that meets organizational compliance standards is used.
Deployed applications should generally not be subject to this sort of testing as a matter of practice. Automated dynamic testing of deployed applications can potentially negatively impact the security (for example, availability or integrity) of the production environment.
Considering the responsibilities of the tester
The role of the individual who is performing the testing is also a factor in selecting a test policy. Security testing is often performed by security specialists. However, if appropriate levels of education related to web application vulnerabilities and tooling have been established in an organization, there are advantages to sharing the security testing responsibility with additional roles.
When developers perform security testing, they have the most insight into the internal code of the application. To leverage this insight, their testing is generally focused on application and code vulnerabilities. Good test policies for developers to use include Developers Essentials and Application Only in addition to the Web Services test policy if the application uses web services. Developers can also leverage static analysis tools like AppScan Source Edition.
When system administrators and web masters perform security testing, there is often a focus on testing the infrastructure and other third-party components. This leverages their knowledge of the configuration of these components.
When internal IS auditors and QA staff test, it is often in support of compliance programs. These compliance programs are often based on internal policies, external compliance requirements, contract terms, or standards that are derived from organizations like SANS or OWASP. Policies for this sort of testing are often customized to specifically meet the goals of these programs.
Considering the type of application
The type of application being tested is also a factor in determining which test policy is appropriate for testing an application.
A very common pattern in web development is to customize and combine multiple off-the-shelf components (commercial and open source). When this pattern is used, it is important to test these components using not only the Application Only tests, but also relevant third-party tests that identify specific known vulnerabilities in these components.
When web services are used in an application, they are often tested to some degree by the application tests. But, it is also important to test them specifically using the web services test policy to increase the coverage of the web service.
Considering the organization's maturity level
Many companies acquire AppScan in the early stages of developing a center of security testing competency. If the security competency team is not experienced in the realities of web application security testing, it is quite common for the initial test runs to be configured using an excessively broad test policy like Complete. This can lead to dissatisfaction with automated dynamic security testing because trying to run a Complete test can potentially take a long time, report many false positives, cause system instability, and create an encyclopedic report. Delivering a massive report that is filled with defects causes a loss of confidence in the testing tools and even in the security testing organization itself.
It is generally a better idea to start out with a smaller test policy for the initial efforts. Often a test policy like Developer's Essentials or The Vital Few is a good choice at this stage as these test policies include tests for important security vulnerabilities that generally have a lower number of false positive results when configured in a minimal manner. As the organization's web application security testing competency and tool experience increases, this initial test policy should be expanded and tailored to specifically support the organizational goals.
IBM Security AppScan is a powerful tool to help identify and correct security vulnerabilities in web applications. Understanding when and how to use the default policies and the components of each policy can help you maximize the effectiveness of AppScan.