Integrate SonarQube into a DevOps environment

Continuous integration and continuous delivery

Learn how to integrate a static code analysis tool using the IBM SmartCloud Continuous Delivery Solution. This article also shows how to integrate SonarQube into a DevOps environment to obtain the feedback from the static analysis tool. Code quality is improved and your project is managed better.

Takehiko Amano (amnt@jp.ibm.com), IT Specialist, IBM

author photoTakehiko Amano is a solution architect at the IBM Software Development Laboratory in Tokyo, Japan. He works as a member of Unleash the Labs initiative. One of his interests is global collaborative software development tools and processes.



27 August 2013

Also available in Chinese

Automation is a key practice for software development improvement. One of the trends in software development is DevOps. There are various definitions of DevOps. However, it is likely that "automation(s)" is part of the definition. While software development is automated, humans also contribute to the process. Code written by one developer differs from code written by another team member. Style differs too. Some developers omit comments in the code. Continuous Integration (CI), Continuous Delivery (CD), Continuous Testing are all words associated with DevOps. It reduces the risk of various coding errors because the code is analyzed systematically. Two best practices are a static code check and to test the code for vulnerabilities with an application security test

This article explains how to deploy a static code check process in the Continuous Delivery process. IBM SmartCloud Continuous Delivery is used as an example to check status source code. However, the method used in this article can be used in various processes, such as Continuous Integration (e.g. IBM® Rational Build Forge®) or the Continuous Deployment (uDeploy) solution from IBM.

IBM SmartCloud Continuous Delivery solution and SonarQube

IBM SmartCloud Continuous Delivery includes most of the key practices of DevOps, and provides additional capabilities, including the workload deployer component which defines deployment patterns and automatic deployment against cloud systems (IBM® PureSystems™, IBM® Workload Deployer, IBM SmartCloud® Provisioning, etc). See the resources section at the bottom of the article for more information.

SonarQube is an open source project for source code static checking. It supports twenty languages. Reports are run in real-time. SonarQube refers to this as Continuous Inspection.

Definition of terminology used in the article.

Because there is no standard definition of "Continuous <process>", it is worthwhile to define the terminology used in this article.

Note: The definitions used in this article vary from publication to publication

Continuous Delivery
(CD) A staged automated process. It starts with the software build and moves through these phases of software development: packaging, automatic deployment to test system, pre-production system for quality assurance, and potentially deployment into production system. Automated system testing is frequently run against the deployed system. 
Continuous Integration (CI)
An automated process of software build and testing typically used for development testing. Continuous Integration is done on a daily basis (or whenever developers bring new source code into the source code repository). The process is similar to Continuous Delivery except that the main focus is software build and unit / feature testing, rather than system testing. 

Some companies may run continuous delivery on daily bases. Continuous integration is included in the continuous delivery process when continuous delivery is run daily.

The delivery pipeline consists of the following steps, listed below, and as shown in Figure 1.

Build
The source code is loaded from a repository. Then, the build process is run. In this step, you can run the unit testing tool and code coverage tool. You can also run code analysis. The delivery pipeline stops if there are errors. 
Publish
The build packages are stored in the library server. The library server also stores relevant information such as the quality status of the build and any associated documentations. 
Deploy
Applications are deployed into physical and/or virtual hosts. If the virtual hosts are not available, the cloud interface will provision virtual machines. 
Verification
Automated tests: integration test (functional test), performance test, security test, are run. 
Production
If you choose to deploy into a production environment, then the build is sent to a production system to obtain stakeholder feedback. In a compliance-oriented software development project the production step may not occur. 

Note: This article focuses on code analysis. The sample starts with the build step and finishes with the verification step.

Figure 1. Typical delivery pipeline process
delivery pipeline steps

Solution architecture

Figure 2 shows the solution architecture. IBM SmartCloud Continuous Delivery solution is the basis of the solution architecture, as shown in Figure 2.

Figure 2. Solution architecture based on SmartCloud Continuous Delivery
Solution architecture

In this example, Ruby Make (Rake) is used as an implementation of the delivery pipeline. Initiated either by an engineer's demand or an event such as scheduler, Jazz Build Engine runs the delivery pipeline. If requested from the client, it also runs the Rake utility. In this article, the delivery pipeline process is called manually from the IBM® Rational Team Concert™ client.

Implement the delivery pipeline onto the solution architecture by following these steps:

  1. Build. The build step is run on the Jazz Build Engine component. The delivery process loads source code from the repository in Rational Team Concert. SonarQube performs static checking, and then finally builds the software. At this stage, results of the static check are stored in the SonarQube server.
  2. Publish. The Jazz Build Engine publishes build artifacts and infrastructure automation code into the Library Server.
  3. Deploy. Jazz Build Engine requests the cloud host to provision any virtual machines needed for testing. The cloud host instructs virtual machines on where to load build artifacts and related files such as setup code (infrastructure automation code). Then, each virtual machine loads software and infrastructure automation code from the library server. Configuring themselves is the final part of the deploy step.
  4. Test. Whether or not the system test passes, the Jazz Build Server runs the system test against provisioned virtual machines.

In this article, the delivery pipeline process (rakefile) is not explored. See the resources section for information about the Ruby Rake utility.

Listing 1. The rakefile static checking test

  desc "Static check the application"
  task :static_check_application => :setup do
    
    log.info "Static checking the source codes ..."

	activity = build_system.start_activity(build, "Static Checking source codes ...")

    build_script_dir = File.join(File.dirname(__FILE__), "..", "JKEBuildScripts")
    current_dir=File.dirname(__FILE__)
    
    cd "#{current_dir}/.."
	sonar_runner_cmd="/opt/sonar-runner-2.2/bin/sonar-runner"
	sh "#{sonar_runner_cmd} -Dproject.settings=#{build_script_dir}/sonar-project.properties"

	build_system.complete_activity(activity)

  end

Rake uses the notion of a task to run a process. The :static_check_application is created in the rakefile. The log.info, and build_system.start_activity Ruby functions are called. This prompts the build system (Jazz Build Engine, in this article) to mark the start of static checking. The main program is:

sh "#{sonar_runner_cmd} -Dproject.settings=#{build_script_dir}/sonar-project.properties"

The command sonar-runner reads a project setting by the option –Dproject.settings. This file specifies the project name and the location of the source code. Listing 2 is a sample file to check the source code.

Listing 2. Sonar-project.properties

# required metadata
sonar.projectKey=my:project
sonar.projectName=JKE project
sonar.projectVersion=1.0

# optional description
sonar.projectDescription=JKEBanking

# path to source directories (required)
sonar.sources=JKEJavaUI/src,JKEBusinessLogic/src,JKEDBAccess/src,JKEServer/src

# The value of the property must be the key of the language.
sonar.language=java

# Additional parameters
sonar.my.property=value

Run the delivery pipeline

Ruby Rakefile runs by requesting the delivery process from Rational Team Concert web client. Figure 3 shows how to start the delivery pipeline. Although the Request Build button shows build, it starts the delivery pipeline run by the Ruby Make process on the Jazz Build Server. After the delivery process starts, the progress and estimated completion time are displayed. This example took 8 minutes from build though system testing. The example in this article is JKEBanking. The project is an artificial banking application created to pass all system tests.

Figure 3. Start the delivery process
Kick off delivery process

Figure 4 shows the delivery pipeline and the delivery process results. SonarQube does the static checking during the delivery process. The validate the deployed application activity runs the system testing.

Figure 4. Delivery pipeline activities
delivery pipeline of sample software delivery.

Figure 5 shows the test results. Zero failures, and zero errors. All tests are successful.

Figure 5. Delivery pipeline system test results
system test result of the delivery pipeline.

Analyze the results

The SonarQube dashboard shown in Figure 6 shows a sample result of the static check. There are 270 violations in the code. two violations are critical!

Figure 6. SonarQube dashboard for the JKEBanking application
Sonar dashboard of analysis

Figure 7 and 8 show the two critical messages. SonarQube lists the criticality and highlights the source code in red. Below the highlighted source code is an explanation of the codes error.

Figure 7. A sample critical error in JKEBanking.java code
A critical message in JKEBanking.java code
Figure 8. A critical message in GenerateData.java code
critical message highlighted in red

Figure 7 shows the message "Avoid catching throwable."..Throwable means it catches all exceptions in this statement. In Figure 8, generateData.java code, the if statement has no statement to execute inside the block. Both are treated as "Critical" in the static code analysis.

Note: Even though this is a sample application to show SonarQube integration, this may happen in a real world situation.

Some applications may be just running by accident. It is important to remind you that system testing does not always go through possible scenarios or use cases. This is one of the reasons for defect escapes after the release. Continuous Inspection reduces the risk of these defect escapes.


Conclusion

Even though the system seems to run perfectly from a system testing perspective, the code quality may not be good enough. SonarQube assists code quality improvement. By including static checking process in the Continuous Integration and Continuous Delivery (DevOps) environments, you can improve software code quality continuously and reduce the risk of failure, after the release of valuable software, significantly.


Acknowledgement

The author would like to express my gratitude to Wei Li who is leading DevOps solution development, and my manager Paul Weiss for encouraging me to publish article.

Resources

Learn

Get products and technologies

  • Download Rational Team Concert from Jazz.net and try it free on up to 10 developers for as long as you want (requires registration). If you'd prefer, you can try it in the sandbox instead, without installing it on your own system.
  • Download a free trial version of Rational software.
  • Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, DevOps, Cloud computing
ArticleID=941831
ArticleTitle=Integrate SonarQube into a DevOps environment
publish-date=08272013