DevOps is a set of principles and practices that focus on improving communication and collaboration between development and operations teams. These practices have become popular with technology experts who wish to develop robust technology solutions for which reliability can be ensured and that can be easily upgraded. The initial focus of DevOps is to help developers and operations professionals to communicate and work more effectively together, but other stakeholders such as the Quality Assurance, testing, and information security (InfoSec) teams are equally important. This article explains how to use DevOps best practices to make your information security robust and effective.
Why information security is a key part of DevOps
DevOps has been extremely successful at enabling developers and operations to work together more effectively. With DevOps, the operations team gets the information that they need to understand how to establish effective and reliable application build, package, and deployment procedures. The information security group has many of the same requirements as the operations group. In addition, InfoSec needs to get the information that they need to ensure that the entire system is secure and reliable. Just as DevOps helps development and operations to work effectively together, DevOps helps development and information security to be more effective as well. Continuous deployment has been a key practice within DevOps and focuses on automating the deployment pipeline through automated build, package, and deployment. Information security can benefit significantly from the deployment pipeline by providing a platform to assess and address security concerns early in the development lifecycle. Effective security should always start with an assessment of the risks involved.
Security risks that DevOps can help address
Risks need to be understood and addressed as part of a software or systems development effort. Security cannot be just added on to a system at the end of a development process. (See Resources for more on building secure software.) Systems need to be designed and developed with security in mind from the very beginning of the development lifecycle. DevOps provides the necessary structure to help address many of the security risks that are inherent in creating any complex technology system.
Security vulnerabilities can be the consequence of a variety of circumstances. For example, inadequate coding practices in C and C++ systems can result in buffer overflow conditions that give a malicious attacker a command prompt with the privileges of the program. Buffer overflow attacks have been used by attackers to gain control of a system and even effectively to gain root privileges.
Runtime issues can occur due to inadequate security controls such as authentication and authorization in the interfaces between components. One common source of issues is incorrect permissions applied during a deployment.
Another area of concern is ensuring that the correct code is deployed. Mistakes during a deployment can be exploited by a malicious attacker.
Configuration issues between interfaces can be exploited by a hacker trying to penetrate a system. Once a system is compromised, inadequate security controls can make it harder to identify unauthorized changes and perform forensics to ascertain what has been changed through error or malicious intent.
InfoSec rarely has the technical expertise and enough adequate test environments early in the development process to conduct effective security- related testing. Many organizations also cannot provision their servers from scratch and deploy the applications as needed, a limitation that means that those responding to a security breach often have a tough job trying to understand exactly what code has been compromised and how the threat can be eliminated.
These risks are all too common. They cause many system outages that have impacted major banking and trading systems and even the trading exchanges themselves. If we cannot simply add in security at the end of a project then how exactly do we build secure systems from the beginning?
Effective source code management to build secure systems
Quality management guru, W. Edwards Deming, wisely noted that quality has to be built in from the beginning. (See Resources for Deming's book.) DevOps depends upon effective source code management to create the automated build, package, and deployment scripts that comprise the deployment pipeline.
Effective source code management includes:
- Version control system
- The ability to reliably baseline the code
- Techniques to model the software architecture using streams and components
- Support for multiple variants in the code.
Having all of the source code in a version control system enables InfoSec to scan the code for vulnerabilities. Robust version control systems also enable the security analysts to understand how the system is constructed, by modeling the component structure as well as the interfaces between components, which are often the target of a malicious attack.
DevOps and the deployment pipeline help to efficiently create and provision test environments that can be used to evaluate and test for vulnerabilities in the interfaces between components. Providing robust test environments in a timely manner enhances security by providing a platform for automated testing that can identify security issues that need to be addressed. Once issues are found in the code, they can be addressed as defects or requests for change and tracked throughout their lifecycle to ensure that identified risks have been addressed.
Robust source code management solutions can help manage the application lifecycle by providing a set of tools and processes that help drive the entire development process from choosing the right requirements to managing defects found while the system is in production. Tracking requirements and defects are an essential aspect of task-based development.
DevOps best practices for robust and effective information security
The following sections describe recommendations from DevOps principles to help build in information security:
- Collaborate with development, operations, and information security teams
- Develop software using a task-based approach
- Automate the application build, package, and deployment
- Instrument the code with libraries
- Identify the weakest link
- Conduct penetration testing
- Programmatically build the infrastructure
- Use automated systems to configure and verify the operating system
- Provide a secure, trusted application base
- Rely on standards and frameworks
- Enable early detection of unauthorized changes
Collaborate with development, operations, and information security teams
The information security team rarely has enough technical expertise required to understand common security vulnerabilities that are inherent in complex systems. Just as DevOps improves communication between development and operations, DevOps also empowers the information security team by allowing them to fully understand the entire application and how it is built, packaged and deployed. This knowledge can help InfoSec maintain a focus that is relevant and effective. By understanding the systems infrastructure, InfoSec can also understand when security has been breached and precisely what needs to be done to address the security weaknesses in the event that the infrastructure itself has been compromised. In many cases, it is essential to rebuild the servers using automated procedures.
Develop software using a task-based approach
Task-based development involves creating work items such as tasks, defects, and requirements and then tracking these work items to the change sets created to develop the code. By tracking requirements to tasks and defects, security issues that could have been introduced as a result of failing to implement an essential requirement can be avoided.
Tracking the requirements to test cases is also an important aspect of task-based development, because it ensures that each requirement is verified and validated. This traceability is essential in preventing defects that could result in security vulnerabilities. Defining and tracking work items helps to manage the complexity of the development effort. Code can also be instrumented to help facilitate security testing.
IBM software that support task-based development:
- IBM Rational Team Concert makes it easier to define and track work items.
- IBM® Rational® Quality Manager, a web-based centralized test management environment, provides a collaborative solution for tracking and metrics reporting.
Automate the application build, package, and deployment
Creating automated procedures to build, package, and deploy the code supports agile iterative development in many ways. Automated build procedures are a prerequisite for continuous integration and continuous delivery. The build procedures should automatically embed immutable version ID into each configuration item (CI) that is used or created by the build procedure. This is also where cryptographic hashes should be created for use in verification during the application deployment described below. Sprint milestone releases can be used for testing and validating requirements that may not be completely understood at project inception. Most importantly, the technical details of how to build each component are implicitly documented and can be reviewed by interested stakeholders including information security. Creating variants in the code facilitates application testing including instrumenting the code to facilitate security related testing.
Instrument the code with libraries
Instrumenting the code involves running the code with libraries that make it easier to scan the runtime code for vulnerabilities. DevOps plays a key role by enabling the code to be built, packaged, and deployed to a test machine for the purpose of identifying security vulnerabilities. In keeping with the DevOps focus to address issues as early as possible, analysis of the security of an application needs to be conducted throughout the lifecycle of the application. Trying to fix a security issue late in the lifecycle is much more difficult than addressing it up front and early in the development effort. By being involved in the full application lifecycle, the security analyst can more effectively identify any issues that exist in the system.
IBM software that helps with application security:
- IBM® Security AppScan® enables you to do static and dynamic testing throughout the application lifecycle.
- IBM® Security Qradar® Vulnerability Manager implements intelligent vulnerability scanning for applications.
Identify the weakest link
As the old adage goes, "a chain is only as strong as its weakest link". By taking a comprehensive view of the system, the security analyst can identify issues that are specific to a particular component or interface between components. Interfaces to legacy systems are often necessary. They are also the weakest link in the chain. After vulnerability has been identified as a risk, the application can be tested to assess the potential security issue. Steps can then be taken to mitigate the risk and address the security vulnerability. These techniques are particularly useful in facilitating early penetration testing.
Conduct penetration testing
The deployment pipeline provides an effective framework for creating the required test environment to conduct penetration testing. With DevOps, penetration testing can occur throughout the entire software and systems lifecycle. By improving the communication between development and information security, the DevOps InfoSec team can design and execute more effective penetration testing within the short time frames that are typically allocated for such purposes. The deployment pipeline also enables information security to gain the technical information that is essential for effectively assessing security vulnerabilities.
Programmatically build the infrastructure
Security experts recognize that almost any system can be compromised and once the environment has been penetrated, malicious code can creep in. It is common for hackers to compromise inadequately secured systems and then use them to attack other machines that are more secure. These types of attacks may simply involve a denial of service attack, which can then be traced back to the machines that were used in the attack. In denial of service attacks, unsuspecting victims may find themselves accused of mounting an attack on another system and required to explain why their machines were used for malicious intent.
When a system is compromised, the best course of action is to completely clean and re-provision the servers. DevOps provides some excellent procedures to build the servers from the operating system to the application base through a fully automated procedure. The ability to programmatically build the infrastructure is known as infrastructure as code. This key practice within DevOps is commonly used to support cloud-based infrastructures. These techniques have also been called agile systems administration, a term that refers to the value that DevOps provides in an agile development environment.
IBM® Tivoli® Provisioning Manager enables a dynamic infrastructure by automating the management of physical servers, virtual servers, software, storage, and networks.
Use automated systems to configure and verify the operating system
Infrastructure as code can be used to implement secure systems by providing automated procedures to configure and verify that the operating system conforms to established industry security standards such as the well-respected Center for Internet Security (CIS) Security benchmark, which defines precisely how the operating system should be configured. (See Resources for a link to the Center for Internet Security.)
Both the provisioning of the operating system and the verification of the security configuration can be fully scripted using the guidance provided in the CIS specification. Scripts enable servers to be fully provisioned and supported through an automated procedure. To ensure a reliable and secure platform, start with servers that are completely secure and that are configured correctly. After your server infrastructure has been provisioned and is verifiable, it is time to use the same principles to deploy the application base in a way that is secure and verifiable. The automated build procedures should ensure that the code can be verified once it is deployed by embedding version IDs into each configuration item including code components, configuration, and properties files.
Cryptographic hashes should be created for use in verifying the application deployment.
Deploying the application itself is just as important to provide a completely secure trusted application base that is not only verifiable, but also enables the detection of security vulnerabilities, including unauthorized changes. Building code that contains embedded version IDs for deployment verification and using cryptography to identify unauthorized changes ensures that you can create a secure trusted application base.
IBM Tivoli Provisioning Manager for OS Deployment enables automated remote deployment of operating systems.
IBM® UrbanCode Deploy provides an automation deployment framework that reduces deployment errors and improves efficiency, correctness, and traceability.
Provide a secure, trusted application base
To secure the trusted application base, build the application code using procedures that securely identify the exact binary code and all other configuration items (including XML and properties configuration files) so that you can definitely prove that the correct code has been deployed and confirm that no unauthorized changes have been made.
These procedures usually make use of cryptographic hashes that are first created during the automated build and packaging of the application. Once the code has been deployed, the hashes can be recalculated to verify that all of the code has been correctly deployed.
These same procedures can be used to establish reliable baselines and to identify any unauthorized changes that could potentially occur as a result of human error or even malicious intent.
Working from a secure, trusted, application base ensures that the systems have been reliably and accurately deployed and makes it easier to identify and address any possible security breaches. When authorized changes are implemented, the baselines must be updated to record and track the changes. These techniques are described in industry standards and frameworks, including the well-respected ITIL v3 framework, which provides guidance on how to build and maintain reliable services.
IBM® Rational® Integrated Development Environments (IDEs) run on the developer's desktop, providing rich capabilities for developing, unit testing, debugging, analyzing, modernizing, and optimizing applications.
Rely on standards and frameworks
Standards and frameworks provide expert guidance on how to implement secure and reliable systems. There are many functions and processes described in these guidelines including the ability to monitor and track code baselines. The ITIL v3 framework describes Service Asset & Configuration Management (SACM), which captures baselines that can be stored in the configuration management system (CMS). These baselines can be monitored by the configuration management database (CMDB) with discrepancies reported to the CMS. The definitive media library (DML) contains the baseline, official releases that can be deployed as part of provisioning a server.
These techniques, including the practices of monitoring baselines using intrusion detection systems (IDS), are well-regarded as industry best practices. The DevOps approach provides the automation and framework for implementing these procedures. Even with the best preventive practices, intrusions can happen. Being able to detect and respond to incidents is essential.
Enable early detection of unauthorized changes
When unauthorized changes occur either because of human error or due to a malicious attack, they should be immediately detected and they should trigger an incident response. The DevOps approach provides the procedures so that unauthorized changes can be easily detected and proactively addressed with minimal impact to the users. They also provide the basis for provisioning servers and deploying systems immediately to restore the system to a reliable baseline. These techniques require collaboration and excellent communication among the key stakeholders including development, operations, Quality Assurance, testing and information security teams.
Several IBM Rational change management software products help monitor software change and configuration management.
DevOps is well known for its focus on helping developers and operations to work together more effectively together. Information security, along with Quality Assurance and testing teams, are also key stakeholders and need to be represented within the DevOps framework. DevOps helps to improve communication and helps to transfer essential technical knowledge from the developers who are writing the code, to the operations and security professionals who are responsible for ensuring that there is uninterrupted service. With the DevOps structure in place, InfoSec can make informed decisions that will help protect the system infrastructure and applications from any unexpected occurrences that might compromise the system, whether due to unintentional error or malicious attack. Keeping systems secure helps to prevent system glitches and also ensures uninterrupted service. The DevOps approach and the deployment pipeline enable your team to roll out features that both meet your business needs and provide the secure trusted application base.
- Learn the latest about the Rational Lifecycle Integration Adapters:
- Start by checking the product overview page.
- Read Using the Rational Adapter for HP ALM 60-day trial with the Rational solution for Collaborative Lifecycle Management on Jazz.net (February 2013).
- See the October 2012 announcement: IBM Rational Lifecycle Integration Adapters extend the Rational ALM solution with integrations to a number of third-party tools.
- Explore the Rational software area on developerWorks for technical resources, best practices, and information about Rational collaborative and integrated solutions for software and systems delivery.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Check the Rational Quality Manager and Rational Test Lab Manager pages on developerWorks for links to product documentation, articles, tutorials, courses, downloads, and other useful areas. Also, explore the Rational Quality Manager Information Center for technical details, check the Rational Quality Manager Roadmap for links to more resources, and browse the IBM Quality Management page to learn more about what else is available.
- Learn more about Rational Team Concert:
- Find Rational Team Concert articles and links to many other resources on IBM developerWorks, and check the product overview page, features and benefits, system requirements, and the user information center.
- Check the Rational Team Concert page on Jazz.net.
- Watch the Using Rational Team Concert in a globally distributed team webcast or a demonstration of the Dashboards and reports, or listen to the podcast about IBM Rational Team Concert and Jazz.
- Explore the Altoro Mutual demonstration-only site, which shows you the application vulnerability scanning capabilities of IBM Security AppScan Standard edition.
- Check the developerWorks AppScan page and the IBM Security AppScan web page to find the editions that interest you most.
- Learn more about Tivoli Provisioning Manager for OS Deployment
- Find out more about Rational Test Workbench.
- Visit the Urban Code Deploy pages to learn more.
- Learn more about Building Secure Software: How to Avoid Security Problems the Right Way (John Viega and Gary McGraw, Addison-Wesley, 2002).
- Learn about building quality in at the beginning in Out of the Crisis (W. Edwards Deming, MIT, 1986).
- Find key resources on cyber security on the Center for Internet Security site.
- Explore DevOps for configuration management in the book Configuration Management Best Practices: Practical Methods that Work in the Real World (Bob Aiello and Leslie Sachs, Addison-Wesley 2010).
- Learn more about agile development in the book Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise (Addison-Wesley, 2011).
- See real world examples in the book User Stories Applied For Agile Software Development (Mike Cohn, Addison-Wesley 2004)
- Check out the Agile Manifesto.
Get products and technologies
- Download the free trial version of Rational Quality Manager, which includes Rational Test Lab Manager.
- Download Rational Team Concert from Jazz.net and try it free on up to 10 developers for as long as you want (requires registration). If you'd prefer, you can try it in the sandbox instead, without installing it on your own system.
- Download the trial version of Tivoli Provisioning Manager for OS Deployment.
- Download Rational Test Workbench, which includes Rational Integration Tester, from Passport Advantage.
- Check the Rational software forums to ask questions and participate in discussions.
- Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
- Join the Rational community to share your Rational software expertise and get connected with your peers.
- Rate or review Rational software. It's quick and easy.