DevOps best practices: Part 2. Ensure robust and effective information security with DevOps

DevOps principles, well known for helping developers and operations to work more effectively together, can be applied to information security, as well. Learn how DevOps can improve information security by applying industry-recognized best practices. Use the DevOps approach to keep systems secure, to prevent system glitches, and to ensure uninterrupted service.

Bob Aiello (bob.aiello@ieee.org), Consultant and editor-in-chief, CM Best Practices Consulting (Division of Yellow Spider, Inc)

author photoBob Aiello is a consultant, a technical editor for CM Crossroads, and the author of Configuration Management Best Practices: Practical Methods that Work in the Real World, Addison-Wesley Professional. Mr. Aiello has more than twenty-five years experience as a technical manager in several top NYC financial services firms where he had company-wide responsibility for CM, often providing hands-on technical support for enterprise source code management tools, SOX/Cobit compliance, build engineering, continuous integration, and automated application deployment. Bob has served as the vice chair of the IEEE 828 Standards working group (CM Planning) and is a member of the IEEE Software and Systems Engineering Standards Committee (S2ESC) management board. Mr. Aiello holds a Masters in industrial psychology from NYU and a B.S. in computer science and math from Hofstra University. You may email Mr. Aiello or link with him.



Leslie Sachs (LeslieASachs@gmail.com), COO, Yellow Spider, Inc.

photo of Leslie SachsLeslie Sachs is a New York state certified school psychologist and the COO of Yellow Spider, Inc. Leslie is the coauthor of Configuration Management Best Practices: Practical Methods that Work in the Real World, Addison-Wesley Professional. Ms. Sachs has more than twenty years of experience in the psychology field and has worked in a variety of clinical and business settings where she has provided many effective interventions designed to improve the social and educational functioning of both individuals and groups. Ms. Sachs has an M.S. in School Psychology from Pace University and interned in Bellevue's Psychiatric Center in New York city. A firm believer in the uniqueness of every individual, she has recently done advanced training with Mel Levine's "All Kinds of Minds" Institute. She is reachable via email and LinkedIn.



10 December 2013

Also available in Chinese Portuguese

Introduction

DevOps is a set of principles and practices that focus on improving communication and collaboration between development and operations teams. These practices have become popular with technology experts who wish to develop robust technology solutions for which reliability can be ensured and that can be easily upgraded. The initial focus of DevOps is to help developers and operations professionals to communicate and work more effectively together, but other stakeholders such as the Quality Assurance, testing, and information security (InfoSec) teams are equally important. This article explains how to use DevOps best practices to make your information security robust and effective.


Why information security is a key part of DevOps

DevOps has been extremely successful at enabling developers and operations to work together more effectively. With DevOps, the operations team gets the information that they need to understand how to establish effective and reliable application build, package, and deployment procedures. The information security group has many of the same requirements as the operations group. In addition, InfoSec needs to get the information that they need to ensure that the entire system is secure and reliable. Just as DevOps helps development and operations to work effectively together, DevOps helps development and information security to be more effective as well. Continuous deployment has been a key practice within DevOps and focuses on automating the deployment pipeline through automated build, package, and deployment. Information security can benefit significantly from the deployment pipeline by providing a platform to assess and address security concerns early in the development lifecycle. Effective security should always start with an assessment of the risks involved.


Security risks that DevOps can help address

Risks need to be understood and addressed as part of a software or systems development effort. Security cannot be just added on to a system at the end of a development process. (See Resources for more on building secure software.) Systems need to be designed and developed with security in mind from the very beginning of the development lifecycle. DevOps provides the necessary structure to help address many of the security risks that are inherent in creating any complex technology system.

Security vulnerabilities can be the consequence of a variety of circumstances. For example, inadequate coding practices in C and C++ systems can result in buffer overflow conditions that give a malicious attacker a command prompt with the privileges of the program. Buffer overflow attacks have been used by attackers to gain control of a system and even effectively to gain root privileges.

Runtime issues can occur due to inadequate security controls such as authentication and authorization in the interfaces between components. One common source of issues is incorrect permissions applied during a deployment.

Another area of concern is ensuring that the correct code is deployed. Mistakes during a deployment can be exploited by a malicious attacker.

Configuration issues between interfaces can be exploited by a hacker trying to penetrate a system. Once a system is compromised, inadequate security controls can make it harder to identify unauthorized changes and perform forensics to ascertain what has been changed through error or malicious intent.

InfoSec rarely has the technical expertise and enough adequate test environments early in the development process to conduct effective security- related testing. Many organizations also cannot provision their servers from scratch and deploy the applications as needed, a limitation that means that those responding to a security breach often have a tough job trying to understand exactly what code has been compromised and how the threat can be eliminated.

These risks are all too common. They cause many system outages that have impacted major banking and trading systems and even the trading exchanges themselves. If we cannot simply add in security at the end of a project then how exactly do we build secure systems from the beginning?


Effective source code management to build secure systems

Quality management guru, W. Edwards Deming, wisely noted that quality has to be built in from the beginning. (See Resources for Deming's book.) DevOps depends upon effective source code management to create the automated build, package, and deployment scripts that comprise the deployment pipeline.

Effective source code management includes:

  • Version control system
  • The ability to reliably baseline the code
  • Techniques to model the software architecture using streams and components
  • Support for multiple variants in the code.

Having all of the source code in a version control system enables InfoSec to scan the code for vulnerabilities. Robust version control systems also enable the security analysts to understand how the system is constructed, by modeling the component structure as well as the interfaces between components, which are often the target of a malicious attack.

DevOps and the deployment pipeline help to efficiently create and provision test environments that can be used to evaluate and test for vulnerabilities in the interfaces between components. Providing robust test environments in a timely manner enhances security by providing a platform for automated testing that can identify security issues that need to be addressed. Once issues are found in the code, they can be addressed as defects or requests for change and tracked throughout their lifecycle to ensure that identified risks have been addressed.

Robust source code management solutions can help manage the application lifecycle by providing a set of tools and processes that help drive the entire development process from choosing the right requirements to managing defects found while the system is in production. Tracking requirements and defects are an essential aspect of task-based development.


DevOps best practices for robust and effective information security

Products and solutions for DevOps, A to Z

IBM products and solutions support the DevOps approach and accelerate adoption. They are built on open standards and designed to work in concert with each other, as well as with open source solutions. You can find handy lists here:

The following sections describe recommendations from DevOps principles to help build in information security:

Collaborate with development, operations, and information security teams

Consider using IBM® Rational Team Concert™ to enable you to collaborate with software development professionals across many disciplines (development, operations, test, and so on).

The information security team rarely has enough technical expertise required to understand common security vulnerabilities that are inherent in complex systems. Just as DevOps improves communication between development and operations, DevOps also empowers the information security team by allowing them to fully understand the entire application and how it is built, packaged and deployed. This knowledge can help InfoSec maintain a focus that is relevant and effective. By understanding the systems infrastructure, InfoSec can also understand when security has been breached and precisely what needs to be done to address the security weaknesses in the event that the infrastructure itself has been compromised. In many cases, it is essential to rebuild the servers using automated procedures.

Develop software using a task-based approach

Task-based development involves creating work items such as tasks, defects, and requirements and then tracking these work items to the change sets created to develop the code. By tracking requirements to tasks and defects, security issues that could have been introduced as a result of failing to implement an essential requirement can be avoided.

Tracking the requirements to test cases is also an important aspect of task-based development, because it ensures that each requirement is verified and validated. This traceability is essential in preventing defects that could result in security vulnerabilities. Defining and tracking work items helps to manage the complexity of the development effort. Code can also be instrumented to help facilitate security testing.

IBM software that support task-based development:

Automate the application build, package, and deployment

Creating automated procedures to build, package, and deploy the code supports agile iterative development in many ways. Automated build procedures are a prerequisite for continuous integration and continuous delivery. The build procedures should automatically embed immutable version ID into each configuration item (CI) that is used or created by the build procedure. This is also where cryptographic hashes should be created for use in verification during the application deployment described below. Sprint milestone releases can be used for testing and validating requirements that may not be completely understood at project inception. Most importantly, the technical details of how to build each component are implicitly documented and can be reviewed by interested stakeholders including information security. Creating variants in the code facilitates application testing including instrumenting the code to facilitate security related testing.

Instrument the code with libraries

Instrumenting the code involves running the code with libraries that make it easier to scan the runtime code for vulnerabilities. DevOps plays a key role by enabling the code to be built, packaged, and deployed to a test machine for the purpose of identifying security vulnerabilities. In keeping with the DevOps focus to address issues as early as possible, analysis of the security of an application needs to be conducted throughout the lifecycle of the application. Trying to fix a security issue late in the lifecycle is much more difficult than addressing it up front and early in the development effort. By being involved in the full application lifecycle, the security analyst can more effectively identify any issues that exist in the system.

IBM software that helps with application security:

Identify the weakest link

Check out IBM® Rational® Test Workbench, a comprehensive test automation solution that supports both legacy and modern architectures.

As the old adage goes, "a chain is only as strong as its weakest link". By taking a comprehensive view of the system, the security analyst can identify issues that are specific to a particular component or interface between components. Interfaces to legacy systems are often necessary. They are also the weakest link in the chain. After vulnerability has been identified as a risk, the application can be tested to assess the potential security issue. Steps can then be taken to mitigate the risk and address the security vulnerability. These techniques are particularly useful in facilitating early penetration testing.

Conduct penetration testing

You can use Security AppScan to help with vulnerability and penetration testing.

The deployment pipeline provides an effective framework for creating the required test environment to conduct penetration testing. With DevOps, penetration testing can occur throughout the entire software and systems lifecycle. By improving the communication between development and information security, the DevOps InfoSec team can design and execute more effective penetration testing within the short time frames that are typically allocated for such purposes. The deployment pipeline also enables information security to gain the technical information that is essential for effectively assessing security vulnerabilities.

Programmatically build the infrastructure

Security experts recognize that almost any system can be compromised and once the environment has been penetrated, malicious code can creep in. It is common for hackers to compromise inadequately secured systems and then use them to attack other machines that are more secure. These types of attacks may simply involve a denial of service attack, which can then be traced back to the machines that were used in the attack. In denial of service attacks, unsuspecting victims may find themselves accused of mounting an attack on another system and required to explain why their machines were used for malicious intent.

When a system is compromised, the best course of action is to completely clean and re-provision the servers. DevOps provides some excellent procedures to build the servers from the operating system to the application base through a fully automated procedure. The ability to programmatically build the infrastructure is known as infrastructure as code. This key practice within DevOps is commonly used to support cloud-based infrastructures. These techniques have also been called agile systems administration, a term that refers to the value that DevOps provides in an agile development environment.

IBM® Tivoli® Provisioning Manager enables a dynamic infrastructure by automating the management of physical servers, virtual servers, software, storage, and networks.

Use automated systems to configure and verify the operating system

Infrastructure as code can be used to implement secure systems by providing automated procedures to configure and verify that the operating system conforms to established industry security standards such as the well-respected Center for Internet Security (CIS) Security benchmark, which defines precisely how the operating system should be configured. (See Resources for a link to the Center for Internet Security.)

Both the provisioning of the operating system and the verification of the security configuration can be fully scripted using the guidance provided in the CIS specification. Scripts enable servers to be fully provisioned and supported through an automated procedure. To ensure a reliable and secure platform, start with servers that are completely secure and that are configured correctly. After your server infrastructure has been provisioned and is verifiable, it is time to use the same principles to deploy the application base in a way that is secure and verifiable. The automated build procedures should ensure that the code can be verified once it is deployed by embedding version IDs into each configuration item including code components, configuration, and properties files.

Cryptographic hashes should be created for use in verifying the application deployment.

Deploying the application itself is just as important to provide a completely secure trusted application base that is not only verifiable, but also enables the detection of security vulnerabilities, including unauthorized changes. Building code that contains embedded version IDs for deployment verification and using cryptography to identify unauthorized changes ensures that you can create a secure trusted application base.

IBM Tivoli Provisioning Manager for OS Deployment enables automated remote deployment of operating systems.

IBM® UrbanCode Deploy provides an automation deployment framework that reduces deployment errors and improves efficiency, correctness, and traceability.

Provide a secure, trusted application base

To secure the trusted application base, build the application code using procedures that securely identify the exact binary code and all other configuration items (including XML and properties configuration files) so that you can definitely prove that the correct code has been deployed and confirm that no unauthorized changes have been made.

These procedures usually make use of cryptographic hashes that are first created during the automated build and packaging of the application. Once the code has been deployed, the hashes can be recalculated to verify that all of the code has been correctly deployed.

These same procedures can be used to establish reliable baselines and to identify any unauthorized changes that could potentially occur as a result of human error or even malicious intent.

Working from a secure, trusted, application base ensures that the systems have been reliably and accurately deployed and makes it easier to identify and address any possible security breaches. When authorized changes are implemented, the baselines must be updated to record and track the changes. These techniques are described in industry standards and frameworks, including the well-respected ITIL v3 framework, which provides guidance on how to build and maintain reliable services.

IBM® Rational® Integrated Development Environments (IDEs) run on the developer's desktop, providing rich capabilities for developing, unit testing, debugging, analyzing, modernizing, and optimizing applications.

Rely on standards and frameworks

Standards and frameworks provide expert guidance on how to implement secure and reliable systems. There are many functions and processes described in these guidelines including the ability to monitor and track code baselines. The ITIL v3 framework describes Service Asset & Configuration Management (SACM), which captures baselines that can be stored in the configuration management system (CMS). These baselines can be monitored by the configuration management database (CMDB) with discrepancies reported to the CMS. The definitive media library (DML) contains the baseline, official releases that can be deployed as part of provisioning a server.

These techniques, including the practices of monitoring baselines using intrusion detection systems (IDS), are well-regarded as industry best practices. The DevOps approach provides the automation and framework for implementing these procedures. Even with the best preventive practices, intrusions can happen. Being able to detect and respond to incidents is essential.

Enable early detection of unauthorized changes

When unauthorized changes occur either because of human error or due to a malicious attack, they should be immediately detected and they should trigger an incident response. The DevOps approach provides the procedures so that unauthorized changes can be easily detected and proactively addressed with minimal impact to the users. They also provide the basis for provisioning servers and deploying systems immediately to restore the system to a reliable baseline. These techniques require collaboration and excellent communication among the key stakeholders including development, operations, Quality Assurance, testing and information security teams.

Several IBM Rational change management software products help monitor software change and configuration management.


Conclusion

DevOps is well known for its focus on helping developers and operations to work together more effectively together. Information security, along with Quality Assurance and testing teams, are also key stakeholders and need to be represented within the DevOps framework. DevOps helps to improve communication and helps to transfer essential technical knowledge from the developers who are writing the code, to the operations and security professionals who are responsible for ensuring that there is uninterrupted service. With the DevOps structure in place, InfoSec can make informed decisions that will help protect the system infrastructure and applications from any unexpected occurrences that might compromise the system, whether due to unintentional error or malicious attack. Keeping systems secure helps to prevent system glitches and also ensures uninterrupted service. The DevOps approach and the deployment pipeline enable your team to roll out features that both meet your business needs and provide the secure trusted application base.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into DevOps on developerWorks


  • Bluemix Developers Community

    Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.

  • DevOps digest

    Updates on continuous delivery of software-driven innovation.

  • DevOps Services

    Software development in the cloud. Register today to create a project.

  • IBM evaluation software

    Evaluate IBM software and solutions, and transform challenges into opportunities.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=DevOps, Security
ArticleID=955941
ArticleTitle=DevOps best practices: Part 2. Ensure robust and effective information security with DevOps
publish-date=12102013