IT security concerns have taken center stage with the seemingly endless stream of bad news about security breaches that have compromised IT systems around the world. The breaches have resulted in bad press for some large reputable companies, financial losses for many, and in some cases, risks to national security. These are legitimate reasons to raise security concerns in an IT organization and take a second look at an organization's security policies and procedures.
Security teams within some organizations, however, are seen as hindering innovation and speed of change. Application delivery teams (development, quality management, operations) see security teams as being at odds with their goals. Application delivery teams are motivated to deliver new and innovative capabilities quickly. Security teams are determined to ensure that new systems and capabilities are secure and that security functions are robust. These goals might seem to be at odds.
With regard to business outcomes, however, the goals are not at odds at all. The business outcomes – improve time to value, deliver innovative business solutions, create high-quality products, increase market share – all require both application delivery teams to continuously deliver capabilities and security teams to continuously secure them. The teams share similar goals, but approach them from different angles.
The tension between these seemingly opposing goals led to the birth of the DevOps movement. DevOps, as its name suggests, was developed to address the lack of collaboration and cooperation between Development (Dev) and Operations (Ops) teams. As with development and security teams, the development and operations teams are also measured differently. Development teams are measured according to their ability to produce new capabilities and to create change. Operations teams are measured according to their ability to keep their systems stable and available to ensure stability.
The DevOps movement seeks to address this challenge and to break down the barriers between development and operations teams. DevOps capabilities require a change in organizational culture (people), a change in how the teams work and collaborate (process), and a shift to automation to reduce manual error prone tasks (tools). DevOps principles are grounded in lean thinking, a set of approaches that led to the development of lean manufacturing principles. DevOps applies these principles to application delivery, fostering collaboration between development and operations teams, reducing waste, minimizing rework, eliminating process bottlenecks, and limiting over-production. DevOps uses the characteristic of movement and change to improve the speed of delivery and increase stability, by making change easy, quick, and repeatable.
As DevOps matures and finds broader adoption in enterprises, the scope of DevOps must be expanded to include all the teams and stakeholders that contribute to application delivery. In the same way that DevOps principles foster collaboration and trust between development and operations teams, they can improve collaboration and trust among development, operations, and security teams.
As security teams scrutinize and secure application delivery processes, it is imperative that they collaborate with the development and operations teams to secure the DevOps delivery pipelines and processes, too. Given its roots in lean thinking, DevOps is not designed to maximize speed at the expense of security. It is designed to provide rapid feedback from the delivery of smaller chunks of capability, in short cycles. This rapid delivery and continuous feedback cycle can help to enhance security. Including security in the DevOps lifecycle ensures that securing the applications and systems being delivered is an ongoing process included in the entire delivery lifecycle, rather than a step that gets added to the end of the delivery cycle.
Just as lean manufacturing revolutionized factory automation and product delivery, DevOps transforms application delivery. The advent of factory automation required the development of practices to secure the product delivery assembly line. Incoming components, line workers, automation specialists, assembly processes, and other elements had to be secured and validated. Similarly, security practitioners need to work with the application delivery teams to secure and validate the application delivery practitioners themselves, the processes, and the automation tools. Security needs to become an integral part of DevOps adoption.
This article explores security vulnerabilities in application delivery. These vulnerabilities must be mitigated, as an organization adopts DevOps. This article also includes guidance on how security teams can work with the application delivery teams to address and mitigate these security risks.
Managing security-related risks
Businesses fear the risk of residual vulnerabilities in all software used in the business. These risks include:
- Vulnerabilities related to the supply chain
- Insider attacks from malicious actors
- Loss or compromise of source code
- Development process subversions
- Errors and mistakes in the development project
- Weaknesses in the design, code, and integration
These risks apply to any style of software development lifecycle or methodology including: waterfall projects, agile projects, or projects that have adopted a broader DevOps approach. Because of the streamlined nature and advanced automation within DevOps projects, events and conditions related to these risks must be detected and responded to in a continuous manner, throughout the delivery lifecycle.
For each kind of risk, special considerations for a DevOps project are described in the following sections.
Vulnerabilities related to the supply chain
Any software project that incorporates software components created outside of the project can be said to have a software development supply chain. The components might be created by suppliers within the company or external to the company or organization that owns or delivers the software project. The security characteristics of the software from the software supply chain have a significant and lasting effect on the security of the software created in the project.
In traditional development projects (including waterfall and iterative projects) it is typical for the development team to evaluate the security characteristics of software from the supply chain. This evaluation involves reviewing component documentation, seeking approval based on licensing and support-ability, and performing security scans.
DevOps development teams gain maximum flexibility by making real-time design, coding, and integration decisions throughout the project lifetime. For this reason, the development teams might select supply chain components that advertise greater functionality and ease of integration and that downplay the security and assurance properties of the components.
To mitigate this limitation, build rigorous quality checks into the software delivery process by adopting continuous testing, a key tenet of DevOps. This practice includes testing in every stage of the delivery cycle. The tests include developer-run code-level tests that share the test results with the consumers of the components; automated functional, performance, integration, and security testing; and manual and automated code reviews of every component delivered. Because DevOps encourages delivering smaller chunks or changes to each component in short cycles, the result is continuous testing of smaller changes to the components as they are delivered. This approach mitigates the associated risk.
Insider attacks from actors
Although the exact numbers remain unknown, the evidence in the marketplace shows that over the past five years the percentage of all cyber-crimes perpetrated by insiders is statistically significant. These attacks can result in source code loss, source code compromise, or subversion of the development process.
These attacks might originate from the direct action of malicious insiders, or as a result of malware infection on networks, workstations, or servers used within the development environment.
To reduce the likelihood of these types of attacks in a traditional development environment, the development infrastructure is typically secured and instrumented for detection and alerting of anomalies. Advanced and streamlined automation within the DevOps environment increases the difficulty of instrumentation and detection of anomalies that might result in source code loss, source code compromise, deployment of malware, or subversion of the development process. This limitation can be mitigated by including security testing – both white box and black box security tests – in the set of testing tasks carried out during the delivery cycle. These security tests, when run in every iteration or sprint, can detect any such malicious attacks.
Virtualized, software-defined infrastructure makes it possible for its configuration to be under change control and thereby auditable. The ability to repeatedly break down and rebuild parts of the DevOp assembly pipeline helps minimize the occurrence of persistent malware on those parts.
Errors and mistakes in the development project
Traditional development projects, whether waterfall or iterative, are generally supported by project management and project tracking tools and systems that provide orchestrated workflow and task completion checkpoints. In particular, the release of the finished software product is preceded by a project review that includes examination of evidence of completion of major tasks and milestones.
Projects that apply a lean (agile) or DevOps approach tend to have shorter delivery cycles in which small components or changes to the software product are delivered more frequently. Although each set of changes might not be delivered to the customer or user, the fast, short cycles can result in less rigorous project reviews and a less careful examination of completion of major tasks and milestones. These shortcuts can make it possible for development project errors and mistakes to slip into the development cycle unchecked. However, the goal of a DevOps project is to deliver smaller components of the software project to the quality assurance team and to the project review processes to reduce the risk of larger project errors and mistakes from occurring in the first place. Catching smaller errors early by delivering smaller software component changes more often, reduces overall risk.
Weaknesses in the design, code, or integration
If supply chain security is adequately managed and if insider attacks and project errors are under control, the most significant remaining risk in development projects of any style is the introduction of weaknesses that can be exploited after the software has been deployed. These weaknesses might be introduced throughout the development project in design, coding, and integration.
Minimize the likelihood of software weaknesses using one of these strategies:
- Perform iterative tests and remediation.
- Implement a "secure by design" strategy.
Iterative test and remediation strategies can work on small-scale projects in which costs and schedules are not constrained, and in which comprehensive testing tools are available. Secure by design development strategies are maturing at the same time that DevOps projects are emerging. A secure by design development strategy as exemplified by the IBM secure engineering framework can be applied to DevOps projects.
Use an approach, described in the following section, that applies the eight essential practices in the IBM secure engineering framework to DevOps style projects.
Setting the stage for addressing security
The adoption of DevOps automation is similar to the transformation of manufacturing systems from being human-intensive to being much more streamlined and automated. Manufacturing processes have evolved from delivering inventory in just the right time and place on a manufacturing floor to positioning the steps on the line, to shifting to controlled, accurate, and high-speed robotic systems, rather than relying on human hands to install, connect, move, and assemble units.
Security risks apply to both non-automated and automated systems. Explore how each risk shows up in manufacturing facilities and in lean, streamlined manufacturing environments.
Examples of vulnerabilities in the supply chain
In manufacturing, the existence of multiple suppliers delivering components to the supply chain introduces vulnerabilities. These suppliers can intentionally or unintentionally supply low-quality or defective components. In traditional supply chains, humans notice when something is wrong with the supplied components and they raise a flag to alert the line. This manual process mitigates risk.
In lean or DevOps projects in which automation is used extensively, the automated processes might or might not detect an issue with the supply, depending on whether the automated elements contain quality assurance checks to validate the incoming supply.
The mitigation in both cases is to manage and verify the incoming supply chain. For lean or DevOps projects, this involves adding testing gates to replace the monitoring function performed by humans in the past. For example, a set of automated tests might be implemented to verify that a new level of an open-source toolkit that has just been received is operating within specified tolerances for use by the including application.
Examples of preventing insider attacks from actors
In manufacturing, line workers can deliberately leave a fitting incorrectly connected, can fail to connect something, can insert foreign objects into the assemblies on the line, or can even sabotage someone else's work and then cover their tracks by wiping clean the device.
In DevOps application delivery environments, automation takes the place of individual practitioners. However, the programmers of the automation tools (for example, the creators of the automation) might also insert behavior into the automation that might deploy malware, sabotage a configuration, or otherwise tamper with the system.
The mitigation in both cases is to have checks and balances between workers or, in the case of the automation, to involve multiple checks and balances in the creation of the automation code. Insider attacks are prevented by scope of control, auditing, and a requirement for multiple sign-offs and approvals prior to release. Similar guards and gates can be created as test cases and built into the automation. Consider that the creation of the automation itself can be a possible attack point. The use of software-defined infrastructure under change management helps mitigate that attack point.
Examples of source code loss or compromise
To apply the manufacturing analogy to software development, source code is either the raw materials used to create the assembly on the assembly line or it is the blueprints and plans that are followed by the workers to create the assembly. In either case, destruction or removal of the code/plans, or tampering with the code/plans would affect the resulting assembly on the assembly line.
For DevOps application delivery environments, improper handling of the source code (raw materials or source used by compilers to build binary components) can result in tampering or compromise. Tampering with design materials or instructions used to develop automation (robotic movements or deployment automation) can cause similar undesired results.
The mitigation on an assembly line is tight control and auditing of both the raw materials and the plans and designs, along with periodic quality assurance testing that the assemblies match the design and that the raw materials have not been tampered with. In the DevOps software delivery model, more automated testing of the assemblies verifies that they conform to the specifications. Furthermore, monitoring, auditing, and enforcing access to design materials, source code, and source code for the automation (robotic behavior of code assembly and deployment), ensure that they do not have any security flaws or vulnerabilities.
Examples of development process subversions
In the analogy of a manufacturing assembly line, the line workers might not follow the assembly line processes and procedures as designed. Every worker in the assembly line has a standard operating procedure (SOP) to follow. Departing from these procedures can result in defective products being produced.
In application delivery environments SOPs exist for the practitioners engaged in software coding, integration, testing, deployment, and similar tasks. Departing from these procedures can result in defective software being delivered. For robots or the automation framework, errors can be caused by faulty programming of the automation.
Recently, a trading error at an international financial services firm that resulted in a US$440 million loss for firm was traced back to a deployment engineer not following the deployment SOP properly. This error went undetected, because the firm had no quality checks, automated or human, to validate that the deployment was done following the proper process.
To mitigate the risk on the assembly line implement sufficient training of the line workers on the processes and procedures, and institute oversight and quality checks that continuously ensure that workers are following the processes. In the application delivery environment, the processes, the oversight, and the quality checks can be automated using tools.
Examples of development project errors and mistakes
In manufacturing, people make mistakes and errors as they work. Work performed by humans is error-prone. Errors can be introduced by the line workers and by the people designing the processes for the line workers.
In application delivery environments, errors come in various forms: typos in code or scripts, errors in documentation, mistakes in data entry, and similar situations.
To mitigate the risk on the assembly line, implement oversight and quality control and create robust systems that can prevent errors or catch them early. In application delivery systems, methods to reduce errors have developed over time. Tests can be embedded in code to validate the code and to validate the appropriate use of the code components in the application. DevOps encourages building "anti-fragile" systems, which are systems that include redundancies, robust error recovery, and self-repair functions. In an extreme implementation of this approach, some companies have adopted an approach in which if any virtual instance of a node detects that it has an error, it replaces itself with a new instance and de-provisions the instance experiencing the error. No attempt is made to fix the error.
Examples of weaknesses in design, code, and integration
In manufacturing, handoffs and communication between designers (architects and mechanical engineers), process engineers (industrial engineers and team leaders), and assemblers (machinists and fitters) result in ill-fitting assemblies, changes during manufacturing, bending parts to fit, swapping for other parts in-flight, and other workarounds. This challenge is exacerbated by the reliance on contractors or suppliers who are typically outside firms. One infamous example of a handoff error is the Apollo 13 incident, in which the change of the voltage requirement for a subsystem of the lunar rocket was not communicated to a contractor, resulting in near-disaster during the lunar flight.
In application delivery, such handoff errors occur as teams hand off their code to other teams developing code or to teams responsible for integration, quality assurance, build, and deployment. The challenge is further exacerbated by the need for multiple suppliers and vendors to complete these handoffs across company boundaries.
To mitigate the risk in manufacturing standards are developed for all vendors to follow. These standards, coupled with extensive documentation and communication about the specifications of components and handoff quality checks, help mitigate these issues. In application development industry standards for component interfaces have been developed, but teams must still rely on contracts and service level agreements (SLAs) to help mitigate these handoff challenges. Standard tools that provide automation, rather than manual handoffs and deployments, help mitigates the risk that handoffs introduce.
The API economy and security
In addition to these security vulnerabilities, the new trend toward an API economy adds additional security concerns. Application programming interfaces (APIs) are programming interfaces made available by service providers to consumers who use these interfaces to include application functions exposed by the API in the application they are delivering. APIs are becoming more and more prevalent, because enterprises see them as a way to create a marketplace around their services. The marketplace might be internal, open to partners, or open to external consumers of their services. For example, a bank might expose APIs to their payment system so that other divisions of the bank or even third-party service providers can include functions in their applications that take advantage of the bank's API-exposed payment system.
As more APIs become available, the risk of security vulnerabilities introduced in the APIs themselves or by rogue users of the APIs can compromise the systems exposed by them. To mitigate the risk, apply strong testing protocols of the APIs and the applications using them.
API providers must ensure that the API developed does not expose them to malicious users who might compromise their systems. These security vulnerabilities might have been intentionally or unintentionally introduced by the rogue developers. API consumers must ensure that the data they access or deliver through the API is secure and that the APIs are appropriately used by their applications, without exposing any security risks. Both suppliers and providers must use proper authentication and provisioning protocols to ensure only valid use of the API is permitted and no third-party is able to misuse the API access provided.
To mitigate this risk with DevOps principles, include rigorous security testing on an ongoing basis to ensure the robustness of the API security. Both API providers and consumers must run rigorous automated tests of the API with each new release of the API. They must engage in continuous testing to ensure that any misuse or breach of the API is detected and addressed in a timely manner. These tests must be included in the deployment process of the API itself, to ensure continuous security testing of the API and of the deployment processes of all the applications that use the API, to ensure that the application is secure. Security testing and security monitoring is a continuum of processing across development, test, and production environments.
Adopting lean principles on the factory floor and in software development leads to reduction of waste and rework. Similarly, just as factory automation led to a new set of potential security risks and required the adoption of methods to mitigate them, the adoption of DevOps practices can also lead to novel security risks. This article describes some of these risks related to DevOps adoption and proposed approaches to mitigate them.
DevOps adoption results in benefits that lead to more efficient and effective software delivery. The security risks introduced by this rapid and agile approach to software development, testing and delivery are well identified and easily addressed. They must not be ignored. Just as ignoring security risks related to factory automation can result in serious quality control challenges, not addressing the relevant security risks exposed by DevOps practices can result in severe quality issues as well. As DevOps evolves from a philosophy with a set of guiding principles to a well-defined set of practices with relevant adoption paths, addressing these security risks must become an inherent part of these practices.
Organizations and teams that adopt DevOps can ensure the mitigation of security risks by including the organizations' security teams into the DevOps lifecycle. These teams need to become the stakeholders responsible for analyzing and determining which risks are relevant to different projects in the organizations and developing strategies to address and mitigate them. Security teams must contribute security-centric quality gates to the DevOps environment as one example point of collaboration.
DevOps introduces an approach of continuous delivery and continuous testing of small sets of capabilities being delivered by an application delivery organization. Security teams can take advantage of this delivery approach as a way to reduce security risks. By continuously securing these smaller releases of functionality, they can identify security vulnerabilities early in the lifecycle and mitigate the effects early.
- Explore the IBM Secure Engineering Framework.
- Learn more about lean software analytics in this blog post by Murray Cantor, IBM Distinguished Engineer.
- Understand DevOps and how your organization can gain real business benefits from it in the book DevOps For Dummies (Sanjeev Sharma)
- Explore more about IBM DevOps and find resources for DevOps developers on the DevOps section of IBM developerWorks.
- Explore the Rational software area on developerWorks for technical resources, best practices, and information about Rational collaborative and integrated solutions for software and systems delivery.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Attend a free developerWorks Live! briefing to get up-to-speed quickly on IBM products and tools, as well as IT industry trends.
- Watch developerWorks on-demand demos, ranging from product installation and setup demos for beginners to advanced functionality for experienced developers.
- Improve your skills. Check the Rational catalog, which includes many types of courses on a wide range of topics. You can take some of them anywhere, anytime, and many of the Getting Started ones are free.
Get products and technologies
- Download a free trial version of Rational software.
- Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment.
- Check the Rational software forums to ask questions and participate in discussions.
- Get connected with your peers and keep up on the latest information in the Rational community.
- Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
- Rate or review Rational software. It's quick and easy.
- Share your knowledge and help others who use Rational software by writing a developerWorks article. Find out what makes a good developerWorks article and how to proceed.
- Follow Rational software on Facebook, Twitter (@ibmrational), and YouTube, and add your comments and requests.
Dig deeper into DevOps on developerWorks
Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.
Updates on continuous delivery of software-driven innovation.
Software development in the cloud. Register today to create a project.
Evaluate IBM software and solutions, and transform challenges into opportunities.