How IBM Cloud VPC transformed its continuous integration and delivery strategy using IBM platform automation tools
The IBM Cloud® Virtual Private Cloud (VPC) team needed a new continuous integration (CI) infrastructure that could support more than 1,200 CI pipelines. The team had more than 500 artifacts to build, scan and test—and this number continued to grow every day.
Given the global nature of the development team, they needed an infrastructure that could ensure high availability to enable effective collaboration across different time zones. Their ideal solution would incorporate standardized processes aligned with corporate directives and DevSecOps recommended practices. It would also be easily adopted by developers and include comprehensive security.
After exploring various in-house solutions, the team identified IBM Cloud® Continuous Delivery (CD) as the most suitable option. The team overcame the challenges of moving from the traditional infrastructure to a cloud-native solution, and they were able to optimize their workload with the new architecture.
The earlier CI infrastructure for IBM Cloud VPC was vast and complex. Due to historical decisions, the original setup consisted of a traditional on-premises deployment that combined both commercial and custom-built tools. The CI system was required to operate nonstop to support hundreds of developers building and testing a wide range of artifacts such as Debian packages, virtual machines and container images.
Maintaining a highly available infrastructure designed for security and compliance, and based on bare metal servers posed significant challenges. It demanded specialized expertise across multiple domains, including network configuration, operating systems, database administration, security scanning and CI orchestration tools. The skill set required went well beyond the typical profile of a DevOps engineer, needing dedicated personnel with deep technical breadth.
To reduce complexity, lower the skill barrier and streamline maintenance, the first major step was to decommission the on-premises bare metal server-based infrastructure in favor of a virtual server hosting solution.
The new environment was built on IBM Cloud VPC with approximately 90 virtual server instances (VSIs), each provisioned with 64 vCPUs and 256 GB of RAM. Each VSI was dedicated to a specific CI function or supporting service. For instance, the CI orchestrator required Kubernetes-based workers, so several VSIs were configured as Kubernetes worker nodes.
The orchestrator also required a high-availability database, which was deployed across multiple VSIs as an active-active cluster. More VSIs were allocated for storage servers, image registries, Domain Name Services (DNS), Virtual Private Network (VPN) connections and load balancers. Also, a queuing engine to manage CI job load was added, along with the orchestrator’s web UI, security scanning tools and various proxy services to enable intercomponent communication.
While the transition to cloud infrastructure eliminated the burden of hardware failures, the overall system remained highly complex from an operational standpoint. The architecture continued to require skills and expertise not typically found within standard DevOps roles.
Moreover, evolving security and compliance requirements required ongoing enhancements to security tool integration, infrastructure hardening and software stack upgrades to remain aligned with industry standards. These demands used the capacity of the CI team. Out of a seven-member team, three engineers were fully dedicated to infrastructure maintenance, which limited the team's ability to focus on improving and evolving the CI pipelines themselves.
The diagram provides an overview of the core components that comprised this cloud-hosted CI infrastructure.
IBM Cloud VPC required a modernized approach that would help reduce complexity, enhance security, optimize operational costs and minimize the skills barrier. The adoption of the IBM Cloud Continuous Delivery Service marked a significant turning point in this evolution.
Previously reliant on dozens of virtual server instances and custom-built components, IBM Cloud VPC moved to a fully cloud-native architecture. This transformation eliminated the need for self-managed networking tools, databases, continuous integration orchestrators and self-managed Kubernetes clusters. Instead, services readily available in the IBM Cloud catalog were used to help deliver a streamlined and efficient solution.
Virtual servers were replaced by two Kubernetes clusters, used for hosting Tekton private workers. The current infrastructure supports a higher workload than the previous VSI-based environment. This system uses 30 Kubernetes nodes (each with 8 vCPUs and 64 GB of RAM) and 44 nodes (each with 16 vCPUs and 128 GB of RAM).
This shift eliminated the dependency on virtual server instances, substantially reducing operational overhead while enabling multiregional failover through Terraform-based automation. If a failure occurs, the entire infrastructure can now be restored within minutes.
As a result, infrastructure costs were reduced significantly. The need for CI team members to dedicate full-time resources to infrastructure maintenance was eliminated. Maintenance activities now require only a few hours per month. This development has enabled the CI team to focus entirely on the creation, enhancement and operation of CI pipelines.
The diagram illustrates the logical components of the newly implemented infrastructure.
As part of this transformation, the CI pipeline was restructured into a shared, multitenant model. This standardization allowed all teams within IBM Cloud VPC to use a common infrastructure and templated pipeline framework with common quality and security gates. This improved efficiency and minimized duplicated efforts.
Developers were granted read-only access to toolchains to help maintain consistency and security across the development lifecycle. At the same time, teams retained the ability to manage their own secrets with IBM Cloud® Secrets Manager to help ensure secure and compliant handling of sensitive information. The new architecture also supports customization of build and test processes, striking an effective balance between developer flexibility and enterprise governance.
With this model, the IBM Cloud VPC CI team retained full responsibility for system availability and pipeline governance. This approach helped to ensure that the recommended practices were maintained across the whole set of VPC services. This approach reinforced the principles of efficiency, security and scalability, all while providing development teams with the tools they needed to succeed.
Security and compliance also improved drastically. IBM Cloud CD service has out-of-the-box integration with all the security scanning tools that the IBM chief information security officer (CISO) recommends for internal tool usage and provides automated compliance evidence collection.
Standardized security gates helped ensure that every build met enterprise-grade requirements, providing proactive vulnerability management. This approach dramatically shifted left security vulnerability management, massively shrinking the number of vulnerabilities found in production.
The operational impact of these changes was significant, reducing the cost of the CI infrastructure by approximately one-third compared to the earlier system. The result was a more agile and efficient software delivery process that directly benefited IBM Cloud VPC and its clients. The VPC CI team, made up of a handful of members, is currently running and maintaining over 1200 pipelines and its number is increasing daily expanding over further shift left, automated scenarios.
IBM Cloud VPC’s transformation exemplifies a core IBM belief: "If we are not using our own products, how can we convince customers to use it?” By adopting IBM Cloud Continuous Delivery Service internally, IBM Cloud not only validates its own technology but also refines its capabilities for broader enterprise adoption. This hands-on experience enables IBM to address real-world challenges, making its solutions more resilient and enterprise ready.
Standardization accelerates efficiency. A unified pipeline model simplifies operations and helps enhance security. By delegating infrastructure management, teams can shift their focus to business innovation. Working across teams can ensure that security, scalability and efficiency remain top priorities.
As a first adopter, IBM Cloud VPC sets the standard for modern cloud-native delivery. For enterprises looking to simplify their CI/CD processes, IBM Cloud Continuous Delivery Service delivers a proven, secure and cost-effective solution.