How to Set Up a Concourse CI/CD Pipeline

Share this post:

The importance of microservices in cloud-native applications

At IBM Cloud Garage, we work closely with clients to provide innovative solutions for their business problems and make them more efficient and effective during this age of digital transformation. The IBM Cloud Garage Method provides clients with the agility to transform their business one step at a time.

Microservices are becoming a critical element of any cloud-native application. Companies migrating to cloud from on-premise applications are increasingly going the microservices route. SOA (service-oriented architecture) was important when we were building monolithic applications but in the cloud-native world, microservices are taking over. One important aspect of microservices is understanding how to divide the application development tasks based on business capabilities. You can refer to the IBM Cloud Architecture center for Microservices to read more about this.

Set up a Concourse CI/CD pipeline to make the development process more agile and deployment more visible

To hit the ground running for any project based on cloud-based applications, it is very important to set up the associated infrastructure for the development of microservices. Quick and high-speed development can be achieved if the team has a way to do Continuous Integration (CI) and Continuous Delivery (CD) of the application. IBM Cloud services like toolchain can help to set this up.

In this blog, we will look into how you can set up a Concourse CI/CD pipeline to make the development process more agile and deployment more visible.

Concourse CI/CD is an open source platform which runs on two main Docker containers. Once Concourse is installed, all you have to do is to use the docker-compose up command to bring up the Concourse server. Concourse uses Postgres as its database. To dive deeper into Concourse, I have created a Github project at this location.

Concourse Landing Page

fly CLI command line tool

fly CLI is a command line tool that you need to use to set up a pipeline on Concourse. fly can be downloaded once the Concourse server instance is running and when you are logged in for the first time. The initial user ID and password by default are “test” and “test,” but you can change them later.

fly CLI will be used to set up, un-pause, pause, and destroy pipelines.

The Concourse pipeline set up needs a set of files that are associated with each repository. Each repo represents a microservice. It is best practice to associate one set of files related to Concourse to one repo.

Figure 1: Pipeline for this blog

Figure 2: ci folder structure

Pipeline.yml

Let’s dive into more details on pipeline.yml. Pipeline.yml is the file that Concourse reads in order to know what resources and jobs are defined. Resources, as the name suggests, are resources to the pipeline. You can configure your Git uri and private key here.

Define the name or resource to which the GitHub repo will be pulled. Another resource can be information about the deployment environment—basically, your PaaS to which the microservice application will be deployed.

Here is an example pipeline.yml resource section:

Figure 3: pipeline.yml resource section

The other section is about the jobs that we need to define to get the build started. Once it is passed, deploy the code to the test and then to the non-prod environments. This will eventually lead the deployment to production environment.

Here is an example pipeline.yml jobs section:

Figure 4: pipeline.yml jobs section

Once the pipeline.yml is ready, make sure you have build.yml and build.sh created. Build.sh is the shell script that is invoked from build.yml. Each node (get, task, and put) in the pipeline is a separate Docker container. When the source code is pulled from the defined repo, it is built in one Docker container and that output is not available to the other node. For example, put will not have access to the build code. So, in the build.sh, we need to manually copy the build directory so the put Docker instance can have access to those files which are created during the build process, especially the WARs and JARs, for the application to be deployed.

Here is an example build.yml:

Figure 5: build.yml

Here is an example of build.sh:

Figure 6: build.sh

Once all files are set up, use fly CLI to set up pipeline as per the instructions on Github repo and see the CI/CD pipeline in action with each developer commit to the repo.

Learn more

Get in touch with IBM Cloud Garage today to get your CI/CD pipelines going. If you would like to see what the IBM Cloud Garage can do for you, please schedule a cost-free, four-hour talk with our experts here.