DevOps Best Practice - Establishing a “Single Source of Truth”
Reedy Feggins 120000A43D Visits (3149)
Establish a “Single Source of Truth”
In a recent post, DevO
This post focuses on Establishing a single source of truth. DevOps expert Paul Duvall recommends that DevOps teams “version everything”; including application code, infrastructure, configuration, data, and even internal system artifacts to gain the capacity to deliver software to users quickly and often.
“Teams that version everything aren't constantly trying to figure out which version of the application code goes with which database and which version of the software application works with which environment. Source files that make up the software system aren't on shared servers, hidden in folders on a laptop, or embedded in a non-versioned database.
Continuous Delivery is focused on more than just code. Instead of maintaining silos with code located on developer’s desktops or release artifacts out on network (or samba) shared folders, we recommend implementing a “single source of truth” for the entire production environment.
To achieve a “single source of truth” requires managing development artifacts (requirements, code, libraries, etc) and deployment artifacts (e.g., WAR and EAR files, configurations, infrastructure changes, release notes, etc…).
Jez Humble and Daâ€‹vid Farley in their book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Auto
These tasks can be automated or as a set of build task, usually provided by the CI servers, so the stage completes in a reasonable length of time. The commit stage should ideally take less than five minutes to run, and certainly no more than ten minutes.
The first step is to compile the latest version of the source code and notify the developers who committed changes since the last successful check-in if there is an error in compilation. If this step fails, we can fail the commit stage immediately and eliminate this instance of the pipeline from further consideration.
Given the nature and storage requirements for these artifact types, organizations usually rely on SCM repositories to support the commit stage and an “Artifact Repo” to store the release artifacts.
In the remainder of this blow will summarize some of the common scenarios from my experience working with Agile and DevOps teams. For those that need additional background, here is a list of references used in this blog.
Managing development artifacts using Rational Team Concert (RTC)
Rational Team Concert (RTC) is a scalable enterprise software development application offering source control, builds, tracking and planning capabilities. RTC has extensive integrations to tools such as other source code tools like Git, application automation tools like UrbanCode, or testing tools like Rational Quality Manager (RQM).
While many teams have seen the benefit of using source code management (SCM) tools to store and version their development artifacts, many teams are creating solutions with technologies that don’t require actual builds to be performed. Using RTC organizations have the ability to version and manage source code changes at an individual developer, team, and project (or program level).
For example take a look at this delivery pipeline example, RTC is being used to support several stages of the delivery pipeline. The team uses RTC for agile planning, version control and integrating with the CI Build Server.
Most agile software development teams typically work on a large base of files that comprise the source code for a software product or system. As a team member, you work within this code to add new code in support of the stories they are working on or to fix defects found during testing. After you verify that your changes are correct (by building and testing the code in a private workspace, for example), you share the changes with the rest of the team, whose members have also been changing files, including ones on which your work depends.
Single Stream Development
While RTC can support most if not all of the different development scenarios, agile teams often want to adopt a “single stream” development as it requires the least amount of setup, administration and oversight.
In this approach the agile team develops along a "straight line", or in other SCM systems this concept would be referred to as “working on main branch”. In this scenario:
Take a look at the example below, extracted from an article written by Dan, see reference 2 above, on setting up Single Stream Development
Note how each workspace is associated with an individual developer, tester, analyst or stakeholder, and all workspaces have the same project/team stream as their delivery target. Using this approach, user has their own sandbox to develop / test any code changes. Once ready, they can check-in and deliver these to the rest of the team; which will appear as incoming changes for the remaining team members.
Jazz SCM Basic
Rational Team Concert™ source control organizes versionable items (files and folders) into components and streams, and provides workspaces where you can view and modify file and folder contents. The is an object in the repository on the Jazz server, used to store items that have been placed under source control. Each workspace is typically associated with a local sandbox on the developers pc which allows them to perform various tasks such as: checkout, check in, and delivery.
Ais a collection of one or more components. A is a collection of related artifacts maintained under change control, such as the binaries, requirements documents, or a set of source code artifacts that can be compiled to build a deliverable.
Artifacts under source control are grouped into components, which can be included in zero or more streams. A component is the fundamental organizational unit in RTC and typically contains at least one Eclipse project, and usually contains several.
Here is an example below, two developers are collaborating by flowing their changes back and forth through the Hyperbola stream. The unit of change is a change set, which provides atomic grouping of changes.
Development changes occur through a repository workspace, which flows to a stream. The workspace contains one or more components of that stream. Artifacts from repository workspaces can be loaded into local (Eclipse-based) workspaces on the client machine, where they can be modified. Repository workspaces are generally created for development and build purposes. Overlapping sandboxes (a sandbox within a sandbox) are not permitted.
Periodically the team will take a component baseline to capture the files and configuration at any particular point in time. Baselines serve as fixed points of reference, and can be used for reverting the component to an earlier configuration, or for initializing streams and other workspaces. A baseline consists of one or more change sets.
A stream snapshot includes one baseline for each component in a workspace or stream.
Agile teams using employ continuous integration practices and tools to allow developers deliver changes to a source code repository and a build system integrates the changes into the build output. Using the build output, continuous delivery deploys the artifacts to the target systems. Because continuous integration increases the number of new builds available for testing, the operations team must update the deployment environments more frequently.
An artifact repo is a repository used to store “release” artifacts, such as War or Ear files. It should provide ability to version these artifacts and have extensive search capabilities. Artifact repos and source code management supporting continuous integration tools serve two different purposes and one cannot be substituted with the other. When working with an artifact repository, you need to make sure it’s secure, is being backed up and accessible for your deployments (including to production).
An artifact repository is needed but the artifact repo is a conceptual element and may not always a distinct tool. For example, here is a delivery pipeline for mobile development process from an recent IBM DeveloperWorks article, see reference 5 above.
Here is a breakdown of the deployment pipeline steps
The release artifacts are created by the build are added to CodeStation, an artifact repository that is part of IBM UrbanCode Deploy, manually, automatically or based on a predefined schedule time / date. CodeStation also tracks the versions of the artifacts and maintains a copy of the versions.
If you do decide that you need repository manager separate from your deployment tool, there are several out there. For example, Nexus is a maven repository manager used for hosting artifacts created in your company, or for caching external artifacts. Check here for Getting started with Nexus.
Nexus provides the ability to manage “released” and “snapshot” artifacts, snapshot artifacts are those build artifacts from components that are currently being developed. It also supports Release repositories that host fixed versions of artifacts so that you can support environments that require a consistent (non-changing) set of components. Nexus uses a naming conventions based on the groupId, artifactId and version (GAV):
See, the Apache Software Foundation for more details.