Agile DevOps: Dynamic configuration

Reduce static configuration in the software delivery life cycle

Between applications, environments, and individual instances, how much time do you spend managing configuration values? Probably much more than you'd expect or would like to admit. Configuration management doesn't have to be such a painful experience. This time, DevOps expert Paul Duvall describes how to manage configuration data to reduce the duplication and misconfiguration that can lead to deployment errors.

Paul Duvall, CTO, Stelligent

Paul DuvallPaul Duvall is the CTO of Stelligent. A featured speaker at many leading software conferences, he has worked in virtually every role on software projects: developer, project manager, architect, and tester. He is the principal author of Continuous Integration: Improving Software Quality and Reducing Risk (Addison-Wesley, 2007) and a 2008 Jolt Award Winner. He is also the author of Startup@Cloud and DevOps in the Cloud LiveLessons (Pearson Education, June 2012). He's contributed to several other books as well. Paul authored the 20-article Automation for the people series on developerWorks. He is passionate about getting high-quality software to users quicker and more often through continuous delivery and the cloud. Read his blog at Stelligent.com.



11 December 2012

About this series

Developers can learn a lot from operations, and operations can learn a lot from developers. This series of articles is dedicated to exploring the practical uses of applying an operations mindset to development, and vice versa — and of considering software products as holistic entities that can be delivered with more agility and frequency than ever before.

Managing the configuration of multiple environments is often the bane of many software delivery systems. Most organizations manage properties by using static property files (for example, local.properties, dev.properties, test.properties, stage.properties, and prod.properties) for different environments. I've designed many software delivery systems that way myself and advocate this approach in "Automation for the people: Speed deployment with automation." It's among the most simple approaches I've seen — but if you need to manage a software delivery system with hundreds or even thousands of individual properties for multiple applications and hundreds of instances, it won't scale and ultimately becomes unmanageable. Most failures result from someone entering a wrong value in any of what might be hundreds of different property files for a single software system.

The situation isn't hopeless, however. By effectively using automation, dynamic configuration, and a cloud infrastructure, your software delivery system can drastically reduce the need for static properties when moving software between environments. You'll learn how in this article.

Typical properties

Table 1 lists some of the typical properties that teams within an organization manage:

Table 1. Typical properties in a software delivery system
PropertyDescription
IP addressUnique addressable location for a node
EndpointDomain name for any node — load balancer, instances, and so on
Domain nameUnique addressable name that users use to access the application; interfaces with the domain registrar and domain service
Database nameName of a database used by the application
PortUnique numbers between 0 and 65535 used for internal and external access to nodes
Usernames and passwordsUsernames and passwords used for headless access to resources
Directories and filesDirectories and files within an operating system
SSH keysSecure Shell key files to access secured resources

This isn't an exhaustive list, but it represents the more commonly used types in a typical software system.


Traditional properties management

In most software organizations, which are composed of silos of development and operations teams, the development team (developers, testers, analysts, and so on) will often:

  • Create a file (such as local.properties) to manage workstation-related properties.
  • Modify properties for the "development" environment in a dev.properties file and commit it to a version-control repository. Because dynamic properties (such as IP addresses) can change, development must continually modify this file when such changes occur.
  • Modify a test.properties file with the same property attributes, but with different values for testers, because they're likely using different IP addresses, files, and directories.

Configuration by the development team usually doesn't end here, because the development side of the organization typically manages other environments in addition to "development" and "test."

In a traditional organization, the development team will hand off the software package to the operations team (DBAs, systems/ops, release engineering, an so on) to deploy the software to downstream environments as part of the software delivery cycle. That team:

  • Modifies properties for the "staging" environment in a stage.properties file and commits it to a version-control repository.
  • Modifies similar properties for the "production" environment in a prod.properties file and commits it to a version-control repository. Because dynamic properties (such as IP addresses) can change, operations must continually update these property files when such changes occur.

I've seen teams that have a hierarchy of properties: local, environment (usually, with at least four environment property files, but often many more), application, subsystem, and instance(s). When they had hundreds of instances, they had hundreds of different property files, making it virtually impossible to manage the constant change in IP addresses, directories, and other properties. This resulted in frequent environment and deployment failures that were often extremely difficult to troubleshoot because the challenge of change management — who changed what and when? — makes it hard to identify the root cause of the error.

Externalized properties

In a well-designed software system that uses the traditional approach, all properties that can differ across environments are externalized into property files. In other words, no setting that could be variable is hard-coded into the application code. But the more properties there are, the more susceptible you are to introducing deployment errors. These errors are costly and difficult to troubleshoot and often cause delay in the "build promotion" process in getting software delivered to users.

Listing 1 is a simple example of a property file:

Listing 1. Configuration defined in a properties file
jboss.home=/usr/local/jboss
jboss.server.hostname=jenkins.example.com
jboss.server.port=8080
jboss.server.name=default

In nontrivial systems, a property file such as this one usually has hundreds of properties. And those properties are usually duplicated across development, test, staging, and production environments — leading to hundreds or thousands of properties to manage when delivering software to production.

Listing 2 shows a Java™ code snippet that retrieves the values from a properties file:

Listing 2. Java code example getting properties
public PropertyReader(final String name, final ClassLoader loader) {
  try {
    InputStream is = loader.getResourceAsStream(convertName(name));
    if (is != null) {
      properties.load(is);
      is.close();
    } else {
      throw new IOException("Couldn't find property file: "
        + convertName(name));
      }
    } catch (IOException problem) {
        System.out.println("Property Reader: problem initializing");
  }
}

This is the most common approach that teams use in managing properties. But today's software delivery systems are capable of operating on an unprecedented scale. Thanks to hardware commoditization, virtualization, and cloud computing, you might be launching (and terminating) hundreds of environments when delivering and managing software systems. Your configuration-management approach must be able to scale commensurately. And (also thanks to hardware commoditization, virtualization, and cloud computing), it can.


How dynamic configuration works

The approach I advocate in this article is to set and retrieve properties dynamically. When you define an environment using infrastructure automation tools (see "Agile DevOps: Infrastructure automation"), you can set configuration items — such as database name, and file and directory names — in a configuration database. You can do this in scripts, because the entire infrastructure is built from the ground up in scripts that are committed to a version-control repository. Furthermore, in a cloud or virtual infrastructure, you can dynamically set and retrieve configuration items such as IP addresses and domain names, which are often statically defined in traditional infrastructures.

The code in Listing 3 is a Ruby script that loads configuration items into a SimpleDB (NoSQL) database:

Listing 3. Writing dynamic configuration items to a NoSQL database
AWS::SimpleDB.consistent_reads do
  domain = sdb.domains["stacks"]
  item = domain.items["#{opts[:itemname]}"]
  
  file.each_line do|line|
    key,value = line.split '='
    item.attributes.set(
      "#{key}" => "#{value}")
  end
end

Username and password

Capistrano is a Ruby-based domain-specific language (DSL) for defining scripted deployments. The code in Listing 4 is from a Capistrano script that defines the database username and password based on the parameter that was defined once by the user:

Listing 4. Setting the username and password in Capistrano
set :dataSourceUsername do
  item = sdb.domains["stacks"].items["wildtracks-config"]
  item.attributes['dataSourceUsername'].values[0].to_s.chomp
end
set :dataSourcePassword do
  item = sdb.domains["stacks"].items["wildtracks-config"]
  item.attributes['dataSourcePassword'].values[0].to_s.chomp
end

CloudFormation

CloudFormation from Amazon Web Services (AWS) is a template language for describing the provisioning of AWS resources. It's a DSL that uses JSON and a declarative approach similar to infrastructure automation tools such as Chef or Puppet. CloudFormation is best used for defining the provisioning of AWS resources, leaving the provisioning of other infrastructure resources to Chef or Puppet scripts. Although other major cloud providers do not currently provide a similar DSL for defining resources, you can often interface with those cloud providers' resources through their APIs.

Because you have complete control of the infrastructure through scripts, the database username and password can be automatically generated and stored in a secure configuration database.

This configuration is used throughout the rest of the software delivery system. The data pushed to this database is retrieved dynamically in upstream environments and used in downstream environments.

IP addresses

The code in Listing 5 is a portion of an AWS CloudFormation JSON script that dynamically sets and associates an external IP address:

Listing 5. Dynamically setting an IP address in CloudFormation
"IPAddress" : {
  "Type" : "AWS::EC2::EIP"
},

"IPAssociation" : {
  "Type" : "AWS::EC2::EIPAssociation",
  "Properties" : {
    "InstanceId" : { "Ref" : "WebServer" },
    "EIP" : { "Ref" : "IPAddress" }
  }
},

You would commit a script such as this one to a version-control repository, along with corresponding automated tests.

Domain name

The CloudFormation JSON code in Listing 6 uses the AWS Route 53 service to dynamically set the A domain record for an application:

Listing 6. Setting a domain in CloudFormation
"JenkinsDNS" : {
  "Type" : "AWS::Route53::RecordSetGroup",
  "Properties" : {
    "HostedZoneName" : { "Fn::Join" : [ "", [ {"Ref" : "HostedZone"}, "." ]]},
    "RecordSets" : [
    {
      "Name" : { "Fn::Join" : ["", [ { "Ref" : "ApplicationName" }, ".", \
      { "Ref" : "HostedZone" }, "." ]]},
      "Type" : "A",
      "TTL"  : "900",
      "ResourceRecords" : [ { "Ref" : "IPAddress" } ]
    }]
  }
},

The script makes it unnecessary to configure this item manually with the Domain Name Service (DNS) provider, which is the traditional method.

Database name

The code in Listing 7 is from a CloudFormation script that defines a parameter called DatabaseName:

Listing 7. Setting the database name in CloudFormation
"Parameters": {
  "DatabaseName": {
    "Description" : "The name of the database",
    "Type": "String"
  }
}

The DatabaseName parameter is passed to an infrastructure script (such as a Puppet manifest) when the script creates a new database for the application. The entire script is committed to a version-control repository, and you write automated tests to verify that the database is accessible.

Port

The code in Listing 8 sets the external ports for the environment (and application):

Listing 8. Setting port access within a security group in CloudFormation
"FrontendGroup" : {
  "Type" : "AWS::EC2::SecurityGroup",
  "Properties" : {
    "GroupDescription" : "Enable SSH and access to Apache and Tomcat",
    "SecurityGroupIngress" : [
      {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"},
      {"IpProtocol" : "tcp", "FromPort" : "8080", "ToPort" : "8080",\
       "CidrIp" : "0.0.0.0/0"},
	  {"IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", \
	    "CidrIp" : "0.0.0.0/0"}
    ]
  }      
},

Get involved

developerWorks Agile transformation provides news, discussions, and training to help you and your organization build a foundation on agile development principles.

The only ports with external access in this example are port 22 (for SSH access), 8080 (for Apache Tomcat), and port 80 (for Apache HTTP Server). The access can be further restricted — through Classless Inter-Domain Routing (CIDR) notation — to networks or even individual nodes (see Resources).


Dynamically speaking

In this article, you learned that configuration that is often statically defined in most software delivery systems can and should be dynamically defined so that you don't need to change it for every target environment. You saw that you can significantly reduce the number of configurable properties required for creating environments.

The next installment introduces an open source platform for continuous software delivery in the cloud. You'll learn the basic steps for running the platform, how to use the delivery pipeline, and how to run jobs to provision environments and run deployments.

Resources

Learn

Get products and technologies

  • IBM Tivoli® Provisioning Manager: Tivoli Provisioning Manager enables a dynamic infrastructure by automating the management of physical servers, virtual servers, software, storage, and networks.
  • IBM Tivoli System Automation for Multiplatforms: Tivoli System Automation for Multiplatforms provides high availability and automation for enterprise-wide applications and IT services.
  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.

Discuss

  • Get involved in the developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.
  • The developerWorks Agile transformation community provides news, discussions, and training to help you and your organization build a foundation on agile development principles.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Agile transformation on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Agile transformation, Open source
ArticleID=850649
ArticleTitle=Agile DevOps: Dynamic configuration
publish-date=12112012