Contents


Enable plug-and-play service discovery with Consul and Docker

Dynamically inject infrastructure awareness into your distributed apps

Comments

Most large cloud deployments today rely on external services: databases, caching, and third-party APIs. Reaching these critical resources can be a challenge in environments where virtual machines are ephemeral components. In this tutorial, you will learn about a drop-in solution that exposes infrastructure knowledge to an application runtime. In addition, you will see how to design a versatile framework to monitor complex server topology for better orchestration of microservices.

My non-intrusive solution to service discovery provides an essential tool for DevOps. It's immediately actionable and doesn't lock us behind frameworks.

Background

Modern app deployment often involves microservices at scale. For example, rather than creating a single monolithic application, you can split an application into single-purpose units that collaborate with one another. In this way, you get modular development with a separation of concerns and horizontal scaling for free.

Is it really for free? Not quite. We still have the hassle of orchestrating all of those moving parts of the infrastructure. And the wide adoption of Docker, the container engine, contributes to the challenge. Although Docker unlocks a workflow to develop, ship, and run programs, many developers hit a wall when considering multi-host deployment or old-school problems such as log management.

The challenge

A typical web application today involves a front end of varying complexity, a back end, a database, and quite often third-party services as well. All of these technologies communicate over the network, and we can take advantage of that fact: The back end is deployed where resources are available, and a database shard spins up nodes for performance considerations. Meanwhile, the whole setup dynamically evolves across the cluster to handle the load.

But how can the back end find the database URL in this changing cloud topology? You need to design a process that gives applications an up-to-date knowledge of the infrastructure.

Introducing Consul

Consul is described on GitHub as "a tool for service discovery, monitoring, and configuration." Consul is one of the open-source projects developed by HashiCorp, the creator of Vagrant. It offers a distributed, highly available system to register services, store shared configuration, and keep an accurate view of multiple data centers. In addition, it's distributed as a simple Go binary, which makes it trivial to deploy.

To make the steps easy to follow (and consistent with our topic), we are going to use Docker.

Once Docker is installed, and thanks to progrium (also known as Jeff Lindsay), the one-liner in Listing 1 is enough to bootstrap a Consul server.

Listing 1. Bootstrap a Consul server
docker run --detach --name consul --hostname consul-server-1 progrium/consul 
-server -bootstrap -ui-dir /ui

# Get  container ip for further interactions
CONSUL_IP=$(docker inspect -f '{{ .NetworkSettings.IPAddress }}' consul)

# The container also runs a web UI at $CONSUL_IP:8500/ui

Note: While the official documentation recommends that you spin up at least three servers to handle failure cases, those considerations are beyond the scope of this tutorial.

You can now query your infrastructure and discover one service: Consul itself (see Listing 2).

Listing 2. Discover the Consul service
curl $CONSUL_IP:8500/v1/catalog/services
{"consul": []}

# we can fetch more details about a specific service
curl $CONSUL_IP:8500/v1/catalog/service/consul
[{"Node":"consul-server-1","Address":"172.17.0.1","ServiceID":"consul",
"ServiceName":"consul","ServiceTags":[],"ServiceAddress":"",
"ServicePort":8300}]

As you can see, Consul stores important facts about services. It covers information and tags, the fundamental data for programmatically accessing remote services.

Declarative services

Let's now look at the role of registration, external services, and Docker in our solution. To illustrate, let's imagine a modern application that stores data in MongoDB and sends emails through Mailgun. The latter is an external service, while we will run the former by ourselves. Read on to see how we can handle both cases.

Registration

In order to expose those valuable properties, you first need to register the service. You will run a Consul agent on each node of your cluster, which is responsible for joining a Consul server, exposing the node's service, and performing a health check (see Listing 3).

Listing 3. Register the service
# download and install the latest version
wget https://dl.bintray.com/mitchellh/consul/0.5.2_linux_amd64.zip -O 
/tmp/consul.zip
cd /usr/local/bin && unzip /tmp/consul.zip

# create state and configuration directories
mkdir -p {/srv/consul,/etc/consul.d}

# check that everything worked
consul --help

With more than 10 million downloads, MongoDB is a popular choice as a document database. Let's use it and save the following file in /etc/consul.d/mongo.json (see Listing 4).

Listing 4. Use MongoDB as a database
{
    "service": {
        "name": "mongo",
        "tags": [
            "database",
            "nosql"
        ],
         "port": 27017,
         "check": {
             "name": "status",
             "script": "mongo --eval 'printjson(rs.status())'",
             "interval": "30s"
         }
     }
}

The syntax offers a concise, readable, and declarative way of defining service properties and your health check. You can pick up those files in a version control system and immediately identify an application's components. The file above declares a service named "mongo" on port 27017. The check section gives the Consul agent a script that tests whether the node is healthy or not. Indeed, when requesting the server for service requirements, you need to be sure that it returns reliable endpoints.

All that remains is starting the actual Mongo server and the local Consul agent (see Listing 5).

Listing 5. Start the Mongo server and the local Consul agent
# launch mongodb server on default port 27017
mongod

# launch local agent
consul agent \
    -join $CONSUL_HOST \  # explicitly provide how to reach the server
    -data-dir /data/consul \  # internal state storage
    -config-dir /etc/consul.d  # configuration directory where services and checks 
    are expected to be defined

Did it work? Let's query the Consul HTTP API (see Listing 6).

Listing 6. Query Consul HTTP API
# fetch infrastructure overview
curl $CONSUL_IP:8500/v1/catalog/nodes
[{"Node":"consul-server-1","Address":"172.17.0.1"},{"Node":"mongo-1","Address"
:"172.17.0.2"}]

# consul correctly registered mongo service
curl $CONSUL_IP:8500/v1/catalog/service/mongo
[{
    "Node": "mongo-1",
    "Address": "172.17.0.2",
    "ServiceID": "mongo",
    "ServiceName": "mongo",
    "ServiceTags": ["database", "no-sql"],
    "ServiceAddress": "",
    "ServicePort": 27017
}]

# it also exposes health state
curl $CONSUL_IP:8500/v1/health/service/mongo
[{
    "Node": {
        "Node":"mongo-1",
    },
    "Service": {
        "ID": "mongo",
        "Service": "mongo",
        "Tags": ["database","no-sql"],
        "Address": "",
    },
    "Checks":[{
        "Node": "mongo-1",
        "CheckID": "service:mongo",
        "Name": "Service 'mongo' check",
        "Status": "passing",
        "Notes": "",
        "Output": "MongoDB shell version: 3.0.3\nconnecting to: test\n{ \"ok\" : 0, 
    \"errmsg\" : \"not running with --replSet\", \"code\" : 76 }\n",
        "ServiceID": "mongo",
        "ServiceName": "mongo"
    },{
        "Node": "mongo-1",
        "CheckID": "serfHealth",
        "Status": "passing",
        "Notes": "",
        "Output": "Agent alive and reachable",
        "ServiceID": "",
        "ServiceName": ""
    }]
}]

Given a Consul agent or server address, any piece of code in the cluster that's capable of HTTP requests is now able to consume that information. Shortly, I will explain how to process it all, but before that, let's see how to register services that are out of our control, and, as a bonus, how to automate the steps above with Docker.

External services

To avoid reinventing the wheel, it's a good idea to integrate third-party services into your application. But in this case, you can't start a Consul agent on the appropriate node. Once again, Consul has you covered (see Listing 7).

Listing 7. Querying Consul HTTP API
# manually register mailgun service through the HTTP API
curl -X PUT -d \
    '{"Datacenter": "dc1", "Node": "mailgun", "Address": "http://www.mailgun.com",
 "Service": {"Service": "email", "Port": 80}, "Check": {"Name": "mailgun api", 
 "http": "www.status.mailgun.com", "interval": "360s", "timeout": "1s"}}' \
    http://$CONSUL_IP:8500/v1/catalog/register

# looks like we're all good !
curl $CONSUL_IP:8500/v1/catalog/services
{"consul":[],"email":[],"mongo":["database","nosql"]}

Because Mailgun is a web service, you use the HTTP field to check API availability. To dive deeper into Consul superpowers, refer to its comprehensive documentation.

Docker integration

So far, a Go binary, a single JSON file, and a few HTTP requests have enabled the service discovery workflow. You are not tied to a particular technology, but as mentioned earlier, this agile setup is especially suitable for microservices.

In this context, Docker lets you package services into a reproducible, self-registering container. Given your existing mongo.json, all it takes is the Dockerfile and Procfile in Listing 8.

Listing 8. Package services into a reproducible self-registering container
# Dockerfile
# start from official mongo image
FROM mongo:3.0

RUN apt-get update && apt-get install -y unzip

# install consul agent
ADD https://dl.bintray.com/mitchellh/consul/0.5.2_linux_amd64.zip /tmp/consul.zip
RUN cd /bin && \
    unzip /tmp/consul.zip&& \
    chmod +x /bin/consul && \
    mkdir -p {/data/consul,/etc/consul.d} && \
    rm /tmp/consul.zip

# copy service and check definition, as we wrote them earlier
ADD mongo.json /etc/consul.d/mongo.json

# Install goreman - foreman clone written in Go language
ADD https://github.com/mattn/goreman/releases/download/v0.0.6
/goreman_linux_amd64.tar.gz /tmp/goreman.tar.gz
RUN tar -xvzf /tmp/goreman.tar.gz -C /usr/local/bin --strip-components 1 && \
    rm -r  /tmp/goreman*

# copy startup script
ADD Procfile /root/Procfile

# launch both mongo server and consul agent
ENTRYPOINT ["goreman"]
CMD ["-f", "/root/Procfile", "start"]

Dockerfiles let us define a single command to run when booting up containers. However, we need now to run both MongoDB and Consul. Goreman let us achieve just that. It reads a configuration file named Procfile, defining multiple processes to manage (lifecycle, environment, logs, etc.). Such an approach in the container world is a debate on its own, and other solutions exist, but for now it does the job in a simple manner.

Listing 9. Procfile
# Procfile
database: mongod
consul: consul agent -join $CONSUL_HOST -data-dir /data/consul -config-dir
/etc/consul.d
Listing 10. Shell commands to build the container
ls
Dockerfile  mongo.json  Procfile

docker build -t article/mongo .
# ...

docker run --detach --name docker-mongo \
    --hostname docker-mongo-2 \  # if not explicitly configured, consul agent 
set its name to the node hostname
    --env CONSUL_HOST=$CONSUL_IP article/mongo

curl $CONSUL_IP:8500/v1/catalog/nodes
[
    {
        "Node": "consul-server-1",
        "Address": "172.17.0.1"
    }, {
        "Node": "docker-mongo-2",
        "Address": "172.17.0.3"
    }, {
        "Node": "mailgun",
        "Address": "http://www.mailgun.com"
    }, {
        "Node": "mongo-1",
        "Address": "172.17.0.2"
    }
]

Awesome! Having Docker and service discovery working together makes you look good!

We can fetch more details by querying $CONSUL_IP:8500/v1/catalog/service/mongo as in Listing 6, and find the service port. Consul exposes the container IP as the service address. This approach works as long as the container exposes the port, even if Docker mapped it to a random value on the host. On multi-host topologies, however, you will need to explicitly map the container's port to the same on the host. To avoid this limitation, consider Weave.

To sum up, here's how you can expose services information throughout several data centers:

  1. Launch at least one Consul server and store its address.
  2. On each node:
    1. Download the Consul binary.
    2. Write service and check definitions in the Consul configuration directory.
    3. Launch the application.
    4. Launch the Consul agent with the address of another agent or server.

Create infrastructure-aware applications

You have now built a convenient and non-intrusive workflow to deploy and register new services. The next logical step is to export this knowledge to dependent applications.

The Twelve-Factor App, a methodology for building software-as-a-service apps, makes a serious case for storing configurations in the environment:

  • Maintain strict separation of the configuration from changing code.
  • Avoid having sensitive information checked into repositories.
  • Keep the language and operating system agnostic.

Now it's time to write a wrapper capable of querying a Consul endpoint for available services, export their connection properties into the environment, and execute the given command. Choosing the Go language gives you a potential cross-platform binary (like the other tools, so far) and access to the official client API (see Listing 11).

Listing 11. Package services into a reproducible self-registering container
package main

import (
    "strconv"
    "strings"
    "flag"
    "log"
    "os"
    "os/exec"
    "fmt"

    "github.com/hashicorp/consul/api"
)

// critical quits on errors with a debug message
func critical(err error) {
    if err != nil {
        log.Printf("error: %v", err)
        os.Exit(1)
    }
}

// inject exports properties into runtime environment
func inject(properties map[string]string) []string {
    // read current process environment
    processEnv := os.Environ()
    // allocate and copy it
    env := make([]string, len(processEnv), len(properties) + len(processEnv))
    copy(env, processEnv)

    for k, v := range properties {
        // format key/value mapping as exec.Command and system style (i.e. KEY=VALUE)
        env = append(env, fmt.Sprintf("%s=%s", k, v))
    }
    return env
}

// discoverServices queries Consul for services data
func discoverServices(addr string, healthyOnly bool) map[string]string {
    servicesEnv := make(map[string]string)
    // initialize consul api client
    consulConf := api.DefaultConfig()
    consulConf.Address = addr
    client, err := api.NewClient(consulConf)
    critical(err)

    // retrieve full list of services throughout our infrastructure
    services, _, err := client.Catalog().Services(&api.QueryOptions{})
    critical(err)
    for name, _ := range services {
        // query healthy services information
        servicesData, _, err := client.Health().Service(name, "", healthyOnly, 
&api.QueryOptions{})
        critical(err)
        // loop over this category of service
        for _, entry := range servicesData {
            // store connection information like environment variables : {"MONGO_HOST":
"172.17.0.5"}
            id := strings.ToUpper(entry.Service.ID)
            servicesEnv[id + "_HOST"] = entry.Node.Address
            servicesEnv[id + "_PORT"] = strconv.Itoa(entry.Service.Port)
        }
    }
    return servicesEnv
}

func main() {
  flag.Parse()
  // keep it consistent and read consul service address from environment
  consulAddress = os.Getenv("CONSUL")
  command = flag.Args()

  log.Printf("inspecting infrastructure")
  services := discoverServices(consulAddress, true)
  env := inject(services)

  log.Printf("running `%s`", strings.Join(command, " "))
  cmd := exec.Command(command[0], command[1:]...)
  cmd.Stdout = os.Stdout
  cmd.Stderr = os.Stderr
  cmd.Env = env

  critical(cmd.Start())
  critical(cmd.Wait())
}

The next command, in Listing 12, compiles this prototype and validates its behavior.

Listing 12. Compile and validate the prototype
# install the single dependency
go get github.com/hashicorp/consul
# compile to `wrapper` (depends on your directory name)
go build ./...

export CONSUL=$CONSUL_IP:8500
./wrapper env

The last command should print something like MONGO_PORT=27017, among other variables. Any command should now be able to read services data from its environment.

Reconfigure the infrastructure dynamically

A situation you are still likely to face is challenging your current implementation. A web app could start like the one above and successfully connect to MongoDB, and bad things could still happen on database failures or migrations. What you want is to dynamically update the application's knowledge when the infrastructure is experiencing either normal or unexpected changes.

While designing a robust solution to this problem might require a tutorial on its own, the Consul Template takes an interesting approach.

Consul Template queries a Consul instance and updates any number of specified templates on the file system. As an added bonus, Consul Template can execute arbitrary commands when a template update completes. Therefore, you can use Consul Template to monitor services (addresses and health) and automatically restart the application whenever a change is detected. Because your wrapper will fetch services data, the runtime environment will mirror the correct state of the infrastructure (see Listing 13).

Listing 13. Use Consul Template to monitor services and restart the application
consul-template \
    -consul $CONSUL \
    -wait 1s  \  # Avoid re-running multiple times on changes
    -template "app.ctmpl:/tmp/app.conf:./wrapper env"

You now enjoy all the benefits of a templated configuration file. Listing 14 is an example adapted from hackathon-starter on GitHub.

Listing 14. Example of a templated configuration file
// app.ctmpl

// store third-party service information in the environment
db: 'mongodb://' + process.env.MONGO_HOST + ':' + process.env.MONGO_PORT + '/test',

// or you can leverage consul-template built-in service discovery
{{ range service "mongo" }}
      db2: 'mongodb://{{ .Address }}:{{ .Port }}/test',
{{ end}}

// Use consul-template to fetch information from consul kv store
// curl -X PUT "http://$CONSUL/v1/kv/hackathon/mailgun_user" -d "xavier"
mailgun: {
    user: '{{ key "hackathon/mailgun_user" }}',
    password: '{{ key "hackathon/mailgun_password" }}'
}

This experience requires more thought. It could be tricky, for example, to restart the application to update its knowledge of services. We could instead send it a specific signal to give it a chance to handle the changes gracefully. However, this requires us to get into the application's code base, and until now, it didn't need to be aware of anything. Moreover, the rise of microservices on fallible cloud providers should encourage us to run stateless, failure-resilient apps.

Nevertheless, the combination of powerful tools with clear contracts allows you to integrate distributed applications into complex infrastructures, without limiting you to a particular provider or application stack.

Conclusion

Service discovery, and more broadly services orchestration, is one of the most exciting challenges of modern development. Big players, along with the developer community, are stepping in and pushing technologies and ideas further.

IBM Cloud, for example, addresses this challenge with workload scheduler, smart databases, monitoring, cost management, data synchronization, REST API, and more. Only a handful of tools can enable developers to focus solely on the loosely coupled modules of their application.

Thanks to Consul and Go, you can to take a step in this direction and build a set of services featuring:

  • Self-registration
  • Self-update
  • Stack agnosticism
  • Drop-in deployment
  • Container friendliness

This tutorial has introduced the basics of a production deployment and shown how a plug-and-play approach to service discovery frees you up to think about the other parts of a modern deployment pipeline without all the usual constraints. Further steps could include extending our wrapper with encryption and offering a consistent integration to safely expose credentials such as service tokens. I hope this tutorial has given you ideas for coping with the challenges of ultra-agile cloud deployment.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Cloud computing
ArticleID=1017284
ArticleTitle=Enable plug-and-play service discovery with Consul and Docker
publish-date=10132015