A Behavior Driven Developer's guide to Infrastructure as Code

Applying BDD to server provisioning using Ansible


In the early 2000s, Kent Beck proclaimed "following these two simple rules can lead us to work much more closely to our potential: Write a failing automated test before you write any code, and remove duplication." Dan North later expanded on this guidance with Behavior Driven Development (BDD).

Dan tried to solve the problem with Test Driven Development (TDD) where developers needed to "know where to start, what to test and what not to test, how much to test in one go, what to call their tests, and how to understand why a test fails." He did so by creating a scenario (or acceptance criteria) template for analyzing a user story and providing a method for executing it (JBehave). This turns requirements (user stories) into living documents driving the behavior of your application. This article expands upon that to use those same techniques to drive Infrastructure as code (IaC) to provision a server for your application using Ansible.

Stories to scenarios

A user story is a chunk of functionality (some people use the word feature) that is of value to the customer.

Martin Fowler and Kent Beck, Planning Extreme Programming (Addison-Wesley Professional, 2000)

Beyond the INVEST (Independent, Negotiable, Valuable, Estimable, Small, Testable) criteria, a user story should be written in a way that is closed, meaning that it should finish "with the achievement of a meaningful goal" (Mike Cohn, "User Stories Applied: For Agile Software Development") to the user. A typical format to use for capturing this is the Connextra user-story template:

As a {role}
I want {goal/desire}
so that {benefit}

From these stories, the scenarios or acceptance criteria can be generated. Often, user stories are the "happy path" through the application. Now that path and its alternatives need to be explored. Dan North uses the given-when-then template for scenarios:

Given some initial context (the givens),
When an event occurs,
then ensure some outcomes.

A popular language based on this template is Gherkin. Once captured, it can become executable through a framework like Cucumber or JBehave.

IT scenarios

Any technical infrastructure must be built in conjunction with the stories and must be developed to support what the stories need.

Martin Fowler & Kent Beck, Planning Extreme Programming (Addison-Wesley Professional, 2000)

Typically, basic infrastructure tasks related to a project are done during the first iteration, called Zero Feature Release (ZFR, or ziffer). Sometimes referred to as a skinny, it is just enough infrastructure to support the application and its development. This infrastructure is normally defined as constraints on the system such as "the application should use couchdb for the database" or "the development team will use Jenkins for continuous integrations." While not all constraints should be expressed with the user story template, it is helpful to understand the source and reason for the constraint later on.

Security stories

Desktop, mobile, or game console end users might not specifically ask that the server application run on the latest version of Apache web server. However, comments that result in that acceptance criteria can appear in user interviews or questionnaires. Loss of user data can be excruciating for the user. When Sony's PlayStation Network was hacked, it left users and politicians asking about the lack of server patching and firewall. Listening to the impact of a security breach, loss of data, downtime, or other IT related events generates scenarios for your infrastructure.

Listing 1. Initial user story
As a user of your service
I want my credit card information to be secured
So that I don't have to cancel and reorder my credit card.

Now the problem with this story is that it is not closed and difficult to elaborate into scenarios. One popular method is to elaborate it with evil user stories. These are elaborated stories that are from the point of view of an evil actor.

Listing 2. Hacker evil use case 1
As a hacker 
I want to port scan the server
So that I can see if vulnerable less secure services are running.
Listing 3. Hacker evil use case 2
As a hacker
I want to leverage vulnerabilities in "out of date" packages
So that I can gain access to the system

These evil user stories can now be elaborated into scenarios which prevent them like:

Listing 4. Hacker evil use case 1: Scenario 1
Given the server has a firewall installed
When a port scan is performed
Then only ssl port 443 is open

A feature of the Gherkin language is the concept of a scenario outline, which applies the scenario to a data set.

Listing 5. Hacker evil use case 2: Scenario outline 1
Scenario outline: Expect security updates to be installed
   Given the server is Ubuntu 14.04
   And the package <package>
   When the version is fetched
   Then It should be equal or later than version <version>

   Examples: Ubuntu 14.04 nginx packages with security updates
     | package      | version          |
     | nginx        | 1.4.6-1ubuntu3.7 |
     | nginx-common | 1.4.6-1ubuntu3.7 |
     | nginx-core   | 1.4.6-1ubuntu3.7 |

Creating a provision project

There are many IaC frameworks available, from Chef to SaltStack. This article leverages the Python-based Ansible. You can install it by running pip install ansible from the command line.

Unfortunately, Ansible on its own doesn’t provide any unit test or linting capability. The command ansible-galaxy init <playbook name> creates a skeleton role project but only provides a mechanism for manual testing. Tools like ServerSpec help but are not enough. Molecule provides additional software development best practices for Ansible role development.

Molecule provides the following basic workflow:

  1. Verify the syntax of the role (ansible-lint).
  2. Create the virtual images used for testing via a Vagrant wrapper. Alternatively, it also supports Docker and OpenStack.
  3. Converge runs the Ansible playbook to provision the image.
  4. Run the role again to verify it doesn’t change anything (Idempotence).
  5. Verify by linting your test and running them.

The integration of linting into the deployment script development process is welcomed. Leveraging ansible-lint, flake8, and rubocop helps keep software somewhat to best practices when collaborating. While it is possible to accomplish all this without Molecule, it is nice to have something with these tools baked in.

Idempotence helps ensure that the script only makes a change that needs to be changed. Running a script twice should not change anything. This sounds trivial but in practice is quite difficult. Leveraging a command or shell task to perform external actions requires additional logic to prevent it from being run twice.

To install Molecule, run the following:

                    pip install molecule

To create a skeleton role project using Molecule:

ansible-galaxy init ansible-role-dw-bdd-example
cd ansible-role-dw-bdd-example
molecule init
mkdir features

The features directory is for storing the GHERKIN feature files. This article uses the following security feature example:

Feature: Security

Story: User's confidential data
As a user of your service
I want methods for my credit card information to be stolen blocked
So that I don't have to cancel and reorder my credit card.

Evil Story: MyDoom Trojan
As a hacker
I want to infect a server with MyDoom
So that I can use it as a socks proxy to gain access to a system

Evil Story: Old nginx packages
As a hacker
I want to leverage vulnerabilities in out of date packages
So that I can gain access to the system

Scenario: Socks proxy is blocked
  Given the server has a firewall installed
  When a list of open ports is fetched
  Then the socks port 1080 is not open

Scenario Outline: Expect security updates to be installed
   Given the server is Ubuntu 14.04
   And the package <package> is installed
   When the version is fetched
   Then It should be equal or later than version <version>

   Examples: Ubuntu 14.04 nginx packages with security updates
     | package      | version          |
     | nginx        | 1.4.6-1ubuntu3.7 |
     | nginx-common | 1.4.6-1ubuntu3.7 |
     | nginx-core   | 1.4.6-1ubuntu3.7 |

Writing the steps

Step definitions are the mechanism of executing the GHERKIN syntax. While GHERKIN is programming language agnostic, the step definitions are written in Python, Ruby, JavaScript, and so on. Each step can take one or more arguments that can be parsed out of the GERKIN statements. Each statement in your scenario will map to the method in your step definitions. There are different Cucumber implementations for each programming language.

By default, Molecule uses the Python-based TestInfra test framework. The easiest way to integrate Cucumber with Molecule without using a separate runner is TestInfra with pytest-bdd. The reason for this is both TestInfra and pytest-bdd are both extensions of the pytest framework. Solutions like behave and other Cucumber implementations require a bit more work to get the same integration. That being said, there is still some work required.

To make TestInfra work with Molecule, the host object using the Connection API needs to be generated. Because Molecule generates an inventory on the fly, add this to the beginning of the script ("test/"):

import testinfra

host = testinfra.get_host(

From pytest-bdd, you need the given, when, then, and scenarios packages:

from pytest_bdd import (

For each scenario that uses a scenario outline, add an example converter:

 'Expect security updates to be installed',
 example_converters=dict(package=str, version=str))
def test_package_scenario():
 scenarios with tables that require type mapping must be referenced
 directly before calling "scenarios()"

After all the example converters has been added calling “scenarios('../features’)” to pull in the remaining scenarios. The pytest-bdd code using the TestInfra host object.

@given('the package <package> is installed')
def package_is_installed(package):
    assert host.package(package).is_installed
    return dict(package=package)

@given('the server is Ubuntu 14.04')
def the_server_is_ubuntu_1404():
    """the server is Ubuntu 14.04."""
    assert host.system_info.type == 'linux'
    assert host.system_info.distribution == 'ubuntu'
    assert host.system_info.release == '14.04'

@when('the server is running')
def the_nginx_server_is_running():
    """the ngingx server is running."""
    assert host.service('nginx').is_running

@when('the version is fetched')
def the_version_is_fetched(package_is_installed):
    """the version is fetched."""
    version = host.package(package_is_installed['package']).version
    package_is_installed['version'] = version

@given('the server has a firewall installed')
def the_server_has_a_firewall_installed():
    """the server has a firewall installed."""
    assert host.package('ufw').is_installed

@when('a list of open ports is fetched')
def a_list_of_open_ports_is_fetched():
    """a list of open ports is fetched."""
    return host.socket.get_listening_sockets()

@then(parsers.parse('the socks port {port:d} is not open'))
def the_socks_port_1080_is_not_open(a_port_scan_is_performed, port):
    """the socks port 1080 is not open."""
    url = 'tcp://' % port
    assert url not in a_port_scan_is_performed

@then('It should be equal or later than version <version>')
def it_should_be_equal_or_later_than_version_version(package_is_installed,
    """It should be equal or later than version <version>."""
    assert package_is_installed['version'] == version

Passing the tests and responding to change

Running molecule test should fail because the Ansible script hasn't been written yet. As Ansible tasks are written, more tests should pass until all have passed. At this point, the Ansible roll is complete. Here is a sample set of Ansible tasks that pass the tests:

- name: Install nginx server
    name: nginx
- name: Block all ports
    state: enabled
    policy: reject
    log: yes
- name: Allow ssh
    rule: allow
    name: OpenSSH
- name: Allow 443
    rule: allow
    port: 443

The key benefit of BDD is that whenever a test fails, it means that you have a defect or your documentation is up to date. For instance, updating the feature documentation with a new security vulnerability means that your documentation is up to date because the feature files are the single source of truth. If that test fails, then it is a defect in your server configuration that needs to be changed.

Downloadable resources

Related topics


Sign in or register to add and subscribe to comments.

Zone=DevOps, Open source
ArticleTitle=A Behavior Driven Developer's guide to Infrastructure as Code