Introducing open source framework and tools built for compliance automation.

Over the past few years, IBM Cloudant has been engaging in more and more compliance-related work. This is a good thing; it gives customers confidence in the systems holding their data, and it gives us a way to critically evaluate and improve our product and processes.

A compliance accreditation is made up of controls. In some cases, lots of controls—all of which need to be consistently and continually executed, with evidence of successful (or otherwise) execution collected for audit. This is complicated by the size and dynamism of a cloud estate, especially for a service the size of IBM Cloudant.

Auditors need to see a lot of evidence data and can “sample” any device at any point within the audit period. As we got more involved in compliance activities, we quickly realised that we needed some automation to scale collection and verification of evidence if we were going to be successful in our compliance activities.

We have built an automation framework to address the problems that compliance at scale presents. It ensures a continuity of evidence, while also helping operationally manage compliance posture by verifying that evidence as it is collected.

The good news is that the framework and tools we’ve built for our compliance automation is now open source!

Introducing Auditree

From the Auditree homepage: “Auditree is an opinionated set of tools to enable the digital transformation of compliance activities. It is designed to be familiar to DevOps teams and software engineers, using tools they are likely already interacting with daily.”

What does that mean? Well, we’ve built a framework and set of associated tools that helps us turn compliance problems into engineering problems. Instead of clipboards and spreadsheets, we have unit tests and data in Git. Cloudant has been using this since 2017, and other teams in IBM have been adopting it since 2018.

It’s no secret that Cloudant uses Chef and a range of continuous integration/continuous delivery tools to manage our estate. We also write a lot of things in Python. We’ve used ideas from CI/CD and configuration management in Auditree.

Evidence is collected from APIs by “fetchers” and verified by “checks.” These are just decorated Python unit tests, so they are generally easy for developers to build, while also being flexible and capable. Checks produce reports, and their status (pass/fail/warn) can be sent to notifiers—Slack, issue trackers etc.

Those decorators manage writing/reading evidence in a “locker.” The locker is a Git repository with some additional metadata tracking provided by the framework. We chose Git because it is tamper evident, gives us history of data in an efficient manner, and, most importantly, familiar to engineers. If data hasn’t changed since the last execution, Git won’t update the file, hence we track metadata such as last retrieval time alongside the evidence. Evidence is given a TTL (time to live) in the locker and only retrieved if that has expired to avoid unnecessary and possibly expensive calls to the providers of evidence.

The framework is designed to fail and recover cleanly, with each execution being a fresh install of the framework and fetchers/checks. We run the set of fetchers/checks every eight hours. This is key for two reasons: 1) It allows us to build up a consistent and comprehensive body of evidence, and 2) it means that our global team has the opportunity to identify and remediate issues or deviations before the next run.

Ecosystem

While the execution of the framework is key—providing both operational oversight and a body of evidence—we’ve found there are other parts of the compliance puzzle that need specific tools to address.

Harvest

Harvest collates evidence (well, any file in Git) over time, which allows you to build and run reports and analysis over the contents of the locker, beyond what checks are doing. This is useful when answering a data request over a period of time or if you want to have periodic roll ups of data for reporting to the organisation.

Plant

While automated data collection is key to scaling compliance activities, there are some pieces of evidence—for example, a penetration test report—that may not be retrievable in this manner. Plant provides a way to manually and correctly place data in the locker so that it can be verified by checks.

Prune

One of the first checks we built was test_abandoned_evidence. This is important because it will catch valid data that hasn’t been updated for a long time—a possible sign of a misconfiguration somewhere. However, while this is important for detecting problems, is also creates an issue—if I no longer use a service, I won’t collect evidence from it, and this check will start to fire. I also can’t just remove the evidence files; they may still be relevant for future audits. We need a way to correctly manage the retirement of evidence, and that is Prune.

Over to you

Auditree is a set of building blocks to help engineers address compliance issues at scale. We’re open sourcing it because we think it is useful, but also to see how others are addressing the same problems. We have a library of fetchers and checks (and a fairly long backlog of things to bring into the open source domain) in Arboretum, and we would welcome contributions from the community there.