Well, hello again, everyone! In case you haven't noticed, there's been a lot of activity in the POWER universe lately!
Power Firmware Updates
POWER systems (and the awesome firmware that runs them) are all over the news. We have Rackspace building a system, we've got some cool new scale out IBM servers here, and bunch of new software/firmware features released for our existing POWER8 systems. Not to mention a new supercomputer on it's way. We also have a team working on the next generation BMC for open compute!
All of this activity in power firmware made us sit back and reevaluate our work flows from an efficiency perspective. I've taken new role within our PowerFW group to really focus on our continuous delivery model, the goal being to get our firmware from development to test to the world much faster. There were 2 obvious areas to focus on initially, the process before a developer's change gets into an internal build, and the process after that build has been created and we want to get it to our customers (both internal and external). We quickly found that continuous delivery is very easy to talk about (automate everything, fail early, ...), but it's a bit more difficult in practice :)
The goal of CI is to test a developer's change before it can break anything else. CI was challenging for us because a lot of our code was stored in a legacy Source Control Management (SCM) system. It's also based on a legacy build environment. But, we rolled up the sleeves, created some cron magic, tossed some Jenkins in, and voila, we were able to pull a pretty awesome CI workflow together. It looks something like this at a high level:
The change (and any defined requisites) is extracted and built. A flash image is generated, which is then loaded into our simulation environment. By default, a base set of tests will be run in simulation but each developer has a mechanism to write their own custom CI tests using our internal, XML-based, Build Verification Test (BVT) suite. The CI infrastructure identifies the area of the development change and looks up any associated tests to run. After the simulation run passes, we have a Jenkins job, which allocates the requested POWER system on which the generated firmware image is flashed and then tested.
For those who haven't worked with Jenkins before, I HIGHLY recommend you take a look. It seems so simple, but it has so much power when it comes to orchestrating your automation needs. Going forward, all new products will be built on top of open-source tools and processes that enable CI out of the box.
CT is the testing we do after a collection of developer changes have been put into an official build. Historically we had a manual test team that would handle these tests. The explosion of systems with OpenPower, plus the general increase in velocity of our dev process made us dig into continuous test. There had been a mix of projects over the years from the team attempting to automate some of the testing but none had ever reached the continuous test level. As a team, we chose three (one for each subsystem, PowerVM, FSP/BMC, HMC) that looked the best suited for our requirements and started to focus on them from a development perspective. We went agile, with monthly sprints and associated user stories. We put all of the code in a git repo, plopped gerrit on top of it, and integrated a CI job using Jenkins (that's right, CI for the CT code!). We quickly had hundreds of test cases automated (tests that used to take weeks to run manually for each of the different releases) and running within daily Jenkins jobs. This had a snowballing affect in that our manual test team was then able to start contributing to the automation due to less manual testing!
We dealt with the legacy SCM as best we could but all new projects in our area now use the above formula as it's base. The OpenPower hostboot project did this from the beginning and reaped benefits, such as CI from the beginning. As a development and test team we have not found anything else that has the flexibility and start-to-delivery time capabilities as this.
And to leave you all with a little gift, here are my personal favorite jenkins plugins: