DevOps: Breaking down silos beyond development and test
In my last blog post, I shared the 5 best practices my teams have learned during our own DevOps Journey:
1. Expand agile practices beyond development and test to include client, business stakeholders, and operations to breakdown silos and improve outcomes.
2. Shift Left with continuous testing using automation and virtualization to eliminate long back-end test cycles and increase quality.
3. Build a delivery pipeline leveraging tool
4. Experiment rapidly by delivering instrumented capabilities, which enable the team to make fact-based decisions and quickly evolve towards an optimal solution.
5. Create a culture of continuous impr
I'm going to share our experiences with the first practice, Expand agile practices beyond development and test to include client, business stakeholders, and operations to breakdown silos and improve outcomes.
Agile adoption is becoming more mainstream and just as with many large clients, my teams are seeing the benefits like reduced cost and better collaboration across the teams. We also seem to struggle with the same challenges. For example, my development and test teams are moving faster, but are we really working on the highest priority market requirements? And how fast can my clients get value from our new capabilities? The bottom line is even if development is moving faster, we don't have business impact until the entire end to end process is working as a well oiled machine.
Moving at a snail's pace
In 2010 when we were delivering our products on a yearly basis, we had long planning cycles. Our product management team would work together over a 3-6 month period. They would gather requirements from many sources, look at market trends, and bring forward a full list of content for development to commit to. This would happen every June and we'd deliver in one big bang release a year later. This approach was problematic in many ways. We were moving way too slow. Development could take up to a quarter to review and consider new requirements and market opportunities, and then another year to get them to market.
Even though we were following agile development practices, we were working in silos. We were not integrated across the business. We reviewed our priorities on a yearly basis and committed to releases which delivered a year later. Because the development team was fully booked, we had no ability to respond to changes in the market, or client requirements, or feedback. Collectively we needed to change. We needed a model where product management, key stakeholders and development all shared a joint understanding of business priorities. We needed the ability to leverage the agility in the development team to respond to change along with a new approach to making business commitments.
Getting in lock step
Like many clients, my development organization has several key stakeholders who can drive conflicting requirements, all of which are the highest priority and critical for the business. It was often these conflicts at the heart of why it took so long to make decisions. We recognized this challenge and as an executive leadership team decided that we needed a strategy. This strategy had to incorporate the market opportunities that we agreed to as a team. We asked the leaders of each market segment to step back and see the best interest of the entire business, not just their view of the world.
With this understanding, we collaborated and used IBM FocalPoint, a product in our portfolio, to document our Strategy, Business Goals, and Business Tactics. We provided direction on the percentage of investment across these Business Tactics. We then shared this with our executive teams and asked them to work together (across product management, development, design and support) to establish specific Investment Themes. These Investment Themes are the specific areas for investment over the upcoming year to meet our Business Goals. Each of the items that sit on our backlog, as well as new strategic opportunities, get categorized into these Investment Themes. We also include reducing Technical Debt as one of our Investment Themes. We prioritize and manage Technical Debt just like other investments. Our plans are now commitments to Investment Themes which we deliver on incrementally, each quarter. They are no longer a list of detailed work items committed to a year in advance.
We are able to move much faster because our business and development leaders are in lock step. They have transparency around priorities and know how our current backlog aligns with those priorities. This gives our leaders confidence in their decision making, thus the ability to make decisions faster. Because our work is clearly prioritized and understood in the context of what is important for our business in both the short and long term, this new Groomed Backlog allows us to make decisions on an ongoing basis when they used to take months.
Respond to change
Once we established this Groomed Backlog, it was critical to keep it realistic and aligned with the Agile approaches used in the development organization. Our team decided to create a Strategic Planning Committee (SPC) made up of Development Directors, Technical Development Leaders, and Product Management Directors to govern this process. The SPC owns decisions on conflicting investment priorities, the delivery cadence (which is now quarterly), along with the ongoing prioritization and re-prioritzation of the backlog. This cross discipline executive team enables us to balance business, development feasibility, client feedback and design first thinking as we move forward with making delivery plans for our releases. The details of the delivery plans are worked by the Project Management Committee (PMC). With the guidance of the SPC, the PMC works on the Epics, User Stories, and Work Items using our Collaborative Lifecycle Management tools. With the Investments Themes we now have full traceability of the Strategy, Business Goals, Business Tactics, Investment Themes down to the Epics, User Stories, work items, test cases and deployments that the development team is delivering.
Time to Development Improvements
In summary, we recognized that the time it took us from identifying a market opportunity we wanted to solve, and into the hands of the development team, was an inhibitor for us. We drew focus to the improvement areas described above by establishing a metric, Time to Development. By focusing on building a clear and transparent strategy, with established priorities and Investment Themes, the product management and development teams are now interlocked and quickly move into the development process vs debating over priorities and requirements. When we started, it took us up to 120 days to move from idea to the start of the development process; now with our ongoing collaboration and Groomed Backlog it is on ongoing process. As a result, we have been able to quickly deliver mobile support in our products and capture several significant client opportunities.
Since when is a party not fun?
One of the biggest challenges we had as we were driving to speed up our delivery process was the transition from development and test into production. There were inconsistencies between the environments in test and the environments in production. These inconsistencies made for lots of not-so-fun 'release parties'. You know the ones where you are confident that you are ready to 'go live' but then when you push, severity 1 problems come out of the woodwork. What should have been a celebration turns into a multi-hour firefight which quite often results in rolling back and delaying the delivery. These issues slowed us down and shined a light on the gaps between development, including test, and operations. We identified the need for better practices, automation, and consistency of our deployments across our development, test, and production environments.
To address these key gaps and bring the fun back to our parties, we implemented a few key practices.
Create DevOps feature teams: Feature teams are a key underpinning of an agile approach. These multi-discipline teams drive the Definition of Done and own the complete deliverable. To address the operational gap, we expanded the definition of our feature teams to include operations. Operations became part of the team from the beginning of the release, involved directly with the development and test teams. Development and test also became part of operations, as self hosted environments are a critical part of our process. Development, test, and operations were all jointly accountable across the entire release for the quality and deployment of the product into our test environments and production environments. As a result, each team grew a much stronger appreciation of what it takes to deploy the product and has a greater incentive to drive more automation across the testing and deployment of the product from the beginning. Additionally, the operations team got more appreciation for the challenges of being a developer.
Define Infrastructure as Code
Stakeholder and Client involvement
It never fails. Every time I speak with leaders of organizations undergoing an agile transformation, they always struggle with getting support from the 'business'. Finding stakeholders and/or end customers who are willing to provide feedback along the way and trust that this agile approach could yield better and faster results is a common challenge. Stakeholder feedback is a pre-req for Agile and devOps take that to the next level. We have to know that what we are delivering is meeting the needs of our users and the feedback needs to go directly to the feature teams that are creating the deliverable. To help us with client engagement we have implemented several best practices.
Code talks- charts walk: Time and time again, I find that teams are more successful when they demo early and often; constantly showing clients, stakeholders, executives, and each other what they are working on. It is a requirement to have an end of sprint demo for each feature team. Some teams choose to do them more often. To share them more broadly, we record them and provide opportunities for feedback. These demo sessions are priceless. They clarify that the team understood the user scenario and they help uncover any unspoken assumptions that cause problems later. I learned this lesson many times. Running code and demos early drives clarity within the team and with clients at a time when fixing the problems is much easier and less expensive. This also provides real time feedback to the feature teams so they can quickly respond.
Experiment often: As we shift more of our work to the cloud, experimentation on a broad level is much more possible. In addition to one on one interactions or traditional beta programs, we now have the ability to deliver new capability to a small subset of users, instrument it, and drive real feedback to the team. We did this recently with several new features on our IBM DevOps Services. As we experimented with different flows, we gained insights on areas which resulted in errors and were able to change them quickly. We also experimented with different approaches to drive users to our services and were able to understand which approaches yielded the most active projects vs those just interested in browsing.
The challenge my team faced as we adopted Agile practices was that we initially focused on breaking down the silos between development and test. While this was a good silo to break down, it did not involve operations, clients input was captured through defects and enhancement requests, and business planning was done in a vacuum and took too long. While, we achieved a good velocity with development and test, we found too many problems too late in the cycle. More importantly, we were not assured that we were client focused and solving the right market problems.
We learned that it is equally important to look across your entire software delivery lifecycle to identify delays between getting an idea to market and receiving client feedback. With the above changes, I can look at our dashboards weekly and have in depth information about our release plans along with expected revenue growth. Aligning the teams, from product management to development to operations and support, around that shared understanding of business priorities and plans is a valuable tool. When a market shift happens or something changes, the entire team can compare it against our business objectives and committed themes to see if we need to change direction. And if we need to change direction, we can make the decision transparently and quickly.
For related information directly from the team on the improvements in our planning process, please check out this Jazz.Net blog on Plan
Next time, I'll focus on shifting left to focus on quality earlier in the lifecycle.