NBCUniversal

NBCUniversal sought to employ a DevOps approach to application development across its complex enterprise, comprising 17 business units, in order to improve code quality, streamline development and lower costs.

NBCUniversal utilizes IBM® UrbanCode™ software and IBM Cloud for Skytap Solutions to drive DevOps, combining continuous integration, delivery, testing, feedback and monitoring into one automated workflow.

Until recently there’s been a general perception that only the new generation of cloud-native organizations, those conceived as market disruptors and built for rapid innovation, have embarked on the DevOps journey. Not so. For every Etsy, Uber or Netflix there are scores of established enterprises, like NBCUniversal, seeking to consistently streamline time-to-market for new applications, increase application code quality and reduce development, testing and deployment costs through DevOps.

These corporations represent a cross-section of industries. And regardless of organizational and IT complexity they’re taking a DevOps approach because agility—the ability to accelerate innovation and respond faster to  changing business requirements—is vital.

The DevOps path that a large enterprise takes likely looks quite different than that of a born-on-the-cloud business, to be sure. Most significant are questions about how best to usher in widespread cultural change and transform long-standing processes and paradigms.

NBCUniversal has grown largely through acquisition and, at present, comprises 17 individual companies and business units, including its international footprint. Characteristic of most large organizations, its application environment is diverse and multi-speed—with a mix of on-premises, private and public cloud environments, with core business applications integrated with new web and mobile applications. The company’s DevOps team needed to bridge wide variability in speed, process, tools, skills and culture between System of Record (often mainframe) developers and System of Engagement (cloud-based, customer-facing applications) developers to achieve integrations across these systems.

John Comas, Manager of Platform DevOps at NBCUniversal, is responsible for the processes and tools underpinning the DevOps software development lifecycle (SDLC) at NBCUniversal. “We needed to provide our application customers with a single, standardized SDLC for a number of reasons: first, to reduce application licensing costs by eliminating multiple instances. Secondly, we wanted to provide a standard, on-premises enterprise-hardened system and migrate as many users over to it as possible, so we could standardize support and training. We also wanted to improve the quality of the code developed, and we wanted to develop it faster, more efficiently and at lower cost,” he says. “Our builds and deployments had high failure rates, and because our application teams did little or no testing there were releases that probably should never have gone into production, resulting in too many outages.”

Undertaking the DevOps journey became the impetus for building a workflow across four groups that were not historically well integrated—the Development, QA, Run and SDLC teams. The journey encompasses technology as well as mindset, speeding innovation by breaking down complex processes into the smallest units of work and running them in parallel whenever possible.

Comas and NBCUniversal’s Senior Director of Service Strategy and Collaboration, Charles Williams, positioned DevOps internally as a shared platform/framework for delivering innovation faster. The IBM UrbanCode platform—UrbanCode Build (application build automation software) and UrbanCode Deploy (application release automation software)—acts as the engine of DevOps, enabling NBCUniversal to combine continuous integration, continuous delivery, continuous testing, continuous feedback and continuous monitoring into one automated workflow, bridging process, culture and technology across the organization.

The UrbanCode platform supports a diverse set of IT environments and delivers comprehensive audit capabilities, which is critical to a complex enterprise, like NBCUniversal. “With the UrbanCode products we're able to provide build-as-a-service for the organization. I can reproduce any build that comes into UrbanCode at the click of a button,” says Comas. “We have a single, unified dashboard where I can see all of the activity. And we have clear records as to what was built, the binaries that relate to that build, what was done with it, how was it tested, what environment it was deployed to—all of that. Give me a date, and if you want to know what was deployed to production on that date I could tell you everything.”

NBCUniversal uses IBM Cloud for Skytap Solutions (ICSS), developed by IBM Business Partner Skytap, in conjunction with IBM UrbanCode Deploy software to automate test environment configuration and management throughout the software development lifecycle and to parallelize regression testing as part of QA automation.

In one case, NBCUniversal’s television group has used ICSS to reduce its regression testing from 6-8 weeks to as little as 3 hours for Compass, a large application used for TV program scheduling. Associated with Compass is an enormous regression test suite comprising 1,000 test cases. Previous to utilizing ICSS, the organization’s QA team would run these tests locally—each test run in series. Now they’ve set up an environment of 50 test machines, all pointing to the Compass QA environment on premise, with 20 tests running per machine in parallel.

Through DevOps, NBCUniversal has broken down what was previously a monolithic process into component parts, with Comas’s DevOps team taking the lead by providing the infrastructure that supports the DevOps tools, managing the tools themselves and managing the best practices in the use of those tools. The team also creates the automations for the development teams.

Advocating for DevOps wasn’t easy at first; not everyone was receptive, so Comas and Williams had some cultural hurdles to overcome. “John and I have been talking about DevOps for a while but it was kind of falling on deaf ears. It was when we started talking about capabilities and then showing capabilities to the app owners and then into the QA environment that we started to see traction,” Williams says. “At one point I think John and I even decided to stop using the word ‘DevOps’, because it didn’t mean anything to most people. Improved speed of delivery in application development; being able to run tests repeatedly and consistently and get results fast—that’s what’s important. When we spoke in those terms we started changing the culture. And when we had a couple of opportunities to demo our agile development capabilities we started hearing, ‘Oh, so this is what DevOps is. This is what I've been wanting.’”

In that sense, tools and technology, like the UrbanCode software, drove culture and process change. “I think the tooling helped fortify what the culture was shifting towards; more agile development, more shift-left testing,” says Williams. “The tools underpinned the DevOps process we envisioned, which is an end-to-end workflow.”
 

NBCUniversal’s television group in New York stands as an example of how the organization's DevOps journey has taken shape. That particular business unit is the television distribution arm of NBCUniversal and is, in part, responsible for scheduling on-air content for all NBCUniversal-owned stations. As noted previously, the Compass application is central to that task.

Comas and his team migrated development for Compass—a massive .net-based app—to the UrbanCode platform some 18 months ago. “I would describe the previous Compass process as like a ‘manu-matic’ process; they would commit their code to the repository, manually click the build button and then deploy it to an environment. Once they deployed it to an environment it was basically left to the QA team to handle,” he says. “So a week or so would go by and they'd get some results, do another build and send it back over to QA, and on and on. So the time-to-market for an application was very high—20 weeks to get a release out. One reason it took so long is that the QA team would do full-scale testing on a release that may or may not have met the business requirements.”

The Compass application team’s goal was to reduce that time by 75 percent, to five weeks. “Once we moved them to the UrbanCode platform they were able to hit that goal. In fact, we’re now looking to reduce time-to-market to two weeks,” Comas says. First, his team integrated QA automation with the Compass workflow, eliminating the need to manually execute numerous QA tests for every deployment. “We started with shift-left testing. We unit-tested the application release and did code coverage analysis on it, while also starting a small functional test where we would observe the different business requirements and create tests which would exercise those. So before the release would even get to the QA team to be fully functional and regression tested we could get immediate feedback if the release was good or not. That, in and of itself, accelerated things greatly.”

NBCUniversal’s DevOps approach also broke down silos in deployment. “Historically, releases could only take place on certain days of the week—typically Tuesday and Thursday evening—and if you missed that window you couldn't do a release until the following window,” Comas says. “So by working with the Compass Run team we can now do deployments to production much more frequently, resulting in smaller, more frequent releases. So they have much greater agility and can meet the business requirements much faster. In short, the code quality has improved, it's going faster and we're able to meet the business requirements much more rapidly.”

From a technology standpoint, Comas says the UrbanCode platform gives his team the ability to integrate a wide range of tools into one automated workflow, eliminating over-reliance on developers. “With UrbanCode, when a developer commits code without adequate unit tests, it's going to be flagged. If a developer didn't have that as part of the workflow you're not going to know that. There'd be no feedback into the process telling you the status of the application. Our Compass workflow is probably the most comprehensive automation we have, and the technology is helping us drive the application quality, improving build quality by an order of magnitude. Not only that, we’re delivering the changes the business wants much faster. Those are the ultimate goals; the reasons why we’re on this DevOps journey.”

For Williams and Comas, DevOps is about building trust and confidence and about instituting predictability, speed and quality in software delivery lifecycle. “We're providing a robust infrastructure guaranteeing a level of quality and a level of service to our Application teams,” says Comas. Indeed, the DevOps team is earning enough trust that business units are now asking them to help with larger projects. “Because we've shown that the capabilities are flexible and expandable and that they allow us to deliver on a very tight timeline we’ve seen a six-fold increase in project volume, from ten to more than 60 applications. We’re providing a path to application production that engenders a level of confidence we’ve never had before.”
 

NBCUniversal is a multinational, multimedia conglomerate headquartered in New York City. The company, with revenues of approximately USD30 billion, is involved primarily in the media and entertainment industries; among its most significant divisions are the NBC television network and the major film studio, Universal Studios. Through its Universal Parks & Resorts division it is also the third-largest operator of amusement parks in the world.