Change IT - Don't Break IT
thartric 120000G744 Comments (3) Visits (4930)
Change control -- this is a process that is vital to the success of any high traffic web site such as with the developerWorks site. The reward for handling changes correctly is that many people don't even notice, however when they do it usually means something didn't go right and now you're into managing a problem.
At developerWorks, we have several controls in place for handling change control. On the design and development side, developers first code and test the fix on a departmental server before promoting it to a test environment. The code change is checked into Rational ClearCase, our source control system. If things go well in the QA and load testing of a fix or enhancement, then it is scheduled for promotion to production. If the change will result in a site outage, those are scheduled for the weekend, otherwise the change can occur on a weekday. A web team member with access to the production servers, separate from the development team, is then scheduled to deploy the change into production. The code files, deployment instructions, complete with documented procedures for deploying the change, test scenarios to execute that confirm it's working properly, and backout steps in case of failure are provided. Hopefully with the right preparation, the backout steps never get used.
It doesn't always work as expected, but at developerWorks we've fortunately seen less than a 5% failure rate with our production deployments.
What questions or best practices do you have on change management?