Skip to main content

The story behind Notes/Domino Maintenance Releases, Fix Packs, and more

Scott M. Vrusho (scott_vrusho@us.ibm.com), Sr. Software Development Manager - Notes/Domino Maintenance Releases, IBM
Scott Vrusho, a native Bostonian, has been working at Lotus and IBM as an Engineer and Development Manager since 1987. Scott was a Notes technical advocate for IBM at the Nagano Japan and Sydney Australia Olympic games. Since 2002, Scott has been a manager in Notes development, where he has been responsible for Notes/Domino Maintenance, Serviceability and more recently the Security response team.

Summary: 

Date:  18 Jan 2009 (Published 24 Oct 2008)
Level:  Introductory
Activity:  5584 views
Comments:  

My name is Scott Vrusho, and I manage Notes/Domino Maintenance. I have been the Development Manager since April 2004, but I assisted for the two years prior to that.

During recent years, we have changed the way we create Maintenance Releases, Fix Packs, and so on. I'm going to take this opportunity to explain the way the process works, but first let me start off by giving a bit of history. In the early 5.x days, we had QMRs, which stood for Quarterly Maintenance Releases. As the name indicates, we shipped these every three months. The three months, though, didn't allot enough time for test, causing an increased regression rate. We dropped the "Q" and switched to Maintenance Releases, shipping every four months and spending the bulk of the extra time in test. That change occurred in 2001.

Shipping maintenance every four months worked well when feature releases were shipping every three years and there was only one current stream to support. With the more rapid onset of feature releases, and lengthier commitments for support of various codestreams, adjustments were made to do more frequent updates for the latest codestream and less frequent updates to the older codestreams. For example, seven months between 5.0.11 and 5.0.12 and another thirteen months before 5.0.13. It was around this time, mid-2002, that a four-, eight-, and twelve-month schedule was put in for the three most recent codestreams. So, 6.5x was every four months, 6.0x every eight months, and 5.x every twelve months.

Critical fixpacks were a delivery vehicle we used for a short period of time in 2003. These were hot-off-the-press fixes for severe issues. We discontinued critical fixpacks in their current incarnation, and we moved to fixpacks in 2005. More on this below...

In late 2004/early 2005, with increased concerns surrounding quality, we took a number of corrective actions. These included bolstering testing in certain areas, increasing interoperability testing with other products, stress testing simulations, and tactical Fix Packs for 6.5.2 and 6.5.3.

In early 2005, we found that the number of hotfixes being built was at an all-time high, however, a small percentage (roughly 5 percent) of the SPRs affected a broader set of customers. This rate was concerning because hotfixes receive limited testing, and re-packaging the same fix over and over is not a good use of our time. Customers were also telling us that they could not deploy maintenance releases as fast as we were putting them out. In fact, we would be lucky to have a server upgrade once a year, let alone a client upgrade every four months. We were shipping maintenance releases faster than customers could deploy them, and instead people wanted hotfixes on the older releases.

This brings us to our current Fix Pack and Maintenance Release strategy. The solution we implemented was to increase the time between Maintenance Releases and thus ship more frequent Server Fix Packs. The goal of the Server Fix Pack is to deliver a smaller number of safe fixes that help address the issues the broader customer base is experiencing or is likely to experience. Instead of getting the 500 or 1000 fixes that an MR delivers, the customer gets 20-50 focused fixes. The fix pack content is focused on bad regressions, crash, hang, security, and data loss issues, but we include them only if the fix is safe, has been deployed in a production environment, doesn't affect translation, and has no new features. Because we deliver a limited number of fixes in a smaller package, customers can deploy with less testing and less risk than they could with a bigger maintenance release. Customers can also spend the time they need to validate a maintenance release before deploying it. Instead of upgrading three times a year, they can now deploy a new MR once a year at most, and they can use Fix Packs to address the bulk of the more pervasive issues. You can read more about these choices in the "Notices" view in the Fix List database on the Web or on the What's New tab. There have also been multiple technotes outlining our strategy and providing more granular detail about the individual fix packs. For example:

But what about the client? The big elephant in the room is the client. It's an interesting problem. We encourage customers to deploy Server Fix Packs proactively so that they have fixes in place for problems before they run into them. The customer response to date, when asked if they would proactively deploy a Client Fix Pack (that, BTW, would work using smart upgrade), has been "No." Now, if the customer wanted or needed a fix for a specific issue, their answer, of course, is different. Admittedly to date, our focus has been on servers and server fix packs. Client Fix Packs are being introduced for the first time in late 2008 starting with 8.0.2 Fix Pack 1. Client Fix Packs are not planned for earlier releases. For earlier releases, we have another delivery mechanism called a CCH, or Cumulative Client Hotfix. A CCH is a hotfix for the client, but instead of just a single fix, we include a set of fixes that multiple customers are asking for. We do more pointed testing on a CCH than on a typical hotfix to ensure that it is a quality deliverable. We do not perform extensive regression testing on the CCH, but the same fix is merged into the next maintenance release where we do perform that level of testing. The CCH gives us an avenue to provide fixes sooner than a Maintenance Release or a Client Fix Pack. The one drawback is that the CCH must be obtained through support channels as it is not posted publicly. On a positive note, CCHs can be deployed using smart upgrade.

Finally, the schedule. You have heard me ramble on a lot in the preceding paragraphs, and I hope I have done a decent job of outlining the challenges and solutions. A sound byte indicating MRs will be every xx months is no longer possible, and, in fact, has not been possible since 2002. But we do post the delivery schedule of the upcoming MR(s) and Fix Pack(s) in the Fix List database on the Upcoming Releases tab.

I hope this information was valuable, and I welcome any feedback.


About the author

Scott Vrusho, a native Bostonian, has been working at Lotus and IBM as an Engineer and Development Manager since 1987. Scott was a Notes technical advocate for IBM at the Nagano Japan and Sydney Australia Olympic games. Since 2002, Scott has been a manager in Notes development, where he has been responsible for Notes/Domino Maintenance, Serviceability and more recently the Security response team.

Comments



Trademarks  |  My developerWorks terms and conditions

Help: Update or add to My dW interests

What's this?

This little timesaver lets you update your My developerWorks profile with just one click! The general subject of this content (AIX and UNIX, Information Management, Lotus, Rational, Tivoli, WebSphere, Java, Linux, Open source, SOA and Web services, Web development, or XML) will be added to the interests section of your profile, if it's not there already. You only need to be logged in to My developerWorks.

And what's the point of adding your interests to your profile? That's how you find other users with the same interests as yours, and see what they're reading and contributing to the community. Your interests also help us recommend relevant developerWorks content to you.

View your My developerWorks profile

Return from help

Help: Remove from My dW interests

What's this?

Removing this interest does not alter your profile, but rather removes this piece of content from a list of all content for which you've indicated interest. In a future enhancement to My developerWorks, you'll be able to see a record of that content.

View your My developerWorks profile

Return from help

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Lotus
ArticleID=365104
ArticleTitle=The story behind Notes/Domino Maintenance Releases, Fix Packs, and more
publish-date=01182009
author1-email=scott_vrusho@us.ibm.com
author1-email-cc=

Table of contents

My developerWorks community

Tags

Help
Use the search field to find all types of content in My developerWorks with that tag.

Use the slider bar to see more or fewer tags.

Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).

My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).

Use the search field to find all types of content in My developerWorks with that tag. Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere). My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).

Lotusphere knows the secret to a successful collaboration event: You.

Special offers