New demands for compliance are having a significant impact on companies that build, customize, or maintain software systems for regulated industries. First, more and more computer systems must comply with government regulations. Businesses that never had to consider compliance issues before now must follow mandates such as HIPAA (Health Insurance Portability and Accountability Act), the Sarbanes-Oxley Act, the FDA's 21 CFR 11 (Part 11 of Title 21 of the Food and Drug Administration's Code of Federal Regulations),¹ or other regulatory requirements. Even companies that have been developing compliant systems for years are finding that more of their internal systems now require compliance -- a trend that is accelerating as businesses seek to integrate systems and databases organization-wide. Today, a typical pharmaceutical company's systems may need to comply with HIPAA as well as regulations from the FDA, the EPA (Environmental Protection Agency), and the SEC (Securities and Exchange Commission).

A second major trend is that companies are beginning to rethink the approaches they have long used to design, build, and deploy regulated systems. They are taking a fresh look at available software development tools and leveraging more COTS (commercial off-the-shelf) packages. And to improve their development processes and product quality and also ensure compliance, they are adopting Good Systems Practices (GSP).

To gain some insight into how these trends are affecting software development efforts, Rational Edge writer Jack Wilber spoke with George J. Grigonis, Jr., senior consultant of QA Edge, Inc.; Matthew Magee, senior systems engineer for IBM Rational; and James Bradburn, principal consultant in the regulatory compliance practice at IBM life sciences. Although the discussion below centers on life sciences, these industry experts agreed that software development organizations serving any regulated industry face similar challenges and are adopting similar approaches and technologies.

Jack Wilber for The Rational Edge: Are development organizations in pharmaceuticals and other regulated industries approaching regulated systems validation differently today than they did in the past?

George Grigonis: Regulatory agencies like the FDA have always expected businesses and nonprofits to use current, sound science and technology practices in their regulated operations. That's why the regulations are largely descriptive rather than prescriptive. The FDA expects organizations to follow good practices, but they let them determine those practices, based on their individual business and manufacturing processes.

Back in the early 1980s, when companies started talking about computer systems validation, they were already validating manufacturing processes, sterile processes, and so on. As a result, computer validation was oriented toward manufacturing and formulation practices, which were predicated largely on "waterfall" engineering methods. They relied on document-centric activities to provide evidence for third parties such as the FDA. That way of thinking quickly became locked in stone and then fossilized. The concepts did not evolve along with uses of computing technology. Now, in the context of, say, utility computing,² the notions of IQ/OQ/PQ³ just don't fit. That doesn't mean you can't use utility computing in your technology base; it means you have to use a different process when assuring the computing service is fit for your specific business use. Many organizations are still trying to force-fit waterfall methods to COTS products, ASPs (application service providers), and so on, but document-centric thinking does not really apply to these technologies. This is in part what drives project costs up astronomically and causes failures; these organizations don't implement solutions that are tailored to their business, and they can't be agile and resilient.

It's time for regulated establishments to examine how they typically acquire and use computing technology and to change by applying current Good Systems Practices that ensure computing environments will be correct, reliable, and fit for their intended purposes. This was computer validation's goal from the beginning.

James Bradburn: Let's consider this subject at three levels. The first layer is FDA guidance, which has been oriented around process control systems and medical devices. Originally, the agency viewed computer systems as a form of equipment, and applied the IQ/OQ/ PQ structure and the waterfall development V-model⁴ approach to these systems. The second layer includes various industry standard organizations, whitepapers, and articles that constitute the industry's foundational background. This encompasses everything from the GAMP (Good Automated Manufacturing Practice) committee to the PDA (Parenteral Drug Association), IEEE (Institute for Electrical and Electronics Engineers), and OECD (Organization for Economic Cooperation and Development), among others. The third layer is the validation policies and procedures that companies build, based on their understanding of the first two layers. The old perspectives still show up in procedure after procedure, because people trained in the equipment-based and waterfall documentation methods can't let go. Unless you do validation a certain way, they think the sky will fall on you.

However, I think we are now seeing views change within the life sciences industry; organizations are moving away from the old, prescriptive, waterfall, V-model approach toward a more risk-based approach that focuses on quality and value. It is refreshing to hear the agency⁵ say that it is logical and acceptable to do risk valuation and have a risk-based decision-making process, and to allocate effort toward those ends. Accordingly, the industry is moving to a more analytical, risk-based approach. But it will take time for that approach to work its way through those three levels -- first across the length and breadth of the agency, and then throughout the industry.

Matthew Magee: I agree that it will take time to get a consistent approach in regulated environments. As I go from organization to organization, I frequently see stovepiped systems and groups, and their procedures and practices differ dramatically. In pharmaceuticals, for example, manufacturing practices are very different from IT system practices, although the general principles for good practices are the same.

Some organizations assemble a huge task force to evaluate all of their validation processes and then build a new approach from scratch. However, this is enormously expensive, and the resulting processes can get stale really quickly -- as evidenced by the condition the industry is in today. A more progressive approach may be to look for a commercial solution that automatically implements Good Systems Practices and is based on existing standards such as IEEE practices, TQM (Total Quality Management) practices, or the Software Engineering Institute's process model, the CMM⁶ (Capability Maturity Model).

JW: What impact would consistent Good Systems Practices have on regulated industries?

GG: It has always been good business practice to make sure your computer systems are correct, reliable, and fit for use. If organizations could impose these practices in every corner of the business, ranging from COTS-based systems engineering environments to procurement departments that specify, select, contract, and oversee computing services, we'd see dramatic quality and efficiency improvements. Unlike the linear waterfall processes for traditional computer validation, Good Systems Practices are not document- or project-driven. They are the institutionalized processes that a corporation uses to acquire, sustain, and operate computing bases. These processes are matched to the technologies they are designed for, and they are in lock-step with technology and IT concept evolution. The issue of whether you actually follow your process -- in other words, "say what you do and do what you say" -- is what most interests the FDA, and the evidence rests with the process tools and artifacts you use. With Good Systems Practices in place, we could stop wasting time on documents we think that the FDA might ask for but are otherwise useless. We could also make our organizations more agile and give ourselves the ability to adapt business processes to market needs. If you're not burdened with heavy documentation, then you can respond more quickly to new demands.

JB: Good Systems Practices is one of the main topics in the life sciences regulatory seminars that IBM conducts. We have to remember that the life sciences industry is pretty wide-ranging -- from global mega-pharmas down to bio-tech startups. Across that industry we see everything from companies with very detailed, conservative, and prescriptive procedures to those that haven't quite figured out what their procedures should be yet. We also see companies that overkill everything, so that validation has become a documentation exercise that does nothing to promote real system quality. These organizations are not only incurring unnecessary costs but also creating more compliance problems. The FDA may avoid dictating what to put in our procedures, but as George noted, whatever we put in them we have to follow. So the FDA often cites companies for not following their own procedures -- because they are too complicated, confusing, or excessively detailed to follow.

Many companies are aware that they are on either the short or long end of this spectrum, and they're seeking the middle. They are asking, "What is an optimum approach to getting quality systems -- one that avoids both regulatory exposure and compliance overkill?" Those who have already gone too far are recognizing the need to rethink, simplify, and reengineer, their processes. And those that realize they aren't doing enough are seeking optimal levels of cost and effort to maintain a validated system.

JW: Are these challenges unique to the life sciences industry?

JB: Our industry has system quality needs that are similar to those of others. The banking industry, for example, is just as concerned with security and data integrity as we are. And companies seeking to achieve or maintain ISO 9000 certification must demonstrate process quality in their operations, and that requires robust, reliable computer systems. Also, businesses in many industries are seeking to comply with Sarbanes-Oxley, whose provisions may focus on financial controls and reporting, but some of the actions required for compliance have ramifications for computer systems. In general, numerous regulations and requirements that pertain to business operations are generating a strong need for IT controls, and they affect how we must demonstrate data integrity, data protection, and data security.

MM: The pharmaceutical and medical devices industries are terrific examples because they represent the extreme end of regulatory necessity. People's health and well-being depend on system validity. But banking, finance, defense, and transportation must also meet certain regulations, and businesses in these industries are held accountable.

Say you are a financial institution, and you have a design flaw in a solution that provides online stock transactions or portfolio management. If the flaw is significant, and you are accountable, then you are in trouble. Company stockholders, shareholders, or investors in your company may turn around and sue you. And with Sarbanes-Oxley, you'll be forced to trace, investigate, and determine the problem's source. Without Good Systems Practices in place, this would be extremely difficult and very expensive.

JW: Some companies have one set of practices for developing regulated computer systems and another for non-regulated systems. Is this a valid strategy?

GG: It is true that many companies in our industry segregate systems. We perform one set of procedures for systems that are FDA-visible and another for those that are not. But in terms of quality, there should be no difference between regulated and non-regulated systems. And, in reality, many of these systems are integrated with one another. So separating system requirements -- doing this for the FDA, this for the SEC, and this for OSHA -- just doesn't make sense.

For example, if you have training records that are now part of HR (human resources), but used to be in another department, then you have to open the HR system to FDA inspection. So it makes sense to establish consistent Good Systems Practices across the business. They'll bring huge benefit to project managers, who will be able to institute logical, interdepartmental workflows. And the business overall will eliminate a great deal of risk.

MM: Yes, and the processes should be consistent regardless of where you are in the organization. Some of today's automated solutions can help you achieve that.

As George noted earlier, regulated industries often look to others to see what is happening, then take what they find and formulate their own solutions. However, as I said before, those solutions can become stale very quickly. But there are methodware solutions that recognize all the methodology tasks and also allow you to contour them to your specific needs. For example, IBM® Rational Unified Process,® or RUP,® is not specifically designed to address a particular regulatory industry's needs, but it offers various plug-ins and tools to help you do that. Rather than a prescriptive approach, you want Good Systems Practices -- and that is essentially what RUP embodies.

JB: One positive trend is that pharmaceutical companies that have traditionally treated non-regulated and regulated systems separately are rethinking this approach. As George mentioned, that kind of segregation is more and more difficult, because companies are using more integrated databases now. But a more compelling reason not to segregate is that validation is about quality, and that is just as important for a payroll system, which is crucial to your operation, as it is for a clinical or manufacturing control system.

Companies can use a single process to ensure reliability and sustainability for all systems; and for regulated systems they can add a very thin layer of documentation -- which they may ultimately decide to make universal. Then you don't have a 30 or 40 percent cost increase if you build a validated system with a documentation-intensive process. Instead, you're using the same efficient process for everything you build or maintain.

JW: It sounds like Good Systems Practices has a lot in common with industry best practices such as continuously ensuring quality. Is that correct?

MM: Good Systems Practices and quality are really the same thing. Organizations like the Project Management Institute⁷ say that in order to enable continuous improvement, you must try not merely to create products but also to improve your capacity to produce high-quality systems.

A process that leverages metrics can help. As a simple example, say a company is approaching the release date for a particular application. With a measurement process in place, the project manager can determine whether the closure rate on defects found during the QA cycle is going down or up. And he can use that simple metric to show an inspector that the team is tracking how well their software development activity is going. Metrics can also help managers and inspectors evaluate the system's stability and overall quality. And project managers can aggregate these metrics in a portfolio to evaluate the organization's consistency and productivity. You can leverage that collective information and the relationships among assets to continuously improve your organization's ability to send high-quality applications into production.

GG: I agree that you can benefit enormously by enabling TQM and metrics-based process improvement. QA and QC teams may have to retool so they can perform different roles in such an environment, but this approach would enable them to do everything they should be doing but don't have time for right now.

I can envision a day when the FDA will not have enough resources to pore over a company's documentation. So instead of asking, "Is this system validated?" they'll ask, "What is the technology process supporting this system? Show me the metrics." The FDA already talks about PAT (process analytical technology) and CAPA (corrective and preventative action) for the manufacturing process. It all relates to TQM; so if you apply TQM to building applications, all your processes will be in perfect alignment.

JB: We keep telling organizations that are going around in circles about compliance to step back and just focus on quality. That's what every agency really wants an industry to do. If you focus on quality processes first, then demonstrating compliance will be much easier.

There are situations in which FDA-regulated companies do not use the pharma-oriented IQ/OQ/PQ and/or V-model approach for their system quality methodology, favoring other cross-industry approaches instead. If your quality procedures reasonably assure that the system functions as intended, and you include controls to assure that it will keep meeting its requirements, then you have a demonstrable quality process.

Just remember that in a regulated environment, you need documented proof of these controls. In addition, to facilitate the inspection, it may help to know how your system quality processes map to the structure that the agency inspector is familiar with. This could aid his or her understanding of your quality approach.

JW: Let's talk more about waterfall versus iterative approaches. Do you think waterfall is on its way out in regulated development environments?

GG: I don't mean to knock waterfall. It has its place. For example, it might be suitable for a unique application that you can't buy off the shelf, one that is specifically tailored to your particular needs. But then you must also have enough time to follow a traditional engineering approach -- which is to lock down requirements before you go into design. This can only work when nothing is dynamic in your development lifecycle, and that is extremely rare. Once things begin to change, the waterfall approach tends to fall apart. For example, before you can complete a project using COTS components, there will likely be a product refresh, so change management is important from the very start of your project. Waterfall thinking doesn't take that into account; configuration management and change control typically don't enter the picture until the design phase.

Sometimes, when you talk about iterative development -- the need to revisit and reevaluate -- to people who are waterfall-centric, they just draw arrows to indicate a backward flow on their waterfall model. But that doesn't work; it just creates a series of disconnected whirlpools resulting in chaos, missed opportunities, lost time, and huge potential for failure to meet an important specification. Good Systems Practices for COTS products have the look and feel of iterative practices that facilitate simultaneous tradeoffs between the system's context of use, design and architecture work, and marketplace influences. You constantly have to go back and reevaluate things -- specifications, elements, testing -- and without an iterative approach and supporting tools, you cannot manage that kind of complex activity with SOP (standard operating procedure) workflow control.

MM: The focus of an iterative development approach is risk management. With a waterfall-based approach, you're not mitigating risk as you go. Waterfall gives you essentially one iteration, and you have limited ability to recover from mistakes. It's not suitable for today's projects, which must deal with rapid change. The new FDA guidance emphasizes risk assessment as a component of product development, and the objective is to focus on issues that present the greatest risk to people who will use the product.

In contrast to a waterfall approach, an iterative approach like RUP emphasizes risk reduction in every phase of the software development lifecycle. It also focuses on risks associated with package deployment and on the risks inherent in integrating COTS packages with existing systems before implementation begins.

JB: We're seeing more and more people adopt risk-based approaches, not only in the design process but also in defining how much testing is enough. Rather than trying to test for every possible thing with a long laundry list of tests, development teams are using a process to determine the systems' and internal modules' relative risk levels. They then use that information to determine what type of testing to do and at what level of detail.

Whether you use a series of waterfall steps or an iterative approach, the goal is still the same: quality in the results.

JW: What other issues should companies consider when implementing COTS packages in regulated systems?

GG: Using COTS products effectively is a matter of adopting good practices that help you evaluate, acquire, assemble, use, and service a computing environment largely based on marketplace products. It requires an awareness of what COTS products are, and SEI has a very good definition.⁸ But even more important is an awareness of what role marketplace dynamics play in the evolution of COTS products, because companies have little control over this. The market and the supplier's business needs drive product evolution; if the supplier is acquired by another company, the product direction may change.

I'd add that tools are absolutely necessary because the developmental, documentation, communication, change management, and quality assurance complexities associated with COTS-based systems are much too labor-intensive for manual oversight and control. Packaged application development is further complicated by open source components, which open the "black box" and make internals accessible and visible; and then you have all the supporting tools and information needed to manage and evolve the open source component. So instead of managing one or several black boxes, you have several black boxes and some white boxes. Tools can play a very important role here.

MM: It's also important to remember that companies in a regulated space influence vendors. When companies partner with vendors to create a particular solution, they can describe what they need and affect the vendor's product direction. For example, we have customer advisory boards that tell us what capabilities they'd like to see in our future products. We are not the only company that does this. Clients should be aware that their COTS products will likely have new releases periodically, and that's why they shouldn't depend on a rigid waterfall methodology. Carnegie Mellon's SEI has a terrific whitepaper by Cecilia Albert and Lisa Brownsword⁹ on how to select and deploy COTS packages, using RUP as a Good Systems Practices model.

If you are going to use a COTS application with an internally developed application, and both systems must comply with one or more regulations, then you must demonstrate a few things: first, that you used Good Systems Practices to select the package; second, that you can show traceability and linkages between assets to prove you are compliant; and third, that you used a common methodology to implement both the systems and the software used to integrate them. You may also need to show that you are partnering with the vendor to extend the COTS package.

JB: COTS application vendors serving the life sciences industry have been learning about computer system validation and are beginning to use testing tools for their own software quality assurance efforts. They're developing applications that have an accompanying set of testing materials to support their customers' compliancy efforts out of the box. Also, now many customers audit the vendors' quality processes and are not compelled to test the entire application themselves. This is another very positive trend.

JW: How can companies find out more about the quality processes behind COTS packages and software development tools they are considering for deployment?

GG: The PDA's Technical Report 32 (TR-32),¹⁰, ¹¹ an audit program, was established to provide software customers in regulated markets insight into what their suppliers do and how they do it.¹² With this information, a regulated company can better assess the risk of using a particular technology, not just from a performance perspective, but also from a project management perspective. If you are trying to make an agile response to a particular business need that includes a COTS product under development, but the vendor doesn't have good project management capability, then you are at risk. You may not get the product when you need it.

JW: George explained why development tools are so important when leveraging COTS applications. But how important are they to the overall process of building quality software and compliant systems?

JB: Automated tools have tremendous value in terms of ensuring quality and generating documentation for validation. They also help to keep systems in a continuous state of validation. Remember, every time you change something, you have to consider the change's impact and do regression testing. You simply cannot do these things thoroughly in a manual world. Technology gives you the ability to understand and test the effects of changes.

That companies are gradually incorporating these tools to lower costs and boost quality is another positive trend -- long overdue, but definitely happening.

GG: I agree with Jim. Today's environments are too complex for manual processes. Automated tools can track and connect artifacts and assets so you don't make a mess of development. Tools also direct workflow, so you have approval-based accountability all along the way. And people cannot jump in and do something they shouldn't do, because tools control data access, too. The FDA expects all of this control and oversight in a structured development environment. Tools are an ideal way to reduce SOP volume, minimize quality assurance labor, and capture and manage the information appropriate to the work being performed.

MM: IBM Rational integrates process and best practices right into its toolset to create a "tool-directed environment." Development teams can establish relationships between specific solution requirements and confirmation tests. And these tools provide all regulated industries thorough documentation that they can show to an inspector to prove they have a common process across their organization that encompasses Good Systems Practices. For example, instead of producing IQ/OQ/PQ tests to substantiate system validation, one of our clients simply runs IBM® Rational SoDA,® which reaches into each of the tool repositories and builds reports to demonstrate the traceability between assets: requirements to test, requirements to configuration management, requirements to change request management, and requirements to production code. Companies that use disparate tools from a variety of vendors would have a very hard time doing this. But this client has successfully walked through audits by showing these reports to substantiate compliance.

Of course, you have to educate people to bring them up to speed on Good Systems Practices. I am not trying to imply that the tools in and of themselves will implement Good Systems Practices, but once the team understands how to use the tools, they'll have a powerful head start on achieving good practices.

JW: What are the benefits of using a toolset that inherently supports -- and is supported by -- a proven methodology?

MM: Among other things, such a toolset can help organizations achieve consistency across all development activities. Development skill sets in many companies are based on old technology. These organizations need to move quickly to new tools and technologies when they are implementing COTS applications or trying to leverage new technologies such as J2EE or component-based development. Links in our RUP product connect the methodware and methodology to specific IBM Rational tools for creating or executing assets, defining requirements, creating models, and documenting change requests. These "tool mentors" educate practitioners new to a specific tool about iterative development and other Good Systems Practices. They tell the practitioner explicitly what to do -- what button to push, what setting to use.

This all goes hand in hand with compliance. You really cannot demonstrate compliance to an inspector unless you have:

• A consistent process in use within the organization -- ideally across all parts of it.
• The ability to document traceability among implementation, design, construction, and deployment assets.

In a tool-directed environment, traceability will happen automatically if you conform to the underlying methodology. And you can actually shorten your time-to-market or system delivery schedules, because you no longer have to establish traceability manually.

JW: What is a good first step for a company that is rethinking the way it ensures compliance? And where can someone go for more information on this topic?

JB: You can start by looking at various system quality methods and practices and deciding which ones might be appropriate for your organization. Specifically, you should consider using an integrated approach across the entire enterprise, adopting a risk-based approach, leveraging technology in your process, leveraging COTS application vendors' quality processes, and gaining a better understanding of the relationship between software quality assurance processes and business process compliance to FDA and other requirements.

GG: In my view, a first step would be to evaluate uniformity of practice across the enterprise for the various technologies you use. A goal is to eliminate segregation between regulated and non-regulated systems and processes -- there should be no difference. Then re-evaluate the disciplines responsible for computing support to make sure the right people are doing the right things. Finally, look outside the company for ways to standardize internal practices; and use Good Systems Practices. Institutionalizing such practices is really the trick.

MM: For more information on the issues we've been discussing, I can suggest a few resources. In addition to the Albert and Brownsword whitepaper I mentioned earlier, there is PDA's Web site at http://www.pda.org, and the Audit Repository Center at http://www.auditcenter.com. I have also written a whitepaper that describes possible strategies for the operation, use, and extension of IBM Rational ClearCase and IBM Rational ClearQuest to address electronic signatures and electronic records compliance, using 21 CFR Part 11 as an example. It can be found at http://www3.software.ibm.com/ibmdl/pub/software/rational/web/whitepapers/2003/germ.pdf. And finally, there is good information on IBM Rational's main compliance page: http://www-306.ibm.com/software/rational/solutions/etrans/compliance.html.

Bios

James Bradburn is a principal in the Life Sciences Division of IBM Corporation. He has been involved for more than thirty years in pharmaceutical and medical device manufacturing, including plant operations and production management. For the past fifteen years, he has focused on information technology supporting quality functions and regulatory compliance. A member of the life sciences regulatory compliance team, which is focused on FDA and EU regulations affecting pharmaceutical and medical device manufacturers, he specializes in Title 21 regulations that impact production and quality systems, including those that relate to computer systems. He has been a frequent speaker on manufacturing systems, 21 CFR 11 compliance, and computer validation at industry conferences and seminars for more than a decade.

George J. Grigonis, Jr., systems quality assurance and systems validation professional, has twenty-one years of experience in pharmaceutical industry process validation and computer validation methods for both regulated and non-regulated environments. Formerly he was affiliated with Merck & Co., Inc. as a manager and consultant in information services. He is presently a senior consultant for both QA Edge, Inc. and Cohasset Associates, Inc. An active member of the PDA, he has served on the committee for computer systems validation and as a leader for both PDA's supplier auditing and qualification task group, which developed the TR-32 program, and PDA's task group on Part 11, which developed the GERM guidance for electronic records management. He is also a member of IEEE Computer Society, an affiliate of the Software Engineering Institute at Carnegie Mellon University, and a member of Pharmaceutical Technology's editorial review board.

Matthew Magee joined Rational Software in 1998 and currently serves the pharmaceutical and regulated industries market for the IBM Rational brand. In addition to developing software for Wall Street and the defense and pharmaceutical industries, his accomplishments include several worldwide deployments of IBM Rational products, and the authoring of Rational's position paper on CFR 21 Part 11 (Electronic Records and Signatures). He holds a B.A. in information systems solutions from Rutgers University.

Back to top

Notes

¹ The Health Insurance Portability and Accountability Act (HIPAA) requires businesses to adopt standards for the security of electronically based health information, to be implemented by health plans, health care clearinghouses, and certain health care providers. The Sarbanes-Oxley Act regulates, among other things, acceptable conduct for public companies regarding retention of business records which to today consist largely of electronic-based information. 21 CFR Part 11, according to the FDA, applies to "records in electronic form that are created, modified, maintained, archived, retrieved, or transmitted under any recordkeeping requirement set forth in Agency regulations."

² For more on utility computing, see http://www-1.ibm.com/services/us/index.wss/rs/imc/al002842?cntxtId=a1000404

³ IQ/OQ/PQ, or Installation Qualification, Operational Qualification, and Performance Qualification, are protocols traditionally used for implementing compliant systems. These terms were derived from practices used to assure that the manufacturing equipment was installed properly and operates according to predetermined specifications and standards.

⁴ In the V-model software development approach, testing and development have equal weight, and each set of development activities is paired with a corresponding set of test and verification activities.

⁵ Unless noted otherwise, the "agency" refers to the FDA.

⁶ The latest development in CMM is the CMMI®. The SEI describes CMMI as, "The Capability Maturity Model® Integration (CMMI®) project is a collaborative effort to provide models for achieving product and process improvement. The primary focus of the project is to build tools to support improvement of processes used to develop and sustain systems and products. The output of the CMMI project is a suite of products, which provide an integrated approach across the enterprise for improving processes, while reducing the redundancy, complexity and cost resulting from the use of separate and multiple capability maturity models (CMM®s)." See http://www.sei.cmu.edu/cmmi/background/conops.html for more information.

⁷ http://www.pmi.org

⁸ From "An Activity Framework for COTS-Based Systems" at http://www.sei.cmu.edu/pub/documents/00.reports/pdf/00tr010.pdf

"A COTS product is a product: sold, leased, or licensed to the general public offered by a vendor trying to profit from it and supported and evolved by the vendor, who retains the intellectual property rights available in multiple, identical copies used without modification of the internals."

⁹ To download this paper, "Evolutionary Process for Integrating COTS-Based Systems (EPIC): An Overview -- Key Elements in Building, Fielding, and Supporting Commercial-off-the-Shelf (COTS) Based Solutions," go to: http://www.sei.cmu.edu/pub/documents/02.reports/pdf/02tr009.pdf

¹⁰ PDA Technical Report 32: "Auditing of Suppliers Providing Computer Products and Services Used for Regulated Pharmaceutical Operations." Supplement to Journal of Pharmaceutical Science and Technology, Vol. 53, No. 6, 1999.

¹¹ Case study: "Computer Supplier Evaluation Practices of the Parenteral Drug Association (PDA)." CMU/SEI-2003-TR-011, May 2003.

¹² To examine audits for specific software tools, including IBM® Rational ClearCase® and IBM® Rational ClearQuest,® visit http://www.auditcenter.com.

About the author

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Bios
• Notes
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into Rational on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers