Migrate data from Rational DOORS to Rational DOORS Next Generation
Planning and implementing a DOORS migration project
Perhaps the most fundamental task for successful product and systems development is gathering requirements and the subsequent creation of a product specification that articulates a shared vision of a final product. The gathered requirements specify all criteria from function to performance. They document known constraints that help to avoid missteps that can get progressively more expensive the longer they remain undetected. The development cycle for any modern project feeds on its intermediate results from iteration to iteration and phase to phase. There is no substitute for cogent requirements to provide focus and clarity to the ongoing project.
Requirements captured in the traditional static document have shortcomings:
- Documents quickly become stale.
- Documents lack a mechanism for detecting and addressing staleness, which puts the project at progressively increasing risk from day one.
- Documents do not scale, so collaboration among many stakeholders is clumsy and error-prone.
Requirements gathering applications like IBM Rational DOORS have dramatically enhanced collaboration and reporting for individual specifications and can scale to large teams. However, there is the burden of thick client installation and maintenance. There is also the lack of a larger ecosystem for common administration, traceability and reporting. This limits the ultimate potential of such first-generation client-server requirements management applications.
IBM Rational Collaborative Lifecycle Management Solution (CLM) is a second-generation suite that combines the requirements management application IBM Rational DOORS Next Generation, the software configuration and process management application IBM Rational Team Concert™, and the quality management application IBM Rational Quality Manager into a tightly integrated eco-system within which traceability linking is second nature.
With dashboards and the IBM Rational Publishing Engine, it is possible to pull information from the entire system in real time for reports that show the status of the project and the impact of changes or issues anywhere in the system. CLM also addresses total cost of ownership with a browser-based user interface while offering collaboration features such as review, visual editing, document generation, reporting, audit history and a common administration interface.
Thus, long-time users of first-generation requirements management applications can turn to second-generation tool suites like CLM to solve team development issues brought on by globalization and other pressures of scale or geographical distribution.
Migration from first-generation requirements applications and information architectures and methods into the CLM suite is a big topic that is outside the scope of this article. The rest of this article concentrates on the migration of requirements and the associated information architecture. CLM issues are mentioned where appropriate.
Terms and definitions
This article presumes that a project exists to create a release of a component, product, system or feature. That list is incomplete, but the intent is obvious. To avoid the clumsy use of any or all of those terms whenever referenced, the remainder of the article uses the term product to represent a new release of any of those options.
A requirement is, at its core, a statement defining a feature or behavior that is mandatory for the new release of the product and is thus defined using the term shall. For example "The moon unit shall include a breathing apparatus in order for life to be sustained." A strongly desired feature is generally specified using should and a nice to have feature using may.
A set of such requirements and any elucidating information is captured within a specification. For example, in both DOORS and DOORS Next Generation a specification is represented by a module artifact that itself groups a set of artifacts.
The most commonly used terms when working with DOORS and DOORS Next Generation are:
- In DOORS, a project is a specialized folder that can contain sub-projects, folders, modules and artifacts. In DOORS Next Generation, a project area is a self-contained area for a team, or specific project, for security purposes. Members of a project area can view any artifact within the project area. Non-members won't see anything.
- A folder can contain projects (DOORS only), subfolders, modules and artifacts. Note that a project that is migrated from DOORS becomes a folder in DOORS Next Generation.
- A specification consisting of a collection of artifacts. A module is itself also an artifact. A module has order and hierarchy.
- A group of artifacts (DOORS Next Generation only.) This is not a containment relationship and it is not a module. A collection has no order or hierarchy.
- An individual requirement or other information such as text, a picture, a table or some other object. In the context of a module, it can be made visible or invisible by a filter. In DOORS, an artifact exists alone in a folder, or it can exist in a module. In DOORS Next Generation, an artifact exists in a folder and can be reused in any number of modules and collections. Thus, DOORS Next Generation supports the concept of shared artifacts. Note that an artifact can contain embedded artifacts, thus offering another form of hierarchy.
- A field in an artifact, akin to a field in a data record or an attribute in a classifier. Displayed as a column in the grid view. Can be made visible or invisible by a view.
- The data type applied to any artifact or attribute. In DOORS, types are local to individual modules, and are copied when the same types and shapes (number and name of attributes) are to be used across multiple modules. In DOORS Next Generation, types are defined globally in a project area and entire type systems are copied between project areas when reuse or standardization is required. Both DOORS and DOORS Next Generation support templates that populate their type systems. DOORS Next Generation, for example, has Systems and Agile requirements templates among others.
- Shorthand for an internal DOORS link, this is a pointer from one artifact to another artifact inside the same module, in another module or between core artifacts without involving modules. These exist in both DOORS and DOORS Next Generation, although core artifact linking is supported in DOORS Next Generation only.
- OSLC Link
- A pointer to a resource in an external OSLC-enabled application such as Rational Quality Manager, Rational Team Concert, DOORS, DOORS Next Generation, etc.
- External Link
- Sometimes called a vanilla external link, this is a URI or equivalent pointing to a resource on another platform or application.
Information architecture and working methods migration
Years of requirements methodology and data evolution represent a significant investment in an information architecture, a systems and tooling architecture, and working methods. Migration is therefore not just about moving data from one database to another. Cost and risk are in fact concentrated more in the organizational transformation than by the act of moving data.
Organizational transformation to a second-generation collaborative suite of tools include: work flow and methods; tooling skills and education; and information architecture elements such as module structures, requirements shape and attributes, data type systems, project organization, folder hierarchies, data and reporting tools (formats and templates), and so on.
Change at any time carries risks such as confusion and loss of inertia. But, migration to a second-generation collaborative lifecycle management environment with the latest in requirements tooling is the perfect time to fix known issues in the information architecture, working methods, work flow and collaboration.
A key analysis result will therefore be an information design and architecture that is carried forward into active work in DOORS Next Generation. New projects in DOORS Next Generation depend on this level of planning while migrated active projects from DOORS benefit further from the alignment of the data models in both databases and conformance to the updated information architecture going forward.
Requirements data and methods life cycle
The requirements data life cycle interacts with requirements migration in three identifiable phases:
- DOORS: The stage during which the original DOORS data is created and actively used. This applies generically to any first-generation requirements tooling.
- TRANSITION: A potentially very long stage during which requirements data is migrated incrementally until there is no more active data in DOORS. This can be years or even decades in an environment where active projects have long life-spans.
- DOORS Next Generation: The stage at which the DOORS database is a historical archive and all current and future work takes place in DOORS Next Generation/CLM.
Note that the TRANSITION stage can be organized so that there are no significant interruptions or disruptions to active teams and projects. It is strongly advised to avoid rushing to the DOORS Next Generation stage and risking damage to active projects along the way. The overall timeline for a DOORS to DOORS NG migration (possibly as part of a larger migration project to CLM) might look like Figure 1.
Figure 1. TRANSITION timeline for migration from DOORS to DOORS Next Generation
A mature DOORS database has experienced a ramp up from its inception to whatever the current state of requirements data might be at the point of migration. Often, after years or even decades, some of the data might be obsolete or has achieved a state of confusion or entanglement (sometimes referred to as spaghetti requirements) that is undesirable for current or new projects. Therefore, it is good practice to maximize efficiency and clarity by migrating only the body of work that is relevant to ongoing and future development.
DOORS data is best moved incrementally, for example project by project or even partial projects as one increment. The whole process is smoother when an incremental migration sequence is embraced from the beginning. The process is nondestructive and DOORS can successfully migrate:
- Folder hierarchies
- OLE objects
The data in a migration package (for example, a whole or partial DOORS project) is maintained in DOORS on the day before its migration and is maintained in DOORS Next Generation on the day after its migration. The migrated data is set to read-only in DOORS and is referenced via backlinks from DOORS Next Generation. This allows the original project in DOORS to be visible from the new version of the project in DOORS Next Generation. You can navigate to historical data and baselines by following the backlink. The backlink opens the thick client or the DOORS Web Client, depending on the local client configuration.
A requirements data migration project
There are four phases to a requirements migration project:
- Analysis: Information architecture, tooling and method migration planning. Includes Estimation of effort and cost for deployment, training, education and the design of a new information architecture. Also includes risk and cost assessment of the shape, size and hygiene of the existing DOORS data, identification of stale data to leave behind and active data to migrate. The end result is a plan for incremental migration into the new information architecture with training and education. The details of data migration include a series of migration packages and the preferred order. Analysis is performed once at the beginning of the overall migration project.
- Preparation: Cleaning and normalization of data, removal or modification of attribute definitions and harmonization of data types. The end result is data that is normalized and compatible with the target project area in DOORS Next Generation. Preparation is performed once per incremental migration package.
- Execution: Export of DOORS data and import into DOORS Next Generation. End result is locked data in DOORS and active data in DOORS Next Generation with links back to the DOORS historical versions. Execution is performed once per incremental migration package.
- Post processing: Optional continued work on types in DOORS Next Generation including harmonization of attribute and artifact types. Administration of users, permissions, team areas and so on. Post processing (if required) is performed once per incremental migration, but can linger as the data and methods in the new information architecture mature.
Figure 2. Typical incremental TRANSITION project timeline
As mentioned in the introduction, migration is best performed as an incremental process. Each increment contains the three phases: preparation, execution and post processing. And, as with any incremental process, the increments (iterations, sprints) are to some extent self-contained and overlapping. This means that each migration increment can run in parallel with another, depending entirely on the available resources. They can also be run serially in order to minimize risk or to accommodate significant overlap in shared types. Concurrency of preparation in DOORS with post processing in DOORS Next Generation is a risk-free way to shorten the time between increments if desired.
The diagram shows that over time, the proportion of archival projects in the doors database grows until it is total with the TRANSITION phase containing as many overlapping incremental migrations as are necessary to move the selected requirements projects into DOORS Next Generation.
Generally, the demarcation from DOORS to TRANSITION is the moment when the analysis begins. The demarcation from TRANSITION to DOORS NG is the moment when there are no more planned migration increments.
The analysis phase is performed once as a precursor to the actual incremental migrations that make up the bulk of the TRANSITION phase. A thorough analysis of existing designs and data and a detailed plan for design migration and incremental project or partial project migrations will make all the difference to future success in DOORS Next Generation.
The overall size of the migration is determined (which projects to move, and when each is to be moved) and the overall effort estimated. To assist with planning, DOORS has integrated utilities that assess projects to determine the number and size of objects, links, OLE Objects, attributes and so on. A risk assessment is performed on the initial plan and data assessment in order to expose potential impacts to the project time line based on resource availability etc. The viability of shifting requirements interchange from DOORS to DOORS Next Generation is assessed and planned as discussed in the sections that follow.
Finally, a target DOORS Next Generation project area is chosen for each incremental migration package. For new project areas, a folder hierarchy and module structure are defined. For existing project areas, the target folder location for the migrated data is chosen or created. Team areas, if needed, are planned and starting in release 6 of DOORS Next Generation, a stream structure can be planned for each project or team area.
Migration utilities document the condition of the DOORS data and type system. DOORS types are module-scoped, so subtle variations tend to creep in over time. These should be identified, and if not intentional (for example, product or regional variations) they must be addressed before migration takes place. This harmonization and consolidation process avoids datatype pollution while exploiting the rich type system support in DOORS Next Generation. DOORS migration support can automatically generate a type system based on the names of attributes, which can save a great deal of the labor of type system mapping. But the data must be clean and normalized for the feature to be effective.
For an existing DOORS Next Generation type system, harmonization is performed if necessary and the data types are aligned to the target type system using their URIs. As mentioned previously, the automated DOORS migration tooling produces a consistent type system across increments, so manual alignment of type systems is required only when migration is intended into a project area that was not initially populated by DOORS migration output.
The DOORS migration metrics utility is used to perform harmonization and consolidation of types. Reports of attributes and enumerations are examined and those that are similar or vary only by case of name must be identified and classified as intentional or in need of harmonization. This achieves a common set of types and eliminates duplication and clutter inside the DOORS Next Generation type system. Once the type system is ready for migration, the increment proceeds in a straightforward manner using the migration support in DOORS, which takes care of type system consistency in DOORS Next Generation, back links, and conversion of data to formats compatible with DOORS Next Generation.
DOORS exports a migration package that carries all data and metadata to enable DOORS Next Generation to maintain a high fidelity representation of the data as it was when it left DOORS. That package is imported into DOORS Next Generation in the same manner as any REQIF package would be.
Migrated data contains links back to the source data in DOORS providing direct navigational access for review of baselines and historical data. Both the DOORS client and the DOORS web client can be used to review the DOORS-based data. Thus, the process incrementally allows teams to work in DOORS Next Generation and continue to have access to non-migrated projects and the historical data for migrated projects in DOORS.
Post processing is an optional phase that occurs if the type system in DOORS Next Generation requires some tweaking, or if the initial team and user set up was not completed during the analysis or preparation phases. After the post processing phase finishes, the team takes ownership of the requirements data inside DOORS Next Generation and all active work is performed there.
Requirements data interchange
Requirements data interchange is a topic in and of itself, but is often confused with Requirements data migration and so is worthy of a brief discussion.
It is common for requirements to be exchanged between original equipment manufacturers and suppliers or partners to specify a component or part to be sourced. To reach an agreed final specification, a cycle of review and modification is required. A set of specific requirements might be repeatedly sent back and forth and loaded into the requirements management tool used at each end of the protocol This process is called round tripping. There are, of course other related scenarios. For example, a specification might be sent to a supplier and then updated at regular intervals. This scenario is called interchange with update and is characterized by requirements traveling in one direction repeatedly without updates in the other direction.
It is immediately obvious that each end of the protocol must support a common requirements interchange format and dialect for this to work. The term dialect is used here to mean the semantics that each tool applies to specific data items within the exchanged package. If they do not match, subtle errors can creep into the agreed specification. DOORS has been around for decades and supports several interchange formats including REQIF, RIF, DOORS eXchange format, CSV (comma-separated values), and DPA (DOORS Project Archive) and DMA (DOORS Module Archive.) DOORS Next Generation, on the other hand, supports only REQIF as of release 5.0.2, as REQIF is the current global standard for requirements interchange.
Requirements data interchange uses round tripping formats such as REQIF, RIF and DOORS eXchange between DOORS databases, and with REQIF between DOORS and DOORS Next Generation databases. CSV round trip support ships in DOORS Next Generation 6.0.1 and Microsoft Word round trip support is under consideration.
As a high level project, migration from one tool to another (for example, DOORS to DOORS Next Generation) can affect requirements interchange. DOORS Next Generation must interchange data using REQIF at this time, and CSV in 6.0.1. This could be a perceived barrier to entry where existing relationships use a different format for the interchange.
However, because DOORS should be retained for historical purposes the option exists to continue these interchanges in DOORS using a format other than REQIF. This would continue until both ends of the interchange agree on another format for round tripping. A specification can be periodically updated from DOORS to DOORS Next Generation with REQIF and linked into the rest of CLM. This extra step does add complexity to the migration and to the interchange and should only be explored if the interchange specification must be linked into the rest of CLM before the exchanged specification has been finalized by agreement. If the specification under review can remain on its own until finalized, then there is no need for a periodic update to DOORS Next Generation and the interchange can continue as it always has in DOORS for the foreseeable future.
The bottom line is that requirements migration is not requirements interchange and vice versa. They have different goals and success criteria and while each iteration of migration has a relatively short life span, interchange can go on indefinitely.
IBM provides a migration framework that supports an incremental process to spread both cost and risk over time. Data migration and architectural and systems transformation both benefit from the staging of planning and execution phases. Incremental data migration further serves to minimize the impact on active projects as compared with a traditional "big bang" product migration.
The goal of migration is to leverage features in DOORS Next Generation while positioning new and migrated data to exploit features in CLM. Existing relationships with suppliers and partners are supported by the persistence of DOORS as a historical record so that the disruptive potential of such a migration is mitigated.
Planning and execution, as with any project in any field, are the keys to successful adoption of this next generation of requirements tools.
- Visit the DOORS Next Generation Learning Center explore DOORS NG with your peers and experts.
- Learn more about DOORS Next Generation on IBM.com or jazz.net.
- Watch the Introduction to IBM Rational DOORS Next Generation demo to understand some benefits of using IBM Rational DOORS Next Generation over documents and spreadsheets.
- After you've migrated to DOORS Next
Generation, you might find these tutorials helpful:
- Read "Take control of your requirements projects with Configuration Management", developerWorks Yianna Papadakis-Kantos, October 2015.
- "IBM Rational DOORS Next Generation: Working with baselines" developerWorks Yianna Papadakis-Kantos, June 2015
- Download DOORS Next Generation for a free 60 day trial.
- Try DOORS Next Generation in the cloud.