How IBM migrated a large GForge installation to Rational environments

Steps and code used to move 500 projects and 7500 users from GForge to Rational Team Concert and Rational Asset Manager

Learn how an IBM team successfully used Open Services for Lifecycle Collaboration (OSLC) and other APIs to migrate 500 projects and 7500 users from Gforge to Rational Team Concert and Rational Asset Manager.

Steven W. Pogue (spogue@us.ibm.com), Senior Technical Staff Member, IBM

author photoSteven W. Pogue is a Senior Technical Staff Member in the IBM Software Group, where he focuses on strategy and technology. He has led several product teams and internal initiatives. In his spare time, he enjoys golf and spending time with his family and friends.



Andrew R. Freed (afreed@us.ibm.com), Senior Software Engineer, IBM

Andrew FreedAndrew R. Freed is a Senior Software Engineer at IBM who has educated more than 2000 end users in Rational Team Concert source control and provided guidance to several projects as they migrated away from Subversion. He has spent a lot of time working with the APIs offered by Rational Team Concert and Rational Asset Manager. In his spare time, he enjoys walking with his wife, son, and three dogs.



08 May 2012

Also available in Chinese

In 2005, IBM launched an internal community development platform using GForge, an open source collaboration platform derived from SourceForge. This allowed IBM to promote the development of reusable, high-value, software components and other assets using open source practices for use in our commercial products. Through campaigns and various other communication channels, along with integration into our enterprise search engine., the platform grew to facilitate more than 1800 projects involving over 13,000 contributors. The types of projects encompassed software components, tools, education modules, and services assets.

In 2010, we began developing a successor platform to leverage the greater capabilities provided by IBM software, along with lowering the cost of maintaining the operational environment. Starting in early 2011, after completing the new platform, we proceeded to promote the migration of active projects, using custom tools to facilitate the migration. We successfully migrated over 500 active projects and 7500 users who were involved with them.

This article recounts our experiences to serve as a guide for how to perform similar large-scale migrations.

The GForge environment

As a variant of SourceForge and other "forge" offerings, GForge provided a single platform to handle the following functions for projects:

  • Manage bug reports and feature requests
  • Manage work schedules
  • Provide hosted source control, using Concurrent Version System (CVS), Apache Subversion (SVN), or adapters to internal software configuration management systems (SCMs), such as IBM® Rational® ClearCase® or IBM CMVC (Configuration Management Version Control)
  • Provide for collaboration through hosted web pages, forums, mailing lists, and wikis
  • Provide a files and document repository for finished deliverables

Although a standard project template was provided as a project was created, each component could be tailored to fit the needs and process of the project and its members.


The new community development environment

When we decided to standardize on application lifecycle management (ALM) within the company, we found that IBM® Rational Team Concert™ and IBM® Rational® Asset Manager gave us greater capabilities than the GForge-based platform (see the Resources section for links to more information).

  • By using Rational Team Concert 3.01, we can manage bug reports and feature requests as work items, manage work schedules by using iterations and release plans, plus provide a rich, hosted SCM environment, along with integrated build facilities.
  • In By using Rational Asset Manager 7.5, we can wrap project deliverables such as software and documentation with their own process for making them available to others, along with the ability to classify, rate, and discuss them.

Both products enable us to integrate them with our corporate social collaboration environment, IBM Connections, to provide a robust way to socialize the development and use of these projects' deliverables. But that's another topic for another day.


Deployment

We deployed both Rational Team Concert Enterprise Edition and Rational Asset Manager Enterprise Edition on a corporate standard IBM® AIX® system running IBM® WebSphere® Application Server Network Deployment and an IBM® DB2® database. We allocated an LPAR to each product for better system management. They were integrated with our corporate directory server for identity management.

System configuration

There were several system-level concerns that needed to be addressed before starting the project migrations, including these:

  • Configuring Rational Team Concert for migration, including licensing and a development process template
  • Configuring Rational Asset Manager for the site and the migrated assets from GForge, including licensing, asset type, and categorization

Configuring Rational Team Concert for migration

Because we were migrating a large number of users from GForge to Rational Team Concert, we needed to consider the licensing model to use. In our case, we decided to use floating developer licenses for all of the migrated users, regardless of their roles, to have the maximum flexibility in role mapping. We could have also chosen to use floating contributor licenses for some of the lesser roles and reserved developer licenses for the more involved roles.

After deciding on the licensing strategy, we acquired and allocated those licenses to the associated license server.

Defining the development process template

When GForge was first deployed within IBM, we established a template of roles and permissions to be used when projects were first created. They included these roles, which are commonly used with forge offerings:

  • Project Administrator
  • Senior Developer
  • Junior Developer
  • Contributor
  • Observer

Each of these roles had various degree of access to source code, trackers, and tasks. The majority of the projects tend to stay with these role permissions. However, GForge also allows you to modify these roles, as well as to create new roles.

By using the Rational Team Concert Eclipse Client, we modeled a Rational Team Concert project area process template that represented the standard GForge template. You can also copy and extend an existing process template if there are development methods (such as scrum processes) that are desirable to carry into the migrated projects. You will use this process area template as a reference later as you do the project-level migration.

For our purposes, we matched existing GForge roles to these new Rational Team Concert user roles so that we could use common names for matching purposes.

Figure 1. Rational Team Concert template editor
Editing project roles with the template editor

Configuring Rational Asset Manager

Licensing

Like Rational Team Concert, we needed to decide on a licensing strategy for Rational Asset Manager for our migrated projects. We needed the flexibility for project members to be either consumers or producers of assets, depending on their project roles. Therefore, we used floating Collaborator and floating Publisher license types and assigned the licenses to a connected Rational License Server.

Predefined asset types and classification schemas

You can use Rational Asset Manager to define asset types and their taxonomy, with flexibility. Within GForge, there were two generic constructs that we wanted to migrate:

  • Packages, which contain all of the files associated with the release of a project,
  • Documents, which contain information that typically crosses release boundaries, such as documentation (instructions) for users and other how-to documents.

Within GForge, the classification of the projects and their assets is managed through the Trove interface. In preparation for the migration, we needed to reflect the Trove classification as category schemas within Rational Asset Manager. By using the Administration Category Schema interface, we were able to define the necessary schemas that represented the Trove classifications. After the migration, the project owners were able to reclassify their assets accordingly.

After we had defined the category schemas, we defined a migration-specific asset type called Migrated Asset by using the Administration Asset Types interface and adding the category schemas to it. That way, when the project packages and documents were migrated, the project owner could reassign types for the migrated assets to a more specific type defined within Rational Asset Manager. For example, we have asset types such as Software Component and Test Tools. In addition, the project owners can decide what type of governance they want for their assets, in terms of publishing and accessing them.


Project-level migration

In the Background and System configuration sections, we explained how we created the site-wide infrastructure and basic project templates that we used for all of the projects that we migrated to Rational software. This section describes the project-specific migration tasks.

We start by going over the APIs that we used during the migration process. Then we show how we created basic project structure and user records. Next, we cover assigning membership and roles to users within a project. After that, we show how we migrated work items and source code into Rational Team Concert and, finally, how we migrated published assets into Rational Asset Manager.

Product APIs

The Rational products that we migrated the GForge data into have extensive APIs that we used to create project structure and import our data. A brief description of these APIs follows.

Rational Team Concert Java API

Rational Team Concert is built as a collection of plug-ins that run on the Eclipse OSGi platform, but there is also a client layer that can run in plain Java environments. In addition, the full source code of Rational Team Concert is available for download from Jazz.net.

The Java API offers access to all parts Rational Team Concert. We used this API to migrate projects, users, and memberships, and we used the plug-in layer rather than the plain Java client. However, there are numerous scenarios documented on Jazz.net that use the plain Java client. Integrating and Extending Rational Team Concert (the SDK) in the Jazz.net Team Wiki gives a complete description of the Java API, including where to download it and how to install it.

Open Services for Lifecycle Collaboration (OSLC)

All of the products built on the Jazz platform, including Rational Team Concert and Rational Asset Manager, expose and manipulate some amount of data by using the Open Services for Lifecycle Collaboration (OSLC) standard. We used OSLC for migrating work items into Rational Team Concert.

You can find a great introduction to using at the two together in Patrick Streule's Jazz Team Blog post, OSLC and Rational Team Concert. We focused on manipulating Rational Team Concert work items with OSLC, thus we found Resource-Oriented Work Item API with OSLC_CM 1.0 very helpful, too.

Rational Asset Manager Java API

Rational Asset Manager offers a plain Java API that provides access to community and user creation, as well as asset manipulation. We used all of these aspects of the API in our migration.

Check the Rational Asset Manager information center for where to get the API and documentation. Under the "Extending production function" section, look for Using Rational Asset Manager Java API.

Migrating project areas

The first part of migrating a project from GForge into Rational software is to set up a "catching" area in Rational Team Concert and Rational Asset Manager. In Rational Team Concert, this is called a project area; in Rational Asset Manager, it is called a community. Both of these areas are mostly empty containers until you add content to them.

Rational Team Concert project area

We needed several pieces of code to create a Rational Team Concert project area:

  1. First, we logged in to the Rational Team Concert repository, as shown in code Listing 1.
  2. Then we gathered the information about what process templates were available for our project area.
  3. Finally, we created the project area.
Listing 1. Rational Team Concert login snippet
import org.eclipse.core.runtime.IProgressMonitor;
import com.ibm.team.repository.client.ITeamRepository;
import com.ibm.team.repository.client.TeamPlatform;
import com.ibm.team.repository.client.ITeamRepository.ILoginHandler;
import com.ibm.team.repository.client.ITeamRepository.ILoginHandler.ILoginInfo;
import com.ibm.team.repository.common.TeamRepositoryException;

public class RTCConnection {
	public ITeamRepository login(IProgressMonitor monitor,
 String repo_address, String userID, String password)
 throws TeamRepositoryException {
		ITeamRepository repository =
 TeamPlatform.getTeamRepositoryService().getTeamRepository(repo_address);
		repository.registerLoginHandler(new LoginHandler(userID, password));
		repository.login(monitor);
		return repository;
	}
}

private static class LoginHandler implements ILoginHandler, ILoginInfo {
	private String fUserId;
	private String fPassword;

	private LoginHandler(String userId, String password) {
		fUserId = userId;
		fPassword = password;
	}

	public String getUserId() {
		return fUserId;
	}
		
	public String getPassword() {
		return fPassword;
	}
		
	public ILoginInfo challenge(ITeamRepository repository) {
		return this;
	}
}

Then we took our logged-in connection and created a Rational Team Concert project area. We required as inputs a name and description for the project area as well as the identifier of the Rational Team Concert process template we wished to use. Note that the ITeamRepository connection must be created with a user that has the JazzAdmin repository role in order to create a project area.

Listing 2. Rational Team Concert project area creation code
 import java.util.List;
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.core.runtime.NullProgressMonitor;
import com.ibm.team.process.client.IProcessClientService;
import com.ibm.team.process.client.IProcessItemService;
import com.ibm.team.process.common.IProcessDefinition;
import com.ibm.team.process.common.IProjectArea;
import com.ibm.team.repository.client.ITeamRepository;
import com.ibm.team.repository.common.TeamRepositoryException;

public class RTCProject {
	public IProjectArea createProject(
ITeamRepository repo, String projectTitle, 
String projectDescription, String processTemplateID) 
throws TeamRepositoryException {

		IProgressMonitor monitor = new NullProgressMonitor();            
		IProcessItemService service = (IprocessItemService)
 repo.getClientLibrary(IProcessItemService.class);
		
		// First create the project area
		IProjectArea area = service.createProjectArea();
		area.setName(projectTitle);
		area.getDescription().setSummary(projectDescription);
		area.setProcessDefinition(
	getProcessDefinition(repo, processTemplateID));
		area = (IProjectArea) service.save(area, monitor);

		// Now initialize the project making it usable.
		// The logged-in user is added as a project administrator.
		area = (IProjectArea) service.getMutableCopy(area);
		area = service.initialize(area, monitor);

		return area; 
	}
	
	private IprocessDefinition getProcessDefinition(
ITeamRepository repo, String processTemplateID) 
throws TeamRepositoryException {
		IProgressMonitor monitor = new NullProgressMonitor();
		IProcessItemService service= (IprocessItemService)
	repo.getClientLibrary(IProcessItemService.class);
		List<IProcessDefinition> definitions = 	service.findAllProcessDefinitions(
	IProcessClientService.ALL_PROPERTIES, monitor);
		
		for (IProcessDefinition definition : definitions) {
			if (definition.getProcessId().equals(processTemplateID)) {
				return definition;
			}
		}
		return null;
	}
}

The newly created project area only has one member, the JazzAdmin who created the project, and this user is also set as a project administrator. Later in this article, you will see how we added users with specific roles to this project area.

Creating the Rational Asset Manager community

Similar to creating the Rational Team Concert project area, we had two main jobs for creating a community in the repository:

  1. Logging in to the repository
  2. Creating a community in the repository

Note:
Someone with repository administrator credentials needs to run this code.

Listing 3. Rational Asset Manager community creation code
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.core.runtime.NullProgressMonitor;

import com.ibm.ram.client.RAMAction;
import com.ibm.ram.client.RAMCommunity;
import com.ibm.ram.client.RAMSession;

public class RamCommunity {

    public RAMSession login(String repo_address, String userID, String password) {
        return new RAMSession(repo_address,userID,password);
    }
    
	public RAMCommunity create(
	RAMSession repo, String projectTitle, String projectDescription) {
		IProgressMonitor monitor = new NullProgressMonitor();  
		RAMCommunity community = repo.createCommunity(projectTitle);
		community.setDescription(projectDescription);
		community.setAction(RAMAction.CREATE);
		repo.put(community, monitor);
		return community; 
	}
}

The next section explains adding members to the community.

Migrating project users

In the previous section, we showed how we created Rational Team Concert project areas and Rational Asset Manager communities. However, as we mentioned previously, these spaces are essentially empty when you create them. So we needed to populate them with users.

Note:
Before adding users to these spaces, we needed to define and customize the roles that we wanted to be available to users. We described this earlier in the System configuration section.

After adding roles, we had three programmatic steps to add users.

  1. We registered the users to the appropriate systems.
  2. We queried the available roles
  3. We assigned some selection of roles to the new users, based on role-name matching.

We allocated a large number of floating licenses for our Rational projects so we could automatically assign floating licenses to all of our new users. Your solution might require a different licensing arrangement.

Registration

Here again, Rational Team Concert requires JazzAdmin authority to create new users and to assign licenses to these users.

Rational Asset Manager requires repository administrator privileges to create users. Additionally, users are prompted to provide addition registration information on their first visits to the Rational Asset Manager web application.

Listing 4. Rational Team Concert user creation code
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.core.runtime.NullProgressMonitor;

import com.ibm.team.repository.client.IContributorManager;
import com.ibm.team.repository.client.ITeamRepository;
import com.ibm.team.repository.client.internal.TeamRepository;
import com.ibm.team.repository.common.IContributor;
import com.ibm.team.repository.common.ILicenseAdminService;
import com.ibm.team.repository.common.ItemNotFoundException;
import com.ibm.team.repository.common.TeamRepositoryException;

public class RTCUser {

	public IContributor register(
ITeamRepository repo, String loginId, String email, String fullName)
throws TeamRepositoryException {
		// See if the user exists already. If so, return existing record.
		IProgressMonitor monitor = new NullProgressMonitor(); 
		IContributorManager cmgr = repo.contributorManager();
		IContributor user = null;
		try {
			user = cmgr.fetchContributorByUserId(loginId, monitor);
			if(user != null) return user;
		} catch (ItemNotFoundException e) {
			// Ignore. There is no user so we need to create them.
		}
		
		user = (IContributor) IContributor.ITEM_TYPE.createItem();
		user.setUserId(loginId);
		user.setEmailAddress(email);
		user.setName(fullName);
		user = cmgr.saveContributor(user, monitor);
		
		// Assign a license to the user
		ILicenseAdminService licenseAdmin = (ILicenseAdminService) 
				((TeamRepository) repo)
	.getServiceInterface(ILicenseAdminService.class);
		licenseAdmin.assignLicense(user, "com.ibm.team.rtc.developer.floating");

		// Return created user
		return user;
	}
}
Listing 5. Rational Asset Manager user creation code
import org.eclipse.core.runtime.NullProgressMonitor;

import com.ibm.ram.client.RAMSession;
import com.ibm.ram.client.RAMUser;

public class RamUser {

	public RAMUser register(
	RAMSession session, String loginId, String email, String fullName) {
		RAMUser ramUser = session.getUser(loginId);
		if(ramUser.isRegistered())
			return ramUser;

		ramUser = session.getUser(loginId);
		ramUser.setName(fullName);
		ramUser.setEmail(email);
		session.put(ramUser, new NullProgressMonitor());
		return ramUser;
	}
}

Membership and role assignments

In both Rational Team Concert and Rational Asset Manager, the pattern for adding members is the same:

  1. Take a given user as input.
  2. Query for the available roles.
  3. Assign the desired roles.

In Rational Team Concert, there is an additional designation for Project Administrators, but in Rational Asset Manager, that is part of the built-in Administrator role.

Rational Asset Manager membership manipulation requires that you use internal APIs.

Note:
These APIs are not supported and are subject to change.

Additionally, these APIs require numeric community IDs, which you can find either by calling getID() on the Rational Asset Manager RAMCommunity object from Listing 3 or by observing the ID parameter in the URL for the Rational Asset Manager community.

Listing 6. Rational Team Concert member addition code
import org.eclipse.core.runtime.NullProgressMonitor;

import com.ibm.team.process.client.IClientProcess;
import com.ibm.team.process.client.IProcessItemService;
import com.ibm.team.process.client.workingcopies.IProcessAreaWorkingCopy;
import com.ibm.team.process.common.IProcessArea;
import com.ibm.team.process.common.IRole;
import com.ibm.team.repository.client.ITeamRepository;
import com.ibm.team.repository.common.IContributor;
import com.ibm.team.repository.common.IContributorHandle;
import com.ibm.team.repository.common.TeamRepositoryException;

public class RTCProjectMember {

	public void create(
ITeamRepository repo, IProcessArea processArea,
 IContributor contributor, String roleID, 
 boolean isAdministrator)
throws TeamRepositoryException{
		// Find roles
		IRole[] roles = new IRole[]{getRole(processArea, roleID)};
		IProcessItemService service = (IprocessItemService)
 repo.getClientLibrary(IProcessItemService.class);
		
		// Join area and assign roles
		IProcessAreaWorkingCopy areaWC = (IprocessAreaWorkingCopy)
 service.getWorkingCopyManager().createPrivateWorkingCopy(processArea);
		areaWC.getTeam().setRoleCast(contributor, roles);
		areaWC.getTeam().addContributorsSettingRoleCast(
				new IContributor[]{contributor}, roles);
		areaWC.save(new NullProgressMonitor());
		
		// Assign administrator if needed
		if(isAdministrator) {
			addAdministrator(contributor, processArea, service);
		}
	}

	private void addAdministrator(
IContributor newAdmin, IProcessArea area, IProcessItemService service) 
throws TeamRepositoryException {
		area = (IProcessArea) service.getMutableCopy(area);
		
		IContributorHandle[] oldAdmins = area.getAdministrators();
		IContributorHandle[] newAdmins = null;
		if(oldAdmins == null || oldAdmins.length == 0) {
			newAdmins = new IContributorHandle[]{newAdmin};
		}
		else {
			//Copy admin array and add to the end
			//unless our admin is already listed
			//(omitted for brevity)
		}
		
		area = (IProcessArea) service.getMutableCopy(area);
		area.setAdministrators(newAdmins);
		area = (IProcessArea) service.save(area, new NullProgressMonitor());
	}
	
	private IRole getRole(
IProcessArea area, String roleId) 
throws TeamRepositoryException {
		ITeamRepository repo = (ITeamRepository) area.getOrigin();
		IProcessItemService service =(IProcessItemService) 
repo.getClientLibrary(IProcessItemService.class);
		IClientProcess clientProcess = service.getClientProcess(area, null);
		IRole[] availableRoles = clientProcess.getRoles(area, null);
		
		for (IRole role : availableRoles) {
			if (role.getId().equals(roleId)) {
				return role;
			}
		}
		return null;
	}
}
Listing 7. Rational Asset Manager member addition code
 import java.net.URL;

import com.ibm.ram.client.RAMSession;
import com.ibm.ram.common.data.UserInformation;
import com.ibm.ram.internal.access.ws.RAM1;
import com.ibm.ram.internal.client.RAMClient;
import com.ibm.ram.internal.common.data.RoleSO;

public class RamCommunityMember {
	
	public void create(
RAMSession repo, int communityId, String userId, String roleName)
 throws Exception {
		RAM1 ramWS = getRAMWebService();

		UserInformation user = ramWS.getUsers(new String[]{userId})[0];
		int[][] communityRoleIds = new int[1][];
		communityRoleIds[0] = new int[]{getRole(communityId, roleName)};
		
		ramWS.updateUser(userId, 
				user.getName(), user.getPhone(), user.getEmail(), 
				0,0, new int[]{communityId}, communityRoleIds);
	}
	
	private int getRole(int communityId, String roleName) throws Exception {
		RAM1 ramWS = getRAMWebService();
		RoleSO[] availableRoles = ramWS.getRoles(communityId);
		for(RoleSO role : availableRoles) {
			if(roleName.equals(role.getName()))
				return role.getId();
		}
		return -1;
	}
	
	private RAM1 getRAMWebService() {
		RAMClient client = new RAMClient(
	new URL(RAM_SERVER_URL), RAM_USER, RAM_PASSWORD, null);
		RAM1 ramWS = client.getRAM1Webservice();
		return ramWS;
	}
}

Adding users to these products through an API means that the users did not receive connection details for the Rational Team Concert project area and Rational Asset Manager community. Therefore, we created additional notes for emailing this information to our users after we added them to the projects.

Migrating project work items

Bug tracking is an important part of any software project, so it was important to be able to migrate bug reports (called bug trackers) from GForge into Rational Team Concert work items. GForge bug trackers follow a standard configuration that allows some customization. Fortunately, Rational Team Concert supports an even broader range of configuration.

We also migrated GForge tasks into Rational Team Concert as Stories. (The process is very similar to that of trackers, so we do not cover it in this article.)

Configuring custom types of work items, attributes, workflows, and editors

Rational Team Concert comes with a strong built-in set of work items, workflows, and editors. These have been rigorously tested, so, if at all possible, stick with them. A simple migration revolves around using the built-in work item types, mapping obvious fields such as title, description, and owner, and appending all other fields to the description.

More thorough mappings are possible if you devote the time to customization. Mapping custom attributes thoroughly makes querying these attributes easier. Otherwise, you are limited to full-text searches on work items. The linked articles linked under "Customization in Rational Team Concert" in the Resources section offer a full treatment of the customization options. A brief summary is included here:

Types
Types are grouped into categories, such as Defect or Story. Categories share a set of attributes and a common workflow and other behavior.
 
Attributes
Attributes are the fields that make up a type. Examples of attributes include title, description, owner, and priority. Attributes can have many types, including various numeric or string options, as well as custom enumerations, and more.
 
Workflow
A workflow is a combination of states, actions, and transitions that are available for work items. An example is a workflow that governs how a work item moves within states, such as New, In Progress, Resolved, and Closed.
 
Editor
An editor offers a way to display and edit the contents of a work item by arranging the various attributes in a visual configuration. Customizing an editor is most important when adding new custom fields. By default, these fields are added to a tab labeled Custom, but you might to move them to the Overview tab.
 

Important:
Do not constrain work items too broadly before the migration. For instance, by setting constraints for required fields constraints can become a problem. Too many constraints will hamstring your initial migration efforts. After you have migrated your bug tracker data, you can set up the constraints that you need for any new work items.

Using OSLC to migrate work items

As described earlier, OSLC is a data format that can be expressed in either XML or JSON (JavaScript Object Notation). The migration for work items revolves around performing various GET, POST, and PUT operations against various Rational Team Concert resources with OSLC payloads. Our examples use XML, but you can use the format most convenient for you.

Caution:
Rational Team Concert will entirely reject any nonconforming work item if it is missing a required property, or it contains a nonexistent attribute, or if there is any other validation error. If this is acceptable, you can construct a single large payload for each migrated work item. Our approach was to make a series of small work item updates in rapid succession, in combination with loading as much into the description field as possible, so that we could maximize the amount of data that we migrated. Migrating multiple small updates is slower than one large update. However, for most migrations (up to a few thousand work items), this performance difference is not significant.

Overall, this was our approach:

  1. Create the initial work item.
  2. Fill in custom fields.
  3. Add comments.
  4. Apply necessary state transitions.
  5. Set the new work item owner

We set the work item owner last to avoid having the work item owner be notified (by email) of every single update that we made.

As you develop work item migration, the Resource-Oriented Work Item API with OSLC_CM 1.0 guide in the Jazz Team Wiki will be invaluable. Here, we will just provide the general XML snippets and URLs that we interacted with.

1. Create the initial work item

This requires a couple of sub-steps that will help us fill in the fields we need to get started.

  1. Get the resource UUID of the project area.
    • One way is to invoke the getItemId().getUuidValue() method on the IProjectArea created earlier.
    • Another way is to make an OSLC call to the server for a list of all projects, and parse the UUID from the response. Listing 8 shows an example, with the UUID in bold.
GET
https://localhost/jazz/oslc/projectareas.xml
Listing 8. OSLC list of projects
<oslc_cm:Collection ... oslc_cm:totalCount="1">
<rtc_cm:Project 
rdf:resource="https://localhost:443/jazz/oslc/projectareas/_bQci8BkFEeC1WKmhMVUJnw"> 
<dc:title>Sample project</dc:title> 
<dc:description> 
This is a project I am migrating work items into 
</dc:description> 
... 
</rtc_cm:Project> 
</oslc_cm:Collection>
  1. Collect a handful of additional resource UUIDs, such as Type, Category, and Owner. Only Type is required for work item creation, but Category and Owner are commonly specified.

Tip:
Notice that each of these following queries references the project resource UUID from above.

GET
https://localhost/jazz/oslc/types/_bQci8BkFEeC1WKmhMVUJnw.xml?oslc_cm.properties=
dc:title,dc:identifier
Listing 9. OSLC list of work item types within a project
<?xml version="1.0" encoding="UTF-8"?><oslc_cm:Collection xmlns:oslc_cm="http://open-
services.net/xmlns/cm/1.0/" oslc_cm:totalCount="2" ...> 
<rtc_cm:Type rdf:resource=
"https://localhost:443/jazz/oslc/types/_bQci8BkFEeC1WKmhMVUJnw/defect"> 
<dc:identifier>defect</dc:identifier> 
<dc:title>Defect</dc:title> 
</rtc_cm:Type> 
<rtc_cm:Type rdf:resource=
"https://localhost:443/jazz/oslc/types/_bQci8BkFEeC1WKmhMVUJnw/task"> 
<dc:identifier>task</dc:identifier> 
<dc:title>Task</dc:title> 
</rtc_cm:Type> 
</oslc_cm:Collection>

Get categories, in Listing 10, gets the category resource UUID, as well as project/team resource UUIDs.

GET
https://localhost/jazz/oslc/categories.xml?oslc_cm.query=rtc_cm:projectArea=
%22_bQci8BkFEeC1WKmhMVUJnw%22&oslc_cm.properties=dc:title,rtc_cm:projectArea,
rtc_cm:defaultTeamArea
Listing 10. OSLC list of work item categories within a project
<?xml version="1.0" encoding="UTF-8"?><oslc_cm:Collection xmlns:oslc_cm="http://open-
services.net/xmlns/cm/1.0/" oslc_cm:totalCount="2" … > 
<rtc_cm:Category 
rdf:resource=
"https://localhost:443/jazz/resource/itemOid/com.ibm.team.workitem.Category/_cQzMsBk
FEeCk3aYr1rys6g"> 
<rtc_cm:projectArea 
rdf:resource="https://localhost:443/jazz/oslc/projectareas/_bQci8BkFEeC1WKmhMVUJnw"/> 
<dc:title>Frontend</dc:title> 
<rtc_cm:defaultTeamArea 
rdf:resource="https://localhost:443/jazz/oslc/teamareas/_cYVYEBkFEeCk3aYr1rys6g"/> 
</rtc_cm:Category> 
<rtc_cm:Category
rdf:resource=
"https://localhost:443/jazz/resource/itemOid/com.ibm.team.workitem.Category/_cZ9IsRk
FEeCk3aYr1rys6g"> 
<rtc_cm:projectArea 
rdf:resource="https://localhost:443/jazz/oslc/projectareas/_bQci8BkFEeC1WKmhMVUJnw"/> 
<dc:title>Backend</dc:title> 
<rtc_cm:defaultTeamArea 
rdf:resource="https://localhost:443/jazz/oslc/teamareas/_cYVYEBkFEeCk3aYr1rys6g"/> 
</rtc_cm:Category> </oslc_cm:Collection>
GET
https://localhost/jazz/oslc/users.xml?oslc_cm.query=
rtc_cm:emailAddress=%22afreed@us.ibm.com%22&
oslc_cm.properties=dc:title,rtc_cm:userId,rtc_cm:emailAddress
Listing 11. OSLC Get user (search by email address)
<?xml version="1.0" encoding="UTF-8"?><oslc_cm:Collection xmlns:oslc_cm="http://open-
services.net/xmlns/cm/1.0/" oslc_cm:totalCount="1" … > 
<rtc_cm:User rdf:resource=
"https://localhost:443/jazz/oslc/users/_AW328LFTEd-SGMrAlkrEIQ"> 
<rtc_cm:userId>afreed@us.ibm.com</rtc_cm:userId> 
<dc:title>Andrew R. Freed</dc:title> 
<rtc_cm:emailAddress>afreed@us.ibm.com</rtc_cm:emailAddress> 
</rtc_cm:User> 
</oslc_cm:Collection>

With this information, we were ready to create a work item with the specified fields.

Tips:
Although we found the user resource UUID at the beginning, we did not assign the work item owner until the last update. We also filled in some work item tags (under dc:subject), which is a convenient way of migrating custom GForge fields without creating new custom Rational Team Concert attributes.

The following POST used a content-type of application/x-oslc-cm-change-request+xml and an accept header of text/xml.

POST
https://localhost/jazz/oslc/contexts/_bQci8BkFEeC1WKmhMVUJnw/workitems
Listing 12. OSLC POST payload for creating a work item
 <?xml version="1.0" encoding="UTF-8"?><oslc_cm:ChangeRequest xmlns:oslc_cm="http://open-
services.net/xmlns/cm/1.0/"> 
    <dc:title xmlns:dc="http://purl.org/dc/terms/">Sample migrated defect</dc:title> 
    <dc:description xmlns:dc="http://purl.org/dc/terms/">Testing the ability to migrate a 
defect</dc:description> 
    <dc:subject xmlns:dc="http://purl.org/dc/terms/">tag1, tag2</dc:subject> 
    <rtc_cm:filedAgainst xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
rdf:resource=
"https://localhost:443/jazz/resource/itemOid/com.ibm.team.workitem.Category/_cZ9IsRk
FEeCk3aYr1rys6g" xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/"/> 
    <dc:type xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
rdf:resource="https://localhost:443/jazz/oslc/types/_bQci8BkFEeC1WKmhMVUJnw/defect" 
xmlns:dc="http://purl.org/dc/terms/"/>
    <rtc_cm:projectArea xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
rdf:resource="https://localhost:443/jazz/oslc/projectareas/_bQci8BkFEeC1WKmhMVUJnw"
xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/"/> 
    <rtc_cm:teamArea xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
rdf:resource="https://localhost:443/jazz/oslc/teamareas/_cYVYEBkFEeCk3aYr1rys6g" 
xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/"/> 
</oslc_cm:ChangeRequest>

The expected response is "HTTP 201 Created," with a Location header that gives the resource UUID path of the newly created work item. Additionally, the body content should be all fields of the new work item. Listing 13 shows most of the work item content. Notice the dc:identifier field, which shows the short ID of this work item.

Listing 13. OSLC work item creation response
Header "location" = "https://localhost:443/jazz/resource/itemOid/com.ibm.team.
workitem.WorkItem/_Qj-IAO4CEeCSaY8h45TMBQ"
<?xml version="1.0" encoding="UTF-8"?> 
<oslc_cm:ChangeRequest xmlns:oslc_cm="http://open-services.net/xmlns/cm/1.0/" 
xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/" ...> 
<dc:identifier>57596</dc:identifier> 
... 
</oslc_cm:ChangeRequest>

2. Fill in custom fields

After creating the initial work item, you can make any number of updates to the work item. Rational Team Concert supports updating only part of the work item, and only the fields that you specify are updated. We took advantage of this to build and upload smaller update payloads. In the example in Listing 14, we set the value of a custom string attribute called customString and a custom integer attribute called customInteger. You can specify as many or as few fields as you want to update.

Updates are done through PUT calls. Generally, you should check for concurrent updates by verifying the ETag of the underlying work item. In the case of migration, we assumed that the migration process is the only one updating work items at this time, and we skipped this check. Additionally, we used a content type header of application/x-oslc-cm-change-request+xml and an accept header of */* (because we ignored the response body). The PUT location is the same as the work item resource UUID path that we gathered above.

PUT
https://localhost:443/jazz/resource/itemOid/com.ibm.team.workitem.WorkItem/
_Qj-IAO4CEeCSaY8h45TMBQ
Listing 14. OSLC PUT payload for updating a custom work item attribute
<?xml version="1.0" encoding="UTF-8"?> 
<oslc_cm:ChangeRequest xmlns:oslc_cm="http://open-services.net/xmlns/cm/1.0/"> 
    <rtc_cm:customString xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/">
my string value</rtc_cm:customString> 
    <rtc_cm:customInteger 
xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/">100</rtc_cm:customInteger> 
</oslc_cm:ChangeRequest>

Expected response: HTTP 204 Updated.

3. Add comments

We replayed comments from old GForge trackers into the work items that we created. The comments all appear to have been created by the user who was running the migration, so we augmented the comment description with additional information from the original commenter. Comments are done through a POST to a URL built from the work item's ID (dc:identifier). Again, we used a content type header of application/x-oslc-cm-change-request+xml and an accept header of */* (because we were ignoring the response body).

POST
https://localhost/jazz/oslc/workitems/57596/rtc_cm:comments
Listing 15. OSLC POST payload for creating a work item comment
<?xml version="1.0" encoding="UTF-8"?> 
<rtc_cm:Comment xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/"> 
    <dc:description xmlns:dc="http://purl.org/dc/terms/">A discussion comment we are
adding to the work item. Originally posted by 
Steve Pogue on July 1, 2010.</dc:description> 
</rtc_cm:Comment>

Expected response: HTTP 201 Created

4. Apply necessary state transitions

All work items are created in a default New state. We wanted to map migrated work items to other states, such as In Progress, Resolved, or Closed. Work item state is changed in two steps:

  1. The first step is to open the process configuration source of your project area and find the ID of the action that moves a work item to your desired end state. In our example, we wanted to move a work item to the Resolved state. Inspecting the process configuration source showed us the Resolve action has this ID:
    bugzillaWorkflow.action.resolve
  2. With this action ID, we generated the URL for the PUT call by appending the action to our original work item resource UUID path. Our payload was a minimal shell of the work item, because we were not updating any fields (only the state). Again, we used a content type header of application/x-oslc-cm-change-request+xml and an accept header of */* (because we ignored the response body).
PUT
https://localhost:443/jazz/resource/itemOid/com.ibm.team.workitem.WorkItem/
_Qj-IAO4CEeCSaY8h45TMBQ?_action=bugzillaWorkflow.action.resolve
Listing 16. OSLC PUT payload for applying a work item transition
<?xml version="1.0" encoding="UTF-8"?> 
<oslc_cm:ChangeRequest xmlns:oslc_cm="http://open-services.net/xmlns/cm/1.0/"> 
</oslc_cm:ChangeRequest>

Expected response: HTTP 204 Updated.

5. Set the new work item owner

Just as we updated the custom fields on the work item, we updated the work item owner in a separate step. This was to spare the work item owner from all of the email notifications generated by the numerous small updates we had done so far. We used the user resource that we retrieved earlier.

Again, we used a content type header of application/x-oslc-cm-change-request+xml and an accept header of */* (because we ignored the response body). The PUT location is the same as the work item resource UUID path that we gathered already.

PUT
https://localhost:443/jazz/resource/itemOid/com.ibm.team.workitem.WorkItem/
_Qj-IAO4CEeCSaY8h45TMBQ
Listing 17. OSLC PUT payload for updating a work item owner.
<?xml version="1.0" encoding="UTF-8"?> 
<oslc_cm:ChangeRequest xmlns:oslc_cm="http://open-services.net/xmlns/cm/1.0/"> 
    <rtc_cm:ownedBy xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
rdf:resource="https://localhost:443/jazz/oslc/users/_AW328LFTEd-SGMrAlkrEIQ" 
xmlns:rtc_cm="http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/"/> 
</oslc_cm:ChangeRequest>

Expected response: HTTP 204 Updated.

Note:
For more tips on using OSLC with work items, consult the Resource-Oriented Work Item API with OSLC_CM 1.0 guide.


Source code management

Source code management (SCM) is at the heart of any software development. When migrating users from one set of development tools to another, it is important to maintain the source control structure and history. In our migration, we helped more than 500 projects move their source code from CVS and Subversion to Rational Team Concert. Source code migration is thoroughly covered in other documents cited in Resources, so this article summarizes only the advice that we gave to the project teams as we helped them through the migration.

Differences between Subversion, CVS, and Rational Team Concert

Subversion (SVN) has two kinds of SCM constructs: local code (working copy) and server code (branches and tags). SVN-based projects typically have a main or current branch or trunk that is shared by all project developers. For the sake of this exercise, CVS is considered functionally equivalent to SVN.

Rational Team Concert (RTC in code and paths) also has local (sandbox) and server (stream) layers. In addition, there is a required intermediate layer called a repository workspace. The repository workspace is a personal resource stored on the server. It is a place for the developer to check changes into before those changes are ready to be delivered to the team (through the shared stream).

Additionally, Rational Team Concert source control requires you to create one or more components to manage your source code. A component is simply a logical grouping of related code artifacts.

A full treatment of the differences between these source control systems is included in the article in the Jazz.net library by Andrew R. Freed, titled Comparing concepts between Subversion and Rational Team Concert.

For understanding our migration, it is sufficient to focus mainly on the similarities. Rational Team Concert streams are analogous to Subversion branches. Rational Team Concert repository workspaces are analogous to personal Subversion branches that are stored on the server. Both SCM systems have a concept of a local workspace or working area. Rational Team Concert introduces components, which are logical groupings of source code within streams and workspaces.

Figure 2. Rational Team Concert source control concepts
Rational Team Concert SCM example

Planning your repository structure

The most important part of the source code migration planning phase is to come up with a plan. Do not build an SCM layout on the fly while running the code import tool.

There is no universal approach for planning repository structure, but the following best practices can guide your planning:

  • Streams naturally correspond to CVS/SVN branches. Therefore, we recommend that you create separate streams for each CVS/SVN branch.
  • It is a good idea to include the developer's name and project name in the repository workspace's name. This simplifies managing and searching for repository workspaces.
  • Components are used to logically group related code artifacts. Typical components for an MVC application could include Build, Backend, Common, and Web.
  • If your project is small, you can also group everything into a single component.

The most important part of the source code migration planning phase is to come up with a plan. Do not build an SCM layout on the fly while running the code import tool. We helped many projects map their old SCM layouts into Rational Team Concert SCM.

Importing source code to Rational Team Concert

Rational Team Concert includes a source code importer that works against a Subversion dump file. If you are importing from CVS, you can create an SVN dump file by using the cvs2svn utility. If you are importing from SVN, you can use svnadmin dump.

The source code importer will replay your entire change history from your SVN dump file into your Rational Team Concert repository. This includes faithful recreation of change sets (file deltas and original author, comment, and date tags) and branch points (baselines are created for all tags and branch points). The importer replays one line of development, so you will run it once for each component for each stream you are importing into.

Before running the importer, double-check your plan. Then create a stream, component (or components), and a repository workspace to import into. You will import into a repository workspace, thereby giving the importer a chance to verify the imported code locally before committing it to the team's stream.

The Jazz.net Library article titled Importing data from Subversion and CVS into Jazz Source Control presents a full treatment of the Rational Team Concert source code import options.


Migrating published files

GForge has a rudimentary file repository. The core concepts of this repository are files, releases (groups of related files), and packages (groups of releases). For instance, you might have a package called Installer that had releases designated as 1.0, 2.0, and nightly, and each release contained the files called installer.zip and README.txt.

Rational Asset Manager has two core concepts: artifacts (files, links, etc) and assets (groups of artifacts). Our migration strategy was to map GForge files to Rational Asset Manager artifacts, GForge packages to Rational Asset Manager assets, and GForge releases to Rational Asset Manager asset versions. In our example, there would be one asset called Installer with three asset versions (1.0, 2.0, and nightly), each containing two file artifacts.

Configuring asset types and constraints

As introduced in the System configuration section, you need to have the Rational Asset Manager community configured before you can add assets to it. At minimum, you must have a set of asset types available. Although we did not want to, can set up constraints on those asset types if you prefer to do that.

In our migration, we provided a single asset type called Migrated Asset with no constraints. We instructed Project Administrators to manually reclassify their assets after the migration and apply constraints as needed. This was largely because it was difficult to ascertain type information from the GForge artifacts in a correct and scalable manner. If you are migrating from a system other than GForge, you might find it easier to classify your assets initially.

You can find more information about asset types and constraints in the Rational Asset Manager information center (see Resources for a link).

Using the GForge SOAP API to get published files

Our GForge installation was derived from GForge 4.5, which included a SOAP API. Yours might differ, of course. Here is an example of how we parsed published files from a GForge repository after generating Java bindings to the GForge Web Services Description Language, or WSDL (com.ibm.gforge.*)

Listing 18. GForge file retrieval snippet
import com.ibm.gforge.GForgeAPILocator;
import com.ibm.gforge.GForgeAPIPortType;
import com.ibm.gforge.FRSFile;
import com.ibm.gforge.FRSPackage;
import com.ibm.gforge.FRSRelease;
import java.io.File;
import java.util.LinkedList;
import java.util.List;

public class GForgeFilesMigrator {
	public List<GForgeRelease> parseGForgePackages(int groupId) {
		GForgeAPIPortType soapConnection = 
new GForgeAPILocator().getGForgeAPIPort();
		String soapSessionSer =
 soapConnection.login(loginName, password);

		List<GForgeRelease> toMigrate = 
new LinkedList<GForgeRelease>();
		FRSPackage[] packages = 
soapConnection.getPackages(soapSessionSer, groupId);
		for(FRSPackage pack : packages) {
			FRSRelease[] releases = 
soapConnection.getReleases(soapSessionSer, groupId, pack.getPackage_id());
			for(FRSRelease release : releases) {
				GForgeRelease gforgeRelease = 
new GForgeRelease();
				gforgeRelease.name = 
pack.getName();
				gforgeRelease.version = 
release.getName();
				gforgeRelease.description = 
"Migrated from GForge";
				//you need to provide a real mapping
				gforgeRelease.type = "..."; 

				FRSFile[] files = 
soapConnection.getFiles(
soapSessionSer, groupId, pack.getPackage_id(), release.getRelease_id());
				List<File> filesToMigrate = new LinkedList<File>();
				for(FRSFile file : files) {
					filesToMigrate.add(loadFile(file.getFile_id()));
				}
				gforgeRelease.files = (File[]) files.toArray(
			new File[files.size()]);
				toMigrate.add(gforgeRelease);
			}
		}
	}
}

class GForgeRelease {
	public File[] files;
	public String name;
	public String description;
	public String version;
	public String type;
}

Using the Rational Asset Manager API to migrate assets

Listing 19 is a brief example on how we created a new Rational Asset Manager asset and attached file artifacts to it (packages and documents).

Listing 19. Rational Asset Manager file migration from GForge code
import java.io.File;

import org.eclipse.core.runtime.NullProgressMonitor;

import com.ibm.ram.client.LocalFileArtifact;
import com.ibm.ram.client.RAMAsset;
import com.ibm.ram.client.RAMFolderArtifact;
import com.ibm.ram.client.RAMSession;
import com.ibm.ram.client.status.RAMStatus;

public class RamAsset {
	public void create(
RAMSession session, GForgeRelease gforgeArtifact, String communityName) {
		// Use session.createAsset(GUID, version) for
		// subsequent releases in the same package
		RAMAsset ramAsset = 
session.createAsset(gforgeArtifact.version);

		ramAsset.setName(gforgeArtifact.name);
		ramAsset.setCommunity(session.getCommunity(communityName));
		ramAsset.setAssetType(session.getAssetType(gforgeArtifact.type));
		ramAsset.setShortDescription(gforgeArtifact.description);
		ramAsset.setDescription(gforgeArtifact.description);
			
		RAMFolderArtifact root = (RAMFolderArtifact)ramAsset.getArtifactsRoot();

		for(File file : gforgeArtifact.files) {
			LocalFileArtifact artifact = new LocalFileArtifact(
	new File(file.getCanonicalPath()));
			artifact.setName(file.getName());
			root.addArtifact(artifact);
		}
			
		RAMStatus status = 
session.put(ramAsset, new NullProgressMonitor());
		if(status.getCode() != RAMStatus.COMPLETE)
			throw new RuntimeException("Failed creating asset");
	}
}

/* Parsing instances of this class out of GForge is omitted */
class GForgeRelease {
	public File[] files;
	public String name;
	public String description;
	public String version;
	public String type;
}

See Using Rational Asset Manager Java API in the information center for a full explanation of the API.


Summary

IBM Rational Team Concert and Rational Asset Manager are both highly configurable. They provide a set of rich APIs that make it easy to replicate workflows from other systems, while providing a rich set of additional functionality. The tools provided by these applications helped us build a smooth and successful migration process. This article describes all of the tools that you need to start your own migration into these products.

Resources

Learn

Get products and technologies

  • Download Rational Team Concert from Jazz.net and try it free on up to 10 projects for as long as you want (requires registration).
  • Download a free trial version of Rational Asset Manager.
  • Evaluate other IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, DevOps
ArticleID=812970
ArticleTitle=How IBM migrated a large GForge installation to Rational environments
publish-date=05082012