Estimating the efforts for IBM Business Process Manager migrations – why it’s not easy!
Werner_Tod 270002UTKA Visits (12030)
Revised: January 29, 2015
This blog entry describes the factors that need to be considered to estimate the necessary effort to migrate an existing environment of Lombardi Teamworks, WebSphere Lombardi Edition, WebSphere Process Server or an older version of IBM Business Process Manager to a new version of IBM Business Process Manager.
Note: It is always strongly recommended for a IBM Business Process Manager migration project to involve experts from IBM Services. The following web site has more information: http
Which factors influence BPM migration efforts?
As already mentioned, it is very difficult to create an accurate estimate for IBM Business Process Manager migration efforts because the sizing depends on so many very different parameters. For example:
Let's look at those parameters in more detail.
Chosen migration method
First of all, you need to make a very conscious decision regarding the migration method that will be applied during a migration process.
Before IBM Business Process Manager V8.5, there were other names for migration methods such as:
Note: An artifact migration has to be done almost always on top of a runtime migration or “Migrating business data and applications." The runtime migration ensures that the existing business data, as well as the running and completed process, task instances, and so on are preserved and attempts to keep the existing applications in the target environment "as is." It does NOT include the migration of applications to the code base of the target version of the product or guarantee that all applications will run without errors in the new environment.
In addition, migrations can either be done locally on the source systems, or they can be performed using additional or new hardware (remote migration).
For all of these migration methods you have to consider things like retaining running instances, keeping existing customizations, migrating existing integrations with back-end systems, and so on.
Also, you will probably need a different set of experts to perform different migration methods. For “Migrating business data and applications” or “Runtime migration” you need infrastructure and integration experts, while you need application development and integration experts for “Application migration” or “Artifact migration”. Skilled application developers might also be required in case an incompatibility of an existing application with the new IBM Business Process Manager causes issues during a runtime migration.
As you can imagine, all of these aspects can add immensely to the overall effort. So the seemingly simple question “How am I going to do the migration?” actually has a huge impact on the migration effort that is necessary to implement it.
Infrastructure layout and topology
Another key factor for the estimation of migration efforts is the topology of the existing installation and the intended topology of the target environment.
It is quite obvious that the migration of one single-cluster IBM Business Process Manager environment is significant less effort that the migration of four three- or four-cluster IBM Business Process Manager environments (Development, Test, Staging, Production). The Production environment maybe even implemented with high availability and disaster recovery (HA/DR) capabilities and spread over more than one location. Starting with IBM Business Process Manager V8.5, it is also possible to change the topology during migration, for example, changing an environment from a single-cluster setup to a three-cluster setup. Topology changes add extra effort to IBM Business Process Manager migrations (architecture and configuration work).
The performance of the source and target systems used in the migration, as well as the performance of the network connecting them, is another important consideration. One of the most time-consuming activities during migration is the migration of data in the database. Thus, the performance of the database server plays a key role. I have seen migrations where this step alone took more than 9 hours to complete. There were others where it even took several days!
System performance also must be considered as part of the migration effort when tuning the target environment of the IBM Business Process Manager migration. Each version of IBM Business Process Manager uses a different version of Java and WebSphere Application Server. As a result, each version needs to be tuned and optimized for scalability and performance in a different way.
Number and complexity of deployed applications
When you migrate old applications to a new version of IBM Business Process Manager, both their number and their complexity need to be considered when sizing the migration effort. In any case, you will need application development experts to actively support the migration; either the original developers of the applications or at least people skilled enough to change them down to the latest detail, if necessary. Only experts with that kind of skill will be able to resolve potential issues with migrated applications, such as APIs that might have changed and thus the application code needs to be adjusted. You might have process and task instances that cause problems during migration, such as dealing with orphaned tasks, failed events, and similar issues.
Some applications might just continue to run in the new environment as they did in the old one. However, you cannot always count on that. So sizing in at least some effort for assessing every migrated application is a good practice.
Number and complexity of implemented customizations
In cases where significant customization is implemented in the migration source environment, for example by customizing the out-of-the-box IBM Business Process Manager Process Portal, it is necessary to plan for some effort to migrate these changes over to the target environment. This scenario also applies to cases where an external UI is implemented for the users of IBM Business Process Manager and leverages the documented product APIs of IBM Business Process Manager. These custom UIs need to be thoroughly tested with the target version of the IBM Business Process Manager migration in order to make sure that they still work.
Amount of available supporting tooling
When you need to migrate a large number of applications you can either validate them manually in the target environment, which includes performing test runs on each one of them, or you can execute a suite of automated regression tests, which is significantly less effort. In the ideal case, those regression tests would have been developed together with the applications. In the next-best case, you would use the opportunity while migrating and developing these tests for future updates as part of the migration testing. In the worst case, you never have any automated tests and always have to manually test the proper function of the applications.
Anything that you can prepare beforehand that helps to validate the migrated environment is immensely helpful and reduces migration effort. Another example would be automated tests for the required back-end connectivity like the proper access to web services.
Availability of necessary skills for the migration
I have already mentioned the importance of skilled application developers for the migration. Another key skill that is required is the database administrator (DBA). There have been migrations that took longer than necessary because key skills were not available when they were needed. That is one of the reasons why a good project manager that looks after these things is one of the foundations for a successful migration project.
Amount and quality of the available migration documentation
As with any other IT project, proper preparation and documentation are key factors for success. You can save a lot of migration effort when you have all of the necessary information available before starting each migration activity. Lists with all of the IP addresses, ports, server names, and so on as well as lists with all application and module names, and so on should be there in every migration project.
Best practices for sizing BPM migrations
Even though the estimating game is not an easy one, there are some heuristics and best practices that can be applied in order to get a good understanding of the size and complexity of a given migration project.
Installing the new environment or environments
The installation of one clustered IBM Business Process Manager environment (base installation only, no complex back-end integration, and no tuning) typically takes between 3 and 5 days. The time depends on the complexity of the environment, such as the number of nodes and the availability of required resources. Again, this is one environment – Development, Test, Staging OR Production. The net effort gets bigger with every environment that is added. Some efforts can be done in parallel, which results in reduced elapsed time, but the net effort remains the same.
This effort does not include the configuration for high availability and disaster recovery (HA and DR) or the performance tuning of the target environments.
Additional configuration and tuning of the environment(s)
As mentioned already, depending on the IBM Business Process Manager product version of the target environments, different settings need to be applied in order to optimize the system performance for IBM Business Process Manager. This effort goes on top of the basic installation effort.
Configuring a target environment for high availability, disaster recovery, or both also introduces very significant extra effort. As a rule-of-thumb, the effort estimate for the environment setup should be doubled compared to a non-HA/DR environment. The actual effort depends on the level of high availability and resiliency to be achieved.
“Migrating business data and applications” (configuration, instances and data)
The migration of the IBM Business Process Manager runtime is a key element in the overall migration procedure. Following the installation of the product binaries and other preparation steps, this phase is where the existing configuration, applications and data get moved. Most issues occur in this phase of the migration, which sometimes requires you to roll back the migration, resolve the issue, and repeat several migration steps. If it is properly planned, rollbacks of migrations should only occur when developing a migration routine using a test or staging environment. It should definitely be avoided for migrations of production environments, which typically happen during rare, well-defined maintenance windows. In developing the runtime migration routine for a given production environment, the goal is to minimize the required effort and elapsed time, so that the migration and the subsequent regression / end-to-end test fit into the maintenance window / migration weekend.
Migrating the databases (part of “Migrating business data and applications”)
To get an estimate for the expected duration of the database migration, it is a good practice to use the time required to fully copy the database (over the same network that is used during the migration) and add some buffer to be on the safe side (for example, 10-20%). This estimate is for runtime migrations. For artifact migrations, the database effort is part of the installation (creation of new databases).
Starting with IBM Business Process Manager V8.5, clones of the existing databases can be used for the migration. This approach adds much more flexibility to the migration routine and greatly simplifies migration rollbacks. Cloning the databases is equivalent to taking a database backup, which is mandatory for migrations anyway, so no additional effort is introduced by this capability.
Migrating existing applications (“Artifact migration”)
In good cases, a runtime migration might successfully move all deployed applications and their running instances to the new environment. In many cases though, applications need to be adjusted in order for them to run with the new IBM Business Process Manager code. It is very hard to estimate how likely it is that an application will need to be modified. Very significant application development skills and knowledge about the changes introduced between the source and target version of the migrated product are required to make this assessment. Therefore, the infrastructure AND application development teams need to be involved in the planning of a migration as early as possible! At some point, each application needs to be migrated in order to fully leverage the capabilities of the new IBM Business Process Manager version and also in order to make it more future-proof in line with the IBM Business Process Manager product strategy.
The following heuristics for the effort to analyze and, if necessary, adjust existing applications have worked well for me (and others) in the past:
Note: These numbers are net effort; not elapsed time! In addition, the tasks cannot be arbitrarily executed in parallel.
Development environments vs. test, staging and production environments
The migration of development environments (Process Centers) is typically rather different from the migration of test, staging and production environments (Process Servers). Process Centers are almost always migrated using “Artifact migration” whereas Process Servers (especially for Production environments) tend to be migrated using “Migrating business data and applications” more often.
In order to get an estimate for the migration efforts for a production environment, take the time for all of the migration activities in the first migrated Process Server environment and determine how similar the environments are. The migration of the test or staging environments is almost always more effort and more complex than the production one because of the following factors:
Testing the migrated binaries, artifacts and processes is very often underestimated! Still, it is a crucial part of the migration project and should be included in the estimate. This applies to all forms of IBM Business Process Manager environments. Obviously, application testing should be done and completed in the development and test environments before rolling out the migrated applications to the staging and production environments. Infrastructure testing has to be done for all environments. Configuration tuning should be tested in development and test before applying the validated tuning to staging and production.
I hope that the insights presented in this article make it easier to answer the rather difficult question: “How long will it take to get me the new IBM Business Process Manager?”
Remember: Always run IBM Business Process Manager migrations as projects, with a project manager, a project plan, and the proper tracking of the project’s progress. Involve IBM Services, a skilled IBM business partner, or both.