Migrating data and Portal Delegated User Registries

Complete the following steps to migrate v5 Public Cloud data and Portal Delegated User Registries.

About this task

Use the migration utility to migrate your API Connect v5 Public Cloud data to a v10 deployment. Your v10 deployment must be up and running, with gateway services, provider organizations, user registries, and so on, ready to receive the migrated data.

Procedure

  1. Optional: Set up a temporary SMTP server to hold automatic emails without sending them during migration.

    During migration, if subscription approval is needed to enable an application to subscribe to a plan, notifications are sent to all users within the organization who have subscription approval permission. In some cases, this results in a large number of emails. To avoid sending these emails, set up a temporary email server prior to running the migration, and use it to collect all notifications rather than sending them to users' email accounts.

    If subscription approvals are not needed during your migration, you can skip this step.
  2. Optional: Set up a temporary LDAP server.

    A temporary LDAP server can be needed if you are temporarily working behind a firewall.

    1. Install a server:
      git clone https://github.com/veo-labs/ldap-server-mock
      npm install
    2. Creating the configuration files:
      • ldap-server-mock-conf.json
        {
          "port": 3004,
          "userLoginAttribute": "cn",
          "searchBase": "dc=test",
          "searchFilter": "(&(objectclass=person)(cn={{username}}))"
        }
      • users.json
        [
          {
            "dn": "cn=user,dc=test",
            "cn": "user-login"
          }
        ]
    3. Run the LDAP server:
      node server.js --conf=./ldap-server-mock-conf.json --database=./users.json
    4. Modifying a current LDAP provider.

      The following file is a dummy user registry file that maps to a read-only test server. The fields in bold font (admin_dn and endpoint) have been changed to fit the dummy server configuration. Change these on all user registries that need to be fixed.

      api_version: 2.0.0
      configuration:
        admin_dn: cn=user,dc=test 
        admin_password: xxxx
        authenticated_bind: "true"
        authentication_method: search_dn
        bind_prefix: ""
        bind_suffix: ""
        protocol_version: "3"
        search_dn_base: ""
        search_dn_filter_prefix: (examplelegacyuid=
        search_dn_filter_suffix: )
        search_dn_scope: sub
      endpoint:
        endpoint: ldap://172.20.20.11:3004
        tls_client_profile_url: idurl://..\..\..\..\admin-org\tls-client-profiles\tls-profile-for-ldap-1.0.0.yml
      integration_url: file://..\..\..\..\integrations\user_registry\ldap.yml
      metadata:
        v5_id: xxxxxx40cf2fc09a9ba0d6c
        v5_scope: '["api"]'
        v5_type: ldap
      name: xxxxxxxxxxxxxx
      title: xxxxxxxxxxxxxxxxxxxxxx
      type: user_registry
  3. Map v5 resource names to v10 resources names as explained in Mapping v5 resource names to v10.

    Most deployments adjust the names or quantities of gateway services, provider organizations, and user registries when setting up a new environment for v10. Be sure to specify any changed v10 destinations for the v5 data in a migration mapping file so that the migration utility can automatically move the data.

  4. If you used the PDUR feature in v5 Public Cloud, map your Portal Delegated User Registries to the equivalent registries that you created in v10, as explained in Mapping v5 PDUR registries to v10.

    API Connect v10 supports new registries and you no longer need the PDUR feature. Instead, you can configure functionally equivalent registries in v10 that are maintained by the Management subsystem and migrate users to the new registries.

  5. Convert the v5 API definitions to formats required by v10 DataPower API Gateway as explained in Migrating v5 APIs to v10 API Gateway format.
    Note: If you are migrating your APIs to a DataPower v5-compatible Gateway, skip this step because there is no need to convert your APIs.
  6. Remove unneeded configuration objects.

    Inspect the set of configuration objects and their attributes, and remove the ones that are not needed. In many cases, some of the configuration objects present in the v5 system are not needed in the new v10 environment. For example, there is no value in preserving inactive users, obsolete apps, obsolete consumer organizations, and so on.

    When deleting objects, make sure that you do not delete objects that have other dependent objects, such as products with subscriptions on them or users that are active. If you delete a consumer organization, make sure you also delete all apps and subscriptions associated with that consumer organization. Be aware of any groups or visibility settings that include these consumer organizations.

  7. Upload the v5 data to v10.

    To upload the v5 data, log in to your v10 deployment with the migration utility and push the data with the migration utility. The admin organization and its data, and then the provider organization and all of its assets, are imported into v10. You can push to v10 as many times as needed to correct errors and provide missing information.

    1. Get the log-in information from your v10 deployment.

      Log in to your v10 deployment and note down the <v10_host_name> from the URL:

      For example, in the following URL, the v10 host name is apicdev10.acme.com:

      https://apicdev10.acme.com/manager/
    2. On the computer where your extracted v5 Public Cloud data is stored, use the migration utility to log in to your v10 deployment as an API Connect administrator:
      Note: On Windows: open a command window with Administrator privileges, and omit the "./" from the command.
      ./apicm login -s <v10_host_name> -r admin/default-idp-1 -u admin

      For example:

      ./apicm login -s apicdev10.acme.com -r admin/default-idp-1 -u admin
    3. Use the migration utility to push your v5 Public Cloud admin organization data to the v10 deployment:

      In the same directory where you ran the apicm: archive:unpack command, run the following command where <v10_host_name> is the host name:

      ./apicm archive:push <v10_host_name> cloud/admin-org

      For example:

      ./apicm archive:push apicdev10.acme.com cloud/admin-org

      The output from this command looks like the following example:

      INFO[2022-06-22T12:54:25-07:00] Saving copy of: saveErrorLog to: logs/archive-push20220622T12-54-25error.log
      INFO[2022-06-22T12:54:25-07:00] Saving copy of: saveInfoLog to: logs/archive-push20220622T12-54-25info.log
      .........
      .........
      INFO[2022-06-22T14:46:35-07:00] archive:push for organization admin completed
    4. On the same computer, use the migration utility to log in to your v10 deployment's provider organization:

      Log in as an Owner of the provider organization.

      ./apicm login -s <v10_host_name> -r provider/default-idp-2 -u provider_org_owner

      For example:

      ./apicm login -s apicdev10.acme.com -r provider/default-idp-2 -u example_user
    5. Use the migration utility to push your v5 Public Cloud provider organization data to your v10 deployment:

      In the same directory where you ran the apicm: archive:unpack command, run the following command where <v10_host_name> is the host name and <v10_provider_organization_name> is the name of the provider organization:

      ./apicm archive:push <v10_host_name> cloud/provider-orgs/<v10_provider_organization_name>/

      For example:

      ./apicm archive:push apicdev10.acme.com cloud/provider-orgs/my-provider-org/

      Table 1 describes optional parameters that you can include when you push your v5 Public Cloud provider organization data to your v10 deployment.

      Table 1. Optional parameters for the apicm archive:push command
      Optional parameter Description
      --catalogs=<cat_1,cat_2,cat_n> Push data for only the specified Catalogs (if you don't want to push the entire provider organization). When a catalog is mapped, specify the v10 Catalog name.
      --dry-run Create a test directory similar to cloud (but titled .workdir-<apim_admin_server>) where you can view how the data will be loaded into your new system. This option also displays the objects that will fail during migration, providing an opportunity to correct errors before actually importing data into v10 (see Migration troubleshooting).
      --no-drafts Do not push any draft (unpublished) APIs or Products. This option can optionally be used with the --catalogs parameter. This option cannot be used with the --drafts-only option.
      --drafts-only Only push draft (unpublished) APIs or Products and do not push any published APIs or Products. This option cannot be used with --catalogs=a,b,c or the --no-drafts option.
      --overwrite Overwrite all existing assets in the v10 deployment. Omit this parameter if you want to retain existing data in v10; for example, if you run the migration in iterations.
      --apigw-only Optional flag. Use when migrating to a DataPower API Gateway.

      The output from this command looks like the following example:

      INFO[2022-06-22T12:54:25-07:00] Saving copy of: saveErrorLog to: logs/archive-push20220622T12-54-25error.log
      INFO[2022-06-22T12:54:25-07:00] Saving copy of: saveInfoLog to: logs/archive-push20220622T12-54-25info.log
      .........
      .........
      INFO[2022-06-22T14:46:35-07:00] archive:push for organization <v10_provider_organization_name> completed
      Troubleshooting:
      • The apicm login token expires after eight hours. If the token expires while apicm archive:push is running, apicm generates errors due to lack of the necessary access permissions. There are two workarounds:
        • Log in again and re-run apicm archive:push. The token is refreshed with each new login.
        • Extend the token expiration time by logging in again with an optional argument: –expireHours=<number_of_hours> (see Token expiration.)
      • If you observe error messages in your log output, consult the Migration troubleshooting topic for information on resolving the problems, and then run the migration again. You can push to v10 as many times as needed to correct errors and provide missing information.
  8. When migration successfully finishes, complete the following steps:
    1. Apply any customized configurations from your v5 deployment (see Customizations).
    2. Optional: Switch the v10 provider organization ownership to the identity that owned it in v5.

      This step might be necessary if you created and migrated the provider organization using a different identity (such as administrative account) than the user identity of the v5 provider org. Note that the UI does not provide this function, but you can use the API /orgs/{org}/transfer-owner to switch ownership.

    3. If Stripe billing integrations were included in your migration, configure the Stripe integrations in your v10 Developer Portal sites; see Configuring Stripe in the Developer Portal for detailed instructions. For more information about the billing feature in v10, see Monetizing your Products.