Using Config Sync to replicate consumer-side catalog data

Use the API Connect Config Sync utility to replicate consumer-side data from a source catalog in an API Connect cluster to a target catalog in a different API Connect cluster, so that API calls can be served by either API Connect cluster interchangeably to enable hybrid deployments and high-availability style scenarios.

API Connect Config Sync (apic-configsync) can be run either as a stand-alone binary, or a schedule with a Kubernetes cronjob, to facilitate the unidirectional replication of consumer-side data (consumer organizations, members, apps, subscriptions, credentials) from a catalog in a source API Connect cluster to a corresponding catalog in a separate, target API Connect cluster.

The following image shows how you can use Config Sync to replicate consumer-side data from a source deployment to a target deployment:

Config Sync can be run as frequently as required, and detects the differences between the replicated resource types in the source and target catalogs so that it efficiently replicates only the new changes that have been made in the source instance.

Changes that are made in a target catalog can be synced back to the source by swapping the configuration, so that the original source catalog is the designated target and the original target is the designated source. This process is known as a reverse sync.

As shown in the following image, Config Sync helps to enable a range of hybrid cloud API topologies such as:

Independent disaster recovery instances
Software and SaaS hybrid scenarios
Geo-replication
Side-by-side version upgrade

Requirements

For API Connect Config Sync to run successfully, the following requirements that must be satisfied:

You are responsible for configuring the target cluster to match the source cluster, so that replication of consumer-side data can take place:
- The target API Connect cluster must be configured correctly including topology setup, provider organization creation, and catalog creation.
- The APIs and Products in the source catalog must be replicated in the target catalog by using either CI/CD or manual procedures.
- The target catalog settings (including any spaces if they are enabled in the source catalog) must be configured to match the configuration of the source catalog. This includes compatible gateways and any extensions, user registries, and any other relevant catalog settings (such as lifecycle approvals, application lifecycle, and so on).
- Compatible onboarding user registries must be configured in the target catalog with the same name and type as in the source catalog. If this is not possible, then use the USER_REGISTRY_NAME_MAPPING variable.
- All OAuth providers must be configured on the target catalog and spaces.
- All API user registries must be configured on the target catalog and spaces.
- All TLS client profiles must be configured on the target catalog and spaces.
The Config Sync utility uses the API Connect Provider REST API, so you must have connectivity (into both the source and target clusters) with sufficient provider organization user permissions.
The metadata.source_id key is reserved for use by Config Sync.
- Because the underlying creation of resources in the target catalog uses the API Connect Provider REST API, the IDs and URLs associated with resources in the target catalog are different from their corresponding resources in the source catalog (only the cluster-agnostic data of a resource is synchronized).
- To track which resources in the source catalog map to which resources in the target catalog, Config Sync uses the metadata field of a particular resource. When a new resource is created in the target catalog, the metadata field of that resource is set to contain a key that is named source_id with a value equal to the ID of the corresponding resource in the source catalog that is being synchronized.
API Connect Config Sync also uses the name field of some resources to determine a match. This is typically done for resources that are not synchronized by the utility but are needed in the target catalog for the synchronization to be successful; for example, User Registries, Consumer Org Groups, Roles, and published Products (the name and version fields are used for product matching).

Note: The first time Config Sync runs, it might take longer to complete than subsequent runs require (depending on the number of resources to synchronize). The first run of the utility creates all new resources in the target cluster; subsequent runs perform only synchronization of any data that was added, updated, or deleted since the last run of the utility.

Limitations

API Connect Config Sync has the following limitations:

Config Sync does not back up data; it merely replicates the contents of a source catalog to a target catalog. You must still take periodic backups to ensure the safety of your data.
A reverse sync can be performed only if the original source cluster and the original target cluster are at the same VRM (Version.Release.Milestone) level. For example, API Connect 10.0.8.1 and API Connect 10.0.8.0 are at the same VRM level even though they are at different fix pack levels. A reverse sync to a different version, release, or milestone is not supported.
Source and target deployments operate independently of each other:
- API rate limiting on Plans is counted independently in the source catalog and the target catalog.
- OIDC tokens are not replicated. OIDC tokens that are generated in the source cluster are not valid within the target cluster.
- Analytics data is tracked independently in the source cluster and the target cluster. Analytics data is not replicated or aggregated across clusters.
- Developer Portal configuration is not replicated. If you want to enable consumer organization access in the target catalog, you must replicate your Developer Portal site for consistency (restore the site with UUID/URL update where needed).
Config Sync operates on a single source-target catalog pair at a time.
If you want to sync multiple different source-target catalog pairs, create a separate cronjob for each pair.

Supported versions of API Connect

Config Sync can synchronize consumer-side resources across different versions of API Connect clusters. The target cluster version must always be greater or equal to the source cluster version.

Consumption reporting

The analytics subsystem records the number of resources that are processed during a config sync. You can see the total number of resources that were processed for a config sync at provider organization and catalog scope in the consumption dashboard of the API Manager UI.

For users on call volume licensing, the amount of data that is synchronized is converted to API calls and is included in your call volume license.