Configuring multiple data center topologies

With the multi-master asynchronous replication, you link a set of catalog service domains. The connected catalog service domains are then synchronized using replication over the links. You can define the links using properties files, at run time with Java Management Extensions (JMX) programs, or with command-line utilities. The set of current links for a domain is stored in the catalog service. You can add and remove links without restarting the catalog service domain that hosts the data grid.

Before you begin

See Planning multiple data center topologies for more information about multi-master replication topologies and design considerations. You can configure links among catalog service domains with the server properties file to form the topology during server startup. You can also configure links at run time.
If you are using loaders in your multi-master replication topology, you must plan how you are going to maintain accurate data between the data centers. The approaches that you can use vary depending on the topology you are using. For more information, see Loader considerations in a multi-master topology.

Important:

When you use multimaster replication (MMR) with eviction enabled or are using the ObjectMap.invalidate() method, the operations on those keys are not flowed to foreign domains. When foreign domain operations do not receive those keys, an entry might be invalidated in one domain, but the entry in the other domain is still there because it has not expired yet. This behavior causes map size output for the two domains to be different. When a domain is restarted and it receives the data from the other domain, the eviction data for that domain is different from the other, since all entries in the restarted domain now have a new access time.

Procedure

Define links in the server properties file for the catalog server of each catalog service domain in the topology, for bootstrap purposes.
See Server properties file for more information about defining this file for the catalog server.

Important: Property names are case-sensitive.
Local Domain name:
Specify the name of the catalog service domain for the current catalog server:
domainName=domain1
An optional list of foreign domain names:
Specify the names of catalog service domains to which you want to link in the multi-master replication topology:
foreignDomains=domain2,domain3,domain4
An optional list of endpoints for the foreign domain names:
Specifies the connection information for the catalog servers of the foreign domains:
domain2.endPoints=hostB1:2809, hostB2:2809
If a foreign domain has multiple catalog servers, specify all of them.
Use the xscmd utility or JMX programming to add or remove links at run time.
The links for a domain are kept in the catalog service in replicated memory. This set of links can be changed at any time by the administrator without requiring a restart of this domain or any other domain. The xscmd utility includes several options for working with links.

The xscmd utility connects to a catalog service and thus a single catalog service domain. Therefore, the xscmd utility can be used to create and destroy links between the domain it attaches to and any other domain.
Use the command line to create a link, for example:
```
xscmd –c establishLink -cep host:2809 -fd dname -fe fdHostA:2809,fdHostB:2809
```
The command establishes a new link between the local domain and the foreign domain named dname. The dname catalog service is running at fdHostA:2809 and fdHostB:2809. The local catalog service domain has a catalog service listener host and port of host:2809. Specify all catalog service endpoints from the foreign domain so that fault tolerance connectivity to the domain is possible. Do not use a single host:port pair for the catalog service of the foreign catalog service domain.

You can use any local catalog service JVM with xscmd and the using the -cep option. If the catalog server is hosted in a WebSphere Application Server deployment manager, then the port is usually 9809.

The ports specified for the foreign domain are not JMX ports. They are the usual ports you would use for eXtreme Scale clients.

After the command to add a new link is issued, the catalog service instructs all containers under its management to begin replicating to the foreign domain. A link is not needed on both sides. It is only necessary to create a link on one side.
Use the command line to remove a link, for example:
```
xscmd –c dismissLink -cep host:2809 -fd dname
```
The command connects to the catalog service for a domain and instructs it to stop replicating to a specific domain. A link needs to be dismissed from one side only.
Attention: You can run the establish or dismiss link commands multiple times. If the link does not enter the correct status or is disjoint, run the command again.

Examples

Figure 1. Example: Link between catalog service domains

Suppose that you want to configure a two-domain setup involving catalog service domains A and B.

Here is the server properties file for the catalog server in domain A:

domainName=A
foreignDomains=B
B.endPoints=hostB1:2809, hostB2:2809

Here is the server properties file for the catalog server in domain B. Notice the similarity between the two property files.

domainName=B
foreignDomains=A
A.endPoints=hostA1:2809,hostA2:2809

After the two domains are started, then any data grids with the following characteristics are replicated between the domains.

Has a private catalog service with a unique domain name
Has the same data grid name as other grids in the domain
Has the same number of partitions as other data grids in the domain
Has a data grid that uses the fixed partition placement strategy. A data grid that is configured to use a per container placement strategy cannot be replicated.
Has the same number of partitions (it might or might not have the same number and types of replicas)
Has the same data types being replicated as other data grids in the domain
Has the same map set name, map names, and dynamic map templates as other data grids in the domain

The replication policy of a catalog service domain is ignored.

The preceding example shows how to configure each domain to have a link to the other domain, but it is necessary only to define a link in one direction. This fact is especially useful in hub and spoke topologies, allowing a much simpler configuration. The hub property file does not require updates as spokes are added, and each spoke file needs only to include hub information. Similarly, a ring topology requires each domain to have only a link to the previous and next domain in the ring.

Figure 2. Example: Hub and spoke topology

The hub and four spokes (domains A, B, C, and D) has server property files like the following examples.

domainName=Hub

Spoke A has the following server properties:

domainName=A
foreignDomains=Hub
Hub.endPoints=hostH1:2809, hostH2:2809

Spoke B has the following server properties:

domainName=B
foreignDomains=Hub
Hub.endPoints=hostH1:2809, hostH2:2809

Spoke C has the following server properties:

domainName=C
foreignDomains=Hub
Hub.endPoints=hostH1:2809, hostH2:2809

Spoke D has the following properties:

domainName=D
foreignDomains=Hub
Hub.endPoints=hostH1:2809, hostH2:2809

What to do next

If you need to check or troubleshoot problems with the links between your catalog service domains, you can use the xscmd utility. For more information about commands to help you troubleshoot your multiple data center configuration, see Troubleshooting multiple data center configurations.
You can provide a custom collision arbiter to resolve collisions between the catalog service domains. See Developing custom arbiters for multi-master replication for more information.