IBM Spectrum Symphony multicluster connection types

The multicluster feature within IBM Spectrum Symphony Advanced Edition allows you to connect multiple lone IBM Spectrum Symphony clusters into a federation cluster. For the multicluster master cluster and lone clusters to communicate, they must establish a connection between the master and the multicluster proxy service (SMCP) enabled on the lone cluster.

 

Historically, IBM Spectrum Symphony multicluster connections were inbound only: a lone cluster initiates connection to the multicluster master cluster. The IBM Spectrum Symphony repository server (RS) also initiates a connection to the multicluster RS.

 

Starting with IBM Spectrum Symphony 7.2.1, you can also use outbound connections, so that the multicluster master initializes connection to the lone clusters, rather than waits for them to come from the lone clusters. Moreover, you can have both inbound and outbound connections configured at the same time; as long as each IBM Spectrum Symphony cluster use one type of connection at a time. For example, if there are two IBM Spectrum Symphony clusters, one lone cluster's proxy can connect inbound to the multicluster master, while the other proxy listens for outbound connection requests from the multicluster master cluster.

 

Using outbound connections can be advantageous in certain business environments. For example, consider the use case in this diagram:

 

The multicluster master and a lone cluster (Cluster1) are on premises clusters, and another lone cluster (Cluster2) is off premise (for example, on the cloud, such as on IBM Cloud). For security, all outbound communications (from on-premises to the cloud) must go through a port-forwarding server in the DMZ. The communication is one way, inbound: hosts on the cloud cannot connect to the hosts on premises; however, hosts on premises can connect to hosts on the cloud. To address this issue, you need to change the connection direction from inbound to outbound.

 

Once you have successfully configured outbound connections, you can continue to use all multicluster features (such as cluster monitoring, service deployment, host repurposing, and workload replacement), with no difference between inbound and outbound connections.

 

Note: There is a limitation in IBM Spectrum Symphony 7.2.1 for the workload replacement feature: the IBM Spectrum Symphony client cannot trigger global workload replacement because there is no support for outbound connection between the IBM Spectrum Symphony client and the multicluster master in version 7.2.1.

 

Configuring outbound connections

Here are the steps to configure outbound connections, so that the multicluster master cluster initializes connection requests, rather than waits for them to come from the lone clusters:

 

1. On the lone cluster, add the SMC_PROXY_INBOUND_CONNECTION  parameter, and give it a value of Y, to the EGO service configuration file: ($EGO_ESRVDIR/esc/conf/services/smcp.xml):

<ego:EnvironmentVariable name="SMC_PROXY_INBOUND_CONNECTION">Y</ego:EnvironmentVariable>

where:

Parameter name	Default value	Valid values
SMC_PROXY_INBOUND_CONNECTION	N	Y: Indicates multicluster outbound connections.   N or undefined: Indicates multicluster inbound connections. Note: The term inbound in this parameter is relative to the lone cluster; that is, SMC_PROXY_INBOUND_CONNECTION means inbound to the lone cluster, outbound from the multicluster master cluster. Therefore, to set outbound connections, this value should be Y.

 

2. Restart EGOSC to reload the new configuration and start the multicluster proxy (SMCP) service:

[root@myhost]# egosh ego restart

 

After EGOSC starts, start the SMCP service:

[root@myhost]# egosh service start SMCP

 

3. From the multicluster master, trigger an invitation to add the lone cluster that you configured in the previous steps:

[root@myhost]# smcadmin cluster add –c cluster_name –m master_list –p vemkd_port

 

       Note that you must have SMC_CLUSTER_CONTROL permission to execute this CLI.

where:

• -c cluster_name
  Specifies the lone cluster name. Type any value for the cluster name (as long as it is a unique name), and this entry will be added into the SmcConfigurations.xml configuration file. When the lone cluster registers with the multicluster master, the master will update the real cluster name.
• -m master_list
  Specifies the fully qualified domain name of the master host of the lone cluster. If you also have master candidate hosts, list them, separating each master with a comma.
• -p vemkd_port
  Specifies the VEMKD listening port for the lone cluster.

For example:

[root@myhost]# smcadmin cluster add -c abcde -m host1.example.com -p 7870
 

4. Check that the multicluster master successfully connects to the lone cluster:

[root@myhost]# smcadmin cluster list

NAME      STATUS     MEMBERSHIP

abcde     ok         nonmember

 

5. Allow the lone cluster to join the federation cluster and participate in multicluster features:

[root@myhost]# smcadmin cluster join -c abcde

join: The system successfully allowed cluster "abcde" to participate in IBM Spectrum Symphony (Multicluster).

[root@myhost]# smcadmin cluster list

NAME      STATUS     MEMBERSHIP

abcde     ok         member

 

6. Verify that the multicluster connection type for the lone cluster is outbound:  

[root@myhost]# smcadmin cluster view -c abcde

Name                              : abcde

Membership                        : member

Status                            : ok

IBM Spectrum Symphony Version     : 7.2.1

Last Heartbeat                    : 2018-06-25 13:21:25

…

Connection Type                   : Outbound

 

Troubleshooting outbound connections

If your outbound connection configuration is unsuccessful, you will see an error status from the CLI output:

[root@myhost]# smcadmin cluster list

NAME     STATUS     MEMBERSHIP 

ca       error      nonmember

 

For even more details about the error, review the IBM Spectrum Symphony multicluster log files:

Failure reason	Error message in log	Log file and location	Suggested resolution
VEMKD for the lone cluster is unreachable	Unable to connect to EGO using URL SMC_PROXY:xxxx : vem_openbyname() failed(vemerrno=25): The cluster is not responding.	$EGO_TOP/smc/logs/smc.master.${ HOSTNAME}.log	1. Check the network of the master host within the lone cluster. 2. Check the VEMKD service on the master cluster.
SMCP is unreachable	Unable to connect to EGO using URL SMC_PROXY:xxxx : vem_locate() failed(vemerrno=0): .	$EGO_TOP/smc/logs/smc.master.${ HOSTNAME}.log	1. Check the EGO service of the SMCP service. 2. Check the TCP socket port of the lone cluster. 3. Check the SMC_PROXY_INBOUND_CONNECTION parameter in the scmp.xml file.
IBM Spectrum Symphony RS is unreachable	IBM Spectrum Symphony (Multicluster) proxy failed to connect to the repository server. Check that the repository server is running.	$EGO_TOP/smc/logs/smc.master.${ HOSTNAME}.log	Check the RS service on the lone cluster.
IBM Spectrum Symphony RS is unreachable	Connecting to RS server failed with error: Transport endpoint is not connected.	$EGO_ESRVDIR/rs/logs/rs.${HOSTNAME}.log	Check the RS service on the lone cluster.

 

The multicluster master will try 10 times every 30 seconds and continues to try every 30 minutes. Therefore, if the status of outbound connections shows “error”, there may be an issue with your outbound connection configuration. Remove the outbound lone cluster configuration so that the multicluster master will stop connecting at those intervals:

[root@myhost]# smcadmin cluster delete -c abacde

delete: The system successfully sent a request to remove the outbound cluster, that does not contain valid configuration settings ("abcde"), from participating in IBM Spectrum Symphony multicluster features.

 

Note: Use the smcadmin cluster delete command only for deleting invalid outbound lone cluster configuration. To remove the inbound lone cluster or the connected outbound lone cluster, run smcadmin cluster retire or smcadmin cluster reject instead.

 

RS and outbound connections

When communication between the multicluster master and multicluster proxy is configured to outbound, communication between the multicluster RS and lone cluster RS will change to outbound automatically. That is, when the multicluster master receives a request to synchronize service packages to the lone cluster, before telling the multicluster proxy to trigger the lone cluster RS to synchronize, the multicluster master will tell the multicluster RS to connect to the lone cluster RS first. The lone cluster RS will then use this connection and synchronize the service packages.

 

You can synchronize service packages from the multicluster RS to lone cluster RS using the multicluster management console (Resources > Service Packages > Deployment) or by running the CLI (smcadmin service sync) as before. There is no difference between inbound and outbound connections.

 

For more information

As always, IBM Knowledge Center provides detailed information about IBM Spectrum Symphony features. To read more about IBM Spectrum Symphony multicluster connections, and now to set up outbound connections, see https://www.ibm.com/support/knowledgecenter/SSZUMP_7.2.1/multicluster_install/smc_connections.html.

Got questions?

We’d love to hear from you. If you’ve got comments or questions, add a comment below. You can also start a topic in our IBM Spectrum Symphony forum.

 

Tags:  ibm_spectrum_symphony_721 multicluster outbound_connections ibm_spectrum_symphony_mul... multicluster_connection_t...