CCM server fails to connect to MQTT Broker

2022-05-04 07:35:02,032 [    Default Executor-thread-32] FATAL .team.repository.service.mqtt.internal.MqttService  - CRJAZ3040E Failed to connect to MQTT Broker at tcp://<server>:1883 (client id=0). Error: Unable to connect to server
Unable to connect to server (32103) - java.net.ConnectException: Connection refused (Connection refused)
    at org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:79)
    at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:650)
    at java.lang.Thread.run(Thread.java:812)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:380)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:236)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:218)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    at java.net.Socket.connect(Socket.java:666)
    at org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:70)
    ... 2 more
2022-05-04 07:35:38,182 [    Default Executor-thread-32] FATAL .team.repository.service.mqtt.internal.MqttService  - CRJAZ3040E Failed to connect to MQTT Broker at tcp://<server>:1883 (client id=0). Error: MqttException
MqttException (0) - java.net.NoRouteToHostException: No route to host (Host unreachable)
    at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:38)
    at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:664)
    at java.lang.Thread.run(Thread.java:812)
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:380)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:236)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:218)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    at java.net.Socket.connect(Socket.java:666)
    at org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:70)
    at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:650)
    ... 1 more

2022-05-04T07:25:46.943-05:00 imaserver tcp.c:360: Set error "Unable to create the endpoint socket." (156)
2022-05-04T07:25:46.943-05:00 logger tcp.c:3029: 2022-05-04T07:25:46.943-05:00 Transport    imaserver  CWLNA1102 E: Unable to start TCP endpoint: Endpoint="AdminEndpoint" Port=9089 Error="Unable to create the endpoint socket." RC=156 TcpError="Address family not supported by protocol" Errno=97.
2022-05-04T07:25:46.944-05:00 imaserver tcp.c:3041: Start TCP endpoint: name=!MQConnectivityEndpoint transport=tcp port=9081 secure=0 secprof=
2022-05-04T07:25:46.944-05:00 imaserver pluginAdmin.c:824: Start plug-in protocol: pluginTerminated=1
2022-05-04T07:25:46.966-05:00 imaserver pluginAdmin.c:1290: /opt/ibm/imaserver/bin/updatePlugins.sh invoked successfully
2022-05-04T07:25:46.966-05:00 imaserver plugin_config.c:131: Configugre plug-in: path=(null)
2022-05-04T07:25:46.966-05:00 imaserver ssl.c:1444: Create TLS context for: endpoint=forwarder profileName=forwarder methodName=TLSv1.2 ciphers=ECDHE-ECDSA-AES128-GCM-SHA256 certFile=memory keyFile=memory isServer=0 sslopt=16778239 useClientCert=0
2022-05-04T07:25:46.968-05:00 imaserver imaSnmpAgentInit.c:161: SNMP Service is not enabled
2022-05-04T07:25:46.968-05:00 imaserver storeMemory.c:1013: Store internal parameters: DiskRootPath /var/messagesight/store/com.ibm.ism/, DiskTransferBlockSize 33554432, DiskClearStoredFiles 0, PhysicalMemSizeBytes 16824401920, CompactMemBytesHWM 20189282304 (60 %), CompactMemBytesLWM 16824401920 (50 %)
2022-05-04T07:25:46.968-05:00 imaserver ismutil.c:1700: Create thread diskUtil: [Read_and_Write_Generations_from_to_Disk] handle=0x7f8754fe9700 data=0x257f680 context=(nil) value=0 affinity=0xff priority=0
2022-05-04T07:25:46.968-05:00 imaserver storeMemory.c:1025: Store internal parameters: DiskBlockSizeBytes 4096
2022-05-04T07:25:46.968-05:00 diskUtil storeDiskUtils.c:1387: The ism_store_diskUtilsThread thread is started (0)
2022-05-04T07:25:46.974-05:00 imaserver ismutil.c:1700: Create thread store: [Store] handle=0x7f8727fff700 data=(nil) context=(nil) value=0 affinity=0xff priority=0
2022-05-04T07:25:46.974-05:00 imaserver storeShmPersist.c:2222: ism_storePersist_init: dirInfo: pdir=0x760d000 , path=/store/persist/ , block=4096, fdir=38, plen=15
2022-05-04T07:25:46.974-05:00 imaserver storeShmPersist.c:2271: PersistThread write buffer size: 20447232 1048576
2022-05-04T07:25:46.978-05:00 store storeMemory.c:12285: The store thread is started
2022-05-04T07:25:47.010-05:00 imaserver storeShmPersist.c:2431: ism_storePersist_init: read  PState: cid=11846, curI=0, curJ=1, genTr=0, Cold=0, len=28
2022-05-04T07:25:47.010-05:00 imaserver storeShmPersist.c:2453: ism_storePersist_init: Header of CPM_0_0
2022-05-04T07:25:47.013-05:00 imaserver storeShmPersist.c:497: line=2456 ,MgmtHeader: StrucId=abcdaaab, GenId=1, State=1, PoolsCount=2, DescriptorStructSize=40, SplitItemStructSize=48, Version=20140101, MemSizeBytes=1051590656, ActiveGenIndex=0, ActiveGenId=2, NextAvailableGenId=3
2022-05-04T07:25:47.013-05:00 imaserver storeShmPersist.c:2459: ism_storePersist_init: Header of CPG_0_0
2022-05-04T07:25:47.014-05:00 imaserver storeShmPersist.c:490: line=2462 , GenHeader: StrucId=abcdaaaa, GenId=2, State=1, PoolsCount=1, DescriptorStructSize=40, SplitItemStructSize=48, Version=20140101, MemSizeBytes=525795328
2022-05-04T07:25:47.014-05:00 imaserver storeShmPersist.c:2465: ism_storePersist_init: Header of ST_0_0
2022-05-04T07:25:47.017-05:00 imaserver storeShmPersist.c:2471: ST_0_0 header: ts=1651629566795449160, cid=11845, len=4096, nst=1
2022-05-04T07:25:47.017-05:00 imaserver storeShmPersist.c:2453: ism_storePersist_init: Header of CPM_0_1
2022-05-04T07:25:47.017-05:00 imaserver storeShmPersist.c:497: line=2456 ,MgmtHeader: StrucId=abcdaaab, GenId=1, State=1, PoolsCount=2, DescriptorStructSize=40, SplitItemStructSize=48, Version=20140101, MemSizeBytes=1051590656, ActiveGenIndex=0, ActiveGenId=2, NextAvailableGenId=3
2022-05-04T07:25:47.017-05:00 imaserver storeShmPersist.c:2459: ism_storePersist_init: Header of CPG_0_1
2022-05-04T07:25:47.023-05:00 imaserver storeShmPersist.c:490: line=2462 , GenHeader: StrucId=abcdaaaa, GenId=2, State=1, PoolsCount=1, DescriptorStructSize=40, SplitItemStructSize=48, Version=20140101, MemSizeBytes=525795328
2022-05-04T07:25:47.023-05:00 imaserver storeShmPersist.c:2465: ism_storePersist_init: Header of ST_0_1
2022-05-04T07:25:47.029-05:00 imaserver storeShmPersist.c:2471: ST_0_1 header: ts=1651632502121505282, cid=11846, len=4096, nst=2
2022-05-04T07:25:47.029-05:00 imaserver storeShmPersist.c:2453: ism_storePersist_init: Header of CPM_1_0
2022-05-04T07:25:47.029-05:00 imaserver storeShmPersist.c:2459: ism_storePersist_init: Header of CPG_1_0
2022-05-04T07:25:47.029-05:00 imaserver storeShmPersist.c:2465: ism_storePersist_init: Header of ST_1_0
2022-05-04T07:25:47.030-05:00 imaserver storeShmPersist.c:2471: ST_1_0 header: ts=0, cid=0, len=0, nst=0
2022-05-04T07:25:47.030-05:00 imaserver storeShmPersist.c:2453: ism_storePersist_init: Header of CPM_1_1
2022-05-04T07:25:47.030-05:00 imaserver storeShmPersist.c:2459: ism_storePersist_init: Header of CPG_1_1
2022-05-04T07:25:47.030-05:00 imaserver storeShmPersist.c:2465: ism_storePersist_init: Header of ST_1_1
2022-05-04T07:25:47.030-05:00 imaserver storeShmPersist.c:2471: ST_1_1 header: ts=0, cid=0, len=0, nst=0
2022-05-04T07:25:47.030-05:00 imaserver storeMemory.c:1171: Mapped store memory at address 0x7f86a2a40000.
2022-05-04T07:25:47.030-05:00 imaserver storeMemory.c:1233: Recovering store from disk persist data.
2022-05-04T07:25:47.030-05:00 imaserver storeShmPersist.c:4667: ism_storePersist_doRecovery: cI=0, cJ=1, cid=11846
2022-05-04T07:25:47.031-05:00 imaserver storeShmPersist.c:4710: ism_storePersist_doRecovery: calling ism_storeDisk_readFile for CPM_0_1 of size 36864
2022-05-04T07:25:47.031-05:00 imaserver storeShmPersist.c:4710: ism_storePersist_doRecovery: calling ism_storeDisk_readFile for CPG_0_1 of size 49152
2022-05-04T07:25:47.032-05:00 diskUtil storeDiskUtils.c:1523: ism_store_diskUtilsThread: A disk task completed successfully: task=2, priority=0, file=/store/persist//CPM_0_1, size=36864.000000, time=0.001196 sec, rate=30823592.958669 Bytes/sec
2022-05-04T07:25:47.032-05:00 diskUtil storeShmPersist.c:4553: ism_storePersist_readCB called with file= CPM_0_1 , rc= 0
2022-05-04T07:25:47.032-05:00 imaserver storeShmPersist.c:4759: ism_storePersist_doRecovery: calling ism_storeDisk_expandGenerationData for file CPM_0_1 of size 1051590656
2022-05-04T07:25:47.033-05:00 diskUtil storeDiskUtils.c:1523: ism_store_diskUtilsThread: A disk task completed successfully: task=2, priority=0, file=/store/persist//CPG_0_1, size=49152.000000, time=0.000904 sec, rate=54360015.660356 Bytes/sec
2022-05-04T07:25:47.033-05:00 diskUtil storeShmPersist.c:4553: ism_storePersist_readCB called with file= CPG_0_1 , rc= 0
2022-05-04T07:25:47.043-05:00 imaserver storeShmPersist.c:497: line=4773 ,MgmtHeader: StrucId=abcdaaab, GenId=1, State=1, PoolsCount=2, DescriptorStructSize=40, SplitItemStructSize=48, Version=20140101, MemSizeBytes=1051590656, ActiveGenIndex=0, ActiveGenId=2, NextAvailableGenId=3
2022-05-04T07:25:47.044-05:00 imaserver storeMemory.c:5533: Store validation completed. StrucId 0xabcdaaab, Version 20140101, GenId 1, DescriptorStructSize 40, SplitItemStructSize 48, TotalMemSizeBytes 2103181312, MemSizeBytes 1051590656, MgmtSmallGranuleSizeBytes 128, MgmtGranuleSizeBytes 1024, PoolsCount 2, InMemGensCount 0, Role 2, SessionId 0050569019ca76a54e60, SessionCount 8, HaveData 1, WasPrimary 1, ActiveGenIndex 0, ActiveGenId 2
2022-05-04T07:25:47.047-05:00 imaserver storeShmPersist.c:4759: ism_storePersist_doRecovery: calling ism_storeDisk_expandGenerationData for file CPG_0_1 of size 525795328
2022-05-04T07:25:47.190-05:00 imaserver storeShmPersist.c:490: line=4770 , GenHeader: StrucId=abcdaaaa, GenId=2, State=1, PoolsCount=1, DescriptorStructSize=40, SplitItemStructSize=48, Version=20140101, MemSizeBytes=525795328
2022-05-04T07:25:47.190-05:00 imaserver storeShmPersist.c:4790: ism_storePersist_doRecovery: calling ism_store_persistReplayFile for cI=0, cJ=1, cid=11846
2022-05-04T07:25:48.153-05:00 imaserver storeShmPersist.c:2600: ism_store_persistReplayFile: cycleId missmatch: found 11844, expected 11846 (nIo=45416)
2022-05-04T07:25:48.153-05:00 imaserver storeShmPersist.c:2695: ism_store_persistReplayFile: Finish playing 50952 STs from 45416 recs of file ST_0_1 ; cI=0, cJ=1, cid=11846
2022-05-04T07:25:48.153-05:00 imaserver storeShmPersist.c:4801: ism_storePersist_doRecovery_4801: INC_CID incremented cid to 11847
2022-05-04T07:25:48.153-05:00 imaserver storeShmPersist.c:4806: ism_storePersist_doRecovery: calling ism_store_persistReplayFile for cI=0, cJ=0, cid=11847
2022-05-04T07:25:48.153-05:00 imaserver storeShmPersist.c:2600: ism_store_persistReplayFile: cycleId missmatch: found 11845, expected 11847 (nIo=0)
2022-05-04T07:25:48.153-05:00 imaserver storeShmPersist.c:2695: ism_store_persistReplayFile: Finish playing 0 STs from 0 recs of file ST_0_0 ; cI=0, cJ=0, cid=11847
2022-05-04T07:25:48.153-05:00 imaserver storeMemory.c:5533: Store validation completed. StrucId 0xabcdaaab, Version 20140101, GenId 1, DescriptorStructSize 40, SplitItemStructSize 48, TotalMemSizeBytes 2103181312, MemSizeBytes 1051590656, MgmtSmallGranuleSizeBytes 128, MgmtGranuleSizeBytes 1024, PoolsCount 2, InMemGensCount 0, Role 2, SessionId 0050569019ca76a54e60, SessionCount 8, HaveData 1, WasPrimary 1, ActiveGenIndex 0, ActiveGenId 2
2022-05-04T07:25:48.153-05:00 imaserver storeMemory.c:1256: Memory store header is valid. Will perform restart recovery
2022-05-04T07:25:48.153-05:00 imaserver storeRecovery.c:1069: Recovery memory has been set to MaxMemoryBytes, curMem= 8412200960
2022-05-04T07:25:48.153-05:00 imaserver storeHighAvailability.c:3141: storeHA(getLocalIf): no local interface found for the given HA.ReplicationNIC (|10.8.180.39|)
2022-05-04T07:25:48.153-05:00 imaserver storeHighAvailability.c:731: ism_storeHA_start: failed to attach to the given HA.LocalReplicationNIC (10.8.180.39) using the given HA.ReplicationProtocol (1)
2022-05-04T07:25:48.153-05:00 imaserver storeHighAvailability.c:740: Set error "An HA configuration parameter is not valid: parameter name=HA.LocalReplicationNIC value=10.8.180.39." (510)
2022-05-04T07:25:48.153-05:00 imaserver storeMemoryHA.c:280: Failed to start the HA component. error code 1000
2022-05-04T07:25:48.154-05:00 imaserver main.c:575: Server init and start process is complete. ServerState=11 adminMode=1 RC=510
2022-05-04T07:25:48.154-05:00 imaserver admin.c:587: In ism_admin_pause(): admnPauseState=0 adminMode=1
2022-05-04T07:25:48.154-05:00 logger admin.c:1612: 2022-05-04T07:25:48.154-05:00 Admin        imaserver  CWLNA6120 E: The IBM MessageSight detects an error. The error code is 510. The error string is "An HA configuration parameter is not valid: parameter name=HA.LocalReplicationNIC value=10.8.180.39.".
2022-05-04T07:34:23.790-05:00 timer1 crlprofile.c:629: Endpoint !MQConnectivityEndpoint not found in configuration database. May have been deleted at some point or a not a tcp endpoint. No need for CRL timer. Canceling timer.
2022-05-04T07:34:23.790-05:00 timer1 crlprofile.c:704: Canceling CRL timer for endpoint !MQConnectivityEndpoint
2022-05-04T07:34:23.790-05:00 timer1 crlprofile.c:728: CRL timer for endpoint !MQConnectivityEndpoint is canceled.