CES network events

The following table lists the events that are created for the CES Network component.

Table 1. Events for the CES network component
Event	Event Type	Severity	Call Home	Details
ces_bond_degraded	STATE_CHANGE DEGRADED	WARNING	no	Message: Some secondaries of the CES-bond {0} went down.
				Description: Some of the CES-bond parts are malfunctioning.
				Cause: Some secondaries of the network bond are not functioning properly.
				User Action: Check the bonding configuration, network configuration, and cabling of the malfunctioning secondaries of the bond.
ces_bond_down	STATE_CHANGE FAILED	ERROR	no	Message: All secondaries of the CES-bond {0} are down.
				Description: All secondaries of a CES-network bond are down.
				Cause: All secondaries of this network bond went down.
				User Action: Check the bonding configuration, network configuration, and cabling of all secondaries of the CES-bond.
ces_bond_up	STATE_CHANGE HEALTHY	INFO	no	Message: All secondaries of the CES bond {0} are working as expected.
				Description: This CES bond is functioning properly.
				Cause: N/A
				User Action: N/A
ces_disable_node_network	INFO	INFO	no	Message: Network was disabled.
				Description: Clean up by using the mmchnode --ces-disable command. Accordingly, the network configuration is modified.
				Cause: Informational message, which is cleaned up by using the mmchnode --ces-disable command.
				User Action: N/A
ces_enable_node_network	INFO	INFO	no	Message: Network was enabled.
				Description: Called to handle any network-specific issues involved after running the mmchnode --ces-enable command. Accordingly, the network configuration is modified.
				Cause: Informational message, which is called by using the mmchnode --ces-enable command.
				User Action: N/A
ces_ips_hostable	TIP	INFO	no	Message: All declared CES-IPs could be hosted on this node.
				Description: All declared CES-IPs could be hosted on this node.
				Cause: N/A
				User Action: N/A
ces_ips_not_hostable	TIP	TIP	no	Message: One ore more CES IP cannot be hosted on this node (no interface).
				Description: One or more CES IP address cannot be hosted on this node because no active network interface for this subnet is available.
				Cause: One ore more CES IP address cannot be hosted on this node.
				User Action: Check whether interfaces are active and the CES group assignment is correct. For more information, run the mmces address list --full- list command.
ces_load_monitord_bad	STATE_CHANGE DEGRADED	WARNING	no	Message: Execution failed. Exit code: ${0}.
				Description: CES monitor daemon cannot run the CES load monitor successfully.
				Cause: An issue with automated ip distribution is detected.
				User Action: If CES IP distribution causes a problem, then set address policy to node affinity and move ips manually.
ces_load_monitord_good	STATE_CHANGE HEALTHY	INFO	no	Message: Execution successfully. Exit code: ${0}.
				Description: CES monitor daemon runs CES load monitor with success.
				Cause: N/A
				User Action: N/A
ces_many_rx_errors	STATE_CHANGE DEGRADED	ERROR	FTDC upload	Message: NIC {0} had many RX errors since the last monitoring cycle.
				Description: NIC used by CES had many RX errors since the last monitoring cycle.
				Cause: The '/proc/net/dev' folder lists much more RX errors for this adapter since the last monitoring cycle.
				User Action: Check the network cabling and network infrastructure.
ces_many_tx_errors	STATE_CHANGE DEGRADED	ERROR	FTDC upload	Message: CES NIC {0} had many TX errors since the last monitoring cycle.
				Description: This CES-related NIC had many TX errors since the last monitoring cycle.
				Cause: The /proc/net/dev lists more TX errors for this adapter since the last monitoring cycle.
				User Action: Check the network cabling and network infrastructure.
ces_monitord_down	STATE_CHANGE DEGRADED	WARNING	no	Message: The CES-IP background monitor is not running. CES-IPs cannot be configured.
				Description: The CES-IP background monitor is not running. It might be terminated.
				Cause: The CES-IP background monitor is not running.
				User Action: If the CES-IP background monitor stops without any known reason, check the local /var file system. Restart it by using the mmces node resume --start command.
ces_monitord_ok	STATE_CHANGE HEALTHY	INFO	no	Message: The CES-IP background monitor is running.
				Description: The CES-IP background monitor is running.
				Cause: N/A
				User Action: N/A
ces_monitord_warn	INFO	WARNING	no	Message: The IBM Storage Scale CES IP assignment monitor (mmcesmonitord) alive check cannot be executed, which can be a timeout issue.
				Description: The CES IP monitor daemon state cannot be determined due to a problem.
				Cause: The CES IP monitor daemon state cannot be determined due to a problem.
				User Action: Find potential issues for this kind of failure in the /var/adm/ras/mmfs.log and /var/adm/ras/mmsysmonitor.log files.
ces_network_affine_ips_not_defined	STATE_CHANGE DEGRADED	WARNING	no	Message: No CES IP addresses can be applied on this node. Check group membership of node and IP addresses.
				Description: No CES IP addresses found, which can be hosted on this node. IPs should be in the global pool or in the group for which this node is a member.
				Cause: No valid CES IP found which could be hosted on this node.
				User Action: Use the mmces address add to add CES IP addresses to the global pool or to a group for which this node is a member.
ces_network_connectivity_down	STATE_CHANGE FAILED	ERROR	no	Message: CES NIC {0} cannot connect to the gateway.
				Description: This CES-related NIC cannot connect to the gateway.
				Cause: The gateway does not respond to the sent connections-checking packets.
				User Action: Check the network configuration of the network adapter, path to the gateway, and gateway itself.
ces_network_connectivity_up	STATE_CHANGE HEALTHY	INFO	no	Message: CES NIC {0} can connect to the gateway and responds to the sent connections-checking packets.
				Description: This CES-related network adapter can connect to the gateway.
				Cause: N/A
				User Action: N/A
ces_network_down	STATE_CHANGE FAILED	ERROR	no	Message: CES NIC {0} is down.
				Description: This CES-related network adapter is down.
				Cause: This network adapter was disabled.
				User Action: Enable this network adapter or check for problems in system logs.
ces_network_found	ADD_ENTITY	INFO	no	Message: CES NIC {0} was found.
				Description: A new CES-related network adapter was found.
				Cause: A new NIC, which is relevant for the CES, is listed by ip a.
				User Action: N/A
ces_network_ips_down	STATE_CHANGE DEGRADED	WARNING	no	Message: No CES IPs were assigned to this node.
				Description: No CES IPs were assigned to any network adapter of this node.
				Cause: No network adapters have the CES-relevant IPs, which makes the node unavailable for the CES clients.
				User Action: If CES is FAILED, then analyse the reason. Ensure if there are sufficient IPs in the CES pool for this node and extend the pool if needed. Also, verify that the root directory has enough free space, as low space can prevent the CES IP assignment. Free up space if required, then run the mmces node suspend command followed by the mmces node resume command.
ces_network_ips_not_assignable	STATE_CHANGE FAILED	ERROR	FTDC upload	Message: No NICs are set up for CES.
				Description: No network adapters are properly configured for CES.
				Cause: There are no network adapters with a static IP, matching any of the IPs from the CES pool.
				User Action: Setup the static IPs and netmasks of the CES NICs in the network interface configuration scripts, or add new matching CES IPs to the pool. The static IPs must not be aliased.
ces_network_ips_not_defined	STATE_CHANGE DEGRADED	WARNING	no	Message: No CES IP addresses have been defined.
				Description: No CES IP addresses have been defined. Run the mmces address add command to add CES IP addresses.
				Cause: No CES IP defined, but at least one CES IP is needed.
				User Action: Run the mmces address add command to add CES IP addresses. Check the group membership of IP addresses and nodes.
ces_network_ips_up	STATE_CHANGE HEALTHY	INFO	no	Message: CES-relevant IPs are served by found NICs.
				Description: CES-relevant IPs are served by network adapters.
				Cause: N/A
				User Action: N/A
ces_network_link_down	STATE_CHANGE FAILED	ERROR	no	Message: Physical link of the CES NIC {0} is down.
				Description: The physical link of this CES-related adapter is down.
				Cause: The LOWER_UP flag is not set for this NIC in the output of ip a.
				User Action: Check the network cabling and network infrastructure.
ces_network_link_up	STATE_CHANGE HEALTHY	INFO	no	Message: Physical link of the CES NIC {0} is up.
				Description: The physical link of this CES-related adapter is up.
				Cause: N/A
				User Action: N/A
ces_network_monitord_bad	STATE_CHANGE DEGRADED	WARNING	no	Message: Execution failed. Exit code: ${0}.
				Description: CES monitor daemon could not run CES network monitor successfully.
				Cause: An issue with automated CES network verification is detected.
				User Action: If protocol I/O is failing, then suspend this node.
ces_network_monitord_good	STATE_CHANGE HEALTHY	INFO	no	Message: Executed successfully. Exit code: {0}.
				Description: The CES monitor daemon runs the network monitor successfully. Automated network verification is working.
				Cause: N/A
				User Action: N/A
ces_network_up	STATE_CHANGE HEALTHY	INFO	no	Message: CES NIC {0} is up.
				Description: This CES-related network adapter is up.
				Cause: N/A
				User Action: N/A
ces_network_vanished	DEL_ENTITY	INFO	no	Message: CES NIC {0} has vanished.
				Description: One of CES-related network adapters cannot be detected anymore.
				Cause: One of the previously monitored NICs is not listed by ip a anymore.
				User Action: N/A
ces_no_rx_errors	STATE_CHANGE HEALTHY	INFO	no	Message: NIC {0} had no or a tiny number of RX errors.
				Description: The NIC used by CES had no or an insignificant number of RX errors.
				Cause: N/A
				User Action: N/A
ces_no_tx_errors	STATE_CHANGE HEALTHY	INFO	no	Message: CES NIC {0} had no or a tiny number of TX errors.
				Description: A CES-related NIC had no or an insignificant number of TX errors.
				Cause: N/A
				User Action: N/A
ces_startup_network	INFO	INFO	no	Message: CES network service was started.
				Description: The CES network has started.
				Cause: CES network IPs are started.
				User Action: N/A
dir_sharedroot_perm_ok	STATE_CHANGE HEALTHY	INFO	no	Message: The permissions of the sharedroot directory are correct: {0}.
				Description: The permissions of the sharedroot directory are sufficient.
				Cause: N/A
				User Action: N/A
dir_sharedroot_perm_problem	STATE_CHANGE DEGRADED	WARNING	no	Message: The permissions of the sharedroot directory are not sufficient {0}.
				Description: The cesSharedRoot directory must have 'rx' permissions for 'group' and 'others', since these are required for NFS 'rpcuser' to access its data inside of cesSharedRoot.
				Cause: The cesSharedRoot directory did not have 'rx' permissions for 'group' and 'others'.
				User Action: Provide 'rx' permissions for 'group' and 'others' for the cesSharedRoot directory.
handle_network_problem_info	INFO	INFO	no	Message: Handle network problem - Problem: {0}, Argument: {1}.
				Description: Information about network-related reconfigurations. For example, enable or disable IPs, and assign or unassign IPs.
				Cause: A change in the network configuration. Details are part of the information message.
				User Action: N/A
move_cesip_from	INFO	INFO	no	Message: Address {0} was moved from this node to {1}.
				Description: A CES IP address was moved from the current node to another node.
				Cause: Rebalancing of CES IP addresses.
				User Action: N/A
move_cesip_to	INFO	INFO	no	Message: Address {0} was moved from {1} to this node.
				Description: A CES IP address was moved from another node to the current node.
				Cause: Rebalancing of CES IP addresses.
				User Action: N/A
move_cesips_info	INFO	INFO	no	Message: A move request for ip addresses was executed. Reason {0}.
				Description: CES IP addresses can be moved in case of node failovers from one node to one or more other nodes. This message is logged on a node that is observing this, not necessarily on any affected node.
				Cause: A CES IP movement was detected.
				User Action: N/A