Custom threshold created but no incidents appear in UI

A user created a simple Threshold to check physical disk space but its not firing, no Incidents are generated from it. Other Thresholds exist, but none generate any Incidents.

Symptoms

The user confirmed that agent side received the threshold in the private_situations.xml file. The user confirmed that agent is firing the event by checking in the agent asfActivity log. The user checks the alarmeventsrc logs for any exceptions. The alarmeventsrc logs show exceptions in alarmforwarder with unknown host "java.net.UnknownHostException". The alarm is unable to send any events to CEM.

Resolving the problem

The user needs to make sure the "global.ingress.domain" definition includes the full hostname. If the shortname of the host was used, ensure that is resolvable within master and worker nodes. You can add an ip address and host alias within the deployment of the alarm service. To add the entry, edit the deployment: kubectl edit deploy ibmcloudappmgmt-alarmeventsrc and under the second spec entry, add the following with the correct yaml spacing:

hostAliases:
  - ip: "x.x.x.x"
    hostnames:
    - "_your-host-shortname_"

where "x.x.x.x" is the ip address and hostAliases is under the deployment spec. For more information, see: Adding entries to Pod /etc/hosts with HostAliases