Error 500 on Alerts page

The following error might be displayed in the Web GUI Event Viewer:

Event dataset error
HEMDP0389E: Load failed
Recovery will be attempted automatically on the next refresh

The following messages might be observed in the SystemOut.log file:

com.micromuse.wave.servlets.ael.Views.getDefaultColumnTitles FINE: customer:Customer
com.micromuse.ncw.tools.NCWToolsConfiguration.getMenus FINE: Getting menu config for user icpadmin and datasources AGG_P
000000c7 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: An exception was thrown by one of the service methods of the servlet [AELConfiguration] in application [isc]. Exception created : [java.lang.IndexOutOfBoundsException: Index: xx, Size: xx
at java.util.ArrayList.rangeCheck(ArrayList.java:670)
at java.util.ArrayList.get(ArrayList.java:446)
at com.micromuse.ncoweb.nova.Nova.getNovaAt(Nova.java:759)
at com.micromuse.ncw.tools.menu.ServerMenuManager.processForSubMenus(ServerMenuManager.java:428)

Where xx in Index: xx is a number, for example, 16 or 18.

When you upgrade Netcool® Operations Insight® on OpenShift®, an error is displayed in the Alerts page.

Error 500 An error occurred while fetching data from the server. The response from the server was 504

While the root cause is unknown, the error can be related to the /home/netcool/app/gui/omnibus_webgui/etc/configstore/ncwMenus/alerts.nova file being corrupted. The file can contain repeated sub-menus, as in the following example:

menu(name="alerts",mnemonic="a",label="Alerts",acl="*")
{
  tool(shortcut="ctrl+a",name="acknowledge",mnemonic="a",label="Acknowledge",acl="*")
  tool(shortcut="ctrl+d",name="deacknowledge",mnemonic="d",label="Deacknowledge",acl="*")
  tool(shortcut="ctrl+n",name="createNewIncident",mnemonic="n",label="Create new incident",acl="*")
  tool(shortcut="ctrl+j",name="addToIncident",mnemonic="",label="Add to incident",acl="*")
  separator(acl="*")
  tool(shortcut="",name="prioritise",mnemonic="p",label="Prioritize",acl="*")
  tool(shortcut="",name="SuppressEscalate",mnemonic="S",label="Suppress/Escalate",acl="*")
  tool(shortcut="",name="takeownership",mnemonic="t",label="Take Ownership",acl="*")
  tool(shortcut="",name="userassign",mnemonic="s",label="User Assign",acl="*")
  tool(shortcut="",name="groupassign",mnemonic="g",label="Group Assign",acl="*")
  tool(shortcut="",name="delete",mnemonic="e",label="Delete",acl="*")
  separator(acl="*")
  menu(name="Ping",acl="*")
  menu(name="Telnet",acl="*")
  tool(shortcut="",name="Tracepath",mnemonic="",label="Tracepath",acl="*")
  menu(name="alerts",mnemonic="a",label="Alerts",acl="*")
  {
    tool(shortcut="ctrl+a",name="acknowledge",mnemonic="a",label="Acknowledge",acl="*")
    tool(shortcut="ctrl+d",name="deacknowledge",mnemonic="d",label="Deacknowledge",acl="*")
    tool(shortcut="ctrl+n",name="createNewIncident",mnemonic="n",label="Create new incident",acl="*")
    tool(shortcut="ctrl+j",name="addToIncident",mnemonic="",label="Add to incident",acl="*")
    separator(acl="*")
    tool(shortcut="",name="prioritise",mnemonic="p",label="Prioritize",acl="*")
    tool(shortcut="",name="SuppressEscalate",mnemonic="S",label="Suppress/Escalate",acl="*")
    tool(shortcut="",name="takeownership",mnemonic="t",label="Take Ownership",acl="*")
    tool(shortcut="",name="userassign",mnemonic="s",label="User Assign",acl="*")
    tool(shortcut="",name="groupassign",mnemonic="g",label="Group Assign",acl="*")
    tool(shortcut="",name="delete",mnemonic="e",label="Delete",acl="*")
    separator(acl="*")
    menu(name="Ping",acl="*")
    menu(name="Telnet",acl="*")
    tool(shortcut="",name="Tracepath",mnemonic="",label="Tracepath",acl="*")
    tool(shortcut="",name="LaunchRunbook",mnemonic="",label="Launch Runbook",acl="*")
  }
  tool(shortcut="",name="LaunchRunbook",mnemonic="",label="Launch Runbook",acl="*")
}

Complete the following steps to resolve the NOVA file corruption error:

1. Optional: The local drive should have the latest data. To refresh the latest data from the database to the local drive, run the following commands.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAModulesUpdate --username console_admin_user_ID --password console_admin_password

   /home/netcool/app/jazz/ui/bin/consolecli.sh RestoreConfigFromDb --username console_admin_user_ID --password console_admin_password

   Where console_admin_user_ID is smadmin and console_admin_password is smadmin's password.
2. Manually fix the corrupted /home/netcool/app/gui/omnibus_webgui/etc/configstore/ncwMenus/alerts.nova file. If possible, use backup copies that were created before corruption or from previous installations. After editing the file, the repeated sub-menus are removed, as in the following example:

   menu(name="alerts",mnemonic="a",label="Alerts",acl="*")
   {
     tool(shortcut="ctrl+a",name="acknowledge",mnemonic="a",label="Acknowledge",acl="*")
     tool(shortcut="ctrl+d",name="deacknowledge",mnemonic="d",label="Deacknowledge",acl="*")
     tool(shortcut="ctrl+n",name="createNewIncident",mnemonic="n",label="Create new incident",acl="*")
     tool(shortcut="ctrl+j",name="addToIncident",mnemonic="",label="Add to incident",acl="*")
     separator(acl="*")
     tool(shortcut="",name="prioritise",mnemonic="p",label="Prioritize",acl="*")
     tool(shortcut="",name="SuppressEscalate",mnemonic="S",label="Suppress/Escalate",acl="*")
     tool(shortcut="",name="takeownership",mnemonic="t",label="Take Ownership",acl="*")
     tool(shortcut="",name="userassign",mnemonic="s",label="User Assign",acl="*")
     tool(shortcut="",name="groupassign",mnemonic="g",label="Group Assign",acl="*")
     tool(shortcut="",name="delete",mnemonic="e",label="Delete",acl="*")
     separator(acl="*")
     menu(name="Ping",acl="*")
     menu(name="Telnet",acl="*")
     tool(shortcut="",name="Tracepath",mnemonic="",label="Tracepath",acl="*")
   } 

3. After fixing the corrupted file, use the ForceHAUpdate parameter instead of the ForceHAModulesUpdate parameter. Ensure that the database is synchronized with local data. Update the database with data from the local drive.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAUpdate --username console_admin_user_ID --password console_admin_password

   Note: Running the command with the ForceHAUpdate parameter restarts the WebSphere® Application Server server. To avoid this restart, configure the webgui pod in debug mode.

   To configure the webgui pod in debug mode, go to the Red Hat® OpenShift UI. Click Workload > StatefulSet > WebGUI pod > Environment. Scroll to the end and click Add more. Add a new variable with name: DEBUG_MODE and value: true. Click Save to restart the webgui pod. If the pod does not restart, manually restart the pod.

The same unknown root cause might corrupt other files, such as the global and system filters in Web GUI.

• /home/netcool/app/gui/omnibus_webgui/etc/data/global/filter.xml
• /home/netcool/app/gui/omnibus_webgui/etc/data/system/filter.xml

You can verify whether the filter files are corrupted in the Web GUI filter builder.

• If the filter builder cannot be loaded, it might be that the filter.xml file is no longer a well-formed XML file, such as, the first three characters of the filter.xml are missing.
• If the filter builder loads, some default filters can be missing, as in the following example:

  <filter
  name="AllEvents"
  description=""
  os="OMNIBUS"
  mode="basic"
  metriclabel="Metric:"
  metricshow="Average"
  metricof="Severity"
  view="Default"
  viewtype="global"
  sql="Severity &gt;= 0">
  </filter>

• Ensure that the filter.xml file contains only one filterbox element, as in the following example:

  <?xml ...> <filterbox> ... </filterbox>

Complete the following steps to resolve the filter file corruption error:

1. Optional: The local drive should have the latest data. To refresh the latest data from the database to the local drive, run the following commands.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAModulesUpdate --username console_admin_user_ID --password console_admin_password

   /home/netcool/app/jazz/ui/bin/consolecli.sh RestoreConfigFromDb --username console_admin_user_ID --password console_admin_password

   Where where console_admin_user_ID is smadmin and console_admin_password is smadmin's password.
2. Manually fix the corrupted filter.xml files. If possible, use backup copies that were created before corruption or from previous installations. The default files are available for comparison and to copy missing default filters.
   • /home/netcool/app/gui/omnibus_webgui/etc/default/data/global/filter.xml
   • /home/netcool/app/gui/omnibus_webgui/etc/default/data/system/filter.xml
   Note: If the default file is copied as a whole file to replace the existing corrupted file, all custom filters will be lost.
3. Ensure that the database is synchronized with local data. Update the database with data from the local drive.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAUpdate --username console_admin_user_ID --password console_admin_password

   Note: Running the command with the ForceHAUpdate parameter restarts the WebSphere Application Server server. To avoid this restart, configure the webgui pod in debug mode.

   To configure the webgui pod in debug mode, go to the Red Hat OpenShift UI. Click Workload > StatefulSet > WebGUI pod > Environment. Scroll to the end and click Add more. Add a new variable with name: DEBUG_MODE and value: true. Click Save to restart the webgui pod. If the pod does not restart, manually restart the pod.

The same unknown root cause might corrupt other files, such as the global and system views in Web GUI.

• /home/netcool/app/gui/omnibus_webgui/etc/data/global/view.xml
• /home/netcool/app/gui/omnibus_webgui/etc/data/system/view.xml

Complete the following steps to resolve the view file corruption error:

1. Optional: The local drive should have the latest data. To refresh the latest data from the database to the local drive, run the following commands.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAModulesUpdate --username console_admin_user_ID --password console_admin_password

   /home/netcool/app/jazz/ui/bin/consolecli.sh RestoreConfigFromDb --username console_admin_user_ID --password console_admin_password

   Where where console_admin_user_ID is smadmin and console_admin_password is smadmin's password.
2. Manually fix the corrupted view.xml files. If possible, use backup copies that were created before corruption or from previous installations. The default files are available for comparison and to copy missing default filters.
   • /home/netcool/app/gui/omnibus_webgui/etc/default/data/global/view.xml
   • /home/netcool/app/gui/omnibus_webgui/etc/default/data/system/view.xml
   Note: If the default file is copied as a whole file to replace the existing corrupted file, all custom views will be lost.
3. Ensure that the database is synchronized with local data. Update the database with data from the local drive.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAUpdate --username console_admin_user_ID --password console_admin_password

   Note: Running the command with the ForceHAUpdate parameter restarts the WebSphere Application Server server. To avoid this restart, configure the webgui pod in debug mode.

   To configure the webgui pod in debug mode, go to the Red Hat OpenShift UI. Click Workload > StatefulSet > WebGUI pod > Environment. Scroll to the end and click Add more. Add a new variable with name: DEBUG_MODE and value: true. Click Save to restart the webgui pod. If the pod does not restart, manually restart the pod.