Error 500 on Alerts page

The following error might be displayed in the Web GUI Event Viewer:

Event dataset error
HEMDP0389E: Load failed
Recovery will be attempted automatically on the next refresh

The following messages might be observed in the SystemOut.log file:

com.micromuse.wave.servlets.ael.Views.getDefaultColumnTitles FINE: customer:Customer
com.micromuse.ncw.tools.NCWToolsConfiguration.getMenus FINE: Getting menu config for user icpadmin and datasources AGG_P
000000c7 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: An exception was thrown by one of the service methods of the servlet [AELConfiguration] in application [isc]. Exception created : [java.lang.IndexOutOfBoundsException: Index: xx, Size: xx
at java.util.ArrayList.rangeCheck(ArrayList.java:670)
at java.util.ArrayList.get(ArrayList.java:446)
at com.micromuse.ncoweb.nova.Nova.getNovaAt(Nova.java:759)
at com.micromuse.ncw.tools.menu.ServerMenuManager.processForSubMenus(ServerMenuManager.java:428)

Where xx in Index: xx is a number, for example, 16 or 18.

When you upgrade Netcool® Operations Insight® on OpenShift®, an error is displayed in the Alerts page.

Error 500 An error occurred while fetching data from the server. The response from the server was 504

While the root cause is unknown, the error can be related to the /home/netcool/app/gui/omnibus_webgui/etc/configstore/ncwMenus/alerts.nova file being corrupted. The file can contain repeated sub-menus, as in the following example:

menu(name="alerts",mnemonic="a",label="Alerts",acl="*")
{
  tool(shortcut="ctrl+a",name="acknowledge",mnemonic="a",label="Acknowledge",acl="*")
  tool(shortcut="ctrl+d",name="deacknowledge",mnemonic="d",label="Deacknowledge",acl="*")
  tool(shortcut="ctrl+n",name="createNewIncident",mnemonic="n",label="Create new incident",acl="*")
  tool(shortcut="ctrl+j",name="addToIncident",mnemonic="",label="Add to incident",acl="*")
  separator(acl="*")
  tool(shortcut="",name="prioritise",mnemonic="p",label="Prioritize",acl="*")
  tool(shortcut="",name="SuppressEscalate",mnemonic="S",label="Suppress/Escalate",acl="*")
  tool(shortcut="",name="takeownership",mnemonic="t",label="Take Ownership",acl="*")
  tool(shortcut="",name="userassign",mnemonic="s",label="User Assign",acl="*")
  tool(shortcut="",name="groupassign",mnemonic="g",label="Group Assign",acl="*")
  tool(shortcut="",name="delete",mnemonic="e",label="Delete",acl="*")
  separator(acl="*")
  menu(name="Ping",acl="*")
  menu(name="Telnet",acl="*")
  tool(shortcut="",name="Tracepath",mnemonic="",label="Tracepath",acl="*")
  menu(name="alerts",mnemonic="a",label="Alerts",acl="*")
  {
    tool(shortcut="ctrl+a",name="acknowledge",mnemonic="a",label="Acknowledge",acl="*")
    tool(shortcut="ctrl+d",name="deacknowledge",mnemonic="d",label="Deacknowledge",acl="*")
    tool(shortcut="ctrl+n",name="createNewIncident",mnemonic="n",label="Create new incident",acl="*")
    tool(shortcut="ctrl+j",name="addToIncident",mnemonic="",label="Add to incident",acl="*")
    separator(acl="*")
    tool(shortcut="",name="prioritise",mnemonic="p",label="Prioritize",acl="*")
    tool(shortcut="",name="SuppressEscalate",mnemonic="S",label="Suppress/Escalate",acl="*")
    tool(shortcut="",name="takeownership",mnemonic="t",label="Take Ownership",acl="*")
    tool(shortcut="",name="userassign",mnemonic="s",label="User Assign",acl="*")
    tool(shortcut="",name="groupassign",mnemonic="g",label="Group Assign",acl="*")
    tool(shortcut="",name="delete",mnemonic="e",label="Delete",acl="*")
    separator(acl="*")
    menu(name="Ping",acl="*")
    menu(name="Telnet",acl="*")
    tool(shortcut="",name="Tracepath",mnemonic="",label="Tracepath",acl="*")
    tool(shortcut="",name="LaunchRunbook",mnemonic="",label="Launch Runbook",acl="*")
  }
  tool(shortcut="",name="LaunchRunbook",mnemonic="",label="Launch Runbook",acl="*")
}

Complete the following steps to resolve the NOVA file corruption error:

1. Optional: The local drive should have the latest data. To refresh the latest data from the database to the local drive, run the following commands.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAModulesUpdate --username console_admin_user_ID --password console_admin_password

   /home/netcool/app/jazz/ui/bin/consolecli.sh RestoreConfigFromDb --username console_admin_user_ID --password console_admin_password

   Where console_admin_user_ID is smadmin and console_admin_password is smadmin's password.
2. Manually fix the corrupted /home/netcool/app/gui/omnibus_webgui/etc/configstore/ncwMenus/alerts.nova file. If possible, use backup copies that were created before corruption or from previous installations. After editing the file, the repeated sub-menus are removed, as in the following example:

   menu(name="alerts",mnemonic="a",label="Alerts",acl="*")
   {
     tool(shortcut="ctrl+a",name="acknowledge",mnemonic="a",label="Acknowledge",acl="*")
     tool(shortcut="ctrl+d",name="deacknowledge",mnemonic="d",label="Deacknowledge",acl="*")
     tool(shortcut="ctrl+n",name="createNewIncident",mnemonic="n",label="Create new incident",acl="*")
     tool(shortcut="ctrl+j",name="addToIncident",mnemonic="",label="Add to incident",acl="*")
     separator(acl="*")
     tool(shortcut="",name="prioritise",mnemonic="p",label="Prioritize",acl="*")
     tool(shortcut="",name="SuppressEscalate",mnemonic="S",label="Suppress/Escalate",acl="*")
     tool(shortcut="",name="takeownership",mnemonic="t",label="Take Ownership",acl="*")
     tool(shortcut="",name="userassign",mnemonic="s",label="User Assign",acl="*")
     tool(shortcut="",name="groupassign",mnemonic="g",label="Group Assign",acl="*")
     tool(shortcut="",name="delete",mnemonic="e",label="Delete",acl="*")
     separator(acl="*")
     menu(name="Ping",acl="*")
     menu(name="Telnet",acl="*")
     tool(shortcut="",name="Tracepath",mnemonic="",label="Tracepath",acl="*")
   } 

3. Ensure that the database is synchronized with local data. Update the database with data from the local drive.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAUpdate --username console_admin_user_ID --password console_admin_password

The same unknown root cause might corrupt other files, such as the global and system filters in Web GUI.

• /home/netcool/app/gui/omnibus_webgui/etc/data/global/filter.xml
• /home/netcool/app/gui/omnibus_webgui/etc/data/system/filter.xml

You can verify whether the filter files are corrupted in the Web GUI filter builder.

• If the filter builder cannot be loaded, it might be that the filter.xml file is no longer a well-formed XML file, such as, the first three characters of the filter.xml are missing.
• If the filter builder loads, some default filters can be missing, as in the following example:

  <filter
  name="AllEvents"
  description=""
  os="OMNIBUS"
  mode="basic"
  metriclabel="Metric:"
  metricshow="Average"
  metricof="Severity"
  view="Default"
  viewtype="global"
  sql="Severity &gt;= 0">
  </filter>

Complete the following steps to resolve the filter file corruption error:

1. Optional: The local drive should have the latest data. To refresh the latest data from the database to the local drive, run the following commands.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAModulesUpdate --username console_admin_user_ID --password console_admin_password

   /home/netcool/app/jazz/ui/bin/consolecli.sh RestoreConfigFromDb --username console_admin_user_ID --password console_admin_password

   Where where console_admin_user_ID is smadmin and console_admin_password is smadmin's password.
2. Manually fix the corrupted filter.xml files. If possible, use backup copies that were created before corruption or from previous installations. The default files are available for comparison and to copy missing default filters.
   • /home/netcool/app/gui/omnibus_webgui/etc/default/data/global/filter.xml
   • /home/netcool/app/gui/omnibus_webgui/etc/default/data/system/filter.xml
   Note: If the default file is copied as a whole file to replace the existing corrupted file, all custom filters will be lost.
3. Ensure that the database is synchronized with local data. Update the database with data from the local drive.

   /home/netcool/app/jazz/ui/bin/consolecli.sh ForceHAUpdate --username console_admin_user_ID --password console_admin_password