Troubleshooting
Problem
Alerts page when accessed throws Error 500 with below message:
Code: FETCH_ERROR_500
Details:
status: 500
url: https://<hostname>/api/p/hdm_datalayer_tabular_alerts_api?properties=severity&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Fbusiness%2Fcriticality%27&properties=state&properties=summary&properties=type.classification&properties=sender.name&properties=resource.name&properties=firstOccurrenceTime&properties=lastOccurrenceTime&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Frunbook%27&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Ftopology%2Fresource%27&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Fseasonal-occurrence%27&properties=relatedContextualStoryIds&properties=relatedStoryIds&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Flad%2Fresolutions%27&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Flad%2Ftemplates%27&properties=suppressed&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Fanomaly%27&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Fincidentcontrol%27&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Frelationship%2Fcausal%27+and+insights.source%3D%27aiops.ibm.com%2Finsight-source%2Frelationship%2Fcausal%2Ftopological-group%27&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Frelationship%2Fcausal%27+and+insights.source%3D%27aiops.ibm.com%2Finsight-source%2Frelationship%2Fcausal%2Fcustom%27&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Frelationship%2Fcausal%27+and+insights.source%3D%27aiops.ibm.com%2Finsight-source%2Frelationship%2Fcausal%2Ftemporal%27&properties=owner&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Fgolden-signal%27&properties=id&properties=signature&properties=%40insights.type%3D%27aiops.ibm.com%2Finsight-type%2Frelationship%2Fcausal-union%27&properties=details.OutageID&maxrows=true&limit=20000&orderby=state%3Adesc&orderby=severity%3Adesc&orderby=lastOccurrenceTime%3Adesc&filter=%28details.OutageID+%21%3D+%27%27%29
responseText: {
"code": "ECONNABORTED",
"message": "Response timeout of 30000ms exceeded",
"level": "fatal",
"req_id": "853bca20-63a0-11ef-b3d0-7b670a9eabcf"
}
Symptom
You will see the below entries in aiops-ir-core-ncodl-api log:
{"name":"request","hostname":"aiops-ir-core-ncodl-api-7f96d5d699-lddwm","pid":17,"subscription":"cfd95b7e-3bc7-4006-a4a8-a73a79c71255","user":"n114905","client":"unknown","req_id":"af4382b0-6416-11ef-aa1a-a7dba1956b91","http":"GET /irdatalayer.aiops.io/active/v1/summary/alerts?groupby=severity","level":30,"stack":null,"req":{"method":"GET","url":"/irdatalayer.aiops.io/active/v1/summary/alerts?groupby=severity"},"res":{"statusCode":500,"header":"HTTP/1.1 500 Internal Server Error\r\nStrict-Transport-Security: max-age=7776000; includeSubDomains\r\nX-Content-Type-Options: nosniff\r\nX-XSS-Protection: 1; mode=block\r\nCache-Control: no-cache, no-store\r\nX-OMaaS-RequestId: af4382b0-6416-11ef-aa1a-a7dba1956b91\r\nContent-Type: application/json; charset=utf-8\r\nContent-Length: 127\r\nETag: W/\"7f-jbOgvHsQM/XI+gAiY5vypKpVXMg\"\r\nVary: Accept-Encoding\r\nDate: Tue, 27 Aug 2024 01:50:06 GMT\r\nConnection: close\r\n\r\n"},"profile":{"miliseconds":43.248329,"tasks":[{"name":"profile-0-rolesSharedKey","taskstart":17.308498,"miliseconds":0.030526},{"name":"profile-1-collabops","taskstart":17.350231,"miliseconds":0.029867}]},"msg":"Request finished","time":"2024-08-27T01:50:06.470Z","v":0}
{"name":"request","hostname":"aiops-ir-core-ncodl-api-7f96d5d699-lddwm","pid":17,"subscription":"cfd95b7e-3bc7-4006-a4a8-a73a79c71255","user":"kubeadmin","client":"unknown","req_id":"eeb2ee40-63bc-11ef-aa1a-a7dba1956b91","http":"GET /irdatalayer.aiops.io/active/v1/stories/b4370edb-6e4b-458f-b0e3-3a504df7546e","level":30,"stack":null,"err":{"message":"Not Found","name":"Error","stack":"Error: Not Found\n at callback (/app/node_modules/superagent/src/node/index.js:901:17)\n at <anonymous> (/app/node_modules/superagent/src/node/index.js:1166:18)\n at IncomingMessage.<anonymous> (/app/node_modules/superagent/src/node/parsers/json.js:19:7)\n at IncomingMessage.emit (node:events:529:35)\n at endReadableNT (node:internal/streams/readable:1400:12)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)"},"msg":"Request ended in error","time":"2024-08-26T15:07:38.182Z","v":0}
Postgres pod logs will have the below entries:
{"level":"info","ts":"2024-08-26T12:57:03Z","logger":"postgres","msg":"record","logging_pod":"ibm-cp-aiops-edb-postgres-1","record":{"log_time":"2024-08-26 12:57:03.941 UTC","user_name":"aiops_irbuc","database_name":"aiops_irbuc","process_id":"1451991","connection_from":"10.130.8.44:42598","session_id":"66cc7b9f.1627d7","session_line_num":"1","command_tag":"initializing","session_start_time":"2024-08-26 12:57:03 UTC","transaction_id":"0","error_severity":"FATAL","sql_state_code":"53300","message":"sorry, too many clients already","backend_type":"client backend"}}
{"level":"info","ts":"2024-08-26T12:57:05Z","msg":"DB not available, will retry","controller":"cluster","controllerGroup":"postgresql.k8s.enterprisedb.io","controllerKind":"Cluster","Cluster":{"name":"ibm-cp-aiops-edb-postgres","namespace":"cp4aiops"},"namespace":"cp4aiops","name":"ibm-cp-aiops-edb-postgres","reconcileID":"a620bcfc-a7ac-4207-be7a-b6f6fceb6071","uuid":"b21164f1-63aa-11ef-8540-0a580a800822","logging_pod":"ibm-cp-aiops-edb-postgres-1","err":"failed to connect to `host=/controller/run user=postgres database=postgres`: server error (FATAL: sorry, too many clients already (SQLSTATE 53300))"}
{"level":"info","ts":"2024-08-26T12:57:07Z","logger":"postgres","msg":"record","logging_pod":"ibm-cp-aiops-edb-postgres-1","record":{"log_time":"2024-08-26 12:57:07.353 UTC","user_name":"postgres","database_name":"aimanager","process_id":"1451883","connection_from":"10.131.0.93:48706","session_id":"66cc7b88.16276b","session_line_num":"1","command_tag":"idle","session_start_time":"2024-08-26 12:56:40 UTC","virtual_transaction_id":"176/0","transaction_id":"0","error_severity":"FATAL","sql_state_code":"57P01","message":"terminating connection due to administrator command","application_name":"PostgreSQL JDBC Driver","backend_type":"client backend"}}
{"level":"info","ts":"2024-08-26T12:58:00Z","logger":"postgres","msg":"record","logging_pod":"ibm-cp-aiops-edb-postgres-2","record":{"log_time":"2024-08-26 12:58:00.135 UTC","user_name":"aiops_irb","database_name":"aiops_irb","process_id":"829762","connection_from":"10.130.8.37:45714","session_id":"66cd64d0.ca942","session_line_num":"1","command_tag":"startup","session_start_time":"2024-08-26 12:58:00 UTC","virtual_transaction_id":"202/3448","transaction_id":"0","error_severity":"FATAL","sql_state_code":"53300","message":"remaining connection slots are reserved for non-replication superuser connections","backend_type":"client backend"}}
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSE9G0Q","label":"IBM Cloud Pak for AIOps"},"ARM Category":[{"code":"a8m3p000000hAHpAAM","label":"Watson AIOps-\u003EAI Manager"},{"code":"a8m3p000000PCISAA4","label":"Watson AIOps-\u003EAI Manager-\u003EAlert Viewer-\u003EAlerts"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Log InLog in to view more of this document
This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.
Was this topic helpful?
Document Information
Modified date:
06 March 2025
UID
ibm17184618