LI82318: ERROR 502 BAD GATEWAY WHEN NAVIGATING THE API MANAGER UI

APAR status

• Closed as program error.

Error description

• Error 502 bad gateway when navigating the API Manager UI
  
  The problem was caused by some cloud webhooks with extremely
  large state_change_history - up to 200,000 state change entries
  - containing many duplicate events (same id and timestamp).
  The webhooks filter field was also very large with many
  duplicate values.
  Performance improved dramatically after cleaning up these tables
  and other database tables.
  
  The following changes were made in v10 to prevent this happening
  again:
  1. Make webhook.state_change_history read-only
  2. Remove duplicates from filter and events fields in requests
  to create or update webhooks.
  3. Before adding a new event to webhook state_change_history,
  remove the oldest events from the state_change_history, keeping
  only the most recent 20 events.
  This is instead of removing the 1 oldest event and it is a
  safeguard to clean up the history size if it has exceeded the
  limit especially in existing deployments where the
  state_change_history could be updated directly via APIM API.
  
  The changes made in v2018 are slightly different, to reduce the
  risk of breaking tools that update webhook state_change_history:
  1. Truncate the state_change_history to the last 21 events in
  requests to update a webhook
  2. Remove duplicates from filter and events fields in requests
  to create or update webhooks.
  3. Before adding a new event to webhook state_change_history,
  remove the oldest events from the state_change_history, keeping
  only the most recent 20 events.
  This is instead of removing the 1 oldest event and it is a
  safeguard to clean up the history size if it has exceeded the
  limit especially in existing deployments where the
  state_change_history could be updated directly via APIM API.
  
  Changes (2) and (3) are the same in v2018 and v10; only change
  (1) is different.
  
  
  RESOLUTION
  
  | Release Stream | Date | Pull Request |
  
  | master           | 14 July 2021  | velox/apim#12298 |
  | v2018.4.1.18 | 16 Aug 2021  | <a
  href="https://github.ibm.com/velox/apim/pull/12312">velox/apim#1
  2312</a> |
  | v10.0.1.5       | 19 Aug 2021  | <a
  href="https://github.ibm.com/velox/apim/pull/12399">velox/apim#1
  2399</a> |
  

Local fix

Problem summary

• Error 502 bad gateway when navigating the API Manager UI
  

Problem conclusion

• The problem was caused by some cloud webhooks with extremely
  large state_change_history - up to 200,000 state change entries
  - containing many duplicate events (same id and timestamp).
  
  The webhooks filter field was also very large with many
  duplicate values.
  Performance improved dramatically after cleaning up these and
  other database tables.
  
  We made <span style="background-color:rgb(255, 255,
  255)">state_change_history field read-only and made sure it
  can't grow beyond 20. This helped us fix the issue where we had
  unbounded state change entries.</span>
  
  The fix was made in v10.0.4, v10.0.1.5 and v2018 fp18.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  API CONNECT ENT

• Fixed component ID

  5725Z2201

Applicable component levels