Troubleshooting guide for the Developer Portal

Use this guide to help you diagnose and resolve Developer Portal issues in IBM® API Connect. For example, why is the Developer Portal not connecting to the Management server, why is my site not created, and why can't I log in?

About

The following sections provide advice about how to diagnose and resolve some common problems. This information can help you to identify whether a specific IBM API Connect Version 10 component is failing, whether it's an environmental problem, and whether you should raise a support request. Some advice includes reviewing specific log files so you can narrow down the source of the problem.
Note: In the The Help icon.Help page of the Cloud Manager, API Manager, and API Designer user interfaces, there's a Product information tile that you can click to find out information about your product versions, as well as Git information about the package versions being used. Note that the API Designer product information is based on its associated management server, but the Git information is based on where it was downloaded from.
  1. Why is the Developer Portal not connecting to the Management server?
  2. Why is my Developer Portal site not being created?
  3. Why can't I register a user?
  4. Why can't I login?
  5. Why are changes to Products not appearing in the Developer Portal?
  6. Why am I having problems with my Developer Portal user interface?
  7. Why am I having problems with my Kubernetes system?
  8. Why am I having problems with my Developer Portal backup?
  9. Why am I having problems installing Drupal 8 based custom modules or sub-themes into the Drupal 9 based Developer Portal?

Why is the Developer Portal not connecting to the Management server?

Problems with connecting to the Management server can be due to connectivity or TLS issues. You should check the following areas: If you need to open a Service Request, include the following information:

Why is my Developer Portal site not being created?

If you are having problems creating a site, you should take the following actions to help diagnose the issue:
  1. Tail the nginx container log to confirm that the site create request reached the portal, for example:
    kubectl logs rb93d4fb118-apic-portal-nginx-7c7b464fb6-vvx5m
  2. Look in the container log for a site create request, for example:
    POST /portal-service-configuration-create HTTP/1.1" 200
    1. If you don't find a site create request in the log, then the create request is not reaching the Developer Portal from the Management. Follow the instructions in Why is the Developer Portal not connecting to the Management server? to diagnose the connectivity issues.
    2. If a site create request is in the log, then further diagnostics can be found in the admin pod log, for example:
      kubectl logs -f rb93d4fb118-apic-portal-www-p4bhj -c admin
      Look in the admin pod log for messages that refer to create_site, as well as the name of the site.

Why can't I register a user?

If you are having problems registering a user, take the following actions to help diagnose the issue:
  • The user registration links in the emails are only valid for 24 hours, so check that the system time is correctly configured across all of the pods.
  • If emails aren't being received, check that the email server is correctly configured; see Configuring an email server for notifications. Also, if possible, check whether other emails such as publish approval alerts or password reset emails are being received.
If you need to open a Service Request, please take the following steps:
  1. Note which user registry type is being used, such as Local User Registry (LUR), or LDAP user registry.
  2. Enable the debug REST server and method trace in the Developer Portal UI. Click Configuration > Development > IBM Development on the administrator dashboard. Then select the Enable method entry / exit trace and Enable API Manager REST interface debug check boxes. Click Save configuration.
  3. Re-create the issue, making a note of the time, the username, and any messages displayed on the screen.
  4. Gather the web container log for the www pod, for example:
    kubectl logs portal-apic-portal-www-zhqxf web > web.log
  5. Gather all of the Management server logs and the logs for the user registry type that is being used; see IBM API Connect MustGather Information for details.

Why can't I login?

If users are having problems with logging in to the Developer Portal, please take the following steps before opening a Service Request:
  1. Note which user registry type is being used, such as Local User Registry (LUR), or LDAP user registry.
  2. Enable the debug REST server and method trace in the Developer Portal UI. Click Configuration > Development > IBM Development on the administrator dashboard. Then select the Enable method entry / exit trace and Enable API Manager REST interface debug check boxes. Click Save configuration.
  3. Recreate the issue, making a note of the time, the username, and any messages printed to the screen.
  4. Gather the web container log for the www pod, for example:
    kubectl logs portal-apic-portal-www-zhqxf web > web.log
  5. Gather all of the Management server logs and the logs for the user registry type that is being used; see IBM API Connect MustGather Information for details.

Why are changes to Products not appearing in the Developer Portal?

To determine whether the problem is with the Portal server (either the nginx or the www pod), the Management server, or the network, please make the following diagnostic checks.
Portal nginx pod
  1. Tail the nginx pod log on the Portal, for example:
    kubectl logs -f portal-apic-portal-nginx-866c977c4c-n9gbf
  2. Make a publish request to the Catalog (note that the request must be a Publish or Retire operation, not a Stage operation).
  3. In the log being tailed, see if the following message appears:
    ...POST /event-create...
If the POST message appears in the nginx log, then the publish webhook that was sent from the Management server did arrive at the Portal. If there is an error message in place of a POST message, then the problem is likely to be with the nginx pod. If there is no message at all, then it's likely that the webhook from the Management server is not reaching the Portal and there is a network problem.
Portal www pod
  1. Tail the www pod admin container log on the Portal, for example:
    kubectl logs -f portal-apic-portal-www-dtbzw -c admin
  2. Make a publish request to the Catalog (note that the request must be a Publish or Retire operation, not a Stage operation).
  3. In the log being tailed, see if the following message appears:
    Received webhook event 'product_lifecycle' for catalog
If no such message appears, it suggests that the webhook is either not arriving at the Portal, or not getting beyond the nginx pod (see the previous section Portal nginx pod). Any errors that the admin container has in processing the webhook will be logged in the www pod admin container log.
Management server apim pod
  1. Tail the apim pod log on the Management server, for example:
    kubectl logs -f apiconnect-apim-v2-86b464c4f5-4cx6r
  2. Make a publish request to the Catalog (note that the request must be a Publish or Retire operation, not a Stage operation).
  3. In the log being tailed, see if there any error messages.
Search for the word 'sending', you should see the message webhook: sending successful if the webhook is sent, or error messages indicating why if the webhook is not sent.

Why am I having problems with my Developer Portal user interface?

If you are having user interface (UI) problems, make the following checks:
  • Try clearing the cache from your browser, try using private browsing, and try using different browsers - see whether there are any differences in behavior.
  • Check your browser window size and screen resolution, and try increasing and decreasing the settings.
  • Investigate the browser developer tools console and network tabs, as they might provide some diagnostic clues.
  • Try clearing the Drupal caches from within the Developer Portal; see Clearing the server caches for details.

If you need to open a Service Request, along with the IBM API Connect MustGather Information logs, also include details of the time and date when the problem was reproduced. In addition, an export of the browser developer tools console and network output would be helpful.

Why am I having problems with my Kubernetes system?

Kubernetes provides a DNS server as one of the pods in the kube-system namespace (unless you have configured your Kubernetes setup differently), and sometimes this DNS server can silently fail. When the Portal www or db pods restart, they check that their own hostname and IP address is in the service that is defined for the pod. If the DNS server is down, this check fails. An indication that the DNS server is down is seeing the following log lines printed endlessly in the admin or db containers:
[     admin stdout] 2cd141dc:2cd141dc:2cd141dc 2018-10-08 07:27:54: service-ready:
[     admin stdout] 2cd141dc:2cd141dc:2cd141dc 2018-10-08 07:27:54: service-all:
[     admin stdout] 2cd141dc:2cd141dc:2cd141dc 2018-10-08 07:27:54: Waiting for sleep in service all check. Pid(s) 352  Seconds 24
[     admin stdout] 2cd141dc:2cd141dc:2cd141dc 2018-10-08 07:27:59: Finished waiting for sleep in service all check. Pid(s) 352  Seconds 29
[     admin stdout] 2cd141dc:2cd141dc:2cd141dc 2018-10-08 07:28:01: WARNING 5: r4c09b98d02-apic-portal-admin-all doesn't include this pod or r4c09b98d02-apic-portal-director does. Checking again in 5 seconds.
To fix this problem, delete the DNS pod in the kube-system namespace to force it to restart, for example:
kubectl -n kube-system delete pod kube-dns-b76db4f7f-n4lvt
Where kube-dns-b76db4f7f-n4lvt is the full name of the DNS pod.
Note: To find out the full name of the DNS pod in your Kubernetes system, run the following command:
kubectl -n kube-system get pods
This command will return data like the following example:
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
default-http-backend-5bccfbdc8-mjnnb 1/1 Running 0 4d
etcd-minikube 1/1 Running 0 4d
kube-addon-manager-minikube 1/1 Running 0 4d
kube-apiserver-minikube 1/1 Running 0 4d
kube-controller-manager-minikube 1/1 Running 0 4d
kube-dns-6f4fd4bdf-7zslq 3/3 Running 0 4d
kube-proxy-mddmj 1/1 Running 0 4d
kube-registry-proxy-99qml 1/1 Running 0 4d
kube-registry-v0-zt8pv 1/1 Running 0 4d
kube-scheduler-minikube 1/1 Running 0 4d
kubernetes-dashboard-77d8b98585-dczcs 1/1 Running 0 4d
nginx-ingress-controller-57bcfc76d6-z775d 1/1 Running 0 4d
storage-provisioner 1/1 Running 0 4d
tiller-deploy-587df449fb-bpwhq 1/1 Running 0 4d

Why am I having problems with my Developer Portal backup?

If you attempt to back up your portal system or site:
apicup subsys exec <portal_subsystem> backup-system|backup-site|backup-all
You might encounter the following message:
curl: (6) Could not resolve host: apic-portal-apic-portal-director
This error is the result of a failure of the SFTP backup file transfer to the backup system, caused by missing authentication with the portal node. To correct this error, ssh from the backup system to the portal node, and accept the authentication key when prompted:
  1. Enter the portal-www admin container, for example: 
    kubectl exec -it WWW_POD_NAME -c admin bash
  2. ssh to your backup server and accept any authentication prompts, for example:
    ssh BACKUP_SERVER_HOSTNAME
When you have successfully authenticated with your backup server, you can close the session. This action needs to be done only once. Portal backups can then run without any issues.

Why am I having problems installing Drupal 8 based custom modules or sub-themes into the Drupal 9 based Developer Portal?

From IBM API Connect 10.0.3.0, the Developer Portal is based on the Drupal 9 content management system. If you want to install Drupal 8 custom modules or sub-themes into the Drupal 9 based Developer Portal, you must ensure that they are compatible with Drupal 9, including any custom code that they contain, and not using any deprecated APIs, for example. There are tools available for checking your custom code, such as drupal_check on GitHub, which checks Drupal code for deprecations.

For example, any Developer Portal sites that contain modules or sub-themes that don't contain a Drupal 9 version declaration will fail to upgrade, and errors like the following output will be seen in the admin logs:
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: Checking theme: emeraldgreen
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: ERROR: Incompatible core_version_requirement '' found for emeraldgreen
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: Checking theme: rubyred
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: ERROR: Incompatible core_version_requirement '8.x' found for rubyred
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: ERROR: Found themes incompatible with Drupal 9: emeraldgreen rubyred
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: ERROR: /tmp/restore_site.355ec8 is NOT Drupal 9 compatible
...
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: Checking module: custom_mod_1
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: ERROR: Incompatible core_version_requirement '' found for custom_mod_1
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: Checking module: custom_mod_2
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: ERROR: Incompatible core_version_requirement '8.x' found for custom_mod_2
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: ERROR: Found modules incompatible with Drupal 9: emeraldgreen rubyred
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: ERROR: site1.com is NOT Drupal 9 compatible
To fix version compatibility errors, all custom modules and sub-themes should declare a core_version_requirement key in their *.info.yml file that indicates Drupal 9 compatibility. For example:
name: Example module
type: module
description: Purely an example
core: 8.x
core_version_requirement: '^8 || ^9'
package: Example module

# Information added by Drupal.org packaging script on 2020-05-31
version: '8.x-1.3'
project: 'example_module'
datestamp: 1590905415
This example specifies that the module is compatible with all versions of Drupal 8 and 9. For more information, see Let Drupal know about your module with an .info.yml file on the drupal.org website.

If you have a backup of a site that you need to restore and are getting the version compatibility error, but the module or theme *.info.yml file cannot be changed easily, then you can modify the site backup.

To modify the site backup, extract the it, edit the relevant files inside it, and then tar the backup file again. Note that this procedure will overwrite the original backup file, so ensure that you keep a separate copy of the original file before you start the extraction. For example:

  1. mkdir /tmp/backup
  2. cd /tmp/backup
  3. tar xfz path_to_backup.tar.gz
  4. Edit the custom module and theme files to make them Drupal 9 compatible, and add the correct core_version_requirement setting.
  5. rm -f path_to_backup.tar.gz
  6. tar cfz path_to_backup.tar.gz
  7. cd /
  8. rm -rf /tmp/backup