Troubleshooting ingress issues on Kubernetes

Use this information to troubleshoot ingress issues for your IBM® App Connect instances that are deployed in a Kubernetes environment.



Resolving errors encountered while trying to access an App Connect Designer instance

After you create an ingress resource for an App Connect Designer instance, if you encounter Found. Redirecting to /auth/icp and 502 bad gateway errors while trying to access the instance, you can resolve these errors as follows:

  1. Run the following command to get the IDs of the ALB (ingress) pods, which are named public-<cluster_id>-alb1-*.
    kubectl get pods -n kube-system | grep alb
  2. Get the ALB pod logs.
    kubectl logs <alb_pod_ID> -n kube-system

  3. Inspect the logs. If you see an error with the text upstream sent too big header while reading response header from upstream, you can increase the header size of the ingress controller by setting the proxy-buffer-size variable to 16k.
  4. To set the proxy-buffer-size value, complete either of the following steps:
    • Set proxy-buffer-size at the ingress level by updating the ALB (ingress) deployment.
      1. List the deployment and note its name, which is in the format public-<cluster_id>-alb1.
        kubectl get deployment -n kube-system | grep alb
      2. Get the deployment's YAML definition and search for the text configmap.
        kubectl get deployment public-<cluster_id>-alb1 -n kube-system -o yaml | grep configmap

        Note the ConfigMap name in the output, which should contain the following text:

        --configmap=<namespaceName>/<configmap_name>

        Example: --configmap=kube-system/ibm-k8s-controller-config

      3. Edit the ConfigMap.
        kubectl edit configmap <configmap_name> -n kube-system

        This command automatically opens the default text editor for your operating system or opens another configured editor. Update the data section as follows:

        data: 
  proxy-buffer-size: 16k

        Save the YAML definition, and then close the text editor to apply your changes.

      4. Restart your ingress pods to pick up the updated data in the ConfigMap.
        kubectl delete pod <alb_pod_ID>
    • Set proxy-buffer-size on the ingress controller to apply this change to all ingresses.
      1. Get the ingress name.
        kubectl get ing -n kube-system

        In the output, the ingress name is shown in the NAME column. For example:

        NAME             CLASS    HOSTS                                                                                                 ADDRESS         PORTS     AGE
k8s-alb-health   <none>   albhealth.acecc-mycluster-cf1234e1b87ad465f7b7cd114438a7e6-0000.us-south.containers.appdomain.cloud   51.115.143.77   80, 443   439d
      2. Edit your ingress definition. 
        kubectl edit ing <ingress_name> -n <namespace>

        In the text editor, update the metadata.annotations section as follows. 

        metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"

        Save the YAML definition and close the text editor to apply your changes.

Resolving network connectivity errors when using a switch server

The ingress resource for a switch server must allow pass-through to enable the switch server and the connectivity agent in the IBM App Connect Enterprise system to communicate securely by using strict SSL self signed certificates. If the ingress is incorrectly configured, you might encounter network security errors when you deploy an integration server or integration runtime that uses callable flows or requires a connection to a private network.

The following example shows the type of error you might see.

2023-02-23 21:37:28.139562: BIP6453E: The connection agent for remote callable flows has encountered a network error when communicating with the Switch server with URL 'wss://switch-server-sw-mycluster.prod.abc.com:443'. The error message is: 'Error: self-signed certificate in certificate chain'.

To help you resolve such errors, complete the following steps:

  1. Confirm that you can successfully connect to your switch server by checking whether the switch server and the deployed integration server or integration runtime use matching certificates.
    1. In your switch server pod, start a terminal session and then run the following command.
      curl -v https://localhost:4443

      You should see output that is similar to this. 

      sh-4.4$ curl -v https://localhost:4443
* Rebuilt URL to: https://localhost:4443/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 4443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
sh-4.4$ curl -v -k https://localhost:4443
* Rebuilt URL to: https://localhost:4443/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 4443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=GB; ST=Hampshire; L=Winchester; O=IBM United Kingdom Limited; OU=App Connect; CN=default-switch-ace-test.apps.acecc-412.abc.com
*  start date: Apr 20 10:05:15 2023 GMT
*  expire date: Apr 20 10:05:15 2033 GMT
*  issuer: C=GB; ST=Hampshire; L=Winchester; O=IBM United Kingdom Limited; OU=App Connect; CN=default-switch-ace-test.apps.acecc-412.abc.om
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/1.1
> Host: localhost:4443
> User-Agent: curl/7.61.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS alert, [no content] (0):
* TLSv1.3 (IN), TLS alert, unknown (628):
* OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
* Closing connection 0
curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
    2. Make a note of the Server certificate details.
    3. From the pod for the deployed integration server or integration runtime, try to connect by using the domain that is configured in your switch server ingress. For example, if your ingress is defined as shown in the following YAML manifest, look for the TLS host name (for example, switchserver.mydomain.com), and then run this curl command.
      curl -k -v https://switchserver.mydomain.com
      apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <switchServerIngressName>
  namespace: <namespaceName>
  labels:
    appconnect.ibm.com/switch: <switchServerCRName>
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
    - hosts:
      - switchserver.mydomain.com
  rules:
    - host: switchserver.mydomain.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: <switchServerCRName>-switch
              port:
                number: 4443

      The output should return the same Server certificate details that were returned when you ran the curl command earlier for the switch server.

      If different Server certificate details are returned, this indicates that your ingress is not working correctly. The ingress might either be re-terminating the connection or sending the traffic to the wrong pod or container.

  2. Check the pod logs for the integration server or integration runtime for this type of message.
    2023-02-23 21:50:28.209484: BIP6469I: The secure connectivity agent for on-premises systems has established a connection to the Switch server with URL 'wss://switch-server-sw-mycluster.prod.abc.com:443'.
2023-02-23 21:50:28.218648: BIP2206I: The integration server component 'switchclient' has been started.