Troubleshooting Secure Tunnel

Learn how to isolate and resolve problems with Secure Tunnel.

Secure Tunnel connector is not running after restart with Podman
fuser hangs when Secure Tunnel install script is launched
When creating a Secure Tunnel connection, the status remains in pending for a long time
Slack is unable to verify the secure tunnel URL
Secure Tunnel connector pod in crash loop state after installation on OpenShift Container Platform cluster

Secure Tunnel connector is not running after a restart with Podman

This issue can occur when you install the Secure Tunnel connector to a host machine on which Podman is installed. When the host machine is rebooted and the Secure Tunnel connector is checked by using the podman ps -a command, the Secure Tunnel connector container does not display running status.

If this issue occurs, the podman-restart service must be activated by using the systemctl command:

systemctl start podman-restart
systemctl enable podman-restart

After entering the command, check podman-restart worked by using the following command:

systemctl status podman-restart

If the Connector is still not running, try restarting the host machine.

fuser hangs when Secure Tunnel install script is launched

After launching the Secure Tunnel install script, fuser hangs and you see something that resembles this

Checking whether docker or podman is installed ...
docker found.
Starting docker engine...
fuser found.

This issue can occur when fuser is having trouble finding a specific available port and so the execution stalls.

Solution: This issue rarely happens but should you encounter it edit the install-connector.sh and add a timeout in the fuser command.

Change PortCheck=$(fuser $HEALTHY_PORT/tcp 2>/dev/null) to PortCheck=$(timeout 2 fuser $HEALTHY_PORT/tcp 2>/dev/null). This change can resolve the issue.

When creating a Secure Tunnel connection, the status remains in pending for a long time

When you use the Custom method to create a Secure Tunnel connection, the Status of the connection is stuck in Pending for a long time. This also occurs when you create a secure tunnel connection that uses a shortcut (for example, ChatOps, VMware or OpenStack).

This issue can occur when the tls secret was not created by the CertManager but the tunnel connection is waiting for it.

Solution: In most cases, re-creating the connection resolves the problem. If that does not work, you need to run the following command to restart the IBM Cert Manager and re-create the certificate CR to resolve this problem. Replace the namespace used here, cp4aiops, with your own namespace.

Find the namespace that IBM Cert Manager is deployed in.

oc get pod -A | grep cert-manager-controller | awk '{print $1}'

Export environment variables:
```
export CERT_MGR_NAMESPACE=<cert_namespace>
export CONNECTION_NAME=<conn_name>
```
Where:
- <cert_namespace> is the namespace that IBM Cert Manager is deployed in.
- <conn_name> is the name of the tunnel connection

Run the following commandL:

for NAME in $(oc -n ${CERT_MGR_NAMESPACE} get pod --no-headers | grep cert-manager | awk '{print $1}')
do
    oc -n ${CERT_MGR_NAMESPACE} delete pod ${NAME}
done
CONNECTION_ID=`oc -n cp4aiops get TunnelConnection ${CONNECTION_NAME} -o=jsonpath={.status.id}`
oc -n cp4aiops delete certificates.certmanager.k8s.io sre-tunnel-${CONNECTION_ID}-server-cert
oc -n cp4aiops delete certificates.certmanager.k8s.io sre-tunnel-${CONNECTION_ID}-connector-cert
oc -n cp4aiops delete certificates.certmanager.k8s.io sre-tunnel-${CONNECTION_ID}-api-cert

Slack is unable to verify the secure tunnel URL

You can encounter an issue with Slack being unable to verify the secure tunnel URL after you set up a ChatOps secure tunnel connection. This issue can occur even when the Secure Tunnel UI shows that the installation was successful.

This issue can occur if you enabled the Egress Network policy with the enableEgressNetworkPolicy: true when you install the Secure Tunnel.

You can use the following command to check whether the egress network policy is enabled. If the egress network policy is enabled the output should be true.

oc -n <the namespace of the aiops> get Tunnel sre-tunnel -o=jsonpath='{.spec.enableEgressNetworkPolicy}'

Solution: Use the following YAML code to help you create an egress network policy in your IBM Cloud Pak for AIOps project (namespace) to allow Secure Tunnel access to the ibm-nginx gateway.

  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: chatopstunnel-egress-ibm-nginx
    namespace: <the namespace of the aiops>
  spec:
    podSelector:
      matchLabels:
        app.kubernetes.io/instance: sre-tunnel
        app.kubernetes.io/managed-by: tunnel
        app.kubernetes.io/name: sre-tunnel-chatopstunnel.app
        tunnel_connection_name: chatopstunnel
    egress:
      - to:
          - podSelector:
              matchLabels:
                app.kubernetes.io/component: ibm-nginx
                icpdsupport/app: gateway
                component: ibm-nginx
    policyTypes:
      - Egress

Secure Tunnel connector pod in crash loop state after installation on OpenShift Container Platform cluster

You might notice an issue where the Secure Tunnel connector pods stay in a crash loop state. The pod logs might show proxy related errors.

This issue can occur when the Secure Tunnel connector is installed on a cluster within an air-gapped (offline) environment that does not have outbound network access.

Solution: For the Secure Tunnel connector that is installed within the air-gapped (offline) environment, create a network connection with the Secure Tunnel server by using an HTTP proxy. You can create the connection by completing the following steps:

From a command line, go to the tunnel-connector-install-scripts directory.

Open the tunnel-connector.yaml file by using the following command:

vi tunnel-connector.yaml

When you open the YAML file, it can resemble the following example YAML file:

 apiVersion: v1
 kind: Secret
 metadata:
   labels:
     app.kubernetes.io/name: sre-tunnel-df012501eede4b02.secret
     app.kubernetes.io/managed-by: tunnel
     app.kubernetes.io/instance: sre-tunnel
     tunnel-network-name: woprod-test
     tunnel_connection_name: woprod-test
     tunnel_connection_id: "df012501eede4b02"
   name: sre-tunnel-df012501eede4b02-config
 data:
   config_connector.yaml: |
     <base64 encoded data>
 type: Opaque

Decode the <base64 encoded data> of the YAML file by using the following command:
```
echo <base64 encoded data> | base64 -d
```
Where <base64 encoded data> is the encoded data from the YAML file.
Update the websocket_http_proxy and websocket_https_proxy properties that are found in the decoded data.
```
websocket_http_proxy: <http proxy endpoint>
websocket_https_proxy: <https proxy endpoint>
```
Where
- <http proxy endpoint> is the HTTP proxy endpoint. This proxy is used by the tunnel connector to create a nonencrypted network web socket connection with the tunnel server. For example, the value can be proxy.example.ibm.com:3128.
- <https proxy endpoint> is the HTTPS proxy endpoint. This proxy is used by the tunnel connector to create an encrypted network web socket connection with the tunnel server. For example, the value can be proxysecure.example.ibm.com:3128.
Encode the data by using the following command.
```
echo '<decoded data>' | base64
```
Where <decoded data> is the decoded data with the two modified properties.

Add the Base64 encoded data to the <base64 encoded data>.

 cat <<EOF | oc -n <namespace> apply -f -
 apiVersion: v1
 kind: Secret
 metadata:
   labels:
     app.kubernetes.io/name: sre-tunnel-df012501eede4b02.secret
     app.kubernetes.io/managed-by: tunnel
     app.kubernetes.io/instance: sre-tunnel
     tunnel-network-name: woprod-test
     tunnel_connection_name: woprod-test
     tunnel_connection_id: "df012501eede4b02"
   name: sre-tunnel-df012501eede4b02-config
 data:
   config_connector.yaml: |
     <base64 encoded data>

 type: Opaque
 EOF

Where

<base64 encoded data> is the Base64 encoded data.
<namespace> is the namespace where Secure Tunnel connector is installed.

Restart the Secure Tunnel connector pods by using the following command:
```
oc delete pod <secure-tunnel-connector-pod> -n <namespace>
```
Where
- <secure-tunnel-connector-pod> is the name of the Secure Tunnel connector pod.
- <namespace> is the namespace where Secure Tunnel connector is installed.