Changing the service type during a helm upgrade

You may encounter a potential issue during a helm release upgrade or rollback that involves a change of service type from NodePort to ClusterIP. This known error will prevent you from performing the upgrade or rollback.

Known Issue when performing a helm upgrade

When performing a helm release upgrade, the error message below is shown after a failed helm upgrade due to a service conversion failure:

Error: UPGRADE FAILED: Service "tutorial-ibm-netcool-probe-syslogd" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

Note: This message is shown if the upgrade is performed from the command-line. In the ICP UI, the message shown is slightly different.

The error is due to the additional parameter nodePort being included in the service type NodePort template, the Helm release upgrade cannot be completed.

Service type ClusterIP

"ports": [
            {
                "name": "http",
                "port": 8888,
                "protocol": "TCP",
                "targetPort": "http"
            }

Service type NodePort

"ports": [
            {
                "name": "http",
                "nodePort": 30240,
                "port": 8888,
                "protocol": "TCP",
                "targetPort": "http"
            }
Important: You cannot rollback to the previous revision after upgrading a helm chart from service type ClusterIP to NodePort. Attempts to roll back to the previous revision with a service type ClusterIP will cause the status of rollback process to be placed in pending.

Known error message during a failed rollback

When performing a helm release rollback, the following error message is shown after a failed helm rollback due to service conversion failure:

Error: Service "tutorial-ibm-netcool-probe-syslogd" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'

Note: This message is shown if the rollback is performed from command-line. In the ICP UI, the message shown is slightly different.

Workarounds To The Issue

There are three methods you can use to avoid the service conversion issue above. You will only need to perform one of these methods:

  • Method 1: Installing the new version of the helm chart with a different release name and update all clients to point to the new probe service endpoint if required. Then delete the old release. This is the recommended method but requires a re-configuration on the client side.
  • Method 2: Manually changing the service type using kubectl edit svc. This method requires more manual steps but preserves the current service name and previous revisions of the helm chart. After performing this workaround, users should be able to perform a helm upgrade.
  • Method 3: Deleting and purging the existing helm release, and then install the new version of helm chart with the same release name.

Method 1: Install a new helm release with a different name

Follow the Helm chart README section to perform a new release install and then delete the old release. Remember to re-configure existing clients to update the probe endpoint.

Method 2: Edit the existing Service of the release

The following workaround requires more manual steps.

Manually edit the service using kubectl

This workaround is intended as an alternative for users who must not delete the existing release due to valid reasons, for example if the Cluster Admin wants to keep track of the changes by keeping records of past revision of an existing release.

A user with Cluster Admin or Admin role is required to change a service type. User Admin will have to edit the service of the probe chart by using kubectl edit svc. Before performing the workaround, user have to identify the release name of the probe chart. It is recommended that user only perform the workaround during scheduled maintenance. As an example, the following steps explain how to apply the changes to the IBM Netcool Probe chart that includes Logstash and Prometheus probes.

  1. To determine the release name of the helm chart, run helm ls and append with --tls for TLS enabled clusters.
  2. Based on the release name, identify the service that is servicing the probe by running kubectl get svc to get the name of the service.
  3. Remove line 25 with the nodePort parameter and replace the service type on line 34 with ClusterIP by using kubectl edit svc "service name". Append the service name of a particular release from Step 2 into the command.

    Precautionary step: Note that, you will not be able to perform a rollback as the workaround is not considered as a helm upgrade. Before editing the service template, you should save the existing service template as a backup by running the following command: kubectl get svc "servicename" -o yaml > service_backup.yaml.

    1 # Please edit the object below. Lines beginning with a '#' will be ignored,
2 # and an empty file will abort the edit. If an error occurs while saving this file will be
3 # reopened with the relevant failures.
4 #
5 apiVersion: v1
6 kind: Service
7 metadata:
8   creationTimestamp: 2018-07-31T04:20:00Z
9   labels:
10     app: ibm-netcool-probe
11     chart: ibm-netcool-probe
12     component: logstashprobe
13     heritage: Tiller
14     release: onetutorial
15   name: onetutorial-ibm-netcool-probe-logstashprobe
16   namespace: default
17   resourceVersion: "4273993"
18   selfLink: /api/v1/namespaces/default/services/onetutorial-ibm-netcool-probe-logstashprobe
19   uid: fdc46a83-9478-11e8-983d-005056a0a011
20 spec:
21   clusterIP: 10.0.0.230
22   externalTrafficPolicy: Cluster
23   ports:
24   - name: webhook-http
25     nodePort: 32025
26     port: 80
27     protocol: TCP
28     targetPort: 80
29   selector:
30     app: ibm-netcool-probe
31     component: logstashprobe
32     release: onetutorial
33   sessionAffinity: None
34   type: NodePort
35 status:
36   loadBalancer: {}
  4. Save and exit to apply the changes to the service template. The new service template for onetutorial-ibm-netcool-probe-logstashprobe will look like this.
    1 # Please edit the object below. Lines beginning with a '#' will be ignored,
2 # and an empty file will abort the edit. If an error occurs while saving this file will be
3 # reopened with the relevant failures.
4 #
5 apiVersion: v1
6 kind: Service
7 metadata:
8   creationTimestamp: 2018-07-31T04:20:00Z
9   labels:
10     app: ibm-netcool-probe
11     chart: ibm-netcool-probe
12     component: logstashprobe
13     heritage: Tiller
14     release: onetutorial
15   name: onetutorial-ibm-netcool-probe-logstashprobe
16   namespace: default
17   resourceVersion: "4274540"
18   selfLink: /api/v1/namespaces/default/services/onetutorial-ibm-netcool-probe-logstashprobe
19   uid: fdc46a83-9478-11e8-983d-005056a0a011
20 spec:
21   clusterIP: 10.0.0.230
22   ports:
23   - name: webhook-http
24     port: 80
25     protocol: TCP
26     targetPort: 80
27   selector:
28     app: ibm-netcool-probe
29     component: logstashprobe
30     release: onetutorial
31   sessionAffinity: None
32   type: ClusterIP
33 status:
34   loadBalancer: {}
  5. When the changes has been applied the following response message is shown:

    service "onetutorial-ibm-netcool-probe-logstashprobe" edited

  6. Repeat the same steps on service onetutorial-ibm-netcool-probe-prometheusprobe.
  7. When the changes have been applied, the following response message is shown:

    service "onetutorial-ibm-netcool-probe-prometheusprobe" edited

  8. Verify the status of the service by using kubectl get svcand verify that the service is able to reach the probe by sending curl test event to the probe.

Method 3: Replace the Helm existing release

The following workaround requires less manual steps.

Delete the existing Helm release and install the new release with the same name

Caution: By deleting the existing release, user will lose the configuration from the revision history of the release.

Note: To determine the release name of the helm chart, run helm ls and append --tls for TLS enabled clusters.

To delete the existing helm release using ICP's menu, use the following steps:
  1. Click on Workloads > Helm Releases.
  2. Under Action, select Delete.

    To delete the existing helm release using command line, use following command:

    helm delete "release name" --purge
  3. Install the new release with the same release name.