Upgrade fails (Streams)

When you upgrade the Streams service instance from a version of Streams that is earlier than 5.5.0 to Streams 5.5.0 or later, the upgrade process fails if Streams jobs exist in the instance.

Symptoms

When you upgrade the Streams service instance from a version of Streams that is earlier than 5.5.0 to Streams 5.5.0 or later, the following error message is displayed.
[ERROR] 2021-01-04T13:23:41.985237Z Error encountered when requesting upgrade: 
Received error while upgrading instance with ID: 1609779641135; error: 
Received status code: 409, message: [PUT /upgrade/instance][409] upgradeServiceInstanceConflict  
&{MessageCode: StatusCode:409 Exception: 
Message:The service instance cannot be upgraded because streams jobs are running. 
You must cancel all jobs before upgrading.}

[ERROR] 2021-01-04T13:23:41.988339Z Execution error:  exit status 1

Causes

This problem can occur if there are running Streams jobs when the upgrade starts.

Resolving the problem

You can check for running jobs (that is, application pods) by entering the following commands, substituting the name of your Streams service instance for myinstance:
oc get pods -l streams-app=active,streams-instance=myinstance
 
streams-user1@rhel7-enodes-1-1 ~$ oc get pods -l streams-app=active,streams-instance=myinstance
NAME               READY   STATUS    RESTARTS   AGE
myinstance-app-0   1/1     Running   0          10h
myinstance-app-1   1/1     Running   0          10h
myinstance-app-2   1/1     Running   0          6h54m
If the query shows that you have running jobs, you must cancel the jobs and then try to upgrade again.
To cancel running jobs, choose one of the following options for each Streams service instance that has running Streams jobs:
  • If the Streams console is enabled:
    1. Open the Streams console:
      1. Determine the URL for the Streams console: 
        https://Cpd_route/streams-console/instances/Instance_name/Project/streams/console
        Where:
        • Cpd_route = Your cluster administrator can provide you with this information by running the oc get route.
        • Instance_name = The name of the Streams instance.
        • Project = The project where the Streams instance is provisioned.
        For example, if your values are as follows:
        • Cpd_route = sample-cpd-sample.mycluster.com
        • Project = sample
        • Instance_name = myinstance
        Your Streams console URL is:
        https://sample-cpd-sample.mycluster.com/streams-console/instances/myinstance/sample/streams/console
      2. Paste the URL in a browser window, and sign in as a Streams service instance administrator.
    2. From the Application Dashboard, click Jobs and view the navigation tree. For the Streams service instance, click Cancel Jobs.
    3. Select all of the jobs, and click Cancel Jobs.
  • If the operations pod is enabled:
    1. As an administrator, use the exec command to access the operations pod, substituting the name of your Streams service instance for myinstance:
      oc exec -it myinstance-ops-0 -- bash
 
[streamsops@myinstance-ops-0 ~]$
    2. Use streamtool commands to cancel the jobs.
      [streamsops@myinstance-ops-0 ~]$ streamtool
[streamtool <?@myinstance>] lsjob
User:admin
Password:********
Instance: myinstance
  Id State   Healthy User          Date                     Name                          Group
   0 Running yes     admin         2020-11-02T15:45:09+0000 application::TradesAppCloud_0 default
   1 Running yes     streams-user1 2020-11-02T19:46:11+0000 Main_1                        default
 
 
[streamtool <admin@myinstance>] canceljob 0
CDISC0021I The following job ID was cancelled: 0. The job was in the myinstance instance.
 
[streamtool <admin@myinstance>] canceljob 1
CDISC0021I The following job ID was cancelled: 1. The job was in the myinstance instance.
    3. Repeat the streamtool lsjob command to verify that no jobs remain.
      streamtool <admin?@myinstance>] lsjob
Id State Healthy User Date Name Group
  • Using the security pod (if neither the Streams console nor the operation pod is enabled) :
    1. As an administrator, use the exec command to access the security pod, substituting the name of your Streams service instance for myinstance:
      oc exec -it myinstance-security-0 -- bash
 
[streamsops@myinstance-security-0 ~]$
    2. Use streamtool commands to cancel the running jobs.
      [streamsops@myinstance-security-0 ~]$ streamtool
[streamtool <?@myinstance>] lsjob
User:admin
Password:********
Instance: myinstance
  Id State   Healthy User          Date                     Name                          Group
   0 Running yes     admin         2020-11-02T15:45:09+0000 application::TradesAppCloud_0 default
   1 Running yes     streams-user1 2020-11-02T19:46:11+0000 Main_1                        default
 
 
[streamtool <admin@myinstance>] canceljob 0
CDISC0021I The following job ID was cancelled: 0. The job was in the myinstance instance.
 
[streamtool <admin@myinstance>] canceljob 1
CDISC0021I The following job ID was cancelled: 1. The job was in the myinstance instance.
    3. Repeat the streamtool lsjob command to verify that no jobs remain.
      streamtool <admin?@myinstance>] lsjob
Id State Healthy User Date Name Group