IBM Support

After upgrade and finalize, the stack versions page shows "Upgrade: In Process"

Troubleshooting


Problem

After upgrade and finalize, the stack versions page shows "Upgrade: In Process". The stack version IBM Open Platform with Apache Hadoop (IOP) 4.0 is still listed as the current version and IOP 4.1 is only listed as installed.

Symptom

After you upgrade and finalize, the stack versions page shows "Upgrade: In Process". The stack version IOP 4.0 is still listed as the current version and IOP 4.1 is only listed as installed.

Cause

The upgrade didn't properly populate the Ambari database.

Diagnosing The Problem

After upgrade and finalize, check the stack versions page. This can be seen on the Ambari user interface. Click "Admin" then "Stack and Version" then the "Version" subtab. Make sure IOP 4.1 is listed as "Current". If it shows "Upgrade: In Process" then you have hit this issue. Also check that the stack version IOP 4.0 is still listed as the current version and IOP 4.1 is only listed as installed.

Resolving The Problem

1a) Run the following command; make sure that you replace <your-cluster-name> with the name of your cluster:

    ambari-server set-current --cluster-name=<your-cluster-name> --version-display-name=IOP-4.1.0.0
1b) After running the command, restart your Ambari server and all the Ambari agents. On the server node, run:
    ambari-server stop && ambari-server start

On all agent nodes, run:
    ambari-agent stop && ambari-agent start

If this causes the stack versions page to show IOP 4.1 as "Current" then the problem is resolved. If not continue with the following instructions.

2a) From the command line, open a postgres command prompt:
    psql -U ambari -d ambari

    Enter "bigdata" when prompted for a password.

2b) Run a query to find the host that is still listed in 'UPGRADING' state
    select h.host_name, h.host_id from hosts h, host_version hv where h.host_id = hv.host_id and hv.state = 'UPGRADING';

    Example output:

    host_name | host_id
    ----------------------+---------
    <upgrading-host> | 2
    (1 row)

You will need the host name in the following steps. In the examples below, <upgrading-host> is used.

3a) From the command line, check the /usr/iop/current folder on the upgrading host for links to 4.0.0.0
    ssh <upgrading-host>

    ll /usr/iop/current | grep 4.0


3b) If there were results from the previous command, then select the 4.1.0.0 components on the upgrading host:
    iop-select set all 4.1.0.0

3c) If there were no results skip to step 4.

3d) Restart the agent on the upgrading host and the ambari server (which may be running on a different host):
    ambari-agent stop && ambari-agent start

    ambari-server stop && ambari-server start

3e) Check the Ambari user interface on the Admin -> Stack and Versions -> Version page to see if it still says "Upgrade: In Process" if it doesn't your upgrade has now been fully completed.

4a) From the command line, open a postgres command prompt:
    psql -U ambari -d ambari

    Enter "bigdata" when prompted for a password.

4b) Run a query to find the components on that host which are still 4.0.0.0
    select hcs.component_name, hcs.version from hostcomponentstate hcs, host_version hv where hcs.host_id = hv.host_id and hv.state = 'UPGRADING' and hcs.version = '4.0.0.0';

4c) If there were results from the previous command, then select the 4.1.0.0 components on the upgrading host:
    iop-select set all 4.1.0.0

4d) If there were no results skip to step 5.

4e) Restart the agent on the upgrading host and the ambari server (which may be running on a different host):
    ambari-agent stop && ambari-agent start

    ambari-server stop && ambari-server start

4f) Check the Ambari user interface on the Admin -> Stack and Versions -> Version page to see if it still says "Upgrade: In Process" if it doesn't your upgrade has now been fully completed.

4g) If the problem persists, from ambari UI restart the component that is still listed as 4.0.0.0 on that node

4h) Check the Ambari user interface on the Admin -> Stack and Versions -> Version page to see if it still says "Upgrade: In Process" if it doesn't your upgrade has now been fully completed.

5a) From the command line, open a postgres command prompt:
    psql -U ambari -d ambari

    Enter "bigdata" when prompted for a password.

5b) Verify that no hosts are listed as 'UPGRADING'
    select h.host_name, h.host_id from hosts h, host_version hv where h.host_id = hv.host_id and hv.state = 'UPGRADING';

5c) If there were results, set the host version state to 'CURRENT' instead of 'UPGRADING':
    update host_version set state = 'CURRENT' where state = 'UPGRADING';

5d) Verify that the cluster is not still listed as 'UPGRADING'
    select * from cluster_version where state = 'UPGRADING'

5e) If there were results, set the cluster version state to 'CURRENT' instead of 'UPGRADING':
    update cluster_version set state = 'CURRENT' where state = 'UPGRADING';

5f) Restart the Ambari server
    ambari-server stop && ambari-server start

5g) Check the Ambari user interface on the Admin -> Stack and Versions -> Version page to see if it still says "Upgrade: In Process" if it doesn't your upgrade has now been fully completed.

[{"Product":{"code":"SSCRJT","label":"IBM Db2 Big SQL"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Install","Platform":[{"code":"PF016","label":"Linux"}],"Version":"4.1.0","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
08 April 2021

UID

swg21966189