After upgrade and finalize, the stack versions page shows "Upgrade: In Process"

Resolving The Problem

1a) Run the following command; make sure that you replace <your-cluster-name> with the name of your cluster:

ambari-server set-current --cluster-name=<your-cluster-name> --version-display-name=IOP-4.1.0.0

1b) After running the command, restart your Ambari server and all the Ambari agents. On the server node, run:

ambari-server stop && ambari-server start

On all agent nodes, run:

ambari-agent stop && ambari-agent start

If this causes the stack versions page to show IOP 4.1 as "Current" then the problem is resolved. If not continue with the following instructions.

2a) From the command line, open a postgres command prompt:

psql -U ambari -d ambari

Enter "bigdata" when prompted for a password.

2b) Run a query to find the host that is still listed in 'UPGRADING' state

select h.host_name, h.host_id from hosts h, host_version hv where h.host_id = hv.host_id and hv.state = 'UPGRADING';

Example output:

host_name | host_id
----------------------+---------
<upgrading-host> | 2
(1 row)

You will need the host name in the following steps. In the examples below, <upgrading-host> is used.

3a) From the command line, check the /usr/iop/current folder on the upgrading host for links to 4.0.0.0

ssh <upgrading-host>

ll /usr/iop/current | grep 4.0

3b) If there were results from the previous command, then select the 4.1.0.0 components on the upgrading host:

iop-select set all 4.1.0.0

3c) If there were no results skip to step 4.

3d) Restart the agent on the upgrading host and the ambari server (which may be running on a different host):

ambari-agent stop && ambari-agent start

ambari-server stop && ambari-server start

3e) Check the Ambari user interface on the Admin -> Stack and Versions -> Version page to see if it still says "Upgrade: In Process" if it doesn't your upgrade has now been fully completed.

4a) From the command line, open a postgres command prompt:

psql -U ambari -d ambari

Enter "bigdata" when prompted for a password.

4b) Run a query to find the components on that host which are still 4.0.0.0

select hcs.component_name, hcs.version from hostcomponentstate hcs, host_version hv where hcs.host_id = hv.host_id and hv.state = 'UPGRADING' and hcs.version = '4.0.0.0';

4c) If there were results from the previous command, then select the 4.1.0.0 components on the upgrading host:

iop-select set all 4.1.0.0

4d) If there were no results skip to step 5.

4e) Restart the agent on the upgrading host and the ambari server (which may be running on a different host):

ambari-agent stop && ambari-agent start

ambari-server stop && ambari-server start

4f) Check the Ambari user interface on the Admin -> Stack and Versions -> Version page to see if it still says "Upgrade: In Process" if it doesn't your upgrade has now been fully completed.

4g) If the problem persists, from ambari UI restart the component that is still listed as 4.0.0.0 on that node

4h) Check the Ambari user interface on the Admin -> Stack and Versions -> Version page to see if it still says "Upgrade: In Process" if it doesn't your upgrade has now been fully completed.

5a) From the command line, open a postgres command prompt:

psql -U ambari -d ambari

Enter "bigdata" when prompted for a password.

5b) Verify that no hosts are listed as 'UPGRADING'

select h.host_name, h.host_id from hosts h, host_version hv where h.host_id = hv.host_id and hv.state = 'UPGRADING';

5c) If there were results, set the host version state to 'CURRENT' instead of 'UPGRADING':

update host_version set state = 'CURRENT' where state = 'UPGRADING';

5d) Verify that the cluster is not still listed as 'UPGRADING'

select * from cluster_version where state = 'UPGRADING'

5e) If there were results, set the cluster version state to 'CURRENT' instead of 'UPGRADING':

update cluster_version set state = 'CURRENT' where state = 'UPGRADING';

5f) Restart the Ambari server

ambari-server stop && ambari-server start

5g) Check the Ambari user interface on the Admin -> Stack and Versions -> Version page to see if it still says "Upgrade: In Process" if it doesn't your upgrade has now been fully completed.