Recovering from updatios when hung ( killing hung process).

Content

If an update is hung it may be only one fileset that will not complete (has hung).


Using the "proctree" command to check the status of the update processes can confirm what is hung, and if appropriate isolate the process to kill.



===========================================================

Steps to recover from a hung process.

Summary:
--------
*) Use "ps -ef" to report the <PID_Number> of the original
command, then use "proctree <PID_Number>" to display the process tree.

*) The last process is most likely the hung process.
A "kill <PID-of_hung-process>" should free the original command and allow it to complete.

*) If the last process is a "sleep" the parent may need to be killed.

Carful review of the process tree is required.
IBM Support should be engaged to review the environment and confirm that killing a process is the appropriate action.



For a hung "updateios" command:
-----------------------------------------------------------

1) Run the updateios command:
$ updateios -dev <dir_with_update_files> -accept -install
(Command already run and has hung)

2) From a new session, enter oem_setup_env
and get the PID # of "updateios":
# ps -ef | grep updateios
padmin 2222222 2220000 0 hh:mm:ss pts/0
0:01 ioscli updateios
-dev <dir_with_updt_files>
-accept -install

# proctree 22222222
1111100 /usr/sbin/inetd
1111200 telnetd -a
1111300 -rksh
2222222 ioscli updateios -dev ... ...
2222400 /usr/sbin/installp ... ...
2222600 <PARENT_OF_HUNG_PROCESS>
yyyyyyy /bin/<PROCESS-THAT-IS-HUNG>
xxxxxxx sleep 5

3)
# kill xxxxxxx (if this process is hung)
# kill -9 xxxxxxx (if kill above does not remove
the process)

Note: If the "xxxxxxx" process is a "sleep"
the process above "yyyyyyy" may need to be killed.

4) # proctree 2222222
(check that updateios is moving forward)

If the <PARENT_OF_HUNG_PROCESS> re-starts
the hung process(child) then the
<PARENT_OF_HUNG_PROCESS> will need to be killed.
A "kill 2222600" may be needed.
Note: Do not kill the "installp" process 2222400.

-----------------------------------------------------------

One line command to display proctree data for
the VIO "updateios":

# ps -ef | grep updateios | grep -v grep | cut -c9-20 | xargs proctree

-----------------------------------------------------------


A hung update is rare, although an example of a hung update
is a 2.2.3.4 update where lwistatus.sh script hangs updating
Systems Director (DirectorCommonAgent, DirectorPlatformAgent).
The steps above may be required to clear the hang, and allow
the full update to complete.


Note:
-----
The "proctree" command can be used in a similar manor to address a hung or appears to be hung snap.
If a snap has hung it may be stuck on a device that has a long timeout, could make the snap take longer then desired.

Carful review of the process table (proctree) may allow killing the problematic process allowing the snap to
complete in a desired time frame.
Use the same technique described in this techdoc replacing the initial search with "snap" not "updateios".


Glen Martin
VIO Development Support