90
Make sure you are monitoring file system space on your VIOS.
Why?
If you run out of space in the root file system, odd things can happen when you try to map virtual devices to virtual adapters with mkvdev.
For example, a colleague of mine was attempting to map a new hdisk to a vhost adapter on a pair of VIOS. The VIOS was running a recent version of code. He received the following error message (see below). It wasn’t a very helpful message. At first I thought it was due to the fact that he had not set the reserve_policy attribute for the new disk to no_reserve on both VIOS. Changing the value for that attribute did not help.
$ ioslevel
2.2.1.3
$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1
*******************************************************************************
The command's response was not recognized. This may or may not indicate a problem.
*******************************************************************************
I enabled debug output from the padmin command line and discovered that the mkdev command was failing for some reason.
$ export CLI_DEBUG=33
$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1
AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"
AIX: "mkdev -V hdisk1 -p vhost0 -l vhdisk1 "
*******************************************************************************
The command's response was not recognized. This may or may not indicate a problem.
*******************************************************************************
I entered oem_setup_env and ran the mkdev command manually. I was presented with a very clear error message describing the problem. The root file system (/) was full!
$ oem_setup_env
# mkdev -V hdisk1 -p vhost0 -l vhdisk1
mkdev: 0514-548 Cannot perform the requested function because
the / filesystem is full or is out of inodes.
# df -g /
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 0.50 0.00 100% 7728 75% /
I found a 45MB core file (generated by cimserver) in the root file system. I moved the file to another file system (with plenty of free space) for further investigation.
# mv core /tmp/core.cg
# df -m .
Filesystem MB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 512.00 45.93 92% 7727 38% /
Then I ran the mkvdev command successfully.
$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1
AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"
AIX: "mkdev -V hdisk1 -p vhost0 -l vhdisk1 "
vhdisk1 Available
AIX: "lsattr -El hdisk1 -a max_transfer |cut -d" " -f2"
$ lsmap -all
AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name" | wc -l -c"
AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name""
AIX: "lsdev -C -l vhost0 -F "physloc""
AIX: "lsdev -p vhost0 -F "name" | wc -l -c"
AIX: "lsdev -p vhost0 -F "name""
SVSA Physloc Client Partition ID
--------------- -------------------------------------------- ------------------
vhost0 U9119.FHA.87654B1-V6-C21 0x0000002e
VTD vhdisk1
Status Available
LUN 0x8100000000000000
Backing device hdisk1
Physloc U5873.001.9SS005Y-P1-C9-T1-W50060E8006D0206A-L1000000000000
Mirrored false
AIX: "lsdev -C -l vhost1 -F "physloc""
AIX: "lsdev -p vhost1 -F "name" | wc -l -c"
AIX: "lsdev -p vhost1 -F "name""
SVSA Physloc Client Partition ID
--------------- -------------------------------------------- ------------------
vhost1 U9119.FHA.87654B1-V6-C22 0x0000002g
VTD NO VIRTUAL TARGET DEVICE FOUND
I found the same issue on the second VIOS i.e. a full root file system due to a core file (from cimserver). I also found no trace of a full file system event in the error report. Perhaps someone had taken it upon themselves to “clean house” at some point and had removed entries from the VIOS error log.
Make sure you monitor file system space on your VIOS. Who knows what else might fail if you run out of space in a critical file system.
