Brian Smith's AIX / UNIX / Linux / Open Source blog
|Modified on by brian_s|
To determine the oslevel on AIX, you can run the "oslevel -s" command. However, what "oslevel -s" reports doesn't always show the entire picture. The OS level reported will be the lowest level of any installed AIX fileset on your server.
For example, if all the filesets on your AIX server are upgraded to AIX TL8 SP3 except for one fileset which is at a lower level, then the oslevel reported will reflect the lower level of that single fileset, which might be something like TL4 SP2. So even though your server is 99.9% AIX TL8 SP3 oslevel would report the lowest level of any installed fileset.
The "oslevel -sq" command will show all of service packs that your AIX server is aware of. If you compare the top line in "oslevel -sq" versus "oslevel -s" they should normally match. If they don't, then you probably have an issue.
If you have a downlevel OS you can figure out which filesets are causing the issue and then fix them.
The first step to figuring out what filesets are causing the problem is to determine if your TL (Technology level) level is incorrect or just your SP (Service Pack) level is incorrect. To do this, compare the highest "oslevel -sq" line with your current "oslevel -s".
If the first 7 characters (####-##) match, but the rest are different, then your TL level is correct, but your SP level is not. For example, if your top line in your "oslevel -sq" output was 6100-07-02-1150, and your "oslevel -s" output was "6100-07-01-1141" then you would know your TL level was correct at TL7, but your SP level was not (oslevel -sq reported SP2, oslevel -s reported SP1). To determine which filesets are the problem if the TL level is correct, but the SP level is wrong, run:
This command will show you all the filesets that are below the SP level of the highest known SP level on the system.
If the TL level doesn''t match, for example if your top line in your "oslevel -sq" output was 6100-07-02-1150, and your "oslevel -s" output was "6100-04-11-1140" then you would know your TL level is incorrect (oslevel -sq reported TL7, oslevel -s reported TL4). To determine which filesets are the problem if the TL level is not correct, run:
This command will show you all the filesets that are below the TL level of the highest know TL level on the system.
Here is a script that will automates this process (note that this script doesn't work with AIX 5.2 or older). It will check out the state of your system and let you know if you have downlevel filesets:
Here is a screenshot of the output:
You might be wondering how you can avoid getting in a downlevel OS situation in the first place... Well usually this issue happens if you use the base media from an older level to install a fileset. For example, if you are at 6.1 TL8 SP3 and a user requests you install a new fileset. You only have the 6.1 TL7 SP2 base media, so you use it to install the requested fileset. If you just do this, your OS level will more than likely be downlevel now and report an incorrect version. What you need to do after installing a fileset from older media is to reinstall the TL8 SP3 update filesets to bring what you just installed up to the correct level. Remember - ALWAYS check the oslevel before and after you do any work related to filesets to make sure what you just did didn't downlevel the OS.
|Modified on by brian_s|
|Modified on by brian_s|
|Modified on by brian_s|
The HMC command line interface is awesome for scripting. However, for day to day administrative tasks it can be hard to use due to the long command line syntax for most commands. I have been using the HMC command line for years, and nearly anytime I want to use a command I have to reference the manual page to lookup the syntax.
I wanted to be able to use the HMC command line for common day to day administrative tasks, so I created a script called EZH - The Easy HMC Command Line Interface.
For more info, see: the EZH Sourceforge website It has full details, a download link, and directions for installation (which is very easy).
Here is an overview of the tasks you can do with EZH. For each task, an example EZH command line is shown as well as the native HMC command line.In each example the managed system name is "p520" and the LPAR name is "aix1".
If you are not familiar with "expect", it is a script/programming language that is designed to automate interactive processes. For example, suppose you need to install a piece of software on many UNIX/Linux servers. The installation program runs from the command line, but must be run interactively. When run, it interactively asks the user for several pieces of information, and then installs. With expect, you could write a script to automate this task so that when the installation program prompts for information expect supplies the information, essentially simulating a user and automating the task.
Expect allows you to turn a normally interactive-only process in to a completely non-interactive, automated task.
The author of the expect language, Don Libes, wrote the definitive book on expect. The book is called "Exploring Expect" by O'Reilly. I would highly recommend the book to anyone wanting to learn expect.
Here is a great picture from the "Exploring Expect" book that sums up the power of Expect:
In this posting I will cover when you should use expect and when you should avoid it.
When approaching a problem, my first rule of thumb when it comes to expect is to avoid using expect unless it is the only solution. Why? Expect is an amazing tool, but can be complicated to use and very fragile. The basic premise of expect is to set it up to look for certain strings of text ("expect"ing them), and when it sees certain text, respond in a certain way. For example, you could write a script that expects the text "Specify directory to install application to: " and when it sees this, type back in "/opt/software/" However, if in the next version of the software the text of the installer changes slightly (i.e. "Specify directory location to install application to: ") , your expect program will no longer see what it is looking for and will fail to work. If there is a different way to get something done other than using an expect script then it is usually a better option in my experience. It would be possible to write an expect program to automate opening vi, editing the file, and then saving and exiting vi. But it would be much easier and more reliable to use a tool such as sed, awk, perl, etc. to edit the file. Make sure you are using the right tool for the job.
My second rule of thumb is to not use expect to automate tasks that deal with passwords. When you look around for expect information and examples, a lot of them deal with automating things like SSH, SFTP, SCP, etc. For example, the Wikipedia page on Expect lists 4 example scripts for using expect. They all deal with automating logging in to a service with password authentication (telnet, ftp, sftp, and ssh). I would not recommend using expect to automate anything like this. There is a very good reason why tools like SFTP and SSH don't allow you to script using a password without a tool like expect: It is a very bad idea! The first problem is you generally need to include the clear text password in your expect script. If anyone gets access to the script, they have access to the password as well. The second big problem when using expect with passwords is the risk that expect will "type" in the password at the wrong time. For example, if you have a expect script setup to expect certain things, and then send the password, and something goes wrong, the script might send the password too early or too late. The password might then end up somewhere inappropriate or visible to other users or in a shell history file. The bottom line is, if you need to automate running remote commands or copying files around, use SSH keys. SSH keys are so much safer than passwords for a variety of reasons, and there are several things you can do to make them a good option for automated tasks. If you MUST use expect to automate a password related task, one method to help the situation would be to have the expect script prompt you for the password when beginning the task and have the user type it in each time. This way the password is not stored in the script file.
Another password related example of what NOT to do with expect: automate changing passwords. Expect even includes an example script with it named "passmass" that will change your password on multiple servers. From a security perspective I think this is a really, really bad idea for the reasons I specified in the previous paragraph. The right tool for this kind of job is the "chpasswd" command. The chpasswd utility even allows you to specify a password hash ("encrypted" password) when setting a users password to make it more secure to script. chpasswd isn't perfect, but in my opinion it is a much better option than expect when it comes to automating changing passwords.
So when should you consider using expect? You should think about expect anytime you have a manual task that needs to be repeated and that only provides a interactive interface to the user. We already covered the example of a interactive software installation program. Another example is any propriety software that forces you to go through a text based menu to do something. Using expect, you could write a script to navigate the menu and automate the task.
Another extremely good way to use expect is when you need to automate an appliance or other closed system that doesn't have the ability to be scripted. To do this, you use expect on a Linux/UNIX machine to connect to the appliance or closed system, and then complete a task. For example, you could write an expect script that would connect to a Cisco switch and run a series of commands on the switch.
Expect is also a good option when creating test cases. If you need to routinely test software functionality then expect might make your life easier.
You can also use expect to not fully automate tasks, but just assist with manual tasks. This is because expect allows you to partially automate tasks while still allowing parts to be manual completed by a real person. An example of this is the AIX command "mkdvd" which burns mksysb images on to a DVD. When you run this command it writes the first DVD, and then if needed it will prompt you to insert additional DVD's. With expect, you could write a script that would email you or page you whenever it was time to put in the next DVD or when the mkdvd command was completed. This script needs to be customized with 2 command lines to email you/page you.
This mkdvd expect script helps with a manual process and this is not something you could do without a tool like expect.
Please post comments with some creative examples of when you have used expect, or horror stories of other people using expect when they shouldn't have :)
One of the fundamental principles of troubleshooting any issue is to look for what has changed between the time things went from working to not working. One thing that could be relevant to any issue is if any software has been recently updated or installed.
AIX provides the "lslpp -h" command which will show fileset installation and update dates for each fileset. Unfortunately it doesn't sort the output of this, so it can be very difficult to look through the output to find filesets that have been recently updated or installed.
Here is a one liner that will sort the output and show you the most recently installed and updated filesets on your AIX system:
If you pipe it to "tail" you can see the 10 most recently installed or updated filesets on your AIX system:
Note: This one liner is assuming the date output will be MM/DD/YY... If you are in a different locale you might need to modify the one liner a little bit or temporarily change the environment variables to set your system to output dates in MM/DD/YY.
Also, in case you are wondering about the "sed 's/70/-70'" and "sed 's/-70/70'" parts of the script.. I noticed that lslpp -h on some servers will list the dates for some filesets as the year 70 (as in the UNIX epoch). Since "70" (1970) is bigger than "13" (2013) these old "1970" filesets were getting listed as the newest. So before the "sort" of the output I search and replace any 70's with -70's so they will be sorted correctly, and then after the sort change all the -70's back to 70. This way the dates are sorted correctly and the output still looks good.
One common problem I have personally made and seen others make while shell scripting is trying to set a variable to be the contents of a file or the output of a multiline command, and then trying to echo the variable to process it further with grep, awk, a while loop, etc. Depending on how you do this you might get some unexpected results because you might have missing newlines and spaces from your output.
In this example, we have a file named "testfile" that contains 5 lines. I then set the testvar variable to contain the contents of the line by running "testvar=`cat testfile`". However, when I attempt to "echo $testvar" all the lines of the file are shown on one line! If I try to grep for "test line 2" it still see's everything on one line.
The same problem happens when trying to set testvar to be the output of "ls -al" and then echoing the variable:
So what is going on here? It all has to do with how command line parameters are parsed by the shell and provided to the echo command.
Here is an example that illustrates what is happening:
Notice how the echo command removed all the extra spaces out of what I had typed? It is doing this because the shell is parsing each word as an argument and providing the arguments to the echo command:
The shell is parsing arguments and providing them to the echo command. As part of parsing the arguments, all of the extra spaces are being removed by the shell before they are passed to the echo command. This is the same thing that was happening when our newlines where removed from the first examples. The shell was parsing everything as arguments and removing all extra new lines and spaces.
How to fix this? It is very easy... Just use double quotes so that a single parameter is passed to the echo command:
By putting quotes around the text, it causes the shell to pass the entire text within the quotes as a single argument to echo which preserves the spaces.
This same technique works when echoing a variable that contains the output of a command or the contents of a file:
Hopefully this post helped you understand how commands are parsed by the shell and why you can sometimes see echo commands unexpectedly removing spaces and newlines.
The man page for the AIX mkvg command states this about how Physical Partition sizes are determined for volume groups when they are created:
The default value for PP size is kind of hard to understand based on the man page description. Below are a couple of tables that show given a hdisk size what the default PP size would be if a volume group was created on it without specifying a PP size.
AIX basically defaults to the smallest PP size possible for the given disk, which is almost certainly not what you want. Especially with original and big volume groups a small PP size puts severe limitations on the total capacity of the PV and volume group.
I recommend using scalable volume groups, and specifying a PP size rather than just taking the default value. Think about what the requirements will be in the future, and pick a PP size for the volume group based on that. Remember that the PP size determines the increments that you will be able to add space to filesystems to, so don't pick too large of a PP size. For example, if you pick a 1 GB (1024 MB) PP size, and have a 4 GB filesystem and you need to add 0.5 GB to, you will be forced to add a full 1 GB since that is the PP size and you would have to add a entire PP. If the PP size had been 512 MB you would have been able to add just 0.5 GB rather than being forced to add a full GB.
To allow the volume group to have the maximum overall capacity in the future for growth, pick the largest PP size that would work for your current and future requirements. 128 MB, 256 MB, and 512 MB PP sizes are good middle ground sizes that will work in most circumstances (again, pick the largest that will work for your environment). Remember that you can not change a volume groups PP size after it is created without backing the data up and recreating it.
Here are the default PP sizes that AIX will default to if you don't specify the PP size yourself:
AIX/VIO: Tracing Virtual SCSI / Shared Storage Pool Disks on AIX to VIO resources (and a script to automate this)Modified on by brian_s
When ever a filesystem is mounted on a UNIX system a mount point must be specified. A mount point is simply a directory that can either be empty or contain files. If the mount point directory contains files these are "covered up" and are no longer accessible when a filesystem is mounted over the mount point. Once the filessytem is unmounted any files that where in the mount point directory become visible again.
When the filesystem is mounted the owner, group, and permissions of the root of the filesystem take effect and override whatever ownership and permissions where set on the mount point directory.
# ls -ald /app1 ##Mount Point Directory (These are the permissions/owner/group I recommend for most mount points)
The first "ls" shows the mount point directory. The filesystem is then mounted, and as you can see both the permissions, owner, and group where changed to the filesystems root directory permissions/owner/group.
I recommend setting the mount point directory owner/group to root:system and having permissions set to something like "r-xr-xr-x" rather than having it set to the same owner/group and permissions that the mounted filesystem has. This way if the filesystem doesn't mount for some reason (for example due to a filesystem issue), users will be prevented from writing files in to the mount point directory. If the mount point ownership/permissions match the filesystem, then if the filesystem doesn't mount you will probably end up with a full root filesystem and then somehow need to merge files between what was written to the mount point and what was already in the filesystem (not fun!)
This is what you don't want:
# ls -ald /app1 ##Mount point permissions, owner, and group identical to what they are when filesystem is mounted
Be careful if you set the mount point permissions too restrictive. In general the mount point should have at least "r-xr-xr-x" permissions. If you set them to something like "rwx------" AIX will show weird permission errors for the parent directory when doing file listings as other users:
# ls -ald /app1 ##Mount point permissions/owner prevent any users from even listing contents of directory
The "lsvg" command has a handy "-i" option, which the man page says "Reads volume group names from standard input." This brief description doesn't explain how useful this option can be.
If you run "lsvg" and pipe the ouput to "lsvg -i" (i.e. "lsvg | lsvg -i") it will list the volume group information for every volume group on the system. You can also use other lsvg options such as "-l" to list all of the LV's/Filesystems from every volume group: "lsvg | lsvg -li".
This is an excellent way to gather LVM information from your system quickly and easily.
Here are a couple of examples:
And another example:
When you use the AIX usermod command to add a user to a new group it requires you to list all of the existing groups, and then the new group. For example, if testuser is a member of group1 and group2, and you need to add the user to group3, the command would be:
usermod -G group1,group2,group3 testuser
If you mistakenly ran:
usermod -G group3 testuser
Then testuser would be removed from group1 and group2 and only be a member of group3.
A challenge can arise if you need to script adding users to groups. This is because AIX usermod does not support "appending" a group to the users existing groups so your script must be aware of what the current groups are so that they are not removed from the account.
The best way I have found to do this is to use a command like this:
usermod -G `id -nG testuser | tr " " ","`,group3 testuser
The backticks on this command line will gather the users current groups by running "id -nG" which returns a space separated list of groups. That output is piped to tr which replaces the spaces with comma's. So in our example the backticks would be replaced by "group1,group2". Immediatly after the backticks we have ,group3 (comma group3) which is the group we would like to add/append. The command that ends up being run after the backticks are evaluated is usermod -G group1,group2,group3 testuser
If you happen to be on Linux, all of this is unneccasary as usermod on Linux has the -a or --append option which has usermod append to the group membership.
|Modified on by brian_s|