I had a new article published on IBM developerWorks today: Getting started with Nmap for system administrators
Brian Smith's AIX / UNIX / Linux / Open Source blog
Does anyone know the status of the IBM Champions for Power Systems program? I nominated someone back in October, 2013, and a couple of people nominated me a year ago back in March, 2013, but I have never heard anything back from IBM in either case and have not been able to get a hold of anyone who is familiar with the current state of the Power Champion program.
Does anyone out there know what the status of the Power Champion program is? Is it still being maintained/updated or is it now defunct?
The latest development versions of PuTTY have a awesome new feature called "Share SSH Connections" (Connection Sharing).
I did a video that covers the basics of this feature and shows a demo of it in action:
Calvin Powers recently interviewed me for the This Week on developerWorks Webcast regarding my recent developerWorks article on Tracing IBM AIX hdisks back to IBM System Storage SAN Volume Controller (SVC) volumes
See the interview over at the This Week on developerWorks Webcast blog.
Here is a quick one liner script you can copy/paste in to your HMC SSH terminal that will generate a HTML report showing all of the managed systems and LPAR's attached to the HMC and their current state.
You could easily change/extend the script to show other information like LPAR CPU settings, memory settings, etc. in the report as well.
This kind of script can be very handy to generate quick reports to send to people who might not have direct access to the HMC. You can also run the one liner on multiple HMC's, and combine the output in to a single HTML file to see information from all your HMC's in a single HTML report.
Here is the one liner script to run on the HMC to generate the report:
Here is sample output of what the generated HTML report looks like:
Here is a quick HMC one line script to run a command on every VIO server attached to the HMC. It uses "viosvrcmd" which uses RMC so there is no SSH keys or anything to setup; it just works as long as RMC is working in your environment.
In this example, it runs the "errlog" (aka errpt) command on every VIO server. You can change this to whatever command you would like to run. The script looks through all managed systems for VIO partitions, and attempts to use viosvrcmd to run the specified command on each VIO server.
A couple of years ago I wrote a script to Trace Virtual SCSI / Shared Storage Pool Disks on AIX to VIO resources. This script automates the process of tracing back your VIO client VSCSI disks back to their VIO resources, and generates a nice HTML web report showing the details.
The original script is written to work in a HMC managed environment.
Steve Bromwich recently contacted me and let me know he had modified the original script to work in IVM environments such as those commonly found in Blade Center environments.
Here is the modified script for IVM use that Steve created: https://www.ibm.com/developerworks/community/blogs/brian/resource/vscsimap_ivm.zip
Steve said that you run the script from the IVM system. You need to have SSH keys setup to your AIX LPAR's that are on the IVM. Within the script you specify the name of the account that should be used (see the line that says "my $ivmuser = ". In addition, Steve also added in some code to show the name of the Volume Group for each client hdisk.
The Shell History file has saved my bacon many, many times. So many times I have used it to figure out what happened with a server after some kind of problem, or to find a command I had run before that I couldn't remember. The Shell history paints a very good picture of what has occurred on a server and is one of the first places I start when troubleshooting a server to see what has recently been happening.
It can also help document the configuration on a server. I remember many years ago we were having issues with a VIO SEA adapter and had to recreate it. Luckily we found the original command that was used to create the SEA in the history file so it was a piece of cake for us to recreate it.
By default the AIX history has a couple of limitations:
Many of these low limits we see today are archaic leftovers from a time long ago when computers had extremely limited resources. For example, one thing that drives me crazy is the default scroll back buffer of the popular PuTTY SSH client is only 200 lines! In an era of desktops with at least several Gigabytes of memory we certainly have room to store more than 200 measly lines of text! Luckily it is very easy to change the default scroll back buffer in PuTTY to something more reasonable like 10 million lines..
So in today's modern AIX environments where servers are often connected to terabytes of storage and often have hundreds of Gigabytes of RAM I think we have room to store a little more than 500 lines of shell history as well.
By default the AIX shell history file doesn't record the date/time stamp for each command that is run. This reduces the usefulness of the history file because the commands in it could have been run 10 minutes ago or 10 months ago, you just can't tell one way or the other. When troubleshooting an issue it can be extremely useful to be able to see that a command in the shell history was run at the same time a problem began.
Luckily it is extremely easy to fix both of these limitations with the default AIX Shell History. Simply add these lines to your global /etc/profile file:
These will increase the history size to 10,000 lines and record date/time stamps for each command run. Use the "-t" flag with the history command to see the date/time stamps next to each command.
Hopefully these simple tips will allow you to get more out of the AIX Shell History.
AIX stores the last time a user changed their password as a "epoch" time stamp, or in other words as the number of seconds since 1970.
For example, if you want to see when the last time root changed their password you can type:
This shows that root changed their password at 1,391,663,150 seconds after 1970. In other words, this really isn't very helpful unless you take this epoch number and convert it to a real date.
Here is a one liner function that will give you a "lastpwchg" command that will show a normal date/time for a users last password change. Just type "lastpwchg" followed by the username you would like to check:
The results look like this when you run it to check a user:
If you found this useful, you might also want to check out this post: Don't let your AIX passwords expire
Sorry for the lack of updates the last month or so.. Things have been very busy.. My son and I were studying for Ham Radio tests we took in January, my wife and I have been taking a class on the weekends, I've been working on a big non-AIX project that is almost done, and we also recently found out that we are expecting a baby boy in June! Anyway, things are starting to settle down again so I'm hoping to get back in the swing of things with the blog again. I also have a couple of article ideas and have started working on a new article. So hopefully there will be more to come in the near future!
Check out my latest article on Power IT Pro, "Compare Files Easily with the comm Command": http://poweritpro.com/system-admin/compare-files-easily-comm-command
Version 1.2 of EZH (Easy HMC Command Line Interface) has been released.
EZH is a script for the IBM HMC console to provide an alternate, easier to use, command line interface for many common commands and the goal of the project is to make the HMC command line interface easier to use for day to day administration tasks. It also includes an optional interactive menu to make it even easier to use.
EZH is Open Source and 100% free, and is very easy to install. For more information and to download, see the project page at http://ezh.sourceforge.net/
Last month an article I wrote on Tracing IBM AIX hdisks back to IBM System Storage SAN Volume Controller (SVC) volumes was published on IBM developerWorks.
The article included a script designed to automate the process of tracing back AIX hdisks back to SVC Volumes so that you could easily see all the SVC related information about the AIX hdisk, including the SVC volume name.
Dan Aldridge modified the script in the article with a few very nice improvements. It no longer depends on pcmpath, supports more versions of the SVC, and even works with SVC Volumes presented through Virtual SCSI (* Requires AIX 6.1 TL7 or later). For more information on what the script does and how to use it, see the original article linked above.
Here is the modified script from Dan:
Check out my latest article on Power IT Pro, "Boost Your Productivity with Single-Line AIX Shell Scripts": http://poweritpro.com/aix/boost-your-productivity-single-line-aix-shell-scripts
Being careful not to shoot yourself in the foot when cleaning up users and home directories on AIX and Linux
It is possible on Linux or AIX to have users that share a home directory. For example, you might have user1, user2, and user3 all have their home directories set to /sharedhome.
Anytime you are deleting users and home directories you need to keep this in mind. You might only need to delete "user1" but want to leave "user2" and "user3" unaffected. But if you delete user1 and its home directory you might end up deleting the shared home directory which would have a big impact on "user2" and "user3".
Let's look at this situation:
AIX and Red Hat both have a "userdel" command that will optionally erase the users home directory as well ("-r" flag).
On most versions of AIX, if you did a "userdel -r" on any one of the 3 users, it would deleted the /sharedhome directory which would impact the remaining 2 users that you didn't want to affect.
On Red Hat Linux, "userdel" tries to be smarter, and verifies that the owner of the home directory matches the user that is being deleted. Thus, if you did a "userdel -r user1" it would happily wipe out /sharedhome, but if you did a "userdel -r user2" or "userdel -r user3" it wouldn't delete /sharedhome because the owner of the directory doesn't match the user being deleted.
Here is a one-liner that will do a better job checking to see if a given directory is a shared home directory between multiple users:
Change "/sharedhome" to whatever directory you would like to check. If this comes back and says it is a shared home directory, then you need to do more research before attempting to delete the home directory.
So anytime you are cleaning up users and home directories, always keep in mind that users might have shared home directories and that under some circumstances AIX and Linux will wipe out these shared home directories which might affect other users on the system.
I've noticed that on some AIX servers there are these files:
These files are there on some servers, but not on others, so I was curious what their purpose was and what created them.
I started by googling the file names, and found a few references to them, but nothing that really explained why they are there or where they come from. So then I tried searching through all man pages with a command like this:
Based on this, it sounded to be like the files were a automatically generated backup file of these files.. The server I was working on didn't have a "/etc/opasswd", or "/etc/ogroup" files, so I tried creating a user, changing a users password, and deleting a user. After all this, the "/etc/opasswd" and "/etc/ogroup" files still didn't exist as I had expected them to....
Next, I decided to look through all the executables on the server and see which of them contained a reference to "/etc/opasswd":
# for dir in `echo $PATH | tr ":" " "`; do for file in `ls -1 $dir 2>/dev/null`; do [ -x $dir/$file ] && [ -f $dir/$file ] && strings $dir/$file | grep -q -i '/etc/opasswd' && echo "$file"; done; done
The only two executables found on the server with references to /etc/opasswd where "pwdck" and "usrck". I also ran "strings" on the libraries under /usr/lib but couldn't find any references to opasswd in them.
So I did some more experimenting and found that the "pwdck" and "usrck" commands do in fact make backup copies of files to /etc/opasswd, /etc/ogroup, /etc/security/opasswd, and /etc/security/ouser. These backup files could be used to help recover the system if pwdck or usrck messed something up on the system.. Mystery solved :)
Over the years I've noticed that a lot of the core utilities on AIX are actually shell scripts.
Here are some examples of these utilities on AIX that are either shell scripts (ksh/csh) or in some cases Perl scripts:
As you can see, there are some pretty important commands in this list. And this is just a small sample of them. On my AIX server I found that there are over 400 scripts included as part of base AIX! You can see a full list of all the scripts that make up your system by running a command like this:
It is pretty cool that so many of the core commands/utilities on AIX are made up of shell scripts. For one, it shows that shell scripts can take on very important and critical tasks. It can also be extremely helpful to be able to review the scripts if you are having any issues with any of these commands. And these scripts can be an excellent learning tool. These are extremely well written and robust scripts many of which have been used for decades on thousands and thousands of servers.
Installp is the main software packaging format used on AIX. When you install a installp fileset on an AIX server it installs files in to the appropriate directories on the AIX server. You can easily see a list of files that a installp package installed on the server by running "lslpp -f fileset" (where fileset is the name of any installed fileset).
When you install a installp fileset there is more going on then just files getting copied to the right place. Most filesets also include several scripts that are run when the software is installed or uninstalled. These scripts can do almost anything from stopping/starting processes, creating files, deleting files, changing configuration files or system settings, etc.
If you are having problems installing, upgrading, or uninstalling a installp fileset you might need to take a look at the filesets installp related scripts to troubleshoot the issue.
It is easy to extract these scripts following these steps. In this example, we are working with the SDDPCM fileset.
1. List the files in the fileset and look for the liblpp.a files:
The ./usr/lpp/devices.sddpcm.61/inst_root/liblpp.a file is for the root part of the install, and the ./usr/lpp/devices.sddpcm.61/liblpp.a is for the usr part of the install.
2. Extract the liblpp.a files:
The files are extracted relative to whatever directory you are currently in.
3. Extract the liblpp.a files:
The liblpp.a files contain the scripts, and can be extracted with the "ar" command:
The scripts have now all been extracted. The "config" and "unconfig" scripts are run at install time and uninstall time, respectively. There are also several other scripts that come in to play and a explaination of what each script does is available at: http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.genprogc%2Fdoc%2Fgenprogc%2Fpkging_sw4_install.htm
A little known feature in SMIT when selecting software to install is the ability to scroll over to the right using the right arrow key to view the actual fileset name rather than just the fileset description. This makes it easier to verify that you are installing the desired fileset.
I had been using AIX for many years before I found this feature. I think it is not very commonly known because even if you have a large terminal window the software selection screen is always the same small size and doesn't ever show this information unless you scroll over with the right arrow key. The message says "Use arrow keys to scroll" but I had always thought you could only scroll up/down in the list.
Here is an animated GIF image that visually shows what I'm talking about (the image is large so it might take a minute to download in your web browser):
Over the years I've come across software that during the installation checks the AIX verison/oslevel before installation. If it detects a AIX oslevel it doesn't recognize, a lot of software will flat out refuse to install. For example, if the software was intended to be run on AIX 5.3 and you are trying to install it on AIX 6.1 the installation program might detect this and refuse to install. Some software might even check for a specific technology level and refuse to install on other TL versions.
If the software installation program is a shell script, the easiest and best way to force an install is to look through the installation shell script and find where it is checking the oslevel. Then either modify or comment out the applicable parts of the shell script to allow it to install on your version of AIX. However, if the installation program is a binary program this won't be a possibility. You can use the "file" command to check if a program is a shell script or a binary executable.
Another option is to try temporarily changing the AIX oslevel command. If the installation program is calling the "oslevel" command to determine what AIX version you are on, you can fake it out by creating your own custom "oslevel" command temporarily. To do this, first rename the /usr/bin/oslevel command to something like "oslevel_original". Then create a new /usr/bin/oslevel shell script file. The shell script will just consist of something like this:
Basically you've just created a fake, hard coded oslevel command that you can have report any OS level that the software installation is expecting. When you run oslevel now it will return whatever you hard coded in to the script. Now you do NOT want to leave your system in this state permanently - this would just be a temporary change during the software installation and then you would want to restore the original oslevel command as soon as possible.
The software might also be calling the "uname -v" and/or "uname -r" commands, and you could also create a temporary shell script to replace uname to temporarily fake out this command as well. For example, if you wanted uname to appear to return the results from an AIX 5.2 server you could use this shell script to replace the uname program:
When you run this modified uname script with the "-v" flag it would display 5, and when you run it with a "-r" flag it would display 2 (i.e. AIX 5.2). Again, this would only be a temporary change you would want to make for the software installation and then you would want to restore the original uname command.
These methods will allow you to fake out the AIX version for the vast majority of software installation programs, however some applications might use different methods to determine the oslevel such as making system calls. In these cases there might not be an easy way to fake the AIX version. If you are having a hard time determining how the installation program is determining the AIX level you could try to run the installation program with "truss" to see what system calls it is making.
Like I mentioned earlier there are a variety of situations you might be in where you need to install software on a version of AIX that the software doesn't support. However, always keep in mind that this would not be supported by the software vendor and the software might not work or have other unexpected results. Also, always remember to restore your original oslevel and/or uname commands if you hard code them to return other information.