As 10Gbit Ethernet (10 GbE) hits data centres around the world, you'll need to know how to optimise it. There are a few superb resources available. IBM PowerVM Technical Webinars - Virtualisation and the World of 10Gbit Ethernet The excellent Webinar Series on Power Systems Virtualisation from IBM has free monthly sessions on Virtualisation. Mr NMon himself -- a.k.a. Nigel Griffiths -- presents some of them. Each webinar runs for about an hour and has an accompanying set of slides. If the UK time zone makes the webinar difficult for you to attend, you can download the sessions in Windows Movie Format (.wmv), usually very shortly after the webinar is completed. Gareth Coates, from Advanced Technology Support (what a great name!), IBM Europe presented Virtualisation and the world of 10Gbit Ethernet. See Session 7 on the Wiki, or download the webinar. The slides are available from here. AIXPert Blog IBM Systems Magazine PowerVM Virtualization Redbook Power IT Pro Vendor Specific offerings Some other links that may be more tailored for vendor-specific solutions are: These are just a few of the documents available. You can easily find your way to others if you need them. It's good to see so many resources available for these newer technologies.
|
I just read this comment about Unix permissions in a whitepaper to do with securing databases. The comment is about Progress databases, which may not exist in your environment, but the comment is worth noting anyway: Though counterintuitive, file modification is controlled by permissions on that file, but file creation and deletion are controlled by permissions on the parent directory. In other words, it is entirely legal to delete a file that you cannot modify if you have write access to the directory without having write access to the file itself.
One of the solutions suggested is this: remove all read, write and execute permissions on all the database extent files and directories.
That came from this whitepaper:
|
When you create a file system using SMITTY, the default action is for the file system NOT to mount after a reboot.  Now it might occur to you that generally when you create a file system, you probably want it to be available after a reboot. So you naturally hit the F4 key (or Esc 4 or just the Tab key) so that you can set the option to True. Here's what to do if you've forgotten that option and found your file system hasn't mounted after a reboot. From the command line: The fix is quite easy using the chfs command, and you don't need to edit /etc/filesystems. Here's some relevant information taken from the command documentation for the chfs command chfs CommandPurposeChanges attributes of a file system. For chfs, the -A flag changes Specifies the attributes for auto-mount.- yes
File system is automatically mounted at system restart.
So for a single file system to mount automatically after a reboot, you'd run: In our example above, that would be: chfs
-A yes /test
Fix the lot of 'em What if you create a few file systems at a time, and have overlooked this mount = true option? Simple! Just run the chfs command for all file systems in a volume group. You can list all the file systems in a volume group using the very simple and elegant lsvgfs command. lsvgfs datavg | xargs -n1 chfs -A yes
Now you can reboot, or simply run mount -a. This will attempt to mount all file systems with the true mount attribute set in /etc/filesystems. What about those which are already mounted? You'll get an error telling you not to try to mount them again. Ignore it.
Geek of the Week award For the really keen AIXers, here's some homework. Which is correct: "filesystems" or "file systems"? One word or two? Include links to official IBM documentation in your answer. First prize wins Geek of the Week award.
|
A mksysb backup of the root volume group (rootvg) includes the file /etc/filesystems. This configuration file includes file systems from other volume groups (like datavg) and maybe NFS mounts. When you restore from the mksysb, those other file system entries must be cleaned up, or you'll get mounting errors when you reboot. Here's how to do that cleanup. But first, a comment from an annoyed reader: "What's the fuss about?"
Response: You may be thinking this is no big deal. You can always ignore errors (sigh!). Well, if you don't clean out those entries from /etc/filesystems, you won't be able to recreate file systems of the same name. If you try it (at least the way I do it), you'll take these steps: - create a volume group
- create a logical volume of type JFS2 (Enhanced Journaled FIle System)
- create an Enhanced Journaled File System on a Previously Defined Logical Volume
And when you get to step 3, you'll be told the file system already exists. Then you'll try to remove it, and you'll lose the whole new logical volume with it and wished you'd listened to me. "OK. You were right (again). What do I do?"
Response: You have a few options. If you're pressed for time jump to option 4. Option 1: Edit /etc/filesystems Yes, you could edit /etc/filesystems and take out the stanzas that are not for rootvg file systems. But this is more of a manual process. If someone has added some user file systems into rootvg, you may accidentally remove the mount points, even if the file systems have been restored with the mksysb. And if you're working on restoring several AIX systems as part of a disaster recovery exercise (hopefully a test one), it can be a lot of work to edit /etc/filesystems for each one and know which stanzas to remove and which to keep. Option 2: Delete ALL file systems (for cowboys) What's that, you say? List all file systems using lsfs, and run rmfs to remove each of them. As for the rootvg file systems, well they're mounted anyway, so rmfs will fail. Yeehaa, cowboy! Now be serious and read on, please. Option 3: export the data volume groups before you create the mksysb The exportvg command will remove the data volume group's file systems from /etc/filesystems. But that requires your datavg file systems to be unmounted first, and if they're in use ... let's just move on to option 4. Option 4: Write a script to remove all non-rootvg file systems The script option is looking good. We want to find all file systems in /etc/filesystems: we can use lsfs and capture the mount points. We'll then have a list of file systems and we want to run rmfs for all that don't belong to rootvg. To list the ones that do belong to rootvg, you can use lsvgfs rootvg. If they're not in that list, out they go. First, here's a sample script. The explanation comes afterwards:
# List all JFS2 file system mount points from lsfs for _fs in $(lsfs -cv
jfs2|awk -F: '{print $1}'|grep -v "^#") do
if lsvgfs rootvg | grep "${_fs}" > /dev/null
then
echo "Keeping rootvg file system: ${_fs}"
else
echo "Removing file system: ${_fs}" rmfs -r ${_fs}
fi
done - lsfs -cv jfs2 lists all JFS2 file systems (including the ones in rootvg)
- -c lists the output in colon delimited format. This makes it easy to pick the file system mount point.
- -v jfs2: shows only JFS2 file systems (not JFS or NFS) . A small modification to the script will capture those types of file systems.
Here are the first few lines of a typical output of this lsfs command: #MountPoint:Device:Vfs:Nodename:Type:Size:Options:AutoMount:Acct /:/dev/hd4:jfs2::bootfs:4194304:rw:yes:no /home:/dev/hd1:jfs2:::5111808:rw:yes:no /usr:/dev/hd2:jfs2::bootfs:5242880:rw:yes:no /var:/dev/hd9var:jfs2::bootfs:1966080:rw:yes:no /tmp:/dev/hd3:jfs2:::4194304:rw:yes:no /opt:/dev/hd10opt:jfs2:::1179648:rw:yes:no /data:/dev/datalv:jfs2:::542880:rw:yes:no
The awk command will extract the first field (the file system mount point). awk -F: '{print $1}'|grep -v "^#"
The grep command excludes the header line which starts with a # #MountPoint:Device:Vfs:Nodename:Type:Size:Options:AutoMount:Acct
So far the script has extracted the file system mount points for all file systems in /etc/filesystems. Then you can check each file system to see if it is found in the list of rootvg file systems (lsvgfs rootvg): if lsvgfs rootvg | grep "${_fs}" > /dev/null It's not there? Then time to remove it (using
rmfs ) and its mount point directory (
-r ). rmfs -r ${_fs} . Cleaner than editing The script is short, and can be used regardless of how many non-rootvg file systems you have created. For that matter, if you have some (naughty) data file systems in rootvg, they won't get wiped out by the script. They would have been recreated as part of the rootvg when you did the mksysb restore. Also, the script approach is cleaner than editing, especially if you have many file systems. Note: This script currently only checks Enhanced Journaled File Systems (JFS2). That's in the line: lsfs -cv jfs2 If you want to search for other file systems, such as nfs, or JFS, the script would need some minor modifications. This script hasn't been run in production (for obvious reasons), and you use it at your own risk etc. I'd be grateful for comments and suggestions. Constructive comments and suggestions can be submitted to the blog post if you have a developerWorks login. Other comments and suggestions can be redirected > /dev/null
|
You can create a file using the AIX touch command. The touch command is used to update a file's timestamp, but if the file doesn't exist, then the touch command creates it. That's an easy way of creating a lock file - the sort of file you use to alert the world that they can't log in because some process is running - or a trigger or flag file whose existence is a signal to some script or program to begin the next process. Trigger Happy The problem with creating an empty lock/flag/trigger file is that no one knows where it came from. You may love creating trigger files, but you don't want to leave a mess for someone else to clean up (or not clean up, because they were too afraid to break something). So you want to be neat and tidy.
"What a neat script!"
When you are creating a lock/trigger/flag file, you could use a very elaborate file name:
# touch
the_day_end_process_is_finished_and_users_can_log_in.flag Or include the date stamp in the name:
# touch dayend.$(date +%Y%m%d) But these approaches have their own difficulties. if you call it something like dayend.20120124 or login.flg then years after your script has been retired (yes, I have seen it happen) the empty files become sacred artifacts of a long lost civilisation: no one knows what they are for but they might be important, so you'd better not touch them. Well, you'd better NOT touch them! Instead, put something useful in them such as the date the files were created, the name of the script that created them, and the host name the script ran on: # echo "$(date +%Y%m%d) This file was created by $0 on $(uname -n)" > myscript.flg
# cat myscript.flg
20120124 This file was created by /usr/local/bin/myscript.sh on myhost
That way, if someone in the future should want to clean up files or turn off a script, a process or a server, there'll be half a chance of working out what all those lock/trigger/flag files were for.
|
Someone asked me to remove all empty directories in a particular path.
Option 1: rm -r You can remove ALL directories and all files using rm
-r You will end up with a nice clean system that way, and a new line on your resumé explaining why you left your job at short notice.
DON'T TRY THIS AT WORK!
For that matter, don't try any of these "solutions" until you've tried them somewhere which doesn't leave you wishing you'd never heard of AIX Down Under. Whenever I run any command to remove files or directories, I waste two seconds staring at the Enter key and thinking about it.
If you wipe out your system, your company and your career, remember one thing: it wasn't me who pressed the Enter key.Option 2: rmdir -pRemember, the aim is to remove empty directories. Now just as you can create a directory structure using mkdir -pyou can also clean up a directory structure using rmdir -pLet me explain. As the documentation for the mkdir command explains, the -p flag creates missing intermediate path name directories. For example, instead of creating a directory hierarchy one at a time:
mkdir
/a
mkdir /a/very
mkdir /a/very/long
mkdir
/a/very/long/directory # well you get the idea
You can do it with a single command using the -p flag:
mkdir
-p /a/very/long/directory/structure/which/nobody/will/ever/find
(Incidentally, there's a command like this called mkdirhier. It creates the directory hierarchy just like mkdir -p, but it doesn't give you the permissions options which are available with mkdir.) Well, you can remove the directory structure using rmdir -p:
rmdir
-p /a/very/long/directory/structure/which/nobody/will/ever/findThe rmdir command will remove directories in reverse (assuming you have write permissions), so that command is like running: rmdir /a/very/long/directory/structure/which/nobody/will/ever/find # "find" directory removedThen rmdir /a/very/long/directory/structure/which/nobody/will/ever # "ever" is gone Then will, then nobody and so on.
rmdir will stop removing directories once it gets to one that is not empty. This option is pretty good, but it does have one drawback: you need to know the bottom of the directory path before you can remove it. If someone asked to remove all the empty directories under a certain branch of the directory structure, you could - saw off the entire branch (using rm -r); or
- find all the empty directories (the dead branches) and just prune them.
If it's the latter, then you need to hunt your way down the very end of the path before you can run a command to clean it up. If you're going to do that, it's probably time to create a script ... or use the find command. Option 3: find command with rmdir
find . -type d -exec rmdir {} \;
The type -d limits the search to directories, and the rmdir will remove them. cd /a/very
find . -type d -exec rmdir {} \;
rmdir: 0653-610 Cannot remove current
directory ..
rmdir: 0653-611 Directory ./long is not empty.
rmdir:
0653-611 Directory ./long/directory is not empty.
find: 0652-081
cannot change directory to </a/very/long/directory/structure>:
: A file or directory in the path name does not exist.
However, as you can see, this crashes on the rocks. I think what's happening here is that the find command is storing a list of directories up front, and then traversring the tree, running rmdir first (which fails for all parent directories, since they're not empty) followed by cd to the sub-directory (which fails for the very last child directory). I'm not sure of this, but I'd be interested in your comments. So, if we were to hard-code this in script form, it would look something like this:
directory_list=". ./long ./long/directory ./long/directory/structure"
for eachdir in $(echo $directory_list)
do
rmdir $eachdir
cd $eachdir
done
The rmdir fails most of the time with the error:
rmdir: 0653-611 Directory [directory name] is not empty
and then does the cd
It gets to remove the bottom directory, since it is empty, and then attempts to cd into it, which produces the error: find: 0652-081
cannot change directory to </a/very/long/directory/structure>:
: A file or directory in the path name does not exist. Option 4: find command with -depth and rmdir
Fortunately, the find command has an option which lets you cd first, and then do the -exec rmdir. In other words, it goes from the bottom of the directory hierarchy and then heads up the tree. The option is -depth, which as the find command documentation explains:
-depth
Always evaluates to the value True. Causes the descent of the directory hierarchy to be done so that all entries in a directory are affected before the directory itself is affected.
find
. -depth -type d -exec rmdir {} \;
rmdir: 0653-610 Cannot remove
current directory ..
This is pretty close to where we want to be. You could ignore that error by redirecting STDERR to /dev/null
but that's a bit inelegant. Perhaps you could start from the directory below the one you're in. Other options
There are many other ways of removing empty directories. Most of them involve the find command. That's a smart way, if you're careful not to find too much and remove too much. Some run commands such ls or du, and check the results are what you'd expect from an empty directory, maybe using wc to check the word count. YUK.Some of the solutions are more elaborate than others, and mostly involve a script. I really wish there were simply an option which let you do
rmdir -r
but only get rid of the empties.
|
Free Blog Downgrade for all our followers After violent battles and verbal exchanges which were unbecoming to AIXers, the team of writers and editors here have finally opted for a minimal theme for the blog. We here in Sydney have been working around the clock (it took a good two minutes) to use a mobile-friendly Minimal Theme. What does this mean for you? - full support for SmartPhone and tablet views
- a Twitter widget
- Trending topics
- ... and plenty more.
AIX Down Under should be faster and slicker. In short, a better experience for you.
|
After restoring a mksysb to a new logical partition the reboot hung with the AIX IPL progress code "0551" "IPL Vary-on is running." After trying slight variations in the profile the boot process hung at "Starting kernel". The rootvg couldn't be varied on, and I didn't know why. The solution was simple, as you'll see. Troubleshooting the boot
As part of the troubleshooting process, I booted the logical partition off the AIX Product Media. I was able to Access the root volume group, and recreate the boot image using bosboot, and set the bootlist. But still the system wouldn't boot off the hard disk. I tried booting off dedicated disk, in case the virtual SCSI disk or drivers were an issue. No luck there. I tried restoring the mksysb onto a different managed system. It worked. I also tried restoring mksysbs from other AIX logical partitions using the same target logical partition profile as the one which was giving me grief. They all worked fine. Perhaps the problem was the source logical partition, from which the mksysb was made. The problem was that I didn't want to make any changes to the original source system until I could be confident that a mksysb restore would work in the event we needed it. After escalating the problem to IBM and doing what seemed to be an endless amount of (pretty interesting, but time-consuming) troubleshooting, we were about to capture boot debugging, but before we did, the IBM support person suggested a solution which was remarkably simple:
Add memory! Even though the 4 GB was sufficient for the other mksysbs which I restored, for this one for some reason it wasn't enough. I assigned an extra 2 GB and the system booted perfectly. In defence of iron casting In a sense I suppose I was too clever by half; a victim of my own experience. A beginner might have suggested we throw some iron at the system: "Why don't you try adding some more memory?" None of the error codes or troubleshooting I did explicitly showed that was where the problem lay, so the beginner's approach wouldn't have been very scientific. But it would have worked.
|
If you want to get a view of the process tree, you could try it using the ps command, but there's a better way: the proctree command. You can see all of its parent processes and its child processes. Here's how it looks. This is an installation of IBM Systems Director Management Server version 6.3.
# ./dirinstall.server -B
/database --DB=1
+=============================================================================+
Start of product installation on myhost
+=============================================================================+
Variables will be used during the
installation:
PRE_INSTALL_CHECKS : 1
PRE_INSTALL_WARN_ABORT : 0
PortNumber : 8421
SecurePortNumber : 8422
AGENT_MANAGER_PORT : 20000
MIGRATE_DATA : 1
UPDATES_PATH : /database/inst.images/packages/updates
-Managed DB2 is supported and its prerequisites are met.
DB_INST_TYPE : 1
DB_DATAPATH : /database
DB_PWD : default.
DB_SERVER : localhost
DB_PORT : default
+=============================================================================+
/tmp free space not enough, less than 1536M, trying to increase it...done
/home free space not enough, less than 1024M, trying to increase it...done
+-----------------------------------------------------------------------------+
Attempting to install Managed DB2...done
Attempting to install sysmgt.cimserver.pegasus Director.install.msg ....done
Attempting to install
sysmgt.cim.providers... To see what commands are being run at this stage, you can first find the Process ID (PID) of the Director installation command, dirinstall.server
ps -ef | grep dirinstall.server
root 8847562 9764866 0 17:07:37
pts/1 0:00 /usr/bin/ksh ./dirinstall.server -B /database --DB=1
Now you have the PID (in this case 8847562), you can run proctree.
# proctree 8847562
3342538 /usr/sbin/srcmstr
4063378 /usr/sbin/sshd
8519734 sshd: root@pts/1
9764866 -ksh
8847562 /usr/bin/ksh ./dirinstall.server -B /database --DB=1
9633942 /usr/sbin/installp -acgNQqXY -d
/database/inst.images/packages sysmgt.cim.provi
4522126 /bin/ksh /usr/lib/instl/instal
/database/inst.images/packages/sysmgt.cim.provid
5963860 /usr/bin/ksh sysmgt.cim.providers.osbase.config
9896132 /bin/ksh
/opt/freeware/cimom/pegasus/etc/orig/rebuild_repository -fm
9175250 cimmofl -aE -W -w -R/opt/freeware/cimom/pegasus/etc
-I/opt/freeware/cimom/pegas
13107380
Going down the tree shows that the dirinstall.server command is calling installp, which is installing some filesets for use by Systems Director.
Heading up the tree to parent processes, you can see that the
dirinstall.server command is being run from the Korn shell (-ksh). This,
in turn, is run out of an ssh (secure shell) session. The ssh daemon
(sshd) was invoked by the System Resource Controller srcmstr.
The proctree command is well worth remembering. It can give you a quick view of the processes going up and down the tree.
|
For network redundancy, you can set up two Shared Ethernet Adapters (SEAs) - one on each Virtual I/O Server (VIOS). The two SEAs work together with one active, the other idle, so the configuration is called failover mode. This is done by a control channel - a heartbeat that works through a virtual ethernet adapter on each SEA. The failover mode is very effective, because it allows you to shut down a single VIOS and automatically have the other SEA on the alternate VIOS take the traffic. With failover mode you can use VLAN tagging, something not available on the Network Interface Backup configuration.
Idle adapter
The failover does have one big drawback. The non-active adapter sits idle while all the traffic passes through the primary adapter. With a relatively expensive 10-GB adapter sitting idle, it seems a bit of a waste. Virtual Switches The Virtual Switch solution required a Network Interface Backup (NIB) configuration on the VIO clients. This would require a primary and a backup virtual Ethernet adapter on each logical partition. This solution was inventive, but I wondered at the time whether it might get a little too complex, especially when you get to a larger number of logical partitions. Load Sharing Now there is an option for configuring SEA failover (not Netwrok Interface Backup) with Load sharing mode. This means you can have both physical network adapters active, instead of one sitting idle in the marketplace, waiting to be called up for duty. This loadsharing configuration leaves any complexity on the VIOS and away from the VIO clients. You have to be running VIOS version 2.2.1.0 or later (check your version by logging in as padmin and running the ioslevel command).This saves the manual load balancing by hand which happens when you use NIB. The load sharing option looks to me to be a much more attractive option than the earlier solutions. For more on Virtual I/O Server Load sharing (as well as some excellent slides on the Shared Storage Pools), see here.
|
Some time ago I experimented with removing the default paging space /dev/hd6 from an LPAR and creating a separate volume group for paging space. This worked ... until I had to restore a mksysb backup, as you may recall from this blog post. Well, what about creating a separate paging space in rootvg instead of in a separate volume group, and then removing hd6? Not exactly sure why you'd do that, but I suppose if you were to create a smaller paging space to replace hd6, in case it's been extended too much, that might be a reason. Whatever your reason for renaming, removing or retiring hd6, if you decide to do it, you should read technote T1010881. Didn't read it did you? (Or if you did, you probably found it more interesting than coming back here). Well never mind. Here's the key sentence:
"Because certain scripts are currently hard-coded to activate /dev/hd6, it is recommended to create the new paging space under the same name." If you change the paging space from hd6 to something else, you have to update /sbin/rc.boot which is a script. The thought of editing /sbin/rc.boot horrifies me. It's just inviting trouble. Do changes get retained when you upgrade the OS to a new release? I hope so, but I don't know. Still, if you do decide to edit the rc.boot file, then make sure you follow the instructions in that technote, or you could be risking a mksysb that can't be cloned or at the very least a system that doesn't activate paging space after the next reboot.
My advice? Just keep hd6 as a default paging space, even if you have other paging spaces in other volume groups. If you're still keen on resizing or relocating the hd6 paging space, see this document.
|
One of the great strengths of AIX is
the Logical Volume Manager (LVM). It may not be completely foolproof,
but it does provide quite a lot of protection from administrator
errors.
Here are a few common LVM tasks that
I've deliberately attempted to make a mess of. See how the LVM
reacts (Don't Try This at Home)
Let's start with three Physical
Volumes. rootvg has one PV (booting from SAN) and datavg has two PVs.
# lspv
hdisk0
00c5a47ecf9edd3e rootvg active
hdisk1
00c5a47ed021ea7d datavg active
hdisk2
00c5a47ef172e84d datavg active
So, let's try to do some damage:
import a volume group that is already
active:
#
importvg -y datavg hdisk1
0516-360
getvgname: The device name is already used; choose a
different
name.
0516-776
importvg: Cannot import hdisk1 as datavg.
"Just a moment, please!"
If you try to remove a disk from the operating system Object Data Manager (ODM) while it's still in an active volume
group, you'll get this warning:
rmdev
-dl hdisk1
Method
error (/etc/methods/ucfgdevice):
0514-062
Cannot perform the requested function because the
specified
device is busy.
What about trying to remove an active
disk from a volume group?
reducevg
datavg hdisk2
0516-016
ldeletepv: Cannot delete physical volume with allocated
partitions.
Use either migratepv to move the partitions or
reducevg
with the -d option to delete the partitions.
0516-884
reducevg: Unable to remove physical volume hdisk2.
"Sorry, I'm busy!" Not getting very far, are we? Alright, let's try to export a volume group that's in use:
exportvg
datavg
0516-764
exportvg: The volume group must be varied off
before
exporting.
OK, I'll vary it off then:
varyoffvg
datavg
0516-012
lvaryoffvg: Logical volume must be closed. If the logical volume
contains a filesystem, the umount command will close the LV device.
So I have to unmount the file systems
in the volume group first. But if they're in use, I'm also protected
from doing damage:
umount
/mksysb
umount:
0506-349 Cannot unmount /dev/lvmksysb: The requested resource is
busy.
The order of exporting a volume group
is:
stop processes using file systems
in the volume group
unmount each file system in the
volume group (and you have to unmount nested file systems such as
/reports/year_end before you can unmount the parent file system
(/reports)
varyoff the volume group
export the volume group.
I can think of many other instances
where the LVM protects me from my own stupidity. I don't have to
remember to increase a logical volume first before expanding a file
system. If I do try to overallocate beyond the MAX Logical Partitions
(shown as MAX LPs in the output of the lslv command), I get an error.
Relax - this is AIX!
Good software should protect you (at least a little)
from glaringly dumb mistakes. The AIX Logical Volume Manager does that. This can give you a little more confidence when
you're breaking the mirror of a logical volume or adding the wrong PV
to a volume group. You can still do damage (like running rm
/dev/* - but don't say you saw it here), and there are in some cases flags which allow you to force the action, but the very existence of those flags are part of the checks and balances. That means you can be a bit more relaxed than
you might have to be on other platforms.
The LVM is another reason why AIX is a
true enterprise operating system.
|
There are no coincidences.
Whenever I hear of any sort of problem at all, I wait to see if an entirely unrelated problem is reported soon after. This is especially worthwhile for performance issues, or when something suddenly stops working. I came across a site where a printer was down. Within half an hour a controller error was reported from a disk subsystem which wasn't used by the system that had the printer go down. Coincidence? I was suspicious.
Temporary problem: permanent solution
We soon found out that the printer was down because all printers on the virutal machine (the LPAR) were unable to print And that was because you couldn't create any new files in /tmp. You could update existing files. This indicated the file system inode map was corrupted. That would require an fsck on /tmp, which would need a reboot of the LPAR. We could fix the /tmp but that still didn't address the cause of what made the file system corrupt in the first place. Usually that indicates a link to the storage had been interrupted or the storage subsystem itself was damaged. Sure enough, we soon found that many other AIX virtual machines and other Power or Wintel systems had storage errors at about the same time. Most of them had redundancy to the disk subsystems, but the ones that didn't have were the ones that suffered some sort of data or OS impact, such as the corrupt /tmp file system.
As it turned out, a SAN switch module had rebooted itself and was the cause of all the problems from disk subsystem error to file system corruption to printers being down. There was some work involved in repairing the switch module and removing it as a single point of failure. And it all began with the report that a printer was down.
|
Nigel Griffiths (of IBM Power Systems Advanced Technology Support, Europe - but he is also known as Mr Nmon) gave a great presentation on Shared Storage Pools at Copenhagen. This was also presented in the excellent Webinar Series on Power Systems Virtualisation.
Here's a small extract on why you would use Shared Storage Pools:
In smaller companies, there is no SAN team. The server guys are the SAN guys. The idea of Shared Storage Pools is to allocated LUNs to the VIOS(s) and a single VIOS command can allocate space to a Virtual Server (an LPAR). You can run a single command using cfgassist (VIOS's smitty) to allocate disk space. You can even use the Virtual Storage Management feature in the HMC GUI.
In the two years since this post was first written, there have been a number of other presentations on Shared Storage Pools. You can get hold of these from the Webinar series linked to in this post. (Links have been updated).
Modified on by AnthonyEnglish 270000RKFN
|
Whenever I see a script with a five or ten minute wait in it, I ask
myself what it's waiting for. I recently saw a script which started a
database, waited 10 minutes (using sleep 600), then started an app
server (sleep 600), then started another application (sleep 600, then
exit!). That script gets a lot of sleep. The
reason behind the delays was to wait for the previous process to
complete. But that could be confirmed in a few ways. Ordinarily, the
previous command will have an exit code which you can test for, or you
could use the Logical AND (&&) operator.
cmd1
&& cmd2
This means only run cmd2 if cmd1 completes successfully (exit code 0). If
the first process is run in background, then you may have reason to
delay before proceeding to the next process. But a ten minute wait is no
guarantee that the first process completed successfully, so it would be
smarter to use a shorter delay, or even set up a loop which checks
periodically that the first process has completed.
In the example script I gave above, the procedure
delayed 600 seconds (that's 10 minutes if you're not a mathematician) ,
then 600 seconds then 600 seconds again, or 30 minutes in total. When
the script was first written it was meant for a single database. But
then the script was used for two other databases, and soon enough a
restart would take 90 minutes out of an outage window.
If you really must do a sleep in your script, set the time as a variable in the start of the script or passed as an argument from the command line:
bigsleep=600 # Wait 10 mins for something (or nothing) to happen
Then you can do the sleep if you really need to:
sleep $bigsleep If you need to run the script without the long delay, you can shorten it by changing the delay time in one place.
|
If you've been following this blog for a while, you might remember a post on the Systems Management Interface Tool (SMIT), better known as SMITTY to his friends. That post covered the frequently-asked question about the difference between smit and smitty. It also provided some handy SMIT Keyboard shortcuts. Well, here are some new keyboard shortcuts that should be real time savers for you. They're not strictly new, as they're about 25 years old, but some of them were new to me, and they may be to you. Some of them are real gems, such as the shortcuts to do with editing fields and moving to the top or bottom of the selection screen. I've grouped the shortcuts into: Cursor movement, Selecting and editing, and Menu navigation / Miscellaneous. This is from a short article I wrote for PowerITPro. The PowerITPro AIX Tips & Tricks are currently published on the 2nd and 4th Tuesday of every month and you can subscribe to them here. Here is a summary of the keyboard shortcuts. You can see the full article SMIT Keyboard Shortcuts on the PowerITPro web site. Cursor movement | Keyboard shortcut | Scroll to the next page | PageDown or Ctrl-V | Scroll to the previous page | PageUp or Esc-V | Scroll to bottom of command output | End (Esc > ) | Scroll to top of command output | Home or Esc < |
Selecting and editing | Keyboard shortcut | List valid choices | F4 (Esc + 4) | Toggle between choices (e.g. true / false) | Tab key | Reset the entry field to original setting | F5 (Esc + 5) | Delete next character | Ctrl + x | Delete to end of line | Ctrl + k | Select entries on multi-select lists | F7 (Esc + 7) | View and edit a long field | F7 (Esc+7) |
Menu navigation and miscellaneous | Keyboard shortcut | Back up one level to previous screen | F3 (Esc + 3) | Escape to shell | F9 (Esc + 9) | Exit from SMIT | F10 (Esc + 0) | Display SMIT FastPath | F8 (Esc + 8) | Refresh screen | F2 (Esc + 2) |
|
It wasn't a hostile question from a bean counter looking to cut costs. It was from a visitor - a school student on work experience (a week or two of trying out life in "the real world"), and I was asked what I did. How could I answer that? "AIX"? Too vague and obscure for someone who's not in the industry. System administration? So does the guy who supports the laptops. How about "I manage stuff that no one understands"? How do you justify your existence in 30 seconds? I looked over my day, and thought about how to answer. I tried to make myself sound intelligent (but not overbearing), and to make the work sound important (but I always had time for a friendly question from a visitor on work experience) and even urgent (but not so much that I left the impression that I wasn't prepared for it). In the end, the visitor left as confused about what I did as I was.
Crushed
After a few nanoseconds of deep reflection, I had to admit the day included a few less glamorous jobs. Like crushing cardboard boxes. Sure, they had contained the latest and greatest equipment, but the boxes did need to be crushed and carted downstairs.
What else? I thought of saying "troubleshooting" but then that sounded like there might be a lot of trouble to be shot. "What sort of trouble?" So what exactly did I do? Can I justify my pay packet in 30 seconds?
How about these? - checking mksysb backups worked
- planning for firmware upgrades on a tape library and its associated drives
- refilling the photocopier
- taking three calls from people who dialled my number by mistake (it's close to the help desk number for a major pay TV provider)
- increasing a file system
- checking emails and working out whether to answer / delete or postpone action
- create Yet Another Spreadsheet to match up the WWPNs for some FC adapters to the totally different format provided by the guy who looks after the SAN switch
- hunting for that suction thing which lets you pull up tiles in a raised floor data centre. Found: suction thing. Lost: 10 minutes.
And I chatted to someone whose AIX 5.2 legacy system with dedicated adapters and processors is about to be extinguished and tried to assure that person I wasn't just hanging out for the hardware the very moment the LPAR gets shut down.
Whose idea was it anyway?
I'm not a conspiracy theorist (usually) but I have to wonder whose idea it was to go asking people throughout the organisation what they did. I mean, techies like me are either going to be stumped for a way of summarising the job or enable the verbose flag and then bury the questioner with detail. If I was looking after the organisation's social media strategy I would have been ready for the question and I could tweet the answer, but instead I'm a techie.
When it comes to giving your job description out, it may not be all that exciting, at least in parts. It's hard to make it sound interesting, essential to the organisation, unique and yet not mysterious or suitable for eccentrics. Perhaps I should have stuck with the troubleshooting answer. Or maybe the cardboard box one.
|
If you mix production and non-prod LPARs on the same managed system, you may be worried that the non-prod LPARs could chew up much-needed production processing juice. What are your options here? Here are some common solutions: - A separate shared processor pool for Prod systems
- Capping the non-prod LPARs
- Implicit capping of non-prod LPARs
This last option is dealt with in a Redpiece (part of the Redbooks) called " SAP Applications on IBM PowerVM". The Redpiece is in draft format..Even if you're not running SAP, it provides a good introduction to PowerVM Virtualisation, which is well worth reading. Before looking at the Implicit capping, here are some of my thoughts on the first two options: Separate shared processor poolsSeparate shared processor pools are not available on all systems, And they do also isolate the LPARs to their own shared processor pool, which is (if you'll pardon the momentary escape into a sporting analogy) a bit like a hard wall separating the lanes in a swimming pool. The two pools are separate, which can be a bit restrictive, especially if you're on the crowded side of the wall. Capping of the non-prod LPARs
This is like a wall in the swimming pool for the non-prod swimmers. The prod swimmers are allowed to swim in and out of the non-prod area, but the non-prod guys have to stay where they are. Good for prod. If you prefer, here's a back yard analogy: you are allowed to wander into your neighbour's yard when you're pressed for space, but your neighbour isn't allowed into yours.
Now the let's see what the Redpiece says about implicit capping.
Implicit capping of non-prod LPARs
First, a word about using DLPAR:
Most LPAR settings can be dynamically altered during runtime by DLPAR (dynamic LPAR) operations using the HMC. Using this functionality, you can manually change the behavior of the LPARs or schedule DLPAR operations using the HMC directly or by external tools.
It's good to have the dynamic LPAR option, but as it requires user intervention or at least scheduling of the DLPAR operation (not available on all systems), you're better to leave it to the hypervisor. So here's the suggestion:
The number of VCPUs [they're referring to Virtual Processors] that are available to a partition limits the number of whole physical CPUs that this LPAR can address, because one virtual CPU represents at maximum one physical CPU. The LPAR is therefore effectively capped at a maximum physical processor utilization that is equal to its number of configured VCPUs.
The configured VCPUs is the entitlement for Virtual Processors.
Here's what they see about the LPAR weight:
By reducing the LPAR weight, you do not actually cap the LPAR; instead, you reduce its ability to compete for additional resources above its actual entitlement. If there is no competition for these resources, the LPAR is only restricted by its number of configured VCPUs, which allows it to be both flexible and harmless to other workloads of higher priority (with greater weights).
The advantage of implicit capping versus explicit capping is: - The flexibility that is gained by low reserved entitlement, which allows more SPLPARs to be defined on the same number of physical CPUs on the one hand and on the other hand a second “implicit entitlement” can be guaranteed.
- The protection of SPLPARs with low weights from being completely cannibalized by uncapped SPLPAR with higher weights.
- Limit the maximum CPUs in hosting environments.
- Can be scheduled as described in “Scheduling dynamic LPAR operations” on page 34, which is not possible with the weight.
I'll have to think through this but there's a lot of very helpful information on virtualisation in this SAP Applications on IBM PowerVM Redpiece. It's worth reading even if you're not running SAP systems. Be sure to read the Micropartitioning design option in section 4.2.
|
The team from IBM in the UK have been running some great webinars, and the next one promises to be a beauty. It's on Virtualisation (British spelling) Best Practices.
A small error in the beginning is a big error in the end.
It's true in philosophy, it's true in maths (British usage), it's true in taking directions and it's true in setting up your virtualised environment. It's much easier to use good planning before implementation than to have to untangle everything afterward. This webinar promises to cover how to: - plan
- document
- implement
- manage
the virtualised environment. The webinar is scheduled for Sept 14th from 10:00 to 11:00 BST (UK time) and you can both register for the free webinar or download it afterwards by connecting to:
I'm looking forward to it.
|
If you thought IBM Watson winning Jeopardy! was just a gimmick, think again. There are people who have seen Watson in action and say: "I want one!" IBM has now established a Watson Solutions team.
I understand that Watson is going to be back in the spotlight very soon, with a re-broadcast of the Jeopardy shows in the week of 12 September. It sounds like there's a lot more planned to apply Watson's technology in various industries.
I found a video of how Watson answers questions very interesting. As they say in the video:. "A computer understands code: ones and zeroes." So how does Watson do it? Have a look:
|
100
