Note AIX 7 Open Beta is now available, fulfilling part of the commitment made in IBM United States Software Announcement 210-056 dated April 13, 2010: "IBM plans to deliver the next version of the AIX operating system, AIX 7, and new releases of PowerVM and PowerHA™ SystemMirror for AIX. ... IBM intends to offer early access to AIX 7 via an Open Beta program and access to the new release of PowerHA SystemMirror via a Beta program."

Note AIX V5.3 will be withdrawn from marketing on April 29, 2011. There are few differences between AIX V5.3 and V6.1. Please beginning planning now for migration off of AIX V5.3. With minor exceptions, the AIX V5.3 best practices below are applicable to AIX V6.1. IBM United States Withdrawal Announcement 910-043, dated April 13, 2010, says: "Effective April 29, 2011, IBM will withdraw from marketing, Software License under the IBM International Program License Agreement. Program Feature number Program name number 5765-G03 AIX 5.3 Operating System ALL 5692-A5L AIX 5.3 SIPO ALL "Effective April 29, 2011, IBM will withdraw from marketing, HIPO features that support IBM internal preload process. Program Feature number Program/Feature name number 5313-HPO AIX 5.3 1053 "Note: The 5313-HPO program is not being withdrawn. "On or after the effective dates of withdrawal, you can no longer order these products directly from IBM or an IBM Distributor." For some time after withdrawal from marketing, IBM will continue to provide software maintenance (SWMA) for AIX V5.3.

AIX V5.3 system administration best practices

A Service Level Agreement provides very important guidance in determining how a server should be configured and managed and, consequently, the best practices to be implemented to configure and manage it. A Service Level Agreement (SLA) is the contract between an IT provider and business users, often established within an ITIL framework or the like. (Within ITIL, Incident, Problem, and Change Management processes are particularly important. IBM offers services to help optimize IT investments, improve performance, achieve availability objectives, and avoid costly problems. See the Integrated Technology Services Infrastructure and systems management web page.) An SLA clearly states the level of IT services which users require. The SLA for a departmental file server is likely to be quite different than that for an enterprise data base server. This page attempts to document best practices which are appropriate across a broad spectrum of servers. Every best practice documented here will not be applicable to every AIX server.

This web page is meant to supplement, not replace, other related best practices which have been published

• Service and support best practices for UNIX servers
• Paging Space Policies and Practice white paper
• Best practices for implementing Oracle on AIX:
  1. Oracle 9i & 10g on IBM AIX 5L: Tips & Considerations
  2. Oracle DB & RAC 11g on IBM AIX: Tips and Considerations
  3. Oracle DB & RAC 10gR2 on IBM AIX 5L: Tips and Considerations
  4. Implementing Oracle 10g RAC with ASM on AIX white paper
  5. Oracle Architecture and Tuning on AIX white paper
  6. Tuning IBM AIX 5L for an Oracle Database white paper
• IBM System p Advanced POWER Virtualization Best Practices Redpaper
• IBM eServer Security Planner and the Strengthening AIX Security: A System-Hardening Approach white paper
• Learn 10 good UNIX usage habits web page

This web page is intended to build upon (not replace) basic UNIX system administration 101

• When installing AIX for the first time (or upgrading to a new release level), be sure to install available fixes. See the AIX V5.3 installation best practices web page for more information. When installing AIX on a LUN on a Storage Area Network (SAN), be aware of considerations unique to that environment. See AIX V5.3 boot from SAN for more information.
• Capture bootable backups periodically (monthly/quarterly?). See AIX V5.3 backup and restore for more information regarding AIX backups.
• Store some bootable backups off site for recovery if the data center is destroyed.
• Test the restore process periodically (yearly?) by restoring the most recent bootable backup. Wouldn't want to discover that a backup can not be successfully restored while in the middle of disaster recovery, would we? Doh! It is best for the restore to be tested by someone other than the person who captured the backup, to confirm that restore procedure documentation is adequate.
• Capture application data backups (volume groups other than rootvg and filesystems not mounted when mksysb is captured).
• Test application data restore process periodically (yearly?) by restoring the most recent backup. It is best for the restore to be tested by someone other than the person who captured the backup, to confirm that restore procedure documentation is adequate.
• Monitor the system for errors. Attempt to discover the root cause of every error and to address the cause to minimize the number of errors which occur, while acknowledging that getting a failed system back in operation must sometimes take precedence over collecting the diagnostic information required to determine failure root cause. The primary AIX error log can be displayed using the errpt command. Please note that an AIX Error Notification exit can be used to take action (eg, send an email) if a particular error occurs. The primary HMC error log can be displayed from the HMC GUI using Service Applications -> Service Focal Point -> Manage Serviceable Events. Please note that Service Applications -> Service Agent -> Customer Notification can be used to configure an HMC to send an email when a new serviceable event is logged. Please note that alog -t console -o will display messages which have appeared on the AIX system console and alog -t boot -o will display messages which were generated as AIX booted up.
• Conduct a post mortem after each application outage. Attempt to answer the following questions and then act upon the answers: (1) Was there any warning of this outage? If so, why was the warning not acted upon in time to prevent the outage? (2) What changes can be made to prevent the outage in the future? (3) Are other servers exposed to similar outages?
• Test operating system, middleware, and application changes on a test server before implementing them in production. The test server's configuration should be as close as possible to the production server's. This is especially important when service level agreements impose availability and/or performance requirements on the production server.
• Implement procedures to protect servers from intrusion (eg, implement and enforce password standards, force passwords to be changed periodically, enable AIX system auditing and implement procedures to review audit logs regularly, etc). See the AIX Security Expert, the AIX Security checklist, and the Install OpenSSH best practice below.
• Run commands, applications, cron jobs, etc as root only when absolutely necessary. (Please note that AIX allows sysadmins to define administrative roles which grant non-root users access to functions which normally require root access. For more information, please see the Administrative roles article in the AIX V5.3 Security manual. And please note that sudo for AIX is available for download from the AIX Toolbox for Linux Applications web site.)
• Install middleware and application software in filesystem(s) dedicated to that purpose. (That is, don't install middleware and applications in /home, /, /usr, /tmp, or /var. Any new filesystem must, of course, have a mount point (eg, /appl) in some existing filesystem.) Even when a vendor packages software in AIX installp format, it is possible to allocate a dedicated filesystem for the software before installing it, if desired. For example, the IBM C compiler for AIX (XL C Enterprise Edition for AIX) is packaged to install in /usr/vac. If a /usr/vac filesystem is allocated and mounted before the C compiler is installed, when installing the compiler the AIX installp command will check the /usr/vac filesystem (rather than the /usr filesystem) for required free space, will allocate more space to the /usr/vac filesystem if free space is insufficient, and will install the compiler's filesets in the directory tree below /usr/vac.
• Implement procedures to collect and archive server performance information for capacity planning purposes.
• Implement procedures to regularly prune files which may grow without limit (eg, /smit.log, /var/adm/wtmp, and /etc/security/failedlogin) and remove stale files from /tmp. ( Note: AIX does not remove files from /tmp at boot time - consider implementing skulker by uncommenting the line in root's crontab which invokes it, after first reviewing the skulker shell script to understand what it does.)
• Maintain a change control log which documents changes and the date & time they were made. Use this change control log when a new problem crops up to identify changes which may have led to the problem. (Please note that the smit and smitty commands log every action in the /smit.log, which can therefore provide useful supplementary detail to a change management log.)
• If appropriate, implement an accounting and charge back system which is fair to all users. To this end, please see the Accounting and Auditing on AIX 5L Redbook and please note that the IBM Tivoli Usage and Accounting Manager supports AIX Advanced Accounting.

Note The contents of this web page solely reflect the personal views of the authors and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management. Please use the Add Comment link at the bottom of the page to provide feedback. Note: Until you log in (using the link in the upper right corner of this web page), you will not see the Add Comment link and you can not add a comment. If you do not already have an IBM ID, use the Register Now link on the sign in page to obtain one. Registration is quick and easy.

Here is an index to the best practices below

1. Don't store local files in / and /usr, minimize custom changes to executable AIX files in / and /usr (eg, /sbin/rc.boot), and if changes must be made in / and /usr, make sure those changes are documented for the benefit of others who might assume responsibility for administration of the AIX image at some future time.
2. Configure /etc/rc.d/rc2.d/Src.local and /etc/rc.d/rc2.d/Krc.local shell scripts which AIX will run at boot and shutdown, respectively.
3. Make sure adequate system dump space is allocated.
4. Prepare the system so you can initiate a stand-alone dump if AIX hangs (won't allow logins - might or might not respond to a ping).
5. Never edit /etc/filesystems to define or modify a filesystem definition.
6. Before manually editing any file in the / and /usr filesystems for the first time, save a copy of the file.
7. Configure the root user so you always know which system you are on and which directory you are in.
8. After installing base AIX, install optional components by installing bundles rather than filesets.
9. Install a few individual filesets in addition to bundles.
10. After installing any new optional AIX filesets, always reinstall the current AIX Technology Level and Service Pack.
11. Before creating any userids for which /bin/ksh is the default shell, update the /usr/lib/security/mkuser.sys file shipped with AIX to install a profile other than the system default when a new userid is created.
12. Install OpenSSH.
13. Update $MANPATH for all users so that the man command can find man pages installed by applications (such as AIX Toolbox for Linux Applications software).
14. Configure TCP/IP to search local /etc/hosts to resolve a host name before trying DNS.
15. Configure AIX to use an NTP server to synchronize the system clock.
16. Configure AIX Workload Manager (WLM) to reserve CPU time for the root userid.
17. When configuring an ASCII console/terminal, configure AIX to set the TERM= shell environment variable correctly during login from that terminal.
18. Add useful shell scripts to /usr/local/bin and make them available to all users.
19. Edit /etc/profile to make useful changes to the shell environments of all users.
20. Install lsof (LiSt Open Files) utility.
21. Consider installing a web browser.
22. Consider installing the Adobe Acrobat reader.
23. Use smitty crcdrfs to create a CD-ROM filesystem with a mount point of /cdrom.

Best Practice:	Configure /etc/rc.d/rc2.d/Src.local and /etc/rc.d/rc2.d/Krc.local shell scripts which AIX will run at boot and shutdown, respectively.
Why:	For reasons described above, it is best not to update any files in / or /usr (such as /sbin/rc.boot) which are shipped in AIX. But sysadmins will certainly want to configure the system to run commands at boot time. Such commands can be placed in /etc/rc.d/rc2.d/Src.local.
How:	Create /etc/rc.d/rc2.d/Src.local and /etc/rc.d/rc2.d/Krc.local shell scripts owned by root.system with 550 permissions. Src.local will be invoked by AIX just after AIX boots up and Krc.local will be invoked by AIX just before AIX shuts down. Write the scripts carefully and test them thoroughly to make sure they will not hang under any circumstances. See the Run level script execution article in the AIX V5.3 Operating system and device management manual for more information regarding run level scripts. Note: This is one of the few exceptions to the Don't store local files in / and /usr best practice, which should be documented in /usr/local/README or equivalent, as suggested in the best practice. Note: Execution of run level scripts is driven by the l2:2:wait:/etc/rc.d/rc 2 line in /etc/inittab. If run level scripts do not get invoked at AIX boot time, confirm that the line is still there. And please note that if a run level script hangs, the AIX boot process will hang at that line in /etc/inittab waiting for the run level script to finish.

Best Practice:	Prepare the system so you can initiate a stand-alone dump if AIX hangs (won't allow logins - might or might not respond to a ping).
Why:	It is too late for preparation when AIX is hung. If AIX hangs and you have no way of initiating a standalone dump, you have no way of collecting diagnostic information to determine why AIX hung.
How:	If the system is in a secure area (where unauthorized personnel can not gain physical access to it), invoke the AIX command sysdumpdev -K. A standalone dump can then be initiated at any time (even if AIX is hung) using any of the methods described in the System Dump Facility article in the AIX V5.3 Kernel Extensions and Device Support Programming Concepts manual. Be sure to follow links from the main article to other articles (eg, Start a System Dump). And there are other ways to initiate a system dump not mentioned in the System Dump Facility article. On POWER5 LPARs managed by a Hardware Management Console, the Dump option of the Restart Partition function can be used to initiate an AIX stand-alone dump, as documented in the Using the Hardware Management Console to restart AIX logical partitions article in the POWER5 Partitioning for AIX with an HMC manual. A system dump can be initiated remotely via a modem or terminal server after enabling the remote reboot facility using the smitty rrbtty fast path. But according to PMR 24881,L6Q, the AIX remote reboot facility does not work for a system (integrated serial) port on a POWER5 system. One should instead enable serial port snoop (see the Enabling serial port snoop article). While an LPAR is dumping, dump progress indicators (0c0, 0c2, 0c9, etc) will appear on the HMC and/or in the LCD display. The various possible indicator values are documented the "Dump progress indicators (dump status codes)" section of the Progress codes overview article in the Power Systems Progress codes manual.

Best Practice:	Never edit /etc/filesystems to define or modify a filesystem definition.
Why:	AIX keeps information regarding each JFS and JFS2 filesystem not only in /etc/filesystems but also in the LVCB of the logical volume on which the filesystem is defined. (The LVCB (logical volume control block) is in the first 512 bytes of a logical volume. Search the AIX V5.3 Information Center for more information about the LVCB.) When a volume group is imported, AIX reads the LVCB of every logical volume in the volume group and adds filesystem definitions to /etc/filesystems. And when a volume group is exported, AIX deletes from /etc/filesystems the definition of filesystems in the volume group. So if /etc/filesystems is edited to change a filesystem definition, the definition will revert to its original state if/when the volume group in which the filesystem resides is exported and re-imported. Not a good thing.
How:	Use smitty manfs or the AIX chfs command to change a filesystem definition. Filesystems are mounted at AIX boot time in the order they appear in /etc/filesystems. Filesystems are added to /etc/filesystems in the order they are defined. See the How to change the order in which AIX V5.3 mounts filesystems web page to understand how to change the order in which filesystems are mounted at AIX boot time without editing /etc/filesystems. Use smitty manfs to remove a filesystem. (There is a documented AIX rmfs command, but be very careful with it. Like the UNIX rm command, rmfs does not prompt for confirmation. It immediately destroys the specified filesystem!)

Best Practice:	After installing any new optional AIX filesets, always reinstall the current AIX Technology Level and Service Pack.
Why:	It is important to keep all system components at a consistent fix level. When an AIX Technology Level or Service Pack is installed, updates are applied only to filesets installed on the system when Technology Level or Service Pack installation occurs. When new optional AIX filesets are installed, they are installed at the fix level available on the base AIX installation media, which is likely below the AIX Technology Level and Service Pack at which the system is currently running.
How:	Use the smit Update Installed Software to Latest Level (Update All) menu (accessed via the smitty update_all fast path) to reinstall AIX Technology Level media. Caution: Use smitty update_all only against AIX Technology Level media, not against other media received from the IBM Support Center which contains miscellaneous fixes, unless instructed to do so by the Support Center.

Best Practice:	Install OpenSSH.
Why:	To avoid use of unsecure protocols such as FTP and telnet, which send unencrypted passwords (including root's password) over TCP/IP networks. See the OpenSSH web page for details.
How:	See How to install, configure, and use SSH on AIX V5.3.

Best Practice:	Update $MANPATH for all users so that the man command can find man pages installed by applications (such as AIX Toolbox for Linux Applications software).
Why:	Useability.
How:	Add the line MANPATH=/opt/freeware/man to /etc/environment. (You did remember to save /etc/environment as /etc/environment.orig first, right? - See the Before manually editing any file in the / and /usr filesystems for the first time, save a copy of the file best practice.) Note: It may be appropriate to add more directories to MANPATH. For example, if you install the IBM XL C/C++ Compiler Man Pages--U.S. English, you should add :/usr/vacpp/man/en_US to MANPATH. If you download and install Apache from the AIX Toolbox for Linux Applications web site, you should add :/opt/freeware/apache/man to MANPATH. There is no need to add the /usr/share/man directory to MANPATH, since the man command searches /usr/share/man before searching the directories listed in $MANPATH. And directories such as /opt/csm/man are not needed in MANPATH since csm filesets (eg, csm.dsh) add soft links to the /usr/share/man directory tree pointing to the man pages in the /opt/csm/man directory tree. Note: Bull freeware installation instructions suggest that an export MANPATH command must be added to /etc/profile, but this is not necessary because ksh exports MANPATH by default if it is set. To confirm that MANPATH is exported by default after adding MANPATH to /etc/environment, login, issue the command export, and observe that MANPATH is displayed (among many other environment variables).

Best Practice:	Configure TCP/IP to search local /etc/hosts to resolve a host name before trying DNS.
Why:	If your network folks hose up DNS, you want to be able to circumvent the problem locally and minimize impact on your servers while they are fixing the problem.
How:	There are two options for name resolution tuning: Add a line containing hosts=local,bind to /etc/netsvc.conf. (You did remember to save /etc/netsvc.conf as /etc/netsvc.conf.orig first, right? - See the Before manually editing any file in the / and /usr filesystems for the first time, save a copy of the file best practice.) This update takes effect as soon as the change is made. Add a line containing NSORDER=local,bind to /etc/environment. (You did remember to save /etc/environment as /etc/environment.orig first, right? - See the Before manually editing any file in the / and /usr filesystems for the first time, save a copy of the file best practice.) This update takes effect only as each daemon is stopped & restarted and as users logoff & log back in. The easiest way to restart all daemons is to reboot AIX. It is not necessary to add an export NSORDER command to /etc/profile for ksh users, since ksh exports NSORDER by default if it is set.
	Note: Leave the loopback/localhost entry in /etc/hosts and add only an entry for your local hostname unless, of course, it becomes necessary to add entries to circumvent a DNS problem. Note: Before and after making this change, issue the command host <ipaddr> for every <ipaddr> defined in /etc/hosts. Update entries in /etc/hosts so that the host <ipaddr> command generates the same output with and without the new line in /etc/netsvc.conf. That is, make sure your /etc/hosts is consistent with DNS.

Best Practice:	Configure AIX Workload Manager (WLM) to reserve CPU time for the root userid.
Why:	So that if/when many application processes are consuming all available CPU, it will still be possible to log in to the server as root and take action quickly. And so that if/when many application processes are consuming most of the available CPU, DLPAR operations will complete promptly.
How:	See How to configure the AIX V5.3 Workload Manager to reserve CPU time for the root userid. Configuring AIX system hang detection is another option to permit quick intervention when all available CPU is consumed.

# Run topas when top is requested
if [ "$SHELL" = "/usr/bin/ksh" ]
then alias top='/usr/bin/topas'
fi
# Avoid problems when using vi to edit large files (files with many lines)
export EXINIT="set ll=20000000"

Best Practice:	Install lsof (LiSt Open Files) utility.
Why:	Useful in diagnosing a variety of functional and performance problems. For example, when a filesystem can not be unmounted because files in it are open, lsof can be used to identify the files which are open and the processes which are holding them open.
How:	See How to install the lsof (LiSt Open Files) utility on AIX V5.3.

Best Practice:	Consider installing the Adobe Acrobat reader.
Why:	A .PDF reader is an important end-user and sysadmin tool.
How:	Download the Adobe Acrobat Reader V7.0.8 for IBM AIX from the Adobe Reader download site. To install Acrobat, run INSTALL in the AdobeReader directory after unpacking the downloaded archive. See the Don't store local files in / and /usr best practice above, which suggests creating a /usr/local filesystem and installing software such as Adobe Acrobat in that filesystem (in, for example, directory /usr/local/Acrobat7). Add a symbolic link from /usr/local/bin/acroread to the Adobe acroread executable. (If you implemented the Add useful shell scripts to /usr/local/bin and make them available to all users best practice, the /usr/local/bin directory will already exist and all users will have /usr/local/bin in their $PATHs.) If no X-desktop is available (that is, no UNIX or Linux workstations with graphics displays are available and OpenText (was Hummingbird) Exceed, Cygwin/X, or equivalent is not available), then consider installing VNC to provide the X-desktop which Acrobat requires.