Brian Smith's AIX / UNIX / Linux / Open Source blog
|Modified on by brian_s|
Below is a script to migrate AIX users to Linux. You run the script on your AIX server, and the script collects the information about the users and groups on the system and prints out the commands you would need to run in Linux to recreate the groups and users. The script doesn't make any changes on the AIX server.
The generated commands are tailored for RHEL 6. If your using a different distro of Linux you might need to slightly tweak the script to generate commands the other distro will like.
The script takes the following in to consideration when generating the commands:
At the top of the script is a place where you can list accounts that you do not want migrated (skip_groups, and skip_users). This is helpful so you can list OS accounts/groups or other accounts/groups that should not be migrated.
Here is example output from the script:
#####Commands to re-create groups on Linux server
Here is the script:
|Modified on by brian_s|
The generic read/write/execute UNIX/Linux file permissions are fairly easy to understand. But when you start getting in to the SUID, SGID, and Sticky bits things get more complicated and harder to understand.
For example, if you came across a file with "--S--S--T" permissions would you know what this means? By reading this short posting and referring to the table linked below, you would be able to quickly and easily decipher any possible file permission you might run in to and translate it in to english :)
Here is a little information on how SUID, SGID, and Sticky bits show up when looking at file permissions:
SUID permissions show up as a "s" in the 3rd column of the user permission set (i.e. rwsr-xr-x)
SGID permissions show up as a "s" in the 3rd column of the group permission set (i.e. rwxr-sr-x)
The Sticky bit shows up as a "t" in the 3rd column of the other permission set (i.e. rwxrwxrwt)
If you see a capital "T" in the 3rd column of the other permission set (i.e. rwxrwxrwT), this means that the sticky bit is set, but that there is no execute ("x") permission for others.
If you see a capital "S" in the 3rd column of the user or group permission sets (i.e. rwSr-xr-x or rwxr-Sr-x), it means that the SUID/SGID permission is set, but that the execute bit isn't set (which would kind of defeat the purpose... but it is still possible to set the permissions to this).
There are a total of 4,096 possible ways to set the permissions on a file. I created a table that shows each possible permission with its numeric mode, what the permissions would look like, and then a plain english explanation.
developerWorks wouldn't let me post the permissions table here in the blog because it would put my posting over the size limit, so here is the link: http://ixbrian.com/unixpermissions.html
What do I mean by scripts that don't actually do anything? I'm suggesting that in some cases it makes sense to write scripts that don't make any changes to the system. Instead, the output of the script is the list of commands that need to be run for the change to be made. After you have reviewed the commands the script generated you can then actually run the commands to make the changes. I picked this technique up from a coworker (Thanks Merrill!)
To actually make the changes on your system you can either copy/paste the command lines the script generated, or if there is a lot of output, you can redirect the output of your script to a file. You can then review the file and if it looks correct run the commands in the file by using a command such as ksh ./outputfile or bash ./outputfile
I like to use this technique for any scripts I am writing that are making intrusive changes on critical systems.
Here are some of the reasons why:
The best of both worlds.
It is very easy to take the script and actually have it make the changes on the system without your intervention. Maybe you feel confident in the script after running it many times or perhaps you are on a test/dev system which isn't that important. All you have to do to actually have the script make the changes is pipe its output to the shell interpretor.
The script will run and output the commands that need to be run and the shell will read this from the pipeline and actually run the commands.
brian_s 270002K5X3 Tags:  security linux aix unix scripting umask script 2 Comments 34,118 ViewsModified on by brian_s
When ever a filesystem is mounted on a UNIX system a mount point must be specified. A mount point is simply a directory that can either be empty or contain files. If the mount point directory contains files these are "covered up" and are no longer accessible when a filesystem is mounted over the mount point. Once the filessytem is unmounted any files that where in the mount point directory become visible again.
When the filesystem is mounted the owner, group, and permissions of the root of the filesystem take effect and override whatever ownership and permissions where set on the mount point directory.
# ls -ald /app1 ##Mount Point Directory (These are the permissions/owner/group I recommend for most mount points)
The first "ls" shows the mount point directory. The filesystem is then mounted, and as you can see both the permissions, owner, and group where changed to the filesystems root directory permissions/owner/group.
I recommend setting the mount point directory owner/group to root:system and having permissions set to something like "r-xr-xr-x" rather than having it set to the same owner/group and permissions that the mounted filesystem has. This way if the filesystem doesn't mount for some reason (for example due to a filesystem issue), users will be prevented from writing files in to the mount point directory. If the mount point ownership/permissions match the filesystem, then if the filesystem doesn't mount you will probably end up with a full root filesystem and then somehow need to merge files between what was written to the mount point and what was already in the filesystem (not fun!)
This is what you don't want:
# ls -ald /app1 ##Mount point permissions, owner, and group identical to what they are when filesystem is mounted
Be careful if you set the mount point permissions too restrictive. In general the mount point should have at least "r-xr-xr-x" permissions. If you set them to something like "rwx------" AIX will show weird permission errors for the parent directory when doing file listings as other users:
# ls -ald /app1 ##Mount point permissions/owner prevent any users from even listing contents of directory
The "ed" editor has for the most part been forgotten as a piece of UNIX history. But it can still be very handy when scripting file edits, especially when the script must work across multiple UNIX variants.
ed is a "line editor". What this basically means is it is designed to work on a teletype (aka keyboard with a printer). This makes it challenging to use as an interactive editor, but it is perfect for scripting. ed is part of the Single UNIX Specification so it should exist on any system claiming to be UNIX, so it is a good utility to rely on in your scripts if they are cross platform.
Many systems such as Linux support the "sed" command with the "-i" flag that specifies the file should be edited in place. Unfortunately systems such as AIX don't support an inline edit mode with sed so you either have to create temporary files (ugly in my opinion) or use something different such as ed.
All ed commands specify a range of lines (or a "," for the entire file), and then a command. For example, within ed you can display lines 1 through 5 with the command "1,5l", or you can display the entire file with ",l".
Here are some example command lines to edit files that can be used from scripts.
First off, we will edit the "sshd_config" file and replace the line "#Port 22" with "Port 222". To do this we use printf to pipe the commands we want to run to ed. In between each ed command we put a "\n" to put a carriage return / new line.
$ cat sshd_config | grep "Port "
So in this example we are telling ed to first run ",s/^#Port 22$/Port222". The first comma means apply to all lines, the "s" means search, the "^#Port 22$" means to look for a line that begins (^) and ends ($) with "Port 22". The "/" means replace, and the "Port 222" is what we want it to be replaced with. The "w" means write the file. The "q" means quit.
Here is an example of adding a line to a file. For example, lets say we want to add the comment line "Updated port to 222 on 8/19/12" to the top of the sshd_config file:
The "1" means to go to line 1, the "i" means insert mode, the "#Updated port to 222 on 8/19/12" is the text we are adding, the "." on the line by itself means to go back to command mode, the "w" is write, and the "q" is quit.
You can also easily delete lines. Lets suppose for example we decide to delete the "Port 222" line from the file:
$ cat sshd_config | grep ^Port
As you can see ed is a powerful utility to edit files from scripts quickly and easily.
Often times System Administrators need to copy a file to remote servers with root authority. One example might be to push out an updated resolv.conf file to all servers. However, often times remote root logins are disabled so it can be difficult to copy files with root authority if root logins are disabled.
Here is a quick and easy method to copy files using root authority even if remote root logins are disabled. For this method to work, you will need the following setp:
Once this is done, you can use a combination of "cat", "sudo", and "ssh" to easily push out a file to mulitple servers with root authority..
Here is an example of pushing out a new resolv.conf file to all servers listed in the "serverlist" file:
It works by catting the file to be transferred and pipping it to the SSH connection. Within the SSH connection we sudo to root and then use cat to write the standard input to the file.
This will also work with binary files - not just text files..
Being careful not to shoot yourself in the foot when cleaning up users and home directories on AIX and Linux
It is possible on Linux or AIX to have users that share a home directory. For example, you might have user1, user2, and user3 all have their home directories set to /sharedhome.
Anytime you are deleting users and home directories you need to keep this in mind. You might only need to delete "user1" but want to leave "user2" and "user3" unaffected. But if you delete user1 and its home directory you might end up deleting the shared home directory which would have a big impact on "user2" and "user3".
Let's look at this situation:
AIX and Red Hat both have a "userdel" command that will optionally erase the users home directory as well ("-r" flag).
On most versions of AIX, if you did a "userdel -r" on any one of the 3 users, it would deleted the /sharedhome directory which would impact the remaining 2 users that you didn't want to affect.
On Red Hat Linux, "userdel" tries to be smarter, and verifies that the owner of the home directory matches the user that is being deleted. Thus, if you did a "userdel -r user1" it would happily wipe out /sharedhome, but if you did a "userdel -r user2" or "userdel -r user3" it wouldn't delete /sharedhome because the owner of the directory doesn't match the user being deleted.
Here is a one-liner that will do a better job checking to see if a given directory is a shared home directory between multiple users:
Change "/sharedhome" to whatever directory you would like to check. If this comes back and says it is a shared home directory, then you need to do more research before attempting to delete the home directory.
So anytime you are cleaning up users and home directories, always keep in mind that users might have shared home directories and that under some circumstances AIX and Linux will wipe out these shared home directories which might affect other users on the system.
Check out my latest article over at Power IT Pro: "Understanding SSH Authentication Key Basics", http://poweritpro.com/security/understanding-ssh-authentication-key-basics
Many people are not aware that you can specify multiple file names on the command line for most commands that manipulate files.
For example, if there were serveral files I wanted to change permissions on I can list all the file names on a single "chmod" command line:
$ chmod 400 /tmp/file1 /home/file5 /etc/testfile
This is more efficient than specifying 3 seperate commands:
$ chmod 400 /tmp/file1
This same technique works for almost any command that deals with files such as: ls, chown, chgrp, chown, grep, cat, which, tail, etc.
Understanding the concept by understanding wildcards
Most UNIX/Linux admins know that you can do a command such as chmod 400 * with the * wildcard. When you do this, what the shell is actually doing is changing the * wildcard in to a list of the file names in the directory before it ever even runs the chmod command. When the shell runs chmod the shell provides it the list of files in the directory as an argument to chmod. So when you do "chmod 400 *" the chmod command doesn't know that you specified a * wildcard. All it is aware of is the list of file names that the shell has provided it as arguments. This can be illustrated by running this command:
$ echo ls -al *
You can see when I ran the command echo ls -al * the shell translated the * in to the list of file names. So when you run ls -al *, the command that is actually being run in the end is ls -al file1 file2 file3 file4 file5 file6
Moving and copying files
If you need to move or copy multiple files in to a directory you can easily do it in a single command. For example, if you wanted to back up several files in to the /tmp directory you can run a command such as cp /etc/passwd /etc/group /etc/resolv.conf /tmp This will copy these 3 files in to the /tmp directory.
When you stop and think about it this makes perfect sense. If you were to run a command such as mv * /tmp as we have already covered what the shell is doing is changing the * wildcard in to the list of file names in the dircetory before ever calling mv. Here is another example showing this by using the echo command to show what is really being run:
$ echo mv * /tmp
Killing processes and editing files with vi
The kill command supports specify multiple pids on a single command line, i.e: kill 3419 456 532
Even the vi command supports specify multiple files to edit on a single command line: vi file1 file2 file3 Once in vi you can run ":n" to move to the next file
But Don't go Overboard
One thing to keep in mind is that there is a limit to how long your command line can be. If you start running in to this limit you need to start looking at utilities such as xargs which can help.
If you are not familiar with "expect", it is a script/programming language that is designed to automate interactive processes. For example, suppose you need to install a piece of software on many UNIX/Linux servers. The installation program runs from the command line, but must be run interactively. When run, it interactively asks the user for several pieces of information, and then installs. With expect, you could write a script to automate this task so that when the installation program prompts for information expect supplies the information, essentially simulating a user and automating the task.
Expect allows you to turn a normally interactive-only process in to a completely non-interactive, automated task.
The author of the expect language, Don Libes, wrote the definitive book on expect. The book is called "Exploring Expect" by O'Reilly. I would highly recommend the book to anyone wanting to learn expect.
Here is a great picture from the "Exploring Expect" book that sums up the power of Expect:
In this posting I will cover when you should use expect and when you should avoid it.
When approaching a problem, my first rule of thumb when it comes to expect is to avoid using expect unless it is the only solution. Why? Expect is an amazing tool, but can be complicated to use and very fragile. The basic premise of expect is to set it up to look for certain strings of text ("expect"ing them), and when it sees certain text, respond in a certain way. For example, you could write a script that expects the text "Specify directory to install application to: " and when it sees this, type back in "/opt/software/" However, if in the next version of the software the text of the installer changes slightly (i.e. "Specify directory location to install application to: ") , your expect program will no longer see what it is looking for and will fail to work. If there is a different way to get something done other than using an expect script then it is usually a better option in my experience. It would be possible to write an expect program to automate opening vi, editing the file, and then saving and exiting vi. But it would be much easier and more reliable to use a tool such as sed, awk, perl, etc. to edit the file. Make sure you are using the right tool for the job.
My second rule of thumb is to not use expect to automate tasks that deal with passwords. When you look around for expect information and examples, a lot of them deal with automating things like SSH, SFTP, SCP, etc. For example, the Wikipedia page on Expect lists 4 example scripts for using expect. They all deal with automating logging in to a service with password authentication (telnet, ftp, sftp, and ssh). I would not recommend using expect to automate anything like this. There is a very good reason why tools like SFTP and SSH don't allow you to script using a password without a tool like expect: It is a very bad idea! The first problem is you generally need to include the clear text password in your expect script. If anyone gets access to the script, they have access to the password as well. The second big problem when using expect with passwords is the risk that expect will "type" in the password at the wrong time. For example, if you have a expect script setup to expect certain things, and then send the password, and something goes wrong, the script might send the password too early or too late. The password might then end up somewhere inappropriate or visible to other users or in a shell history file. The bottom line is, if you need to automate running remote commands or copying files around, use SSH keys. SSH keys are so much safer than passwords for a variety of reasons, and there are several things you can do to make them a good option for automated tasks. If you MUST use expect to automate a password related task, one method to help the situation would be to have the expect script prompt you for the password when beginning the task and have the user type it in each time. This way the password is not stored in the script file.
Another password related example of what NOT to do with expect: automate changing passwords. Expect even includes an example script with it named "passmass" that will change your password on multiple servers. From a security perspective I think this is a really, really bad idea for the reasons I specified in the previous paragraph. The right tool for this kind of job is the "chpasswd" command. The chpasswd utility even allows you to specify a password hash ("encrypted" password) when setting a users password to make it more secure to script. chpasswd isn't perfect, but in my opinion it is a much better option than expect when it comes to automating changing passwords.
So when should you consider using expect? You should think about expect anytime you have a manual task that needs to be repeated and that only provides a interactive interface to the user. We already covered the example of a interactive software installation program. Another example is any propriety software that forces you to go through a text based menu to do something. Using expect, you could write a script to navigate the menu and automate the task.
Another extremely good way to use expect is when you need to automate an appliance or other closed system that doesn't have the ability to be scripted. To do this, you use expect on a Linux/UNIX machine to connect to the appliance or closed system, and then complete a task. For example, you could write an expect script that would connect to a Cisco switch and run a series of commands on the switch.
Expect is also a good option when creating test cases. If you need to routinely test software functionality then expect might make your life easier.
You can also use expect to not fully automate tasks, but just assist with manual tasks. This is because expect allows you to partially automate tasks while still allowing parts to be manual completed by a real person. An example of this is the AIX command "mkdvd" which burns mksysb images on to a DVD. When you run this command it writes the first DVD, and then if needed it will prompt you to insert additional DVD's. With expect, you could write a script that would email you or page you whenever it was time to put in the next DVD or when the mkdvd command was completed. This script needs to be customized with 2 command lines to email you/page you.
This mkdvd expect script helps with a manual process and this is not something you could do without a tool like expect.
Please post comments with some creative examples of when you have used expect, or horror stories of other people using expect when they shouldn't have :)