Learn 10 more good UNIX usage habits

Become a UNIX command-line wizard

John Fronckowiak
Published on May 27, 2008

Let's face it: Bad habits are hard to break. But habits that you've just become comfortable with can be even more difficult to overcome. Sometimes, a fresh look at things may provide you with an "A-ha, I didn't know you could do that!" moment. Building on Michael Stutz's excellent article, "Learn 10 good UNIX usage habits," this article suggests 10 more UNIX command-line commands, tools, and techniques that may make you more productive as a UNIX command-line wizard.

The 10 additional good habits you should adopt are:

• Use file name completion.
• Use history expansion.
• Reuse previous arguments.
• Manage directory navigation with pushd and popd.
• Find large files.
• Create temporary files without an editor.
• Use the curl command-line utility.
• Make the most of regular expressions.
• Determine the current user.
• Process data with awk.

Use file name completion

Wouldn't it be great if you didn't have to type a long, convoluted file name at the command prompt? Well, you don't, as it turns out. You can configure the most popular UNIX shells for file name completion, instead. This functionality works a bit differently in each shell, so I show you how to use file name completion in the most popular shells. File name completion allows you to type faster and avoid errors. Lazy? Perhaps. More efficient? Definitely!

Which shell am I running?

What happens if you don’t know which shell you're currently running? Although this trick isn't officially part of the 10 more good habits, it's still pretty useful. As shown in Listing 1, you can use the echo $0 or ps –p $$ command to display the shell you're using. In my case, I'm running the Bash shell.

Listing 1. Determine your shell

$ echo $0
-bash
$ ps –p $$
PID TTY           TIME CMD
6344 ttys000    0:00.02 –bash

C shell

The C shell supports the most straightforward file name completion. Setting the filec variable enables the functionality. (You can use the command set filec). After you start typing the name of a file, you can click Escape, and the shell fills in the name of the file—or as much as it can. For example, say you have files named file1, file2, and file3. If you type f, then click Escape, file will be filled out, and you'll have to type the 1, 2, or 3 to complete the appropriate file name.

Bash

The Bash shell also provides file name completion but uses the Tab key instead of the Escape key. You don't need to set anything to enable file name completion in the Bash shell; it's set by default. Bash also implements an additional feature. After typing a portion of a file name, then clicking Tab, if you reach that point at which multiple files satisfy your request and you need to add text to select one of the files, you can click Tab twice more for a list of the files that match what you have typed so far. Using the earlier examples of files named file1, file2, and file3, start by typing f. When you click Tab once, Bash completes file; clicking Tab one more time expands the list of file1 file2 file3.

Korn shell

For Korn shell users, file name completion depends on the value of the EDITOR variable. If EDITOR is set to vi, you type part of name, and then click Escape followed by a backslash (\) character. If EDITOR is set to emacs, you type part of the name, and then click the Escape key twice to complete the file name.

Use history expansion

What happens if you're using the same file name for a series of commands? Well, there's a shortcut that can quickly retrieve the last file name you used. As shown in Listing 2, the !$ command returns the file name that the previous command used. The file this-is-a-long-lunch-menu-file.txt is searched for occurrences of the word pickles. After searching, the vi command is used to edit the this-is-a-long-lunch-menu-file.txt file without the need for retyping the file name. You use the bang, or exclamation point (!), to access the history, and the dollar sign ($) returns the last field of the previous command. It's a great tool if you are using long file names repeatedly.

Listing 2. Using !$ to retrieve the last file name used with a command

$ grep pickles this-is-a-long-lunch-menu-file.txt
pastrami on rye with pickles and onions
$ vi !$

Reuse previous arguments

The !$ command returns the last argument used with a command. But what happens if you have a command that used arguments and you want to reuse just one of them? The !:1 operator returns the argument used in a command. The example in Listing 3 shows how you can use this operator in combination with the !$ operator. In the first command, a file is renamed to a more meaningful name, but to preserve use of the original file name, a symbolic link is created. The file kxp12.c is renamed in a more readable manner, then the link command is used to create a symbolic link back to the original file name, in case it's still used elsewhere. The !$ operator returns the file_system_access.c argument, and the !:1 operator returns the kxp12.c argument, which is the first argument of the previous command.

Listing 3. Using !$ and !:1 in combination

$ mv kxp12.c file_system_access.c
$ ln –s !$ !:1

Manage directory navigation with pushd and popd

UNIX supports a wide variety of directory-navigation tools. Two of my favorite productivity tools are pushd and popd. You're certainly aware that the cd command changes your current directory. What happens if you have several directories to navigate, but you want to be able to quickly return to a location? The pushd and popd commands create a virtual directory stack, with the pushd command changing your current directory and storing it on the stack, and the popd command removing the directory from the top of the stack and returning you to that location. You can use the dirs command to display the current directory stack without pushing or popping a new directory. Listing 4 shows how you can use the pushd and popd commands to quickly navigate the directory tree.

Listing 4. Using pushd and popd to navigate the directory tree

$ pushd .
~ ~
$ pushd /etc
/etc ~ ~
$ pushd /var
/var /etc ~ ~
$ pushd /usr/local/bin
/usr/local/bin /var /etc ~ ~
$ dirs
/usr/local/bin /var /etc ~ ~
$ popd
/var /etc ~ ~
$ popd
/etc ~ ~
$ popd
~ ~
$ popd

The pushd and popd commands also support parameters to manipulate the directory stack. Using the +n or -n parameter, where n is a number, you can rotate the stack left or right, as shown in Listing 5.

Listing 5. Rotating the directory stack

$ dirs
/usr/local/bin /var /etc ~ ~
$ pushd +1
/var /etc ~ ~ /usr/local/bin
$ pushd -1
~ /usr/local/bin /var /etc ~

Find large files

Need to find out where all your free disk space went? Here are a couple of tools you can use to manage your storage. As shown in Listing 6, the df command shows you the total number of blocks used on each available volume and the percentage of free space.

Listing 6. Determining volume usage

$ df
Filesystem                            512-blocks      Used  Available Capacity  Mounted on
/dev/disk0s2                           311909984 267275264   44122720    86%    /
devfs                                        224       224          0   100%    /dev
fdesc                                          2         2          0   100%    /dev
map -hosts                                     0         0          0   100%    /net
map auto_home                                  0         0          0   100%    /home

Want to find the largest files? Use the find command with the -size parameter. Listing 7 shows how to use the find command to find files larger than 10MB. Note that the -size parameter takes a size in kilobytes.

Listing 7. Find all files larger than 10MB

$ find / -size +10000k –xdev –exec ls –lh {}\;

Create temporary files without an editor

This is a simple one: You need to quickly create a simple temporary file but don't want to fire up your editor. Use the cat command with the > file-redirection operator. As shown in Listing 8, using the cat command without a file name simply echoes anything typed to standard input; the > redirection captures that to the specified file. Note that you must provide the end-of-file character when you're finished typing—typically, Ctrl-D.

Listing 8. Quickly create a temporary file

$ cat > my_temp_file.txt
This is my temp file text
^D
$ cat my_temp_file.txt
This is my temp file text

Need to do the same thing but append to an existing file instead of creating a new one? As shown in Listing 9, use the >> operator, instead. The >> file-redirection operator appends to an existing file.

Listing 9. Quickly append to a file

$ cat >> my_temp_file.txt
More text
^D
$ cat my_temp_file.txt
This is my temp file text
More text

Use the curl command-line utility

I can access the Web from the command line? Are you crazy? No, it's just curl! The curl command lets you retrieve data from a server using the HTTP, HTTPS, FTP, FTPS, Gopher, DICT, TELNET, LDAP, or FILE protocols. As shown in Listing 10, I can use the curl command to access the current local conditions of the National Weather Service for my location (Buffalo, NY). When combined with the grep command, I can retrieve the conditions in Buffalo. Use the -s command-line option to suppress curl processing output.

Listing 10. Retrieve the current weather conditions with curl

$ curl –s http://www.srh.noaa.gov/data/ALY/RWRALY | grep BUFFALO
BUFFALO        MOSUNNY   43  22  43 NE13      30.10R

As shown in Listing 11, you can also use the curl command to download HTTP-hosted files. Use the -o parameter to specify where the output is saved.

Listing 11. Use curl to download HTTP-hosted files

$ curl -o archive.tar http://www.somesite.com/archive.tar

This is really just a hint of what you can do with curl. You can start exploring a bit more simply by typing man curl at your command prompt to display the complete usage information for the curl command.

Make the most of regular expressions

Many UNIX commands use regular expressions as arguments. Technically speaking, a regular expression is a string (that is, a sequence of characters composed of letters, numbers, and symbols) that represents a pattern defining zero or more strings. A regular expression uses meta-characters (for example, the asterisk [*] and question mark [?] symbols) to match parts of or whole other strings. A regular expression doesn't have to contain wildcards, but wildcards can make regular expressions useful for searching for patterns and manipulating files. Table 1 shows some basic regular expression sequences.

Table 1. Regular expression sequences

Listing 12 shows some of the basic regular expressions used with the grep command.

Listing 12. Using regular expressions with grep

$ # Lists your mail
$ grep '^From: ' /usr/mail/$USER   
$ # Any line with at least one letter  
$ grep '[a-zA-Z]'  search-file.txt
$ # Anything not a letter or number
$ grep '[^a-zA-Z0-9] search-file.txt
$ # Find phone numbers in the form 999-9999 
$ grep '[0-9]\{3\}-[0-9]\{4\}' search-file.txt
$ # Find lines with exactly one character
$ grep '^.$' search-file.txt
$ #  Find any line that starts with a period "."          
$ grep '^\.' search-file.txt 
$ # Find lines that  start with a "." and 2 lowercase letters
$ grep '^\.[a-z][a-z]' search-file.txt

Many books have been written just about regular expressions. For a more in-depth look at command-line regular expressions, I suggest the developerWorks article, "Speaking UNIX, Part 9: Regular expressions."

Determine the current user

At times, you may have an administrative script that you want to make sure a certain user has or has not executed. To find out, you can use the whoami command to return the name of the current user. Listing 13 shows the whoami command run on its own; Listing 14 shows an excerpt from a Bash script using whoami to make sure the current user isn't root.

Listing 13. Using whoami from the command line

$ whoami
John

Listing 14. Using whoami in a script

if [ $(whoami) = "root" ]
then
   echo "You cannot run this script as root."
   exit 1
fi

Process data with awk

The awk command always seems to live in the shadows of Perl, but it can be a quick, useful tool for simple command-line-based data manipulation. Listing 15 shows how to get started with the awk command. To get the length of each line in the file text, use the length() function. To see if the string ing is present in the file text, use the index() function, which returns the location of the first occurrence of ing so that you can use it for further string processing. To tokenize (that is, split a line into word-length pieces) a string, use the split() function.

Listing 15. Basic awk processing

$ cat text
testing the awk command
$ awk '{ i = length($0); print i }' text
23
$ awk '{ i = index($0,”ing”); print i}' text
5
$ awk 'BEGIN { i = 1 } { n = split($0,a," "); while (i <= n) {print a[i]; i++;} }' text
testing 
the
awk
command

Printing specified fields of text file is a simple awk task. In Listing 16, the sales file consists of each salesperson's name followed by a monthly sales figure. You can use the awk command to quickly total the sales for each month. By default, awk treats each comma-separated value as a different field. You use the $n operators to access each individual field.

Listing 16. Using awk for data summarization

$cat sales
Gene,12,23,7
Dawn,10,25,15
Renee,15,13,18
David,8,21,17
$ awk -F, '{print $1,$2+$3+$4}' sales
Gene 42
Dawn 50
Renee 46
David 46

The awk command can be complex and used in a wide variety of situations. To explore the awk command more fully, start with the command man awk in addition to the resources mentioned in the Related topics.

Conclusion

Becoming a command-line wizard takes a bit of practice. It's easy to keep doing things the same way simply because you're used to it. Expanding your command-line resources can provide a big increase in your productivity and propel you toward becoming a UNIX command line wizard!

Downloadable resources

• PDF of this content

Related topics

• UNIX tips: Read other articles in the "UNIX tips" series.
• "Common threads: Awk by example, Part 1," "Common threads: Awk by example, Part 2," "Common threads: Awk by example, Part 3" (Daniel Robbins, developerWorks, 2001-2002): Check out a good introduction to using awk.
• "Speaking UNIX, Part 9: Regular expressions" (Martin Streicher, developerWorks, 17 Apr 2007): Learn to use regular expressions to discern the relevant from the irrelevant.
• New to AIX and UNIX: New to IBM® AIX® and UNIX? Visit the this page to learn more.