![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
You're a "systems programmer" -- you write code to keep servers healthy and to supply your application developer colleagues with the low-level functionality they need. Where do you get the information you need? Most programming references focus on client or "application" issues, and administration books often avoid programming in favor of "configuration."
I hope one of the sources you'll find useful is this new "Server clinic" column. Every month, I'll solve a programming problem or class of problems common in the "care and feeding" of servers.
This first installment introduces Expect as the one language you most need to know. You're probably acquainted with Expect already. It's at least equally probable, though, that you've never seen the full range of tasks Expect manages. Expect achieves a kind of universality for Linux systems programming that no other language -- not even C or Java or bash -- matches. While future columns will exhibit solutions in all kinds of languages, Expect is likely to be the one that appears most often.
What makes Expect "universal"? Understand, first, that Expect is a proper superset of the Tcl/Tk programming language. Tcl is a high-level language used in a wide variety of programs. It's often lumped with Perl, Python, Ruby, and others as a "scripting" language. In the year 2002, it's wisest to put aside certain historical accidents and simply see all of these as highly productive open source languages. Tcl in particular has been popular in the computer-assisted design (CAD) segment, and with such networking equipment vendors as Cisco and Nortel. Like the other "scripting" languages, Tcl's built-in facilities apply to most common problems in text processing, database management, networking, arithmetic, and so on.
Tcl is Expect's base. Any Tcl program is automatically an Expect program. It's important to emphasize this for a couple of reasons:
- Many people know Expect only as a "tool" and never learned that it is a full-fledged programming language.
- Many of the programmers who do recognize Expect's general-purpose abilities are stuck in 1994.
Expect's author is Don Libes of the (U.S.) National Institute of Standards and Technology. In 1994, he published a superb book on Expect. Still in its first edition, it has no competitors; the book was written so well that no publisher has put out another. Most remarkably, Exploring Expect (see Resources later in this article) hasn't needed updating. Its lucidity and precision have aged quite well.
The problem in this is that Expect's underlying Tcl base has advanced a great deal in the last eight years. When Expect was originally written, Tcl didn't aspire to be a general-purpose programming language. Since that time, Tcl has:
- Learned how to handle full eight-bit data and even Unicode conveniently
- Added a handy TCP/IP abstraction;
- Acquired abilities in data and time calculations and formatting
- Improved and rationalized its string-handling
So, remember: if Perl or Java or C can solve a problem, then Tcl, and therefore Expect, probably can too.
One thing Tcl does "out of the box" better than any of the others, though, is construction of graphical user interfaces (GUIs). While the standard ActiveTcl binary distribution download from ActiveState Tools Corporation for Linux is only about ten megabytes, it includes not only Expect, but a full-blown integrated GUI toolkit. Examples below will show how succinctly this toolkit, called "Tk," expresses GUI solutions.
Unique solution to hard problems
Expect's Tcl/Tk base applies across an unusually wide range of programming. Keep in mind that Expect does everything that Tcl/Tk does. Along with this, Expect adds three broad categories of additional functionality:
- Expanded debugging options
- Convenience commands for describing character-oriented dialogues
- Unparalleled management of troublesome character-oriented terminals
The first of these is conventional. Expect has a variety of "switches" to log or report aspects of its operation.
The purpose of Expect is to automate character-oriented interactions. You already do a lot of this yourself. Every time you write a command-line pipeline or redirect input-output (I/O) streams, you're having the computer manage work that you would otherwise have to type in.
Expect deepens this control in two ways: first, it provides a language for expressing dialogue complexities. Instead of just having a fixed "script" that you feed as input to an application, Expect makes every keystroke of the interaction programmable.
More crucially, though, as Libes expresses it, "Expect is, after
all, a tool made for dealing with crappy interfaces." In
particular, Expect has complete knowledge of managing
applications that resist I/O redirection. The classic
example of this is the command-line passwd
program. Everyone with server responsibilities sooner or
later needs to automate password updates. A first attempt
might be to run something like this as the root user:
Failed automation of passwd
passwd $user << HERE $newpassword $newpassword HERE |
As everyone who tries it soon discovers,
this simply doesn't work. The shell's < and << redirections
don't work with programs such as passwd.
Expect does, though. Expect knows how to talk to all
character-oriented applications, even ones that, like
passwd, manipulate terminal settings.
And that's what completes Expect's universality. In
principal, other languages or libraries could offer
knowledge of terminal characteristics. Perl's
Expect.pm module, for example, goes a long
way in that direction. After over a decade of production
use, though, no other serious competitor has yet arisen.
That's why you should learn Expect. You're going to work with programs with "crappy interfaces" -- there are a lot of them around -- and Expect can eliminate hours and even days of development time in getting them to do what you need. At the same time, you can also use Expect for all the jobs more often done with bash or Perl.
Everything else you should know about Expect
There are a few other things you should know about Expect. This column closes with an explanation of Expect's limits, a peek at working Expect code that solves common problems, and references that can lead you deeper into Expect programming.
Expect does more than most people realize; that's the main theme of this column. It also does less. Systems programmers often need to automate things like FTP operations, e-mail emission or handling, and GUI testing. With the first two of these, Expect doesn't help. More precisely, although it's possible to use Expect to automate FTP and e-mail, and it was common several years ago, Expect now has no special advantages in these domains. Other languages and approaches work as well or better than an Expect-oriented coding. Future installments of this "Server clinic" column will show examples of easy networking automations.
One famous use of Expect is in testing. Expect is the basis for the DejaGnu system used in quality control of several high-profile products, including gcc. However, although Expect can be used to build GUIs, and Expect is crucial in several test frameworks, Expect does not, in general, play a role in test frameworks for GUI systems.
Return for a moment, to the passwd
problem mentioned above. What is Expect's perspective on it?
To understand the Expect source, it's easiest for the moment to neglect security considerations. The programs below need to be run as the "root" user. Expect offers useful facilities for more secure operation; it's easier to understand those, though, after mastering Expect's basics.
You already know that simple I/O redirection doesn't work with
passwd. What kind of Expect program
gives better results?
Simple Expect program to update a password
# Invoke as "change_password <user> <newpassword>".
package require expect
# Define a [proc] that can be re-used in many
# applications.
proc update_one_password {user newpassword} {
spawn passwd $user
expect "password: "
exp_send $newpassword\n
# passwd insists on verifying the change,
# so repeat the password.
expect "password: "
exp_send $newpassword\n
}
eval update_one_password $argv
|
This is all it takes to automate a program that languages other than Expect find almost impossibly difficult. With only a few lines more, you can batch updates for hundreds of users at a time. This is a common need; I've often been called in to restore servers with deeply corrupted password files, and this illustrates one way to start:
A simple iteration
...
set default_password lizard5
set list [exec cat list_of_accounts]
foreach account $list {
update_one_password $account $default_password
puts "Password for '$account' has been reset."
}
|
Putting a GUI face on an Expect automation takes only a few lines more. Imagine that you want to give a non-programmer an application that makes it easy to update a password. Leaving aside again the security consideration, it's as easy to do as:
A simple iteration
...
package require Tk
frame .account
frame .password
label .account.label -text Account
entry .account.entry -textvariable account
label .password.label -text Password
# Show only '*', not the real characters of
# the entered password.
entry .password.entry -textvariable password -show *
button .button -text "Update account" -command {
update_one_password $account $password
}
pack .account .password .button -side top
pack .account.label .account.entry -side left
pack .password.label .password.entry -side left
|
This small working application has this visual appearance:
Screen shot of simple Expect password manager
Expect has unique capabilities systems programmers often need. At the same time, Expect is a fine general-purpose programming language with strengths in networking and GUI construction. If you have to choose only one language for your daily work, Expect is nearly ideal.
Let me know how you use Expect and what you'd like it to do for you. Over the next few months, this "Server clinic" column will be back to look at high-end version control, network proxies, and more automations. Until then, good health to you and your servers.
- Participate in the discussion forum.
- Check out the other installments of Server clinic.
- The Tcl-ers' Wiki page
on Expect is a collaboratively maintained resource that
includes fresh links to the Expect home page, various archives,
tutorials, and related information.
- "What is Tcl"
explains the language that Expect extends.
- The Wiki page on Tk
presents the GUI toolkit integrated into Expect.
-
ActiveTcl
is a no-charge branded distribution of Tcl/Tk that includes
Expect, along with several other extensions to the core Tcl.
-
Exploring Expect
not only definitively describes Expect, but also briefly
introduces Tcl and Tk.
-
DejaGnu is a highly
influential and widely used testing framework.
-
Android is the name
of an interesting test framework for GUI applications. There
are slender connections between Android and Expect.
-
Tcl/Tk quick start is a developerWorks tutorial that introduces the Tcl/Tk scripting language, including its history, key features of the language and interpreter, some extensions, and several examples of Tcl/Tk in action.
-
Tcl for WebSphere Application Server administrators shows how to use the WebSphere Control Program's scriptable interface to deploy EJBs, configure server startup, invoke external processes, and much more.
- Find more Linux articles in the developerWorks Linux zone.
Table of contents
Local resources
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
