Level: Introductory Cameron Laird (claird@phaseit.net), Vice president, Phaseit, Inc.
09 Apr 2002 Cameron Laird opens his new monthly column with an overview of the popular Expect tool, a language capable of far more than most programmers and administrators realize. Expect is so apt for the general-purpose work needed to keep servers healthy, in fact, that it can serve as your one (almost) universal programming language.
You're a "systems programmer" -- you write code to
keep servers healthy and to supply your application developer
colleagues with the low-level functionality they need. Where
do you get the information you need? Most programming references
focus on client or "application" issues, and administration books
often avoid programming in favor of "configuration." I hope one of the sources you'll find useful is this new
"Server clinic" column. Every month,
I'll solve a programming problem or class of problems common in the "care and feeding" of servers. This first installment introduces Expect as the one language
you most need to know. You're probably acquainted with Expect
already. It's at least equally probable, though, that you've never
seen the full range of tasks Expect manages. Expect achieves a
kind of universality for Linux systems programming that no other
language -- not even C or Java or bash -- matches. While future
columns will exhibit solutions in all kinds of languages,
Expect is likely to be the one that appears most often. Built
on Tcl
What makes Expect
"universal"? Understand, first, that Expect is a proper
superset of the Tcl/Tk programming language. Tcl is a
high-level language used in a wide variety of programs.
It's often lumped with Perl, Python, Ruby, and others
as a "scripting" language. In the year 2002, it's wisest
to put aside certain historical accidents and simply see all of
these as highly productive open source languages. Tcl in
particular has been popular in the computer-assisted design
(CAD) segment, and with such networking equipment vendors as
Cisco and Nortel. Like the other "scripting" languages, Tcl's
built-in facilities apply to most common problems in text
processing, database management, networking, arithmetic, and
so on. Tcl is Expect's base. Any Tcl program is automatically an
Expect program. It's important to emphasize this for a couple
of reasons:
- Many people know Expect only as a "tool" and
never learned that it is a full-fledged programming
language.
- Many of the programmers who do recognize Expect's
general-purpose abilities are stuck in 1994.
Expect's author is Don Libes of the (U.S.) National Institute
of Standards and Technology. In 1994, he published a superb
book on Expect. Still in its first edition, it has no
competitors; the book was written so well that no publisher
has put out another. Most remarkably, Exploring
Expect (see Resources later in this article) hasn't needed updating. Its lucidity and
precision have aged quite well. The problem in this is that Expect's underlying Tcl base
has advanced a great deal in the last eight years. When
Expect was originally written, Tcl didn't aspire to be a
general-purpose programming language. Since that time,
Tcl has:
- Learned how to handle full eight-bit data and
even Unicode conveniently
- Added a handy TCP/IP abstraction;
- Acquired abilities in data and time calculations
and formatting
- Improved and rationalized its string-handling
So, remember: if Perl or Java or C can solve a problem,
then Tcl, and therefore Expect, probably can too. One thing Tcl does "out of the box" better than any of
the others, though, is construction of graphical user
interfaces (GUIs). While the standard ActiveTcl
binary distribution download from ActiveState Tools
Corporation for Linux is only about ten megabytes, it
includes not only Expect, but a full-blown integrated
GUI toolkit. Examples below will show how succinctly
this toolkit, called "Tk," expresses GUI solutions.
Unique solution to hard problems
Expect's Tcl/Tk base applies across an unusually wide range
of programming. Keep in mind that Expect does everything
that Tcl/Tk does. Along with this, Expect adds three
broad categories of additional functionality:
- Expanded debugging options
- Convenience commands for describing
character-oriented dialogues
- Unparalleled management of troublesome
character-oriented terminals
The first of these is conventional. Expect has a variety of
"switches" to log or report aspects of its operation. The purpose of Expect is to automate character-oriented
interactions. You already do a lot of this yourself.
Every time you write a command-line pipeline or
redirect input-output (I/O) streams, you're having the computer
manage work that you would otherwise have to type in. Expect deepens this control in two ways: first, it provides a
language for expressing dialogue complexities. Instead of
just having a fixed "script" that you feed as input to an
application, Expect makes every keystroke of the interaction
programmable. More crucially, though, as Libes expresses it, "Expect is, after
all, a tool made for dealing with crappy interfaces." In
particular, Expect has complete knowledge of managing
applications that resist I/O redirection. The classic
example of this is the command-line passwd
program. Everyone with server responsibilities sooner or
later needs to automate password updates. A first attempt
might be to run something like this as the root user:
Failed automation
of passwd
passwd $user << HERE
$newpassword
$newpassword
HERE
|
As everyone who tries it soon discovers,
this simply doesn't work. The shell's < and << redirections
don't work with programs such as passwd. Expect does, though. Expect knows how to talk to all
character-oriented applications, even ones that, like
passwd, manipulate terminal settings. And that's what completes Expect's universality. In
principal, other languages or libraries could offer
knowledge of terminal characteristics. Perl's
Expect.pm module, for example, goes a long
way in that direction. After over a decade of production
use, though, no other serious competitor has yet arisen. That's why you should learn Expect. You're going to work
with programs with "crappy interfaces" -- there are a lot
of them around -- and Expect can eliminate hours and even
days of development time in getting them to do what you
need. At the same time, you can also use Expect for all
the jobs more often done with bash or Perl.
Everything else you should know about Expect
There are a few other things you should know about Expect.
This column closes with an explanation of Expect's limits,
a peek at working Expect code that solves common problems,
and references that can lead you deeper into Expect programming. Expect does more than most people realize; that's the main
theme of this column. It also does less. Systems
programmers often need to automate things like FTP operations, e-mail
emission or handling, and GUI testing. With the first two
of these, Expect doesn't help. More precisely, although
it's possible to use Expect to automate FTP and e-mail,
and it was common several years ago, Expect now has no
special advantages in these domains. Other languages and
approaches work as well or better than an Expect-oriented
coding. Future installments of this "Server clinic" column
will show examples of easy networking automations. One famous use of Expect is in testing. Expect is the
basis for the DejaGnu system used in quality control of
several high-profile products, including gcc. However,
although Expect can be used to build GUIs, and Expect is
crucial in several test frameworks, Expect does not,
in general, play a role in test frameworks for GUI systems. Return for a moment, to the passwd
problem mentioned above. What is Expect's perspective on it? To understand the Expect source, it's easiest for the moment to
neglect security considerations. The programs below need to be
run as the "root" user. Expect offers useful facilities for
more secure operation; it's easier to understand those, though,
after mastering Expect's basics. You already know that simple I/O redirection doesn't work with
passwd. What kind of Expect program
gives better results?
Simple Expect
program to update a password
# Invoke as "change_password <user> <newpassword>".
package require expect
# Define a [proc] that can be re-used in many
# applications.
proc update_one_password {user newpassword} {
spawn passwd $user
expect "password: "
exp_send $newpassword\n
# passwd insists on verifying the change,
# so repeat the password.
expect "password: "
exp_send $newpassword\n
}
eval update_one_password $argv
|
This is all it takes to automate a program that languages
other than Expect find almost impossibly difficult. With
only a few lines more, you can batch updates for
hundreds of users at a time. This is a common need; I've
often been called in to restore servers with deeply
corrupted password files, and this illustrates one way to
start:
A simple
iteration
...
set default_password lizard5
set list [exec cat list_of_accounts]
foreach account $list {
update_one_password $account $default_password
puts "Password for '$account' has been reset."
}
|
GUIs, too
Putting a GUI face on an Expect automation takes only a few
lines more. Imagine that you want to give a non-programmer
an application that makes it easy to update a password.
Leaving aside again the security consideration, it's as easy
to do as:
A simple
iteration
...
package require Tk
frame .account
frame .password
label .account.label -text Account
entry .account.entry -textvariable account
label .password.label -text Password
# Show only '*', not the real characters of
# the entered password.
entry .password.entry -textvariable password -show *
button .button -text "Update account" -command {
update_one_password $account $password
}
pack .account .password .button -side top
pack .account.label .account.entry -side left
pack .password.label .password.entry -side left
|
This small working application has this visual appearance:
Screen
shot of simple Expect password manager
Conclusion
Expect has unique capabilities systems programmers often need.
At the same time, Expect is a fine general-purpose programming
language with strengths in networking and GUI construction.
If you have to choose only one language for your daily work,
Expect is nearly ideal. Let me know how you use Expect and what you'd like it
to do for you. Over the next few months, this "Server clinic" column
will be back to look at high-end version control, network
proxies, and more automations. Until then, good health to
you and your servers.
Resources - Participate in the discussion forum.
- Check out the other installments of Server clinic.
- The Tcl-ers' Wiki page
on Expect is a collaboratively maintained resource that
includes fresh links to the Expect home page, various archives,
tutorials, and related information.
- "What is Tcl"
explains the language that Expect extends.
- The Wiki page on Tk
presents the GUI toolkit integrated into Expect.
-
ActiveTcl
is a no-charge branded distribution of Tcl/Tk that includes
Expect, along with several other extensions to the core Tcl.
-
Exploring Expect
not only definitively describes Expect, but also briefly
introduces Tcl and Tk.
-
DejaGnu is a highly
influential and widely used testing framework.
-
Android is the name
of an interesting test framework for GUI applications. There
are slender connections between Android and Expect.
-
Tcl/Tk quick start is a developerWorks tutorial that introduces the Tcl/Tk scripting language, including its history, key features of the language and interpreter, some extensions, and several examples of Tcl/Tk in action.
-
Tcl for WebSphere Application Server administrators shows how to use the WebSphere Control Program's scriptable interface to deploy EJBs, configure server startup, invoke external processes, and much more.
- Find more Linux articles in the developerWorks Linux zone.
About the author  | | | Cameron Laird is a full-time consultant for Phaseit, Inc. He writes and speaks frequently on open source and other technical topics. Cameron thanks Reinhard Max of SuSE for his help in the preparation of this column. You can contact Cameron at claird@phaseit.net. |
Rate this page
| 
