If you manage systems and networks, you need Expect.

More precisely, why would you want to be without Expect? It saves hours common tasks otherwise demand. Even if you already depend on Expect, though, you might not be aware of the capabilities described below.

Expect automates command-line interactions

You don't have to understand all of Expect to begin profiting from the tool; let's start with a concrete example of how Expect can simplify your work on AIX® or other operating systems:

Suppose you have logins on several UNIX® or UNIX-like hosts and you need to change the passwords of these accounts, but the accounts are not synchronized by Network Information Service (NIS), Lightweight Directory Access Protocol (LDAP), or some other mechanism that recognizes you're the same person logging in on each machine. Logging in to a specific host and running the appropriate passwd command doesn't take long—probably only a minute, in most cases. And you must log in "by hand," right, because there's no way to script your password?

Wrong. In fact, the standard Expect distribution (full distribution) includes a command-line tool (and a manual page describing its use!) that precisely takes over this chore. passmass (see Resources) is a short script written in Expect that makes it as easy to change passwords on twenty machines as on one. Rather than retyping the same password over and over, you can launch passmass once and let your desktop computer take care of updating each individual host. You save yourself enough time to get a bit of fresh air, and multiple opportunities for the frustration of mistyping something you've already entered.

The limits of Expect

This passmass application is an excellent model—it illustrates many of Expect's general properties:

• It's a great return on investment: The utility is already written, freely downloadable, easy to install and use, and saves time and effort.
• Its contribution is "superficial," in some sense. If everything were "by the book"—if you had NIS or some other domain authentication or single sign-on system in place—or even if login could be scripted, there'd be no need for passmass. The world isn't polished that way, though, and Expect is very handy for grabbing on to all sorts of sharp edges that remain. Maybe Expect will help you create enough free time to rationalize your configuration so that you no longer need Expect. In the meantime, take advantage of it.
• As distributed, passmass only logs in by way of telnet, rlogin, or slogin. I hope all current developerWorks readers have abandoned these protocols for ssh, which passmasss does not fully support.
• On the other hand, almost everything having to do with Expect is clearly written and freely available. It only takes three simple lines (at most) to enhance passmass to respect ssh and other options.

You probably know enough already to begin to write or modify your own Expect tools. As it turns out, the passmass distribution actually includes code to log in by means of ssh, but omits the command-line parsing to reach that code. Here's one way you might modify the distribution source to put ssh on the same footing as telnet and the other protocols:
Listing 1. Modified passmass fragment that accepts the -ssh argument

            ...
         } "-rlogin" {
            set login "rlogin"
            continue
        } "-slogin" {
            set login "slogin"
            continue
        } "-ssh" {
            set login "ssh"
            continue
        } "-telnet" {
            set login "telnet"
            continue
           ...
      

What is Expect?

What exactly is Expect, and how can you make it work for you?

"Expect" refers to several distinct concepts, which many frequent users of Expect distinguish only muddily:

• Expect is a specific, high-level, and general-purpose programming language, with syntax identical to that of Tcl, augmented by a few special-purpose commands Tcl lacks.
• Expect is an executable that implements that language in the sense that it correctly processes input written in the Expect language.
• The expect command names one of the commands with which Expect extends Tcl.
• Expect is a Tcl package. That means, roughly, that any Tcl application can load Expect capabilities at run time.
• Expect is a library based on C source code, which goes into both the Expect loadable package and the Expect executable.
• Expect is the abstract notion of a tool that:
  • Automates terminal interactions, even when passwords or other special items are involved
  • Implements a "dialogue" model, which encodes a simple rhythm of message and response
  There's nothing specific to Tcl in this abstraction and, in fact, there are now several independent implementations of the Expect model in such languages as Python, Perl, and so on. While the examples of this article are all expressed in Tcl-based Expect, they can all be written in other languages. There's no reason to let familiarity with Tcl, or its lack, limit your use of Expect.

Be aware that I've slightly bent the truth above in regard to a few technical details beyond the focus of the current article; for example, the conventional Expect executable not only extends the Tcl command set, but it also recognizes a few extra command-line arguments on launching. These don't bear on the main theme, though: A little Expect can do more for system managers than most of them realize.

With this background, "passmass is an Expect application" means:

• passmass is a text file written in the Expect language.
• If you execute expect passmass ... on a host where Expect and passmass have been appropriately installed, you'll receive correct passmass functionality.

Note that several variations on execution are possible: It's possible to load Expect interactively within a Tcl interpreter, run passmass, create passmass as a stand-alone executable, and so on. Again, these alternatives are outside the main focus of this article.

A network management example

Let's consider a bigger challenge, one more typical of daily operations in network operations centers: retrieval of current configurations of a collection of managed Cisco switches. While some sites use SNMP or HTTP for these operations, it's far more common to use a console or in-band telnet session to obtain this information. Many administrators believe the only practical way to do this is to type something like Listing 2.
Listing 2. Typical "manual" config introspection

        telnet $MY_ROUTER
        [User: admin]
        [password: ...]
        CCNA01# show running-config
        [... Current configuration:
         ... version 12.0 ...
         FIFTY LINES MORE OF CONFIGURATION DETAIL
         ...
         End]

    

It's not so, though; this can actually be entirely automated, to the point of being invoked, by cron, a mailback server, or a similar job-control mechanism (see Listing 3).
Listing 3. Automation of config introspection

      #!/usr/bin/expect

      # initialize host, password, ...

      package require Expect
      set prompt {[00m# }
      spawn telnet $host
      expect {User: }
      send admin\r
      expect password:
      send $password\r
      expect -exact $prompt
      send "show running-config\r"
      expect -exact $prompt
      send exit\r
  

Common confusions

This example is typical of Expect's use in several ways that deserve mention. First, there is no widely available canned solution that's useful. Quite a few administrators seem to come at Expect with the expectation that it's a single-use tool for exactly the problem they're currently facing. That's not Expect. Expect is not like, say, an electronic stud finder, something that's complete in itself for locating studs. Expect is more like a hand drill: You have to combine it with the right drill bit or driver bit or hole saw or other attachment to have it be useful.

In much the same way, Expect involves at least a bit of customization. Of course, I mostly use Expect on problems no other tool comes close to solving as easily. The case of running-config shows how just a few lines of Expect can yield an impressive harvest of automated information. Even this small example raises several of the issues that recur with Expect. This particular automation, for instance, requires embedding an administrative password in the plaintext Expect script; you need to decide for yourself whether this is appropriate for your environment.

There are at least a few more ways this case is typical. The $prompt was a surprise to me. What shows on the screen is just CCNA01# ; as is typical for many devices, though, this prompt actually embeds invisible control characters. Luckily, Expect has helpful debugging switches to report on its interactions; that's how I determined exactly what characters CCNA01 was emitting.

Also, as written, Listing 3 returns the entire session, including the configuration report I'm after, plus the login and exit framing it. Another common misunderstanding about Expect is implicit in the common request to "retrieve a command's value." Expect simply doesn't have those semantics. When you sit at a keyboard and enter a command, you conceive its action in terms of the display you see before the next prompt. For a networking protocol such as telnet, though, there's really no distinction between results and prompts; those are meanings you impute as an observer.

In complete consistency with this model of network communications, Expect doesn't directly distinguish results and prompts. All Expect knows are dialogues, sequences of characters it sends and others it expects. In practice, therefore, I use a sliver of regular expression parsing to give me just the details I'm after. Expect boasts extensive regular expression capabilities, and the formatting is easy to code. For now, though, let's focus on the broad theme of automation of processes that aren't conventionally scriptable.

Concurrency

Remember, Expect is, among other things, a powerful, general-purpose language. Real Expect applications often parse command-line arguments, tabulate results in columns, retrieve historical data from databases, display graphical user interfaces (GUIs), and much more. These are all conventional computations, though, ones eliminated from the examples of this article. The focus here is to understand the unique values of Expect correctly.

Network administrators equipped with a solution like Listing 3 frequently think next of a "horizontal" enhancement: retrieval of such a report from dozens or thousands of similar machines. Expect has looping constructs—foreach, while, and so on—that make this easy.

It can also quickly turn unsatisfying. Suppose you're responsible for a hundred LAN hosts—not an uncommon situation. You have an Expect script that automatically logs in to each in turn and retrieves an important datum. Now you abstract that script slightly to iterate over the whole collection.

The problem is that the resulting script is likely to take a long time to run. It logs in to the first host, requests the result, receives the result, logs out, turns to the next, requests a new result, and so on and so on. The delays make a person wish there were some way to ask for all the results at once, and collect them as they come in, in whichever order network lags, differential loading, and other delays cause them to arrive.

There is such a way. As it happens, Expect has excellent facilities for managing multiple dialogues simultaneously. Here's an example of a program that logs in several times, issues long-running commands on each of the logins, and then receives the results in the order they arrive; this case has been arranged so that the results return in the order opposite to that in which they were launched (see Listing 4).
Listing 4. Concurrent retrieval of reports from several logins

       #!/home/claird/local/ActiveTcl/bin/tclsh 
       
       package require Expect
       
       log_user 0
       
       # Initialize user, passphrase, ... here.
       
           # Sequentially login and issue time-consuming commands to all
           # hosts.
       for {set i 0; set delay 8} {$delay > 0} {incr i; incr delay -1} {
           spawn ssh $user@$host
           set sid($i) $spawn_id
           expect rsa':
           send $passphrase
           expect "Last login"
           expect -ex $prompt
           set active($sid($i)) $i
           send -i $sid($i) "echo `date`; sleep $delay; echo `date`; echo \
           '$delay done on $i.'\r"
       }
       
       while {[llength [array names active]]} {
           expect -i [array names active] -ex $prompt {
                puts "RECEIVED:  $::expect_out(buffer)"
                send -i $expect_out(spawn_id) exit\r
                expect -i $expect_out(spawn_id) eof
                unset active($expect_out(spawn_id))
           }
       }
  

       RECEIVED:  echo `date`; sleep 1; echo `date`; echo '1 done on 7.'
       Mon Apr 23 22:15:15 UTC 2007
       Mon Apr 23 22:15:16 UTC 2007
       1 done on 7.
       RECEIVED:  echo `date`; sleep 2; echo `date`; echo '2 done on 6.'
       Mon Apr 23 22:15:15 UTC 2007
       Mon Apr 23 22:15:17 UTC 2007
       2 done on 6.
       RECEIVED:  echo `date`; sleep 3; echo `date`; echo '3 done on 5.'
       Mon Apr 23 22:15:15 UTC 2007
       Mon Apr 23 22:15:18 UTC 2007
       3 done on 5.
          ...
       RECEIVED:  echo `date`; sleep 8; echo `date`; echo '8 done on 0.'
       Mon Apr 23 22:15:14 UTC 2007
       Mon Apr 23 22:15:22 UTC 2007
       8 done on 0.
     

At a programming level, notice that all these logins happen to be by means of passphrase-based ssh to the same host, with identical user and password credentials. In a more realistic example, a few more lines of code would manage a range of different hosts, and perhaps different accounts and login protocols for each. There's little point in sleeping for a few seconds and then returning; sleep is a good model, though, for the commands that are useful to network administrators.

Even this example doesn't exhaust Expect's capabilities. Expect also has the ability to manage TCP/IP dialogues directly, with a built-in socket programming interface, and it can semi-automate—switch back and forth between modes in which a user enters part of the dialogue and one where Expect drives everything automatically.

These are subjects for another time, though. The aim of this article is just to demonstrate how much more there is to Expect than most administrators realize, and specifically how study of the tool can immediately solve common problems in network management. The ability to script password and passphrase entry and to control multiple connections concurrently are powerful.

Summary

Expect makes all sorts of automations practical that otherwise seem impossible: scripting of password entry, logging in a remote user's session and returning control to him or her, and much more. As widely as it is used, though, Expect is almost equally misunderstood. Correct understanding of a few Expect basics—how to invoke it, its dialogue model, its programming aids, and so on—shines a light on its enormous potential in system and network administration.

Resources

Learn

• Expect: The Expect home page is conservative. To a large extent, Expect still does the same work it did when first released fifteen years ago, and its Web site caters more to those focused on long-lasting value than glossy impressions.
  
  
• PDSLIB: Among the many open source packages with AIX binaries, PDSLIB has Expect available for download. Also, note that the official AIX FAQ, last updated in 2000, includes a paragraph on Expect.
  
  
• ActiveTcl: Many varieties of UNIX are distributed with Expect, but often these binaries are not current. For most purposes, the easiest way to update Expect is to install ActiveTcl. ActiveTcl includes all Expect run-time capabilities, and it is available for Windows®, Mac OS X, Linux®, Solaris, AIX and HP-UX.
  
  
• Exploring Expect: Only one book has ever been published that focuses on Expect. It was written so well that it has gone without changes for over two decades, and it still sells a few thousand copies annually, according to its publisher. This remarkable achievement is characteristic of the Expect mentality: although Expect is now far more powerful than it was in 1994, when this book was first published, its essential ideas remain unchanged and applicable to computing in the year 2007. Don Libes authored both Expect and Exploring Expect, and continues to maintain the former. The book is dense with valuable detail that goes far beyond the current article. The goal of this article is, on one level, just to make clear how much more power is available to those who have the book in hand.
  
  
• The manual page for passmass, along with all the rest of the example tools and documentation standard for the latest Expect, is available online.
  
  
• "Productivity tips" (Michael Stutz, developerWorks, September 2006): This article recommends use of Expect for automation of command-line interactions and many other useful command-line secrets for increasing productivity. Why do I take ten times as many words as Stutz to reach the same end? Simply because of the number of misunderstandings I've seen result when newcomers first encounter Expect. Expect is rich enough that it's important to understand it fully, and my aim is to help toward that goal.
  
  
• This Wiki page on Expect leads to several interesting commentaries, including details on how to protect Expect scripts, how to retrieve the results of a single command, and much more.
  
  
• "Expect exceeds expectations" (Cameron Laird, developerWorks, April 2002): This introduction to Expect emphasizes its universality, that is, its potential for expression of solutions for all sorts of programming requirements.
  
  
• Android: Many newcomers to Expect wonder in their enthusiasm for the tool whether it automates GUI interactions with the deftness it handles command-line dialogues. It does not. GUI application-level scripting is a hard problem beyond the scope of this article. Among the many projects that have attacked it, "Android" names an interesting test framework for GUI applications. There are slender connections between Android and Expect.
  
  
• Tcl: Expect extends Tcl: It is Tcl, plus a few extra facilities for terminal management, dialogue parsing, and debugging. "What is Tcl?" Tcl is a high-level, general-purpose language that's "lightweight", portable, and easy to learn. This means that Expect is generally available where Tcl is—on all varieties of UNIX and Windows, in particular—that Expect applications can re-use Tcl programming, and that Tcl capabilities are automatically present in Expect.
  
  
• Debugging Expect programs: This page highlights a few key techniques—autoexpect, exp_internal, and interact, mostly—that are specific to Expect.
  
  
• Automating the Management of Network Devices through the Command Line: The present article emphasizes that Expect does jobs that are otherwise impossible, or at least impractical—scripting password entries, and so on. This isn't strictly true; it's possible with Perl, bash, and several other languages to insert delays to automate passwords. Time delays are inherently fragile in a way that Expect need not be. Programming in terms of time delays invariably introduces mysteries that complicate maintenance. This article shows the technique at its best, along with a comparison with Expect style.
  
  
• " Speaking UNIX, Part 9: Regular Expressions" (Martin Streicher, developerWorks, April 2007): Part of getting the most out of Expect is judicious use of regular expressions. This provides a good introduction to regular expressions. Also, I maintain a site on issues related to regular expressions.
  
  
• Check out other articles and tutorials written by Cameron Laird:
  • Across developerWorks and IBM
  
  
• Popular content: See what AIX and UNIX content your peers find interesting.
  
  
• AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX?: Visit the "New to AIX and UNIX" page to learn more about AIX and UNIX.
  
  
• AIX 5L™ Wiki: Discover a collaborative environment for technical information related to AIX.
  
  
• Search the AIX and UNIX library by topic:
  • System administration
  • Application development
  • Performance
  • Porting
  • Security
  • Tips
  • Tools and utilities
  • Java™ technology
  • Linux
  • Open source
  
  
• Safari bookstore: Visit this e-reference library to find specific technical resources.
  
  
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
  
  
• Podcasts: Tune in and catch up with IBM technical experts.
  
  

Get products and technologies

• IBM trial software: Build your next development project with software for download directly from developerWorks.
  
  

Discuss

• Participate in the developerWorks blogs and get involved in the developerWorks community.
  
  
• Participate in the AIX and UNIX forums:
  • AIX 5L—technical forum
  • AIX for Developers Forum
  • Cluster Systems Management
  • IBM Support Assistant
  • Performance Tools—technical
  • Virtualization—technical
  • More AIX and UNIX forums
  
  

About the author

Cameron is a full-time developer for independent consultancy Phaseit, Inc., based on the Texas Gulf Coast, just outside Houston. He frequently writes on Tcl and other "scripting" languages, including Python, Perl, ksh, and others less well known. In addition, he writes on GUI toolkits, data management, and testing methods. His own development projects most often are highly-reliable networked Web and industrial applications. He persists in regarding XML as a variant of SGML. He can be reached at Cameron@Lairds.com.

Rate this article

Comments

Back to top

Table of contents

• Expect automates command-line interactions
• The limits of Expect
• What is Expect?
• A network management example
• Common confusions
• Concurrency
• Summary
• Resources
• About the author
• Comments

Dig deeper into AIX and UNIX on developerWorks

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers