expect-lite
Automation for the rest of us
Features
Expect-lite has many features which make expect-style script writing easy. Such as:- Automatic login to remote host (supporting telnet, ssh with password, and ssh with keys)
- Simplicity using
one
character for send and expect '>' '<'
This creates many lines of expect with minimal "code"
- Regular expression (regex) evaluation in expected results
- Non-Regex evaluation also available in expected results
- Static
and
Dynamic
Variables with
math
functions and pseudo arrays
- Constants, passed on the command line which override variables in the script
- Leverages Bash shell
- Include mechanism to read other script files (sourcing)
- Embedded Expect - using native expect embedded in an expect-lite script
- Not Expect - a feature to look for unexpected text/strings
- Conditional Branching a if/then/else with a c-like syntax
- Simple looping using conditional branching, a repeat loop can be constructed with infinite loop protection
- Scripts can be executable with the help of a small embedded bash script
- User
defined
prompt
allows interaction with a wider range of hosts (routers, switches, etc)
- Multiple Sessions enabling interaction with a client and a server in one script
- Instant-Interact empowers on the spot debugging
- Color enhances output, differentiates log messages new in version 3.7.0
Automatic
Login
expect-lite was born into a multi-host environment, and therefore
there was the early requirement to remote login to a host (usually a
Linux machine) allocated by a sharing facility such as the commercial
package, LSF. Therefore, a remote host must be specified on the command
line where the script will wake up and begin executing. Three methods of remote login are supported:
- telnet
- ssh with password
- ssh with keys (no password)
Even when no remote host is required, it is best to log into the localhost, since the underlying Expect has problems with synchronization (between send and expect strings) when a remote host is not specified. The provided shell script 'setup_local_ssh.sh' will setup the localhost with ssh keys. It is only necessary to run it once:
./setup_local_ssh.shBased on the command line arguments passed, expect-lite will do the following:
- Automatically log into the remote host via telnet or ssh (if ssh keys have been previously setup)
- Change directory to the passed directory (if included)
In the following example of expect-lite:
./expect-lite remote_host=remote-host-018 cmd_file=basic_ping_test.elt user_dir=/etcOr the shorter version
./expect-lite r=remote-host-018 c=basic_ping_test.elt d=/etcOr the shorter version which is more linux like
./expect-lite -r remote-host-018 -c basic_ping_test.elt -d /etc
expect-lite will log into remote-host-018, change directory to /etc/ and begin to send commands from basic_ping_test.elt to the remote-host. The parameters of remote_host and cmd_file are required, as expect-lite must know what host to log into, and what script (aka command file) to run.
Environment Variables
The following environment variables can be used as alternatives to specifiying command line arguments, remote_host, cmd_file, and user_dir respectively:
- EL_REMOTE_HOST
- EL_CMD_FILE
- EL_USER_DIR
Special
Character Sequences
Although basic expect-lite scripts can be created
by simply cutting and pasting text from a terminal window into a
script, and adding '>' '<' characters, such as:>pwdMuch more functionality is available. expect-lite interprets punctuation characters at the beginning of each line in the script:
</home/user
-
Char
Action
>
send string to the remote host >>
send string to remote host, without waiting for prompt (see implementation details) <
string/regex MUST be received from the remote host in the alloted timeout or the script will FAIL! (see test failure) << literal string MUST be received (similar to '<' without regex evaluation) -<
if string/regex IS received from the remote host the script will FAIL! (see Not Expect)
#
used to indicate comment lines, and have no effect ;
are also used to indicate comment lines, but are printed to stdout (for logging) ;<colour> add custom colour comment lines, colour may be blue, ltblue, gray, ltgray, cyan, black, pink, purple, red, green, yellow ;;
similar to above, but no extra newlines are printed (useful for printing script help)
@num
changes the expect timeout to num of seconds
$var=
static variable assignment at script invocation (see variables for details) +$var=
dynamic variable assignment (see variables) +$var increment value of $var by 1 decimal (see repeat loop) -$var decrement value of $var by 1 decimal =$var
math functions, perform bitwise and arithmetic operations: << >> & | ^ * / % + - (see math functions)
!
indicates an embedded expect line (see embedded expect) ?
c-style if/then/else in the format ?cond?action::else_action (see conditionals)
%
label - used for jumping to labels (see conditionals)
~filename
includes a expect-lite automation file, useful for creation of common variable files, or 'subprograms/subroutines'
Sometimes control characters must be sent. Currently all control characters are supported (from ^@ to ^\):
- ^C or break, when it is desired to stop a running program e.g. >^C or >>^C
- ^] or escape from telnet, e.g. >^] This will bring the script to the telnet prompt, then use: >quit
- ^<any char> will send a control character to the remote-host, such as ^D to logout
Lastly, any line beginning with an asterisk '*' will be interpreted as an expect-lite directive. Directives change the behaviour of expect-lite.
-
Expect-lite Directives
*~filename Include a fail script, which expect-lite runs only if the main script fails (see fail script) */prompt/
Set a user defined prompt (see user defined prompt)
*TERM
*PASS
Immediately terminates script, but returns 0 (pass) (see stopping execution early)
*FAIL
Immediately fails script, and returns 1 (fail) (see stopping execution early)
**SHELL=<shell>
Configuration directive which sets the shell immediately after automatic login to remote host (see setting the shell)
*INTERACT
Places user in interact mode, which pauses the script and turns control of the keyboard over to the user (see Tips & Techniques: Interact)
*FORK <session> Multiple session support. Directs expect-lite to open a new session and spawns a newshell (see multiple sessions) *SHOW VARS
Debug information, displays all expect-lite variables. Can be used in interact mode. (to be deprecated) *EOLS LF
*EOLS CRLFControls end of line sequence sent to remote host, either line feed, or carriage return & line feed.
Logging Directives *INFO
*NOINFOEnable/disable informational messages (see managing logging) *EXP_INFO
*NOEXP_INFOEnable/disable "expected" messages (addtionally use -V flag to enable) *WARN
*NOWARNEnable/disable warning messages *DEBUG
*NODEBUGEnable/disable debug messages *NOCOLOR Disables color output, best when logging output
General Tips for writing scripts
Here are some simple tips when script writing:- Use reasonable timeouts, if 30 seconds is needed to get a response, set the timeout at 45 or 60 seconds, not 600.
- There is no cost to changing the timeout, timeout
values
can
also be variables
- Beware of expect-lite using regex, when creating lines like: <0.005 secs (5 micro secs)
- The parentheses is used by the regex engine, instead escape these characters: <0.005 secs \(5 micro secs\) (see Use of Regex in expect-lite)
- or use '<<' which does not use regex, and does not require escaping: <<0.005 secs (5 micro secs) (see using non-regex evaluation)
- Use the expect character '<' or '<<' often. Check for valid results when possible. A script which expects nothing will never fail! (see Test Failure)
- Use printable comments ';' and ';;' often. Think of it as writing a note to oneself, it will make reading log files much easier. As of version 3.7.1 printable comments default to blue, however using a colour right after the semicolon will change the colour for that comment.
Debugging Expect-lite Scripts
Expect-lite was written to permit quick and easy automation of repetitive tasks. However, sometimes a little additional debugging is required.Instant-Interact (as of version 3.5.0), provides instant debugging assistance. Pressing ^\ (control+backslash) will force expect-lite to drop into interact mode, a mode which turns control of the keyboard over to the user, so that one may type directly to the process on the remote host. To return to the script, type '+++' (three pluses). During interactive mode, timeouts do not apply, and the script will wait indefinitely for the user to exit this debugging mode. Switching between multiple sessions (see *FORK) is possible via typing the *FORK command in interact mode (see *FORK debugging).
Additionally, the include scripts, debug.inc and show_vars.inc, have been added to the release (as of version 3.1.3) to assist in debugging scripts (see also *SHOW VARS)
See the Tips and Techniques Guide for additional expect-lite script debugging information.
Use of Regex in expect-lite
Support of regular expressions in expect-lite are limited by:- Support of regex in standard expect (e.g. including
anchors,
char-classes and repeats)
- Support of regex meta characters (e.g. \t \n are supported, \d is not)
- Expect-lite only evaluates lines using regex which
begin
with
'<' '-<' and '+' (dynamic variables)
> /sbin/ifconfig eth0The periods are preceded by a backslash to indicate to regex that a period must be returned rather than the regex dot '.' which indicates any character.
<inet addr:10\.29\.200\.[0-9][0-9][ ]
Because regex is always enabled for expected results, some 'escaping' of characters must be done when using '<'. The following characters must be escaped with a backslash to convey their literal meaning:
| Example |
Escaped |
Character |
| (abc) | \(abc\) | parenthesis |
| [abc] |
\[abc\]
|
square brackets |
| \ |
\\ |
backslash |
| . | \. |
period |
| $ | \$ |
dollar sign |
| + | \+ | plus |
| * | \* | asterisk (or star) |
| | | \| | pipe |
Using << for non-regex evaluation
There are times when the use of regex is not desired, or 'escaping' characters is burdensome. As of version 3.6.0, expect lines can begin with '<<' which will not use regex, but rather literal evaluation of expected results. That said, wild-carding is still permitted such as '<<wild*permitted' would match 'wild-carding is still permitted'.Handling HTML/XML Tags in expect lines
The << may cause a backward compatibility problem, if the script was expecting an HTML tag, such as
>grep body my_html_file.htmlIn the above example, the script is expecting a body tag with 1 or more characters between body and the closing greater-than of the tag. However the new feature will not interpret this line as regex, but rather interpret the beginning of the line as <<, and evaluate it as a literal. The method to force expect-lite to evaluate the line as regex, is to change the line is slightly:
<<body.+>
>grep body my_html_file.htmlBy making the first expected character as a regex character class, it signals to expect-lite that this line should be evaluated as regex, rather than a literal.
<[<]body.+>
Using Variables
Expect-lite supports two types of variables:- Static, which are bound at invocation
- Dynamic, which are bound during execution of the
script
Sometimes it is desirable to bind a variable during the execution of the script. Dynamic variables fill this need by utilizing Expect's built-in regex capture mechanism. Only the portion of the match inside the parenthesis will be bound to the variable. The format of the line is as follows:
+$somevar=text that is not put into the var (text which is put into the var)
Dynamic Variables are always bound to Expect output, meaning text which is returned from the remote host. Therefore something must be sent to the remote host, before text can be returned. A typical usage would be:
> command
+$myvar=command output (capture value) more command output
If the value of the var is successfully captured then expect-lite will print:
Assigned Var:somevar=sometext which is put into the var
If, however the dynamic var is not successfully captured, expect-lite will print:
Assigned Var:somevar=__NO_STRING_CAPTURED__Avoid using an overly general capture as it will tend to capture too much, or the wrong info. e.g:
> commandInstead use a specific capture when possible. When capturing the current directory, the current directory will always start with '/', and the path will be made up of a known character set:
+$myvar=\n(.*)
> echo $PWD
+$mypwd=\n(/[a-zA-Z0-9/\-_]+)
Expect-lite will print to stdout the value of the assigned variable, assisting the script writer in understanding the value that is bound to the variable during runtime. The value of "__NO_STRING_CAPTURED__" indicates that the regex pattern used did not match the return data. It is best to start using this feature with short scripts targeted at capturing the desired information. Examples can be found in the example section of this document.
Math
Functions with Variables
There are times when writing a script it may be necessary to perform
arithmetic or bitwise operations, also known as math functions.
Expect-lite inherits expect/tcl math functions and natively supports
math/bitwise operations on a variable, using the following slightly odd
notation:$myvar=1In the above example, 2 will be multiplied by 10 and added to value of $myvar (1). The result of 21 will be placed in $myvar, overwriting the previous value. Since multiplication has a higher precedence than addition, (2 * 10) will be performed before the addition to $myvar.
=$myvar + 2 * 10
Parenthesis may be used to enforce user defined precedence. In the following example, the result will be 23:
$myvar=1
=$myvar + 2 * (10 + 1)
Both bitwise and arithmetic operators are supported with the following precedence (left to right):
| bitwise |
<< >> & ^ | | shift left,
shift
right, bit
and, bit exclusive or, or |
| arithmetic | * /
% + - |
multiply,
divide,
modulo, add,
subtract |
If $myvar does not exist before the math function, it will be initialized to blank and then math functions will be performed. In the following example the result, $myvar, will be 20:
=$myvar + 2 * 10However, realistically, only the '+' operator works as expected in the above example. Using other operators will more than likely yield a syntax error:
Warning: Expect-lite: Unable to interpret =$answer1 / 2 * (10 + 1)
syntax error in expression " / 2 * (10 + 1)": unexpected operator /
Because of the expect/tcl inheritance of math functions, all operations will be performed as integers (or whole numbers) unless a decimal (or real) number is used explicitly. In the following example the result is 2.5:
$five=5However, the following (because of the use of integers) example will have the result of 3:
=$myvar + $five / 2.0
$five=5
=$myvar + $five / 2
Math functions can be used with static and dynamic variables.
Pseudo Arrays
Expect-lite supports pseudo arrays, which are different from a real array (e.g. item[1], item[2],...). Pseudo arrays are made possible by the fact that variables on the left side of the equals can be dereferenced at assignment time. For example, below, the variable $count will be dereferenced (or resolved) prior to assignment:$myarray$count=some $valueWhen placed in a loop with a looping variable of $count, a pseudo array of $myarray1, $myarray2, $myarray3,... will be created. Individual values can be retrieved from the array by using the index (or $count) such as $myarray23.
The pseudo array name (variable + index) must be a valid variable name, in the set of characters (A to Z, a to z, 0 to 9, and underscore) and no spaces.
Constants
Expect-lite Constants are represented as variables which are passed into expect-lite at runtime. Constants will override any script variable already defined inside the script, are immutable, and cannot be changed. Constants can be used to change the behaviour of the script.For example, one might invoke the following test as:
./expect-lite remote_host=remote-host-018 cmd_file=basic_ping_test.elt \
user_dir=/etc local_eth=eth2
Inside the script basic_ping_test.txt any reference to $local_eth would be set to eth2, thus allowing the script actions to be changed based on the constant passed at invocation. Constants will override any script variable already defined inside the script.
Shell Variables
Expect-lite allows the use of shell variables, which can be more powerful than the built-in variable mechanism. However shell variables will only be resolved by the shell. For example, assigning current working directory to a shell var:> PWD=`pwd`However, the following test would fail, since $PWD is a shell variable, not an expect-lite variable:
> echo $PWD
> PWD=`pwd`Shell variables (or environment variables) must be dereferenced by the shell. Similarly, expect-lite variables must be dereferenced by expect-lite.
>pwd
< $PWD
It is possible to read a shell variable into an expect-lite variable by using the dynamic variable method:
> echo $PWD
+$mypwd=\n(/[a-zA-Z0-9/\-_]+)
Environment Variables starting with EL_
When expect-lite starts up it will automatically search the environment variables (aka shell variables), and import any found starting with EL_ as constants. This enables setting environment variables as passwords, or remote hosts.For example, the environment variable EL_sudopass might be set to "mysecret". In the script, the variable would be used as:
>sudo tcpdump -i eth0
<<[sudo] password
>>$EL_sudopass
Using
Conditionals & Labels
Conditional (if/then/else) statements are natively supported in
expect-lite. The conditional uses a c-style syntax with a question mark
(?) at the beginning of the line, and double colon to indicate the else
statement, using the format ?cond?action::else_actionAlthough spaces are not required around the conditional characters (? and ::), it is recommended for ease of reading. The comparison operators are: '==', '!=' ,'>=', '<=', '>' and '<'. If the compared values can be evaluated as a numbers, then larger and less than will yield expected results. A simple conditional example:
$age=56In the above example if $age is larger than or equal to 55 then the action 'echo "freedom at 55!". If $age is less than 55 then the action 'echo "keep working!"' will be sent.
? $age >= 55 ? >echo "freedom at 55!" :: > echo "keep working!"
Each action or else_action is as if it began on a separate line. Since in the above example '>' is used (e.g. >echo) the echo line will be sent to the remote host. Any expect-lite action (see Special Characters) can be placed after the second question mark. For example, an include file may be executed based on a conditional:
$platform=ppcIn order to make log files more understandable, a message will be printed when a conditional is evaluated, such as:
? $platform == i386 ? ~connect_i386.inc :: ~connect_ppc.inc
If: expr { "ppc" == "i386" } |then ~connect_i386.inc|else ~connect_ppc.inc|result0
The message prints: what was evaluated (expr { "ppc" == "i386"
}), 'then', and 'else' actions, as well as a result (true or
false)Labels
Conditionals are limited to a single line. Sometimes this is too limiting, as it would be nice to have several commands be executed based on the success of a conditional. To support this, the concept of labels has been introduced. A label is defined as having the first character a '%'. Although the label line itself does nothing, it provides a location to the conditional to Jump To Label. The following is a simple example of using a conditional in conjunction with a label:
$kmodules_inserted=trueWhen a label is the action of a conditional, there is an implied Jump to Label. Lines between the conditional and the label will be skipped. The Jump to Label action is no longer limited to only jumping forward. (see Looping with Conditionals)
? $kmodules_inserted == false ? %SKIP_CHECK_KMODS
>lsmod | grep nfs
<nfsd
<exportfs
<sunrpc
%SKIP_CHECK_KMODS
Multiple labels with the same name are permitted. For example, a conditional action may be Jump to Label %SKIP. Although multiple labels of %SKIP may occur in the script, in this example, expect-lite will jump to the next line containing %SKIP. Of course it is easier for the script reader if clearer label names are used. Labels may contain spaces, as in this example:
; === Test conditional jump to labelTo assist reading the logs, the action Jump to Label will generate a message to standard out (captured in the log). From the above example, the following would be printed:
? $jump == true ?%move along, nothing to see
>echo "1"
>echo "2"
>echo "3"
%move along, nothing to see
Jumping to label:%move along, nothing to see
Looping with
Conditionals & Labels
Simple looping is now supported by allowing jump to label
backwards. The Repeat
Loop is the easiest loop to implement:; ======== Incrementing Loop ========Because of jump to label can jump backwards, it is important to assign unique looping labels, such as %REPEAT_INC_LOOP. Unexpected results will occur if non-unique loop label names are used. Non-looping labels, as illustrated in the previous section, are not required to be unique.
$max=5
$count=3
%REPEAT_INC_LOOP
>echo $count
# increment variable
+$count
?if $count <= $max ?%REPEAT_INC_LOOP
Also included in the above example is incrementing an expect-lite variable: +$count
This will add 1 to the value of $count. If $count is not an integer, the value of $count will remain unchanged (can't add 1 to a string).
As part of the looping enhancement, there is infinite loop protection. The maximum amount of looping is defined in expect-lite itself with the variable _el_infinite_loop. This value is decremented with each iteration of all loops for the entire script. Typically this would be in the range of 100 to 1000 to be safe. For example, if a complex expect-lite script had 4 loops each with 100 iterations, the _el_infinite_loop should be set larger than 400.
Expect-lite & Bash
Although not limited to working with bash, bash is invoked upon logging into the remote host, and therefore will be discussed more here.Since the bash shell is well documented, and supported, and therefore can be leveraged to assist in expect-lite's limitations such as looping and branching (see limitations below). A simple bash loop inside expect-lite can be created for example:
$set=1 2 3 4 5In the above example, $set is an expect-lite variable, not a bash variable. An expect-lite variable is always declared before its use (e.g. $set= 1 2 3 4 5). If a variable cannot be dereference by expect-lite it is passed to the shell. The loop will execute after the final "}" line is sent to the remote-host. Because of this "execute after" effect, an expect-lite '<' line can not be used within the bash loop.
>for i in $set #expanded by expect-lite
>{
>echo $i #expanded by bash
>}
Another example is the bash while loop:
$max=5Conditionals via bash are also supported, as in this example:
>i=0
>while [ $i -lt $max ]
>do
>echo $i
>let i+=1
>done
$max=4More complex bash assists can be constructed. For example, one may want to set a looping variable based on the processor type of machine at the time (some hosts may be faster than others). The answer is echoed, and then captured using a dynamic variable. Note that that 'proc' is prepended to the answer, as it enables the capture to be much more specific.
>if [ $max -ne 5 ]; then
>echo "max is not equal to 5"
>else
>echo "we have a winner"
>fi
>PROC=`uname -i`Note that 'proc' is outside the parenthesis, and therefore will not be captured into the expect-lite dynamic variable $loop_counter.
>if [ "$PROC" == "i386" ]; then
>echo proc20
>else
>echo proc100
>fi
+$loop_counter=\nproc([0-9]+)
Using Bash to create executable expect-lite scripts
It is possible to make expect-lite scripts executable, with the help of a small embedded bash helper script. Insert the following at the top of the expect-lite script:
#!/usr/bin/env bashThe bash scriptlet does the following:
# make this auto-runnable!
# Little bootstrap bash script to run kick start expect-lite script
# Convert --parms to expect-lite param=value format
PARAMS=`echo $* | /bin/sed -r 's;--([a-z]+) ([0-9a-zA-Z]+);\1=\2;g'`
echo $PARAMS
expect-lite r=none c=$0 $PARAMS
exit $?
- Converts command line arguments from --arg value to arg=value format, making the script more linux-like
- Call the expect-lite script with default arguments (in this example that is r=none) and converted arguments
- Exit with the same exit code of the expect-lite script (0 success, 1 failure)
The final step to making the script executable is to use a chmod command:
chmod 775 my_expect-lite_script.eltNow the expect-lite script can be executed with --arguments or the standard expect-lite argument format (mixed and matched).
$ ./my_expect-lite_script.elt --r host-15 --user_name craig --max_loop_count 100This bash scriptlet example can be found in expect-lite.proj/Examples/test_masquerade.txt
Include Files
Include files are a quick way to develop script snippets which can be included into larger scripts or to include a common variable file. When an include file is executed, it is as if the file were just pasted into the script file, and therefore has access to the variable space of the main script, and can modify that variable space as well. In this example, a common variable file is "sourced":# Source common variable file
~asic_vars.inc
Common functions, such as telnet'ing to the DUT, are a good use of include files:
; === Connect to DUTInclude filenames can also be assigned in a variable, such that the file names can be declared at the top of the script but used later within the script. For example:
~dut_connect.inc
# Source Var file to be used
$asic_include=asic_vars.inc
...
~$asic_include
Embedded Expect
There are situations when expect-lite cannot provide a solution. Rather than force the user to abandon the simplicity of expect-lite, an embedded expect is supported. This functionality is provided to assist expect-lite rather than replace it. Lines beginning with '!' will be interpreted as native expect (see limitations below).Embedded Expect requires the script writer to know native expect. It runs with some variable protection, and provides access to expect-lite variables and constants, as well as the expect variable timeout. This permits the script writer to query the timeout, store it away, and then reset the timeout later.
# Preserve old timeout using embedded expectExpect-lite variable/values are stored in the tcl array user_namespace() with the variable name as the index name. Constants are stored similarly in the tcl array cli_namespace().
!set user_namespace(TIMEOUT) $timeout
# set timeout to 15 seconds - max time to wait for login prompts
@15
>telnet $dut_ip
<login
>>root
<assword
>>my_secret
>
# Set the timeout back to old timeout
@$TIMEOUT
Expect code is collected in islands and executed after an island shore is reached. The island shore is represented by lines that start with any special character, except '#' and blank lines. However, it is recommended that the '>' be used, as this implies a "wait for prompt."
Branching and looping are supported in embedded expect e.g.:
!if { $arch == "ppc" } {
! puts "\narch is $arch\n"
!}
>
And looping:!for {set j 1} {$j<6} {incr j} {
! if {$j == 1} {set type "abc" }
! if {$j == 2} {set type "def" }
! if {$j == 3} {set type "hij" }
! if {$j == 4} {set type "lmn" }
! if {$j == 5} {set type "qrs" }
!
! puts "$j>$type"
!}
>
Provides access to expect send & expect commands:!send "ls\n"Embedded expect can fail the entire expect-lite script by calling the built-in _el_fail_test function:
!expect -re "test" { puts "got test"} timeout {puts "got nothing"}
!if { $arch == "ppc" } {
! puts "\narch IS $arch\n"
! } else {
! puts "\narch is NOT ppc, but $arch\n"
! _el_fail_test
! }
>
TCL files can be sourced, and functions can be declared and called from
within embedded expect.Limitations of Embedded Expect:
- Limited support of expect global variables. To declare a variable at the top level as global use "set ::<var> <value>" The double colon '::' makes var global (this is a TCL standard)
- tcl/expect switch command not supported
- expect statements must be on one line (as in the
example
above)
Please remember, this functionality is to provide assistance to expect-lite rather than replace it. (see What's Missing)
Expect-lite Directives '*'
Expect-lite directives are indicated by lines which start with the
asterisk, '*'. The directives change the behaviour of expect-lite in a
user defined manner. These directives include:- fail script - run when the script fails *~include_fail_script
- user defined prompt - for those non-unix/linux remote hosts */regex/
- stopping the script - *TERM & * FAIL
- setting the default shell - **SHELL
- multiple sessions - for hard to script environments *FORK
- managing logging - *INFO, *NOINFO, *EXP_INFO, *NOEXP_INFO, *WARN, *NOWARN, *DEBUG, *NODEBUG, *NOCOLOR
- debugging aid - *INTERACT (see debugging tips)
- print all expect-lite variables - *SHOW VARS
- control end of line character(s) - *EOLS LF, *EOLS CRLF
Fail Script *~include_fail_script
The purpose of the fail script is to clean up after the failed script, reseting times, deleting temporary files and such. It is a special include script which is declared near the top of the script:*~clean_up.incShould the script fail, the fail script will be "sourced".
The scope of the fail script is always local to the running script. This allows normal include scripts to declare a separate fail script from the main script. Thus enabling a different clean up mechanism for the include script.
The fail script mechanism can also be used for script development. One the script fails, it is possible to drop expect-lite into interactive mode allowing the script developer to interact the DUT interactively. debug.inc in the following example:
*INTERACTThe above debug fail script places the developer into interactive mode. Once the developer has completed with interactive mode, '+++' is typed and the script continues. In this particular example, a flag is set _el(continue), which allows the main script to continue as if there were no failure.
>
# allow script to continue
!set _el(continue) 1
>
; === Continuing Script
>
User Defined Prompt */prompt/
With each '>' command an implied "wait for prompt" occurs (see implementation details). The predefined prompts are based on standard shell prompts (>%$#). However, it is quite possible that expect-lite will not be interacting with a shell, but another application (such as gdb) or device which does not issue a shell-like prompt.As of version 3.1.5, the user may define a custom prompt using the following command:
*/my_prompt /The succeeding '>' lines will interpret a prompt as the standard shell prompts and 'my_prompt '. The user defined prompt definition between the slashes is interpreted as regex, and therefore regex rules such as escaping applies. For example, to set a user defined prompt for gdb the following would be used:
*/\(gdb\) /Only one (1) user defined prompt can be active at a time, however multiple prompts can be represented using the regex OR '|', for example, creating a gdb prompt and my_prompt:
*/\(gdb\) |my_prompt /Another application of the user defined prompt is to make telnet logins easier:
*/login: |ssword: /Clearing a user defined prompt. There may be times where a previously user defined prompt is causing problems by output which falsely triggers the user defined prompt. It is possible to clear the user defined prompt by:
>telnet remote-host-018
<Connected to
>$username
>$password
>
*//Debug the user defined prompt by using the -vv or --verbose cli parameter. In the example below, the user defined prompt has been incorrectly */\(gbb\) / while running gdb (a linux debugger). The debug output shows the user defined prompt and the string expect-lite is searching for the prompt.
(gdb) Warning:Prompt Timed Out!
DEBUG: User Defined Prompt:<<\(gbb\) >>
Not Found in<<
(gdb)
>>
The scope of the user defined prompt is global, that is, it extends to the main script as well as all include files referenced.
Setting the default shell with **SHELL=<shell>
expect-lite has been tested with bash, however it should operate with other shells. Based on the user's preference, a shell can be selected with the **SHELL directive. Because the shell is invoked immediately after login to the remote host, this is a configuration directive with is read before the script executes. Therefore, it can be placed anywhere in the script, and is not limited to the first line.Stopping the script early with *TERM & *FAIL
There are times when it is desirable to terminate the execution of a script early (earlier than the end of the script). For example, while debugging a script, it is convenient to stop execution mid way in the script while focusing a troublesome section. Or to perform a test, such as this is Monday, and terminate the script.*TERM will stop execution and return 0, or Pass. This is useful for debugging. *FAIL will stop execution and return 1, or Fail. If a fail script has been declared earlier, it will be executed before terminating execution.
Debugging by examining expect-lite variables with *SHOW VARS
It can be useful when debugging a script to see what the value of variables are at some point in the script. This can be especially useful when used with *INTERACT, or pausing the script, and looking at the value of the variables.Multiple Sessions *FORK
Until version release 3.5.0, expect-lite has been intentionally limited to a single session keeping it simple. However, there are certain environments where it might be advantageous for a single script to control/monitor both a client (e.g wget) and a server (httpd log). Without multiple session support this type of environment would be difficult to automate with expect-lite.What is a new session? A new session starts with a new shell on the remote host. All commands '>' and received text '<' are constrained to that session. It is possible to use multiple sessions on the localhost using r=none, however r=none support is limited due to timing issues. If multiple sessions on the localhost (the same host where expect-lite is running), it is better to ssh to the localhost by specifying r=localhost. This loop back method ensures that the timing between commands and received text stay in sync.
With the version of 3.5.0, expect-lite supports nearly limitless multiple sessions. However, it is a good guideline to use as few sessions as needed. For backward compatibility, the first session is the "default" session. Additional sessions are assigned a name at invocation with the *FORK <session_name> directive:
*FORK Serverexpect-lite will print an "INFO" line stating the current session name
INFO: FORK session is: Server
Session names may not contain spaces, and must be unique for each session. To switch back to a previously started session, re-use the session name. The session name "default" (without quotes) is reserved for the first session.
*FORK defaultPrint the current session name by using the *FORK with no session name:
INFO: FORK session is: default
*FORK*FORK and variables. It is possible to assign a session name to a variable, and then create a new session with that name. For example:
INFO: FORK session is: Server
$my_session=ClientAll sessions will be automatically closed when the script terminates.
*FORK $my_session
INFO: FORK session is: Client
Debugging in a multi-session environment: When in INTERACT, either via the *INTERACT command in the script or via instant-interact (by pressing ^\), the *FORK command is still available, permitting the switching of sessions while in INTERACT mode. For example:
expect-lite directive: *INTERACT
Press '+++' to end interact session & return to script
% *FORK CONSOLE
INFO: FORK session is: CONSOLE, Active sessions are: default CONSOLE
#
With the addition of multiple sessions, decoding the output will become more difficult. Remember, although multiple sessions are supported, expect-lite is still single threaded, meaning commands will always be executed the order as they appear in the script.
Managing Logging *INFO, *NOINFO, *EXP_INFO, *NOEXP_INFO, *WARN, *NOWARN, *DEBUG, *NODEBUG, *NOCOLOR
Expect-lite provides additional information (logging) which makes it easier to debug failures. As of version 3.7.0, the output is coloured based on three levels of logging: info, warnings, and debug. By default, info and warnings are turned on and exp_info is turned off. However, using the logging directives: *INFO, *NOINFO, *WARN, *NOWARN, *DEBUG, *NODEBUG, these messages can be managed. Additionally, *NOCOLOR, will turn off color output.Example of the different logging levels are:
- INFO: Entering FORK sessions, Constant substitution, Conditional Statement Results, Jumping to labels, Including files, Dynamic Variable Assignments, User Defined Prompt assignments
- EXP_INFO: prints "expected" text (lines with < or <<). Enabled with -V flag on the command line.
- WARN: Instant-Interact feature disabled, Wait-for-prompt timeouts, Using localhost (r=none), Unable to parse Conditional, Dynamic Variable Capture timeouts, Unable to parse math variable statement, Unable to parse FORK session name, Unable to parse Conditional Statement
- DEBUG: Additional debugging information for: User Defined Prompt, FORK session instantiation/switching, '<' searching, Dynamic Variable capture
In addition to the three logging levels above, there is another more serious level of ERROR, which can not be suppressed, and will terminate expect-lite with a non-zero return code. Examples of ERRORs are: unable to connect or lost connection with remote host, include file not found, bad timeout value (non-numeric) or infinite loop detection.
It may be tempting to turn off logging once a script is debugged. However, if the script is being run in an automated environment (like crontab), it is recommended that INFO and WARN be turned on, as the additional information will make debugging failures much easier.
Controlling End of Line Sequence with *EOLS
There are some remote devices which require carriage return + line feed at the end of each line sent. In past versions of expect-lite, only line feed was appended to each '>' or '>>' line. This limitation made it difficult to automate tasks with this class of remote devices.With version 3.7.0, the end of line sequence can be controlled with the following expect-lite directives: *EOLS LF & *EOLS CRLF. The default is *EOLS LF.
These directives can be placed anywhere in the script and the setting remains in effect until another *EOLS directive is encountered. For example, enabling a script to control a linux machine (using *EOLS LF), and later in the script controlling a remote power-switch device (using *EOLS CRLF).
Examples of Expect-lite
Below are some examples to better illustrate what can be accomplished with expect-lite:Setting the IP address on a secondary interface of the remote host (be sure to escape the dot's in the IP address)
$local_eth=eth1Starting a telnet to another host
$ip_addr=192\.168\.10\.2
# change timeout value to 10 sec
@10
>ifconfig $local_eth $ip_addr
#check to ensure that we set the right IP address
>ifconfig
<$ip_addr
@2Starting a ssh session to another host
>telnet remote-host-018
<Connected to
<login
>>root
<assword
>>secret_password
# issue a command once logged in
>pwd
>^]
>quit
>ssh root@host-021Use of 'screen' command to keep a command alive after test completes
<assword:
>secret_password
# issue a command once logged in
>ls
>exit
; ==== Use Screen command for to keep application running ====Assigning a dynamic variable $host using regex to capture the hostname from 'uname -a' command
>screen
> $pmm
# Check that HW is alive
>ping
<Successfully pinged the PM H/W
;purple ==== Keep pmm application active, and detach from this screen
>>^A
>>^D
<detached
>
>uname -aAssigning multiple dynamic variables using regex to capture environment variables $HOME and $SHELL, while expecting $TERM=xterm
+$host=Linux ([a-z\-0-9]+)
Assigned Var:host=remote-host-008
>
>env | sortManage multiple Screen commands with a dynamic variable and regex capture
+$home=\nHOME=([a-z/]+)
+$shell=\nSHELL=([a-z/]+)
<TERM=xterm
Assigned Var:home=/home/joe
Assigned Var:shell=/bin/bash
>
# show the last screen in the listUsing Regex in expect-lite to allow multiple responses
>screen -list | grep pts | sort | tail -1
# regex capture to dynamically assign the result to var $myscreen
+$myscreen=([0-9]+\.pts.[0-9]+.[a-z\- 0-9]+)
# connect to $myscreen
> screen -r $myscreen
# inside existing screen, kick it to get a prompt
>>^M
> sleep 3
> ls
>
>>^A
>>^D
<detached
>
>idCreate a temporary file with unique name including script name and date. The predefined variable $arg0 is contains the expect-lite script name.
# allow either groups users or wheel
<(users|wheel)
# date stamp of script run time
>echo `date +_%Y_%m_%d_%H-%M-%S`
+$DATE=\n(_[0-9_-]+)
$temp_file=$arg0$DATE
; === Create a temp file using bash "here doc" method
>cat > $temp_file <<'+++'
>line 1
>line 2
>+++
>
Test Failure?!?
When will expect-lite fail a test? It will only fail a test when an expected result (after issuing a command) does not appear in the specified timeout period. For example in the following command file:$ip_addr=192.168.10.99The first line will be sent (blindly), since there is no expected return. On line 3, the ifconfig command is sent again, and the script is looking for a desired result of 192.168.10.99 (using the var $ip_addr). If for some reason the ip address is different than ip address was returned, the script would fail.
>/sbin/ifconfig eth0 $ip_addr
>/sbin/ifconfig eth0
< $ip_addr
Not Expect
A test can also fail should text appear that is unexpected (such as an error). This does not clear the expect input buffer, and should be used before a valid expect. How long should the script wait for the un-expected? In order to reduce delays in script running time, the Not Expect feature only waits for 100ms. This is usually enough time to detect quick error responses such as "file not found".
For example: Fail device doesn't exist
>ls -l /dev/linuxIn another example: We don't like Mondays
-<No such file
; === fail if today is 'Monday'Since the expect input buffer is not consumed, valid expects can still be performed. in the following example, ttyS2 will still be found.
>date +%A
-<Monday
>ls -1 /dev/ttyS*It is recommended that Not Expect lines be used before valid expect lines.
-<ttyX
<ttyS2
How can it help?
Expect-lite can be used to quickly create automated tests. It can collapse complicated tests into a single command line. It can also be used in nightly regression testing, performing simple functional tests providing confidence that core functionality has not been broken by the previous day's submissions.Limitations
Expect-lite is limited to what a person could do in one (1) terminal window (or xterm). It cannot start a program in one window and run a different program in a separate window. However it does support multiple sessions via *FORK which should meet the needs of most.
Standard programming constructs such as looping and branching are supported as of version 3.0.0. Real Expect can do these things and more. That said, looping is limited to a repeat loop and branching can be accomplished with Labels.
Expect-lite has been purposely limited to keep scripts simple, easy to use and maintain. Although more complex scripts can be created, basic expect-lite scripts can be created by simply cutting and pasting text from a terminal window into a script and adding '>' '<' characters.
Implementation Details
Variables
Variable names must be restricted to the following set of characters [A-Za-z0-9_]. Variable values may include spaces, and quoting is not required nor permitted. For example:$bell=For whom the bell tolls
>echo $bell
For whom the bell tolls
Implied "wait for prompt"
Using the '>' character implies waiting for the prompt before sending the command. The definition of a shell prompt a: $, #, % or >. It is often useful to follow a '<' with a '>' for force the script to wait for the prompt:> $some_long_commandPrompt detection can be problematic, and by no means is expect-lite's method perfect. Expect-lite looks for a prompt character ($, #, % or >) followed by a space at the end of line. This should work for most standard prompts, however coloured or fancy prompts may not be detected. As of version 3.1.5, user defined prompt is supported.
<critical value
# wait for prompt
>
It may be useful to create files on the fly which will be used in the test. Since expect-lite does not read/write files directly, this function can be done with the linux command 'cat' and the bash here doc method. Creating a regex file for example:
; ==== Create Test File on the flyIn the above example the script initiates a cat to a file (referenced by $regex_test_file ). Normally the '>' would be used to send lines to the remote host. However, there is an inherent "wait for prompt", which will politely wait for the next line. The last '>' is there to wait for the prompt to return after the cat command.
> cat > $regex_test_file << '+++'
>T00001/a(?$XL.*)/tag=0x08000001
>T00002/b(?$XR.*)/tag=0x08000002
>T00003/c(?$XO[0-7]*)/tag=0x08000003
>T00004/d(?$XH[0-9A-F]*)/tag=0x08000004
>T00005/e(?$XD[0-9]*)/tag=0x08000005
>+++
>
Predefined variables
Expect-lite has minimized the use of predefined variables to allow the user the widest latitude in selecting variable names. However a few predefined variables are useful and listed below:| Predefined
variable |
Usage |
| $arg0 | expect-lite script name. Set to the c= or cmd_file= value. Useful for creating temporary files unique to the test script. |
| $timeout_multiplier | Typically used as a constant (passed in from the command line) which the integer value is multiplied by each of the timeouts in the script. i.e. @10 x $timeout_multiplier = real timeout. The value default is 1. |
Timeouts
Setting the expect-lite timeout to 0 isn't actually zero as this will cause the keyboard input buffer on the remote machine to overflow (and lose characters). Expect-lite adds a 50ms delay between lines when the @0 is used to prevent the overflow.Expect
Input
Buffer
Expect by default places strings that are sent back (from the remote
host) into an expect buffer ($expect_out(buffer)). Expect-lite
leverages this built in function, and uses it to perform multiple
searches through the buffer when using the '<' , '+<' or
'-<'
actions. The actions '>' and '>>' will clear
expect buffer to
prevent spurious matches.Timing is everything: delay_wait_for_host
When communicating with a remote host, enough time must be provided for the remote host to respond. There is an internal global value of $delay_wait_for_host (in mili-seconds) which can be tuned to your environment. This value is also used by the Expect Not feature to determine how long Expect-Not should wait for the unexpected.Unsynchronized messages disclaimer
Effort has been put into making the output as synchronous as possible. However because what is seen on stdout or screen is data returned/echoed from the remote host and printable comments are directly printed to stdout, it is possible to have a printable comment appear in the stdout before the action it references is returned from the remote host. This lack of synchronization can sometimes be minimized by adding an additional '>' after the printable comment as shown in the following:; === starting long command
>
>start_long_command
Instant-Interact: More than just a pretty prompt
When the user presses ^\ (control+backslash) during the running of a script, expect-lite will drop into interactive mode. In version 3.5.0, this is more than the underlying expect's standard interact mode, as Integrated Debugger (IDE) commands can be entered as well. This feature requires an additional tcl package Tclx, which is automatically detected at script runtime. If the Tclx package is not available, the instant-interact feature will be disabled, and a warning will be printed.Control will be returned to the script when '+++' is typed. In addition to returning to the script, the current command in the script will be "Passed", preventing it from failing the script. This is useful when debugging a script, the user notices that the response is not as expected, and presses ^\ to move to interact mode. Returning from interact mode, the line which would have failed the script (because of the response not being as expected) will be "Passed" and the script will continue. This Passing action is line oriented, and other '<' lines in the script can still fail the script.
There is an experimental (read: could be better tested) IDE to access to expect-lite commands or directives while the script is paused with instant-interact. In this IDE, expect-lite script commands can be typed, even assigning variables or displaying them inside the paused script, for example:
$MYVAR=today
*SHOW VARS
Var:0 Value:0
Var:MYVAR Value:today
Var:TEST Value:/proc/cpuinfo
>>pwd
pwd
/home/user
*FORK: what
happens
behind the scenes
A new *FORK session actually creates an additional login(s) to the
remote host from the localhost using the same connect method as the
first (or default) session. It then sets a sane prompt, and waits for
the first script command in that session. This provides multiple
sessions on the remote host, which then could be used to drive separate
programs (see test_fork_web_client_server.txt for example). Graphically
it would appear as:Localhost Remote Host
+----New_Session1--->Client App
+----New_Session2--->Server App
What's missing
In order to maintain an easy-to-learn, simple-to-use expect like language, looping has been intentionally kept simple and intuitive. Most expect-lite scripts will be to be top down, single pass scripts, making the scripts easy to create and read. These limitations preserve the KISS principle.If more complexity is required, consider using embedded expect inside expect-lite, or writing a native expect script.