All QMTest functionality is available using the qmtest command.
qmtest [ option ...] command [ command-option ...] [ argument ...]
These options can be used with any QMTest command, and must precede the command name on the command line.
All options are available in a "long form" prefixed with "--" (two hyphens). Some options also may be specified in a "short form" consisting of a single hyphen and a one-letter abbreviation. Short-form options may be combined; for example, -abc is equivalent to -a -b -c.
-D
path, --tdb
pathUse the test database located in the directory given by
path. This flag overrides the value of
the environment variable QMTEST_DB_PATH
. If
neither this flag nor the environment variable is specified,
QMTest assumes that the current directory should be used as the
database. See Section 3.1.6.
-h
, --help
Display help information, listing commands and general options for the qmtest command.
--version
Describe the version of QMTest in use.
Additional options are available for specific commands; these
are presented with each command. Options specific to a command must
follow the command on the command line. Specify the
--help
(-h
) option after the
command for a description of the command and a list of of available
options for that command.
Create a new extension instance.
qmtest create [ option ...] kind descriptor
The qmtest create creates a new extension instance. For example, this command can be used to create a new test or resource. For a list of the kinds of extensions supported by QMTest, run qmtest extensions. The kind must be one of these extension kinds.
The descriptor specifies an extension class and (optionally) attributes for that extension class. The form of the descriptor is class(attributes), where the attributes are of the form attr = "val". If there are no attributes, the parentheses may be omitted.
The class may be either the path to an extension file or a QMTest class name in the form module.class. If the class is the path to an extension file (such as an existing test or resource file), the class name used is the one provided in the file; otherwise the class named used is the name provided on the command line.
The attributes used to construct the extension instance come
from three sources: the attributes in the extension file (if the
class is the path to an extension
file), the --attribute
options provided
on the command line, and the explicit attributes provided in the
descriptor. If multiple values for the the same attribute name
are provided, the value used is taken from the first source in the
following list for which there is a value: the rightmost attribute
provided in the descriptor, the extension file, or the rightmost
--attribute
present on the command
line.
The new extension file is written to the file specified with
the --output
option, or to the standard
output if no --output
is specified.
The create command accepts these options:
-a
name=value, --attribute
name=valueSet the target class argument name to value. The set of valid argument names and valid values is dependent on the extension class in use.
-o
file, --output
fileWrite a description of the extension instance to file.
This command:
qmtest create -a format=stats -o rs result_stream text_result_stream.TextResultStream(filename="rs")creates a file called rs containing an instance of
TextResultStream
.
Create a new target.
qmtest create-target [ option ...] name class [ group ]
The qmtest create-target command creates a new target. A target is an entity that runs tests; normally, a target corresponds to a particular machine.
The target's name and class must be specified. An optional group may also be specified. When QMTest decides which target to use to run a particular tests, it will select a target that matches the test's requested target group.
The create-target command accepts these options:
-a
name=value, --attribute
name=valueSet the target class argument name to value. The set of valid argument names and valid values is dependent on the target class in use.
-T
file, --targets
fileWrite the target description to the indicated file. If there are already targets listed in file, they will be preserved, except that any target with the same name as the new target will be removed. If this option is not present, the file used will be the QMTest/targets file in the test database directory.
Create a new test database.
qmtest create-tdb [ option ...]
The qmtest create-tdb command creates a new, empty test database. A test database is a directory in which QMTest stores configuration files, tests, and other data. Certain test database classes may also store data elsewhere, such as in an external relational database.
The test database is created in the directory specified by
--tdb
(-D
) option or by
setting the QMTEST_DB_PATH
environment variable.
If no database path is specified, QMTest assumes that the
current directory is the test database.
By default, QMTest creates a new test database that uses the standard XML-based implementation. (See Section 4.5 for information about writing a test database class.)
The create-tdb command accepts these options:
-a
name=value, --attribute
name=valueSet the database attribute name to value. The set of attribute names and valid values is dependent on the database class in use. The default database class accepts no attributes.
-c
class, --class
classUse the test database class given by class. The class may have the general form described in Section 3.4.2. Once you create a test database, you cannot change the test database implementation it uses. If you do not use this option, QMTest will use the default test database implementation, which uses an XML file format to store tests.
Start the graphical user interface.
qmtest gui [ option ...]
The qmtest gui starts the graphical user interface. The graphical user interface is accessed through a web browser. You must have a web browser that supports JavaScript to use the graphical interface. QMTest has been tested with recent versions of Internet Explorer and Netscape Navigator. Other web browsers may or may nor work with QMTest.
The gui command accepts these options:
-A
address, --address
addressBind the server to the indicated internet address, which should be a dotted quad. By default, the server binds itself to the address 127.0.0.1, which is the address of the local machine. If you specify another address, the server will be accessible to users on other machines. QMTest does not perform any authentication of remote users, so you should not use this option unless you have a firewall in place that blocks all untrusted users.
-c
name=value, --context
name=valueFor details about this option, see the description of the qmtest run command.
-C
file, --load-context
fileFor details about this option, see the description of the qmtest run command.
--daemon
Run the QMTest GUI as a daemon. In this mode, QMTest will detach from the controlling terminal and run in the background until explicitly shutdown.
-j
count, --concurrency
countFor details about this option, see the description of the qmtest run command.
--no-browser
Do not attempt to start a web browser when starting the GUI. QMTest will still print out the URL at which the server can be accessed. You can then connect to this URL manually using the browser of your choice.
-O
file, --outcomes
fileFor details about this option, see the description of the qmtest run command.
--pid-file path
Specify the path to which the QMTest GUI will write its process ID. This option is useful if you want to run QMTest as a daemon. If this option is not provided, no PID file is written. If you specify this option, but path is the empty string, QMTest will check the .qmrc configuration file for a pid-file entry. If there is no such entry, QMTest will use an appropriate platform-specific default value.
--port port
Specify the port on which the QMTest GUI will listen for connections. If this option is not provided, QMTest will select an available port automatically.
-T
file, --targets
fileFor details about this option, see the description of the qmtest run command.
List available extension classes.
qmtest extensions [ option ...]
The qmtest extensions lists available extension classes and provides a brief description of each class. You can use this command to list all of the available extension classes, or to list all of the available extension classes of a particular type. For example, you can use this command to list all of the available test classes.
The extensions command accepts these options:
-k
kind, --kind
kindList the available extension classes of the indicated kind. The kind must be one of test, resource, target, or database.
Register an extension class.
qmtest register kind class-name
The qmtest register registers an extension class with QMTest. As part of this process, QMTest will load your extension class. If the extension class cannot be loaded, QMTest will tell you what went wrong.
QMTest will search for your extension class in the
directories it would search when running tests, including those
given by the environment variable
QMTEST_CLASS_PATH
.
The kind argument tells QMTest what kind of extension class you are registering. If you invoke qmtest register with no arguments it will provide you with a list of the available extension kinds.
The class-name argument gives the
name of the class in the form
module.Class
. QMTest will look for a file
whose basename is the module name and whose extension is either
py, pyc, or
pyo.
Run tests or test suites.
qmtest run [ option ...] [test-name | suite-name]...
The qmtest run command runs tests and displays the results. If no test or suite names are specified, QMTest runs all of the tests in the test database. If test or suite names are specified, only those tests or suites are run. Tests listed more than once (directly or by inclusion in a test suite) are run only once.
The run command accepts these options:
-c
name=value, --context
name=valueAdd a property to the test execution context. The name of the property is name, and its value is set to the string value.
This option may be specified multiple times.
-C
file, --load-context
fileRead properties for the test execution context from the file file.
The file should be a text file with one context property on each line, in the format name=value. Leading and trailing whitespace on each line are ignored. Also, blank lines and lines that begin with "#" (a hash mark) are ignored as comments.
This option may be specified more than once, and used in
conjunction with the --context
option. All of the context properties specified are added to
the eventual context. If a property is set more than once,
the last value provided is the one used.
If this option is not specified, but a file named
context exists in the current directory,
that file is read. The properties specified in this file are
processed first; the values in this file can be overridden by
subsequent uses of the --context
option on the command line.
-f
format, --format
formatControl the format used when displaying results. The
format specified must be one of full,
brief, stats,
batch, or none. The
brief format is the default if QMTest was
invoked interactively; the batch format is
the default otherwise. In the full format,
QMTest displays any annotations provided in test results. In
the brief mode only the causes of failures
are shown; detailed annotations are not shown. In the
stats format, no details about failing tests
are displayed; only statistics showing the number of passing
and failing tests are displayed. In the
batch mode, the summary is displayed first,
followed by detailed results for tests with unexpected
outcomes. In the none mode, no results are
displayed, but a results file is still created, unless the
--no-output
option is also
provided.
-j
count, --concurrency
countRun tests in multiple count concurrent processes on the local computer. On multiprocessor machines, the processes may be scheduled to run in parallel on different processors. QMTest automatically collects results from the processes and presents combines test results and summary. By default, one process is used.
This option may not be combined with the --targets
(-T
) option.
--no-output
Do not produce a test results file.
-o
file, --output
fileWrite full test results to file.
Specify "-" (a hyphen) to write results to
standard output. If neither this option nor
--no-output
is specified, the results
are written to the file named results.qmr
in the current directory.
-O
file, --outcomes
fileTreat file as a set of expected outcomes. The file must have be a results file created either by qmtest run, or by saving results in the graphical user interface. QMTest will expect the results of the current test run to match those specified in the file and will highlight differences from those results.
--random
Run the tests in a random order.
This option can be used to find hidden dependencies between tests in the testsuite. (You may not notice the dependencies if you always run the tests in the same order.)
--rerun
fileRerun only those tests that had unexpected outcomes.
The tests run are determined as follows. QMTest starts
with all of the tests specified on the command line, or, if no
tests are explicitly specified, all of the tests in the
database. If no expectations file is specified (see the
description of the --outcomes
option),
then all tests that passed in the results file indicated by the
--rerun
option are removed form the
set of eligible tests. If an expectations file is specified,
then the tests removed are tests whose outcome in the results
file indicated by the --rerun
option
is the same as in the expectations file.
The --rerun
provides a simple
way of rerunning failing tests. If you run your tests and
notice failures, you might try to fix those failing tests.
Then, you can rerun the failing tests to see if you succeeded
by using the --rerun
option.
--result-stream descriptor
Specify an additional output result stream. The descriptor is in the format described in Section 3.4.2.
--seed
integerIf the --random
is used, QMTest
randomizes the order in which tests are run, subject to the
constraints described in Section 3.2.1. By default, the
random number generator is seeded using the system time.
For debugging purposes, it is sometimes necessary to obtain
a reproducible sequence of tests. Use the
--seed
option to specify the seed for the
random number generator.
Note that even with the same random number seed, if tests are run in parallel, scheduling uncertainty may still produce variation in the order in which tests are run.
-T
file, --targets
fileUse targets specified in target specification file file. If this option is not present, the QMTest/targets in the test database directory will be used. If that file is not present, the tests will be run in serial on the local machine.
The qmtest summarize displays information stored in a results file.
qmtest summarize [ option ...] [test-name | suite-name]...
The qmtest summarize extracts information stored in a results file and displays this information on the console. The information is formatted just as if the tests had just been run, but QMTest does not actually run the tests.
The summarize command accepts the following options:
-f
format, --format
formatFor details about this option, see the description of the qmtest run command.
-O
file, --outcomes
fileFor details about this option, see the description of the qmtest run command.
--result-stream descriptor
Specify an additional output result stream. The descriptor is in the format described in Section 3.4.2.
QMTest recognizes the following environment variables:
QMTEST_CLASS_PATH
If this environment variable is set, it should contain a
list of directories in the same format as used for the system's
PATH
environment variable. These directories
are searched (before the directories that QMTest searches by
default) when looking for extension classes such as test classes
and database classes.
QMTEST_DB_PATH
If this environment variable is set, its value is used as
the location of the test database, unless the
--tdb
(-D
) option is
used. If this environment variable is not set and the
--tdb
option is not used, the current
directory is used as the test database.
These configuration variables are used by QMTest. You should define them in the [qmtest] section of your QM configuration file.
The default path to use when creating a PID file with the
--pid-file
option. (See Section 3.4.5 for more information about this
option.) If this entry is not present, an appropriate
platform-specific default value is used.
If QMTest successfully performed the action requested,
QMTest returns 0. For the qmtest run or
qmtest summarize commands, success implies
not only that the tests ran, but also that all of the tests passed
(if the --outcomes
option was not used) or
had their expected outcomes (if the
--outcomes
option was used).
If either the run command or the
summarize command was used, and at least one
test failed (if the --outcomes
option was
not used) or had an unexpected outcome (if the
--outcomes
option was used), qmtest
returns 1.
If QMTest could not perform the action requested, qmtest returns 2.