Introduction

Atheist is a simple framework for running test cases. You write small files in Python language using a set of predefined functions and classes. In many senses, the concept is quite similar to make or the SCons framework although Atheist isn’t a building system at all.

Features:

ToDo:

  • Plugins per project.
  • Limit/perf testing.
  • Remote tests.
  • Test deployment.
  • Remote management.
  • Distributed tests.

Test objects

The Test object is the minimal testing unit. Each Test instance defines an individual execution (a shell command) that may be checked for its success termination. The Test constructor accepts many parameters that may change the test exception behavior in several ways. The only mandatory parameter is cmd that is the command to exec.

The Test object is responsible to execute the command and check its termination value. A very basic example:

Test('true')

Test files

The Test instances need to be defined into text source files (with extension .test). Although these files are written in a subset of Python language, they may be seen as declarative programs. You tell Atheist what you want to test and even the order, but the decision about when to run the corresponding action is taken by Atheist; some of them may be never done.

The file does not define sequential imperative sentences. For example, if you write this in a .test file:

Test('ls /')
print "hello"

the print statement will be executed BEFORE the ls command.

Key-args

Any Test constructor accepts the next key-val parameters. All of them are optional. Beside the parameter name appears its type and default value.

check – type: bool, default: True

If ‘check’ is False there is no consequences when the Task fails and it is not considered in the final stats.

cwd – type: str

Directory for the task execution.

delay – type: int, default: 0

Waits for ‘delay’ seconds before launch the task actions.

desc – type: str

One-liner textual task description.

detach – type: bool, default: False

When detach is False the next task does not starts until the current one ends. When detach is True the next task is executed even the current one is running.

env – type: str:str map

A dictionary with shell environment variables.

expected – type: int

Expected program return code. It may be negative if the process is killed by a signal.

tid – type: str

It is a unique string Task IDentifier. You can get a reference to the task later giving this value to the get_task() function.

must_fail – type: bool, default: False

When you expect the program terminates with error but the return code is not known (i.e: is different that zero). You should check other conditions (stdout, stderr, generated files, etc) to differentiate alternative fails.

parent – type: CompositeTask

Use this to aggregate current test in a already defined CompositeTask.

template – type: Template list

A list of templates. See Templates.

timeout – type: int, default: 5

The maximum task execution time (in seconds). When the timeout finish, Atheist sends the programmed signal to the process. To avoid timed termination (daemonic task) give timeout=0.

save_stderr – type: bool, default: False

Store the process stderr in an actual file. If the stderr parameter is not set, Atheist will create an unique name for the file.

save_stdout – type: bool, default: False

Store the process stdout in an actual file. If the stdout parameter is not set, Atheist will create an unique name for the file.

shell – type: bool, default: False

Execute the command within a shell session. bash is the used shell.

signal – type: int, default: SIGKILL

It is the signal that Atheist sends to the process when the timeout finish.

stdout – type: str

Is the file name to save the process stdout. Setting this parameters implies save_stdout = True.

todo – type: bool, default: False

It indicates the task is not fully verified and it is possible it can fail unduly. This has not effect when the task ends successfully.

Not all of these key-args are available for all Task classes. See Task’s, Test’s, Commands, Daemons....

Task results

The execution of any task return a value:

  • FAIL: The task ran normally but the user requirements or conditions did not achieved. The test failed.
  • OK: The task ran successfully and all required conditions and/or return value was right.
  • NOEXEC: The task was skipped or it was not executed.
  • ERROR: The task implementation is wrong and task execution failed itself.
  • UNKNOWN: The task was executed but the its result is not known.
  • TODO: The task implementation is unstable and it may produce false failures.

Templates

The template is a set of predefined values for the Test key-values. You may use the same configuration for many tests avoiding repeat them. This is an example:

t1 = Template(timeout=6, expected=-9)
Test('foo', templates=[t1])
Test('bar', templates=[t1])

Both tests will be automatically killed after 6 seconds and the expected return value is -9. That means these process receive the SIGKILL(9) signal. You may specify several templates as a list.

Conditions

The conditions are predicates (actually functors) that check for specific checks. Conditions may be specified to be checked before (pre-conditions) or after (post-conditions) the task execution. If any of the condition fails then the task fails. This is an example:

t = Test('foo')
t.pre  += FileExists('/path/to/foofile')
t.post += FileContains('path/to/barfile', 'some text')

Available builtin conditions

AtheistVersion(version)
Checks the installed atheist version is equal or newer than the given number. This is useful to assure recent or experimental features.
Callback(*args)
Call the specified function with the given arguments. You must avoid the use of this condition as possible.
DebPkgInstalled(package)
Checks the Debian package is installed.
DirExists(path)
Checks the directory path exists.
EnvVarDefined(name[, value])
Checks the environment variable name exists, and optionally has the value value.
FileExists(filename)
Checks the directory filename exists.
FileContains(val[, filename=task.stdout, times=1])
Checks the file filename exists and contains val, that may be a string or a string list. The file must contain at least times ocurrences of val. Default value for filename is the stdout of the corresponding task, and it implies save_stdout=True. This implies also a FileExists condition for that file.
FileEquals(filename1[, filename2=task.stdout])
Checks the contents of filename1 and filename2 are identical. Default value for filename2 is the stdout of the current task, and it implies save_stdout=True.
OpenPort(port[, host='localhost'[, proto='tcp']])
Checks the port is open, that is: a process is listen in it.
OutContains(val)
Checks the task output contains val, that may be a string or a string list.
ProcessRunning(pid)
Checks the given PID belongs to a running process.

Condition decorators

Not(condition)

It is True when condition is evaluated as False.

Example:

t = Test('rm foo_file')
t.pre += Not(FileExists(foo_file))
Poll(condition[, interval=1[, timeout=5]])

Checks condition each interval seconds stopping after timeout seconds or its value become True.

In the next example, the task waits for the nc server become ready before continue (a maximum of 5 seconds):

t = Test('nc -l -p 2000')
t.post += Poll(OpenPort(2000))
Or(condition1, condition2, ...)
It is the boolean operator “or”.

Condition decorators may be combined. Next example shows a task that waits for a environment variable is removed before execute the command:

t = Test('command')
t.pre += Poll(Not(EnvVarDefined('WAIT_FLAG')), timeout=10)

Note the effect of Poll(Not(condition)) is not the same of Not(Poll(condition)).

Task’s, Test’s, Commands, Daemons...

Task is the base class for all executable items. Test is-a Task that run a shell command but other kind of Task are possible:

Command

It is a non-checked Test. Command is exactly the same than a Test with parameter check=False.

The Commands (or other non-checked tests) are not considered in results counting.

Daemon

Command shortcut for detached commands. Predefined parameters are:

  • detach = True
  • expected = -9 (sigkilled)
  • timeout = None (runs “forerver”)
  • check = False
UnitTestCase
Wrapper to add standard unitest.TestCase in atheist testcases.
DocTest
Wrapper to add standard doctest in atheist testcases.
TestFunc
Check return value of arbitrary Python functions of methods. Return value 0 means success, other case means a code error. For unit testing prefer UnitTestCase instead of this.

Task detaching and termination

TaskTerminator

It’s a special test to kill and ensure the termination of other tasks.

Next example runs a netcat server during 10 seconds and then kill it:

nc = Daemon("nc -l -p 2000")
TaskTerminator(nc, delay=10)

Function utilities

get_task(name)
Returns the task which tid attribute is name. [include a sample here]

Variable substitutions

The testfiles may include some substitutable variable. That is useful to locate related files relevant in the test. You must write the symbol ‘$’ preceding each one of the next words:

basedir
is the name of the directory from atheist was executed. Usually this is a tests directory into your project.
dirname
is the name of the directory where the testfile is.
fname
is the path to the testfile without its extension (.test).
testname
is just the name of the testfile, without extension and directory path.

For example, for next vars.test file:

Test("echo $basedir $dirname $fname $testname")

When you run atheist, you get:

~/sample$ atheist -i2 tests/vars.test
[ OK ] Test case: ./tests/substitutions.test
[ OK ] +- T-1   ( 0: 0)  echo . ./tests ./tests/vars vars
Total:  ALL OK!!  - 0.24s - 1 test

setup and teardown

You may write tasks to execute before and after each test file. To do this just put this tasks in files called _setup.test and _teardown.test.

Clean-up

When your task create files you may track them for automatic cleaning. Just add filenames to the task gen attribute. Here’s an example:

t = Test('touch foo')
t.gen += 'foo'

You may specify one o more filenames (a string list).

If you want the generated files don’t be automatically removed for manual inspection of results, you must specify the --dirty option (see below). To clean-up these files later, specify -C option.

Invoking Atheist

-h, --help
Show basic help information.
-a TASK_ARGS, --task-args=TASK_ARGS
Colon-separated options for the tests.
-b BASE_DIR, --base-dir=BASE_DIR
Change working directory.
-C, --clean-only
Don’t execute nothing, only remove generated files.
--cols=SCREEN_WIDTH
Set terminal width (in chars).
-d, --describe
Describe tasks but does not execute nothing.
--dirty
Don’t remove generated files after test execution.
-e, --stderr
Print test process stderr.
-f, --stdout-on-fail
print task output but only if it fail.
-g, --gen-template
Generate a test file template with default values.
-i REPORT_DETAIL, --report-detail=REPORT_DETAIL
Report verbosity (0:nothing, [1:case], 2:task, 3:composite, 4:condition)
-j, --skip-setup
Skip _setup and _teardown files.
-k, --keep-going
Continue even with failed tests.
-l, --list
List tests but do not execute them.
--log=LOG
List tasks but do not execute them.
--notify-jabber=JABBER
Notify failed tests to the given jabber account (may be given several times).
--notify-smtp=MAIL
Notify failed tests to the given email address (may be given several times).
-r RANDOM, --random=RANDOM
Run testcases in random order using the specified seed.
-o, --stdout
Print test process stdout.
-p PLUGINPATH, --plugin_dir=PLUGINPATH

Load plugins on that directory (may be given several times).

Print test process stdout.

-q, --quiet
Do not show result summary nor warnings, only totals.
-s INLINE, --script=INLINE
Specifies command line script.
-t, --time-tag
Include time info in the logs.
-u, --until-failure
Repeat test until something fails.
-v, --verbose
Increase verbosity.
-w WORKERS, --workers=WORKERS
Number of simultaneous tasks. Value ‘0’ allows atheist choose number. Default is 1.
-x EXT, --extension=EXT
Extension for the test files.

Logging control

[ToDo]

Result report

[ToDo]

Notifications (other reporters)

You may use Atheist to monitor the well working of any application. It may send you notifications when some is wrong. The easiest way is to run a testcase with cron specifying one --notify-* command-line argument. At the moment, two notificators are implemented:

Email

The destination is a email account using the SMTP protocol. Atheist does not require a SMTP server or smarthost. You only need to configure an email account that will be used by Atheist to send mail to any destination. That information must be written in the .atheist configuration file. This is an example:

[smtp]
host = smtp.server.org
port = 587
user = atheist.notify@server.org
pasw = somesecret
Jabber

You must indicate a jabber account that will be used by Atheist to send notification messages to any other jabber account. The destination account must accept this contact previously. The next is an example for the configuration in the .atheist file:

[jabber]
user = atheist.notify@server.org
pasw = anothersecret

To ask for a notification you just need to specify a test file and one or more destinations. For example:

$ atheist --notify-jabber John.Doe@jabber.info test/some_test.test

It is possible to give several –notify-* arguments in the same command-line to send notifications to different destinations.