Atheist is a simple framework for running test cases. You write small files in Python language using a set of predefined functions and classes. In many senses, the concept is quite similar to make or the SCons framework although Atheist isn’t a building system at all.
Features:
ToDo:
The Test object is the minimal testing unit. Each Test instance defines an individual execution (a shell command) that may be checked for its success termination. The Test constructor accepts many parameters that may change the test exception behavior in several ways. The only mandatory parameter is cmd that is the command to exec.
The Test object is responsible to execute the command and check its termination value. A very basic example:
Test('true')
The Test instances need to be defined into text source files (with extension .test). Although these files are written in a subset of Python language, they may be seen as declarative programs. You tell Atheist what you want to test and even the order, but the decision about when to run the corresponding action is taken by Atheist; some of them may be never done.
The file does not define sequential imperative sentences. For example, if you write this in a .test file:
Test('ls /')
print "hello"
the print statement will be executed BEFORE the ls command.
Any Test constructor accepts the next key-val parameters. All of them are optional. Beside the parameter name appears its type and default value.
check – type: bool, default: True
If ‘check’ is False there is no consequences when the Task fails and it is not considered in the final stats.
cwd – type: str
Directory for the task execution.
delay – type: int, default: 0
Waits for ‘delay’ seconds before launch the task actions.
desc – type: str
One-liner textual task description.
detach – type: bool, default: False
When detach is False the next task does not starts until the current one ends. When detach is True the next task is executed even the current one is running.
env – type: str:str map
A dictionary with shell environment variables.
expected – type: int
Expected program return code. It may be negative if the process is killed by a signal.
tid – type: str
It is a unique string Task IDentifier. You can get a reference to the task later giving this value to the get_task() function.
must_fail – type: bool, default: False
When you expect the program terminates with error but the return code is not known (i.e: is different that zero). You should check other conditions (stdout, stderr, generated files, etc) to differentiate alternative fails.
parent – type: CompositeTask
Use this to aggregate current test in a already defined CompositeTask.
template – type: Template list
A list of templates. See Templates.
timeout – type: int, default: 5
The maximum task execution time (in seconds). When the timeout finish, Atheist sends the programmed signal to the process. To avoid timed termination (daemonic task) give timeout=0.
save_stderr – type: bool, default: False
Store the process stderr in an actual file. If the stderr parameter is not set, Atheist will create an unique name for the file.
save_stdout – type: bool, default: False
Store the process stdout in an actual file. If the stdout parameter is not set, Atheist will create an unique name for the file.
shell – type: bool, default: False
Execute the command within a shell session. bash is the used shell.
signal – type: int, default: SIGKILL
It is the signal that Atheist sends to the process when the timeout finish.
stdout – type: str
Is the file name to save the process stdout. Setting this parameters implies save_stdout = True.
todo – type: bool, default: False
It indicates the task is not fully verified and it is possible it can fail unduly. This has not effect when the task ends successfully.
Not all of these key-args are available for all Task classes. See Task’s, Test’s, Commands, Daemons....
The execution of any task return a value:
The template is a set of predefined values for the Test key-values. You may use the same configuration for many tests avoiding repeat them. This is an example:
t1 = Template(timeout=6, expected=-9)
Test('foo', templates=[t1])
Test('bar', templates=[t1])
Both tests will be automatically killed after 6 seconds and the expected return value is -9. That means these process receive the SIGKILL(9) signal. You may specify several templates as a list.
The conditions are predicates (actually functors) that check for specific checks. Conditions may be specified to be checked before (pre-conditions) or after (post-conditions) the task execution. If any of the condition fails then the task fails. This is an example:
t = Test('foo')
t.pre += FileExists('/path/to/foofile')
t.post += FileContains('path/to/barfile', 'some text')
It is True when condition is evaluated as False.
Example:
t = Test('rm foo_file')
t.pre += Not(FileExists(foo_file))
Checks condition each interval seconds stopping after timeout seconds or its value become True.
In the next example, the task waits for the nc server become ready before continue (a maximum of 5 seconds):
t = Test('nc -l -p 2000')
t.post += Poll(OpenPort(2000))
Condition decorators may be combined. Next example shows a task that waits for a environment variable is removed before execute the command:
t = Test('command')
t.pre += Poll(Not(EnvVarDefined('WAIT_FLAG')), timeout=10)
Note the effect of Poll(Not(condition)) is not the same of Not(Poll(condition)).
Task is the base class for all executable items. Test is-a Task that run a shell command but other kind of Task are possible:
It is a non-checked Test. Command is exactly the same than a Test with parameter check=False.
The Commands (or other non-checked tests) are not considered in results counting.
Command shortcut for detached commands. Predefined parameters are:
It’s a special test to kill and ensure the termination of other tasks.
Next example runs a netcat server during 10 seconds and then kill it:
nc = Daemon("nc -l -p 2000")
TaskTerminator(nc, delay=10)
The testfiles may include some substitutable variable. That is useful to locate related files relevant in the test. You must write the symbol ‘$’ preceding each one of the next words:
For example, for next vars.test file:
Test("echo $basedir $dirname $fname $testname")
When you run atheist, you get:
~/sample$ atheist -i2 tests/vars.test
[ OK ] Test case: ./tests/substitutions.test
[ OK ] +- T-1 ( 0: 0) echo . ./tests ./tests/vars vars
Total: ALL OK!! - 0.24s - 1 test
You may write tasks to execute before and after each test file. To do this just put this tasks in files called _setup.test and _teardown.test.
When your task create files you may track them for automatic cleaning. Just add filenames to the task gen attribute. Here’s an example:
t = Test('touch foo')
t.gen += 'foo'
You may specify one o more filenames (a string list).
If you want the generated files don’t be automatically removed for manual inspection of results, you must specify the --dirty option (see below). To clean-up these files later, specify -C option.
Load plugins on that directory (may be given several times).
Print test process stdout.
[ToDo]
[ToDo]
You may use Atheist to monitor the well working of any application. It may send you notifications when some is wrong. The easiest way is to run a testcase with cron specifying one --notify-* command-line argument. At the moment, two notificators are implemented:
The destination is a email account using the SMTP protocol. Atheist does not require a SMTP server or smarthost. You only need to configure an email account that will be used by Atheist to send mail to any destination. That information must be written in the .atheist configuration file. This is an example:
[smtp]
host = smtp.server.org
port = 587
user = atheist.notify@server.org
pasw = somesecret
You must indicate a jabber account that will be used by Atheist to send notification messages to any other jabber account. The destination account must accept this contact previously. The next is an example for the configuration in the .atheist file:
[jabber]
user = atheist.notify@server.org
pasw = anothersecret
To ask for a notification you just need to specify a test file and one or more destinations. For example:
$ atheist --notify-jabber John.Doe@jabber.info test/some_test.test
It is possible to give several –notify-* arguments in the same command-line to send notifications to different destinations.