The deal.II testsuite consists of two parts, the build tests and the regression tests. While the build tests just check if the library can be compiled on different systems and with different (versions of) compilers, the regression tests are actually run and their output compared with previously stored. These two testsuites are described below.
With our build tests, we check if deal.II can be compiled on different systems and with different compilers as well as different configuration options. Results are collected in a database and can be accessed online.
Running the build test suite is simple and we encourage deal.II
users with configurations not found on the test suite page to
participate. Assuming you checked out deal.II into the directory
dealtest
, running it is as simple as (you may want to pass
different options than the ones given as BUILDTESTFLAGS
to
./configure
):
cd dealtest make clean svn update make build-test BUILDTESTFLAGS='-with-umfpack -with-lapack' mail build-tests@dealii.org < build-test-log
After cleaning up the directory and updating the library to the
most recent version, the make
command configures the
library (here for use of UMFPack and LAPACK libraries) and
compiles everything. The result is in the file
build-test-log
, which is sent to the build test demon
in the end. A status indicator should appear on the build test website
after some time (results are collected and processed by a program that is
run periodically, but not immediately after a mail has been received).
If your compiler is not in the standard path or has a strange
name (like mycc
for C and myCC
for C++),
your template would be something like this (assuming you work with the
bash
shell):
export PATH=/my/compiler/path:$PATH export CXX=myCC cd dealtest make clean svn update make build-test BUILDTESTFLAGS='CC=mycc CXX=myCC' mail build-tests@dealii.org < build-test-log
Most of our tests are built in dedicated directories, i.e. work on copies of deal.II that we don't usually use to work on every day. We do this, because we want to see if the version in the subversion repository can be compiled, not the version that we have made changes in.
In this case, one would check out a new version, and do essentially the same things as above:
export PATH=/my/compiler/path:$PATH export CXX=myCC svn co http://www.dealii.org/svn/dealii/trunk/deal.II cd deal.II svn update make build-test BUILDTEST=yes BUILDTESTFLAGS='CC=mycc CXX=myCC' mail build-tests@dealii.org < build-test-log cd .. rm -rf deal.IIThe only difference is that one has to give the option
BUILDTEST=yes
to the call to make
. This is so
because for the make file to work it depends on a file that is created by
./configure
; the latter, however, hasn't run on this pristine
copy yet.
deal.II has a testsuite that, at the time this article is written, has some 1650 small programs that we run every time we make a change to make sure that no existing functionality is broken. The expected output is also stored in our subversion archive, and when you run a test you are notified if a test fails. These days, every time we add a significant piece of functionality, we add at least one new test to the testsuite, and we also do so if we fix a bug, in both cases to make sure that future changes do not break what we have just checked in. In addition, some machines run the tests every night and send the results back home; this is then converted into a webpage showing the status of our regression tests.
If you develop parts of deal.II, want to add something, or fix a bug in it, we encourage you to use our testsuite. This page documents some aspects of it.
To run the testsuite, go into your deal.II directory and do this:
cd /path/to/deal.II svn checkout http://www.dealii.org/svn/dealii/trunk/testsThis should generate a
tests/
directory parallel to
the base/
, lac/
, etc directories. Then
do this:
cd tests ./configureThis assumes that you have previously configured your deal.II installation and sets up a few things, in particular it determines which system you are on and which compiler you are using, in order to make sure that the stored correct output for your system and compiler is used to compare against the output of the tests when you run them.
Once you have done this, you may simply type
make
. This runs all the tests there are, but stops at
the first one that fails to either execute properly or for which
the output does not match the expected output found in the subversion
archive. This is helpful if you want to figure out if any test is
failing at all. Typical output looks like this:
deal.II/tests> make cd base ; make make[1]: Entering directory `/ices/bangerth/p/deal.II/1/deal.II/tests/base' =====linking======= logtest.exe =====Running======= logtest.exe =====Checking====== logtest.output =====OK============ logtest.OK =====linking======= reference.exe =====Running======= reference.exe =====Checking====== reference.output =====OK============ reference.OK =====linking======= quadrature_test.exe ...You should be aware that because of the number of tests we have, running the entire testsuite can easily take an hour, even on a fast system. (On the other hand, only a large testsuite can offer comprehensive coverage of a software as big as deal.II.) This time can be reduced, however, on multicore machines if you use the command
make -jN
where N
is an integer equal to or
slight larger than the number of processor cores you have, as this
instructs make
to run several tests at the same time.
Sometimes, you know that for whatever reason one test
always fails on your system, or has already failed before you made
any changes to the library that could have caused tests to
fail. We also sometimes check in tests that we know presently
fail, just to remind us that we need to work on a fix, if we don't
have the time to debug the problem properly right away. In this
case, you will not want the testsuite to stop at the first test
that fails, but will want to run all tests first and then inspect
the output to find any fails. There are make targets for this
as well. The usual way we use the testsuite is to run all tests
like so
(the same applies as above: make -jN
can be used on multicore
machines):
deal.II/tests> make report | tee report =======Report: base ======= make[1]: Entering directory `/ices/bangerth/p/deal.II/1/deal.II/tests/base' 2005-03-10 21:58 + anisotropic_1 2005-03-10 21:58 + anisotropic_2 2005-03-10 21:58 + auto_derivative_function 2005-03-10 21:58 + data_out_base 2005-03-10 21:58 + hierarchical 2005-03-10 21:58 + logtest 2005-03-10 21:58 + polynomial1d 2005-03-10 21:58 + polynomial_test 2005-03-10 21:58 + quadrature_selector ...This generates a report (that we "tee" into a file called "report" and show on screen at the same time). It shows the time at which the tests was run, an indicator of success, and the name of a test. The indicator is either a plus, which means that the test compiled and linked successfully and that the output compared successfully against the stored results. Or it is a minus, indicating that the test failed, which could mean that it didn't compile, it didn't link, or the results were wrong. Since it is often hard to see visually which tests have a minus (we should have used a capital X instead), this command
grep " - " reportpicks out the lines that have a minus surrounded by spaces.
If you want to do a little more than just that, you should consider running
make report+mail | tee reportinstead. This does all the same stuff, but also mails the test result to our central mail result server which will in regular intervals (at least once a day) munge these mails and present them on our test site. This way, people can get an overview of what tests fail. You may even consider running tests nightly through a cron-job with this command, to have regular test runs.
If a test failed, you have to find out what exactly went
wrong. For this, you will want to go into the directory of that
test, and figure out in more detail what went wrong. For example,
if above test hierarchical
would have failed, you
would want to go into the base
directory (this is
given in the line with the equals signs; there are tests in other
directories as well) and then type
make hierarchical/exeto compile and link the executable. (For each test there is a not only a file with suffic
.cc
but also a subdirectory with the
same name, in which we store among other things the executable for that
test, under the name exe
.) If this fails, i.e. if
you can't compile or link, then you probably already know where
the problem is, and how to fix it. If you could compile and link
the test, you will want to make sure that it executes correctly
and produces an output file:
make hierarchical/output(As you see, the output file is also stored in the subdirectory with the test's name.) If this produces errors or triggers assertions, then you will want to use a debugger on the executable to figure out what happens. On the other hand, if you are sure that this also worked, you will want to compare the output with the stored output from subversion:
make hierarchical/OKIf the output isn't equal, then you'll see something like this:
=====Checking====== hierarchical/output +++++Error+++++++++ hierarchical/OK. Use make verbose=on for the diffsBecause the diffs between the output we get and the output we expected can sometimes be very large, you don't get to see it by default. However, following the suggestion printed, if you type
make hierarchical/OK verbose=onyou get to see it all:
=====Checking====== hierarchical/output 12c12 < DEAL::0.333 1.667 0.333 -0.889 0.296 -0.988 0.329 -0.999 0.333 -1.000 0.333 -1.000 --- > DEAL::0.333 0.667 0.333 -0.889 0.296 -0.988 0.329 -0.999 0.333 -1.000 0.333 -1.000 +++++Error+++++++++ hierarchical/OKIn this case, the second number on line 12 is off by one. To find the reason for this, you again should use a debugger or other suitable means, but that of course depends on what changes you have made last and that could have caused this discrepancy.
As mentioned above, we add a new test every time we add new functionality to the library or fix a bug. If you want to contribute code to the library, you should do this as well. Here's how: you need a testcase, a subdirectory with the same name as the test, an entry in the Makefile, and an expected output.
For the testcase, we usually start from a template like this:
//---------------------------- my_new_test.cc --------------------------- // $Id: testsuite.html 21182 2010-06-09 20:15:09Z bangerth $ // Version: $Name$ // // Copyright (C) 2005 by the deal.II authors // // This file is subject to QPL and may not be distributed // without copyright and license information. Please refer // to the file deal.II/doc/license.html for the text and // further information on this license. // //---------------------------- my_new_test.cc --------------------------- // a short (a few lines) description of what the program does #include "../tests.h" #include#include // all include files you need here int main () { std::ofstream logfile("my_new_test/output"); deallog.attach(logfile); deallog.depth_console(0); // your testcode here: int i=0; deallog << i << std::endl; return 0; }
You open an output file in a directory with the same
name as your test, and then write
all output you generate to it,
through the deallog
stream. The deallog
stream works like any
other std::ostream
except that it does a few more
things behind the scenes that are helpful in this context. In
above case, we only write a zero to the output
file. Most tests actually write computed data to the output file
to make sure that whatever we compute is what we got when the
test was first written.
There are a number of directories where you can put a new test.
Extensive tests of individual classes or groups of classes
have traditionally been into the base/
,
lac/
, deal.II/
, fe/
,
hp/
, or multigrid/
directories, depending on
where the classes that are tested are located.
We have started to create more atomic tests which
are usually very small and test only a single aspect of the
library, often only a single function. These tests go into the
bits/
directory and often have names that are
composed of the name of the class being tested and a two-digit
number, e.g., dof_tools_11
. There are
directories for PETSc and Trilinos wrapper functionality.
You have to create a subdirectory with the same name as your test to hold the output from the test.
One convenient way to create this subdirectory with the correct properties is to use svn copy.
svn copy existing_test_directory my_new_test
In order for the Makefiles to pick up your new test, you have to
add it there. In all the directories under tests/
where tests reside, there is a separate Makefile that contains a
list of the tests to be run in a variable called
tests_x
. You should add your test to the bottom of
this list, by adding the base name of your testfile, i.e., without
the extension .cc
. The entries can contain
wildcards: for example, in the tests/bits/
directory,
the tests_x
variable contains the entry
petsc_*
which at the time of this writing tests 120
different tests that all match this pattern.
If you have done this, you can try to run
make my_new_test/outputThis should compile, link, and run your test. Running your test should generate the desired output file.
If you run your new test executable, you will get an output file
mytestname/output
that should be used to compare all future
runs with. If the test
is relatively simple, it is often a good idea to look at the
output and make sure that the output is actually what you had
expected. However, if you do complex operations, this may
sometimes be impossible, and in this case we are quite happy with
any reasonable output file just to make sure that future
invokations of the test yield the same results.
The next step is to copy this output file to the place where the
scripts can find it when they compare with newer runs. For this, you first
have to understand how correct results are verified. It works in the
following way: for each test, we have subdirectories
testname/cmp
where we store the expected results in a file
testname/cmp/generic
. If you create a new test, you should
therefore create this directory, and copy the output of your program,
testname/output
to testname/cmp/generic
.
Why generic
? The reason is that sometimes test results
differ slightly from platform to platform, for example because numerical
roundoff is different due to different floating point implementations on
different CPUs. What this means is that sometimes a single stored output is
not enough to verify that a test functioned properly: if you happen to be
on a platform different from the one on which the generic output was
created, your test will always fail even though it produces almost exactly
the same output.
To avoid this, what the makefiles do is to first check whether an output
file is stored for this test and your particular configuration (platform
and compiler). If this isn't the case, it goes through a hierarchy of files
with related configurations, and only if none of them does it take the
generic output file. It then compares the output of your test run with the
first file it found in this process. To make things a bit clearer, if you
are, for example, on a i686-pc-linux-gnu
box and use
gcc4.0
as your compiler, then the following files will be
sought (in this order):
testname/cmp/i686-pc-linux-gnu+gcc4.0 testname/cmp/i686-pc-linux-gnu+gcc3.4 testname/cmp/i686-pc-linux-gnu+gcc3.3 testname/cmp/generic(This list is generated by the
tests/hierarchy.pl
script.)
Your output will then be compared with the first one that is actually
found. The virtue of this is that we don't have to store the output files
from all possible platforms (this would amount to gigabytes of data), but
that we only have store an output file for gcc4.0 if it differs from that
of gcc3.4, and for gcc3.4 if it differs from gcc3.3. If all of them are the
same, we would only have the generic output file.
Most of the time, you will be able to generate output files only
for your own platform and compiler, and that's alright: someone
else will create the output files for other platforms
eventually. You only have to copy your output file to
testname/cmp/generic
.
At this point you can run
make my_new_test/OKwhich should compare the present output with what you have just copied into the compare directory. This should, of course, succeed, since the two files should be identical.
On the other hand, if you realize that an existing test fails on your
system, but that the differences (as shown when running with
verbose=on
, see above) are only marginal and around the 6th or
8th digit, then you should check in your output file for the platform you
work on. For this, you could copy testname/output
to
testname/cmp/myplatform+compiler
, but your life can be easier
if you simply type
make my_new_test/refwhich takes your output and copies it to the right place automatically.
Tests are a way to make sure everything keeps working. If they aren't automated, they are no good. We are therefore very interested in getting new tests. If you have subversion write access already, you have to add the new test and the expected output file, and to commit them together with the changed Makefile, like so:
svn add bits/my_new_test.cc svn add bits/my_new_test svn add bits/my_new_test/cmp svn add bits/my_new_test/cmp/generic svn commit -m "New test" bits/my_new_test* bits/MakefileIn addition, you should do the following in order to avoid that the files generated while running the testsuite show up in the output of
svn
status
commands:
svn propset svn:ignore "obj.* exe output status OK" bits/my_new_test svn commit -m "Ignore generated files." bits/my_new_testNote that the list of files given in quotes to the propset command extends over several lines.
If you don't have subversion write access, talk to us on the mailing list; writing testcases is a worthy and laudable task, and we would like to encourage it by giving people the opportunity to contribute!
The deal.II mailing list