The deal.II Testsuite

The deal.II testsuite consists of two parts, the build tests and the regression tests. While the build tests just check if the library can be compiled on different systems and with different (versions of) compilers, the regression tests are actually run and their output compared with previously stored. These two testsuites are described below.

The build tests

With our build tests, we check if deal.II can be compiled on different systems and with different compilers as well as different configuration options. Results are collected in a database and can be accessed online.

Running the build test suite is simple and we encourage deal.II users with configurations not found on the test suite page to participate. Assuming you checked out deal.II into the directory dealtest, running it is as simple as (you may want to pass different options than the ones given as BUILDTESTFLAGS to ./configure):


    cd dealtest
    make clean
    svn update
    make build-test BUILDTESTFLAGS='-with-umfpack -with-lapack'
    mail build-tests@dealii.org < build-test-log

    

After cleaning up the directory and updating the library to the most recent version, the make command configures the library (here for use of UMFPack and LAPACK libraries) and compiles everything. The result is in the file build-test-log, which is sent to the build test demon in the end. A status indicator should appear on the build test website after some time (results are collected and processed by a program that is run periodically, but not immediately after a mail has been received).

Configuring for a special compiler

If your compiler is not in the standard path or has a strange name (like mycc for C and myCC for C++), your template would be something like this (assuming you work with the bash shell):


    export PATH=/my/compiler/path:$PATH
    export CXX=myCC

    cd dealtest
    make clean
    svn update
    make build-test BUILDTESTFLAGS='CC=mycc CXX=myCC'
    mail build-tests@dealii.org < build-test-log

    

Dedicated tests

Most of our tests are built in dedicated directories, i.e. work on copies of deal.II that we don't usually use to work on every day. We do this, because we want to see if the version in the subversion repository can be compiled, not the version that we have made changes in.

In this case, one would check out a new version, and do essentially the same things as above:


    export PATH=/my/compiler/path:$PATH
    export CXX=myCC

    svn co http://www.dealii.org/svn/dealii/trunk/deal.II
    cd deal.II
    svn update
    make build-test BUILDTEST=yes BUILDTESTFLAGS='CC=mycc CXX=myCC'
    mail build-tests@dealii.org < build-test-log
    cd ..
    rm -rf deal.II

    
The only difference is that one has to give the option BUILDTEST=yes to the call to make. This is so because for the make file to work it depends on a file that is created by ./configure; the latter, however, hasn't run on this pristine copy yet.

The regression tests

deal.II has a testsuite that, at the time this article is written, has some 1650 small programs that we run every time we make a change to make sure that no existing functionality is broken. The expected output is also stored in our subversion archive, and when you run a test you are notified if a test fails. These days, every time we add a significant piece of functionality, we add at least one new test to the testsuite, and we also do so if we fix a bug, in both cases to make sure that future changes do not break what we have just checked in. In addition, some machines run the tests every night and send the results back home; this is then converted into a webpage showing the status of our regression tests.

If you develop parts of deal.II, want to add something, or fix a bug in it, we encourage you to use our testsuite. This page documents some aspects of it.

Running it

To run the testsuite, go into your deal.II directory and do this:

       cd /path/to/deal.II
       svn checkout http://www.dealii.org/svn/dealii/trunk/tests
    
This should generate a tests/ directory parallel to the base/, lac/, etc directories. Then do this:
       cd tests
       ./configure
    
This assumes that you have previously configured your deal.II installation and sets up a few things, in particular it determines which system you are on and which compiler you are using, in order to make sure that the stored correct output for your system and compiler is used to compare against the output of the tests when you run them.

Once you have done this, you may simply type make. This runs all the tests there are, but stops at the first one that fails to either execute properly or for which the output does not match the expected output found in the subversion archive. This is helpful if you want to figure out if any test is failing at all. Typical output looks like this:

      deal.II/tests> make
      cd base ; make
      make[1]: Entering directory `/ices/bangerth/p/deal.II/1/deal.II/tests/base'
      =====linking======= logtest.exe
      =====Running======= logtest.exe
      =====Checking====== logtest.output
      =====OK============ logtest.OK
      =====linking======= reference.exe
      =====Running======= reference.exe
      =====Checking====== reference.output
      =====OK============ reference.OK
      =====linking======= quadrature_test.exe
      ...
    
You should be aware that because of the number of tests we have, running the entire testsuite can easily take an hour, even on a fast system. (On the other hand, only a large testsuite can offer comprehensive coverage of a software as big as deal.II.) This time can be reduced, however, on multicore machines if you use the command make -jN where N is an integer equal to or slight larger than the number of processor cores you have, as this instructs make to run several tests at the same time.

Sometimes, you know that for whatever reason one test always fails on your system, or has already failed before you made any changes to the library that could have caused tests to fail. We also sometimes check in tests that we know presently fail, just to remind us that we need to work on a fix, if we don't have the time to debug the problem properly right away. In this case, you will not want the testsuite to stop at the first test that fails, but will want to run all tests first and then inspect the output to find any fails. There are make targets for this as well. The usual way we use the testsuite is to run all tests like so (the same applies as above: make -jN can be used on multicore machines):

      deal.II/tests> make report | tee report
      =======Report: base =======
      make[1]: Entering directory `/ices/bangerth/p/deal.II/1/deal.II/tests/base'
      2005-03-10 21:58 + anisotropic_1
      2005-03-10 21:58 + anisotropic_2
      2005-03-10 21:58 + auto_derivative_function
      2005-03-10 21:58 + data_out_base
      2005-03-10 21:58 + hierarchical
      2005-03-10 21:58 + logtest
      2005-03-10 21:58 + polynomial1d
      2005-03-10 21:58 + polynomial_test
      2005-03-10 21:58 + quadrature_selector
      ...
    
This generates a report (that we "tee" into a file called "report" and show on screen at the same time). It shows the time at which the tests was run, an indicator of success, and the name of a test. The indicator is either a plus, which means that the test compiled and linked successfully and that the output compared successfully against the stored results. Or it is a minus, indicating that the test failed, which could mean that it didn't compile, it didn't link, or the results were wrong. Since it is often hard to see visually which tests have a minus (we should have used a capital X instead), this command
      grep " - " report
    
picks out the lines that have a minus surrounded by spaces.

If you want to do a little more than just that, you should consider running

      make report+mail | tee report
    
instead. This does all the same stuff, but also mails the test result to our central mail result server which will in regular intervals (at least once a day) munge these mails and present them on our test site. This way, people can get an overview of what tests fail. You may even consider running tests nightly through a cron-job with this command, to have regular test runs.

If a test failed, you have to find out what exactly went wrong. For this, you will want to go into the directory of that test, and figure out in more detail what went wrong. For example, if above test hierarchical would have failed, you would want to go into the base directory (this is given in the line with the equals signs; there are tests in other directories as well) and then type

      make hierarchical/exe
    
to compile and link the executable. (For each test there is a not only a file with suffic .cc but also a subdirectory with the same name, in which we store among other things the executable for that test, under the name exe.) If this fails, i.e. if you can't compile or link, then you probably already know where the problem is, and how to fix it. If you could compile and link the test, you will want to make sure that it executes correctly and produces an output file:
      make hierarchical/output
    
(As you see, the output file is also stored in the subdirectory with the test's name.) If this produces errors or triggers assertions, then you will want to use a debugger on the executable to figure out what happens. On the other hand, if you are sure that this also worked, you will want to compare the output with the stored output from subversion:
      make hierarchical/OK
    
If the output isn't equal, then you'll see something like this:
      =====Checking====== hierarchical/output
      +++++Error+++++++++ hierarchical/OK. Use make verbose=on for the diffs
    
Because the diffs between the output we get and the output we expected can sometimes be very large, you don't get to see it by default. However, following the suggestion printed, if you type
      make hierarchical/OK verbose=on
    
you get to see it all:
      =====Checking====== hierarchical/output
      12c12
      < DEAL::0.333 1.667 0.333 -0.889 0.296 -0.988 0.329 -0.999 0.333 -1.000 0.333 -1.000
      ---
      > DEAL::0.333 0.667 0.333 -0.889 0.296 -0.988 0.329 -0.999 0.333 -1.000 0.333 -1.000
      +++++Error+++++++++ hierarchical/OK
    
In this case, the second number on line 12 is off by one. To find the reason for this, you again should use a debugger or other suitable means, but that of course depends on what changes you have made last and that could have caused this discrepancy.

Adding new tests

As mentioned above, we add a new test every time we add new functionality to the library or fix a bug. If you want to contribute code to the library, you should do this as well. Here's how: you need a testcase, a subdirectory with the same name as the test, an entry in the Makefile, and an expected output.

The testcase

For the testcase, we usually start from a template like this:

//----------------------------  my_new_test.cc  ---------------------------
//    $Id: testsuite.html 21182 2010-06-09 20:15:09Z bangerth $
//    Version: $Name$ 
//
//    Copyright (C) 2005 by the deal.II authors 
//
//    This file is subject to QPL and may not be  distributed
//    without copyright and license information. Please refer
//    to the file deal.II/doc/license.html for the  text  and
//    further information on this license.
//
//----------------------------  my_new_test.cc  ---------------------------


// a short (a few lines) description of what the program does

#include "../tests.h"
#include 
#include 

// all include files you need here


int main () 
{
  std::ofstream logfile("my_new_test/output");
  deallog.attach(logfile);
  deallog.depth_console(0);

  // your testcode here:
  int i=0;
  deallog << i << std::endl;

  return 0;
}
    

You open an output file in a directory with the same name as your test, and then write all output you generate to it, through the deallog stream. The deallog stream works like any other std::ostream except that it does a few more things behind the scenes that are helpful in this context. In above case, we only write a zero to the output file. Most tests actually write computed data to the output file to make sure that whatever we compute is what we got when the test was first written.

There are a number of directories where you can put a new test. Extensive tests of individual classes or groups of classes have traditionally been into the base/, lac/, deal.II/, fe/, hp/, or multigrid/ directories, depending on where the classes that are tested are located.

We have started to create more atomic tests which are usually very small and test only a single aspect of the library, often only a single function. These tests go into the bits/ directory and often have names that are composed of the name of the class being tested and a two-digit number, e.g., dof_tools_11. There are directories for PETSc and Trilinos wrapper functionality.

A directory with the same name as the test

You have to create a subdirectory with the same name as your test to hold the output from the test.

One convenient way to create this subdirectory with the correct properties is to use svn copy.

    svn copy existing_test_directory my_new_test
    

An entry in the Makefile

In order for the Makefiles to pick up your new test, you have to add it there. In all the directories under tests/ where tests reside, there is a separate Makefile that contains a list of the tests to be run in a variable called tests_x. You should add your test to the bottom of this list, by adding the base name of your testfile, i.e., without the extension .cc. The entries can contain wildcards: for example, in the tests/bits/ directory, the tests_x variable contains the entry petsc_* which at the time of this writing tests 120 different tests that all match this pattern.

If you have done this, you can try to run

      make my_new_test/output
    
This should compile, link, and run your test. Running your test should generate the desired output file.

An expected output

If you run your new test executable, you will get an output file mytestname/output that should be used to compare all future runs with. If the test is relatively simple, it is often a good idea to look at the output and make sure that the output is actually what you had expected. However, if you do complex operations, this may sometimes be impossible, and in this case we are quite happy with any reasonable output file just to make sure that future invokations of the test yield the same results.

The next step is to copy this output file to the place where the scripts can find it when they compare with newer runs. For this, you first have to understand how correct results are verified. It works in the following way: for each test, we have subdirectories testname/cmp where we store the expected results in a file testname/cmp/generic. If you create a new test, you should therefore create this directory, and copy the output of your program, testname/output to testname/cmp/generic.

Why generic? The reason is that sometimes test results differ slightly from platform to platform, for example because numerical roundoff is different due to different floating point implementations on different CPUs. What this means is that sometimes a single stored output is not enough to verify that a test functioned properly: if you happen to be on a platform different from the one on which the generic output was created, your test will always fail even though it produces almost exactly the same output.

To avoid this, what the makefiles do is to first check whether an output file is stored for this test and your particular configuration (platform and compiler). If this isn't the case, it goes through a hierarchy of files with related configurations, and only if none of them does it take the generic output file. It then compares the output of your test run with the first file it found in this process. To make things a bit clearer, if you are, for example, on a i686-pc-linux-gnu box and use gcc4.0 as your compiler, then the following files will be sought (in this order):

testname/cmp/i686-pc-linux-gnu+gcc4.0 
testname/cmp/i686-pc-linux-gnu+gcc3.4 
testname/cmp/i686-pc-linux-gnu+gcc3.3
testname/cmp/generic
    
(This list is generated by the tests/hierarchy.pl script.) Your output will then be compared with the first one that is actually found. The virtue of this is that we don't have to store the output files from all possible platforms (this would amount to gigabytes of data), but that we only have store an output file for gcc4.0 if it differs from that of gcc3.4, and for gcc3.4 if it differs from gcc3.3. If all of them are the same, we would only have the generic output file.

Most of the time, you will be able to generate output files only for your own platform and compiler, and that's alright: someone else will create the output files for other platforms eventually. You only have to copy your output file to testname/cmp/generic.

At this point you can run

      make my_new_test/OK
    
which should compare the present output with what you have just copied into the compare directory. This should, of course, succeed, since the two files should be identical.

On the other hand, if you realize that an existing test fails on your system, but that the differences (as shown when running with verbose=on, see above) are only marginal and around the 6th or 8th digit, then you should check in your output file for the platform you work on. For this, you could copy testname/output to testname/cmp/myplatform+compiler, but your life can be easier if you simply type

      make my_new_test/ref
    
which takes your output and copies it to the right place automatically.

Checking in

Tests are a way to make sure everything keeps working. If they aren't automated, they are no good. We are therefore very interested in getting new tests. If you have subversion write access already, you have to add the new test and the expected output file, and to commit them together with the changed Makefile, like so:

      svn add bits/my_new_test.cc 
      svn add bits/my_new_test
      svn add bits/my_new_test/cmp
      svn add bits/my_new_test/cmp/generic
      svn commit -m "New test" bits/my_new_test* bits/Makefile
    
In addition, you should do the following in order to avoid that the files generated while running the testsuite show up in the output of svn status commands:
      svn propset svn:ignore "obj.*
        exe
        output
        status
        OK" bits/my_new_test
      svn commit -m "Ignore generated files." bits/my_new_test
    
Note that the list of files given in quotes to the propset command extends over several lines.

If you don't have subversion write access, talk to us on the mailing list; writing testcases is a worthy and laudable task, and we would like to encourage it by giving people the opportunity to contribute!

The deal.II mailing list

Valid HTML 4.01!