[Ericsson AB]

1 Common Test Basics

1.1 Introduction

The Common Test framework (CT) is a tool which can support implementation and automated execution of test cases towards different types of target systems. The framework is based on the OTP Test Server. Test cases can be run individually or in batches. Common Test also features a distributed testing mode with central control and logging. This feature makes it possible to test multiple systems independently in one common session. This can be very useful e.g. for running automated large-scale regression tests.

The SUT (system under test) may consist of one or several target nodes. CT contains a generic test server which together with other test utilities is used to perform test case execution. It is possible to start the tests from the CT GUI or from an OS- or Erlang shell prompt. Test suites are files (Erlang modules) that contain the test cases (Erlang functions) to be executed. Support modules provide functions that the test cases utilize in order to carry out the tests.

The main idea is that CT based test programs connect to the target system(s) via standard O&M interfaces. CT provides implementations and wrappers of some of these O&M interfaces and will be extended with more interfaces later. There are a number of target independent interfaces supported in CT such as Generic Telnet, FTP etc. which can be specialized or used directly for controlling instruments, traffic generators etc.

For white-box testing of Erlang code, the test programs can of course call Erlang API functions directly. Black-box testing of Erlang code can use Erlang RPC as well as standard O&M interfaces if desired.

A test case can handle several connections towards one or several target systems, instruments and traffic generators in parallel in order to perform the necessary actions for a test. The handling of many connections in parallel is one of the major strengths of CT!

1.2 Test Suite Organisation

The test suites are organized in test directories and each test suite may have a separate data directory. Typically, these files and directories are version controlled similarly to other forms of source code (possibly by means of some CVS-like tool). However, CT does not itself put any requirements on (or has any form of awareness of) possible file and directory versions.

1.3 Support Libraries

Support libraries are functions that are useful for all test suites or for test suites in a specific functional area or subsystem. So as well as the general support libraries provided by the CT framework there might be a need for customized support libraries on a subsystem level.

1.4 Scripting Suite and Test Cases

Testing is performed by running test suites (a set of test cases) or individual test cases. A test suite is implemented as an Erlang module named <suite_name>_SUITE.erl which contains a number of test cases. A test case is an Erlang function which tests one or more things. The test case is the smallest unit that the CT test server deals with.

The test suite file must conform to a certain interface which is specified by the CT test server. See the section on writing test suites for more information.

A test case is considered successful if it returns to the caller, no matter what the returned value is. A few return values have special meaning however (such as {skip,Reason} which indicates that the test case is skipped, {comment,Comment} which prints a comment in the log for the test case and {save_config,Config} which makes the CT test server pass Config to the next test case). A test case failure is specified as a runtime error (a crash), no matter what the reason for termination is. If you use Erlang pattern matching effectively, you can take advantage of this property. The result will be concise and readable test case functions that look much more like scripts than actual programs. Simple example:

      session(_Config) ->
          {started,ServerId} = my_server:start(),
          {clients,[]} = my_server:get_clients(ServerId),
          MyId = self(),
          connected = my_server:connect(ServerId, MyId),
          {clients,[MyId]} = my_server:get_clients(ServerId),
          disconnected = my_server:disconnect(ServerId, MyId),
          {clients,[]} = my_server:get_clients(ServerId),
          stopped = my_server:stop(ServerId).
    

As a test suite runs, all information (including output to stdout) is recorded in several different log files. A minimum of information is displayed in the user console (only start and stop information, plus a note for each failed test case).

The result from each test case is recorded in an HTML log file which is created for the particular test run. Each test case is represented by a row in a table that shows the total execution time, whether the case was successful or if it failed or was skipped, plus a possible user comment. For a failed test case, the reason for termination is printed. The HTML file has a link to the logfile for each test case (which may also be viewed with an HTML browser).

1.5 External Interfaces

The CT test server requires some default functions in a test suite. Each suite module should define and export the following functions (most are however optional):

all()
Returns a list of all test cases in the suite. (Mandatory)
suite()
Default suite configuration. (Optional)
groups()
For declaring test case groups. (Optional)
init_per_suite(Config)
Executed before the first test case in a suite. (Optional)
end_per_suite(Config)
Executed after the last test case in a suite. (Optional)
init_per_group(Config)
Executed before the first test case in a group. (Optional)
end_per_group(Config)
Executed after the last test case in a group. (Optional)
init_per_testcase(TC, Config)
Executed before each test case. (Optional)
end_per_testcase(TC, Config)
Executed after each test case. (Optional)

For each test case the CT test server needs these functions:

Testcasename()
Returns a key-value list of test case configuration/information. (Optional)
Testcasename(Config)
The actual test case function.

common_test 1.4.5
Copyright © 1991-2009 Ericsson AB