funcargs: advanced test setup and parametrization

what are "funcargs" and what are they good for?

Named parameters of a test function are called funcargs for short. A Funcarg can be a simple number of a complex object. To perform a test function call each parameter is setup by a factory function. To call a test function repeatedly with different funcargs sets test parameters can be generated.

funcarg factories: setting up test function arguments

Test functions can specify one ore more arguments ("funcargs") and a test module or plugin can define factory functions that provide the function argument. Let's look at a simple self-contained example that you can put into a test module:

If you run this with py.test test_simplefactory.py you see something like this:

This means that the test function was called with a myfuncarg value of 42 and the assert fails accordingly. Here is how py.test calls the test function:

  1. py.test discovers the test_function because of the test_ prefix. The test function needs a function argument named myfuncarg. A matching factory function is discovered by looking for the name pytest_funcarg__myfuncarg.
  2. pytest_funcarg__myfuncarg(request) is called and returns the value for myfuncarg.
  3. test_function(42) call is executed.

Note that if you misspell a function argument or want to use one that isn't available, you'll see an error with a list of available function arguments.

factory functions receive a request object which they can use to register setup/teardown functions or access meta data about a test.

funcarg factory request objects

Request objects are passed to funcarg factories and allow to access test configuration, test context and useful caching and finalization helpers. Here is a list of attributes:

request.function: python function object requesting the argument

request.cls: class object where the test function is defined in or None.

request.module: module object where the test function is defined in.

request.config: access to command line opts and general config

request.param: if exists was passed by a previous metafunc.addcall

managing fixtures across test modules and test runs

Calling request.cached_setup() helps you to manage fixture objects across several scopes. For example, for creating a Database object that is to be setup only once during a test session you can use the helper like this:

requesting values of other funcargs

request.getfuncargvalue(name) calls another funcarg factory function. You can use this function if you want to decorate a funcarg, i.e. you want to provide the "normal" value but add something extra. If a factory cannot be found a request.Error exception will be raised.

generating parametrized tests

You can parametrize multiple runs of the same test function by adding new test function calls with different function argument values. Let's look at a simple self-contained example:

If you run this with py.test test_example.py you'll get:

Here is what happens in detail:

  1. pytest_generate_tests(metafunc) hook is called once for each test function. It adds ten new function calls with explicit function arguments.
  2. execute tests: test_func(numiter) is called ten times with ten different arguments.

test generators and metafunc objects

metafunc objects are passed to the pytest_generate_tests hook. They help to inspect a testfunction and to generate tests according to test configuration or values specified in the class or module where a test function is defined:

metafunc.funcargnames: set of required function arguments for given function

metafunc.function: underlying python test function

metafunc.cls: class object where the test function is defined in or None.

metafunc.module: the module object where the test function is defined in.

metafunc.config: access to command line opts and general config

the metafunc.addcall() method

funcargs can be a dictionary of argument names mapped to values - providing it is called direct parametrization.

If you provide an id` it will be used for reporting and identification purposes. If you don't supply an id the stringified counter of the list of added calls will be used. id values needs to be unique between all invocations for a given test function.

param if specified will be seen by any funcarg factory as a request.param attribute. Setting it is called indirect parametrization.

Indirect parametrization is preferable if test values are expensive to setup or can only be created in certain environments. Test generators and thus addcall() invocations are performed during test collection which is separate from the actual test setup and test run phase. With distributed testing collection and test setup/run happens in different process.

Tutorial Examples

To see how you can implement custom paramtrization schemes, see e.g. parametrizing tests, generalized (blog post).

To enable creation of test support code that can flexibly register setup/teardown functions see the blog post about the monkeypatch funcarg.

If you find issues or have further suggestions for improving the mechanism you are welcome to checkout contact possibilities page.

application specific test setup and fixtures

Here is a basic useful step-wise example for handling application specific test setup. The goal is to have one place where we have the glue and test support code for bootstrapping and configuring application objects and allow test modules and test functions to stay ignorant of involved details.

step 1: use and implement a test/app-specific "mysetup"

Let's write a simple test function living in a test file test_sample.py that uses a mysetup funcarg for accessing test specific setup.

To run this test py.test needs to find and call a factory to obtain the required mysetup function argument. The test function interacts with the provided application specific setup.

To provide the mysetup function argument we write down a factory method in a local plugin by putting the following code into a local conftest.py:

To run the example we represent our application by putting a pseudo MyApp object into myapp.py:

You can now run the test with py.test test_sample.py which will show this failure:

This means that our mysetup object was successfully instantiated, we asked it to provide an application instance and checking its question method resulted in the wrong answer. If you are confused as to what the concrete question or answers actually mean, please see here :) Otherwise proceed to step 2.

step 2: adding a command line option

If you provide a "funcarg" from a plugin you can easily make methods depend on command line options or environment settings. To add a command line option we update the conftest.py of the previous example to add a command line option and to offer a new mysetup method:

Now any test function can use the mysetup.getsshconnection() method like this:

Running py.test test_ssh.py without specifying a command line option will result in a skipped test_function:

Note especially how the test function could stay clear knowing about how to construct test state values or when to skip and with what message. The test function can concentrate on actual test code and test state factories can interact with execution of tests.

If you specify a command line option like py.test --ssh=python.org the test will get un-skipped and actually execute.

example: specifying and selecting acceptance tests

and the actual test function example:

If you run this test without specifying a command line option the test will get skipped with an appropriate message. Otherwise you can start to add convenience and test support methods to your AcceptFuncarg and drive running of tools or applications and provide ways to do assertions about the output.

example: decorating a funcarg in a test module

For larger scale setups it's sometimes useful to decorare a funcarg just for a particular test module. We can extend the accept example by putting this in our test module:

Our module level factory will be invoked first and it can ask its request object to call the next factory and then decorate its result. This mechanism allows us to stay ignorant of how/where the function argument is provided - in our example from a conftest plugin.

sidenote: the temporary directory used here are instances of the py.path.local class which provides many of the os.path methods in a convenient way.