# Testing

Our aim is that all new functionality is adequately tested. Adding tests for existing functionality is highly recommended before any major reimplementation (refactoring, etcetera).

We use py.test for (unit) testing. You need at least pytest >= 2.2.

To run tests in the current directory and its subdirectories:

    $ py.test  # runs all tests
    $ py.test storage/test_dtd.py  # runs just a single test module

We use several py.test features to simplify testing, and to suppress errors in circumstances where the tests cannot possibly succeed (limitations of tests and missing dependencies).

## Skipping tests

Pytest allows tests, test classes, and modules to be skipped or marked as “expected to fail” (xfail). Generally you should skip only if the test cannot run at all (throws uncaught exception); otherwise xfail is preferred as it provides more test coverage.

### importorskip

Use the builtin `importorskip()` function to skip a test module if a dependency cannot be imported:

    from pytest import importorskip
    importorskip("vobject")

If vobject can be imported, it will be; otherwise it raises an exception that causes pytest to skip the entire module rather than failing.

### skipif

Use the `skipif` decorator to mark tests to be skipped unless certain criteria are met. The following skips a test if the version of mymodule is too old:

    import mymodule

    @pytest.mark.skipif("mymodule.__version__ < '1.2'")
    def test_function():
        ...

You can apply this decorator to classes as well as functions and methods.

It is also possible to skip an entire test module by creating a `pytestmark` static variable in the module:

    # mark entire module as skipped for py.test if no indexer available
    pytestmark = pytest.mark.skipif("noindexer")

### xfail

Use the `xfail` decorator to mark tests as expected to fail. This allows you to do the following:

- Build tests for functionality that we haven’t implemented yet
- Mark tests that will fail on certain platforms or Python versions
- Mark tests that we should fix but haven’t got round to fixing yet

The simplest form is the following:

    from pytest import pytest.mark

    @mark.xfail
    def test_function():
        ...

You can also pass parameters to the decorator to mark expected failure only under some condition (like skipif), to document the reason failure is expected, or to actually skip the test:

    @mark.xfail("sys.version_info >= (3,0)")  # only expect failure for Python 3
    @mark.xfail(..., reason="Not implemented")  # provide a reason for the xfail
    @mark.xfail(..., run=False)  # skip the test but still regard it as xfailed

## Testing for Warnings

### deprecated_call

The builtin `deprecated_call()` function checks that a function that we run raises a DeprecationWarning:

    from pytest import deprecated_call

    def test_something():
        deprecated_call(function_to_run, arguments_for_function)

### recwarn

The recwarn plugin allows us to test for other warnings. Note that `recwarn` is a funcargs plugin, which means that you need it in your test function parameters:

    def test_example(recwarn):
        # do something
        w = recwarn.pop()
        # w.{message,category,filename,lineno}
        assert 'something' in str(w.message)

# Command Line Functional Testing

Functional tests allow us to validate the operation of the tools on the command line. The execution by a user is simulated using reference data files and the results are captured for comparison.

The tests are simple to craft and use some naming magic to make it easy to refer to test files, stdout and stderr.

## File name magic

We use a special naming convention to make writing tests quick and easy. Thus in the case of testing the following command:

    $ moz2po -t template.dtd translations.po translated.dtd

Our test would be written like this:

    $ moz2po -t $one $two $out

Where `$one` and `$two` are the input files and `$out` is the result file that the test framework will validate.

The files would be called:

File | Function | Variable | File naming conventions
--- | --- | --- | ---
test_moz2po_help.sh | Test script |  | test_${command}_${description}.sh
test_moz2po_help/one.dtd | Input | $one | ${testname}/${variable}.${extension}
test_moz2po_help/two.po | Input | $two | ${testname}/${variable}.${extension}
test_moz2po_help/out.dtd | Output | $out | ${testname}/${variable}.${extension}
test_moz2po_help/stdout.txt | Output | $stdout | ${testname}/${variable}.${extension}
test_moz2po_help/stderr.txt | Output | $stderr | ${testname}/${variable}.${extension}

Note

A test filename must start with `test_` and end in `.sh`. The rest of the name may only use ASCII alphanumeric characters and underscore `_`.

The test file is placed in the `tests/` directory while data files are placed in the `tests/data/${testname}` directory.

There are three standard output files:

1. `$out` - the output from the command
2. `$stdout` - any output given to the user
3. `$stderr` - any error output

The output files are available for checking at the end of the test execution and a test will fail if there are differences between the reference output and that achieved in the test run.

You do not need to define reference output for all three, if one is missing then checks will be against `/dev/null`.

There can be any number of input files. They need to be named using only ASCII characters without any punctuation. While you can give them any name we recommend using numbered positions such as one, two, three. These are converted into variables in the test framework so ensure that none of your choices clash with existing bash commands and variables.

Your test script can access variables for all of your files so e.g. `moz2po_conversion/one.dtd` will be referenced as `$one` and output `moz2po_conversion/out.dtd` as `$out`.

## Writing

The tests are normal bash scripts so they can be executed on their own. A template for a test is as follows:

    #!/bin/bash

    # Import the test framework
    source $(basename $0)/test.inc.sh

    # You can put any extra preperation here

    # Your actual command line to test No need for redirecting to /dev/stdout as
    # the test framework will do that automatically
    myprogram $one $two -o $out

    # Check that the results of the test match your reference resulst
    check_results  # does start_check and diff_all

    # OR do the following
    # start_checks - begin checking
    # has_stdout|has_stderr|has $file - checks that the file exists we don't care for content
    # startswith $file|startswith_stderr|startswith_stdout - the output starts with some expression
    # startswithi $file|startswithi_stderr|startswithi_stdout - case insensitive startswith
    # end_checks

For simple tests, where we diff output and do the correct checking of output files, simply use `check_results`. More complex tests need to wrap tests in `start_checks` and `end_checks`.

    start_checks
    has $out
    containsi_stdout "Parsed:"
    end_checks

You can make use of the following commands in the `start_checks` scenario:

Command | Description
--- | ---
has $file | $file was output and it not empty
has_stdout | stdout is not empty
has_stderr | stderr is not empty
startswith $file “String” | $file starts with “String”
startswithi $file “String” | $file starts with “String” ignoring case
startswith_stdout “String” | stdout starts with “String”
startswithi_stdout “String” | stdout starts with “String” ignoring case
startswith_stderr “String” | stderr starts with “String”
startswithi_stderr “String” | stderr starts with “String” ignoring case
contains $file “String” | $file contains “String”
containsi $file “String” | $file contains “String” ignoring case
contains_stdout “String” | stdout contains “String”
containsi_stdout “String” | stdout contains “String” ignoring case
contains_stderr “String” | stderr contains “String”
containsi_stderr “String” | stderr contains “String” ignoring case
endswith $file “String” | $file ends with “String”
endswithi $file “String” | $file ends with “String” ignoring case
endswith_stdout “String” | stdout ends with “String”
endswithi_stdout “String” | stdout ends with “String” ignoring case
endswith_stderr “String” | stderr ends with “String”
endswithi_stderr “String” | stderr ends with “String” ignoring case

### –prep

If you use the –prep options on any test then the test will change behavior. It won’t validate the results against your reference data but will instead create your reference data. This makes it easy to generate your expected result files when you are setting up your test.