| :mod:`test` --- Regression tests package for Python |
| =================================================== |
| |
| .. module:: test |
| :synopsis: Regression tests package containing the testing suite for Python. |
| .. sectionauthor:: Brett Cannon <brett@python.org> |
| |
| |
| The :mod:`test` package contains all regression tests for Python as well as the |
| modules :mod:`test.support` and :mod:`test.regrtest`. |
| :mod:`test.support` is used to enhance your tests while |
| :mod:`test.regrtest` drives the testing suite. |
| |
| Each module in the :mod:`test` package whose name starts with ``test_`` is a |
| testing suite for a specific module or feature. All new tests should be written |
| using the :mod:`unittest` or :mod:`doctest` module. Some older tests are |
| written using a "traditional" testing style that compares output printed to |
| ``sys.stdout``; this style of test is considered deprecated. |
| |
| |
| .. seealso:: |
| |
| Module :mod:`unittest` |
| Writing PyUnit regression tests. |
| |
| Module :mod:`doctest` |
| Tests embedded in documentation strings. |
| |
| |
| .. _writing-tests: |
| |
| Writing Unit Tests for the :mod:`test` package |
| ---------------------------------------------- |
| |
| It is preferred that tests that use the :mod:`unittest` module follow a few |
| guidelines. One is to name the test module by starting it with ``test_`` and end |
| it with the name of the module being tested. The test methods in the test module |
| should start with ``test_`` and end with a description of what the method is |
| testing. This is needed so that the methods are recognized by the test driver as |
| test methods. Also, no documentation string for the method should be included. A |
| comment (such as ``# Tests function returns only True or False``) should be used |
| to provide documentation for test methods. This is done because documentation |
| strings get printed out if they exist and thus what test is being run is not |
| stated. |
| |
| A basic boilerplate is often used:: |
| |
| import unittest |
| from test import support |
| |
| class MyTestCase1(unittest.TestCase): |
| |
| # Only use setUp() and tearDown() if necessary |
| |
| def setUp(self): |
| ... code to execute in preparation for tests ... |
| |
| def tearDown(self): |
| ... code to execute to clean up after tests ... |
| |
| def test_feature_one(self): |
| # Test feature one. |
| ... testing code ... |
| |
| def test_feature_two(self): |
| # Test feature two. |
| ... testing code ... |
| |
| ... more test methods ... |
| |
| class MyTestCase2(unittest.TestCase): |
| ... same structure as MyTestCase1 ... |
| |
| ... more test classes ... |
| |
| def test_main(): |
| support.run_unittest(MyTestCase1, |
| MyTestCase2, |
| ... list other tests ... |
| ) |
| |
| if __name__ == '__main__': |
| test_main() |
| |
| This boilerplate code allows the testing suite to be run by :mod:`test.regrtest` |
| as well as on its own as a script. |
| |
| The goal for regression testing is to try to break code. This leads to a few |
| guidelines to be followed: |
| |
| * The testing suite should exercise all classes, functions, and constants. This |
| includes not just the external API that is to be presented to the outside |
| world but also "private" code. |
| |
| * Whitebox testing (examining the code being tested when the tests are being |
| written) is preferred. Blackbox testing (testing only the published user |
| interface) is not complete enough to make sure all boundary and edge cases |
| are tested. |
| |
| * Make sure all possible values are tested including invalid ones. This makes |
| sure that not only all valid values are acceptable but also that improper |
| values are handled correctly. |
| |
| * Exhaust as many code paths as possible. Test where branching occurs and thus |
| tailor input to make sure as many different paths through the code are taken. |
| |
| * Add an explicit test for any bugs discovered for the tested code. This will |
| make sure that the error does not crop up again if the code is changed in the |
| future. |
| |
| * Make sure to clean up after your tests (such as close and remove all temporary |
| files). |
| |
| * If a test is dependent on a specific condition of the operating system then |
| verify the condition already exists before attempting the test. |
| |
| * Import as few modules as possible and do it as soon as possible. This |
| minimizes external dependencies of tests and also minimizes possible anomalous |
| behavior from side-effects of importing a module. |
| |
| * Try to maximize code reuse. On occasion, tests will vary by something as small |
| as what type of input is used. Minimize code duplication by subclassing a |
| basic test class with a class that specifies the input:: |
| |
| class TestFuncAcceptsSequences(unittest.TestCase): |
| |
| func = mySuperWhammyFunction |
| |
| def test_func(self): |
| self.func(self.arg) |
| |
| class AcceptLists(TestFuncAcceptsSequences): |
| arg = [1, 2, 3] |
| |
| class AcceptStrings(TestFuncAcceptsSequences): |
| arg = 'abc' |
| |
| class AcceptTuples(TestFuncAcceptsSequences): |
| arg = (1, 2, 3) |
| |
| |
| .. seealso:: |
| |
| Test Driven Development |
| A book by Kent Beck on writing tests before code. |
| |
| |
| .. _regrtest: |
| |
| Running tests using :mod:`test.regrtest` |
| ---------------------------------------- |
| |
| :mod:`test.regrtest` can be used as a script to drive Python's regression test |
| suite. Running the script by itself automatically starts running all regression |
| tests in the :mod:`test` package. It does this by finding all modules in the |
| package whose name starts with ``test_``, importing them, and executing the |
| function :func:`test_main` if present. The names of tests to execute may also |
| be passed to the script. Specifying a single regression test (:program:`python |
| regrtest.py` :option:`test_spam.py`) will minimize output and only print |
| whether the test passed or failed and thus minimize output. |
| |
| Running :mod:`test.regrtest` directly allows what resources are available for |
| tests to use to be set. You do this by using the :option:`-u` command-line |
| option. Run :program:`python regrtest.py` :option:`-uall` to turn on all |
| resources; specifying :option:`all` as an option for :option:`-u` enables all |
| possible resources. If all but one resource is desired (a more common case), a |
| comma-separated list of resources that are not desired may be listed after |
| :option:`all`. The command :program:`python regrtest.py` |
| :option:`-uall,-audio,-largefile` will run :mod:`test.regrtest` with all |
| resources except the :option:`audio` and :option:`largefile` resources. For a |
| list of all resources and more command-line options, run :program:`python |
| regrtest.py` :option:`-h`. |
| |
| Some other ways to execute the regression tests depend on what platform the |
| tests are being executed on. On Unix, you can run :program:`make` |
| :option:`test` at the top-level directory where Python was built. On Windows, |
| executing :program:`rt.bat` from your :file:`PCBuild` directory will run all |
| regression tests. |
| |
| |
| :mod:`test.support` --- Utility functions for tests |
| =================================================== |
| |
| .. module:: test.support |
| :synopsis: Support for Python regression tests. |
| |
| |
| The :mod:`test.support` module provides support for Python's regression |
| tests. |
| |
| This module defines the following exceptions: |
| |
| |
| .. exception:: TestFailed |
| |
| Exception to be raised when a test fails. This is deprecated in favor of |
| :mod:`unittest`\ -based tests and :class:`unittest.TestCase`'s assertion |
| methods. |
| |
| |
| .. exception:: ResourceDenied |
| |
| Subclass of :exc:`unittest.SkipTest`. Raised when a resource (such as a |
| network connection) is not available. Raised by the :func:`requires` |
| function. |
| |
| The :mod:`test.support` module defines the following constants: |
| |
| |
| .. data:: verbose |
| |
| :const:`True` when verbose output is enabled. Should be checked when more |
| detailed information is desired about a running test. *verbose* is set by |
| :mod:`test.regrtest`. |
| |
| |
| .. data:: is_jython |
| |
| :const:`True` if the running interpreter is Jython. |
| |
| |
| .. data:: TESTFN |
| |
| Set to the name that a temporary file could use. Any temporary file that is |
| created should be closed and unlinked (removed). |
| |
| The :mod:`test.support` module defines the following functions: |
| |
| |
| .. function:: forget(module_name) |
| |
| Remove the module named *module_name* from ``sys.modules`` and deletes any |
| byte-compiled files of the module. |
| |
| |
| .. function:: is_resource_enabled(resource) |
| |
| Return :const:`True` if *resource* is enabled and available. The list of |
| available resources is only set when :mod:`test.regrtest` is executing the |
| tests. |
| |
| |
| .. function:: requires(resource, msg=None) |
| |
| Raise :exc:`ResourceDenied` if *resource* is not available. *msg* is the |
| argument to :exc:`ResourceDenied` if it is raised. Always returns |
| :const:`True` if called by a function whose ``__name__`` is ``'__main__'``. |
| Used when tests are executed by :mod:`test.regrtest`. |
| |
| |
| .. function:: findfile(filename) |
| |
| Return the path to the file named *filename*. If no match is found |
| *filename* is returned. This does not equal a failure since it could be the |
| path to the file. |
| |
| |
| .. function:: run_unittest(*classes) |
| |
| Execute :class:`unittest.TestCase` subclasses passed to the function. The |
| function scans the classes for methods starting with the prefix ``test_`` |
| and executes the tests individually. |
| |
| It is also legal to pass strings as parameters; these should be keys in |
| ``sys.modules``. Each associated module will be scanned by |
| ``unittest.TestLoader.loadTestsFromModule()``. This is usually seen in the |
| following :func:`test_main` function:: |
| |
| def test_main(): |
| support.run_unittest(__name__) |
| |
| This will run all tests defined in the named module. |
| |
| |
| .. function:: check_warnings(*filters, quiet=None) |
| |
| A convenience wrapper for ``warnings.catch_warnings()`` that makes |
| it easier to test that a warning was correctly raised with a single |
| assertion. It is approximately equivalent to calling |
| ``warnings.catch_warnings(record=True)``. |
| |
| It accepts 2-tuples ``("message regexp", WarningCategory)`` as positional |
| arguments. If there's some ``*filters`` defined, or if the optional keyword |
| argument ``quiet`` is :const:`False`, it checks if the warnings are |
| effective. If some filter did not catch any warning, the test fails. If some |
| warnings are not caught, the test fails, too. To disable these checks, set |
| argument ``quiet`` to :const:`True`. |
| |
| Without argument, it defaults to:: |
| |
| check_warnings(("", Warning), quiet=True) |
| |
| Additionally, on entry to the context manager, a :class:`WarningRecorder` |
| instance is returned. The underlying warnings list is available via the |
| recorder object's :attr:`warnings` attribute, while the attributes of the |
| last raised warning are also accessible directly on the object. If no |
| warning has been raised, then the latter attributes will all be |
| :const:`None`. |
| |
| A :meth:`reset` method is also provided on the recorder object. This |
| method simply clears the warnings list. |
| |
| The context manager may be used like this:: |
| |
| import warnings |
| |
| with check_warnings(quiet=False): |
| exec('assert(False, "Hey!")') |
| warnings.warn(UserWarning("Hide me!")) |
| |
| with check_warnings(("assertion is always true", SyntaxWarning), |
| ("", UserWarning)): |
| exec('assert(False, "Hey!")') |
| warnings.warn(UserWarning("Hide me!")) |
| |
| with check_warnings(quiet=True) as w: |
| warnings.simplefilter("always") |
| warnings.warn("foo") |
| assert str(w.args[0]) == "foo" |
| warnings.warn("bar") |
| assert str(w.args[0]) == "bar" |
| assert str(w.warnings[0].args[0]) == "foo" |
| assert str(w.warnings[1].args[0]) == "bar" |
| w.reset() |
| assert len(w.warnings) == 0 |
| |
| .. versionchanged:: 2.7 |
| New optional attributes ``*filters`` and ``quiet``. |
| |
| |
| .. function:: captured_stdout() |
| |
| This is a context manager than runs the :keyword:`with` statement body using |
| a :class:`StringIO.StringIO` object as sys.stdout. That object can be |
| retrieved using the ``as`` clause of the :keyword:`with` statement. |
| |
| Example use:: |
| |
| with captured_stdout() as s: |
| print("hello") |
| assert s.getvalue() == "hello" |
| |
| |
| .. function:: import_module(name, deprecated=False) |
| |
| This function imports and returns the named module. Unlike a normal |
| import, this function raises :exc:`unittest.SkipTest` if the module |
| cannot be imported. |
| |
| Module and package deprecation messages are suppressed during this import |
| if *deprecated* is :const:`True`. |
| |
| .. versionadded:: 3.1 |
| |
| |
| .. function:: import_fresh_module(name, fresh=(), blocked=(), deprecated=False) |
| |
| This function imports and returns a fresh copy of the named Python module |
| by removing the named module from ``sys.modules`` before doing the import. |
| Note that unlike :func:`reload`, the original module is not affected by |
| this operation. |
| |
| *fresh* is an iterable of additional module names that are also removed |
| from the ``sys.modules`` cache before doing the import. |
| |
| *blocked* is an iterable of module names that are replaced with :const:`0` |
| in the module cache during the import to ensure that attempts to import |
| them raise :exc:`ImportError`. |
| |
| The named module and any modules named in the *fresh* and *blocked* |
| parameters are saved before starting the import and then reinserted into |
| ``sys.modules`` when the fresh import is complete. |
| |
| Module and package deprecation messages are suppressed during this import |
| if *deprecated* is :const:`True`. |
| |
| This function will raise :exc:`unittest.SkipTest` is the named module |
| cannot be imported. |
| |
| Example use:: |
| |
| # Get copies of the warnings module for testing without |
| # affecting the version being used by the rest of the test suite |
| # One copy uses the C implementation, the other is forced to use |
| # the pure Python fallback implementation |
| py_warnings = import_fresh_module('warnings', blocked=['_warnings']) |
| c_warnings = import_fresh_module('warnings', fresh=['_warnings']) |
| |
| .. versionadded:: 3.1 |
| |
| |
| The :mod:`test.support` module defines the following classes: |
| |
| .. class:: TransientResource(exc, **kwargs) |
| |
| Instances are a context manager that raises :exc:`ResourceDenied` if the |
| specified exception type is raised. Any keyword arguments are treated as |
| attribute/value pairs to be compared against any exception raised within the |
| :keyword:`with` statement. Only if all pairs match properly against |
| attributes on the exception is :exc:`ResourceDenied` raised. |
| |
| |
| .. class:: EnvironmentVarGuard() |
| |
| Class used to temporarily set or unset environment variables. Instances can |
| be used as a context manager and have a complete dictionary interface for |
| querying/modifying the underlying ``os.environ``. After exit from the |
| context manager all changes to environment variables done through this |
| instance will be rolled back. |
| |
| .. versionchanged:: 3.1 |
| Added dictionary interface. |
| |
| .. method:: EnvironmentVarGuard.set(envvar, value) |
| |
| Temporarily set the environment variable ``envvar`` to the value of |
| ``value``. |
| |
| |
| .. method:: EnvironmentVarGuard.unset(envvar) |
| |
| Temporarily unset the environment variable ``envvar``. |
| |
| |
| .. class:: WarningsRecorder() |
| |
| Class used to record warnings for unit tests. See documentation of |
| :func:`check_warnings` above for more details. |