1 

     2 :mod:`test` --- Regression tests package for Python

     3 ===================================================

     4 

     5 .. module:: test

     6    :synopsis: Regression tests package containing the testing suite for Python.

     7 .. sectionauthor:: Brett Cannon <brett@python.org>

     8 

     9 

    10 The :mod:`test` package contains all regression tests for Python as well as the

    11 modules :mod:`test.test_support` and :mod:`test.regrtest`.

    12 :mod:`test.test_support` is used to enhance your tests while

    13 :mod:`test.regrtest` drives the testing suite.

    14 

    15 Each module in the :mod:`test` package whose name starts with ``test_`` is a

    16 testing suite for a specific module or feature. All new tests should be written

    17 using the :mod:`unittest` or :mod:`doctest` module.  Some older tests are

    18 written using a "traditional" testing style that compares output printed to

    19 ``sys.stdout``; this style of test is considered deprecated.

    20 

    21 

    22 .. seealso::

    23 

    24    Module :mod:`unittest`

    25       Writing PyUnit regression tests.

    26 

    27    Module :mod:`doctest`

    28       Tests embedded in documentation strings.

    29 

    30 

    31 .. _writing-tests:

    32 

    33 Writing Unit Tests for the :mod:`test` package

    34 ----------------------------------------------

    35 

    36 It is preferred that tests that use the :mod:`unittest` module follow a few

    37 guidelines. One is to name the test module by starting it with ``test_`` and end

    38 it with the name of the module being tested. The test methods in the test module

    39 should start with ``test_`` and end with a description of what the method is

    40 testing. This is needed so that the methods are recognized by the test driver as

    41 test methods. Also, no documentation string for the method should be included. A

    42 comment (such as ``# Tests function returns only True or False``) should be used

    43 to provide documentation for test methods. This is done because documentation

    44 strings get printed out if they exist and thus what test is being run is not

    45 stated.

    46 

    47 A basic boilerplate is often used::

    48 

    49    import unittest

    50    from test import test_support

    51 

    52    class MyTestCase1(unittest.TestCase):

    53 

    54        # Only use setUp() and tearDown() if necessary

    55 

    56        def setUp(self):

    57            ... code to execute in preparation for tests ...

    58 

    59        def tearDown(self):

    60            ... code to execute to clean up after tests ...

    61 

    62        def test_feature_one(self):

    63            # Test feature one.

    64            ... testing code ...

    65 

    66        def test_feature_two(self):

    67            # Test feature two.

    68            ... testing code ...

    69 

    70        ... more test methods ...

    71 

    72    class MyTestCase2(unittest.TestCase):

    73        ... same structure as MyTestCase1 ...

    74 

    75    ... more test classes ...

    76 

    77    def test_main():

    78        test_support.run_unittest(MyTestCase1,

    79                                  MyTestCase2,

    80                                  ... list other tests ...

    81                                 )

    82 

    83    if __name__ == '__main__':

    84        test_main()

    85 

    86 This boilerplate code allows the testing suite to be run by :mod:`test.regrtest`

    87 as well as on its own as a script.

    88 

    89 The goal for regression testing is to try to break code. This leads to a few

    90 guidelines to be followed:

    91 

    92 * The testing suite should exercise all classes, functions, and constants. This

    93   includes not just the external API that is to be presented to the outside world

    94   but also "private" code.

    95 

    96 * Whitebox testing (examining the code being tested when the tests are being

    97   written) is preferred. Blackbox testing (testing only the published user

    98   interface) is not complete enough to make sure all boundary and edge cases are

    99   tested.

   100 

   101 * Make sure all possible values are tested including invalid ones. This makes

   102   sure that not only all valid values are acceptable but also that improper values

   103   are handled correctly.

   104 

   105 * Exhaust as many code paths as possible. Test where branching occurs and thus

   106   tailor input to make sure as many different paths through the code are taken.

   107 

   108 * Add an explicit test for any bugs discovered for the tested code. This will

   109   make sure that the error does not crop up again if the code is changed in the

   110   future.

   111 

   112 * Make sure to clean up after your tests (such as close and remove all temporary

   113   files).

   114 

   115 * If a test is dependent on a specific condition of the operating system then

   116   verify the condition already exists before attempting the test.

   117 

   118 * Import as few modules as possible and do it as soon as possible. This

   119   minimizes external dependencies of tests and also minimizes possible anomalous

   120   behavior from side-effects of importing a module.

   121 

   122 * Try to maximize code reuse. On occasion, tests will vary by something as small

   123   as what type of input is used. Minimize code duplication by subclassing a basic

   124   test class with a class that specifies the input::

   125 

   126      class TestFuncAcceptsSequences(unittest.TestCase):

   127 

   128          func = mySuperWhammyFunction

   129 

   130          def test_func(self):

   131              self.func(self.arg)

   132 

   133      class AcceptLists(TestFuncAcceptsSequences):

   134          arg = [1,2,3]

   135 

   136      class AcceptStrings(TestFuncAcceptsSequences):

   137          arg = 'abc'

   138 

   139      class AcceptTuples(TestFuncAcceptsSequences):

   140          arg = (1,2,3)

   141 

   142 

   143 .. seealso::

   144 

   145    Test Driven Development

   146       A book by Kent Beck on writing tests before code.

   147 

   148 

   149 .. _regrtest:

   150 

   151 Running tests using :mod:`test.regrtest`

   152 ----------------------------------------

   153 

   154 :mod:`test.regrtest` can be used as a script to drive Python's regression test

   155 suite. Running the script by itself automatically starts running all regression

   156 tests in the :mod:`test` package. It does this by finding all modules in the

   157 package whose name starts with ``test_``, importing them, and executing the

   158 function :func:`test_main` if present. The names of tests to execute may also be

   159 passed to the script. Specifying a single regression test (:program:`python

   160 regrtest.py` :option:`test_spam.py`) will minimize output and only print whether

   161 the test passed or failed and thus minimize output.

   162 

   163 Running :mod:`test.regrtest` directly allows what resources are available for

   164 tests to use to be set. You do this by using the :option:`-u` command-line

   165 option. Run :program:`python regrtest.py` :option:`-uall` to turn on all

   166 resources; specifying :option:`all` as an option for :option:`-u` enables all

   167 possible resources. If all but one resource is desired (a more common case), a

   168 comma-separated list of resources that are not desired may be listed after

   169 :option:`all`. The command :program:`python regrtest.py`

   170 :option:`-uall,-audio,-largefile` will run :mod:`test.regrtest` with all

   171 resources except the :option:`audio` and :option:`largefile` resources. For a

   172 list of all resources and more command-line options, run :program:`python

   173 regrtest.py` :option:`-h`.

   174 

   175 Some other ways to execute the regression tests depend on what platform the

   176 tests are being executed on. On Unix, you can run :program:`make` :option:`test`

   177 at the top-level directory where Python was built. On Windows, executing

   178 :program:`rt.bat` from your :file:`PCBuild` directory will run all regression

   179 tests.

   180 

   181 

   182 :mod:`test.test_support` --- Utility functions for tests

   183 ========================================================

   184 

   185 .. module:: test.test_support

   186    :synopsis: Support for Python regression tests.

   187 

   188 .. note::

   189 

   190    The :mod:`test.test_support` module has been renamed to :mod:`test.support`

   191    in Python 3.0.  The :term:`2to3` tool will automatically adapt imports when

   192    converting your sources to 3.0.

   193 

   194 

   195 

   196 

   197 The :mod:`test.test_support` module provides support for Python's regression

   198 tests.

   199 

   200 This module defines the following exceptions:

   201 

   202 

   203 .. exception:: TestFailed

   204 

   205    Exception to be raised when a test fails. This is deprecated in favor of

   206    :mod:`unittest`\ -based tests and :class:`unittest.TestCase`'s assertion

   207    methods.

   208 

   209 

   210 .. exception:: TestSkipped

   211 

   212    Subclass of :exc:`TestFailed`. Raised when a test is skipped. This occurs when a

   213    needed resource (such as a network connection) is not available at the time of

   214    testing.

   215 

   216 

   217 .. exception:: ResourceDenied

   218 

   219    Subclass of :exc:`TestSkipped`. Raised when a resource (such as a network

   220    connection) is not available. Raised by the :func:`requires` function.

   221 

   222 The :mod:`test.test_support` module defines the following constants:

   223 

   224 

   225 .. data:: verbose

   226 

   227    :const:`True` when verbose output is enabled. Should be checked when more

   228    detailed information is desired about a running test. *verbose* is set by

   229    :mod:`test.regrtest`.

   230 

   231 

   232 .. data:: have_unicode

   233 

   234    :const:`True` when Unicode support is available.

   235 

   236 

   237 .. data:: is_jython

   238 

   239    :const:`True` if the running interpreter is Jython.

   240 

   241 

   242 .. data:: TESTFN

   243 

   244    Set to the path that a temporary file may be created at. Any temporary that is

   245    created should be closed and unlinked (removed).

   246 

   247 The :mod:`test.test_support` module defines the following functions:

   248 

   249 

   250 .. function:: forget(module_name)

   251 

   252    Removes the module named *module_name* from ``sys.modules`` and deletes any

   253    byte-compiled files of the module.

   254 

   255 

   256 .. function:: is_resource_enabled(resource)

   257 

   258    Returns :const:`True` if *resource* is enabled and available. The list of

   259    available resources is only set when :mod:`test.regrtest` is executing the

   260    tests.

   261 

   262 

   263 .. function:: requires(resource[, msg])

   264 

   265    Raises :exc:`ResourceDenied` if *resource* is not available. *msg* is the

   266    argument to :exc:`ResourceDenied` if it is raised. Always returns true if called

   267    by a function whose ``__name__`` is ``'__main__'``. Used when tests are executed

   268    by :mod:`test.regrtest`.

   269 

   270 

   271 .. function:: findfile(filename)

   272 

   273    Return the path to the file named *filename*. If no match is found *filename* is

   274    returned. This does not equal a failure since it could be the path to the file.

   275 

   276 

   277 .. function:: run_unittest(*classes)

   278 

   279    Execute :class:`unittest.TestCase` subclasses passed to the function. The

   280    function scans the classes for methods starting with the prefix ``test_`` and

   281    executes the tests individually.

   282 

   283    It is also legal to pass strings as parameters; these should be keys in

   284    ``sys.modules``. Each associated module will be scanned by

   285    ``unittest.TestLoader.loadTestsFromModule()``. This is usually seen in the

   286    following :func:`test_main` function::

   287 

   288       def test_main():

   289           test_support.run_unittest(__name__)

   290 

   291    This will run all tests defined in the named module.

   292 

   293 

   294 .. function:: check_warnings()

   295 

   296    A convenience wrapper for ``warnings.catch_warnings()`` that makes

   297    it easier to test that a warning was correctly raised with a single

   298    assertion. It is approximately equivalent to calling

   299    ``warnings.catch_warnings(record=True)``.

   300 

   301    The main difference is that on entry to the context manager, a

   302    :class:`WarningRecorder` instance is returned instead of a simple list.

   303    The underlying warnings list is available via the recorder object's

   304    :attr:`warnings` attribute, while the attributes of the last raised

   305    warning are also accessible directly on the object. If no warning has

   306    been raised, then the latter attributes will all be :const:`None`.

   307 

   308    A :meth:`reset` method is also provided on the recorder object. This

   309    method simply clears the warning list.

   310 

   311    The context manager is used like this::

   312 

   313       with check_warnings() as w:

   314           warnings.simplefilter("always")

   315           warnings.warn("foo")

   316           assert str(w.message) == "foo"

   317           warnings.warn("bar")

   318           assert str(w.message) == "bar"

   319           assert str(w.warnings[0].message) == "foo"

   320           assert str(w.warnings[1].message) == "bar"

   321           w.reset()

   322           assert len(w.warnings) == 0

   323 

   324    .. versionadded:: 2.6

   325 

   326 

   327 .. function:: captured_stdout()

   328 

   329    This is a context manager than runs the :keyword:`with` statement body using

   330    a :class:`StringIO.StringIO` object as sys.stdout.  That object can be

   331    retrieved using the ``as`` clause of the :keyword:`with` statement.

   332 

   333    Example use::

   334 

   335       with captured_stdout() as s:

   336           print "hello"

   337       assert s.getvalue() == "hello"

   338 

   339    .. versionadded:: 2.6

   340 

   341 

   342 The :mod:`test.test_support` module defines the following classes:

   343 

   344 .. class:: TransientResource(exc[, **kwargs])

   345 

   346    Instances are a context manager that raises :exc:`ResourceDenied` if the

   347    specified exception type is raised.  Any keyword arguments are treated as

   348    attribute/value pairs to be compared against any exception raised within the

   349    :keyword:`with` statement.  Only if all pairs match properly against

   350    attributes on the exception is :exc:`ResourceDenied` raised.

   351 

   352    .. versionadded:: 2.6

   353 .. class:: EnvironmentVarGuard()

   354 

   355    Class used to temporarily set or unset environment variables.  Instances can be

   356    used as a context manager.

   357 

   358    .. versionadded:: 2.6

   359 

   360 

   361 .. method:: EnvironmentVarGuard.set(envvar, value)

   362 

   363    Temporarily set the environment variable ``envvar`` to the value of ``value``.

   364 

   365 

   366 .. method:: EnvironmentVarGuard.unset(envvar)

   367 

   368    Temporarily unset the environment variable ``envvar``.

   369 

   370 .. class:: WarningsRecorder()

   371 

   372    Class used to record warnings for unit tests. See documentation of

   373    :func:`check_warnings` above for more details.

   374 

   375    .. versionadded:: 2.6

   376