Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 1 | \section{\module{test} --- |
| 2 | Regression tests package for Python} |
| 3 | |
| 4 | \declaremodule{standard}{test} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 5 | \sectionauthor{Brett Cannon}{brett@python.org} |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 6 | \modulesynopsis{Regression tests package containing the testing suite |
| 7 | for Python.} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 8 | |
| 9 | |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 10 | The \module{test} package contains all regression tests for Python as |
| 11 | well as the modules \module{test.test_support} and |
| 12 | \module{test.regrtest}. \module{test.test_support} is used to enhance |
| 13 | your tests while \module{test.regrtest} drives the testing suite. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 14 | |
| 15 | Each module in the \module{test} package whose name starts with |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 16 | \samp{test_} is a testing suite for a specific module or feature. |
| 17 | All new tests should be written using the \refmodule{unittest} module; |
| 18 | using \refmodule{unittest} is not required but makes the tests more |
| 19 | flexible and maintenance of the tests easier. Some older tests are |
| 20 | written to use \refmodule{doctest} and a ``traditional'' testing |
| 21 | style; these styles of tests will not be covered. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 22 | |
| 23 | \begin{seealso} |
| 24 | \seemodule{unittest}{Writing PyUnit regression tests.} |
| 25 | \seemodule{doctest}{Tests embedded in documentation strings.} |
| 26 | \end{seealso} |
| 27 | |
| 28 | |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 29 | \subsection{Writing Unit Tests for the \module{test} package% |
| 30 | \label{writing-tests}} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 31 | |
| 32 | It is preferred that tests for the \module{test} package use the |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 33 | \refmodule{unittest} module and follow a few guidelines. |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 34 | One is to have the name of all the test methods start with \samp{test_} as |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 35 | well as the module's name. |
| 36 | This is needed so that the methods are recognized by the test driver as |
| 37 | test methods. |
| 38 | Also, no documentation string for the method should be included. |
| 39 | A comment (such as |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 40 | \samp{\# Tests function returns only True or False}) should be used to provide |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 41 | documentation for test methods. |
| 42 | This is done because documentation strings get printed out if they exist and |
| 43 | thus what test is being run is not stated. |
| 44 | |
| 45 | A basic boilerplate is often used: |
| 46 | |
| 47 | \begin{verbatim} |
| 48 | import unittest |
| 49 | from test import test_support |
| 50 | |
| 51 | class MyTestCase1(unittest.TestCase): |
| 52 | |
| 53 | # Only use setUp() and tearDown() if necessary |
| 54 | |
| 55 | def setUp(self): |
| 56 | ... code to execute in preparation for tests ... |
| 57 | |
| 58 | def tearDown(self): |
| 59 | ... code to execute to clean up after tests ... |
| 60 | |
| 61 | def test_feature_one(self): |
| 62 | # Test feature one. |
| 63 | ... testing code ... |
| 64 | |
| 65 | def test_feature_two(self): |
| 66 | # Test feature two. |
| 67 | ... testing code ... |
| 68 | |
| 69 | ... more test methods ... |
| 70 | |
| 71 | class MyTestCase2(unittest.TestCase): |
| 72 | ... same structure as MyTestCase1 ... |
| 73 | |
| 74 | ... more test classes ... |
| 75 | |
| 76 | def test_main(): |
| 77 | test_support.run_unittest(MyTestCase1, |
| 78 | MyTestCase2, |
| 79 | ... list other tests ... |
| 80 | ) |
| 81 | |
| 82 | if __name__ == '__main__': |
| 83 | test_main() |
| 84 | \end{verbatim} |
| 85 | |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 86 | This boilerplate code allows the testing suite to be run by |
| 87 | \module{test.regrtest} as well as on its own as a script. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 88 | |
| 89 | The goal for regression testing is to try to break code. |
| 90 | This leads to a few guidelines to be followed: |
| 91 | |
| 92 | \begin{itemize} |
| 93 | \item The testing suite should exercise all classes, functions, and |
| 94 | constants. |
| 95 | This includes not just the external API that is to be presented to the |
| 96 | outside world but also "private" code. |
| 97 | \item Whitebox testing (examining the code being tested when the tests are |
| 98 | being written) is preferred. |
| 99 | Blackbox testing (testing only the published user interface) is not |
| 100 | complete enough to make sure all boundary and edge cases are tested. |
| 101 | \item Make sure all possible values are tested including invalid ones. |
| 102 | This makes sure that not only all valid values are acceptable but also |
| 103 | that improper values are handled correctly. |
| 104 | \item Exhaust as many code paths as possible. |
| 105 | Test where branching occurs and thus tailor input to make sure as many |
| 106 | different paths through the code are taken. |
| 107 | \item Add an explicit test for any bugs discovered for the tested code. |
| 108 | This will make sure that the error does not crop up again if the code is |
| 109 | changed in the future. |
| 110 | \item Make sure to clean up after your tests (such as close and remove all |
| 111 | temporary files). |
| 112 | \item Import as few modules as possible and do it as soon as possible. |
| 113 | This minimizes external dependencies of tests and also minimizes possible |
| 114 | anomalous behavior from side-effects of importing a module. |
| 115 | \item Try to maximize code reuse. |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 116 | On occasion, tests will vary by something as small as what type |
| 117 | of input is used. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 118 | Minimize code duplication by subclassing a basic test class with a class |
| 119 | that specifies the input: |
| 120 | \begin{verbatim} |
| 121 | class TestFuncAcceptsSequences(unittest.TestCase): |
| 122 | |
| 123 | func = mySuperWhammyFunction |
| 124 | |
| 125 | def test_func(self): |
| 126 | self.func(self.arg) |
| 127 | |
| 128 | class AcceptLists(TestFuncAcceptsSequences): |
| 129 | arg = [1,2,3] |
| 130 | |
| 131 | class AcceptStrings(TestFuncAcceptsSequences): |
| 132 | arg = 'abc' |
| 133 | |
| 134 | class AcceptTuples(TestFuncAcceptsSequences): |
| 135 | arg = (1,2,3) |
| 136 | \end{verbatim} |
| 137 | \end{itemize} |
| 138 | |
| 139 | \begin{seealso} |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 140 | \seetitle{Test Driven Development} |
| 141 | {A book by Kent Beck on writing tests before code.} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 142 | \end{seealso} |
| 143 | |
| 144 | |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 145 | \subsection{Running tests Using \module{test.regrtest} \label{regrtest}} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 146 | |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 147 | \module{test.regrtest} can be used as a script to drive Python's |
| 148 | regression test suite. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 149 | Running the script by itself automatically starts running all |
| 150 | regression tests in the \module{test} package. |
| 151 | It does this by finding all modules in the package whose name starts with |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 152 | \samp{test_}, importing them, and executing the function |
| 153 | \function{test_main()} if present. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 154 | The names of tests to execute may also be passed to the script. |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 155 | Specifying a single regression test (\program{python regrtest.py} |
| 156 | \programopt{test_spam.py}) will minimize output and only print whether |
| 157 | the test passed or failed and thus minimize output. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 158 | |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 159 | Running \module{test.regrtest} directly allows what resources are |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 160 | available for tests to use to be set. |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 161 | You do this by using the \programopt{-u} command-line option. |
| 162 | Run \program{python regrtest.py} \programopt{-uall} to turn on all |
| 163 | resources; specifying \programopt{all} as an option for |
| 164 | \programopt{-u} enables all possible resources. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 165 | If all but one resource is desired (a more common case), a |
| 166 | comma-separated list of resources that are not desired may be listed after |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 167 | \programopt{all}. |
| 168 | The command \program{python regrtest.py} |
| 169 | \programopt{-uall,-audio,-largefile} will run \module{test.regrtest} |
| 170 | with all resources except the \programopt{audio} and |
| 171 | \programopt{largefile} resources. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 172 | For a list of all resources and more command-line options, run |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 173 | \program{python regrtest.py} \programopt{-h}. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 174 | |
| 175 | Some other ways to execute the regression tests depend on what platform the |
| 176 | tests are being executed on. |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 177 | On \UNIX{}, you can run \program{make} \programopt{test} at the |
| 178 | top-level directory where Python was built. |
| 179 | On Windows, executing \program{rt.bat} from your \file{PCBuild} |
| 180 | directory will run all regression tests. |
| 181 | |
| 182 | |
| 183 | \section{\module{test.test_support} --- |
| 184 | Utility functions for tests} |
| 185 | |
| 186 | \declaremodule[test.testsupport]{standard}{test.test_support} |
| 187 | \modulesynopsis{Support for Python regression tests.} |
| 188 | |
| 189 | The \module{test.test_support} module provides support for Python's |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 190 | regression tests. |
| 191 | |
Fred Drake | 60e868a | 2003-09-06 17:51:16 +0000 | [diff] [blame] | 192 | This module defines the following exceptions: |
| 193 | |
| 194 | \begin{excdesc}{TestFailed} |
| 195 | Exception to be raised when a test fails. |
| 196 | \end{excdesc} |
| 197 | |
| 198 | \begin{excdesc}{TestSkipped} |
| 199 | Subclass of \exception{TestFailed}. |
| 200 | Raised when a test is skipped. |
| 201 | This occurs when a needed resource (such as a network connection) is not |
| 202 | available at the time of testing. |
| 203 | \end{excdesc} |
| 204 | |
| 205 | \begin{excdesc}{ResourceDenied} |
| 206 | Subclass of \exception{TestSkipped}. |
| 207 | Raised when a resource (such as a network connection) is not available. |
| 208 | Raised by the \function{requires()} function. |
| 209 | \end{excdesc} |
| 210 | |
| 211 | |
| 212 | The \module{test.test_support} module defines the following constants: |
| 213 | |
| 214 | \begin{datadesc}{verbose} |
| 215 | \constant{True} when verbose output is enabled. |
| 216 | Should be checked when more detailed information is desired about a running |
| 217 | test. |
| 218 | \var{verbose} is set by \module{test.regrtest}. |
| 219 | \end{datadesc} |
| 220 | |
| 221 | \begin{datadesc}{have_unicode} |
| 222 | \constant{True} when Unicode support is available. |
| 223 | \end{datadesc} |
| 224 | |
| 225 | \begin{datadesc}{is_jython} |
| 226 | \constant{True} if the running interpreter is Jython. |
| 227 | \end{datadesc} |
| 228 | |
| 229 | \begin{datadesc}{TESTFN} |
| 230 | Set to the path that a temporary file may be created at. |
| 231 | Any temporary that is created should be closed and unlinked (removed). |
| 232 | \end{datadesc} |
| 233 | |
| 234 | |
| 235 | The \module{test.test_support} module defines the following functions: |
| 236 | |
| 237 | \begin{funcdesc}{forget}{module_name} |
| 238 | Removes the module named \var{module_name} from \code{sys.modules} and deletes |
| 239 | any byte-compiled files of the module. |
| 240 | \end{funcdesc} |
| 241 | |
| 242 | \begin{funcdesc}{is_resource_enabled}{resource} |
| 243 | Returns \constant{True} if \var{resource} is enabled and available. |
| 244 | The list of available resources is only set when \module{test.regrtest} |
| 245 | is executing the tests. |
| 246 | \end{funcdesc} |
| 247 | |
| 248 | \begin{funcdesc}{requires}{resource\optional{, msg}} |
| 249 | Raises \exception{ResourceDenied} if \var{resource} is not available. |
| 250 | \var{msg} is the argument to \exception{ResourceDenied} if it is raised. |
| 251 | Always returns true if called by a function whose \code{__name__} is |
| 252 | \code{'__main__'}. |
| 253 | Used when tests are executed by \module{test.regrtest}. |
| 254 | \end{funcdesc} |
| 255 | |
| 256 | \begin{funcdesc}{findfile}{filename} |
| 257 | Return the path to the file named \var{filename}. |
| 258 | If no match is found \var{filename} is returned. |
| 259 | This does not equal a failure since it could be the path to the file. |
| 260 | \end{funcdesc} |
| 261 | |
| 262 | \begin{funcdesc}{run_unittest}{*classes} |
| 263 | Execute \class{unittest.TestCase} subclasses passed to the function. |
| 264 | The function scans the classes for methods starting with the prefix |
| 265 | \samp{test_} and executes the tests individually. |
| 266 | This is the preferred way to execute tests. |
| 267 | \end{funcdesc} |
| 268 | |
| 269 | \begin{funcdesc}{run_suite}{suite\optional{, testclass}} |
| 270 | Execute the \class{unittest.TestSuite} instance \var{suite}. |
| 271 | The optional argument \var{testclass} accepts one of the test classes in the |
| 272 | suite so as to print out more detailed information on where the testing suite |
| 273 | originated from. |
| 274 | \end{funcdesc} |