Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 1 | \section{\module{test} --- |
| 2 | Regression tests package for Python} |
| 3 | |
| 4 | \declaremodule{standard}{test} |
| 5 | |
| 6 | \sectionauthor{Brett Cannon}{brett@python.org} |
| 7 | |
| 8 | |
| 9 | \modulesynopsis{Regression tests package containing the testing suite for |
| 10 | Python.} |
| 11 | |
| 12 | |
| 13 | The \module{test} package contains all regression tests for Python as well as |
| 14 | the modules \module{test_support} and \module{regrtest.py}. |
| 15 | \module{test_support} is used to enhance your tests while \module{regrtest.py} |
| 16 | drives the testing suite. |
| 17 | |
| 18 | Each module in the \module{test} package whose name starts with |
| 19 | \code{'test_'} is a testing suite for a specific module or feature. |
| 20 | All new tests should be written using the \module{unittest} module; using |
| 21 | \module{unittest} is not required but makes the tests more flexible and |
| 22 | maintenance of the tests easier. |
| 23 | Some older tests are written to use \module{doctest} and a ``traditional'' |
| 24 | testing style; these styles of tests will not be covered. |
| 25 | |
| 26 | \begin{seealso} |
| 27 | \seemodule{unittest}{Writing PyUnit regression tests.} |
| 28 | \seemodule{doctest}{Tests embedded in documentation strings.} |
| 29 | \end{seealso} |
| 30 | |
| 31 | |
Raymond Hettinger | 951976d | 2003-09-06 04:39:54 +0000 | [diff] [blame] | 32 | \subsection{\module{test.test_support} --- Utility functions for tests} |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 33 | \declaremodule[test.testsupport]{standard}{test.test_support} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 34 | |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 35 | The \module{test.test_support} module contains functions for assisting |
| 36 | with writing regression tests. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 37 | |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 38 | The \module{test.test_support} module defines the following exceptions: |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 39 | |
| 40 | \begin{excdesc}{TestFailed} |
| 41 | Exception to be raised when a test fails. |
| 42 | \end{excdesc} |
| 43 | |
| 44 | \begin{excdesc}{TestSkipped} |
| 45 | Subclass of \exception{TestFailed}. |
| 46 | Raised when a test is skipped. |
| 47 | This occurs when a needed resource (such as a network connection) is not |
| 48 | available at the time of testing. |
| 49 | \end{excdesc} |
| 50 | |
| 51 | \begin{excdesc}{ResourceDenied} |
| 52 | Subclass of \exception{TestSkipped}. |
| 53 | Raised when a resource (such as a network connection) is not available. |
| 54 | Raised by the \function{requires} function. |
| 55 | \end{excdesc} |
| 56 | |
| 57 | |
| 58 | The \module{test_support} module defines the following constants: |
| 59 | |
| 60 | \begin{datadesc}{verbose} |
| 61 | \constant{True} when verbose output is enabled. |
| 62 | Should be checked when more detailed information is desired about a running |
| 63 | test. |
| 64 | \var{verbose} is set by \module{regrtest.py}. |
| 65 | \end{datadesc} |
| 66 | |
| 67 | \begin{datadesc}{have_unicode} |
| 68 | \constant{True} when Unicode support is available. |
| 69 | \end{datadesc} |
| 70 | |
| 71 | \begin{datadesc}{is_jython} |
| 72 | \constant{True} if the running interpreter is Jython. |
| 73 | \end{datadesc} |
| 74 | |
| 75 | \begin{datadesc}{TESTFN} |
| 76 | Set to the path that a temporary file may be created at. |
| 77 | Any temporary that is created should be closed and unlinked (removed). |
| 78 | \end{datadesc} |
| 79 | |
| 80 | |
| 81 | The \module{test_support} module defines the following functions: |
| 82 | |
| 83 | \begin{funcdesc}{forget}{module_name} |
| 84 | Removes the module named \var{module_name} from \module{sys.modules} and deletes |
| 85 | any byte-compiled files of the module. |
| 86 | \end{funcdesc} |
| 87 | |
| 88 | \begin{funcdesc}{is_resource_enabled}{resource} |
| 89 | Returns \constant{True} if \var{resource} is enabled and available. |
| 90 | The list of available resources is only set when \module{regrtest.py} is |
| 91 | executing the tests. |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 92 | \end{funcdesc} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 93 | |
| 94 | \begin{funcdesc}{requires}{resource\optional{, msg}} |
| 95 | Raises \exception{ResourceDenied} if \var{resource} is not available. |
| 96 | \var{msg} is the argument to \exception{ResourceDenied} if it is raised. |
| 97 | Always returns true if called by a function whose \var{__name__} is |
| 98 | \code{"__main__"}. |
| 99 | Used when tests are executed by \module{regrtest.py}. |
| 100 | \end{funcdesc} |
| 101 | |
| 102 | \begin{funcdesc}{findfile}{filename} |
| 103 | Return the path to the file named \var{filename}. |
| 104 | If no match is found \var{filename} is returned. |
| 105 | This does not equal a failure since it could be the path to the file. |
| 106 | \end{funcdesc} |
| 107 | |
| 108 | \begin{funcdesc}{run_unittest}{*classes} |
| 109 | Execute \class{unittest.TestCase} subclasses passed to the function. |
| 110 | The function scans the classes for methods starting with the name |
| 111 | \code{"test_"} and executes the tests individually. |
| 112 | This is the preferred way to execute tests. |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 113 | \end{funcdesc} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 114 | |
| 115 | \begin{funcdesc}{run_suite}{suite\optional{, testclass=None}} |
| 116 | Execute the \class{unittest.TestSuite} instance, \var{suite}. |
| 117 | The optional argument \var{testclass} accepts one of the test classes in the |
| 118 | suite so as to print out more detailed information on where the testing suite |
| 119 | originated from. |
| 120 | \end{funcdesc} |
| 121 | |
| 122 | |
| 123 | |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 124 | \subsection{Writing Unit Tests for the \module{test} package% |
| 125 | \label{writing-tests}} |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 126 | |
| 127 | It is preferred that tests for the \module{test} package use the |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 128 | \refmodule{unittest} module and follow a few guidelines. |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 129 | One is to have the name of all the test methods start with \code{"test_"} as |
| 130 | well as the module's name. |
| 131 | This is needed so that the methods are recognized by the test driver as |
| 132 | test methods. |
| 133 | Also, no documentation string for the method should be included. |
| 134 | A comment (such as |
Fred Drake | 9f545c4 | 2003-05-09 19:10:12 +0000 | [diff] [blame] | 135 | \code{\# Tests function returns only True or False}) should be used to provide |
Brett Cannon | 066f392 | 2003-05-07 22:02:17 +0000 | [diff] [blame] | 136 | documentation for test methods. |
| 137 | This is done because documentation strings get printed out if they exist and |
| 138 | thus what test is being run is not stated. |
| 139 | |
| 140 | A basic boilerplate is often used: |
| 141 | |
| 142 | \begin{verbatim} |
| 143 | import unittest |
| 144 | from test import test_support |
| 145 | |
| 146 | class MyTestCase1(unittest.TestCase): |
| 147 | |
| 148 | # Only use setUp() and tearDown() if necessary |
| 149 | |
| 150 | def setUp(self): |
| 151 | ... code to execute in preparation for tests ... |
| 152 | |
| 153 | def tearDown(self): |
| 154 | ... code to execute to clean up after tests ... |
| 155 | |
| 156 | def test_feature_one(self): |
| 157 | # Test feature one. |
| 158 | ... testing code ... |
| 159 | |
| 160 | def test_feature_two(self): |
| 161 | # Test feature two. |
| 162 | ... testing code ... |
| 163 | |
| 164 | ... more test methods ... |
| 165 | |
| 166 | class MyTestCase2(unittest.TestCase): |
| 167 | ... same structure as MyTestCase1 ... |
| 168 | |
| 169 | ... more test classes ... |
| 170 | |
| 171 | def test_main(): |
| 172 | test_support.run_unittest(MyTestCase1, |
| 173 | MyTestCase2, |
| 174 | ... list other tests ... |
| 175 | ) |
| 176 | |
| 177 | if __name__ == '__main__': |
| 178 | test_main() |
| 179 | \end{verbatim} |
| 180 | |
| 181 | This boilerplate code allows the testing suite to be run by \module{regrtest.py} |
| 182 | as well as on its own as a script. |
| 183 | |
| 184 | The goal for regression testing is to try to break code. |
| 185 | This leads to a few guidelines to be followed: |
| 186 | |
| 187 | \begin{itemize} |
| 188 | \item The testing suite should exercise all classes, functions, and |
| 189 | constants. |
| 190 | This includes not just the external API that is to be presented to the |
| 191 | outside world but also "private" code. |
| 192 | \item Whitebox testing (examining the code being tested when the tests are |
| 193 | being written) is preferred. |
| 194 | Blackbox testing (testing only the published user interface) is not |
| 195 | complete enough to make sure all boundary and edge cases are tested. |
| 196 | \item Make sure all possible values are tested including invalid ones. |
| 197 | This makes sure that not only all valid values are acceptable but also |
| 198 | that improper values are handled correctly. |
| 199 | \item Exhaust as many code paths as possible. |
| 200 | Test where branching occurs and thus tailor input to make sure as many |
| 201 | different paths through the code are taken. |
| 202 | \item Add an explicit test for any bugs discovered for the tested code. |
| 203 | This will make sure that the error does not crop up again if the code is |
| 204 | changed in the future. |
| 205 | \item Make sure to clean up after your tests (such as close and remove all |
| 206 | temporary files). |
| 207 | \item Import as few modules as possible and do it as soon as possible. |
| 208 | This minimizes external dependencies of tests and also minimizes possible |
| 209 | anomalous behavior from side-effects of importing a module. |
| 210 | \item Try to maximize code reuse. |
| 211 | On occasion tests will vary by something as small as what type of input |
| 212 | they take. |
| 213 | Minimize code duplication by subclassing a basic test class with a class |
| 214 | that specifies the input: |
| 215 | \begin{verbatim} |
| 216 | class TestFuncAcceptsSequences(unittest.TestCase): |
| 217 | |
| 218 | func = mySuperWhammyFunction |
| 219 | |
| 220 | def test_func(self): |
| 221 | self.func(self.arg) |
| 222 | |
| 223 | class AcceptLists(TestFuncAcceptsSequences): |
| 224 | arg = [1,2,3] |
| 225 | |
| 226 | class AcceptStrings(TestFuncAcceptsSequences): |
| 227 | arg = 'abc' |
| 228 | |
| 229 | class AcceptTuples(TestFuncAcceptsSequences): |
| 230 | arg = (1,2,3) |
| 231 | \end{verbatim} |
| 232 | \end{itemize} |
| 233 | |
| 234 | \begin{seealso} |
| 235 | \seetitle{Test Driven Development}{A book by Kent Beck on writing tests before |
| 236 | code} |
| 237 | \end{seealso} |
| 238 | |
| 239 | |
| 240 | |
| 241 | \subsection{Running tests Using \module{regrtest.py} \label{regrtest}} |
| 242 | |
| 243 | \module{regrtest.py} is the script used to drive Python's regression test |
| 244 | suite. |
| 245 | Running the script by itself automatically starts running all |
| 246 | regression tests in the \module{test} package. |
| 247 | It does this by finding all modules in the package whose name starts with |
| 248 | \code{test_}, importing them, and executing the function \function{test_main} |
| 249 | if present. |
| 250 | The names of tests to execute may also be passed to the script. |
| 251 | Specifying a single regression test (\code{python regrtest.py test_spam.py}) |
| 252 | will minimize output and only print whether the test passed or failed and thus |
| 253 | minimize output. |
| 254 | |
| 255 | Running \module{regrtest.py} directly allows what resources are |
| 256 | available for tests to use to be set. |
| 257 | You do this by using the \code{-u} command-line option. |
| 258 | Run \code{python regrtest.py -uall} to turn on all resources; |
| 259 | specifying \code{all} as an option for \code{-u} enables all possible |
| 260 | resources. |
| 261 | If all but one resource is desired (a more common case), a |
| 262 | comma-separated list of resources that are not desired may be listed after |
| 263 | \code{all}. |
| 264 | The command \code{python regrtest.py -uall,-audio,-largefile} will run |
| 265 | \module{regrtest.py} with all resources except the audio and largefile |
| 266 | resources. |
| 267 | For a list of all resources and more command-line options, run |
| 268 | \code{python regrtest.py -h}. |
| 269 | |
| 270 | Some other ways to execute the regression tests depend on what platform the |
| 271 | tests are being executed on. |
| 272 | On \UNIX{}, you can run \code{make test} at the top-level directory |
| 273 | where Python was built. |
| 274 | On Windows, executing \code{rt.bat} from your PCBuild directory will run all |
| 275 | regression tests. |
| 276 | |