Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 1 | Writing Python Regression Tests |
| 2 | ------------------------------- |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 3 | Skip Montanaro |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 4 | (skip@mojam.com) |
| 5 | |
| 6 | |
| 7 | Introduction |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 8 | |
| 9 | If you add a new module to Python or modify the functionality of an existing |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 10 | module, you should write one or more test cases to exercise that new |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame^] | 11 | functionality. There are different ways to do this within the regression |
| 12 | testing facility provided with Python; any particular test should use only |
| 13 | one of these options. Each option requires writing a test module using the |
| 14 | conventions of the the selected option: |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 15 | |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame^] | 16 | - PyUnit based tests |
| 17 | - doctest based tests |
| 18 | - "traditional" Python test modules |
| 19 | |
| 20 | Regardless of the mechanics of the testing approach you choose, |
| 21 | you will be writing unit tests (isolated tests of functions and objects |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 22 | defined by the module) using white box techniques. Unlike black box |
| 23 | testing, where you only have the external interfaces to guide your test case |
| 24 | writing, in white box testing you can see the code being tested and tailor |
| 25 | your test cases to exercise it more completely. In particular, you will be |
| 26 | able to refer to the C and Python code in the CVS repository when writing |
| 27 | your regression test cases. |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 28 | |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame^] | 29 | PyUnit based tests |
| 30 | |
| 31 | The PyUnit framework is based on the ideas of unit testing as espoused |
| 32 | by Kent Beck and the Extreme Programming (XP) movement. The specific |
| 33 | interface provided by the framework is tightly based on the JUnit |
| 34 | Java implementation of Beck's original SmallTalk test framework. Please |
| 35 | see the documentation of the unittest module for detailed information on |
| 36 | the interface and general guidelines on writing PyUnit based tests. |
| 37 | |
| 38 | The test_support helper module provides a single function for use by |
| 39 | PyUnit based tests in the Python regression testing framework: |
| 40 | run_unittest() takes a unittest.TestCase derived class as a parameter |
| 41 | and runs the tests defined in that class. All test methods in the |
| 42 | Python regression framework have names that start with "test_" and use |
| 43 | lower-case names with words separated with underscores. |
| 44 | |
| 45 | doctest based tests |
| 46 | |
| 47 | Tests written to use doctest are actually part of the docstrings for |
| 48 | the module being tested. Each test is written as a display of an |
| 49 | interactive session, including the Python prompts, statements that would |
| 50 | be typed by the user, and the output of those statements (including |
| 51 | tracebacks!). The module in the test package is simply a wrapper that |
| 52 | causes doctest to run over the tests in the module. The test for the |
| 53 | doctest module provides a convenient example: |
| 54 | |
| 55 | import doctest |
| 56 | doctest.testmod(doctest, verbose=1) |
| 57 | |
| 58 | See the documentation for the doctest module for information on |
| 59 | writing tests using the doctest framework. |
| 60 | |
| 61 | "traditional" Python test modules |
| 62 | |
| 63 | The mechanics of how the "traditional" test system operates are fairly |
| 64 | straightforward. When a test case is run, the output is compared with the |
| 65 | expected output that is stored in .../Lib/test/output. If the test runs to |
| 66 | completion and the actual and expected outputs match, the test succeeds, if |
| 67 | not, it fails. If an ImportError or test_support.TestSkipped error is |
| 68 | raised, the test is not run. |
| 69 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 70 | |
| 71 | Executing Test Cases |
| 72 | |
| 73 | If you are writing test cases for module spam, you need to create a file |
| 74 | in .../Lib/test named test_spam.py and an expected output file in |
| 75 | .../Lib/test/output named test_spam ("..." represents the top-level |
| 76 | directory in the Python source tree, the directory containing the configure |
| 77 | script). From the top-level directory, generate the initial version of the |
| 78 | test output file by executing: |
| 79 | |
| 80 | ./python Lib/test/regrtest.py -g test_spam.py |
| 81 | |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame^] | 82 | (If your test does not generate any output when run successfully, this |
| 83 | step may be skipped; no file containing expected output will be needed |
| 84 | in this case.) |
| 85 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 86 | Any time you modify test_spam.py you need to generate a new expected |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 87 | output file. Don't forget to desk check the generated output to make sure |
| 88 | it's really what you expected to find! To run a single test after modifying |
| 89 | a module, simply run regrtest.py without the -g flag: |
| 90 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 91 | ./python Lib/test/regrtest.py test_spam.py |
| 92 | |
| 93 | While debugging a regression test, you can of course execute it |
| 94 | independently of the regression testing framework and see what it prints: |
| 95 | |
| 96 | ./python Lib/test/test_spam.py |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 97 | |
| 98 | To run the entire test suite, make the "test" target at the top level: |
| 99 | |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 100 | make test |
| 101 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 102 | On non-Unix platforms where make may not be available, you can simply |
| 103 | execute the two runs of regrtest (optimized and non-optimized) directly: |
| 104 | |
| 105 | ./python Lib/test/regrtest.py |
| 106 | ./python -O Lib/test/regrtest.py |
| 107 | |
| 108 | |
| 109 | Test cases generate output based upon values computed by the test code. |
| 110 | When executed, regrtest.py compares the actual output generated by executing |
| 111 | the test case with the expected output and reports success or failure. It |
| 112 | stands to reason that if the actual and expected outputs are to match, they |
| 113 | must not contain any machine dependencies. This means your test cases |
| 114 | should not print out absolute machine addresses (e.g. the return value of |
| 115 | the id() builtin function) or floating point numbers with large numbers of |
| 116 | significant digits (unless you understand what you are doing!). |
| 117 | |
| 118 | |
| 119 | Test Case Writing Tips |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 120 | |
| 121 | Writing good test cases is a skilled task and is too complex to discuss in |
| 122 | detail in this short document. Many books have been written on the subject. |
| 123 | I'll show my age by suggesting that Glenford Myers' "The Art of Software |
| 124 | Testing", published in 1979, is still the best introduction to the subject |
| 125 | available. It is short (177 pages), easy to read, and discusses the major |
| 126 | elements of software testing, though its publication predates the |
| 127 | object-oriented software revolution, so doesn't cover that subject at all. |
| 128 | Unfortunately, it is very expensive (about $100 new). If you can borrow it |
| 129 | or find it used (around $20), I strongly urge you to pick up a copy. |
| 130 | |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 131 | The most important goal when writing test cases is to break things. A test |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 132 | case that doesn't uncover a bug is much less valuable than one that does. |
| 133 | In designing test cases you should pay attention to the following: |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 134 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 135 | * Your test cases should exercise all the functions and objects defined |
| 136 | in the module, not just the ones meant to be called by users of your |
| 137 | module. This may require you to write test code that uses the module |
| 138 | in ways you don't expect (explicitly calling internal functions, for |
| 139 | example - see test_atexit.py). |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 140 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 141 | * You should consider any boundary values that may tickle exceptional |
| 142 | conditions (e.g. if you were writing regression tests for division, |
| 143 | you might well want to generate tests with numerators and denominators |
| 144 | at the limits of floating point and integer numbers on the machine |
| 145 | performing the tests as well as a denominator of zero). |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 146 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 147 | * You should exercise as many paths through the code as possible. This |
| 148 | may not always be possible, but is a goal to strive for. In |
| 149 | particular, when considering if statements (or their equivalent), you |
| 150 | want to create test cases that exercise both the true and false |
| 151 | branches. For loops, you should create test cases that exercise the |
| 152 | loop zero, one and multiple times. |
| 153 | |
| 154 | * You should test with obviously invalid input. If you know that a |
| 155 | function requires an integer input, try calling it with other types of |
| 156 | objects to see how it responds. |
| 157 | |
| 158 | * You should test with obviously out-of-range input. If the domain of a |
| 159 | function is only defined for positive integers, try calling it with a |
| 160 | negative integer. |
| 161 | |
| 162 | * If you are going to fix a bug that wasn't uncovered by an existing |
| 163 | test, try to write a test case that exposes the bug (preferably before |
| 164 | fixing it). |
| 165 | |
Fred Drake | 44b6bd2 | 2000-10-23 16:37:14 +0000 | [diff] [blame] | 166 | * If you need to create a temporary file, you can use the filename in |
| 167 | test_support.TESTFN to do so. It is important to remove the file |
| 168 | when done; other tests should be able to use the name without cleaning |
| 169 | up after your test. |
| 170 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 171 | |
| 172 | Regression Test Writing Rules |
| 173 | |
| 174 | Each test case is different. There is no "standard" form for a Python |
| 175 | regression test case, though there are some general rules: |
| 176 | |
| 177 | * If your test case detects a failure, raise TestFailed (found in |
| 178 | test_support). |
| 179 | |
| 180 | * Import everything you'll need as early as possible. |
| 181 | |
| 182 | * If you'll be importing objects from a module that is at least |
| 183 | partially platform-dependent, only import those objects you need for |
| 184 | the current test case to avoid spurious ImportError exceptions that |
| 185 | prevent the test from running to completion. |
| 186 | |
| 187 | * Print all your test case results using the print statement. For |
| 188 | non-fatal errors, print an error message (or omit a successful |
| 189 | completion print) to indicate the failure, but proceed instead of |
| 190 | raising TestFailed. |
| 191 | |
Tim Peters | a48b526 | 2000-08-23 05:28:45 +0000 | [diff] [blame] | 192 | * Use "assert" sparingly, if at all. It's usually better to just print |
| 193 | what you got, and rely on regrtest's got-vs-expected comparison to |
| 194 | catch deviations from what you expect. assert statements aren't |
| 195 | executed at all when regrtest is run in -O mode; and, because they |
| 196 | cause the test to stop immediately, can lead to a long & tedious |
| 197 | test-fix, test-fix, test-fix, ... cycle when things are badly broken |
| 198 | (and note that "badly broken" often includes running the test suite |
| 199 | for the first time on new platforms or under new implementations of |
| 200 | the language). |
| 201 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 202 | |
| 203 | Miscellaneous |
| 204 | |
| 205 | There is a test_support module you can import from your test case. It |
| 206 | provides the following useful objects: |
| 207 | |
| 208 | * TestFailed - raise this exception when your regression test detects a |
| 209 | failure. |
| 210 | |
Fred Drake | 62c53dd | 2000-08-21 16:55:57 +0000 | [diff] [blame] | 211 | * TestSkipped - raise this if the test could not be run because the |
| 212 | platform doesn't offer all the required facilities (like large |
| 213 | file support), even if all the required modules are available. |
| 214 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 215 | * findfile(file) - you can call this function to locate a file somewhere |
| 216 | along sys.path or in the Lib/test tree - see test_linuxaudiodev.py for |
| 217 | an example of its use. |
| 218 | |
| 219 | * verbose - you can use this variable to control print output. Many |
| 220 | modules use it. Search for "verbose" in the test_*.py files to see |
| 221 | lots of examples. |
| 222 | |
Tim Peters | a48b526 | 2000-08-23 05:28:45 +0000 | [diff] [blame] | 223 | * use_large_resources - true iff tests requiring large time or space |
| 224 | should be run. |
| 225 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 226 | * fcmp(x,y) - you can call this function to compare two floating point |
| 227 | numbers when you expect them to only be approximately equal withing a |
| 228 | fuzz factor (test_support.FUZZ, which defaults to 1e-6). |
| 229 | |
Tim Peters | a48b526 | 2000-08-23 05:28:45 +0000 | [diff] [blame] | 230 | NOTE: Always import something from test_support like so: |
| 231 | |
| 232 | from test_support import verbose |
| 233 | |
| 234 | or like so: |
| 235 | |
| 236 | import test_support |
| 237 | ... use test_support.verbose in the code ... |
| 238 | |
| 239 | Never import anything from test_support like this: |
| 240 | |
| 241 | from test.test_support import verbose |
| 242 | |
| 243 | "test" is a package already, so can refer to modules it contains without |
| 244 | "test." qualification. If you do an explicit "test.xxx" qualification, that |
| 245 | can fool Python into believing test.xxx is a module distinct from the xxx |
| 246 | in the current package, and you can end up importing two distinct copies of |
| 247 | xxx. This is especially bad if xxx=test_support, as regrtest.py can (and |
| 248 | routinely does) overwrite its "verbose" and "use_large_resources" |
| 249 | attributes: if you get a second copy of test_support loaded, it may not |
| 250 | have the same values for those as regrtest intended. |
| 251 | |
| 252 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 253 | Python and C statement coverage results are currently available at |
| 254 | |
| 255 | http://www.musi-cal.com/~skip/python/Python/dist/src/ |
| 256 | |
| 257 | As of this writing (July, 2000) these results are being generated nightly. |
| 258 | You can refer to the summaries and the test coverage output files to see |
| 259 | where coverage is adequate or lacking and write test cases to beef up the |
| 260 | coverage. |