Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 1 | +++++++++++++++++++++++++++++++ |
| 2 | Writing Python Regression Tests |
| 3 | +++++++++++++++++++++++++++++++ |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 4 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 5 | :Author: Skip Montanaro |
Christian Heimes | 895627f | 2007-12-08 17:28:33 +0000 | [diff] [blame] | 6 | :Contact: skip@pobox.com |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 7 | |
| 8 | Introduction |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 9 | ============ |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 10 | |
| 11 | If you add a new module to Python or modify the functionality of an existing |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 12 | module, you should write one or more test cases to exercise that new |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 13 | functionality. There are different ways to do this within the regression |
| 14 | testing facility provided with Python; any particular test should use only |
| 15 | one of these options. Each option requires writing a test module using the |
Walter Dörwald | 88af4df | 2003-02-03 20:22:27 +0000 | [diff] [blame] | 16 | conventions of the selected option: |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 17 | |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 18 | - unittest_ based tests |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 19 | - doctest_ based tests |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 20 | - "traditional" Python test modules |
| 21 | |
| 22 | Regardless of the mechanics of the testing approach you choose, |
| 23 | you will be writing unit tests (isolated tests of functions and objects |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 24 | defined by the module) using white box techniques. Unlike black box |
| 25 | testing, where you only have the external interfaces to guide your test case |
| 26 | writing, in white box testing you can see the code being tested and tailor |
| 27 | your test cases to exercise it more completely. In particular, you will be |
| 28 | able to refer to the C and Python code in the CVS repository when writing |
| 29 | your regression test cases. |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 30 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 31 | .. _unittest: http://www.python.org/doc/current/lib/module-unittest.html |
| 32 | .. _doctest: http://www.python.org/doc/current/lib/module-doctest.html |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 33 | |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 34 | unittest-based tests |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 35 | ------------------ |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 36 | The unittest_ framework is based on the ideas of unit testing as espoused |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 37 | by Kent Beck and the `Extreme Programming`_ (XP) movement. The specific |
| 38 | interface provided by the framework is tightly based on the JUnit_ |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 39 | Java implementation of Beck's original SmallTalk test framework. Please |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 40 | see the documentation of the unittest_ module for detailed information on |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 41 | the interface and general guidelines on writing unittest-based tests. |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 42 | |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 43 | The test_support helper module provides a function for use by |
| 44 | unittest-based tests in the Python regression testing framework, |
| 45 | ``run_unittest()``. This is the primary way of running tests in the |
| 46 | standard library. You can pass it any number of the following: |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 47 | |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 48 | - classes derived from or instances of ``unittest.TestCase`` or |
| 49 | ``unittest.TestSuite``. These will be handed off to unittest for |
| 50 | converting into a proper TestSuite instance. |
| 51 | |
| 52 | - a string; this must be a key in sys.modules. The module associated with |
| 53 | that string will be scanned by ``unittest.TestLoader.loadTestsFromModule``. |
| 54 | This is usually seen as ``test_support.run_unittest(__name__)`` in a test |
| 55 | module's ``test_main()`` function. This has the advantage of picking up |
| 56 | new tests automatically, without you having to add each new test case |
| 57 | manually. |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 58 | |
Barry Warsaw | 5ca5374 | 2002-04-23 21:39:00 +0000 | [diff] [blame] | 59 | All test methods in the Python regression framework have names that |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 60 | start with "``test_``" and use lower-case names with words separated with |
Barry Warsaw | 5ca5374 | 2002-04-23 21:39:00 +0000 | [diff] [blame] | 61 | underscores. |
Fred Drake | b2ad1c8 | 2001-09-28 20:05:25 +0000 | [diff] [blame] | 62 | |
Guido van Rossum | 1c48654 | 2002-08-22 20:08:14 +0000 | [diff] [blame] | 63 | Test methods should *not* have docstrings! The unittest module prints |
| 64 | the docstring if there is one, but otherwise prints the function name |
| 65 | and the full class name. When there's a problem with a test, the |
| 66 | latter information makes it easier to find the source for the test |
| 67 | than the docstring. |
| 68 | |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 69 | All unittest-based tests in the Python test suite use boilerplate that |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 70 | looks like this (with minor variations):: |
Fred Drake | b2ad1c8 | 2001-09-28 20:05:25 +0000 | [diff] [blame] | 71 | |
| 72 | import unittest |
Barry Warsaw | 04f357c | 2002-07-23 19:04:11 +0000 | [diff] [blame] | 73 | from test import test_support |
Fred Drake | b2ad1c8 | 2001-09-28 20:05:25 +0000 | [diff] [blame] | 74 | |
Barry Warsaw | 5ca5374 | 2002-04-23 21:39:00 +0000 | [diff] [blame] | 75 | class MyTestCase1(unittest.TestCase): |
Guido van Rossum | 1c48654 | 2002-08-22 20:08:14 +0000 | [diff] [blame] | 76 | |
| 77 | # Define setUp and tearDown only if needed |
| 78 | |
| 79 | def setUp(self): |
| 80 | unittest.TestCase.setUp(self) |
| 81 | ... additional initialization... |
| 82 | |
| 83 | def tearDown(self): |
| 84 | ... additional finalization... |
| 85 | unittest.TestCase.tearDown(self) |
| 86 | |
| 87 | def test_feature_one(self): |
| 88 | # Testing feature one |
| 89 | ...unit test for feature one... |
| 90 | |
| 91 | def test_feature_two(self): |
| 92 | # Testing feature two |
| 93 | ...unit test for feature two... |
| 94 | |
| 95 | ...etc... |
Fred Drake | b2ad1c8 | 2001-09-28 20:05:25 +0000 | [diff] [blame] | 96 | |
Barry Warsaw | 5ca5374 | 2002-04-23 21:39:00 +0000 | [diff] [blame] | 97 | class MyTestCase2(unittest.TestCase): |
Guido van Rossum | 1c48654 | 2002-08-22 20:08:14 +0000 | [diff] [blame] | 98 | ...same structure as MyTestCase1... |
| 99 | |
| 100 | ...etc... |
Barry Warsaw | 5ca5374 | 2002-04-23 21:39:00 +0000 | [diff] [blame] | 101 | |
Fred Drake | b2ad1c8 | 2001-09-28 20:05:25 +0000 | [diff] [blame] | 102 | def test_main(): |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 103 | test_support.run_unittest(__name__) |
Fred Drake | b2ad1c8 | 2001-09-28 20:05:25 +0000 | [diff] [blame] | 104 | |
| 105 | if __name__ == "__main__": |
| 106 | test_main() |
| 107 | |
| 108 | This has the advantage that it allows the unittest module to be used |
| 109 | as a script to run individual tests as well as working well with the |
| 110 | regrtest framework. |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 111 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 112 | .. _Extreme Programming: http://www.extremeprogramming.org/ |
| 113 | .. _JUnit: http://www.junit.org/ |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 114 | |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 115 | doctest based tests |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 116 | ------------------- |
| 117 | Tests written to use doctest_ are actually part of the docstrings for |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 118 | the module being tested. Each test is written as a display of an |
| 119 | interactive session, including the Python prompts, statements that would |
| 120 | be typed by the user, and the output of those statements (including |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 121 | tracebacks, although only the exception msg needs to be retained then). |
| 122 | The module in the test package is simply a wrapper that causes doctest |
| 123 | to run over the tests in the module. The test for the difflib module |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 124 | provides a convenient example:: |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 125 | |
Barry Warsaw | 1bfab7b | 2002-07-23 19:13:45 +0000 | [diff] [blame] | 126 | import difflib |
| 127 | from test import test_support |
Tim Peters | a0a6222 | 2001-09-09 06:12:01 +0000 | [diff] [blame] | 128 | test_support.run_doctest(difflib) |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 129 | |
| 130 | If the test is successful, nothing is written to stdout (so you should not |
| 131 | create a corresponding output/test_difflib file), but running regrtest |
Tim Peters | a0a6222 | 2001-09-09 06:12:01 +0000 | [diff] [blame] | 132 | with -v will give a detailed report, the same as if passing -v to doctest. |
| 133 | |
| 134 | A second argument can be passed to run_doctest to tell doctest to search |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 135 | ``sys.argv`` for -v instead of using test_support's idea of verbosity. This |
Tim Peters | a0a6222 | 2001-09-09 06:12:01 +0000 | [diff] [blame] | 136 | is useful for writing doctest-based tests that aren't simply running a |
| 137 | doctest'ed Lib module, but contain the doctests themselves. Then at |
| 138 | times you may want to run such a test directly as a doctest, independent |
| 139 | of the regrtest framework. The tail end of test_descrtut.py is a good |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 140 | example:: |
Tim Peters | a0a6222 | 2001-09-09 06:12:01 +0000 | [diff] [blame] | 141 | |
| 142 | def test_main(verbose=None): |
Barry Warsaw | 1bfab7b | 2002-07-23 19:13:45 +0000 | [diff] [blame] | 143 | from test import test_support, test_descrtut |
| 144 | test_support.run_doctest(test_descrtut, verbose) |
Tim Peters | a0a6222 | 2001-09-09 06:12:01 +0000 | [diff] [blame] | 145 | |
| 146 | if __name__ == "__main__": |
| 147 | test_main(1) |
| 148 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 149 | If run via regrtest, ``test_main()`` is called (by regrtest) without |
| 150 | specifying verbose, and then test_support's idea of verbosity is used. But |
| 151 | when run directly, ``test_main(1)`` is called, and then doctest's idea of |
| 152 | verbosity is used. |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 153 | |
| 154 | See the documentation for the doctest module for information on |
| 155 | writing tests using the doctest framework. |
| 156 | |
| 157 | "traditional" Python test modules |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 158 | --------------------------------- |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 159 | The mechanics of how the "traditional" test system operates are fairly |
| 160 | straightforward. When a test case is run, the output is compared with the |
| 161 | expected output that is stored in .../Lib/test/output. If the test runs to |
| 162 | completion and the actual and expected outputs match, the test succeeds, if |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 163 | not, it fails. If an ``ImportError`` or ``test_support.TestSkipped`` error |
| 164 | is raised, the test is not run. |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 165 | |
| 166 | Executing Test Cases |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 167 | ==================== |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 168 | If you are writing test cases for module spam, you need to create a file |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 169 | in .../Lib/test named test_spam.py. In addition, if the tests are expected |
| 170 | to write to stdout during a successful run, you also need to create an |
| 171 | expected output file in .../Lib/test/output named test_spam ("..." |
| 172 | represents the top-level directory in the Python source tree, the directory |
| 173 | containing the configure script). If needed, generate the initial version |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 174 | of the test output file by executing:: |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 175 | |
| 176 | ./python Lib/test/regrtest.py -g test_spam.py |
| 177 | |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 178 | from the top-level directory. |
Fred Drake | a6daad2 | 2001-05-23 04:57:49 +0000 | [diff] [blame] | 179 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 180 | Any time you modify test_spam.py you need to generate a new expected |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 181 | output file. Don't forget to desk check the generated output to make sure |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 182 | it's really what you expected to find! All in all it's usually better |
| 183 | not to have an expected-out file (note that doctest- and unittest-based |
| 184 | tests do not). |
| 185 | |
| 186 | To run a single test after modifying a module, simply run regrtest.py |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 187 | without the -g flag:: |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 188 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 189 | ./python Lib/test/regrtest.py test_spam.py |
| 190 | |
| 191 | While debugging a regression test, you can of course execute it |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 192 | independently of the regression testing framework and see what it prints:: |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 193 | |
| 194 | ./python Lib/test/test_spam.py |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 195 | |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 196 | To run the entire test suite: |
| 197 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 198 | - [UNIX, + other platforms where "make" works] Make the "test" target at the |
| 199 | top level:: |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 200 | |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 201 | make test |
| 202 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 203 | - [WINDOWS] Run rt.bat from your PCBuild directory. Read the comments at |
| 204 | the top of rt.bat for the use of special -d, -O and -q options processed |
| 205 | by rt.bat. |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 206 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 207 | - [OTHER] You can simply execute the two runs of regrtest (optimized and |
| 208 | non-optimized) directly:: |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 209 | |
| 210 | ./python Lib/test/regrtest.py |
| 211 | ./python -O Lib/test/regrtest.py |
| 212 | |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 213 | But note that this way picks up whatever .pyc and .pyo files happen to be |
| 214 | around. The makefile and rt.bat ways run the tests twice, the first time |
| 215 | removing all .pyc and .pyo files from the subtree rooted at Lib/. |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 216 | |
| 217 | Test cases generate output based upon values computed by the test code. |
| 218 | When executed, regrtest.py compares the actual output generated by executing |
| 219 | the test case with the expected output and reports success or failure. It |
| 220 | stands to reason that if the actual and expected outputs are to match, they |
| 221 | must not contain any machine dependencies. This means your test cases |
| 222 | should not print out absolute machine addresses (e.g. the return value of |
| 223 | the id() builtin function) or floating point numbers with large numbers of |
| 224 | significant digits (unless you understand what you are doing!). |
| 225 | |
| 226 | |
| 227 | Test Case Writing Tips |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 228 | ====================== |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 229 | Writing good test cases is a skilled task and is too complex to discuss in |
| 230 | detail in this short document. Many books have been written on the subject. |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 231 | I'll show my age by suggesting that Glenford Myers' `"The Art of Software |
| 232 | Testing"`_, published in 1979, is still the best introduction to the subject |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 233 | available. It is short (177 pages), easy to read, and discusses the major |
| 234 | elements of software testing, though its publication predates the |
| 235 | object-oriented software revolution, so doesn't cover that subject at all. |
| 236 | Unfortunately, it is very expensive (about $100 new). If you can borrow it |
| 237 | or find it used (around $20), I strongly urge you to pick up a copy. |
| 238 | |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 239 | The most important goal when writing test cases is to break things. A test |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 240 | case that doesn't uncover a bug is much less valuable than one that does. |
| 241 | In designing test cases you should pay attention to the following: |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 242 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 243 | * Your test cases should exercise all the functions and objects defined |
| 244 | in the module, not just the ones meant to be called by users of your |
| 245 | module. This may require you to write test code that uses the module |
| 246 | in ways you don't expect (explicitly calling internal functions, for |
| 247 | example - see test_atexit.py). |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 248 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 249 | * You should consider any boundary values that may tickle exceptional |
| 250 | conditions (e.g. if you were writing regression tests for division, |
| 251 | you might well want to generate tests with numerators and denominators |
| 252 | at the limits of floating point and integer numbers on the machine |
| 253 | performing the tests as well as a denominator of zero). |
Skip Montanaro | 47c60ec | 2000-06-30 06:08:35 +0000 | [diff] [blame] | 254 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 255 | * You should exercise as many paths through the code as possible. This |
| 256 | may not always be possible, but is a goal to strive for. In |
| 257 | particular, when considering if statements (or their equivalent), you |
| 258 | want to create test cases that exercise both the true and false |
| 259 | branches. For loops, you should create test cases that exercise the |
| 260 | loop zero, one and multiple times. |
| 261 | |
| 262 | * You should test with obviously invalid input. If you know that a |
| 263 | function requires an integer input, try calling it with other types of |
| 264 | objects to see how it responds. |
| 265 | |
| 266 | * You should test with obviously out-of-range input. If the domain of a |
| 267 | function is only defined for positive integers, try calling it with a |
| 268 | negative integer. |
| 269 | |
| 270 | * If you are going to fix a bug that wasn't uncovered by an existing |
| 271 | test, try to write a test case that exposes the bug (preferably before |
| 272 | fixing it). |
| 273 | |
Fred Drake | 44b6bd2 | 2000-10-23 16:37:14 +0000 | [diff] [blame] | 274 | * If you need to create a temporary file, you can use the filename in |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 275 | ``test_support.TESTFN`` to do so. It is important to remove the file |
Fred Drake | 44b6bd2 | 2000-10-23 16:37:14 +0000 | [diff] [blame] | 276 | when done; other tests should be able to use the name without cleaning |
| 277 | up after your test. |
| 278 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 279 | .. _"The Art of Software Testing": |
| 280 | http://www.amazon.com/exec/obidos/ISBN=0471043281 |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 281 | |
| 282 | Regression Test Writing Rules |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 283 | ============================= |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 284 | Each test case is different. There is no "standard" form for a Python |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 285 | regression test case, though there are some general rules (note that |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 286 | these mostly apply only to the "classic" tests; unittest_- and doctest_- |
| 287 | based tests should follow the conventions natural to those frameworks):: |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 288 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 289 | * If your test case detects a failure, raise ``TestFailed`` (found in |
| 290 | ``test.test_support``). |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 291 | |
| 292 | * Import everything you'll need as early as possible. |
| 293 | |
| 294 | * If you'll be importing objects from a module that is at least |
| 295 | partially platform-dependent, only import those objects you need for |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 296 | the current test case to avoid spurious ``ImportError`` exceptions |
| 297 | that prevent the test from running to completion. |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 298 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 299 | * Print all your test case results using the ``print`` statement. For |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 300 | non-fatal errors, print an error message (or omit a successful |
| 301 | completion print) to indicate the failure, but proceed instead of |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 302 | raising ``TestFailed``. |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 303 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 304 | * Use ``assert`` sparingly, if at all. It's usually better to just print |
Tim Peters | a48b526 | 2000-08-23 05:28:45 +0000 | [diff] [blame] | 305 | what you got, and rely on regrtest's got-vs-expected comparison to |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 306 | catch deviations from what you expect. ``assert`` statements aren't |
Tim Peters | a48b526 | 2000-08-23 05:28:45 +0000 | [diff] [blame] | 307 | executed at all when regrtest is run in -O mode; and, because they |
| 308 | cause the test to stop immediately, can lead to a long & tedious |
| 309 | test-fix, test-fix, test-fix, ... cycle when things are badly broken |
| 310 | (and note that "badly broken" often includes running the test suite |
| 311 | for the first time on new platforms or under new implementations of |
| 312 | the language). |
| 313 | |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 314 | Miscellaneous |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 315 | ============= |
Barry Warsaw | 1bfab7b | 2002-07-23 19:13:45 +0000 | [diff] [blame] | 316 | There is a test_support module in the test package you can import for |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 317 | your test case. Import this module using either:: |
Barry Warsaw | 04f357c | 2002-07-23 19:04:11 +0000 | [diff] [blame] | 318 | |
| 319 | import test.test_support |
| 320 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 321 | or:: |
Barry Warsaw | 04f357c | 2002-07-23 19:04:11 +0000 | [diff] [blame] | 322 | |
| 323 | from test import test_support |
| 324 | |
| 325 | test_support provides the following useful objects: |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 326 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 327 | * ``TestFailed`` - raise this exception when your regression test detects |
| 328 | a failure. |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 329 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 330 | * ``TestSkipped`` - raise this if the test could not be run because the |
Fred Drake | 62c53dd | 2000-08-21 16:55:57 +0000 | [diff] [blame] | 331 | platform doesn't offer all the required facilities (like large |
| 332 | file support), even if all the required modules are available. |
| 333 | |
Brett Cannon | e6b7033 | 2003-04-30 01:42:35 +0000 | [diff] [blame] | 334 | * ``ResourceDenied`` - this is raised when a test requires a resource that |
| 335 | is not available. Primarily used by 'requires'. |
| 336 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 337 | * ``verbose`` - you can use this variable to control print output. Many |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 338 | modules use it. Search for "verbose" in the test_*.py files to see |
| 339 | lots of examples. |
| 340 | |
Brett Cannon | e6b7033 | 2003-04-30 01:42:35 +0000 | [diff] [blame] | 341 | * ``forget(module_name)`` - attempts to cause Python to "forget" that it |
| 342 | loaded a module and erase any PYC files. |
| 343 | |
| 344 | * ``is_resource_enabled(resource)`` - Returns a boolean based on whether |
| 345 | the resource is enabled or not. |
| 346 | |
| 347 | * ``requires(resource [, msg])`` - if the required resource is not |
| 348 | available the ResourceDenied exception is raised. |
| 349 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 350 | * ``verify(condition, reason='test failed')``. Use this instead of:: |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 351 | |
| 352 | assert condition[, reason] |
| 353 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 354 | ``verify()`` has two advantages over ``assert``: it works even in -O |
| 355 | mode, and it raises ``TestFailed`` on failure instead of |
| 356 | ``AssertionError``. |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 357 | |
Brett Cannon | e6b7033 | 2003-04-30 01:42:35 +0000 | [diff] [blame] | 358 | * ``have_unicode`` - true if Unicode is available, false otherwise. |
| 359 | |
| 360 | * ``is_jython`` - true if the interpreter is Jython, false otherwise. |
| 361 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 362 | * ``TESTFN`` - a string that should always be used as the filename when |
| 363 | you need to create a temp file. Also use ``try``/``finally`` to |
| 364 | ensure that your temp files are deleted before your test completes. |
| 365 | Note that you cannot unlink an open file on all operating systems, so |
| 366 | also be sure to close temp files before trying to unlink them. |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 367 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 368 | * ``sortdict(dict)`` - acts like ``repr(dict.items())``, but sorts the |
| 369 | items first. This is important when printing a dict value, because |
| 370 | the order of items produced by ``dict.items()`` is not defined by the |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 371 | language. |
| 372 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 373 | * ``findfile(file)`` - you can call this function to locate a file |
| 374 | somewhere along sys.path or in the Lib/test tree - see |
Guido van Rossum | 33d2689 | 2007-08-05 15:29:28 +0000 | [diff] [blame] | 375 | test_ossaudiodev.py for an example of its use. |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 376 | |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 377 | * ``fcmp(x,y)`` - you can call this function to compare two floating |
| 378 | point numbers when you expect them to only be approximately equal |
| 379 | withing a fuzz factor (``test_support.FUZZ``, which defaults to 1e-6). |
Skip Montanaro | e9e5dcd | 2000-07-19 17:19:49 +0000 | [diff] [blame] | 380 | |
Thomas Wouters | 89f507f | 2006-12-13 04:49:30 +0000 | [diff] [blame] | 381 | * ``check_syntax_error(testcase, statement)`` - make sure that the |
| 382 | statement is *not* correct Python syntax. |
Brett Cannon | e6b7033 | 2003-04-30 01:42:35 +0000 | [diff] [blame] | 383 | |
Tim Peters | a48b526 | 2000-08-23 05:28:45 +0000 | [diff] [blame] | 384 | |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 385 | Some Non-Obvious regrtest Features |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 386 | ================================== |
Tim Peters | f5f6c43 | 2001-05-23 07:46:36 +0000 | [diff] [blame] | 387 | * Automagic test detection: When you create a new test file |
| 388 | test_spam.py, you do not need to modify regrtest (or anything else) |
| 389 | to advertise its existence. regrtest searches for and runs all |
| 390 | modules in the test directory with names of the form test_xxx.py. |
| 391 | |
| 392 | * Miranda output: If, when running test_spam.py, regrtest does not |
| 393 | find an expected-output file test/output/test_spam, regrtest |
| 394 | pretends that it did find one, containing the single line |
| 395 | |
| 396 | test_spam |
| 397 | |
| 398 | This allows new tests that don't expect to print anything to stdout |
| 399 | to not bother creating expected-output files. |
| 400 | |
| 401 | * Two-stage testing: To run test_spam.py, regrtest imports test_spam |
| 402 | as a module. Most tests run to completion as a side-effect of |
| 403 | getting imported. After importing test_spam, regrtest also executes |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 404 | ``test_spam.test_main()``, if test_spam has a ``test_main`` attribute. |
Fred Drake | b2ad1c8 | 2001-09-28 20:05:25 +0000 | [diff] [blame] | 405 | This is rarely required with the "traditional" Python tests, and |
| 406 | you shouldn't create a module global with name test_main unless |
| 407 | you're specifically exploiting this gimmick. This usage does |
Guido van Rossum | d8faa36 | 2007-04-27 19:54:29 +0000 | [diff] [blame] | 408 | prove useful with unittest-based tests as well, however; defining |
Neal Norwitz | deaba57 | 2002-11-27 15:47:10 +0000 | [diff] [blame] | 409 | a ``test_main()`` which is run by regrtest and a script-stub in the |
| 410 | test module ("``if __name__ == '__main__': test_main()``") allows |
Fred Drake | b2ad1c8 | 2001-09-28 20:05:25 +0000 | [diff] [blame] | 411 | the test to be used like any other Python test and also work |
| 412 | with the unittest.py-as-a-script approach, allowing a developer |
| 413 | to run specific tests from the command line. |