Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 1 | \section{\module{unittest} --- |
| 2 | Unit testing framework} |
| 3 | |
| 4 | \declaremodule{standard}{unittest} |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 5 | \modulesynopsis{Unit testing framework for Python.} |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 6 | \moduleauthor{Steve Purcell}{stephen\textunderscore{}purcell@yahoo.com} |
| 7 | \sectionauthor{Steve Purcell}{stephen\textunderscore{}purcell@yahoo.com} |
| 8 | \sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org} |
| 9 | |
| 10 | |
| 11 | The Python unit testing framework, often referred to as ``PyUnit,'' is |
| 12 | a Python language version of JUnit, by Kent Beck and Erich Gamma. |
| 13 | JUnit is, in turn, a Java version of Kent's Smalltalk testing |
| 14 | framework. Each is the de facto standard unit testing framework for |
| 15 | its respective language. |
| 16 | |
| 17 | PyUnit supports test automation, sharing of setup and shutdown code |
| 18 | for tests, aggregation of tests into collections, and independence of |
| 19 | the tests from the reporting framework. The \module{unittest} module |
| 20 | provides classes that make it easy to support these qualities for a |
| 21 | set of tests. |
| 22 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 23 | To achieve this, PyUnit supports some important concepts: |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 24 | |
| 25 | \begin{definitions} |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 26 | \term{test fixture} |
| 27 | A \dfn{test fixture} represents the preparation needed to perform one |
| 28 | or more tests, and any associate cleanup actions. This may involve, |
| 29 | for example, creating temporary or proxy databases, directories, or |
| 30 | starting a server process. |
| 31 | |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 32 | \term{test case} |
| 33 | A \dfn{test case} is the smallest unit of testing. It checks for a |
| 34 | specific response to a particular set of inputs. PyUnit provides a |
| 35 | base class, \class{TestCase}, which may be used to create new test |
| 36 | cases. |
| 37 | |
| 38 | \term{test suite} |
| 39 | A \dfn{test suite} is a collection of test cases, test suites, or |
| 40 | both. It is used to aggregate tests that should be executed |
| 41 | together. |
| 42 | |
| 43 | \term{test runner} |
| 44 | A \dfn{test runner} is a component which orchestrates the execution of |
| 45 | tests and provides the outcome to the user. The runner may use a |
| 46 | graphical interface, a textual interface, or return a special value to |
| 47 | indicate the results of executing the tests. |
| 48 | \end{definitions} |
| 49 | |
| 50 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 51 | The test case and test fixture concepts are supported through the |
| 52 | \class{TestCase} and \class{FunctionTestCase} classes; the former |
| 53 | should be used when creating new tests, and the later can be used when |
| 54 | integrating existing test code with a PyUnit-driven framework. When |
| 55 | building test fixtures using \class{TestCase}, the \method{setUp()} |
| 56 | and \method{tearDown()} methods can be overridden to provide |
| 57 | initialization and cleanup for the fixture. With |
| 58 | \class{FunctionTestCase}, existing functions can be passed to the |
| 59 | constructor for these purposes. When the test is run, the |
| 60 | fixture initialization is run first; if it succeeds, the cleanup |
| 61 | method is run after the test has been executed, regardless of the |
| 62 | outcome of the test. Each instance of the \class{TestCase} will only |
| 63 | be used to run a single test method, so a new fixture is created for |
| 64 | each test. |
| 65 | |
| 66 | Test suites are implemented by the \class{TestSuite} class. This |
| 67 | class allows individual tests and test suites to be aggregated; when |
| 68 | the suite is executed, all tests added directly to the suite and in |
| 69 | ``child'' test suites are run. |
| 70 | |
| 71 | A test runner is an object that provides a single method, |
| 72 | \method{run()}, which accepts a \class{TestCase} or \class{TestSuite} |
| 73 | object as a parameter, and returns a result object. The class |
| 74 | \class{TestResult} is provided for use as the result object. PyUnit |
| 75 | provide the \class{TextTestRunner} as an example test runner which |
| 76 | reports test results on the standard error stream by default. |
| 77 | Alternate runners can be implemented for other environments (such as |
| 78 | graphical environments) without any need to derive from a specific |
| 79 | class. |
| 80 | |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 81 | |
| 82 | \begin{seealso} |
| 83 | \seetitle[http://pyunit.sourceforge.net/]{PyUnit Web Site}{The |
| 84 | source for further information on PyUnit.} |
| 85 | \seetitle[http://www.XProgramming.com/testfram.htm]{Simple Smalltalk |
| 86 | Testing: With Patterns}{Kent Beck's original paper on |
| 87 | testing frameworks using the pattern shared by |
| 88 | \module{unittest}.} |
| 89 | \end{seealso} |
| 90 | |
| 91 | |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 92 | \subsection{Organizing test code |
| 93 | \label{organizing-tests}} |
| 94 | |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 95 | The basic building blocks of unit testing are \dfn{test cases} --- |
| 96 | single scenarios that must be set up and checked for correctness. In |
| 97 | PyUnit, test cases are represented by instances of the |
| 98 | \class{TestCase} class in the \refmodule{unittest} module. To make |
| 99 | your own test cases you must write subclasses of \class{TestCase}, or |
| 100 | use \class{FunctionTestCase}. |
| 101 | |
| 102 | An instance of a \class{TestCase}-derived class is an object that can |
| 103 | completely run a single test method, together with optional set-up |
| 104 | and tidy-up code. |
| 105 | |
| 106 | The testing code of a \class{TestCase} instance should be entirely |
| 107 | self contained, such that it can be run either in isolation or in |
| 108 | arbitrary combination with any number of other test cases. |
| 109 | |
| 110 | The simplest test case subclass will simply override the |
| 111 | \method{runTest()} method in order to perform specific testing code: |
| 112 | |
| 113 | \begin{verbatim} |
| 114 | import unittest |
| 115 | |
| 116 | class DefaultWidgetSizeTestCase(unittest.TestCase): |
| 117 | def runTest(self): |
| 118 | widget = Widget("The widget") |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 119 | self.failUnless(widget.size() == (50,50), 'incorrect default size') |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 120 | \end{verbatim} |
| 121 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 122 | Note that in order to test something, we use the one of the |
| 123 | \method{assert*()} or \method{fail*()} methods provided by the |
| 124 | \class{TestCase} base class. If the test fails when the test case |
| 125 | runs, an exception will be raised, and the testing framework will |
| 126 | identify the test case as a \dfn{failure}. Other exceptions that do |
| 127 | not arise from checks made through the \method{assert*()} and |
| 128 | \method{fail*()} methods are identified by the testing framework as |
| 129 | dfn{errors}. |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 130 | |
| 131 | The way to run a test case will be described later. For now, note |
| 132 | that to construct an instance of such a test case, we call its |
| 133 | constructor without arguments: |
| 134 | |
| 135 | \begin{verbatim} |
| 136 | testCase = DefaultWidgetSizeTestCase() |
| 137 | \end{verbatim} |
| 138 | |
| 139 | Now, such test cases can be numerous, and their set-up can be |
| 140 | repetitive. In the above case, constructing a ``Widget'' in each of |
| 141 | 100 Widget test case subclasses would mean unsightly duplication. |
| 142 | |
| 143 | Luckily, we can factor out such set-up code by implementing a method |
| 144 | called \method{setUp()}, which the testing framework will |
| 145 | automatically call for us when we run the test: |
| 146 | |
| 147 | \begin{verbatim} |
| 148 | import unittest |
| 149 | |
| 150 | class SimpleWidgetTestCase(unittest.TestCase): |
| 151 | def setUp(self): |
| 152 | self.widget = Widget("The widget") |
| 153 | |
| 154 | class DefaultWidgetSizeTestCase(SimpleWidgetTestCase): |
| 155 | def runTest(self): |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 156 | self.failUnless(self.widget.size() == (50,50), |
| 157 | 'incorrect default size') |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 158 | |
| 159 | class WidgetResizeTestCase(SimpleWidgetTestCase): |
| 160 | def runTest(self): |
| 161 | self.widget.resize(100,150) |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 162 | self.failUnless(self.widget.size() == (100,150), |
| 163 | 'wrong size after resize') |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 164 | \end{verbatim} |
| 165 | |
| 166 | If the \method{setUp()} method raises an exception while the test is |
| 167 | running, the framework will consider the test to have suffered an |
| 168 | error, and the \method{runTest()} method will not be executed. |
| 169 | |
| 170 | Similarly, we can provide a \method{tearDown()} method that tidies up |
| 171 | after the \method{runTest()} method has been run: |
| 172 | |
| 173 | \begin{verbatim} |
| 174 | import unittest |
| 175 | |
| 176 | class SimpleWidgetTestCase(unittest.TestCase): |
| 177 | def setUp(self): |
| 178 | self.widget = Widget("The widget") |
| 179 | |
| 180 | def tearDown(self): |
| 181 | self.widget.dispose() |
| 182 | self.widget = None |
| 183 | \end{verbatim} |
| 184 | |
| 185 | If \method{setUp()} succeeded, the \method{tearDown()} method will be |
| 186 | run regardless of whether or not \method{runTest()} succeeded. |
| 187 | |
| 188 | Such a working environment for the testing code is called a |
| 189 | \dfn{fixture}. |
| 190 | |
| 191 | Often, many small test cases will use the same fixture. In this case, |
| 192 | we would end up subclassing \class{SimpleWidgetTestCase} into many |
| 193 | small one-method classes such as |
| 194 | \class{DefaultWidgetSizeTestCase}. This is time-consuming and |
| 195 | discouraging, so in the same vein as JUnit, PyUnit provides a simpler |
| 196 | mechanism: |
| 197 | |
| 198 | \begin{verbatim} |
| 199 | import unittest |
| 200 | |
| 201 | class WidgetTestCase(unittest.TestCase): |
| 202 | def setUp(self): |
| 203 | self.widget = Widget("The widget") |
| 204 | |
| 205 | def tearDown(self): |
| 206 | self.widget.dispose() |
| 207 | self.widget = None |
| 208 | |
| 209 | def testDefaultSize(self): |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 210 | self.failUnless(self.widget.size() == (50,50), |
| 211 | 'incorrect default size') |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 212 | |
| 213 | def testResize(self): |
| 214 | self.widget.resize(100,150) |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 215 | self.failUnless(self.widget.size() == (100,150), |
| 216 | 'wrong size after resize') |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 217 | \end{verbatim} |
| 218 | |
| 219 | Here we have not provided a \method{runTest()} method, but have |
| 220 | instead provided two different test methods. Class instances will now |
| 221 | each run one of the \method{test*()} methods, with \code{self.widget} |
| 222 | created and destroyed separately for each instance. When creating an |
| 223 | instance we must specify the test method it is to run. We do this by |
| 224 | passing the method name in the constructor: |
| 225 | |
| 226 | \begin{verbatim} |
| 227 | defaultSizeTestCase = WidgetTestCase("testDefaultSize") |
| 228 | resizeTestCase = WidgetTestCase("testResize") |
| 229 | \end{verbatim} |
| 230 | |
| 231 | Test case instances are grouped together according to the features |
| 232 | they test. PyUnit provides a mechanism for this: the \class{test |
| 233 | suite}, represented by the class \class{TestSuite} in the |
| 234 | \refmodule{unittest} module: |
| 235 | |
| 236 | \begin{verbatim} |
| 237 | widgetTestSuite = unittest.TestSuite() |
| 238 | widgetTestSuite.addTest(WidgetTestCase("testDefaultSize")) |
| 239 | widgetTestSuite.addTest(WidgetTestCase("testResize")) |
| 240 | \end{verbatim} |
| 241 | |
| 242 | For the ease of running tests, as we will see later, it is a good |
| 243 | idea to provide in each test module a callable object that returns a |
| 244 | pre-built test suite: |
| 245 | |
| 246 | \begin{verbatim} |
| 247 | def suite(): |
| 248 | suite = unittest.TestSuite() |
| 249 | suite.addTest(WidgetTestCase("testDefaultSize")) |
| 250 | suite.addTest(WidgetTestCase("testResize")) |
| 251 | return suite |
| 252 | \end{verbatim} |
| 253 | |
| 254 | or even: |
| 255 | |
| 256 | \begin{verbatim} |
| 257 | class WidgetTestSuite(unittest.TestSuite): |
| 258 | def __init__(self): |
| 259 | unittest.TestSuite.__init__(self,map(WidgetTestCase, |
| 260 | ("testDefaultSize", |
| 261 | "testResize"))) |
| 262 | \end{verbatim} |
| 263 | |
| 264 | (The latter is admittedly not for the faint-hearted!) |
| 265 | |
| 266 | Since it is a common pattern to create a \class{TestCase} subclass |
| 267 | with many similarly named test functions, there is a convenience |
| 268 | function called \function{makeSuite()} provided in the |
| 269 | \refmodule{unittest} module that constructs a test suite that |
| 270 | comprises all of the test cases in a test case class: |
| 271 | |
| 272 | \begin{verbatim} |
| 273 | suite = unittest.makeSuite(WidgetTestCase,'test') |
| 274 | \end{verbatim} |
| 275 | |
| 276 | Note that when using the \function{makeSuite()} function, the order in |
| 277 | which the various test cases will be run by the test suite is the |
| 278 | order determined by sorting the test function names using the |
| 279 | \function{cmp()} built-in function. |
| 280 | |
| 281 | Often it is desirable to group suites of test cases together, so as to |
| 282 | run tests for the whole system at once. This is easy, since |
| 283 | \class{TestSuite} instances can be added to a \class{TestSuite} just |
| 284 | as \class{TestCase} instances can be added to a \class{TestSuite}: |
| 285 | |
| 286 | \begin{verbatim} |
| 287 | suite1 = module1.TheTestSuite() |
| 288 | suite2 = module2.TheTestSuite() |
| 289 | alltests = unittest.TestSuite((suite1, suite2)) |
| 290 | \end{verbatim} |
| 291 | |
| 292 | You can place the definitions of test cases and test suites in the |
| 293 | same modules as the code they are to test (e.g.\ \file{widget.py}), |
| 294 | but there are several advantages to placing the test code in a |
| 295 | separate module, such as \file{widgettests.py}: |
| 296 | |
| 297 | \begin{itemize} |
| 298 | \item The test module can be run standalone from the command line. |
| 299 | \item The test code can more easily be separated from shipped code. |
| 300 | \item There is less temptation to change test code to fit the code. |
| 301 | it tests without a good reason. |
| 302 | \item Test code should be modified much less frequently than the |
| 303 | code it tests. |
| 304 | \item Tested code can be refactored more easily. |
| 305 | \item Tests for modules written in C must be in separate modules |
| 306 | anyway, so why not be consistent? |
| 307 | \item If the testing strategy changes, there is no need to change |
| 308 | the source code. |
| 309 | \end{itemize} |
| 310 | |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 311 | |
| 312 | \subsection{Re-using old test code |
| 313 | \label{legacy-unit-tests}} |
| 314 | |
| 315 | Some users will find that they have existing test code that they would |
| 316 | like to run from PyUnit, without converting every old test function to |
| 317 | a \class{TestCase} subclass. |
| 318 | |
| 319 | For this reason, PyUnit provides a \class{FunctionTestCase} class. |
| 320 | This subclass of \class{TestCase} can be used to wrap an existing test |
| 321 | function. Set-up and tear-down functions can also optionally be |
| 322 | wrapped. |
| 323 | |
| 324 | Given the following test function: |
| 325 | |
| 326 | \begin{verbatim} |
| 327 | def testSomething(): |
| 328 | something = makeSomething() |
| 329 | assert something.name is not None |
| 330 | # ... |
| 331 | \end{verbatim} |
| 332 | |
| 333 | one can create an equivalent test case instance as follows: |
| 334 | |
| 335 | \begin{verbatim} |
| 336 | testcase = unittest.FunctionTestCase(testSomething) |
| 337 | \end{verbatim} |
| 338 | |
| 339 | If there are additional set-up and tear-down methods that should be |
| 340 | called as part of the test case's operation, they can also be provided: |
| 341 | |
| 342 | \begin{verbatim} |
| 343 | testcase = unittest.FunctionTestCase(testSomething, |
| 344 | setUp=makeSomethingDB, |
| 345 | tearDown=deleteSomethingDB) |
| 346 | \end{verbatim} |
| 347 | |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 348 | \strong{Note:} PyUnit supports the use of \exception{AssertionError} |
| 349 | as an indicator of test failure, but does not recommend it. Future |
| 350 | versions may treat \exception{AssertionError} differently. |
| 351 | |
| 352 | |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 353 | \subsection{Classes and functions |
| 354 | \label{unittest-contents}} |
| 355 | |
| 356 | \begin{classdesc}{TestCase}{} |
| 357 | Instances of the \class{TestCase} class represent the smallest |
| 358 | testable units in a set of tests. This class is intended to be used |
| 359 | as a base class, with specific tests being implemented by concrete |
| 360 | subclasses. This class implements the interface needed by the test |
| 361 | runner to allow it to drive the test, and methods that the test code |
| 362 | can use to check for and report various kinds of failures. |
| 363 | \end{classdesc} |
| 364 | |
| 365 | \begin{classdesc}{FunctionTestCase}{testFunc\optional{, |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 366 | setUp\optional{, tearDown\optional{, description}}}} |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 367 | This class implements the portion of the \class{TestCase} interface |
| 368 | which allows the test runner to drive the test, but does not provide |
| 369 | the methods which test code can use to check and report errors. |
| 370 | This is used to create test cases using legacy test code, allowing |
| 371 | it to be integrated into a \refmodule{unittest}-based test |
| 372 | framework. |
| 373 | \end{classdesc} |
| 374 | |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 375 | \begin{classdesc}{TestSuite}{\optional{tests}} |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 376 | This class represents an aggregation of individual tests cases and |
| 377 | test suites. The class presents the interface needed by the test |
| 378 | runner to allow it to be run as any other test case, but all the |
| 379 | contained tests and test suites are executed. Additional methods |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 380 | are provided to add test cases and suites to the aggregation. If |
| 381 | \var{tests} is given, it must be a sequence of individual tests that |
| 382 | will be added to the suite. |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 383 | \end{classdesc} |
| 384 | |
| 385 | \begin{classdesc}{TestLoader}{} |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 386 | This class is responsible for loading tests according to various |
| 387 | criteria and returning them wrapped in a \class{TestSuite}. |
| 388 | It can load all tests within a given module or \class{TestCase} |
| 389 | class. When loading from a module, it considers all |
| 390 | \class{TestCase}-derived classes. For each such class, it creates |
| 391 | an instance for each method with a name beginning with the string |
| 392 | \samp{test}. |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 393 | \end{classdesc} |
| 394 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 395 | \begin{datadesc}{defaultTestLoader} |
| 396 | Instance of the \class{TestLoader} class which can be shared. If no |
| 397 | customization of the \class{TestLoader} is needed, this instance can |
| 398 | always be used instead of creating new instances. |
| 399 | \end{datadesc} |
| 400 | |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 401 | \begin{classdesc}{TextTestRunner}{\optional{stream\optional{, |
| 402 | descriptions\optional{, verbosity}}}} |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 403 | A basic test runner implementation which prints results on standard |
| 404 | output. It has a few configurable parameters, but is essentially |
| 405 | very simple. Graphical applications which run test suites should |
| 406 | provide alternate implementations. |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 407 | \end{classdesc} |
| 408 | |
| 409 | \begin{funcdesc}{main}{\optional{module\optional{, |
| 410 | defaultTest\optional{, argv\optional{, |
| 411 | testRunner\optional{, testRunner}}}}}} |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 412 | A command-line program that runs a set of tests; this is primarily |
| 413 | for making test modules conveniently executable. The simplest use |
| 414 | for this function is: |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 415 | |
| 416 | \begin{verbatim} |
| 417 | if __name__ == '__main__': |
| 418 | unittest.main() |
| 419 | \end{verbatim} |
| 420 | \end{funcdesc} |
| 421 | |
| 422 | |
| 423 | \subsection{TestCase Objects |
| 424 | \label{testcase-objects}} |
| 425 | |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 426 | Each \class{TestCase} instance represents a single test, but each |
| 427 | concrete subclass may be used to define multiple tests --- the |
| 428 | concrete class represents a single test fixture. The fixture is |
| 429 | created and cleaned up for each test case. |
| 430 | |
| 431 | \class{TestCase} instances provide three groups of methods: one group |
| 432 | used to run the test, another used by the test implementation to |
| 433 | check conditions and report failures, and some inquiry methods |
| 434 | allowing information about the test itself to be gathered. |
| 435 | |
| 436 | Methods in the first group are: |
| 437 | |
| 438 | \begin{methoddesc}[TestCase]{setUp}{} |
| 439 | Method called to prepare the test fixture. This is called |
| 440 | immediately before calling the test method; any exception raised by |
| 441 | this method will be considered an error rather than a test failure. |
| 442 | The default implementation does nothing. |
| 443 | \end{methoddesc} |
| 444 | |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 445 | \begin{methoddesc}[TestCase]{tearDown}{} |
| 446 | Method called immediately after the test method has been called and |
| 447 | the result recorded. This is called even if the test method raised |
| 448 | an exception, so the implementation in subclasses may need to be |
| 449 | particularly careful about checking internal state. Any exception |
| 450 | raised by this method will be considered an error rather than a test |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 451 | failure. This method will only be called if the \method{setUp()} |
| 452 | succeeds, regardless of the outcome of the test method. |
| 453 | The default implementation does nothing. |
| 454 | \end{methoddesc} |
| 455 | |
| 456 | \begin{methoddesc}[TestCase]{run}{\optional{result}} |
| 457 | Run the test, collecting the result into the test result object |
| 458 | passed as \var{result}. If \var{result} is omitted or \code{None}, |
| 459 | a temporary result object is created and used, but is not made |
| 460 | available to the caller. This is equivalent to simply calling the |
| 461 | \class{TestCase} instance. |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 462 | \end{methoddesc} |
| 463 | |
| 464 | \begin{methoddesc}[TestCase]{debug}{} |
| 465 | Run the test without collecting the result. This allows exceptions |
| 466 | raised by the test to be propogated to the caller, and can be used |
| 467 | to support running tests under a debugger. |
| 468 | \end{methoddesc} |
| 469 | |
| 470 | |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 471 | The test code can use any of the following methods to check for and |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 472 | report failures. |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 473 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 474 | \begin{methoddesc}[TestCase]{assert_}{expr\optional{, msg}} |
| 475 | \methodline{failUnless}{expr\optional{, msg}} |
| 476 | Signal a test failure if \var{expr} is false; the explanation for |
| 477 | the error will be \var{msg} if given, otherwise it will be |
| 478 | \code{None}. |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 479 | \end{methoddesc} |
| 480 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 481 | \begin{methoddesc}[TestCase]{assertEqual}{first, second\optional{, msg}} |
| 482 | \methodline{failUnlessEqual}{first, second\optional{, msg}} |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 483 | Test that \var{first} and \var{second} are equal. If the values do |
| 484 | not compare equal, the test will fail with the explanation given by |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 485 | \var{msg}, or \code{None}. Note that using \method{failUnlessEqual()} |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 486 | improves upon doing the comparison as the first parameter to |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 487 | \method{failUnless()}: the default value for \var{msg} can be |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 488 | computed to include representations of both \var{first} and |
| 489 | \var{second}. |
| 490 | \end{methoddesc} |
| 491 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 492 | \begin{methoddesc}[TestCase]{assertNotEqual}{first, second\optional{, msg}} |
| 493 | \methodline{failIfEqual}{first, second\optional{, msg}} |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 494 | Test that \var{first} and \var{second} are not equal. If the values |
| 495 | do compare equal, the test will fail with the explanation given by |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 496 | \var{msg}, or \code{None}. Note that using \method{failIfEqual()} |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 497 | improves upon doing the comparison as the first parameter to |
| 498 | \method{failUnless()} is that the default value for \var{msg} can be |
| 499 | computed to include representations of both \var{first} and |
| 500 | \var{second}. |
| 501 | \end{methoddesc} |
| 502 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 503 | \begin{methoddesc}[TestCase]{assertRaises}{exception, callable, \moreargs} |
| 504 | \methodline{failUnlessRaises}{exception, callable, \moreargs} |
| 505 | Test that an exception is raised when \var{callable} is called with |
| 506 | any positional or keyword arguments that are also passed to |
| 507 | \method{assertRaises()}. The test passes if \var{exception} is |
| 508 | raised, is an error if another exception is raised, or fails if no |
| 509 | exception is raised. To catch any of a group of exceptions, a tuple |
| 510 | containing the exception classes may be passed as \var{exception}. |
| 511 | \end{methoddesc} |
| 512 | |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 513 | \begin{methoddesc}[TestCase]{failIf}{expr\optional{, msg}} |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 514 | The inverse of the \method{failUnless()} method is the |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 515 | \method{failIf()} method. This signals a test failure if \var{expr} |
| 516 | is true, with \var{msg} or \code{None} for the error message. |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 517 | \end{methoddesc} |
| 518 | |
| 519 | \begin{methoddesc}[TestCase]{fail}{\optional{msg}} |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 520 | Signals a test failure unconditionally, with \var{msg} or |
| 521 | \code{None} for the error message. |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 522 | \end{methoddesc} |
| 523 | |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 524 | \begin{memberdesc}[TestCase]{failureException} |
| 525 | This class attribute gives the exception raised by the |
| 526 | \method{test()} method. If a test framework needs to use a |
| 527 | specialized exception, possibly to carry additional information, it |
| 528 | must subclass this exception in order to ``play fair'' with the |
| 529 | framework. The initial value of this attribute is |
| 530 | \exception{AssertionError}. |
| 531 | \end{memberdesc} |
| 532 | |
Fred Drake | 29be701 | 2001-04-10 22:25:06 +0000 | [diff] [blame] | 533 | |
| 534 | Testing frameworks can use the following methods to collect |
| 535 | information on the test: |
| 536 | |
| 537 | \begin{methoddesc}[TestCase]{countTestCases}{} |
| 538 | Return the number of tests represented by the this test object. For |
| 539 | \class{TestCase} instances, this will always be \code{1}, but this |
| 540 | method is also implemented by the \class{TestSuite} class, which can |
| 541 | return larger values. |
| 542 | \end{methoddesc} |
| 543 | |
| 544 | \begin{methoddesc}[TestCase]{defaultTestResult}{} |
| 545 | Return the default type of test result object to be used to run this |
| 546 | test. |
| 547 | \end{methoddesc} |
| 548 | |
| 549 | \begin{methoddesc}[TestCase]{id}{} |
| 550 | Return a string identifying the specific test case. This is usually |
| 551 | the full name of the test method, including the module and class |
| 552 | names. |
| 553 | \end{methoddesc} |
| 554 | |
| 555 | \begin{methoddesc}[TestCase]{shortDescription}{} |
| 556 | Returns a one-line description of the test, or \code{None} if no |
| 557 | description has been provided. The default implementation of this |
| 558 | method returns the first line of the test method's docstring, if |
| 559 | available, or \code{None}. |
| 560 | \end{methoddesc} |
| 561 | |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 562 | |
| 563 | \subsection{TestSuite Objects |
| 564 | \label{testsuite-objects}} |
| 565 | |
| 566 | \class{TestSuite} objects behave much like \class{TestCase} objects, |
| 567 | except they do not actually implement a test. Instead, they are used |
| 568 | to aggregate tests into groups that should be run together. Some |
| 569 | additional methods are available to add tests to \class{TestSuite} |
| 570 | instances: |
| 571 | |
| 572 | \begin{methoddesc}[TestSuite]{addTest}{test} |
| 573 | Add a \class{TestCase} or \class{TestSuite} to the set of tests that |
| 574 | make up the suite. |
| 575 | \end{methoddesc} |
| 576 | |
| 577 | \begin{methoddesc}[TestSuite]{addTests}{tests} |
| 578 | Add all the tests from a sequence of \class{TestCase} and |
| 579 | \class{TestSuite} instances to this test suite. |
| 580 | \end{methoddesc} |
| 581 | |
Fred Drake | 4d17b30 | 2001-09-06 15:51:56 +0000 | [diff] [blame^] | 582 | The \method{run()} method is also slightly different: |
| 583 | |
| 584 | \begin{methoddesc}[TestSuite]{run}{result} |
| 585 | Run the tests associated with this suite, collecting the result into |
| 586 | the test result object passed as \var{result}. Note that unlike |
| 587 | \method{TestCase.run()}, \method{TestSuite.run()} requires the |
| 588 | result object to be passed in. |
| 589 | \end{methoddesc} |
| 590 | |
| 591 | In the typical usage of a \class{TestSuite} object, the \method{run()} |
| 592 | method is invoked by a \class{TestRunner} rather than by the end-user |
| 593 | test harness. |
| 594 | |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 595 | |
| 596 | \subsection{TestResult Objects |
| 597 | \label{testresult-objects}} |
| 598 | |
| 599 | A \class{TestResult} object stores the results of a set of tests. The |
| 600 | \class{TestCase} and \class{TestSuite} classes ensure that results are |
| 601 | properly stored; test authors do not need to worry about recording the |
| 602 | outcome of tests. |
| 603 | |
| 604 | Testing frameworks built on top of \refmodule{unittest} may want |
| 605 | access to the \class{TestResult} object generated by running a set of |
| 606 | tests for reporting purposes; a \class{TestResult} instance is |
| 607 | returned by the \method{TestRunner.run()} method for this purpose. |
| 608 | |
| 609 | Each instance holds the total number of tests run, and collections of |
| 610 | failures and errors that occurred among those test runs. The |
| 611 | collections contain tuples of \code{(\var{testcase}, |
| 612 | \var{exceptioninfo})}, where \var{exceptioninfo} is a tuple as |
| 613 | returned by \function{sys.exc_info()}. |
| 614 | |
| 615 | \class{TestResult} instances have the following attributes that will |
| 616 | be of interest when inspecting the results of running a set of tests: |
| 617 | |
| 618 | \begin{memberdesc}[TestResult]{errors} |
| 619 | A list containing pairs of \class{TestCase} instances and the |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 620 | \function{sys.exc_info()} results for tests which raised an |
| 621 | exception but did not signal a test failure. |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 622 | \end{memberdesc} |
| 623 | |
| 624 | \begin{memberdesc}[TestResult]{failures} |
| 625 | A list containing pairs of \class{TestCase} instances and the |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 626 | \function{sys.exc_info()} results for tests which signalled a |
| 627 | failure in the code under test. |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 628 | \end{memberdesc} |
| 629 | |
| 630 | \begin{memberdesc}[TestResult]{testsRun} |
| 631 | The number of tests which have been started. |
| 632 | \end{memberdesc} |
| 633 | |
| 634 | \begin{methoddesc}[TestResult]{wasSuccessful}{} |
| 635 | Returns true if all tests run so far have passed, otherwise returns |
| 636 | false. |
| 637 | \end{methoddesc} |
| 638 | |
| 639 | |
| 640 | The following methods of the \class{TestResult} class are used to |
| 641 | maintain the internal data structures, and mmay be extended in |
| 642 | subclasses to support additional reporting requirements. This is |
Fred Drake | 8ee679f | 2001-07-14 02:50:55 +0000 | [diff] [blame] | 643 | particularly useful in building tools which support interactive |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 644 | reporting while tests are being run. |
| 645 | |
| 646 | \begin{methoddesc}[TestResult]{startTest}{test} |
| 647 | Called when the test case \var{test} is about to be run. |
| 648 | \end{methoddesc} |
| 649 | |
| 650 | \begin{methoddesc}[TestResult]{stopTest}{test} |
| 651 | Called when the test case \var{test} has been executed, regardless |
| 652 | of the outcome. |
| 653 | \end{methoddesc} |
| 654 | |
| 655 | \begin{methoddesc}[TestResult]{addError}{test, err} |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 656 | Called when the test case \var{test} raises an exception without |
| 657 | signalling a test failure. \var{err} is a tuple of the form |
| 658 | returned by \function{sys.exc_info()}: \code{(\var{type}, |
| 659 | \var{value}, \var{traceback})}. |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 660 | \end{methoddesc} |
| 661 | |
| 662 | \begin{methoddesc}[TestResult]{addFailure}{test, err} |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 663 | Called when the test case \var{test} signals a failure. |
Fred Drake | 0056a42 | 2001-04-12 04:50:06 +0000 | [diff] [blame] | 664 | \var{err} is a tuple of the form returned by |
| 665 | \function{sys.exc_info()}: \code{(\var{type}, \var{value}, |
| 666 | \var{traceback})}. |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 667 | \end{methoddesc} |
| 668 | |
| 669 | \begin{methoddesc}[TestResult]{addSuccess}{test} |
| 670 | This method is called for a test that does not fail; \var{test} is |
| 671 | the test case object. |
| 672 | \end{methoddesc} |
| 673 | |
| 674 | |
| 675 | One additional method is available for \class{TestResult} objects: |
| 676 | |
| 677 | \begin{methoddesc}[TestResult]{stop}{} |
| 678 | This method can be called to signal that the set of tests being run |
| 679 | should be aborted. Once this has been called, the |
| 680 | \class{TestRunner} object return to its caller without running any |
| 681 | additional tests. This is used by the \class{TextTestRunner} class |
| 682 | to stop the test framework when the user signals an interrupt from |
Fred Drake | 8ee679f | 2001-07-14 02:50:55 +0000 | [diff] [blame] | 683 | the keyboard. Interactive tools which provide runners can use this |
| 684 | in a similar manner. |
Fred Drake | b9ad228 | 2001-04-07 05:41:39 +0000 | [diff] [blame] | 685 | \end{methoddesc} |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 686 | |
| 687 | |
| 688 | \subsection{TestLoader Objects |
| 689 | \label{testloader-objects}} |
| 690 | |
| 691 | The \class{TestLoader} class is used to create test suites from |
| 692 | classes and modules. Normally, there is no need to create an instance |
| 693 | of this class; the \refmodule{unittest} module provides an instance |
| 694 | that can be shared as the \code{defaultTestLoader} module attribute. |
| 695 | Using a subclass or instance would allow customization of some |
| 696 | configurable properties. |
| 697 | |
| 698 | \class{TestLoader} objects have the following methods: |
| 699 | |
| 700 | \begin{methoddesc}[TestLoader]{loadTestsFromTestCase}{testCaseClass} |
| 701 | Return a suite of all tests cases contained in the |
| 702 | \class{TestCase}-derived class \class{testCaseClass}. |
| 703 | \end{methoddesc} |
| 704 | |
| 705 | \begin{methoddesc}[TestLoader]{loadTestsFromModule}{module} |
| 706 | Return a suite of all tests cases contained in the given module. |
| 707 | This method searches \var{module} for classes derived from |
| 708 | \class{TestCase} and creates an instance of the class for each test |
| 709 | method defined for the class. |
| 710 | |
| 711 | \strong{Warning:} While using a hierarchy of |
| 712 | \class{Testcase}-derived classes can be convenient in sharing |
| 713 | fixtures and helper functions, defining test methods on base classes |
| 714 | that are not intended to be instantiated directly does not play well |
| 715 | with this method. Doing so, however, can be useful when the |
| 716 | fixtures are different and defined in subclasses. |
| 717 | \end{methoddesc} |
| 718 | |
| 719 | \begin{methoddesc}[TestLoader]{loadTestsFromName}{name\optional{, module}} |
| 720 | Return a suite of all tests cases given a string specifier. |
| 721 | |
Fred Drake | 4d17b30 | 2001-09-06 15:51:56 +0000 | [diff] [blame^] | 722 | The specifier \var{name} is a ``dotted name'' that may resolve |
| 723 | either to a module, a test case class, a test method within a test |
| 724 | case class, or a callable object which returns a \class{TestCase} or |
| 725 | \class{TestSuite} instance. For example, if you have a module |
| 726 | \module{SampleTests} containing a \class{TestCase}-derived class |
| 727 | \class{SampleTestCase} with three test methods (\method{test_one()}, |
| 728 | \method{test_two()}, and \method{test_three()}), the specifier |
| 729 | \code{'SampleTests.SampleTestCase'} would cause this method to |
| 730 | return a suite which will run all three test methods. Using the |
| 731 | specifier \code{'SampleTests.SampleTestCase.test_two'} would cause |
| 732 | it to return a test suite which will run only the |
| 733 | \method{test_two()} test method. The specifier can refer to modules |
| 734 | and packages which have not been imported; they will be imported as |
| 735 | a side-effect. |
Fred Drake | 62a2669 | 2001-04-12 19:34:38 +0000 | [diff] [blame] | 736 | |
| 737 | The method optionally resolves \var{name} relative to a given module. |
| 738 | \end{methoddesc} |
| 739 | |
| 740 | \begin{methoddesc}[TestLoader]{loadTestsFromNames}{names\optional{, module}} |
| 741 | Similar to \method{loadTestsFromName()}, but takes a sequence of |
| 742 | names rather than a single name. The return value is a test suite |
| 743 | which supports all the tests defined for each name. |
| 744 | \end{methoddesc} |
| 745 | |
| 746 | \begin{methoddesc}[TestLoader]{getTestCaseNames}{testCaseClass} |
| 747 | Return a sorted sequence of method names found within |
| 748 | \var{testCaseClass}. |
| 749 | \end{methoddesc} |
| 750 | |
| 751 | |
| 752 | The following attributes of a \class{TestLoader} can be configured |
| 753 | either by subclassing or assignment on an instance: |
| 754 | |
| 755 | \begin{memberdesc}[TestLoader]{testMethodPrefix} |
| 756 | String giving the prefix of method names which will be interpreted |
| 757 | as test methods. The default value is \code{'test'}. |
| 758 | \end{memberdesc} |
| 759 | |
| 760 | \begin{memberdesc}[TestLoader]{sortTestMethodsUsing} |
| 761 | Function to be used to compare method names when sorting them in |
| 762 | \method{getTestCaseNames()}. The default value is the built-in |
| 763 | \function{cmp()} function; it can be set to \code{None} to disable |
| 764 | the sort. |
| 765 | \end{memberdesc} |
| 766 | |
| 767 | \begin{memberdesc}[TestLoader]{suiteClass} |
| 768 | Callable object that constructs a test suite from a list of tests. |
| 769 | No methods on the resulting object are needed. The default value is |
| 770 | the \class{TestSuite} class. |
| 771 | \end{memberdesc} |