Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 1 | \section{\module{doctest} --- |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 2 | Test interactive Python examples} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 3 | |
| 4 | \declaremodule{standard}{doctest} |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 5 | \moduleauthor{Tim Peters}{tim@python.org} |
| 6 | \sectionauthor{Tim Peters}{tim@python.org} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 7 | \sectionauthor{Moshe Zadka}{moshez@debian.org} |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 8 | \sectionauthor{Edward Loper}{edloper@users.sourceforge.net} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 9 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 10 | \modulesynopsis{A framework for verifying interactive Python examples.} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 11 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 12 | The \module{doctest} module searches for pieces of text that look like |
| 13 | interactive Python sessions, and then executes those sessions to |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 14 | verify that they work exactly as shown. There are several common ways to |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 15 | use doctest: |
| 16 | |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 17 | \begin{itemize} |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 18 | \item To check that a module's docstrings are up-to-date by verifying |
| 19 | that all interactive examples still work as documented. |
| 20 | \item To perform regression testing by verifying that interactive |
| 21 | examples from a test file or a test object work as expected. |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 22 | \item To write tutorial documentation for a package, liberally |
Tim Peters | cac5e7b | 2004-09-25 00:11:43 +0000 | [diff] [blame] | 23 | illustrated with input-output examples. Depending on whether |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 24 | the examples or the expository text are emphasized, this has |
| 25 | the flavor of "literate testing" or "executable documentation". |
| 26 | \end{itemize} |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 27 | |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 28 | Here's a complete but small example module: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 29 | |
| 30 | \begin{verbatim} |
| 31 | """ |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 32 | This is the "example" module. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 33 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 34 | The example module supplies one function, factorial(). For example, |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 35 | |
| 36 | >>> factorial(5) |
| 37 | 120 |
| 38 | """ |
| 39 | |
| 40 | def factorial(n): |
| 41 | """Return the factorial of n, an exact integer >= 0. |
| 42 | |
| 43 | If the result is small enough to fit in an int, return an int. |
| 44 | Else return a long. |
| 45 | |
| 46 | >>> [factorial(n) for n in range(6)] |
| 47 | [1, 1, 2, 6, 24, 120] |
| 48 | >>> [factorial(long(n)) for n in range(6)] |
| 49 | [1, 1, 2, 6, 24, 120] |
| 50 | >>> factorial(30) |
| 51 | 265252859812191058636308480000000L |
| 52 | >>> factorial(30L) |
| 53 | 265252859812191058636308480000000L |
| 54 | >>> factorial(-1) |
| 55 | Traceback (most recent call last): |
| 56 | ... |
| 57 | ValueError: n must be >= 0 |
| 58 | |
| 59 | Factorials of floats are OK, but the float must be an exact integer: |
| 60 | >>> factorial(30.1) |
| 61 | Traceback (most recent call last): |
| 62 | ... |
| 63 | ValueError: n must be exact integer |
| 64 | >>> factorial(30.0) |
| 65 | 265252859812191058636308480000000L |
| 66 | |
| 67 | It must also not be ridiculously large: |
| 68 | >>> factorial(1e100) |
| 69 | Traceback (most recent call last): |
| 70 | ... |
| 71 | OverflowError: n too large |
| 72 | """ |
| 73 | |
| 74 | \end{verbatim} |
| 75 | % allow LaTeX to break here. |
| 76 | \begin{verbatim} |
| 77 | |
| 78 | import math |
| 79 | if not n >= 0: |
| 80 | raise ValueError("n must be >= 0") |
| 81 | if math.floor(n) != n: |
| 82 | raise ValueError("n must be exact integer") |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 83 | if n+1 == n: # catch a value like 1e300 |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 84 | raise OverflowError("n too large") |
| 85 | result = 1 |
| 86 | factor = 2 |
| 87 | while factor <= n: |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 88 | result *= factor |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 89 | factor += 1 |
| 90 | return result |
| 91 | |
| 92 | def _test(): |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 93 | import doctest |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 94 | doctest.testmod() |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 95 | |
| 96 | if __name__ == "__main__": |
| 97 | _test() |
| 98 | \end{verbatim} |
| 99 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 100 | If you run \file{example.py} directly from the command line, |
| 101 | \module{doctest} works its magic: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 102 | |
| 103 | \begin{verbatim} |
| 104 | $ python example.py |
| 105 | $ |
| 106 | \end{verbatim} |
| 107 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 108 | There's no output! That's normal, and it means all the examples |
| 109 | worked. Pass \programopt{-v} to the script, and \module{doctest} |
| 110 | prints a detailed log of what it's trying, and prints a summary at the |
| 111 | end: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 112 | |
| 113 | \begin{verbatim} |
| 114 | $ python example.py -v |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 115 | Trying: |
| 116 | factorial(5) |
| 117 | Expecting: |
| 118 | 120 |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 119 | ok |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 120 | Trying: |
| 121 | [factorial(n) for n in range(6)] |
| 122 | Expecting: |
| 123 | [1, 1, 2, 6, 24, 120] |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 124 | ok |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 125 | Trying: |
| 126 | [factorial(long(n)) for n in range(6)] |
| 127 | Expecting: |
| 128 | [1, 1, 2, 6, 24, 120] |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 129 | ok |
| 130 | \end{verbatim} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 131 | |
| 132 | And so on, eventually ending with: |
| 133 | |
| 134 | \begin{verbatim} |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 135 | Trying: |
| 136 | factorial(1e100) |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 137 | Expecting: |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 138 | Traceback (most recent call last): |
| 139 | ... |
| 140 | OverflowError: n too large |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 141 | ok |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 142 | 1 items had no tests: |
| 143 | __main__._test |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 144 | 2 items passed all tests: |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 145 | 1 tests in __main__ |
| 146 | 8 tests in __main__.factorial |
| 147 | 9 tests in 3 items. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 148 | 9 passed and 0 failed. |
| 149 | Test passed. |
| 150 | $ |
| 151 | \end{verbatim} |
| 152 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 153 | That's all you need to know to start making productive use of |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 154 | \module{doctest}! Jump in. The following sections provide full |
| 155 | details. Note that there are many examples of doctests in |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 156 | the standard Python test suite and libraries. Especially useful examples |
| 157 | can be found in the standard test file \file{Lib/test/test_doctest.py}. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 158 | |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 159 | \subsection{Simple Usage: Checking Examples in |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 160 | Docstrings\label{doctest-simple-testmod}} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 161 | |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 162 | The simplest way to start using doctest (but not necessarily the way |
| 163 | you'll continue to do it) is to end each module \module{M} with: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 164 | |
| 165 | \begin{verbatim} |
| 166 | def _test(): |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 167 | import doctest |
Tim Peters | 06cc847 | 2004-09-25 00:49:53 +0000 | [diff] [blame] | 168 | doctest.testmod() |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 169 | |
| 170 | if __name__ == "__main__": |
| 171 | _test() |
| 172 | \end{verbatim} |
| 173 | |
Tim Peters | 06cc847 | 2004-09-25 00:49:53 +0000 | [diff] [blame] | 174 | \module{doctest} then examines docstrings in module \module{M}. |
Martin v. Löwis | 4581cfa | 2002-11-22 08:23:09 +0000 | [diff] [blame] | 175 | |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 176 | Running the module as a script causes the examples in the docstrings |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 177 | to get executed and verified: |
| 178 | |
| 179 | \begin{verbatim} |
| 180 | python M.py |
| 181 | \end{verbatim} |
| 182 | |
| 183 | This won't display anything unless an example fails, in which case the |
| 184 | failing example(s) and the cause(s) of the failure(s) are printed to stdout, |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 185 | and the final line of output is |
Tim Peters | 06cc847 | 2004-09-25 00:49:53 +0000 | [diff] [blame] | 186 | \samp{***Test Failed*** \var{N} failures.}, where \var{N} is the |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 187 | number of examples that failed. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 188 | |
Fred Drake | 7eb1463 | 2001-02-17 17:32:41 +0000 | [diff] [blame] | 189 | Run it with the \programopt{-v} switch instead: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 190 | |
| 191 | \begin{verbatim} |
| 192 | python M.py -v |
| 193 | \end{verbatim} |
| 194 | |
Fred Drake | 8836e56 | 2003-07-17 15:22:47 +0000 | [diff] [blame] | 195 | and a detailed report of all examples tried is printed to standard |
| 196 | output, along with assorted summaries at the end. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 197 | |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 198 | You can force verbose mode by passing \code{verbose=True} to |
Fred Drake | 5d2f515 | 2003-06-28 03:09:06 +0000 | [diff] [blame] | 199 | \function{testmod()}, or |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 200 | prohibit it by passing \code{verbose=False}. In either of those cases, |
Tim Peters | 06cc847 | 2004-09-25 00:49:53 +0000 | [diff] [blame] | 201 | \code{sys.argv} is not examined by \function{testmod()} (so passing |
| 202 | \programopt{-v} or not has no effect). |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 203 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 204 | For more information on \function{testmod()}, see |
| 205 | section~\ref{doctest-basic-api}. |
| 206 | |
| 207 | \subsection{Simple Usage: Checking Examples in a Text |
| 208 | File\label{doctest-simple-testfile}} |
| 209 | |
| 210 | Another simple application of doctest is testing interactive examples |
| 211 | in a text file. This can be done with the \function{testfile()} |
| 212 | function: |
| 213 | |
| 214 | \begin{verbatim} |
| 215 | import doctest |
Tim Peters | 06cc847 | 2004-09-25 00:49:53 +0000 | [diff] [blame] | 216 | doctest.testfile("example.txt") |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 217 | \end{verbatim} |
| 218 | |
Tim Peters | 06cc847 | 2004-09-25 00:49:53 +0000 | [diff] [blame] | 219 | That short script executes and verifies any interactive Python |
| 220 | examples contained in the file \file{example.txt}. The file content |
| 221 | is treated as if it were a single giant docstring; the file doesn't |
| 222 | need to contain a Python program! For example, perhaps \file{example.txt} |
| 223 | contains this: |
| 224 | |
| 225 | \begin{verbatim} |
| 226 | The ``example`` module |
| 227 | ====================== |
| 228 | |
| 229 | Using ``factorial`` |
| 230 | ------------------- |
| 231 | |
| 232 | This is an example text file in reStructuredText format. First import |
| 233 | ``factorial`` from the ``example`` module: |
| 234 | |
| 235 | >>> from example import factorial |
| 236 | |
| 237 | Now use it: |
| 238 | |
| 239 | >>> factorial(6) |
| 240 | 120 |
| 241 | \end{verbatim} |
| 242 | |
| 243 | Running \code{doctest.testfile("example.txt")} then finds the error |
| 244 | in this documentation: |
| 245 | |
| 246 | \begin{verbatim} |
| 247 | File "./example.txt", line 14, in example.txt |
| 248 | Failed example: |
| 249 | factorial(6) |
| 250 | Expected: |
| 251 | 120 |
| 252 | Got: |
| 253 | 720 |
| 254 | \end{verbatim} |
| 255 | |
| 256 | As with \function{testmod()}, \function{testfile()} won't display anything |
| 257 | unless an example fails. If an example does fail, then the failing |
| 258 | example(s) and the cause(s) of the failure(s) are printed to stdout, using |
| 259 | the same format as \function{testmod()}. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 260 | |
| 261 | By default, \function{testfile()} looks for files in the calling |
| 262 | module's directory. See section~\ref{doctest-basic-api} for a |
| 263 | description of the optional arguments that can be used to tell it to |
| 264 | look for files in other locations. |
| 265 | |
| 266 | Like \function{testmod()}, \function{testfile()}'s verbosity can be |
| 267 | set with the \programopt{-v} command-line switch or with the optional |
Tim Peters | 06cc847 | 2004-09-25 00:49:53 +0000 | [diff] [blame] | 268 | keyword argument \var{verbose}. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 269 | |
| 270 | For more information on \function{testfile()}, see |
| 271 | section~\ref{doctest-basic-api}. |
| 272 | |
| 273 | \subsection{How It Works\label{doctest-how-it-works}} |
| 274 | |
| 275 | This section examines in detail how doctest works: which docstrings it |
| 276 | looks at, how it finds interactive examples, what execution context it |
| 277 | uses, how it handles exceptions, and how option flags can be used to |
| 278 | control its behavior. This is the information that you need to know |
| 279 | to write doctest examples; for information about actually running |
| 280 | doctest on these examples, see the following sections. |
| 281 | |
| 282 | \subsubsection{Which Docstrings Are Examined?\label{doctest-which-docstrings}} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 283 | |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 284 | The module docstring, and all function, class and method docstrings are |
| 285 | searched. Objects imported into the module are not searched. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 286 | |
Fred Drake | 7eb1463 | 2001-02-17 17:32:41 +0000 | [diff] [blame] | 287 | In addition, if \code{M.__test__} exists and "is true", it must be a |
| 288 | dict, and each entry maps a (string) name to a function object, class |
| 289 | object, or string. Function and class object docstrings found from |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 290 | \code{M.__test__} are searched, and strings are treated as if they |
| 291 | were docstrings. In output, a key \code{K} in \code{M.__test__} appears |
| 292 | with name |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 293 | |
| 294 | \begin{verbatim} |
Fred Drake | 8836e56 | 2003-07-17 15:22:47 +0000 | [diff] [blame] | 295 | <name of M>.__test__.K |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 296 | \end{verbatim} |
| 297 | |
| 298 | Any classes found are recursively searched similarly, to test docstrings in |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 299 | their contained methods and nested classes. |
| 300 | |
| 301 | \versionchanged[A "private name" concept is deprecated and no longer |
Tim Peters | 2603960 | 2004-08-13 01:49:12 +0000 | [diff] [blame] | 302 | documented]{2.4} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 303 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 304 | \subsubsection{How are Docstring Examples |
| 305 | Recognized?\label{doctest-finding-examples}} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 306 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 307 | In most cases a copy-and-paste of an interactive console session works |
| 308 | fine, but doctest isn't trying to do an exact emulation of any specific |
| 309 | Python shell. All hard tab characters are expanded to spaces, using |
| 310 | 8-column tab stops. If you don't believe tabs should mean that, too |
| 311 | bad: don't use hard tabs, or write your own \class{DocTestParser} |
| 312 | class. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 313 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 314 | \versionchanged[Expanding tabs to spaces is new; previous versions |
| 315 | tried to preserve hard tabs, with confusing results]{2.4} |
| 316 | |
| 317 | \begin{verbatim} |
| 318 | >>> # comments are ignored |
| 319 | >>> x = 12 |
| 320 | >>> x |
| 321 | 12 |
| 322 | >>> if x == 13: |
| 323 | ... print "yes" |
| 324 | ... else: |
| 325 | ... print "no" |
| 326 | ... print "NO" |
| 327 | ... print "NO!!!" |
| 328 | ... |
| 329 | no |
| 330 | NO |
| 331 | NO!!! |
| 332 | >>> |
| 333 | \end{verbatim} |
| 334 | |
| 335 | Any expected output must immediately follow the final |
| 336 | \code{'>\code{>}>~'} or \code{'...~'} line containing the code, and |
| 337 | the expected output (if any) extends to the next \code{'>\code{>}>~'} |
| 338 | or all-whitespace line. |
| 339 | |
| 340 | The fine print: |
| 341 | |
| 342 | \begin{itemize} |
| 343 | |
| 344 | \item Expected output cannot contain an all-whitespace line, since such a |
| 345 | line is taken to signal the end of expected output. If expected |
| 346 | output does contain a blank line, put \code{<BLANKLINE>} in your |
| 347 | doctest example each place a blank line is expected. |
| 348 | \versionchanged[\code{<BLANKLINE>} was added; there was no way to |
| 349 | use expected output containing empty lines in |
| 350 | previous versions]{2.4} |
| 351 | |
| 352 | \item Output to stdout is captured, but not output to stderr (exception |
| 353 | tracebacks are captured via a different means). |
| 354 | |
| 355 | \item If you continue a line via backslashing in an interactive session, |
| 356 | or for any other reason use a backslash, you should use a raw |
| 357 | docstring, which will preserve your backslashes exactly as you type |
| 358 | them: |
| 359 | |
| 360 | \begin{verbatim} |
| 361 | >>> def f(x): |
| 362 | ... r'''Backslashes in a raw docstring: m\n''' |
| 363 | >>> print f.__doc__ |
| 364 | Backslashes in a raw docstring: m\n |
| 365 | \end{verbatim} |
| 366 | |
| 367 | Otherwise, the backslash will be interpreted as part of the string. |
Tim Peters | 39c5de0 | 2004-09-25 01:22:29 +0000 | [diff] [blame] | 368 | For example, the "{\textbackslash}" above would be interpreted as a |
| 369 | newline character. Alternatively, you can double each backslash in the |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 370 | doctest version (and not use a raw string): |
| 371 | |
| 372 | \begin{verbatim} |
| 373 | >>> def f(x): |
| 374 | ... '''Backslashes in a raw docstring: m\\n''' |
| 375 | >>> print f.__doc__ |
| 376 | Backslashes in a raw docstring: m\n |
| 377 | \end{verbatim} |
| 378 | |
| 379 | \item The starting column doesn't matter: |
| 380 | |
| 381 | \begin{verbatim} |
| 382 | >>> assert "Easy!" |
| 383 | >>> import math |
| 384 | >>> math.floor(1.9) |
| 385 | 1.0 |
| 386 | \end{verbatim} |
| 387 | |
| 388 | and as many leading whitespace characters are stripped from the |
| 389 | expected output as appeared in the initial \code{'>\code{>}>~'} line |
| 390 | that started the example. |
| 391 | \end{itemize} |
| 392 | |
| 393 | \subsubsection{What's the Execution Context?\label{doctest-execution-context}} |
| 394 | |
| 395 | By default, each time \module{doctest} finds a docstring to test, it |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 396 | uses a \emph{shallow copy} of \module{M}'s globals, so that running tests |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 397 | doesn't change the module's real globals, and so that one test in |
| 398 | \module{M} can't leave behind crumbs that accidentally allow another test |
| 399 | to work. This means examples can freely use any names defined at top-level |
Tim Peters | 0481d24 | 2001-10-02 21:01:22 +0000 | [diff] [blame] | 400 | in \module{M}, and names defined earlier in the docstring being run. |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 401 | Examples cannot see names defined in other docstrings. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 402 | |
| 403 | You can force use of your own dict as the execution context by passing |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 404 | \code{globs=your_dict} to \function{testmod()} or |
| 405 | \function{testfile()} instead. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 406 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 407 | \subsubsection{What About Exceptions?\label{doctest-exceptions}} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 408 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 409 | No problem, provided that the traceback is the only output produced by |
| 410 | the example: just paste in the traceback. Since tracebacks contain |
| 411 | details that are likely to change rapidly (for example, exact file paths |
| 412 | and line numbers), this is one case where doctest works hard to be |
| 413 | flexible in what it accepts. |
| 414 | |
| 415 | Simple example: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 416 | |
| 417 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 418 | >>> [1, 2, 3].remove(42) |
| 419 | Traceback (most recent call last): |
| 420 | File "<stdin>", line 1, in ? |
| 421 | ValueError: list.remove(x): x not in list |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 422 | \end{verbatim} |
| 423 | |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 424 | That doctest succeeds if \exception{ValueError} is raised, with the |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 425 | \samp{list.remove(x): x not in list} detail as shown. |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 426 | |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 427 | The expected output for an exception must start with a traceback |
| 428 | header, which may be either of the following two lines, indented the |
| 429 | same as the first line of the example: |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 430 | |
| 431 | \begin{verbatim} |
| 432 | Traceback (most recent call last): |
| 433 | Traceback (innermost last): |
| 434 | \end{verbatim} |
| 435 | |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 436 | The traceback header is followed by an optional traceback stack, whose |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 437 | contents are ignored by doctest. The traceback stack is typically |
| 438 | omitted, or copied verbatim from an interactive session. |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 439 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 440 | The traceback stack is followed by the most interesting part: the |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 441 | line(s) containing the exception type and detail. This is usually the |
| 442 | last line of a traceback, but can extend across multiple lines if the |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 443 | exception has a multi-line detail: |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 444 | |
| 445 | \begin{verbatim} |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 446 | >>> raise ValueError('multi\n line\ndetail') |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 447 | Traceback (most recent call last): |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 448 | File "<stdin>", line 1, in ? |
| 449 | ValueError: multi |
| 450 | line |
| 451 | detail |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 452 | \end{verbatim} |
| 453 | |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 454 | The last three lines (starting with \exception{ValueError}) are |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 455 | compared against the exception's type and detail, and the rest are |
| 456 | ignored. |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 457 | |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 458 | Best practice is to omit the traceback stack, unless it adds |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 459 | significant documentation value to the example. So the last example |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 460 | is probably better as: |
| 461 | |
| 462 | \begin{verbatim} |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 463 | >>> raise ValueError('multi\n line\ndetail') |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 464 | Traceback (most recent call last): |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 465 | ... |
| 466 | ValueError: multi |
| 467 | line |
| 468 | detail |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 469 | \end{verbatim} |
| 470 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 471 | Note that tracebacks are treated very specially. In particular, in the |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 472 | rewritten example, the use of \samp{...} is independent of doctest's |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 473 | \constant{ELLIPSIS} option. The ellipsis in that example could be left |
| 474 | out, or could just as well be three (or three hundred) commas or digits, |
| 475 | or an indented transcript of a Monty Python skit. |
| 476 | |
| 477 | Some details you should read once, but won't need to remember: |
| 478 | |
| 479 | \begin{itemize} |
| 480 | |
| 481 | \item Doctest can't guess whether your expected output came from an |
| 482 | exception traceback or from ordinary printing. So, e.g., an example |
| 483 | that expects \samp{ValueError: 42 is prime} will pass whether |
| 484 | \exception{ValueError} is actually raised or if the example merely |
| 485 | prints that traceback text. In practice, ordinary output rarely begins |
| 486 | with a traceback header line, so this doesn't create real problems. |
| 487 | |
| 488 | \item Each line of the traceback stack (if present) must be indented |
| 489 | further than the first line of the example, \emph{or} start with a |
| 490 | non-alphanumeric character. The first line following the traceback |
| 491 | header indented the same and starting with an alphanumeric is taken |
| 492 | to be the start of the exception detail. Of course this does the |
| 493 | right thing for genuine tracebacks. |
| 494 | |
Tim Peters | 1fbf9c5 | 2004-09-04 17:21:02 +0000 | [diff] [blame] | 495 | \item When the \constant{IGNORE_EXCEPTION_DETAIL} doctest option is |
| 496 | is specified, everything following the leftmost colon is ignored. |
| 497 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 498 | \end{itemize} |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 499 | |
Tim Peters | 39c5de0 | 2004-09-25 01:22:29 +0000 | [diff] [blame] | 500 | \versionchanged[The ability to handle a multi-line exception detail, |
| 501 | and the \constant{IGNORE_EXCEPTION_DETAIL} doctest option, |
| 502 | were added]{2.4} |
Tim Peters | 0e44807 | 2004-08-26 01:02:08 +0000 | [diff] [blame] | 503 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 504 | \subsubsection{Option Flags and Directives\label{doctest-options}} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 505 | |
Tim Peters | cf53355 | 2004-08-26 04:50:38 +0000 | [diff] [blame] | 506 | A number of option flags control various aspects of doctest's |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 507 | behavior. Symbolic names for the flags are supplied as module constants, |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 508 | which can be or'ed together and passed to various functions. The names |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 509 | can also be used in doctest directives (see below). |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 510 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 511 | The first group of options define test semantics, controlling |
| 512 | aspects of how doctest decides whether actual output matches an |
| 513 | example's expected output: |
| 514 | |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 515 | \begin{datadesc}{DONT_ACCEPT_TRUE_FOR_1} |
| 516 | By default, if an expected output block contains just \code{1}, |
| 517 | an actual output block containing just \code{1} or just |
| 518 | \code{True} is considered to be a match, and similarly for \code{0} |
| 519 | versus \code{False}. When \constant{DONT_ACCEPT_TRUE_FOR_1} is |
| 520 | specified, neither substitution is allowed. The default behavior |
| 521 | caters to that Python changed the return type of many functions |
| 522 | from integer to boolean; doctests expecting "little integer" |
| 523 | output still work in these cases. This option will probably go |
| 524 | away, but not for several years. |
| 525 | \end{datadesc} |
| 526 | |
| 527 | \begin{datadesc}{DONT_ACCEPT_BLANKLINE} |
| 528 | By default, if an expected output block contains a line |
| 529 | containing only the string \code{<BLANKLINE>}, then that line |
| 530 | will match a blank line in the actual output. Because a |
| 531 | genuinely blank line delimits the expected output, this is |
| 532 | the only way to communicate that a blank line is expected. When |
| 533 | \constant{DONT_ACCEPT_BLANKLINE} is specified, this substitution |
| 534 | is not allowed. |
| 535 | \end{datadesc} |
| 536 | |
| 537 | \begin{datadesc}{NORMALIZE_WHITESPACE} |
| 538 | When specified, all sequences of whitespace (blanks and newlines) are |
| 539 | treated as equal. Any sequence of whitespace within the expected |
| 540 | output will match any sequence of whitespace within the actual output. |
| 541 | By default, whitespace must match exactly. |
| 542 | \constant{NORMALIZE_WHITESPACE} is especially useful when a line |
| 543 | of expected output is very long, and you want to wrap it across |
| 544 | multiple lines in your source. |
| 545 | \end{datadesc} |
| 546 | |
| 547 | \begin{datadesc}{ELLIPSIS} |
| 548 | When specified, an ellipsis marker (\code{...}) in the expected output |
| 549 | can match any substring in the actual output. This includes |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 550 | substrings that span line boundaries, and empty substrings, so it's |
| 551 | best to keep usage of this simple. Complicated uses can lead to the |
| 552 | same kinds of "oops, it matched too much!" surprises that \regexp{.*} |
| 553 | is prone to in regular expressions. |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 554 | \end{datadesc} |
| 555 | |
Tim Peters | 1fbf9c5 | 2004-09-04 17:21:02 +0000 | [diff] [blame] | 556 | \begin{datadesc}{IGNORE_EXCEPTION_DETAIL} |
| 557 | When specified, an example that expects an exception passes if |
| 558 | an exception of the expected type is raised, even if the exception |
| 559 | detail does not match. For example, an example expecting |
| 560 | \samp{ValueError: 42} will pass if the actual exception raised is |
| 561 | \samp{ValueError: 3*14}, but will fail, e.g., if |
| 562 | \exception{TypeError} is raised. |
| 563 | |
| 564 | Note that a similar effect can be obtained using \constant{ELLIPSIS}, |
| 565 | and \constant{IGNORE_EXCEPTION_DETAIL} may go away when Python releases |
| 566 | prior to 2.4 become uninteresting. Until then, |
| 567 | \constant{IGNORE_EXCEPTION_DETAIL} is the only clear way to write a |
| 568 | doctest that doesn't care about the exception detail yet continues |
| 569 | to pass under Python releases prior to 2.4 (doctest directives |
| 570 | appear to be comments to them). For example, |
| 571 | |
| 572 | \begin{verbatim} |
| 573 | >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL |
| 574 | Traceback (most recent call last): |
| 575 | File "<stdin>", line 1, in ? |
| 576 | TypeError: object doesn't support item assignment |
| 577 | \end{verbatim} |
| 578 | |
| 579 | passes under Python 2.4 and Python 2.3. The detail changed in 2.4, |
| 580 | to say "does not" instead of "doesn't". |
| 581 | |
| 582 | \end{datadesc} |
| 583 | |
Tim Peters | 38330fe | 2004-08-30 16:19:24 +0000 | [diff] [blame] | 584 | \begin{datadesc}{COMPARISON_FLAGS} |
| 585 | A bitmask or'ing together all the comparison flags above. |
| 586 | \end{datadesc} |
| 587 | |
Tim Peters | f33683f | 2004-08-26 04:52:46 +0000 | [diff] [blame] | 588 | The second group of options controls how test failures are reported: |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 589 | |
Edward Loper | 71f55af | 2004-08-26 01:41:51 +0000 | [diff] [blame] | 590 | \begin{datadesc}{REPORT_UDIFF} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 591 | When specified, failures that involve multi-line expected and |
| 592 | actual outputs are displayed using a unified diff. |
| 593 | \end{datadesc} |
| 594 | |
Edward Loper | 71f55af | 2004-08-26 01:41:51 +0000 | [diff] [blame] | 595 | \begin{datadesc}{REPORT_CDIFF} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 596 | When specified, failures that involve multi-line expected and |
| 597 | actual outputs will be displayed using a context diff. |
| 598 | \end{datadesc} |
| 599 | |
Edward Loper | 71f55af | 2004-08-26 01:41:51 +0000 | [diff] [blame] | 600 | \begin{datadesc}{REPORT_NDIFF} |
Tim Peters | c6cbab0 | 2004-08-22 19:43:28 +0000 | [diff] [blame] | 601 | When specified, differences are computed by \code{difflib.Differ}, |
| 602 | using the same algorithm as the popular \file{ndiff.py} utility. |
| 603 | This is the only method that marks differences within lines as |
| 604 | well as across lines. For example, if a line of expected output |
| 605 | contains digit \code{1} where actual output contains letter \code{l}, |
| 606 | a line is inserted with a caret marking the mismatching column |
| 607 | positions. |
| 608 | \end{datadesc} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 609 | |
Edward Loper | a89f88d | 2004-08-26 02:45:51 +0000 | [diff] [blame] | 610 | \begin{datadesc}{REPORT_ONLY_FIRST_FAILURE} |
| 611 | When specified, display the first failing example in each doctest, |
| 612 | but suppress output for all remaining examples. This will prevent |
| 613 | doctest from reporting correct examples that break because of |
| 614 | earlier failures; but it might also hide incorrect examples that |
| 615 | fail independently of the first failure. When |
| 616 | \constant{REPORT_ONLY_FIRST_FAILURE} is specified, the remaining |
| 617 | examples are still run, and still count towards the total number of |
| 618 | failures reported; only the output is suppressed. |
| 619 | \end{datadesc} |
| 620 | |
Tim Peters | 38330fe | 2004-08-30 16:19:24 +0000 | [diff] [blame] | 621 | \begin{datadesc}{REPORTING_FLAGS} |
| 622 | A bitmask or'ing together all the reporting flags above. |
| 623 | \end{datadesc} |
| 624 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 625 | "Doctest directives" may be used to modify the option flags for |
| 626 | individual examples. Doctest directives are expressed as a special |
| 627 | Python comment following an example's source code: |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 628 | |
| 629 | \begin{productionlist}[doctest] |
| 630 | \production{directive} |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 631 | {"\#" "doctest:" \token{directive_options}} |
| 632 | \production{directive_options} |
| 633 | {\token{directive_option} ("," \token{directive_option})*} |
| 634 | \production{directive_option} |
| 635 | {\token{on_or_off} \token{directive_option_name}} |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 636 | \production{on_or_off} |
| 637 | {"+" | "-"} |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 638 | \production{directive_option_name} |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 639 | {"DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...} |
| 640 | \end{productionlist} |
| 641 | |
| 642 | Whitespace is not allowed between the \code{+} or \code{-} and the |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 643 | directive option name. The directive option name can be any of the |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 644 | option flag names explained above. |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 645 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 646 | An example's doctest directives modify doctest's behavior for that |
| 647 | single example. Use \code{+} to enable the named behavior, or |
| 648 | \code{-} to disable it. |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 649 | |
| 650 | For example, this test passes: |
| 651 | |
| 652 | \begin{verbatim} |
| 653 | >>> print range(20) #doctest: +NORMALIZE_WHITESPACE |
| 654 | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, |
| 655 | 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] |
| 656 | \end{verbatim} |
| 657 | |
| 658 | Without the directive it would fail, both because the actual output |
| 659 | doesn't have two blanks before the single-digit list elements, and |
| 660 | because the actual output is on a single line. This test also passes, |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 661 | and also requires a directive to do so: |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 662 | |
| 663 | \begin{verbatim} |
| 664 | >>> print range(20) # doctest:+ELLIPSIS |
| 665 | [0, 1, ..., 18, 19] |
| 666 | \end{verbatim} |
| 667 | |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 668 | Multiple directives can be used on a single physical line, separated |
| 669 | by commas: |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 670 | |
| 671 | \begin{verbatim} |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 672 | >>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 673 | [0, 1, ..., 18, 19] |
| 674 | \end{verbatim} |
| 675 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 676 | If multiple directive comments are used for a single example, then |
| 677 | they are combined: |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 678 | |
| 679 | \begin{verbatim} |
| 680 | >>> print range(20) # doctest: +ELLIPSIS |
| 681 | ... # doctest: +NORMALIZE_WHITESPACE |
| 682 | [0, 1, ..., 18, 19] |
| 683 | \end{verbatim} |
| 684 | |
| 685 | As the previous example shows, you can add \samp{...} lines to your |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 686 | example containing only directives. This can be useful when an |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame] | 687 | example is too long for a directive to comfortably fit on the same |
| 688 | line: |
| 689 | |
| 690 | \begin{verbatim} |
| 691 | >>> print range(5) + range(10,20) + range(30,40) + range(50,60) |
| 692 | ... # doctest: +ELLIPSIS |
| 693 | [0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59] |
| 694 | \end{verbatim} |
| 695 | |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 696 | Note that since all options are disabled by default, and directives apply |
| 697 | only to the example they appear in, enabling options (via \code{+} in a |
| 698 | directive) is usually the only meaningful choice. However, option flags |
| 699 | can also be passed to functions that run doctests, establishing different |
| 700 | defaults. In such cases, disabling an option via \code{-} in a directive |
| 701 | can be useful. |
| 702 | |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 703 | \versionchanged[Constants \constant{DONT_ACCEPT_BLANKLINE}, |
| 704 | \constant{NORMALIZE_WHITESPACE}, \constant{ELLIPSIS}, |
Tim Peters | 1fbf9c5 | 2004-09-04 17:21:02 +0000 | [diff] [blame] | 705 | \constant{IGNORE_EXCEPTION_DETAIL}, |
Edward Loper | a89f88d | 2004-08-26 02:45:51 +0000 | [diff] [blame] | 706 | \constant{REPORT_UDIFF}, \constant{REPORT_CDIFF}, |
Tim Peters | 38330fe | 2004-08-30 16:19:24 +0000 | [diff] [blame] | 707 | \constant{REPORT_NDIFF}, \constant{REPORT_ONLY_FIRST_FAILURE}, |
| 708 | \constant{COMPARISON_FLAGS} and \constant{REPORTING_FLAGS} |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 709 | were added; by default \code{<BLANKLINE>} in expected output |
| 710 | matches an empty line in actual output; and doctest directives |
| 711 | were added]{2.4} |
| 712 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 713 | \subsubsection{Warnings\label{doctest-warnings}} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 714 | |
Tim Peters | 2dc8205 | 2004-09-25 01:30:16 +0000 | [diff] [blame] | 715 | \module{doctest} is serious about requiring exact matches in expected |
| 716 | output. If even a single character doesn't match, the test fails. This |
| 717 | will probably surprise you a few times, as you learn exactly what Python |
| 718 | does and doesn't guarantee about output. For example, when printing a |
| 719 | dict, Python doesn't guarantee that the key-value pairs will be printed |
| 720 | in any particular order, so a test like |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 721 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 722 | % Hey! What happened to Monty Python examples? |
| 723 | % Tim: ask Guido -- it's his example! |
| 724 | \begin{verbatim} |
| 725 | >>> foo() |
| 726 | {"Hermione": "hippogryph", "Harry": "broomstick"} |
| 727 | \end{verbatim} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 728 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 729 | is vulnerable! One workaround is to do |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 730 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 731 | \begin{verbatim} |
| 732 | >>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"} |
| 733 | True |
| 734 | \end{verbatim} |
| 735 | |
| 736 | instead. Another is to do |
| 737 | |
| 738 | \begin{verbatim} |
| 739 | >>> d = foo().items() |
| 740 | >>> d.sort() |
| 741 | >>> d |
| 742 | [('Harry', 'broomstick'), ('Hermione', 'hippogryph')] |
| 743 | \end{verbatim} |
| 744 | |
| 745 | There are others, but you get the idea. |
| 746 | |
| 747 | Another bad idea is to print things that embed an object address, like |
| 748 | |
| 749 | \begin{verbatim} |
| 750 | >>> id(1.0) # certain to fail some of the time |
| 751 | 7948648 |
Tim Peters | 39c5de0 | 2004-09-25 01:22:29 +0000 | [diff] [blame] | 752 | >>> class C: pass |
| 753 | >>> C() # the default repr() for instances embeds an address |
| 754 | <__main__.C instance at 0x00AC18F0> |
| 755 | \end{verbatim} |
| 756 | |
| 757 | The \constant{ELLIPSIS} directive gives a nice approach for the last |
| 758 | example: |
| 759 | |
| 760 | \begin{verbatim} |
| 761 | >>> C() #doctest: +ELLIPSIS |
| 762 | <__main__.C instance at 0x...> |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 763 | \end{verbatim} |
| 764 | |
| 765 | Floating-point numbers are also subject to small output variations across |
| 766 | platforms, because Python defers to the platform C library for float |
| 767 | formatting, and C libraries vary widely in quality here. |
| 768 | |
| 769 | \begin{verbatim} |
| 770 | >>> 1./7 # risky |
| 771 | 0.14285714285714285 |
| 772 | >>> print 1./7 # safer |
| 773 | 0.142857142857 |
| 774 | >>> print round(1./7, 6) # much safer |
| 775 | 0.142857 |
| 776 | \end{verbatim} |
| 777 | |
| 778 | Numbers of the form \code{I/2.**J} are safe across all platforms, and I |
| 779 | often contrive doctest examples to produce numbers of that form: |
| 780 | |
| 781 | \begin{verbatim} |
| 782 | >>> 3./4 # utterly safe |
| 783 | 0.75 |
| 784 | \end{verbatim} |
| 785 | |
| 786 | Simple fractions are also easier for people to understand, and that makes |
| 787 | for better documentation. |
| 788 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 789 | \subsection{Basic API\label{doctest-basic-api}} |
| 790 | |
| 791 | The functions \function{testmod()} and \function{testfile()} provide a |
| 792 | simple interface to doctest that should be sufficient for most basic |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 793 | uses. For a less formal introduction to these two functions, see |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 794 | sections \ref{doctest-simple-testmod} and |
| 795 | \ref{doctest-simple-testfile}. |
| 796 | |
| 797 | \begin{funcdesc}{testfile}{filename\optional{, module_relative}\optional{, |
| 798 | name}\optional{, package}\optional{, |
| 799 | globs}\optional{, verbose}\optional{, |
| 800 | report}\optional{, optionflags}\optional{, |
| 801 | extraglobs}\optional{, raise_on_error}} |
| 802 | |
| 803 | All arguments except \var{filename} are optional, and should be |
| 804 | specified in keyword form. |
| 805 | |
| 806 | Test examples in the file named \var{filename}. Return |
| 807 | \samp{(\var{failure_count}, \var{test_count})}. |
| 808 | |
| 809 | Optional argument \var{module_relative} specifies how the filename |
| 810 | should be interpreted: |
| 811 | |
| 812 | \begin{itemize} |
| 813 | \item If \var{module_relative} is \code{True} (the default), then |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 814 | \var{filename} specifies an OS-independent module-relative |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 815 | path. By default, this path is relative to the calling |
| 816 | module's directory; but if the \var{package} argument is |
| 817 | specified, then it is relative to that package. To ensure |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 818 | OS-independence, \var{filename} should use \code{/} characters |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 819 | to separate path segments, and may not be an absolute path |
| 820 | (i.e., it may not begin with \code{/}). |
| 821 | \item If \var{module_relative} is \code{False}, then \var{filename} |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 822 | specifies an OS-specific path. The path may be absolute or |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 823 | relative; relative paths are resolved with respect to the |
| 824 | current working directory. |
| 825 | \end{itemize} |
| 826 | |
| 827 | Optional argument \var{name} gives the name of the test; by default, |
| 828 | or if \code{None}, \code{os.path.basename(\var{filename})} is used. |
| 829 | |
| 830 | Optional argument \var{package} is a Python package or the name of a |
| 831 | Python package whose directory should be used as the base directory |
| 832 | for a module-relative filename. If no package is specified, then |
| 833 | the calling module's directory is used as the base directory for |
| 834 | module-relative filenames. It is an error to specify \var{package} |
| 835 | if \var{module_relative} is \code{False}. |
| 836 | |
| 837 | Optional argument \var{globs} gives a dict to be used as the globals |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 838 | when executing examples. A new shallow copy of this dict is |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 839 | created for the doctest, so its examples start with a clean slate. |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 840 | By default, or if \code{None}, a new empty dict is used. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 841 | |
| 842 | Optional argument \var{extraglobs} gives a dict merged into the |
| 843 | globals used to execute examples. This works like |
| 844 | \method{dict.update()}: if \var{globs} and \var{extraglobs} have a |
| 845 | common key, the associated value in \var{extraglobs} appears in the |
| 846 | combined dict. By default, or if \code{None}, no extra globals are |
| 847 | used. This is an advanced feature that allows parameterization of |
| 848 | doctests. For example, a doctest can be written for a base class, using |
| 849 | a generic name for the class, then reused to test any number of |
| 850 | subclasses by passing an \var{extraglobs} dict mapping the generic |
| 851 | name to the subclass to be tested. |
| 852 | |
| 853 | Optional argument \var{verbose} prints lots of stuff if true, and prints |
| 854 | only failures if false; by default, or if \code{None}, it's true |
| 855 | if and only if \code{'-v'} is in \code{sys.argv}. |
| 856 | |
| 857 | Optional argument \var{report} prints a summary at the end when true, |
| 858 | else prints nothing at the end. In verbose mode, the summary is |
| 859 | detailed, else the summary is very brief (in fact, empty if all tests |
| 860 | passed). |
| 861 | |
| 862 | Optional argument \var{optionflags} or's together option flags. See |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 863 | section~\ref{doctest-options}. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 864 | |
| 865 | Optional argument \var{raise_on_error} defaults to false. If true, |
| 866 | an exception is raised upon the first failure or unexpected exception |
| 867 | in an example. This allows failures to be post-mortem debugged. |
| 868 | Default behavior is to continue running examples. |
| 869 | |
| 870 | \versionadded{2.4} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 871 | \end{funcdesc} |
| 872 | |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 873 | \begin{funcdesc}{testmod}{\optional{m}\optional{, name}\optional{, |
| 874 | globs}\optional{, verbose}\optional{, |
| 875 | isprivate}\optional{, report}\optional{, |
| 876 | optionflags}\optional{, extraglobs}\optional{, |
Tim Peters | 8278860 | 2004-09-13 15:03:17 +0000 | [diff] [blame] | 877 | raise_on_error}\optional{, exclude_empty}} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 878 | |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 879 | All arguments are optional, and all except for \var{m} should be |
| 880 | specified in keyword form. |
| 881 | |
| 882 | Test examples in docstrings in functions and classes reachable |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 883 | from module \var{m} (or module \module{__main__} if \var{m} is not |
| 884 | supplied or is \code{None}), starting with \code{\var{m}.__doc__}. |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 885 | |
| 886 | Also test examples reachable from dict \code{\var{m}.__test__}, if it |
| 887 | exists and is not \code{None}. \code{\var{m}.__test__} maps |
| 888 | names (strings) to functions, classes and strings; function and class |
| 889 | docstrings are searched for examples; strings are searched directly, |
| 890 | as if they were docstrings. |
| 891 | |
| 892 | Only docstrings attached to objects belonging to module \var{m} are |
| 893 | searched. |
| 894 | |
| 895 | Return \samp{(\var{failure_count}, \var{test_count})}. |
| 896 | |
| 897 | Optional argument \var{name} gives the name of the module; by default, |
| 898 | or if \code{None}, \code{\var{m}.__name__} is used. |
| 899 | |
Tim Peters | 8278860 | 2004-09-13 15:03:17 +0000 | [diff] [blame] | 900 | Optional argument \var{exclude_empty} defaults to false. If true, |
| 901 | objects for which no doctests are found are excluded from consideration. |
| 902 | The default is a backward compatibility hack, so that code still |
| 903 | using \method{doctest.master.summarize()} in conjunction with |
| 904 | \function{testmod()} continues to get output for objects with no tests. |
| 905 | The \var{exclude_empty} argument to the newer \class{DocTestFinder} |
| 906 | constructor defaults to true. |
| 907 | |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 908 | Optional arguments \var{extraglobs}, \var{verbose}, \var{report}, |
| 909 | \var{optionflags}, \var{raise_on_error}, and \var{globs} are the same as |
| 910 | for function \function{testfile()} above, except that \var{globs} |
| 911 | defaults to \code{\var{m}.__dict__}. |
| 912 | |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 913 | Optional argument \var{isprivate} specifies a function used to |
| 914 | determine whether a name is private. The default function treats |
| 915 | all names as public. \var{isprivate} can be set to |
| 916 | \code{doctest.is_private} to skip over names that are |
| 917 | private according to Python's underscore naming convention. |
| 918 | \deprecated{2.4}{\var{isprivate} was a stupid idea -- don't use it. |
| 919 | If you need to skip tests based on name, filter the list returned by |
| 920 | \code{DocTestFinder.find()} instead.} |
| 921 | |
| 922 | \versionchanged[The parameter \var{optionflags} was added]{2.3} |
| 923 | |
Tim Peters | 8278860 | 2004-09-13 15:03:17 +0000 | [diff] [blame] | 924 | \versionchanged[The parameters \var{extraglobs}, \var{raise_on_error} |
| 925 | and \var{exclude_empty} were added]{2.4} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 926 | \end{funcdesc} |
| 927 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 928 | \subsection{Unittest API\label{doctest-unittest-api}} |
| 929 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 930 | As your collection of doctest'ed modules grows, you'll want a way to run |
| 931 | all their doctests systematically. Prior to Python 2.4, \module{doctest} |
| 932 | had a barely documented \class{Tester} class that supplied a rudimentary |
| 933 | way to combine doctests from multiple modules. \class{Tester} was feeble, |
| 934 | and in practice most serious Python testing frameworks build on the |
| 935 | \module{unittest} module, which supplies many flexible ways to combine |
| 936 | tests from multiple sources. So, in Python 2.4, module{doctest}'s |
| 937 | \class{Tester} class is deprecated, and \module{doctest} provides several |
| 938 | functions that can be used to create \module{unittest} test suites from |
| 939 | modules and text files containing doctests. These test suites can then be |
| 940 | run using \module{unittest} test runners: |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 941 | |
| 942 | \begin{verbatim} |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 943 | import unittest |
| 944 | import doctest |
| 945 | import my_module_with_doctests, and_another |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 946 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 947 | suite = unittest.TestSuite() |
| 948 | for mod in my_module_with_doctests, and_another: |
| 949 | suite.addTest(doctest.DocTestSuite(mod)) |
| 950 | runner = unittest.TextTestRunner() |
| 951 | runner.run(suite) |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 952 | \end{verbatim} |
| 953 | |
| 954 | \begin{funcdesc}{DocFileSuite}{*paths, **kw} |
| 955 | Convert doctest tests from one or more text files to a |
| 956 | \class{\refmodule{unittest}.TestSuite}. |
| 957 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 958 | The returned \class{unittest.TestSuite} is to be run by the unittest |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 959 | framework and runs the interactive examples in each file. If an |
| 960 | example in any file fails, then the synthesized unit test fails, and |
| 961 | a \exception{failureException} exception is raised showing the |
| 962 | name of the file containing the test and a (sometimes approximate) |
| 963 | line number. |
| 964 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 965 | Pass one or more paths (as strings) to text files to be examined. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 966 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 967 | Options may be provided as keyword arguments: |
| 968 | |
| 969 | Optional argument \var{module_relative} specifies how |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 970 | the the filenames in \var{paths} should be interpreted: |
| 971 | |
| 972 | \begin{itemize} |
| 973 | \item If \var{module_relative} is \code{True} (the default), then |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 974 | each filename specifies an OS-independent module-relative |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 975 | path. By default, this path is relative to the calling |
| 976 | module's directory; but if the \var{package} argument is |
| 977 | specified, then it is relative to that package. To ensure |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 978 | OS-independence, each filename should use \code{/} characters |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 979 | to separate path segments, and may not be an absolute path |
| 980 | (i.e., it may not begin with \code{/}). |
| 981 | \item If \var{module_relative} is \code{False}, then each filename |
Tim Peters | b2b26ac | 2004-09-25 01:51:49 +0000 | [diff] [blame] | 982 | specifies an OS-specific path. The path may be absolute or |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 983 | relative; relative paths are resolved with respect to the |
| 984 | current working directory. |
| 985 | \end{itemize} |
| 986 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 987 | Optional argument \var{package} is a Python package or the name |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 988 | of a Python package whose directory should be used as the base |
| 989 | directory for module-relative filenames. If no package is |
| 990 | specified, then the calling module's directory is used as the base |
| 991 | directory for module-relative filenames. It is an error to specify |
| 992 | \var{package} if \var{module_relative} is \code{False}. |
| 993 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 994 | Optional argument \var{setUp} specifies a set-up function for |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 995 | the test suite. This is called before running the tests in each |
| 996 | file. The \var{setUp} function will be passed a \class{DocTest} |
| 997 | object. The setUp function can access the test globals as the |
| 998 | \var{globs} attribute of the test passed. |
| 999 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1000 | Optional argument \var{tearDown} specifies a tear-down function |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1001 | for the test suite. This is called after running the tests in each |
| 1002 | file. The \var{tearDown} function will be passed a \class{DocTest} |
| 1003 | object. The setUp function can access the test globals as the |
| 1004 | \var{globs} attribute of the test passed. |
| 1005 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1006 | Optional argument \var{globs} is a dictionary containing the |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1007 | initial global variables for the tests. A new copy of this |
| 1008 | dictionary is created for each test. By default, \var{globs} is |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1009 | a new empty dictionary. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1010 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1011 | Optional argument \var{optionflags} specifies the default |
| 1012 | doctest options for the tests, created by or-ing together |
| 1013 | individual option flags. See section~\ref{doctest-options}. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1014 | |
| 1015 | \versionadded{2.4} |
| 1016 | \end{funcdesc} |
| 1017 | |
| 1018 | \begin{funcdesc}{DocTestSuite}{\optional{module}\optional{, |
| 1019 | globs}\optional{, extraglobs}\optional{, |
| 1020 | test_finder}\optional{, setUp}\optional{, |
| 1021 | tearDown}\optional{, checker}} |
| 1022 | Convert doctest tests for a module to a |
| 1023 | \class{\refmodule{unittest}.TestSuite}. |
| 1024 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1025 | The returned \class{unittest.TestSuite} is to be run by the unittest |
| 1026 | framework and runs each doctest in the module. If any of the doctests |
| 1027 | fail, then the synthesized unit test fails, and a |
| 1028 | \exception{failureException} exception is raised showing the name of the |
| 1029 | file containing the test and a (sometimes approximate) line number. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1030 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1031 | Optional argument \var{module} provides the module to be tested. It |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1032 | can be a module object or a (possibly dotted) module name. If not |
| 1033 | specified, the module calling this function is used. |
| 1034 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1035 | Optional argument \var{globs} is a dictionary containing the |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1036 | initial global variables for the tests. A new copy of this |
| 1037 | dictionary is created for each test. By default, \var{globs} is |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1038 | a new empty dictionary. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1039 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1040 | Optional argument \var{extraglobs} specifies an extra set of |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1041 | global variables, which is merged into \var{globs}. By default, no |
| 1042 | extra globals are used. |
| 1043 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1044 | Optional argument \var{test_finder} is the \class{DocTestFinder} |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1045 | object (or a drop-in replacement) that is used to extract doctests |
| 1046 | from the module. |
| 1047 | |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1048 | Optional arguments \var{setUp}, \var{tearDown}, and \var{optionflags} |
| 1049 | are the same as for function \function{DocFileSuite()} above. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1050 | |
| 1051 | \versionadded{2.3} |
| 1052 | \versionchanged[The parameters \var{globs}, \var{extraglobs}, |
| 1053 | \var{test_finder}, \var{setUp}, \var{tearDown}, and |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1054 | \var{optionflags} were added; this function now uses the same search |
| 1055 | technique as \function{testmod()}]{2.4} |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1056 | \end{funcdesc} |
| 1057 | |
| 1058 | \subsection{Advanced API\label{doctest-advanced-api}} |
| 1059 | |
| 1060 | The basic API is a simple wrapper that's intended to make doctest easy |
Tim Peters | 8c0a2cf | 2004-09-25 03:02:23 +0000 | [diff] [blame] | 1061 | to use. It is fairly flexible, and should meet most users' needs; |
| 1062 | however, if you require more fine-grained control over testing, or |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1063 | wish to extend doctest's capabilities, then you should use the |
| 1064 | advanced API. |
| 1065 | |
| 1066 | The advanced API revolves around two container classes, which are used |
| 1067 | to store the interactive examples extracted from doctest cases: |
| 1068 | |
| 1069 | \begin{itemize} |
| 1070 | \item \class{Example}: A single python statement, paired with its |
| 1071 | expected output. |
| 1072 | \item \class{DocTest}: A collection of \class{Example}s, typically |
| 1073 | extracted from a single docstring or text file. |
| 1074 | \end{itemize} |
| 1075 | |
| 1076 | Additional processing classes are defined to find, parse, and run, and |
| 1077 | check doctest examples: |
| 1078 | |
| 1079 | \begin{itemize} |
| 1080 | \item \class{DocTestFinder}: Finds all docstrings in a given module, |
| 1081 | and uses a \class{DocTestParser} to create a \class{DocTest} |
| 1082 | from every docstring that contains interactive examples. |
| 1083 | \item \class{DocTestParser}: Creates a \class{DocTest} object from |
| 1084 | a string (such as an object's docstring). |
| 1085 | \item \class{DocTestRunner}: Executes the examples in a |
| 1086 | \class{DocTest}, and uses an \class{OutputChecker} to verify |
| 1087 | their output. |
| 1088 | \item \class{OutputChecker}: Compares the actual output from a |
| 1089 | doctest example with the expected output, and decides whether |
| 1090 | they match. |
| 1091 | \end{itemize} |
| 1092 | |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1093 | The relationships among these processing classes are summarized in the |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1094 | following diagram: |
| 1095 | |
| 1096 | \begin{verbatim} |
| 1097 | list of: |
| 1098 | +------+ +---------+ |
| 1099 | |module| --DocTestFinder-> | DocTest | --DocTestRunner-> results |
| 1100 | +------+ | ^ +---------+ | ^ (printed) |
| 1101 | | | | Example | | | |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1102 | v | | ... | v | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1103 | DocTestParser | Example | OutputChecker |
| 1104 | +---------+ |
| 1105 | \end{verbatim} |
| 1106 | |
| 1107 | \subsubsection{DocTest Objects\label{doctest-DocTest}} |
| 1108 | \begin{classdesc}{DocTest}{examples, globs, name, filename, lineno, |
| 1109 | docstring} |
| 1110 | A collection of doctest examples that should be run in a single |
| 1111 | namespace. The constructor arguments are used to initialize the |
| 1112 | member variables of the same names. |
| 1113 | \versionadded{2.4} |
| 1114 | \end{classdesc} |
| 1115 | |
| 1116 | \class{DocTest} defines the following member variables. They are |
| 1117 | initialized by the constructor, and should not be modified directly. |
| 1118 | |
| 1119 | \begin{memberdesc}{examples} |
| 1120 | A list of \class{Example} objects encoding the individual |
| 1121 | interactive Python examples that should be run by this test. |
| 1122 | \end{memberdesc} |
| 1123 | |
| 1124 | \begin{memberdesc}{globs} |
| 1125 | The namespace (aka globals) that the examples should be run in. |
| 1126 | This is a dictionary mapping names to values. Any changes to the |
| 1127 | namespace made by the examples (such as binding new variables) |
| 1128 | will be reflected in \member{globs} after the test is run. |
| 1129 | \end{memberdesc} |
| 1130 | |
| 1131 | \begin{memberdesc}{name} |
| 1132 | A string name identifying the \class{DocTest}. Typically, this is |
| 1133 | the name of the object or file that the test was extracted from. |
| 1134 | \end{memberdesc} |
| 1135 | |
| 1136 | \begin{memberdesc}{filename} |
| 1137 | The name of the file that this \class{DocTest} was extracted from; |
| 1138 | or \code{None} if the filename is unknown, or if the |
| 1139 | \class{DocTest} was not extracted from a file. |
| 1140 | \end{memberdesc} |
| 1141 | |
| 1142 | \begin{memberdesc}{lineno} |
| 1143 | The line number within \member{filename} where this |
| 1144 | \class{DocTest} begins, or \code{None} if the line number is |
| 1145 | unavailable. This line number is zero-based with respect to the |
| 1146 | beginning of the file. |
| 1147 | \end{memberdesc} |
| 1148 | |
| 1149 | \begin{memberdesc}{docstring} |
| 1150 | The string that the test was extracted from, or `None` if the |
| 1151 | string is unavailable, or if the test was not extracted from a |
| 1152 | string. |
| 1153 | \end{memberdesc} |
| 1154 | |
| 1155 | \subsubsection{Example Objects\label{doctest-Example}} |
| 1156 | \begin{classdesc}{Example}{source, want\optional{, |
| 1157 | exc_msg}\optional{, lineno}\optional{, |
| 1158 | indent}\optional{, options}} |
| 1159 | A single interactive example, consisting of a Python statement and |
| 1160 | its expected output. The constructor arguments are used to |
| 1161 | initialize the member variables of the same names. |
| 1162 | \versionadded{2.4} |
| 1163 | \end{classdesc} |
| 1164 | |
| 1165 | \class{Example} defines the following member variables. They are |
| 1166 | initialized by the constructor, and should not be modified directly. |
| 1167 | |
| 1168 | \begin{memberdesc}{source} |
| 1169 | A string containing the example's source code. This source code |
| 1170 | consists of a single Python statement, and always ends with a |
| 1171 | newline; the constructor adds a newline when necessary. |
| 1172 | \end{memberdesc} |
| 1173 | |
| 1174 | \begin{memberdesc}{want} |
| 1175 | The expected output from running the example's source code (either |
| 1176 | from stdout, or a traceback in case of exception). \member{want} |
| 1177 | ends with a newline unless no output is expected, in which case |
| 1178 | it's an empty string. The constructor adds a newline when |
| 1179 | necessary. |
| 1180 | \end{memberdesc} |
| 1181 | |
| 1182 | \begin{memberdesc}{exc_msg} |
| 1183 | The exception message generated by the example, if the example is |
| 1184 | expected to generate an exception; or \code{None} if it is not |
| 1185 | expected to generate an exception. This exception message is |
| 1186 | compared against the return value of |
| 1187 | \function{traceback.format_exception_only()}. \member{exc_msg} |
| 1188 | ends with a newline unless it's \code{None}. The constructor adds |
| 1189 | a newline if needed. |
| 1190 | \end{memberdesc} |
| 1191 | |
| 1192 | \begin{memberdesc}{lineno} |
| 1193 | The line number within the string containing this example where |
| 1194 | the example begins. This line number is zero-based with respect |
| 1195 | to the beginning of the containing string. |
| 1196 | \end{memberdesc} |
| 1197 | |
| 1198 | \begin{memberdesc}{indent} |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1199 | The example's indentation in the containing string, i.e., the |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1200 | number of space characters that preceed the example's first |
| 1201 | prompt. |
| 1202 | \end{memberdesc} |
| 1203 | |
| 1204 | \begin{memberdesc}{options} |
| 1205 | A dictionary mapping from option flags to \code{True} or |
| 1206 | \code{False}, which is used to override default options for this |
| 1207 | example. Any option flags not contained in this dictionary are |
| 1208 | left at their default value (as specified by the |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1209 | \class{DocTestRunner}'s \member{optionflags}). |
| 1210 | By default, no options are set. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1211 | \end{memberdesc} |
| 1212 | |
| 1213 | \subsubsection{DocTestFinder objects\label{doctest-DocTestFinder}} |
| 1214 | \begin{classdesc}{DocTestFinder}{\optional{verbose}\optional{, |
| 1215 | parser}\optional{, recurse}\optional{, |
| 1216 | exclude_empty}} |
| 1217 | A processing class used to extract the \class{DocTest}s that are |
| 1218 | relevant to a given object, from its docstring and the docstrings |
| 1219 | of its contained objects. \class{DocTest}s can currently be |
| 1220 | extracted from the following object types: modules, functions, |
| 1221 | classes, methods, staticmethods, classmethods, and properties. |
| 1222 | |
| 1223 | The optional argument \var{verbose} can be used to display the |
| 1224 | objects searched by the finder. It defaults to \code{False} (no |
| 1225 | output). |
| 1226 | |
| 1227 | The optional argument \var{parser} specifies the |
| 1228 | \class{DocTestParser} object (or a drop-in replacement) that is |
| 1229 | used to extract doctests from docstrings. |
| 1230 | |
| 1231 | If the optional argument \var{recurse} is false, then |
| 1232 | \method{DocTestFinder.find()} will only examine the given object, |
| 1233 | and not any contained objects. |
| 1234 | |
| 1235 | If the optional argument \var{exclude_empty} is false, then |
| 1236 | \method{DocTestFinder.find()} will include tests for objects with |
| 1237 | empty docstrings. |
| 1238 | |
| 1239 | \versionadded{2.4} |
| 1240 | \end{classdesc} |
| 1241 | |
| 1242 | \class{DocTestFinder} defines the following method: |
| 1243 | |
Tim Peters | 7a08214 | 2004-09-25 00:10:53 +0000 | [diff] [blame] | 1244 | \begin{methoddesc}{find}{obj\optional{, name}\optional{, |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1245 | module}\optional{, globs}\optional{, extraglobs}} |
| 1246 | Return a list of the \class{DocTest}s that are defined by |
| 1247 | \var{obj}'s docstring, or by any of its contained objects' |
| 1248 | docstrings. |
| 1249 | |
| 1250 | The optional argument \var{name} specifies the object's name; this |
| 1251 | name will be used to construct names for the returned |
| 1252 | \class{DocTest}s. If \var{name} is not specified, then |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1253 | \code{\var{obj}.__name__} is used. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1254 | |
| 1255 | The optional parameter \var{module} is the module that contains |
| 1256 | the given object. If the module is not specified or is None, then |
| 1257 | the test finder will attempt to automatically determine the |
| 1258 | correct module. The object's module is used: |
| 1259 | |
| 1260 | \begin{itemize} |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1261 | \item As a default namespace, if \var{globs} is not specified. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1262 | \item To prevent the DocTestFinder from extracting DocTests |
| 1263 | from objects that are imported from other modules. (Contained |
| 1264 | objects with modules other than \var{module} are ignored.) |
| 1265 | \item To find the name of the file containing the object. |
| 1266 | \item To help find the line number of the object within its file. |
| 1267 | \end{itemize} |
| 1268 | |
| 1269 | If \var{module} is \code{False}, no attempt to find the module |
| 1270 | will be made. This is obscure, of use mostly in testing doctest |
| 1271 | itself: if \var{module} is \code{False}, or is \code{None} but |
| 1272 | cannot be found automatically, then all objects are considered to |
| 1273 | belong to the (non-existent) module, so all contained objects will |
| 1274 | (recursively) be searched for doctests. |
| 1275 | |
| 1276 | The globals for each \class{DocTest} is formed by combining |
| 1277 | \var{globs} and \var{extraglobs} (bindings in \var{extraglobs} |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1278 | override bindings in \var{globs}). A new shallow copy of the globals |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1279 | dictionary is created for each \class{DocTest}. If \var{globs} is |
| 1280 | not specified, then it defaults to the module's \var{__dict__}, if |
| 1281 | specified, or \code{\{\}} otherwise. If \var{extraglobs} is not |
| 1282 | specified, then it defaults to \code{\{\}}. |
| 1283 | \end{methoddesc} |
| 1284 | |
| 1285 | \subsubsection{DocTestParser objects\label{doctest-DocTestParser}} |
| 1286 | \begin{classdesc}{DocTestParser}{} |
| 1287 | A processing class used to extract interactive examples from a |
| 1288 | string, and use them to create a \class{DocTest} object. |
| 1289 | \versionadded{2.4} |
| 1290 | \end{classdesc} |
| 1291 | |
| 1292 | \class{DocTestParser} defines the following methods: |
| 1293 | |
| 1294 | \begin{methoddesc}{get_doctest}{string, globs, name, filename, lineno} |
| 1295 | Extract all doctest examples from the given string, and collect |
| 1296 | them into a \class{DocTest} object. |
| 1297 | |
| 1298 | \var{globs}, \var{name}, \var{filename}, and \var{lineno} are |
| 1299 | attributes for the new \class{DocTest} object. See the |
| 1300 | documentation for \class{DocTest} for more information. |
| 1301 | \end{methoddesc} |
| 1302 | |
| 1303 | \begin{methoddesc}{get_examples}{string\optional{, name}} |
| 1304 | Extract all doctest examples from the given string, and return |
| 1305 | them as a list of \class{Example} objects. Line numbers are |
| 1306 | 0-based. The optional argument \var{name} is a name identifying |
| 1307 | this string, and is only used for error messages. |
| 1308 | \end{methoddesc} |
| 1309 | |
| 1310 | \begin{methoddesc}{parse}{string\optional{, name}} |
| 1311 | Divide the given string into examples and intervening text, and |
| 1312 | return them as a list of alternating \class{Example}s and strings. |
| 1313 | Line numbers for the \class{Example}s are 0-based. The optional |
| 1314 | argument \var{name} is a name identifying this string, and is only |
| 1315 | used for error messages. |
| 1316 | \end{methoddesc} |
| 1317 | |
| 1318 | \subsubsection{DocTestRunner objects\label{doctest-DocTestRunner}} |
| 1319 | \begin{classdesc}{DocTestRunner}{\optional{checker}\optional{, |
| 1320 | verbose}\optional{, optionflags}} |
| 1321 | A processing class used to execute and verify the interactive |
| 1322 | examples in a \class{DocTest}. |
| 1323 | |
| 1324 | The comparison between expected outputs and actual outputs is done |
| 1325 | by an \class{OutputChecker}. This comparison may be customized |
| 1326 | with a number of option flags; see section~\ref{doctest-options} |
| 1327 | for more information. If the option flags are insufficient, then |
| 1328 | the comparison may also be customized by passing a subclass of |
| 1329 | \class{OutputChecker} to the constructor. |
| 1330 | |
| 1331 | The test runner's display output can be controlled in two ways. |
| 1332 | First, an output function can be passed to |
| 1333 | \method{TestRunner.run()}; this function will be called with |
| 1334 | strings that should be displayed. It defaults to |
| 1335 | \code{sys.stdout.write}. If capturing the output is not |
| 1336 | sufficient, then the display output can be also customized by |
| 1337 | subclassing DocTestRunner, and overriding the methods |
| 1338 | \method{report_start}, \method{report_success}, |
| 1339 | \method{report_unexpected_exception}, and \method{report_failure}. |
| 1340 | |
| 1341 | The optional keyword argument \var{checker} specifies the |
| 1342 | \class{OutputChecker} object (or drop-in replacement) that should |
| 1343 | be used to compare the expected outputs to the actual outputs of |
| 1344 | doctest examples. |
| 1345 | |
| 1346 | The optional keyword argument \var{verbose} controls the |
| 1347 | \class{DocTestRunner}'s verbosity. If \var{verbose} is |
| 1348 | \code{True}, then information is printed about each example, as it |
| 1349 | is run. If \var{verbose} is \code{False}, then only failures are |
| 1350 | printed. If \var{verbose} is unspecified, or \code{None}, then |
| 1351 | verbose output is used iff the command-line switch \programopt{-v} |
| 1352 | is used. |
| 1353 | |
| 1354 | The optional keyword argument \var{optionflags} can be used to |
| 1355 | control how the test runner compares expected output to actual |
| 1356 | output, and how it displays failures. For more information, see |
| 1357 | section~\ref{doctest-options}. |
| 1358 | |
| 1359 | \versionadded{2.4} |
| 1360 | \end{classdesc} |
| 1361 | |
| 1362 | \class{DocTestParser} defines the following methods: |
| 1363 | |
| 1364 | \begin{methoddesc}{report_start}{out, test, example} |
| 1365 | Report that the test runner is about to process the given example. |
| 1366 | This method is provided to allow subclasses of |
| 1367 | \class{DocTestRunner} to customize their output; it should not be |
| 1368 | called directly. |
| 1369 | |
| 1370 | \var{example} is the example about to be processed. \var{test} is |
| 1371 | the test containing \var{example}. \var{out} is the output |
| 1372 | function that was passed to \method{DocTestRunner.run()}. |
| 1373 | \end{methoddesc} |
| 1374 | |
| 1375 | \begin{methoddesc}{report_success}{out, test, example, got} |
| 1376 | Report that the given example ran successfully. This method is |
| 1377 | provided to allow subclasses of \class{DocTestRunner} to customize |
| 1378 | their output; it should not be called directly. |
| 1379 | |
| 1380 | \var{example} is the example about to be processed. \var{got} is |
| 1381 | the actual output from the example. \var{test} is the test |
| 1382 | containing \var{example}. \var{out} is the output function that |
| 1383 | was passed to \method{DocTestRunner.run()}. |
| 1384 | \end{methoddesc} |
| 1385 | |
| 1386 | \begin{methoddesc}{report_failure}{out, test, example, got} |
| 1387 | Report that the given example failed. This method is provided to |
| 1388 | allow subclasses of \class{DocTestRunner} to customize their |
| 1389 | output; it should not be called directly. |
| 1390 | |
| 1391 | \var{example} is the example about to be processed. \var{got} is |
| 1392 | the actual output from the example. \var{test} is the test |
| 1393 | containing \var{example}. \var{out} is the output function that |
| 1394 | was passed to \method{DocTestRunner.run()}. |
| 1395 | \end{methoddesc} |
| 1396 | |
| 1397 | \begin{methoddesc}{report_unexpected_exception}{out, test, example, exc_info} |
| 1398 | Report that the given example raised an unexpected exception. |
| 1399 | This method is provided to allow subclasses of |
| 1400 | \class{DocTestRunner} to customize their output; it should not be |
| 1401 | called directly. |
| 1402 | |
| 1403 | \var{example} is the example about to be processed. |
| 1404 | \var{exc_info} is a tuple containing information about the |
| 1405 | unexpected exception (as returned by \function{sys.exc_info()}). |
| 1406 | \var{test} is the test containing \var{example}. \var{out} is the |
| 1407 | output function that was passed to \method{DocTestRunner.run()}. |
| 1408 | \end{methoddesc} |
| 1409 | |
| 1410 | \begin{methoddesc}{run}{test\optional{, compileflags}\optional{, |
| 1411 | out}\optional{, clear_globs}} |
| 1412 | Run the examples in \var{test} (a \class{DocTest} object), and |
| 1413 | display the results using the writer function \var{out}. |
| 1414 | |
| 1415 | The examples are run in the namespace \code{test.globs}. If |
| 1416 | \var{clear_globs} is true (the default), then this namespace will |
| 1417 | be cleared after the test runs, to help with garbage collection. |
| 1418 | If you would like to examine the namespace after the test |
| 1419 | completes, then use \var{clear_globs=False}. |
| 1420 | |
| 1421 | \var{compileflags} gives the set of flags that should be used by |
| 1422 | the Python compiler when running the examples. If not specified, |
| 1423 | then it will default to the set of future-import flags that apply |
| 1424 | to \var{globs}. |
| 1425 | |
| 1426 | The output of each example is checked using the |
| 1427 | \class{DocTestRunner}'s output checker, and the results are |
| 1428 | formatted by the \method{DocTestRunner.report_*} methods. |
| 1429 | \end{methoddesc} |
| 1430 | |
| 1431 | \begin{methoddesc}{summarize}{\optional{verbose}} |
| 1432 | Print a summary of all the test cases that have been run by this |
| 1433 | DocTestRunner, and return a tuple \samp{(\var{failure_count}, |
| 1434 | \var{test_count})}. |
| 1435 | |
| 1436 | The optional \var{verbose} argument controls how detailed the |
| 1437 | summary is. If the verbosity is not specified, then the |
| 1438 | \class{DocTestRunner}'s verbosity is used. |
| 1439 | \end{methoddesc} |
| 1440 | |
| 1441 | \subsubsection{OutputChecker objects\label{doctest-OutputChecker}} |
| 1442 | |
| 1443 | \begin{classdesc}{OutputChecker}{} |
| 1444 | A class used to check the whether the actual output from a doctest |
| 1445 | example matches the expected output. \class{OutputChecker} |
| 1446 | defines two methods: \method{check_output}, which compares a given |
| 1447 | pair of outputs, and returns true if they match; and |
| 1448 | \method{output_difference}, which returns a string describing the |
| 1449 | differences between two outputs. |
| 1450 | \versionadded{2.4} |
| 1451 | \end{classdesc} |
| 1452 | |
| 1453 | \class{OutputChecker} defines the following methods: |
| 1454 | |
| 1455 | \begin{methoddesc}{check_output}{want, got, optionflags} |
| 1456 | Return \code{True} iff the actual output from an example |
| 1457 | (\var{got}) matches the expected output (\var{want}). These |
| 1458 | strings are always considered to match if they are identical; but |
| 1459 | depending on what option flags the test runner is using, several |
| 1460 | non-exact match types are also possible. See |
| 1461 | section~\ref{doctest-options} for more information about option |
| 1462 | flags. |
| 1463 | \end{methoddesc} |
| 1464 | |
| 1465 | \begin{methoddesc}{output_difference}{example, got, optionflags} |
| 1466 | Return a string describing the differences between the expected |
| 1467 | output for a given example (\var{example}) and the actual output |
| 1468 | (\var{got}). \var{optionflags} is the set of option flags used to |
| 1469 | compare \var{want} and \var{got}. |
| 1470 | \end{methoddesc} |
| 1471 | |
| 1472 | \subsection{Debugging\label{doctest-debugging}} |
| 1473 | |
| 1474 | Doctest provides three mechanisms for debugging doctest examples: |
| 1475 | |
| 1476 | \begin{enumerate} |
| 1477 | \item The \function{debug()} function converts a specified doctest |
| 1478 | to a Python script, and executes that script using \module{pdb}. |
| 1479 | \item The \class{DebugRunner} class is a subclass of |
| 1480 | \class{DocTestRunner} that raises an exception for the first |
| 1481 | failing example, containing information about that example. |
| 1482 | This information can be used to perform post-mortem debugging on |
| 1483 | the example. |
| 1484 | \item The unittest cases generated by \function{DocTestSuite()} |
| 1485 | support the \method{debug} method defined by |
| 1486 | \class{unittest.TestCase}. |
| 1487 | \end{enumerate} |
| 1488 | |
| 1489 | \begin{funcdesc}{debug}{module, name} |
| 1490 | Debug a single doctest docstring. |
| 1491 | |
| 1492 | Provide the \var{module} (or dotted name of the module) containing |
| 1493 | the docstring to be debugged and the fully qualified dotted |
| 1494 | \var{name} of the object with the docstring to be debugged. |
| 1495 | |
| 1496 | The doctest examples are extracted (see function \function{testsource()}), |
| 1497 | and written to a temporary file. The Python debugger, \refmodule{pdb}, |
| 1498 | is then invoked on that file. |
| 1499 | \versionadded{2.3} |
| 1500 | \end{funcdesc} |
| 1501 | |
| 1502 | \begin{classdesc}{DebugRunner}{\optional{checker}\optional{, |
| 1503 | verbose}\optional{, optionflags}} |
| 1504 | |
| 1505 | A subclass of \class{DocTestRunner} that raises an exception as |
| 1506 | soon as a failure is encountered. If an unexpected exception |
| 1507 | occurs, an \exception{UnexpectedException} exception is raised, |
| 1508 | containing the test, the example, and the original exception. If |
| 1509 | the output doesn't match, then a \exception{DocTestFailure} |
| 1510 | exception is raised, containing the test, the example, and the |
| 1511 | actual output. |
| 1512 | |
| 1513 | For information about the constructor parameters and methods, see |
| 1514 | the documentation for \class{DocTestRunner} in |
| 1515 | section~\ref{doctest-advanced-api}. |
| 1516 | \end{classdesc} |
| 1517 | |
| 1518 | \begin{excclassdesc}{DocTestFailure}{test, example, got} |
| 1519 | An exception thrown by \class{DocTestRunner} to signal that a |
| 1520 | doctest example's actual output did not match its expected output. |
| 1521 | The constructor arguments are used to initialize the member |
| 1522 | variables of the same names. |
| 1523 | \end{excclassdesc} |
| 1524 | \exception{DocTestFailure} defines the following member variables: |
| 1525 | \begin{memberdesc}{test} |
| 1526 | The \class{DocTest} object that was being run when the example failed. |
| 1527 | \end{memberdesc} |
| 1528 | \begin{memberdesc}{example} |
| 1529 | The \class{Example} that failed. |
| 1530 | \end{memberdesc} |
| 1531 | \begin{memberdesc}{got} |
| 1532 | The example's actual output. |
| 1533 | \end{memberdesc} |
| 1534 | |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 1535 | \begin{funcdesc}{testsource}{module, name} |
| 1536 | Extract the doctest examples from a docstring. |
| 1537 | |
| 1538 | Provide the \var{module} (or dotted name of the module) containing the |
| 1539 | tests to be extracted and the \var{name} (within the module) of the object |
| 1540 | with the docstring containing the tests to be extracted. |
| 1541 | |
| 1542 | The doctest examples are returned as a string containing Python |
| 1543 | code. The expected output blocks in the examples are converted |
| 1544 | to Python comments. |
| 1545 | \versionadded{2.3} |
| 1546 | \end{funcdesc} |
| 1547 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1548 | \begin{excclassdesc}{UnexpectedException}{test, example, got} |
| 1549 | An exception thrown by \class{DocTestRunner} to signal that a |
| 1550 | doctest example raised an unexpected exception. The constructor |
| 1551 | arguments are used to initialize the member variables of the same |
| 1552 | names. |
| 1553 | \end{excclassdesc} |
| 1554 | \exception{UnexpectedException} defines the following member variables: |
| 1555 | \begin{memberdesc}{test} |
| 1556 | The \class{DocTest} object that was being run when the example failed. |
| 1557 | \end{memberdesc} |
| 1558 | \begin{memberdesc}{example} |
| 1559 | The \class{Example} that failed. |
| 1560 | \end{memberdesc} |
| 1561 | \begin{memberdesc}{exc_info} |
| 1562 | A tuple containing information about the unexpected exception, as |
| 1563 | returned by \function{sys.exc_info()}. |
| 1564 | \end{memberdesc} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 1565 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1566 | \begin{funcdesc}{register_optionflag}{name} |
| 1567 | Create a new option flag with a given name, and return the new |
| 1568 | flag's integer value. \function{register_optionflag()} can be |
| 1569 | used when subclassing \class{OutputChecker} or |
| 1570 | \class{DocTestRunner} to create new options that are supported by |
| 1571 | your subclasses. \function{register_optionflag} should always be |
| 1572 | called using the following idiom: |
| 1573 | \begin{verbatim} |
| 1574 | MY_FLAG = register_optionflag('MY_FLAG') |
| 1575 | \end{verbatim} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 1576 | \end{funcdesc} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 1577 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1578 | \subsection{Soapbox\label{doctest-soapbox}} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 1579 | |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1580 | As mentioned in the introduction, \module{doctest} has grown to have |
| 1581 | three primary uses: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 1582 | |
| 1583 | \begin{enumerate} |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1584 | \item Checking examples in docstrings. |
| 1585 | \item Regression testing. |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1586 | \item Executable documentation / literate testing. |
Fred Drake | c115835 | 2001-06-11 14:55:01 +0000 | [diff] [blame] | 1587 | \end{enumerate} |
| 1588 | |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1589 | These uses have different requirements, and it is important to |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1590 | distinguish them. In particular, filling your docstrings with obscure |
| 1591 | test cases makes for bad documentation. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 1592 | |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1593 | When writing a docstring, choose docstring examples with care. |
| 1594 | There's an art to this that needs to be learned---it may not be |
| 1595 | natural at first. Examples should add genuine value to the |
| 1596 | documentation. A good example can often be worth many words. |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 1597 | If done with care, the examples will be invaluable for your users, and |
| 1598 | will pay back the time it takes to collect them many times over as the |
| 1599 | years go by and things change. I'm still amazed at how often one of |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1600 | my \refmodule{doctest} examples stops working after a "harmless" |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 1601 | change. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 1602 | |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1603 | Doctest also makes an excellent tool for regression testing, especially if |
| 1604 | you don't skimp on explanatory text. By interleaving prose and examples, |
| 1605 | it becomes much easier to keep track of what's actually being tested, and |
| 1606 | why. When a test fails, good prose can make it much easier to figure out |
| 1607 | what the problem is, and how it should be fixed. It's true that you could |
| 1608 | write extensive comments in code-based testing, but few programmers do. |
| 1609 | Many have found that using doctest approaches instead leads to much clearer |
| 1610 | tests. Perhaps this is simply because doctest makes writing prose a little |
| 1611 | easier than writing code, while writing comments in code is a little |
| 1612 | harder. I think it goes deeper than just that: the natural attitude |
| 1613 | when writing a doctest-based test is that you want to explain the fine |
| 1614 | points of your software, and illustrate them with examples. This in |
| 1615 | turn naturally leads to test files that start with the simplest features, |
| 1616 | and logically progress to complications and edge cases. A coherent |
| 1617 | narrative is the result, instead of a collection of isolated functions |
| 1618 | that test isolated bits of functionality seemingly at random. It's |
| 1619 | a different attitude, and produces different results, blurring the |
| 1620 | distinction between testing and explaining. |
| 1621 | |
| 1622 | Regression testing is best confined to dedicated objects or files. There |
| 1623 | are several options for organizing tests: |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1624 | |
| 1625 | \begin{itemize} |
Tim Peters | 3f79125 | 2004-09-25 03:50:35 +0000 | [diff] [blame^] | 1626 | \item Write text files containing test cases as interactive examples, |
| 1627 | and test the files using \function{testfile()} or |
| 1628 | \function{DocFileSuite()}. This is recommended, although is |
| 1629 | easiest to do for new projects, designed from the start to use |
| 1630 | doctest. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1631 | \item Define functions named \code{_regrtest_\textit{topic}} that |
| 1632 | consist of single docstrings, containing test cases for the |
| 1633 | named topics. These functions can be included in the same file |
| 1634 | as the module, or separated out into a separate test file. |
| 1635 | \item Define a \code{__test__} dictionary mapping from regression test |
| 1636 | topics to docstrings containing test cases. |
Edward Loper | b3666a3 | 2004-09-21 03:00:51 +0000 | [diff] [blame] | 1637 | \end{itemize} |