Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 1 | \section{\module{doctest} --- |
| 2 | Test docstrings represent reality} |
| 3 | |
| 4 | \declaremodule{standard}{doctest} |
| 5 | \moduleauthor{Tim Peters}{tim_one@users.sourceforge.net} |
| 6 | \sectionauthor{Tim Peters}{tim_one@users.sourceforge.net} |
| 7 | \sectionauthor{Moshe Zadka}{moshez@debian.org} |
| 8 | |
| 9 | \modulesynopsis{A framework for verifying examples in docstrings.} |
| 10 | |
| 11 | The \module{doctest} module searches a module's docstrings for text that looks |
| 12 | like an interactive Python session, then executes all such sessions to verify |
| 13 | they still work exactly as shown. Here's a complete but small example: |
| 14 | |
| 15 | \begin{verbatim} |
| 16 | """ |
| 17 | This is module example. |
| 18 | |
| 19 | Example supplies one function, factorial. For example, |
| 20 | |
| 21 | >>> factorial(5) |
| 22 | 120 |
| 23 | """ |
| 24 | |
| 25 | def factorial(n): |
| 26 | """Return the factorial of n, an exact integer >= 0. |
| 27 | |
| 28 | If the result is small enough to fit in an int, return an int. |
| 29 | Else return a long. |
| 30 | |
| 31 | >>> [factorial(n) for n in range(6)] |
| 32 | [1, 1, 2, 6, 24, 120] |
| 33 | >>> [factorial(long(n)) for n in range(6)] |
| 34 | [1, 1, 2, 6, 24, 120] |
| 35 | >>> factorial(30) |
| 36 | 265252859812191058636308480000000L |
| 37 | >>> factorial(30L) |
| 38 | 265252859812191058636308480000000L |
| 39 | >>> factorial(-1) |
| 40 | Traceback (most recent call last): |
| 41 | ... |
| 42 | ValueError: n must be >= 0 |
| 43 | |
| 44 | Factorials of floats are OK, but the float must be an exact integer: |
| 45 | >>> factorial(30.1) |
| 46 | Traceback (most recent call last): |
| 47 | ... |
| 48 | ValueError: n must be exact integer |
| 49 | >>> factorial(30.0) |
| 50 | 265252859812191058636308480000000L |
| 51 | |
| 52 | It must also not be ridiculously large: |
| 53 | >>> factorial(1e100) |
| 54 | Traceback (most recent call last): |
| 55 | ... |
| 56 | OverflowError: n too large |
| 57 | """ |
| 58 | |
| 59 | \end{verbatim} |
| 60 | % allow LaTeX to break here. |
| 61 | \begin{verbatim} |
| 62 | |
| 63 | import math |
| 64 | if not n >= 0: |
| 65 | raise ValueError("n must be >= 0") |
| 66 | if math.floor(n) != n: |
| 67 | raise ValueError("n must be exact integer") |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 68 | if n+1 == n: # catch a value like 1e300 |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 69 | raise OverflowError("n too large") |
| 70 | result = 1 |
| 71 | factor = 2 |
| 72 | while factor <= n: |
| 73 | try: |
| 74 | result *= factor |
| 75 | except OverflowError: |
| 76 | result *= long(factor) |
| 77 | factor += 1 |
| 78 | return result |
| 79 | |
| 80 | def _test(): |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 81 | import doctest |
| 82 | return doctest.testmod() |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 83 | |
| 84 | if __name__ == "__main__": |
| 85 | _test() |
| 86 | \end{verbatim} |
| 87 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 88 | If you run \file{example.py} directly from the command line, |
| 89 | \module{doctest} works its magic: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 90 | |
| 91 | \begin{verbatim} |
| 92 | $ python example.py |
| 93 | $ |
| 94 | \end{verbatim} |
| 95 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 96 | There's no output! That's normal, and it means all the examples |
| 97 | worked. Pass \programopt{-v} to the script, and \module{doctest} |
| 98 | prints a detailed log of what it's trying, and prints a summary at the |
| 99 | end: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 100 | |
| 101 | \begin{verbatim} |
| 102 | $ python example.py -v |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 103 | Trying: |
| 104 | factorial(5) |
| 105 | Expecting: |
| 106 | 120 |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 107 | ok |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 108 | Trying: |
| 109 | [factorial(n) for n in range(6)] |
| 110 | Expecting: |
| 111 | [1, 1, 2, 6, 24, 120] |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 112 | ok |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 113 | Trying: |
| 114 | [factorial(long(n)) for n in range(6)] |
| 115 | Expecting: |
| 116 | [1, 1, 2, 6, 24, 120] |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 117 | ok |
| 118 | \end{verbatim} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 119 | |
| 120 | And so on, eventually ending with: |
| 121 | |
| 122 | \begin{verbatim} |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 123 | Trying: |
| 124 | factorial(1e100) |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 125 | Expecting: |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 126 | Traceback (most recent call last): |
| 127 | ... |
| 128 | OverflowError: n too large |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 129 | ok |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 130 | 2 items passed all tests: |
| 131 | 1 tests in example |
| 132 | 8 tests in example.factorial |
| 133 | 9 tests in 2 items. |
| 134 | 9 passed and 0 failed. |
| 135 | Test passed. |
| 136 | $ |
| 137 | \end{verbatim} |
| 138 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 139 | That's all you need to know to start making productive use of |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 140 | \module{doctest}! Jump in. The following sections provide full |
| 141 | details. Note that there are many examples of doctests in |
| 142 | the standard Python test suite and libraries. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 143 | |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 144 | \subsection{Simple Usage} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 145 | |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 146 | The simplest way to start using doctest (but not necessarily the way |
| 147 | you'll continue to do it) is to end each module \module{M} with: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 148 | |
| 149 | \begin{verbatim} |
| 150 | def _test(): |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 151 | import doctest |
| 152 | return doctest.testmod() |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 153 | |
| 154 | if __name__ == "__main__": |
| 155 | _test() |
| 156 | \end{verbatim} |
| 157 | |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 158 | \module{doctest} then examines docstrings in the module calling |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 159 | \function{testmod()}. |
Martin v. Löwis | 4581cfa | 2002-11-22 08:23:09 +0000 | [diff] [blame] | 160 | |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 161 | Running the module as a script causes the examples in the docstrings |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 162 | to get executed and verified: |
| 163 | |
| 164 | \begin{verbatim} |
| 165 | python M.py |
| 166 | \end{verbatim} |
| 167 | |
| 168 | This won't display anything unless an example fails, in which case the |
| 169 | failing example(s) and the cause(s) of the failure(s) are printed to stdout, |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 170 | and the final line of output is |
Tim Peters | 2603960 | 2004-08-13 01:49:12 +0000 | [diff] [blame] | 171 | \samp{'***Test Failed*** \var{N} failures.'}, where \var{N} is the |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 172 | number of examples that failed. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 173 | |
Fred Drake | 7eb1463 | 2001-02-17 17:32:41 +0000 | [diff] [blame] | 174 | Run it with the \programopt{-v} switch instead: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 175 | |
| 176 | \begin{verbatim} |
| 177 | python M.py -v |
| 178 | \end{verbatim} |
| 179 | |
Fred Drake | 8836e56 | 2003-07-17 15:22:47 +0000 | [diff] [blame] | 180 | and a detailed report of all examples tried is printed to standard |
| 181 | output, along with assorted summaries at the end. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 182 | |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 183 | You can force verbose mode by passing \code{verbose=True} to |
Fred Drake | 5d2f515 | 2003-06-28 03:09:06 +0000 | [diff] [blame] | 184 | \function{testmod()}, or |
Tim Peters | c2388a2 | 2004-08-10 01:41:28 +0000 | [diff] [blame] | 185 | prohibit it by passing \code{verbose=False}. In either of those cases, |
Fred Drake | 5d2f515 | 2003-06-28 03:09:06 +0000 | [diff] [blame] | 186 | \code{sys.argv} is not examined by \function{testmod()}. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 187 | |
Fred Drake | 5d2f515 | 2003-06-28 03:09:06 +0000 | [diff] [blame] | 188 | In any case, \function{testmod()} returns a 2-tuple of ints \code{(\var{f}, |
Fred Drake | 7eb1463 | 2001-02-17 17:32:41 +0000 | [diff] [blame] | 189 | \var{t})}, where \var{f} is the number of docstring examples that |
| 190 | failed and \var{t} is the total number of docstring examples |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 191 | tried. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 192 | |
| 193 | \subsection{Which Docstrings Are Examined?} |
| 194 | |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 195 | The module docstring, and all function, class and method docstrings are |
| 196 | searched. Objects imported into the module are not searched. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 197 | |
Fred Drake | 7eb1463 | 2001-02-17 17:32:41 +0000 | [diff] [blame] | 198 | In addition, if \code{M.__test__} exists and "is true", it must be a |
| 199 | dict, and each entry maps a (string) name to a function object, class |
| 200 | object, or string. Function and class object docstrings found from |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 201 | \code{M.__test__} are searched, and strings are treated as if they |
| 202 | were docstrings. In output, a key \code{K} in \code{M.__test__} appears |
| 203 | with name |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 204 | |
| 205 | \begin{verbatim} |
Fred Drake | 8836e56 | 2003-07-17 15:22:47 +0000 | [diff] [blame] | 206 | <name of M>.__test__.K |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 207 | \end{verbatim} |
| 208 | |
| 209 | Any classes found are recursively searched similarly, to test docstrings in |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 210 | their contained methods and nested classes. |
| 211 | |
| 212 | \versionchanged[A "private name" concept is deprecated and no longer |
Tim Peters | 2603960 | 2004-08-13 01:49:12 +0000 | [diff] [blame] | 213 | documented]{2.4} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 214 | |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 215 | |
| 216 | \subsection{What's the Execution Context?} |
| 217 | |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 218 | By default, each time \function{testmod()} finds a docstring to test, it |
| 219 | uses a \emph{shallow copy} of \module{M}'s globals, so that running tests |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 220 | doesn't change the module's real globals, and so that one test in |
| 221 | \module{M} can't leave behind crumbs that accidentally allow another test |
| 222 | to work. This means examples can freely use any names defined at top-level |
Tim Peters | 0481d24 | 2001-10-02 21:01:22 +0000 | [diff] [blame] | 223 | in \module{M}, and names defined earlier in the docstring being run. |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 224 | Examples cannot see names defined in other docstrings. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 225 | |
| 226 | You can force use of your own dict as the execution context by passing |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 227 | \code{globs=your_dict} to \function{testmod()} instead. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 228 | |
| 229 | \subsection{What About Exceptions?} |
| 230 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 231 | No problem, provided that the traceback is the only output produced by |
| 232 | the example: just paste in the traceback. Since tracebacks contain |
| 233 | details that are likely to change rapidly (for example, exact file paths |
| 234 | and line numbers), this is one case where doctest works hard to be |
| 235 | flexible in what it accepts. |
| 236 | |
| 237 | Simple example: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 238 | |
| 239 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 240 | >>> [1, 2, 3].remove(42) |
| 241 | Traceback (most recent call last): |
| 242 | File "<stdin>", line 1, in ? |
| 243 | ValueError: list.remove(x): x not in list |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 244 | \end{verbatim} |
| 245 | |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 246 | That doctest succeeds if \exception{ValueError} is raised, with the |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 247 | \samp{list.remove(x): x not in list} detail as shown. |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 248 | |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 249 | The expected output for an exception must start with a traceback |
| 250 | header, which may be either of the following two lines, indented the |
| 251 | same as the first line of the example: |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 252 | |
| 253 | \begin{verbatim} |
| 254 | Traceback (most recent call last): |
| 255 | Traceback (innermost last): |
| 256 | \end{verbatim} |
| 257 | |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 258 | The traceback header is followed by an optional traceback stack, whose |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 259 | contents are ignored by doctest. The traceback stack is typically |
| 260 | omitted, or copied verbatim from an interactive session. |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 261 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 262 | The traceback stack is followed by the most interesting part: the |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 263 | line(s) containing the exception type and detail. This is usually the |
| 264 | last line of a traceback, but can extend across multiple lines if the |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 265 | exception has a multi-line detail: |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 266 | |
| 267 | \begin{verbatim} |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 268 | >>> raise ValueError('multi\n line\ndetail') |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 269 | Traceback (most recent call last): |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 270 | File "<stdin>", line 1, in ? |
| 271 | ValueError: multi |
| 272 | line |
| 273 | detail |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 274 | \end{verbatim} |
| 275 | |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 276 | The last three lines (starting with \exception{ValueError}) are |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 277 | compared against the exception's type and detail, and the rest are |
| 278 | ignored. |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 279 | |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 280 | Best practice is to omit the traceback stack, unless it adds |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 281 | significant documentation value to the example. So the last example |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 282 | is probably better as: |
| 283 | |
| 284 | \begin{verbatim} |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 285 | >>> raise ValueError('multi\n line\ndetail') |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 286 | Traceback (most recent call last): |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 287 | ... |
| 288 | ValueError: multi |
| 289 | line |
| 290 | detail |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 291 | \end{verbatim} |
| 292 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 293 | Note that tracebacks are treated very specially. In particular, in the |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 294 | rewritten example, the use of \samp{...} is independent of doctest's |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 295 | \constant{ELLIPSIS} option. The ellipsis in that example could be left |
| 296 | out, or could just as well be three (or three hundred) commas or digits, |
| 297 | or an indented transcript of a Monty Python skit. |
| 298 | |
| 299 | Some details you should read once, but won't need to remember: |
| 300 | |
| 301 | \begin{itemize} |
| 302 | |
| 303 | \item Doctest can't guess whether your expected output came from an |
| 304 | exception traceback or from ordinary printing. So, e.g., an example |
| 305 | that expects \samp{ValueError: 42 is prime} will pass whether |
| 306 | \exception{ValueError} is actually raised or if the example merely |
| 307 | prints that traceback text. In practice, ordinary output rarely begins |
| 308 | with a traceback header line, so this doesn't create real problems. |
| 309 | |
| 310 | \item Each line of the traceback stack (if present) must be indented |
| 311 | further than the first line of the example, \emph{or} start with a |
| 312 | non-alphanumeric character. The first line following the traceback |
| 313 | header indented the same and starting with an alphanumeric is taken |
| 314 | to be the start of the exception detail. Of course this does the |
| 315 | right thing for genuine tracebacks. |
| 316 | |
Tim Peters | 1fbf9c5 | 2004-09-04 17:21:02 +0000 | [diff] [blame] | 317 | \item When the \constant{IGNORE_EXCEPTION_DETAIL} doctest option is |
| 318 | is specified, everything following the leftmost colon is ignored. |
| 319 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 320 | \end{itemize} |
Tim Peters | 41a65ea | 2004-08-13 03:55:05 +0000 | [diff] [blame] | 321 | |
Tim Peters | 0e44807 | 2004-08-26 01:02:08 +0000 | [diff] [blame] | 322 | \versionchanged[The ability to handle a multi-line exception detail |
| 323 | was added]{2.4} |
| 324 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 325 | |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 326 | \subsection{Option Flags and Directives\label{doctest-options}} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 327 | |
Tim Peters | cf53355 | 2004-08-26 04:50:38 +0000 | [diff] [blame] | 328 | A number of option flags control various aspects of doctest's |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 329 | behavior. Symbolic names for the flags are supplied as module constants, |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 330 | which can be or'ed together and passed to various functions. The names |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 331 | can also be used in doctest directives (see below). |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 332 | |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 333 | The first group of options define test semantics, controlling |
| 334 | aspects of how doctest decides whether actual output matches an |
| 335 | example's expected output: |
| 336 | |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 337 | \begin{datadesc}{DONT_ACCEPT_TRUE_FOR_1} |
| 338 | By default, if an expected output block contains just \code{1}, |
| 339 | an actual output block containing just \code{1} or just |
| 340 | \code{True} is considered to be a match, and similarly for \code{0} |
| 341 | versus \code{False}. When \constant{DONT_ACCEPT_TRUE_FOR_1} is |
| 342 | specified, neither substitution is allowed. The default behavior |
| 343 | caters to that Python changed the return type of many functions |
| 344 | from integer to boolean; doctests expecting "little integer" |
| 345 | output still work in these cases. This option will probably go |
| 346 | away, but not for several years. |
| 347 | \end{datadesc} |
| 348 | |
| 349 | \begin{datadesc}{DONT_ACCEPT_BLANKLINE} |
| 350 | By default, if an expected output block contains a line |
| 351 | containing only the string \code{<BLANKLINE>}, then that line |
| 352 | will match a blank line in the actual output. Because a |
| 353 | genuinely blank line delimits the expected output, this is |
| 354 | the only way to communicate that a blank line is expected. When |
| 355 | \constant{DONT_ACCEPT_BLANKLINE} is specified, this substitution |
| 356 | is not allowed. |
| 357 | \end{datadesc} |
| 358 | |
| 359 | \begin{datadesc}{NORMALIZE_WHITESPACE} |
| 360 | When specified, all sequences of whitespace (blanks and newlines) are |
| 361 | treated as equal. Any sequence of whitespace within the expected |
| 362 | output will match any sequence of whitespace within the actual output. |
| 363 | By default, whitespace must match exactly. |
| 364 | \constant{NORMALIZE_WHITESPACE} is especially useful when a line |
| 365 | of expected output is very long, and you want to wrap it across |
| 366 | multiple lines in your source. |
| 367 | \end{datadesc} |
| 368 | |
| 369 | \begin{datadesc}{ELLIPSIS} |
| 370 | When specified, an ellipsis marker (\code{...}) in the expected output |
| 371 | can match any substring in the actual output. This includes |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 372 | substrings that span line boundaries, and empty substrings, so it's |
| 373 | best to keep usage of this simple. Complicated uses can lead to the |
| 374 | same kinds of "oops, it matched too much!" surprises that \regexp{.*} |
| 375 | is prone to in regular expressions. |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 376 | \end{datadesc} |
| 377 | |
Tim Peters | 1fbf9c5 | 2004-09-04 17:21:02 +0000 | [diff] [blame] | 378 | \begin{datadesc}{IGNORE_EXCEPTION_DETAIL} |
| 379 | When specified, an example that expects an exception passes if |
| 380 | an exception of the expected type is raised, even if the exception |
| 381 | detail does not match. For example, an example expecting |
| 382 | \samp{ValueError: 42} will pass if the actual exception raised is |
| 383 | \samp{ValueError: 3*14}, but will fail, e.g., if |
| 384 | \exception{TypeError} is raised. |
| 385 | |
| 386 | Note that a similar effect can be obtained using \constant{ELLIPSIS}, |
| 387 | and \constant{IGNORE_EXCEPTION_DETAIL} may go away when Python releases |
| 388 | prior to 2.4 become uninteresting. Until then, |
| 389 | \constant{IGNORE_EXCEPTION_DETAIL} is the only clear way to write a |
| 390 | doctest that doesn't care about the exception detail yet continues |
| 391 | to pass under Python releases prior to 2.4 (doctest directives |
| 392 | appear to be comments to them). For example, |
| 393 | |
| 394 | \begin{verbatim} |
| 395 | >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL |
| 396 | Traceback (most recent call last): |
| 397 | File "<stdin>", line 1, in ? |
| 398 | TypeError: object doesn't support item assignment |
| 399 | \end{verbatim} |
| 400 | |
| 401 | passes under Python 2.4 and Python 2.3. The detail changed in 2.4, |
| 402 | to say "does not" instead of "doesn't". |
| 403 | |
| 404 | \end{datadesc} |
| 405 | |
Tim Peters | 38330fe | 2004-08-30 16:19:24 +0000 | [diff] [blame] | 406 | \begin{datadesc}{COMPARISON_FLAGS} |
| 407 | A bitmask or'ing together all the comparison flags above. |
| 408 | \end{datadesc} |
| 409 | |
Tim Peters | f33683f | 2004-08-26 04:52:46 +0000 | [diff] [blame] | 410 | The second group of options controls how test failures are reported: |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 411 | |
Edward Loper | 71f55af | 2004-08-26 01:41:51 +0000 | [diff] [blame] | 412 | \begin{datadesc}{REPORT_UDIFF} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 413 | When specified, failures that involve multi-line expected and |
| 414 | actual outputs are displayed using a unified diff. |
| 415 | \end{datadesc} |
| 416 | |
Edward Loper | 71f55af | 2004-08-26 01:41:51 +0000 | [diff] [blame] | 417 | \begin{datadesc}{REPORT_CDIFF} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 418 | When specified, failures that involve multi-line expected and |
| 419 | actual outputs will be displayed using a context diff. |
| 420 | \end{datadesc} |
| 421 | |
Edward Loper | 71f55af | 2004-08-26 01:41:51 +0000 | [diff] [blame] | 422 | \begin{datadesc}{REPORT_NDIFF} |
Tim Peters | c6cbab0 | 2004-08-22 19:43:28 +0000 | [diff] [blame] | 423 | When specified, differences are computed by \code{difflib.Differ}, |
| 424 | using the same algorithm as the popular \file{ndiff.py} utility. |
| 425 | This is the only method that marks differences within lines as |
| 426 | well as across lines. For example, if a line of expected output |
| 427 | contains digit \code{1} where actual output contains letter \code{l}, |
| 428 | a line is inserted with a caret marking the mismatching column |
| 429 | positions. |
| 430 | \end{datadesc} |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 431 | |
Edward Loper | a89f88d | 2004-08-26 02:45:51 +0000 | [diff] [blame] | 432 | \begin{datadesc}{REPORT_ONLY_FIRST_FAILURE} |
| 433 | When specified, display the first failing example in each doctest, |
| 434 | but suppress output for all remaining examples. This will prevent |
| 435 | doctest from reporting correct examples that break because of |
| 436 | earlier failures; but it might also hide incorrect examples that |
| 437 | fail independently of the first failure. When |
| 438 | \constant{REPORT_ONLY_FIRST_FAILURE} is specified, the remaining |
| 439 | examples are still run, and still count towards the total number of |
| 440 | failures reported; only the output is suppressed. |
| 441 | \end{datadesc} |
| 442 | |
Tim Peters | 38330fe | 2004-08-30 16:19:24 +0000 | [diff] [blame] | 443 | \begin{datadesc}{REPORTING_FLAGS} |
| 444 | A bitmask or'ing together all the reporting flags above. |
| 445 | \end{datadesc} |
| 446 | |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 447 | A "doctest directive" is a trailing Python comment on a line of a doctest |
| 448 | example: |
| 449 | |
| 450 | \begin{productionlist}[doctest] |
| 451 | \production{directive} |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 452 | {"\#" "doctest:" \token{directive_options}} |
| 453 | \production{directive_options} |
| 454 | {\token{directive_option} ("," \token{directive_option})*} |
| 455 | \production{directive_option} |
| 456 | {\token{on_or_off} \token{directive_option_name}} |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 457 | \production{on_or_off} |
| 458 | {"+" | "-"} |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 459 | \production{directive_option_name} |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 460 | {"DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...} |
| 461 | \end{productionlist} |
| 462 | |
| 463 | Whitespace is not allowed between the \code{+} or \code{-} and the |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 464 | directive option name. The directive option name can be any of the |
| 465 | option names explained above. |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 466 | |
| 467 | The doctest directives appearing in a single example modify doctest's |
| 468 | behavior for that single example. Use \code{+} to enable the named |
| 469 | behavior, or \code{-} to disable it. |
| 470 | |
| 471 | For example, this test passes: |
| 472 | |
| 473 | \begin{verbatim} |
| 474 | >>> print range(20) #doctest: +NORMALIZE_WHITESPACE |
| 475 | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, |
| 476 | 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] |
| 477 | \end{verbatim} |
| 478 | |
| 479 | Without the directive it would fail, both because the actual output |
| 480 | doesn't have two blanks before the single-digit list elements, and |
| 481 | because the actual output is on a single line. This test also passes, |
Tim Peters | a07bcd4 | 2004-08-26 04:47:31 +0000 | [diff] [blame] | 482 | and also requires a directive to do so: |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 483 | |
| 484 | \begin{verbatim} |
| 485 | >>> print range(20) # doctest:+ELLIPSIS |
| 486 | [0, 1, ..., 18, 19] |
| 487 | \end{verbatim} |
| 488 | |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 489 | Multiple directives can be used on a single physical line, separated |
| 490 | by commas: |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 491 | |
| 492 | \begin{verbatim} |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 493 | >>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 494 | [0, 1, ..., 18, 19] |
| 495 | \end{verbatim} |
| 496 | |
Edward Loper | 6cc1350 | 2004-09-19 01:16:44 +0000 | [diff] [blame^] | 497 | If multiple directives are used for a single example, then they are |
| 498 | combined: |
| 499 | |
| 500 | \begin{verbatim} |
| 501 | >>> print range(20) # doctest: +ELLIPSIS |
| 502 | ... # doctest: +NORMALIZE_WHITESPACE |
| 503 | [0, 1, ..., 18, 19] |
| 504 | \end{verbatim} |
| 505 | |
| 506 | As the previous example shows, you can add \samp{...} lines to your |
| 507 | example containing only directives. This can also be useful when an |
| 508 | example is too long for a directive to comfortably fit on the same |
| 509 | line: |
| 510 | |
| 511 | \begin{verbatim} |
| 512 | >>> print range(5) + range(10,20) + range(30,40) + range(50,60) |
| 513 | ... # doctest: +ELLIPSIS |
| 514 | [0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59] |
| 515 | \end{verbatim} |
| 516 | |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 517 | Note that since all options are disabled by default, and directives apply |
| 518 | only to the example they appear in, enabling options (via \code{+} in a |
| 519 | directive) is usually the only meaningful choice. However, option flags |
| 520 | can also be passed to functions that run doctests, establishing different |
| 521 | defaults. In such cases, disabling an option via \code{-} in a directive |
| 522 | can be useful. |
| 523 | |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 524 | \versionchanged[Constants \constant{DONT_ACCEPT_BLANKLINE}, |
| 525 | \constant{NORMALIZE_WHITESPACE}, \constant{ELLIPSIS}, |
Tim Peters | 1fbf9c5 | 2004-09-04 17:21:02 +0000 | [diff] [blame] | 526 | \constant{IGNORE_EXCEPTION_DETAIL}, |
Edward Loper | a89f88d | 2004-08-26 02:45:51 +0000 | [diff] [blame] | 527 | \constant{REPORT_UDIFF}, \constant{REPORT_CDIFF}, |
Tim Peters | 38330fe | 2004-08-30 16:19:24 +0000 | [diff] [blame] | 528 | \constant{REPORT_NDIFF}, \constant{REPORT_ONLY_FIRST_FAILURE}, |
| 529 | \constant{COMPARISON_FLAGS} and \constant{REPORTING_FLAGS} |
Tim Peters | 026f8dc | 2004-08-19 16:38:58 +0000 | [diff] [blame] | 530 | were added; by default \code{<BLANKLINE>} in expected output |
| 531 | matches an empty line in actual output; and doctest directives |
| 532 | were added]{2.4} |
| 533 | |
Tim Peters | 8a3b69c | 2004-08-12 22:31:25 +0000 | [diff] [blame] | 534 | |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 535 | \subsection{Advanced Usage} |
| 536 | |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 537 | Several module level functions are available for controlling how doctests |
| 538 | are run. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 539 | |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 540 | \begin{funcdesc}{debug}{module, name} |
| 541 | Debug a single docstring containing doctests. |
| 542 | |
| 543 | Provide the \var{module} (or dotted name of the module) containing the |
| 544 | docstring to be debugged and the \var{name} (within the module) of the |
| 545 | object with the docstring to be debugged. |
| 546 | |
| 547 | The doctest examples are extracted (see function \function{testsource()}), |
| 548 | and written to a temporary file. The Python debugger, \refmodule{pdb}, |
Fred Drake | 8836e56 | 2003-07-17 15:22:47 +0000 | [diff] [blame] | 549 | is then invoked on that file. |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 550 | \versionadded{2.3} |
| 551 | \end{funcdesc} |
| 552 | |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 553 | \begin{funcdesc}{testmod}{\optional{m}\optional{, name}\optional{, |
| 554 | globs}\optional{, verbose}\optional{, |
| 555 | isprivate}\optional{, report}\optional{, |
| 556 | optionflags}\optional{, extraglobs}\optional{, |
Tim Peters | 8278860 | 2004-09-13 15:03:17 +0000 | [diff] [blame] | 557 | raise_on_error}\optional{, exclude_empty}} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 558 | |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 559 | All arguments are optional, and all except for \var{m} should be |
| 560 | specified in keyword form. |
| 561 | |
| 562 | Test examples in docstrings in functions and classes reachable |
| 563 | from module \var{m} (or the current module if \var{m} is not supplied |
| 564 | or is \code{None}), starting with \code{\var{m}.__doc__}. |
| 565 | |
| 566 | Also test examples reachable from dict \code{\var{m}.__test__}, if it |
| 567 | exists and is not \code{None}. \code{\var{m}.__test__} maps |
| 568 | names (strings) to functions, classes and strings; function and class |
| 569 | docstrings are searched for examples; strings are searched directly, |
| 570 | as if they were docstrings. |
| 571 | |
| 572 | Only docstrings attached to objects belonging to module \var{m} are |
| 573 | searched. |
| 574 | |
| 575 | Return \samp{(\var{failure_count}, \var{test_count})}. |
| 576 | |
| 577 | Optional argument \var{name} gives the name of the module; by default, |
| 578 | or if \code{None}, \code{\var{m}.__name__} is used. |
| 579 | |
| 580 | Optional argument \var{globs} gives a dict to be used as the globals |
| 581 | when executing examples; by default, or if \code{None}, |
| 582 | \code{\var{m}.__dict__} is used. A new shallow copy of this dict is |
| 583 | created for each docstring with examples, so that each docstring's |
| 584 | examples start with a clean slate. |
| 585 | |
| 586 | Optional argument \var{extraglobs} gives a dict merged into the |
| 587 | globals used to execute examples. This works like |
| 588 | \method{dict.update()}: if \var{globs} and \var{extraglobs} have a |
| 589 | common key, the associated value in \var{extraglobs} appears in the |
| 590 | combined dict. By default, or if \code{None}, no extra globals are |
| 591 | used. This is an advanced feature that allows parameterization of |
| 592 | doctests. For example, a doctest can be written for a base class, using |
| 593 | a generic name for the class, then reused to test any number of |
| 594 | subclasses by passing an \var{extraglobs} dict mapping the generic |
| 595 | name to the subclass to be tested. |
| 596 | |
| 597 | Optional argument \var{verbose} prints lots of stuff if true, and prints |
| 598 | only failures if false; by default, or if \code{None}, it's true |
| 599 | if and only if \code{'-v'} is in \code{sys.argv}. |
| 600 | |
| 601 | Optional argument \var{report} prints a summary at the end when true, |
| 602 | else prints nothing at the end. In verbose mode, the summary is |
| 603 | detailed, else the summary is very brief (in fact, empty if all tests |
| 604 | passed). |
| 605 | |
| 606 | Optional argument \var{optionflags} or's together option flags. See |
| 607 | see section \ref{doctest-options}. |
| 608 | |
| 609 | Optional argument \var{raise_on_error} defaults to false. If true, |
| 610 | an exception is raised upon the first failure or unexpected exception |
| 611 | in an example. This allows failures to be post-mortem debugged. |
| 612 | Default behavior is to continue running examples. |
| 613 | |
Tim Peters | 8278860 | 2004-09-13 15:03:17 +0000 | [diff] [blame] | 614 | Optional argument \var{exclude_empty} defaults to false. If true, |
| 615 | objects for which no doctests are found are excluded from consideration. |
| 616 | The default is a backward compatibility hack, so that code still |
| 617 | using \method{doctest.master.summarize()} in conjunction with |
| 618 | \function{testmod()} continues to get output for objects with no tests. |
| 619 | The \var{exclude_empty} argument to the newer \class{DocTestFinder} |
| 620 | constructor defaults to true. |
| 621 | |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 622 | Optional argument \var{isprivate} specifies a function used to |
| 623 | determine whether a name is private. The default function treats |
| 624 | all names as public. \var{isprivate} can be set to |
| 625 | \code{doctest.is_private} to skip over names that are |
| 626 | private according to Python's underscore naming convention. |
| 627 | \deprecated{2.4}{\var{isprivate} was a stupid idea -- don't use it. |
| 628 | If you need to skip tests based on name, filter the list returned by |
| 629 | \code{DocTestFinder.find()} instead.} |
| 630 | |
| 631 | \versionchanged[The parameter \var{optionflags} was added]{2.3} |
| 632 | |
Tim Peters | 8278860 | 2004-09-13 15:03:17 +0000 | [diff] [blame] | 633 | \versionchanged[The parameters \var{extraglobs}, \var{raise_on_error} |
| 634 | and \var{exclude_empty} were added]{2.4} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 635 | \end{funcdesc} |
| 636 | |
| 637 | \begin{funcdesc}{testsource}{module, name} |
| 638 | Extract the doctest examples from a docstring. |
| 639 | |
| 640 | Provide the \var{module} (or dotted name of the module) containing the |
| 641 | tests to be extracted and the \var{name} (within the module) of the object |
| 642 | with the docstring containing the tests to be extracted. |
| 643 | |
| 644 | The doctest examples are returned as a string containing Python |
| 645 | code. The expected output blocks in the examples are converted |
| 646 | to Python comments. |
| 647 | \versionadded{2.3} |
| 648 | \end{funcdesc} |
| 649 | |
| 650 | \begin{funcdesc}{DocTestSuite}{\optional{module}} |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 651 | Convert doctest tests for a module to a |
| 652 | \class{\refmodule{unittest}.TestSuite}. |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 653 | |
| 654 | The returned \class{TestSuite} is to be run by the unittest framework |
| 655 | and runs each doctest in the module. If any of the doctests fail, |
| 656 | then the synthesized unit test fails, and a \exception{DocTestTestFailure} |
| 657 | exception is raised showing the name of the file containing the test and a |
| 658 | (sometimes approximate) line number. |
| 659 | |
| 660 | The optional \var{module} argument provides the module to be tested. It |
| 661 | can be a module object or a (possibly dotted) module name. If not |
Fred Drake | 8836e56 | 2003-07-17 15:22:47 +0000 | [diff] [blame] | 662 | specified, the module calling this function is used. |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 663 | |
| 664 | Example using one of the many ways that the \refmodule{unittest} module |
| 665 | can use a \class{TestSuite}: |
| 666 | |
| 667 | \begin{verbatim} |
| 668 | import unittest |
| 669 | import doctest |
| 670 | import my_module_with_doctests |
| 671 | |
| 672 | suite = doctest.DocTestSuite(my_module_with_doctests) |
| 673 | runner = unittest.TextTestRunner() |
| 674 | runner.run(suite) |
| 675 | \end{verbatim} |
| 676 | |
| 677 | \versionadded{2.3} |
Fred Drake | 8836e56 | 2003-07-17 15:22:47 +0000 | [diff] [blame] | 678 | \warning{This function does not currently search \code{M.__test__} |
Raymond Hettinger | 943277e | 2003-07-17 14:47:12 +0000 | [diff] [blame] | 679 | and its search technique does not exactly match \function{testmod()} in |
| 680 | every detail. Future versions will bring the two into convergence.} |
Raymond Hettinger | 92f21b1 | 2003-07-11 22:32:18 +0000 | [diff] [blame] | 681 | \end{funcdesc} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 682 | |
| 683 | |
| 684 | \subsection{How are Docstring Examples Recognized?} |
| 685 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 686 | In most cases a copy-and-paste of an interactive console session works |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 687 | fine, but doctest isn't trying to do an exact emulation of any specific |
| 688 | Python shell. All hard tab characters are expanded to spaces, using |
| 689 | 8-column tab stops. If you don't believe tabs should mean that, too |
| 690 | bad: don't use hard tabs, or write your own \class{DocTestParser} |
| 691 | class. |
| 692 | |
| 693 | \versionchanged[Expanding tabs to spaces is new; previous versions |
| 694 | tried to preserve hard tabs, with confusing results]{2.4} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 695 | |
| 696 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 697 | >>> # comments are ignored |
| 698 | >>> x = 12 |
| 699 | >>> x |
| 700 | 12 |
| 701 | >>> if x == 13: |
| 702 | ... print "yes" |
| 703 | ... else: |
| 704 | ... print "no" |
| 705 | ... print "NO" |
| 706 | ... print "NO!!!" |
| 707 | ... |
| 708 | no |
| 709 | NO |
| 710 | NO!!! |
| 711 | >>> |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 712 | \end{verbatim} |
| 713 | |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 714 | Any expected output must immediately follow the final |
| 715 | \code{'>\code{>}>~'} or \code{'...~'} line containing the code, and |
| 716 | the expected output (if any) extends to the next \code{'>\code{>}>~'} |
| 717 | or all-whitespace line. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 718 | |
| 719 | The fine print: |
| 720 | |
| 721 | \begin{itemize} |
| 722 | |
| 723 | \item Expected output cannot contain an all-whitespace line, since such a |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 724 | line is taken to signal the end of expected output. If expected |
| 725 | output does contain a blank line, put \code{<BLANKLINE>} in your |
| 726 | doctest example each place a blank line is expected. |
| 727 | \versionchanged[\code{<BLANKLINE>} was added; there was no way to |
| 728 | use expected output containing empty lines in |
| 729 | previous versions]{2.4} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 730 | |
| 731 | \item Output to stdout is captured, but not output to stderr (exception |
| 732 | tracebacks are captured via a different means). |
| 733 | |
Martin v. Löwis | 92816de | 2004-05-31 19:01:00 +0000 | [diff] [blame] | 734 | \item If you continue a line via backslashing in an interactive session, |
| 735 | or for any other reason use a backslash, you should use a raw |
| 736 | docstring, which will preserve your backslahses exactly as you type |
| 737 | them: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 738 | |
| 739 | \begin{verbatim} |
Tim Peters | 336689b | 2004-07-23 02:48:24 +0000 | [diff] [blame] | 740 | >>> def f(x): |
Martin v. Löwis | 92816de | 2004-05-31 19:01:00 +0000 | [diff] [blame] | 741 | ... r'''Backslashes in a raw docstring: m\n''' |
| 742 | >>> print f.__doc__ |
| 743 | Backslashes in a raw docstring: m\n |
| 744 | \end{verbatim} |
Tim Peters | 336689b | 2004-07-23 02:48:24 +0000 | [diff] [blame] | 745 | |
Martin v. Löwis | 92816de | 2004-05-31 19:01:00 +0000 | [diff] [blame] | 746 | Otherwise, the backslash will be interpreted as part of the string. |
Edward Loper | 19b1958 | 2004-08-25 23:07:03 +0000 | [diff] [blame] | 747 | E.g., the "{\textbackslash}" above would be interpreted as a newline |
Martin v. Löwis | 92816de | 2004-05-31 19:01:00 +0000 | [diff] [blame] | 748 | character. Alternatively, you can double each backslash in the |
| 749 | doctest version (and not use a raw string): |
| 750 | |
| 751 | \begin{verbatim} |
Tim Peters | 336689b | 2004-07-23 02:48:24 +0000 | [diff] [blame] | 752 | >>> def f(x): |
Martin v. Löwis | 92816de | 2004-05-31 19:01:00 +0000 | [diff] [blame] | 753 | ... '''Backslashes in a raw docstring: m\\n''' |
| 754 | >>> print f.__doc__ |
| 755 | Backslashes in a raw docstring: m\n |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 756 | \end{verbatim} |
| 757 | |
Tim Peters | f0768c8 | 2001-02-20 10:57:30 +0000 | [diff] [blame] | 758 | \item The starting column doesn't matter: |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 759 | |
| 760 | \begin{verbatim} |
Tim Peters | c4089d8 | 2001-02-17 18:03:25 +0000 | [diff] [blame] | 761 | >>> assert "Easy!" |
| 762 | >>> import math |
| 763 | >>> math.floor(1.9) |
| 764 | 1.0 |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 765 | \end{verbatim} |
| 766 | |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 767 | and as many leading whitespace characters are stripped from the |
| 768 | expected output as appeared in the initial \code{'>\code{>}>~'} line |
Tim Peters | 83e259a | 2004-08-13 21:55:21 +0000 | [diff] [blame] | 769 | that started the example. |
Fred Drake | 7eb1463 | 2001-02-17 17:32:41 +0000 | [diff] [blame] | 770 | \end{itemize} |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 771 | |
| 772 | \subsection{Warnings} |
| 773 | |
| 774 | \begin{enumerate} |
| 775 | |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 776 | \item \module{doctest} is serious about requiring exact matches in expected |
| 777 | output. If even a single character doesn't match, the test fails. This |
| 778 | will probably surprise you a few times, as you learn exactly what Python |
| 779 | does and doesn't guarantee about output. For example, when printing a |
| 780 | dict, Python doesn't guarantee that the key-value pairs will be printed |
| 781 | in any particular order, so a test like |
| 782 | |
| 783 | % Hey! What happened to Monty Python examples? |
Tim Peters | f0768c8 | 2001-02-20 10:57:30 +0000 | [diff] [blame] | 784 | % Tim: ask Guido -- it's his example! |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 785 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 786 | >>> foo() |
| 787 | {"Hermione": "hippogryph", "Harry": "broomstick"} |
| 788 | >>> |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 789 | \end{verbatim} |
| 790 | |
| 791 | is vulnerable! One workaround is to do |
| 792 | |
| 793 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 794 | >>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"} |
Martin v. Löwis | ccabed3 | 2003-11-27 19:48:03 +0000 | [diff] [blame] | 795 | True |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 796 | >>> |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 797 | \end{verbatim} |
| 798 | |
| 799 | instead. Another is to do |
| 800 | |
| 801 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 802 | >>> d = foo().items() |
| 803 | >>> d.sort() |
| 804 | >>> d |
| 805 | [('Harry', 'broomstick'), ('Hermione', 'hippogryph')] |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 806 | \end{verbatim} |
| 807 | |
| 808 | There are others, but you get the idea. |
| 809 | |
| 810 | Another bad idea is to print things that embed an object address, like |
| 811 | |
| 812 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 813 | >>> id(1.0) # certain to fail some of the time |
| 814 | 7948648 |
| 815 | >>> |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 816 | \end{verbatim} |
| 817 | |
| 818 | Floating-point numbers are also subject to small output variations across |
| 819 | platforms, because Python defers to the platform C library for float |
| 820 | formatting, and C libraries vary widely in quality here. |
| 821 | |
| 822 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 823 | >>> 1./7 # risky |
| 824 | 0.14285714285714285 |
| 825 | >>> print 1./7 # safer |
| 826 | 0.142857142857 |
| 827 | >>> print round(1./7, 6) # much safer |
| 828 | 0.142857 |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 829 | \end{verbatim} |
| 830 | |
| 831 | Numbers of the form \code{I/2.**J} are safe across all platforms, and I |
| 832 | often contrive doctest examples to produce numbers of that form: |
| 833 | |
| 834 | \begin{verbatim} |
Fred Drake | 19f3c52 | 2001-02-22 23:15:05 +0000 | [diff] [blame] | 835 | >>> 3./4 # utterly safe |
| 836 | 0.75 |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 837 | \end{verbatim} |
| 838 | |
| 839 | Simple fractions are also easier for people to understand, and that makes |
| 840 | for better documentation. |
| 841 | |
Skip Montanaro | 1dc98c4 | 2001-06-08 14:40:28 +0000 | [diff] [blame] | 842 | \item Be careful if you have code that must only execute once. |
| 843 | |
| 844 | If you have module-level code that must only execute once, a more foolproof |
Fred Drake | c115835 | 2001-06-11 14:55:01 +0000 | [diff] [blame] | 845 | definition of \function{_test()} is |
Skip Montanaro | 1dc98c4 | 2001-06-08 14:40:28 +0000 | [diff] [blame] | 846 | |
| 847 | \begin{verbatim} |
| 848 | def _test(): |
| 849 | import doctest, sys |
Martin v. Löwis | 4581cfa | 2002-11-22 08:23:09 +0000 | [diff] [blame] | 850 | doctest.testmod() |
Skip Montanaro | 1dc98c4 | 2001-06-08 14:40:28 +0000 | [diff] [blame] | 851 | \end{verbatim} |
Tim Peters | 6ebe61f | 2003-06-27 20:48:05 +0000 | [diff] [blame] | 852 | |
| 853 | \item WYSIWYG isn't always the case, starting in Python 2.3. The |
Fred Drake | 5d2f515 | 2003-06-28 03:09:06 +0000 | [diff] [blame] | 854 | string form of boolean results changed from \code{'0'} and |
| 855 | \code{'1'} to \code{'False'} and \code{'True'} in Python 2.3. |
Tim Peters | 6ebe61f | 2003-06-27 20:48:05 +0000 | [diff] [blame] | 856 | This makes it clumsy to write a doctest showing boolean results that |
| 857 | passes under multiple versions of Python. In Python 2.3, by default, |
| 858 | and as a special case, if an expected output block consists solely |
Fred Drake | 5d2f515 | 2003-06-28 03:09:06 +0000 | [diff] [blame] | 859 | of \code{'0'} and the actual output block consists solely of |
| 860 | \code{'False'}, that's accepted as an exact match, and similarly for |
| 861 | \code{'1'} versus \code{'True'}. This behavior can be turned off by |
Tim Peters | 6ebe61f | 2003-06-27 20:48:05 +0000 | [diff] [blame] | 862 | passing the new (in 2.3) module constant |
| 863 | \constant{DONT_ACCEPT_TRUE_FOR_1} as the value of \function{testmod()}'s |
| 864 | new (in 2.3) optional \var{optionflags} argument. Some years after |
| 865 | the integer spellings of booleans are history, this hack will |
| 866 | probably be removed again. |
| 867 | |
Fred Drake | c115835 | 2001-06-11 14:55:01 +0000 | [diff] [blame] | 868 | \end{enumerate} |
| 869 | |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 870 | |
| 871 | \subsection{Soapbox} |
| 872 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 873 | The first word in ``doctest'' is ``doc,'' and that's why the author |
| 874 | wrote \refmodule{doctest}: to keep documentation up to date. It so |
| 875 | happens that \refmodule{doctest} makes a pleasant unit testing |
| 876 | environment, but that's not its primary purpose. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 877 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 878 | Choose docstring examples with care. There's an art to this that |
| 879 | needs to be learned---it may not be natural at first. Examples should |
| 880 | add genuine value to the documentation. A good example can often be |
| 881 | worth many words. If possible, show just a few normal cases, show |
| 882 | endcases, show interesting subtle cases, and show an example of each |
| 883 | kind of exception that can be raised. You're probably testing for |
| 884 | endcases and subtle cases anyway in an interactive shell: |
| 885 | \refmodule{doctest} wants to make it as easy as possible to capture |
| 886 | those sessions, and will verify they continue to work as designed |
| 887 | forever after. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 888 | |
Fred Drake | 7a6b4f0 | 2003-07-17 16:00:01 +0000 | [diff] [blame] | 889 | If done with care, the examples will be invaluable for your users, and |
| 890 | will pay back the time it takes to collect them many times over as the |
| 891 | years go by and things change. I'm still amazed at how often one of |
| 892 | my \refmodule{doctest} examples stops working after a ``harmless'' |
| 893 | change. |
Tim Peters | 7688229 | 2001-02-17 05:58:44 +0000 | [diff] [blame] | 894 | |
| 895 | For exhaustive testing, or testing boring cases that add no value to the |
Fred Drake | 7eb1463 | 2001-02-17 17:32:41 +0000 | [diff] [blame] | 896 | docs, define a \code{__test__} dict instead. That's what it's for. |