blob: 895182137636dc6d89efeec372c837ee76b29479 [file] [log] [blame]
Tim Peters76882292001-02-17 05:58:44 +00001\section{\module{doctest} ---
2 Test docstrings represent reality}
3
4\declaremodule{standard}{doctest}
5\moduleauthor{Tim Peters}{tim_one@users.sourceforge.net}
6\sectionauthor{Tim Peters}{tim_one@users.sourceforge.net}
7\sectionauthor{Moshe Zadka}{moshez@debian.org}
8
9\modulesynopsis{A framework for verifying examples in docstrings.}
10
11The \module{doctest} module searches a module's docstrings for text that looks
12like an interactive Python session, then executes all such sessions to verify
13they still work exactly as shown. Here's a complete but small example:
14
15\begin{verbatim}
16"""
17This is module example.
18
19Example supplies one function, factorial. For example,
20
21>>> factorial(5)
22120
23"""
24
25def factorial(n):
26 """Return the factorial of n, an exact integer >= 0.
27
28 If the result is small enough to fit in an int, return an int.
29 Else return a long.
30
31 >>> [factorial(n) for n in range(6)]
32 [1, 1, 2, 6, 24, 120]
33 >>> [factorial(long(n)) for n in range(6)]
34 [1, 1, 2, 6, 24, 120]
35 >>> factorial(30)
36 265252859812191058636308480000000L
37 >>> factorial(30L)
38 265252859812191058636308480000000L
39 >>> factorial(-1)
40 Traceback (most recent call last):
41 ...
42 ValueError: n must be >= 0
43
44 Factorials of floats are OK, but the float must be an exact integer:
45 >>> factorial(30.1)
46 Traceback (most recent call last):
47 ...
48 ValueError: n must be exact integer
49 >>> factorial(30.0)
50 265252859812191058636308480000000L
51
52 It must also not be ridiculously large:
53 >>> factorial(1e100)
54 Traceback (most recent call last):
55 ...
56 OverflowError: n too large
57 """
58
59\end{verbatim}
60% allow LaTeX to break here.
61\begin{verbatim}
62
63 import math
64 if not n >= 0:
65 raise ValueError("n must be >= 0")
66 if math.floor(n) != n:
67 raise ValueError("n must be exact integer")
Raymond Hettinger92f21b12003-07-11 22:32:18 +000068 if n+1 == n: # catch a value like 1e300
Tim Peters76882292001-02-17 05:58:44 +000069 raise OverflowError("n too large")
70 result = 1
71 factor = 2
72 while factor <= n:
73 try:
74 result *= factor
75 except OverflowError:
76 result *= long(factor)
77 factor += 1
78 return result
79
80def _test():
Tim Petersc2388a22004-08-10 01:41:28 +000081 import doctest
82 return doctest.testmod()
Tim Peters76882292001-02-17 05:58:44 +000083
84if __name__ == "__main__":
85 _test()
86\end{verbatim}
87
Fred Drake7a6b4f02003-07-17 16:00:01 +000088If you run \file{example.py} directly from the command line,
89\module{doctest} works its magic:
Tim Peters76882292001-02-17 05:58:44 +000090
91\begin{verbatim}
92$ python example.py
93$
94\end{verbatim}
95
Fred Drake7a6b4f02003-07-17 16:00:01 +000096There's no output! That's normal, and it means all the examples
97worked. Pass \programopt{-v} to the script, and \module{doctest}
98prints a detailed log of what it's trying, and prints a summary at the
99end:
Tim Peters76882292001-02-17 05:58:44 +0000100
101\begin{verbatim}
102$ python example.py -v
Tim Peters76882292001-02-17 05:58:44 +0000103Trying: factorial(5)
104Expecting: 120
105ok
Tim Peters76882292001-02-17 05:58:44 +0000106Trying: [factorial(n) for n in range(6)]
107Expecting: [1, 1, 2, 6, 24, 120]
108ok
109Trying: [factorial(long(n)) for n in range(6)]
110Expecting: [1, 1, 2, 6, 24, 120]
Tim Peters41a65ea2004-08-13 03:55:05 +0000111ok
112\end{verbatim}
Tim Peters76882292001-02-17 05:58:44 +0000113
114And so on, eventually ending with:
115
116\begin{verbatim}
117Trying: factorial(1e100)
118Expecting:
Tim Petersc2388a22004-08-10 01:41:28 +0000119 Traceback (most recent call last):
120 ...
121 OverflowError: n too large
Tim Peters76882292001-02-17 05:58:44 +0000122ok
Tim Peters76882292001-02-17 05:58:44 +00001232 items passed all tests:
124 1 tests in example
125 8 tests in example.factorial
1269 tests in 2 items.
1279 passed and 0 failed.
128Test passed.
129$
130\end{verbatim}
131
Fred Drake7a6b4f02003-07-17 16:00:01 +0000132That's all you need to know to start making productive use of
Tim Peters41a65ea2004-08-13 03:55:05 +0000133\module{doctest}! Jump in. The following sections provide full
134details. Note that there are many examples of doctests in
135the standard Python test suite and libraries.
Tim Peters76882292001-02-17 05:58:44 +0000136
Tim Petersc2388a22004-08-10 01:41:28 +0000137\subsection{Simple Usage}
Tim Peters76882292001-02-17 05:58:44 +0000138
Tim Peters41a65ea2004-08-13 03:55:05 +0000139The simplest way to start using doctest (but not necessarily the way
140you'll continue to do it) is to end each module \module{M} with:
Tim Peters76882292001-02-17 05:58:44 +0000141
142\begin{verbatim}
143def _test():
Tim Petersc2388a22004-08-10 01:41:28 +0000144 import doctest
145 return doctest.testmod()
Tim Peters76882292001-02-17 05:58:44 +0000146
147if __name__ == "__main__":
148 _test()
149\end{verbatim}
150
Tim Petersc2388a22004-08-10 01:41:28 +0000151\module{doctest} then examines docstrings in the module calling
Tim Peters41a65ea2004-08-13 03:55:05 +0000152\function{testmod()}.
Martin v. Löwis4581cfa2002-11-22 08:23:09 +0000153
Tim Petersc2388a22004-08-10 01:41:28 +0000154Running the module as a script causes the examples in the docstrings
Tim Peters76882292001-02-17 05:58:44 +0000155to get executed and verified:
156
157\begin{verbatim}
158python M.py
159\end{verbatim}
160
161This won't display anything unless an example fails, in which case the
162failing example(s) and the cause(s) of the failure(s) are printed to stdout,
Tim Petersc2388a22004-08-10 01:41:28 +0000163and the final line of output is
Tim Peters26039602004-08-13 01:49:12 +0000164\samp{'***Test Failed*** \var{N} failures.'}, where \var{N} is the
Tim Petersc2388a22004-08-10 01:41:28 +0000165number of examples that failed.
Tim Peters76882292001-02-17 05:58:44 +0000166
Fred Drake7eb14632001-02-17 17:32:41 +0000167Run it with the \programopt{-v} switch instead:
Tim Peters76882292001-02-17 05:58:44 +0000168
169\begin{verbatim}
170python M.py -v
171\end{verbatim}
172
Fred Drake8836e562003-07-17 15:22:47 +0000173and a detailed report of all examples tried is printed to standard
174output, along with assorted summaries at the end.
Tim Peters76882292001-02-17 05:58:44 +0000175
Tim Petersc2388a22004-08-10 01:41:28 +0000176You can force verbose mode by passing \code{verbose=True} to
Fred Drake5d2f5152003-06-28 03:09:06 +0000177\function{testmod()}, or
Tim Petersc2388a22004-08-10 01:41:28 +0000178prohibit it by passing \code{verbose=False}. In either of those cases,
Fred Drake5d2f5152003-06-28 03:09:06 +0000179\code{sys.argv} is not examined by \function{testmod()}.
Tim Peters76882292001-02-17 05:58:44 +0000180
Fred Drake5d2f5152003-06-28 03:09:06 +0000181In any case, \function{testmod()} returns a 2-tuple of ints \code{(\var{f},
Fred Drake7eb14632001-02-17 17:32:41 +0000182\var{t})}, where \var{f} is the number of docstring examples that
183failed and \var{t} is the total number of docstring examples
184attempted.
Tim Peters76882292001-02-17 05:58:44 +0000185
186\subsection{Which Docstrings Are Examined?}
187
Tim Peters8a3b69c2004-08-12 22:31:25 +0000188The module docstring, and all function, class and method docstrings are
189searched. Objects imported into the module are not searched.
Tim Peters76882292001-02-17 05:58:44 +0000190
Fred Drake7eb14632001-02-17 17:32:41 +0000191In addition, if \code{M.__test__} exists and "is true", it must be a
192dict, and each entry maps a (string) name to a function object, class
193object, or string. Function and class object docstrings found from
Tim Peters8a3b69c2004-08-12 22:31:25 +0000194\code{M.__test__} are searched, and strings are treated as if they
195were docstrings. In output, a key \code{K} in \code{M.__test__} appears
196with name
Tim Peters76882292001-02-17 05:58:44 +0000197
198\begin{verbatim}
Fred Drake8836e562003-07-17 15:22:47 +0000199<name of M>.__test__.K
Tim Peters76882292001-02-17 05:58:44 +0000200\end{verbatim}
201
202Any classes found are recursively searched similarly, to test docstrings in
Tim Peters8a3b69c2004-08-12 22:31:25 +0000203their contained methods and nested classes.
204
205\versionchanged[A "private name" concept is deprecated and no longer
Tim Peters26039602004-08-13 01:49:12 +0000206 documented]{2.4}
Tim Peters8a3b69c2004-08-12 22:31:25 +0000207
Tim Peters76882292001-02-17 05:58:44 +0000208
209\subsection{What's the Execution Context?}
210
Tim Peters41a65ea2004-08-13 03:55:05 +0000211By default, each time \function{testmod()} finds a docstring to test, it
212uses a \emph{shallow copy} of \module{M}'s globals, so that running tests
Tim Peters76882292001-02-17 05:58:44 +0000213doesn't change the module's real globals, and so that one test in
214\module{M} can't leave behind crumbs that accidentally allow another test
215to work. This means examples can freely use any names defined at top-level
Tim Peters0481d242001-10-02 21:01:22 +0000216in \module{M}, and names defined earlier in the docstring being run.
Tim Peters41a65ea2004-08-13 03:55:05 +0000217Examples cannot see names defined in other docstrings.
Tim Peters76882292001-02-17 05:58:44 +0000218
219You can force use of your own dict as the execution context by passing
Tim Peters41a65ea2004-08-13 03:55:05 +0000220\code{globs=your_dict} to \function{testmod()} instead.
Tim Peters76882292001-02-17 05:58:44 +0000221
222\subsection{What About Exceptions?}
223
Tim Peters41a65ea2004-08-13 03:55:05 +0000224No problem: just paste in the expected traceback. Since
225tracebacks contain details that are likely to change
226rapidly (for example, exact file paths and line numbers), this is one
227case where doctest works hard to be flexible in what it accepts.
228This makes the full story involved, but you really don't have
229to remember much. Simple example:
Tim Peters76882292001-02-17 05:58:44 +0000230
231\begin{verbatim}
Fred Drake19f3c522001-02-22 23:15:05 +0000232>>> [1, 2, 3].remove(42)
233Traceback (most recent call last):
234 File "<stdin>", line 1, in ?
235ValueError: list.remove(x): x not in list
Tim Peters76882292001-02-17 05:58:44 +0000236\end{verbatim}
237
Edward Loper19b19582004-08-25 23:07:03 +0000238That doctest succeeds if \exception{ValueError} is raised, with the
239\samp{list.remove(x): x not in list} detail as shown.\footnote{The
240 doctest also succeeds if it prints the exact text of the traceback
241 message; otherwise, it fails.}
Tim Peters41a65ea2004-08-13 03:55:05 +0000242
Edward Loper19b19582004-08-25 23:07:03 +0000243The expected output for an exception must start with a traceback
244header, which may be either of the following two lines, indented the
245same as the first line of the example:
Tim Peters41a65ea2004-08-13 03:55:05 +0000246
247\begin{verbatim}
248Traceback (most recent call last):
249Traceback (innermost last):
250\end{verbatim}
251
Edward Loper19b19582004-08-25 23:07:03 +0000252The traceback header is followed by an optional traceback stack, whose
253contents are ignored by doctest. Each line of the traceback stack
254must be indented further than the first line of the example, \emph{or}
255start with a non-alphanumeric character. Typically, the traceback
256stack is either omitted or copied verbatim from an interactive
257session.
258
259The traceback stack is followed by the most interesting part: the
260line(s) containing the exception type and detail. This is usually the
261last line of a traceback, but can extend across multiple lines if the
262exception has a multi-line detail, as illustrated in the following
263example:
Tim Peters41a65ea2004-08-13 03:55:05 +0000264
265\begin{verbatim}
Edward Loper19b19582004-08-25 23:07:03 +0000266>>> raise ValueError('multi\n line\ndetail')
Tim Peters41a65ea2004-08-13 03:55:05 +0000267Traceback (most recent call last):
Edward Loper19b19582004-08-25 23:07:03 +0000268 File "<stdin>", line 1, in ?
269ValueError: multi
270 line
271detail
Tim Peters41a65ea2004-08-13 03:55:05 +0000272\end{verbatim}
273
Edward Loper19b19582004-08-25 23:07:03 +0000274The last three (starting with \exception{ValueError}) lines are
275compared against the exception's type and detail, and the rest are
276ignored.
Tim Peters41a65ea2004-08-13 03:55:05 +0000277
Edward Loper19b19582004-08-25 23:07:03 +0000278Best practice is to omit the traceback stack, unless it adds
Tim Peters41a65ea2004-08-13 03:55:05 +0000279significant documentation value to the example. So the example above
280is probably better as:
281
282\begin{verbatim}
Edward Loper19b19582004-08-25 23:07:03 +0000283>>> raise ValueError('multi\n line\ndetail')
Tim Peters41a65ea2004-08-13 03:55:05 +0000284Traceback (most recent call last):
Edward Loper19b19582004-08-25 23:07:03 +0000285 ...
286ValueError: multi
287 line
288detail
Tim Peters41a65ea2004-08-13 03:55:05 +0000289\end{verbatim}
290
291Note the tracebacks are treated very specially. In particular, in the
292rewritten example, the use of \samp{...} is independent of doctest's
293\constant{ELLIPSIS} option. The ellipsis in that example could
294be left out, or could just as well be three (or three hundred) commas.
295
Tim Peters026f8dc2004-08-19 16:38:58 +0000296\subsection{Option Flags and Directives\label{doctest-options}}
Tim Peters8a3b69c2004-08-12 22:31:25 +0000297
Tim Peters83e259a2004-08-13 21:55:21 +0000298A number of option flags control various aspects of doctest's comparison
Tim Peters026f8dc2004-08-19 16:38:58 +0000299behavior. Symbolic names for the flags are supplied as module constants,
Tim Peters83e259a2004-08-13 21:55:21 +0000300which can be or'ed together and passed to various functions. The names
Tim Peters026f8dc2004-08-19 16:38:58 +0000301can also be used in doctest directives (see below).
Tim Peters8a3b69c2004-08-12 22:31:25 +0000302
303\begin{datadesc}{DONT_ACCEPT_TRUE_FOR_1}
304 By default, if an expected output block contains just \code{1},
305 an actual output block containing just \code{1} or just
306 \code{True} is considered to be a match, and similarly for \code{0}
307 versus \code{False}. When \constant{DONT_ACCEPT_TRUE_FOR_1} is
308 specified, neither substitution is allowed. The default behavior
309 caters to that Python changed the return type of many functions
310 from integer to boolean; doctests expecting "little integer"
311 output still work in these cases. This option will probably go
312 away, but not for several years.
313\end{datadesc}
314
315\begin{datadesc}{DONT_ACCEPT_BLANKLINE}
316 By default, if an expected output block contains a line
317 containing only the string \code{<BLANKLINE>}, then that line
318 will match a blank line in the actual output. Because a
319 genuinely blank line delimits the expected output, this is
320 the only way to communicate that a blank line is expected. When
321 \constant{DONT_ACCEPT_BLANKLINE} is specified, this substitution
322 is not allowed.
323\end{datadesc}
324
325\begin{datadesc}{NORMALIZE_WHITESPACE}
326 When specified, all sequences of whitespace (blanks and newlines) are
327 treated as equal. Any sequence of whitespace within the expected
328 output will match any sequence of whitespace within the actual output.
329 By default, whitespace must match exactly.
330 \constant{NORMALIZE_WHITESPACE} is especially useful when a line
331 of expected output is very long, and you want to wrap it across
332 multiple lines in your source.
333\end{datadesc}
334
335\begin{datadesc}{ELLIPSIS}
336 When specified, an ellipsis marker (\code{...}) in the expected output
337 can match any substring in the actual output. This includes
Tim Peters026f8dc2004-08-19 16:38:58 +0000338 substrings that span line boundaries, and empty substrings, so it's
339 best to keep usage of this simple. Complicated uses can lead to the
340 same kinds of "oops, it matched too much!" surprises that \regexp{.*}
341 is prone to in regular expressions.
Tim Peters8a3b69c2004-08-12 22:31:25 +0000342\end{datadesc}
343
344\begin{datadesc}{UNIFIED_DIFF}
345 When specified, failures that involve multi-line expected and
346 actual outputs are displayed using a unified diff.
347\end{datadesc}
348
349\begin{datadesc}{CONTEXT_DIFF}
350 When specified, failures that involve multi-line expected and
351 actual outputs will be displayed using a context diff.
352\end{datadesc}
353
Tim Petersc6cbab02004-08-22 19:43:28 +0000354\begin{datadesc}{NDIFF_DIFF}
355 When specified, differences are computed by \code{difflib.Differ},
356 using the same algorithm as the popular \file{ndiff.py} utility.
357 This is the only method that marks differences within lines as
358 well as across lines. For example, if a line of expected output
359 contains digit \code{1} where actual output contains letter \code{l},
360 a line is inserted with a caret marking the mismatching column
361 positions.
362\end{datadesc}
Tim Peters8a3b69c2004-08-12 22:31:25 +0000363
Tim Peters026f8dc2004-08-19 16:38:58 +0000364A "doctest directive" is a trailing Python comment on a line of a doctest
365example:
366
367\begin{productionlist}[doctest]
368 \production{directive}
Johannes Gijsbersc8906182004-08-20 14:37:05 +0000369 {"\#" "doctest:" \token{on_or_off} \token{directive_name}}
Tim Peters026f8dc2004-08-19 16:38:58 +0000370 \production{on_or_off}
371 {"+" | "-"}
372 \production{directive_name}
373 {"DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...}
374\end{productionlist}
375
376Whitespace is not allowed between the \code{+} or \code{-} and the
377directive name. The directive name can be any of the option names
378explained above.
379
380The doctest directives appearing in a single example modify doctest's
381behavior for that single example. Use \code{+} to enable the named
382behavior, or \code{-} to disable it.
383
384For example, this test passes:
385
386\begin{verbatim}
387>>> print range(20) #doctest: +NORMALIZE_WHITESPACE
388[0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
38910, 11, 12, 13, 14, 15, 16, 17, 18, 19]
390\end{verbatim}
391
392Without the directive it would fail, both because the actual output
393doesn't have two blanks before the single-digit list elements, and
394because the actual output is on a single line. This test also passes,
395and requires a directive to do so:
396
397\begin{verbatim}
398>>> print range(20) # doctest:+ELLIPSIS
399[0, 1, ..., 18, 19]
400\end{verbatim}
401
402Only one directive per physical line is accepted. If you want to
403use multiple directives for a single example, you can add
404\samp{...} lines to your example containing only directives:
405
406\begin{verbatim}
407>>> print range(20) #doctest: +ELLIPSIS
408... #doctest: +NORMALIZE_WHITESPACE
409[0, 1, ..., 18, 19]
410\end{verbatim}
411
412Note that since all options are disabled by default, and directives apply
413only to the example they appear in, enabling options (via \code{+} in a
414directive) is usually the only meaningful choice. However, option flags
415can also be passed to functions that run doctests, establishing different
416defaults. In such cases, disabling an option via \code{-} in a directive
417can be useful.
418
Tim Peters8a3b69c2004-08-12 22:31:25 +0000419\versionchanged[Constants \constant{DONT_ACCEPT_BLANKLINE},
420 \constant{NORMALIZE_WHITESPACE}, \constant{ELLIPSIS},
Tim Petersc6cbab02004-08-22 19:43:28 +0000421 \constant{UNIFIED_DIFF}, \constant{CONTEXT_DIFF}, and
422 \constant{NDIFF_DIFF}
Tim Peters026f8dc2004-08-19 16:38:58 +0000423 were added; by default \code{<BLANKLINE>} in expected output
424 matches an empty line in actual output; and doctest directives
425 were added]{2.4}
426
Tim Peters8a3b69c2004-08-12 22:31:25 +0000427
Tim Peters76882292001-02-17 05:58:44 +0000428\subsection{Advanced Usage}
429
Raymond Hettinger92f21b12003-07-11 22:32:18 +0000430Several module level functions are available for controlling how doctests
431are run.
Tim Peters76882292001-02-17 05:58:44 +0000432
Raymond Hettinger92f21b12003-07-11 22:32:18 +0000433\begin{funcdesc}{debug}{module, name}
434 Debug a single docstring containing doctests.
435
436 Provide the \var{module} (or dotted name of the module) containing the
437 docstring to be debugged and the \var{name} (within the module) of the
438 object with the docstring to be debugged.
439
440 The doctest examples are extracted (see function \function{testsource()}),
441 and written to a temporary file. The Python debugger, \refmodule{pdb},
Fred Drake8836e562003-07-17 15:22:47 +0000442 is then invoked on that file.
Raymond Hettinger92f21b12003-07-11 22:32:18 +0000443 \versionadded{2.3}
444\end{funcdesc}
445
Tim Peters83e259a2004-08-13 21:55:21 +0000446\begin{funcdesc}{testmod}{\optional{m}\optional{, name}\optional{,
447 globs}\optional{, verbose}\optional{,
448 isprivate}\optional{, report}\optional{,
449 optionflags}\optional{, extraglobs}\optional{,
450 raise_on_error}}
Raymond Hettinger92f21b12003-07-11 22:32:18 +0000451
Tim Peters83e259a2004-08-13 21:55:21 +0000452 All arguments are optional, and all except for \var{m} should be
453 specified in keyword form.
454
455 Test examples in docstrings in functions and classes reachable
456 from module \var{m} (or the current module if \var{m} is not supplied
457 or is \code{None}), starting with \code{\var{m}.__doc__}.
458
459 Also test examples reachable from dict \code{\var{m}.__test__}, if it
460 exists and is not \code{None}. \code{\var{m}.__test__} maps
461 names (strings) to functions, classes and strings; function and class
462 docstrings are searched for examples; strings are searched directly,
463 as if they were docstrings.
464
465 Only docstrings attached to objects belonging to module \var{m} are
466 searched.
467
468 Return \samp{(\var{failure_count}, \var{test_count})}.
469
470 Optional argument \var{name} gives the name of the module; by default,
471 or if \code{None}, \code{\var{m}.__name__} is used.
472
473 Optional argument \var{globs} gives a dict to be used as the globals
474 when executing examples; by default, or if \code{None},
475 \code{\var{m}.__dict__} is used. A new shallow copy of this dict is
476 created for each docstring with examples, so that each docstring's
477 examples start with a clean slate.
478
479 Optional argument \var{extraglobs} gives a dict merged into the
480 globals used to execute examples. This works like
481 \method{dict.update()}: if \var{globs} and \var{extraglobs} have a
482 common key, the associated value in \var{extraglobs} appears in the
483 combined dict. By default, or if \code{None}, no extra globals are
484 used. This is an advanced feature that allows parameterization of
485 doctests. For example, a doctest can be written for a base class, using
486 a generic name for the class, then reused to test any number of
487 subclasses by passing an \var{extraglobs} dict mapping the generic
488 name to the subclass to be tested.
489
490 Optional argument \var{verbose} prints lots of stuff if true, and prints
491 only failures if false; by default, or if \code{None}, it's true
492 if and only if \code{'-v'} is in \code{sys.argv}.
493
494 Optional argument \var{report} prints a summary at the end when true,
495 else prints nothing at the end. In verbose mode, the summary is
496 detailed, else the summary is very brief (in fact, empty if all tests
497 passed).
498
499 Optional argument \var{optionflags} or's together option flags. See
500 see section \ref{doctest-options}.
501
502 Optional argument \var{raise_on_error} defaults to false. If true,
503 an exception is raised upon the first failure or unexpected exception
504 in an example. This allows failures to be post-mortem debugged.
505 Default behavior is to continue running examples.
506
507 Optional argument \var{isprivate} specifies a function used to
508 determine whether a name is private. The default function treats
509 all names as public. \var{isprivate} can be set to
510 \code{doctest.is_private} to skip over names that are
511 private according to Python's underscore naming convention.
512 \deprecated{2.4}{\var{isprivate} was a stupid idea -- don't use it.
513 If you need to skip tests based on name, filter the list returned by
514 \code{DocTestFinder.find()} instead.}
515
516 \versionchanged[The parameter \var{optionflags} was added]{2.3}
517
518 \versionchanged[The parameters \var{extraglobs} and \var{raise_on_error}
519 were added]{2.4}
Raymond Hettinger92f21b12003-07-11 22:32:18 +0000520\end{funcdesc}
521
522\begin{funcdesc}{testsource}{module, name}
523 Extract the doctest examples from a docstring.
524
525 Provide the \var{module} (or dotted name of the module) containing the
526 tests to be extracted and the \var{name} (within the module) of the object
527 with the docstring containing the tests to be extracted.
528
529 The doctest examples are returned as a string containing Python
530 code. The expected output blocks in the examples are converted
531 to Python comments.
532 \versionadded{2.3}
533\end{funcdesc}
534
535\begin{funcdesc}{DocTestSuite}{\optional{module}}
Fred Drake7a6b4f02003-07-17 16:00:01 +0000536 Convert doctest tests for a module to a
537 \class{\refmodule{unittest}.TestSuite}.
Raymond Hettinger92f21b12003-07-11 22:32:18 +0000538
539 The returned \class{TestSuite} is to be run by the unittest framework
540 and runs each doctest in the module. If any of the doctests fail,
541 then the synthesized unit test fails, and a \exception{DocTestTestFailure}
542 exception is raised showing the name of the file containing the test and a
543 (sometimes approximate) line number.
544
545 The optional \var{module} argument provides the module to be tested. It
546 can be a module object or a (possibly dotted) module name. If not
Fred Drake8836e562003-07-17 15:22:47 +0000547 specified, the module calling this function is used.
Raymond Hettinger92f21b12003-07-11 22:32:18 +0000548
549 Example using one of the many ways that the \refmodule{unittest} module
550 can use a \class{TestSuite}:
551
552 \begin{verbatim}
553 import unittest
554 import doctest
555 import my_module_with_doctests
556
557 suite = doctest.DocTestSuite(my_module_with_doctests)
558 runner = unittest.TextTestRunner()
559 runner.run(suite)
560 \end{verbatim}
561
562 \versionadded{2.3}
Fred Drake8836e562003-07-17 15:22:47 +0000563 \warning{This function does not currently search \code{M.__test__}
Raymond Hettinger943277e2003-07-17 14:47:12 +0000564 and its search technique does not exactly match \function{testmod()} in
565 every detail. Future versions will bring the two into convergence.}
Raymond Hettinger92f21b12003-07-11 22:32:18 +0000566\end{funcdesc}
Tim Peters76882292001-02-17 05:58:44 +0000567
568
569\subsection{How are Docstring Examples Recognized?}
570
Fred Drake7a6b4f02003-07-17 16:00:01 +0000571In most cases a copy-and-paste of an interactive console session works
Tim Peters83e259a2004-08-13 21:55:21 +0000572fine, but doctest isn't trying to do an exact emulation of any specific
573Python shell. All hard tab characters are expanded to spaces, using
5748-column tab stops. If you don't believe tabs should mean that, too
575bad: don't use hard tabs, or write your own \class{DocTestParser}
576class.
577
578\versionchanged[Expanding tabs to spaces is new; previous versions
579 tried to preserve hard tabs, with confusing results]{2.4}
Tim Peters76882292001-02-17 05:58:44 +0000580
581\begin{verbatim}
Fred Drake19f3c522001-02-22 23:15:05 +0000582>>> # comments are ignored
583>>> x = 12
584>>> x
58512
586>>> if x == 13:
587... print "yes"
588... else:
589... print "no"
590... print "NO"
591... print "NO!!!"
592...
593no
594NO
595NO!!!
596>>>
Tim Peters76882292001-02-17 05:58:44 +0000597\end{verbatim}
598
Fred Drake19f3c522001-02-22 23:15:05 +0000599Any expected output must immediately follow the final
600\code{'>\code{>}>~'} or \code{'...~'} line containing the code, and
601the expected output (if any) extends to the next \code{'>\code{>}>~'}
602or all-whitespace line.
Tim Peters76882292001-02-17 05:58:44 +0000603
604The fine print:
605
606\begin{itemize}
607
608\item Expected output cannot contain an all-whitespace line, since such a
Tim Peters83e259a2004-08-13 21:55:21 +0000609 line is taken to signal the end of expected output. If expected
610 output does contain a blank line, put \code{<BLANKLINE>} in your
611 doctest example each place a blank line is expected.
612 \versionchanged[\code{<BLANKLINE>} was added; there was no way to
613 use expected output containing empty lines in
614 previous versions]{2.4}
Tim Peters76882292001-02-17 05:58:44 +0000615
616\item Output to stdout is captured, but not output to stderr (exception
617 tracebacks are captured via a different means).
618
Martin v. Löwis92816de2004-05-31 19:01:00 +0000619\item If you continue a line via backslashing in an interactive session,
620 or for any other reason use a backslash, you should use a raw
621 docstring, which will preserve your backslahses exactly as you type
622 them:
Tim Peters76882292001-02-17 05:58:44 +0000623
624\begin{verbatim}
Tim Peters336689b2004-07-23 02:48:24 +0000625>>> def f(x):
Martin v. Löwis92816de2004-05-31 19:01:00 +0000626... r'''Backslashes in a raw docstring: m\n'''
627>>> print f.__doc__
628Backslashes in a raw docstring: m\n
629\end{verbatim}
Tim Peters336689b2004-07-23 02:48:24 +0000630
Martin v. Löwis92816de2004-05-31 19:01:00 +0000631 Otherwise, the backslash will be interpreted as part of the string.
Edward Loper19b19582004-08-25 23:07:03 +0000632 E.g., the "{\textbackslash}" above would be interpreted as a newline
Martin v. Löwis92816de2004-05-31 19:01:00 +0000633 character. Alternatively, you can double each backslash in the
634 doctest version (and not use a raw string):
635
636\begin{verbatim}
Tim Peters336689b2004-07-23 02:48:24 +0000637>>> def f(x):
Martin v. Löwis92816de2004-05-31 19:01:00 +0000638... '''Backslashes in a raw docstring: m\\n'''
639>>> print f.__doc__
640Backslashes in a raw docstring: m\n
Tim Peters76882292001-02-17 05:58:44 +0000641\end{verbatim}
642
Tim Petersf0768c82001-02-20 10:57:30 +0000643\item The starting column doesn't matter:
Tim Peters76882292001-02-17 05:58:44 +0000644
645\begin{verbatim}
Tim Petersc4089d82001-02-17 18:03:25 +0000646 >>> assert "Easy!"
647 >>> import math
648 >>> math.floor(1.9)
649 1.0
Tim Peters76882292001-02-17 05:58:44 +0000650\end{verbatim}
651
Fred Drake19f3c522001-02-22 23:15:05 +0000652and as many leading whitespace characters are stripped from the
653expected output as appeared in the initial \code{'>\code{>}>~'} line
Tim Peters83e259a2004-08-13 21:55:21 +0000654that started the example.
Fred Drake7eb14632001-02-17 17:32:41 +0000655\end{itemize}
Tim Peters76882292001-02-17 05:58:44 +0000656
657\subsection{Warnings}
658
659\begin{enumerate}
660
Tim Peters76882292001-02-17 05:58:44 +0000661\item \module{doctest} is serious about requiring exact matches in expected
662 output. If even a single character doesn't match, the test fails. This
663 will probably surprise you a few times, as you learn exactly what Python
664 does and doesn't guarantee about output. For example, when printing a
665 dict, Python doesn't guarantee that the key-value pairs will be printed
666 in any particular order, so a test like
667
668% Hey! What happened to Monty Python examples?
Tim Petersf0768c82001-02-20 10:57:30 +0000669% Tim: ask Guido -- it's his example!
Tim Peters76882292001-02-17 05:58:44 +0000670\begin{verbatim}
Fred Drake19f3c522001-02-22 23:15:05 +0000671>>> foo()
672{"Hermione": "hippogryph", "Harry": "broomstick"}
673>>>
Tim Peters76882292001-02-17 05:58:44 +0000674\end{verbatim}
675
676is vulnerable! One workaround is to do
677
678\begin{verbatim}
Fred Drake19f3c522001-02-22 23:15:05 +0000679>>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"}
Martin v. Löwisccabed32003-11-27 19:48:03 +0000680True
Fred Drake19f3c522001-02-22 23:15:05 +0000681>>>
Tim Peters76882292001-02-17 05:58:44 +0000682\end{verbatim}
683
684instead. Another is to do
685
686\begin{verbatim}
Fred Drake19f3c522001-02-22 23:15:05 +0000687>>> d = foo().items()
688>>> d.sort()
689>>> d
690[('Harry', 'broomstick'), ('Hermione', 'hippogryph')]
Tim Peters76882292001-02-17 05:58:44 +0000691\end{verbatim}
692
693There are others, but you get the idea.
694
695Another bad idea is to print things that embed an object address, like
696
697\begin{verbatim}
Fred Drake19f3c522001-02-22 23:15:05 +0000698>>> id(1.0) # certain to fail some of the time
6997948648
700>>>
Tim Peters76882292001-02-17 05:58:44 +0000701\end{verbatim}
702
703Floating-point numbers are also subject to small output variations across
704platforms, because Python defers to the platform C library for float
705formatting, and C libraries vary widely in quality here.
706
707\begin{verbatim}
Fred Drake19f3c522001-02-22 23:15:05 +0000708>>> 1./7 # risky
7090.14285714285714285
710>>> print 1./7 # safer
7110.142857142857
712>>> print round(1./7, 6) # much safer
7130.142857
Tim Peters76882292001-02-17 05:58:44 +0000714\end{verbatim}
715
716Numbers of the form \code{I/2.**J} are safe across all platforms, and I
717often contrive doctest examples to produce numbers of that form:
718
719\begin{verbatim}
Fred Drake19f3c522001-02-22 23:15:05 +0000720>>> 3./4 # utterly safe
7210.75
Tim Peters76882292001-02-17 05:58:44 +0000722\end{verbatim}
723
724Simple fractions are also easier for people to understand, and that makes
725for better documentation.
726
Skip Montanaro1dc98c42001-06-08 14:40:28 +0000727\item Be careful if you have code that must only execute once.
728
729If you have module-level code that must only execute once, a more foolproof
Fred Drakec1158352001-06-11 14:55:01 +0000730definition of \function{_test()} is
Skip Montanaro1dc98c42001-06-08 14:40:28 +0000731
732\begin{verbatim}
733def _test():
734 import doctest, sys
Martin v. Löwis4581cfa2002-11-22 08:23:09 +0000735 doctest.testmod()
Skip Montanaro1dc98c42001-06-08 14:40:28 +0000736\end{verbatim}
Tim Peters6ebe61f2003-06-27 20:48:05 +0000737
738\item WYSIWYG isn't always the case, starting in Python 2.3. The
Fred Drake5d2f5152003-06-28 03:09:06 +0000739 string form of boolean results changed from \code{'0'} and
740 \code{'1'} to \code{'False'} and \code{'True'} in Python 2.3.
Tim Peters6ebe61f2003-06-27 20:48:05 +0000741 This makes it clumsy to write a doctest showing boolean results that
742 passes under multiple versions of Python. In Python 2.3, by default,
743 and as a special case, if an expected output block consists solely
Fred Drake5d2f5152003-06-28 03:09:06 +0000744 of \code{'0'} and the actual output block consists solely of
745 \code{'False'}, that's accepted as an exact match, and similarly for
746 \code{'1'} versus \code{'True'}. This behavior can be turned off by
Tim Peters6ebe61f2003-06-27 20:48:05 +0000747 passing the new (in 2.3) module constant
748 \constant{DONT_ACCEPT_TRUE_FOR_1} as the value of \function{testmod()}'s
749 new (in 2.3) optional \var{optionflags} argument. Some years after
750 the integer spellings of booleans are history, this hack will
751 probably be removed again.
752
Fred Drakec1158352001-06-11 14:55:01 +0000753\end{enumerate}
754
Tim Peters76882292001-02-17 05:58:44 +0000755
756\subsection{Soapbox}
757
Fred Drake7a6b4f02003-07-17 16:00:01 +0000758The first word in ``doctest'' is ``doc,'' and that's why the author
759wrote \refmodule{doctest}: to keep documentation up to date. It so
760happens that \refmodule{doctest} makes a pleasant unit testing
761environment, but that's not its primary purpose.
Tim Peters76882292001-02-17 05:58:44 +0000762
Fred Drake7a6b4f02003-07-17 16:00:01 +0000763Choose docstring examples with care. There's an art to this that
764needs to be learned---it may not be natural at first. Examples should
765add genuine value to the documentation. A good example can often be
766worth many words. If possible, show just a few normal cases, show
767endcases, show interesting subtle cases, and show an example of each
768kind of exception that can be raised. You're probably testing for
769endcases and subtle cases anyway in an interactive shell:
770\refmodule{doctest} wants to make it as easy as possible to capture
771those sessions, and will verify they continue to work as designed
772forever after.
Tim Peters76882292001-02-17 05:58:44 +0000773
Fred Drake7a6b4f02003-07-17 16:00:01 +0000774If done with care, the examples will be invaluable for your users, and
775will pay back the time it takes to collect them many times over as the
776years go by and things change. I'm still amazed at how often one of
777my \refmodule{doctest} examples stops working after a ``harmless''
778change.
Tim Peters76882292001-02-17 05:58:44 +0000779
780For exhaustive testing, or testing boring cases that add no value to the
Fred Drake7eb14632001-02-17 17:32:41 +0000781docs, define a \code{__test__} dict instead. That's what it's for.