blob: c01606dca6ae739533fe4472ac3c21f885d8ff47 [file] [log] [blame]
Tim Peters4fd9e2f2001-08-18 00:05:50 +00001# Module doctest.
Tim Peters8a7d2d52001-01-16 07:10:57 +00002# Released to the public domain 16-Jan-2001,
3# by Tim Peters (tim.one@home.com).
4
5# Provided as-is; use at your own risk; no warranty; no promises; enjoy!
6
7"""Module doctest -- a framework for running examples in docstrings.
8
9NORMAL USAGE
10
11In normal use, end each module M with:
12
13def _test():
14 import doctest, M # replace M with your module's name
15 return doctest.testmod(M) # ditto
16
17if __name__ == "__main__":
18 _test()
19
20Then running the module as a script will cause the examples in the
21docstrings to get executed and verified:
22
23python M.py
24
25This won't display anything unless an example fails, in which case the
26failing example(s) and the cause(s) of the failure(s) are printed to stdout
27(why not stderr? because stderr is a lame hack <0.2 wink>), and the final
28line of output is "Test failed.".
29
30Run it with the -v switch instead:
31
32python M.py -v
33
34and a detailed report of all examples tried is printed to stdout, along
35with assorted summaries at the end.
36
37You can force verbose mode by passing "verbose=1" to testmod, or prohibit
38it by passing "verbose=0". In either of those cases, sys.argv is not
39examined by testmod.
40
41In any case, testmod returns a 2-tuple of ints (f, t), where f is the
42number of docstring examples that failed and t is the total number of
43docstring examples attempted.
44
45
46WHICH DOCSTRINGS ARE EXAMINED?
47
48+ M.__doc__.
49
50+ f.__doc__ for all functions f in M.__dict__.values(), except those
Tim Peters7402f792001-10-02 03:53:41 +000051 with private names and those defined in other modules.
Tim Peters8a7d2d52001-01-16 07:10:57 +000052
53+ C.__doc__ for all classes C in M.__dict__.values(), except those with
Tim Peters7402f792001-10-02 03:53:41 +000054 private names and those defined in other modules.
Tim Peters8a7d2d52001-01-16 07:10:57 +000055
56+ If M.__test__ exists and "is true", it must be a dict, and
57 each entry maps a (string) name to a function object, class object, or
58 string. Function and class object docstrings found from M.__test__
59 are searched even if the name is private, and strings are searched
60 directly as if they were docstrings. In output, a key K in M.__test__
61 appears with name
62 <name of M>.__test__.K
63
64Any classes found are recursively searched similarly, to test docstrings in
65their contained methods and nested classes. Private names reached from M's
66globals are skipped, but all names reached from M.__test__ are searched.
67
68By default, a name is considered to be private if it begins with an
69underscore (like "_my_func") but doesn't both begin and end with (at least)
70two underscores (like "__init__"). You can change the default by passing
71your own "isprivate" function to testmod.
72
73If you want to test docstrings in objects with private names too, stuff
74them into an M.__test__ dict, or see ADVANCED USAGE below (e.g., pass your
75own isprivate function to Tester's constructor, or call the rundoc method
76of a Tester instance).
77
Tim Peters8a7d2d52001-01-16 07:10:57 +000078WHAT'S THE EXECUTION CONTEXT?
79
80By default, each time testmod finds a docstring to test, it uses a *copy*
81of M's globals (so that running tests on a module doesn't change the
82module's real globals, and so that one test in M can't leave behind crumbs
83that accidentally allow another test to work). This means examples can
84freely use any names defined at top-level in M. It also means that sloppy
85imports (see above) can cause examples in external docstrings to use
86globals inappropriate for them.
87
88You can force use of your own dict as the execution context by passing
89"globs=your_dict" to testmod instead. Presumably this would be a copy of
90M.__dict__ merged with the globals from other imported modules.
91
92
93WHAT IF I WANT TO TEST A WHOLE PACKAGE?
94
95Piece o' cake, provided the modules do their testing from docstrings.
96Here's the test.py I use for the world's most elaborate Rational/
97floating-base-conversion pkg (which I'll distribute some day):
98
99from Rational import Cvt
100from Rational import Format
101from Rational import machprec
102from Rational import Rat
103from Rational import Round
104from Rational import utils
105
106modules = (Cvt,
107 Format,
108 machprec,
109 Rat,
110 Round,
111 utils)
112
113def _test():
114 import doctest
115 import sys
116 verbose = "-v" in sys.argv
117 for mod in modules:
118 doctest.testmod(mod, verbose=verbose, report=0)
119 doctest.master.summarize()
120
121if __name__ == "__main__":
122 _test()
123
124IOW, it just runs testmod on all the pkg modules. testmod remembers the
125names and outcomes (# of failures, # of tries) for each item it's seen, and
126passing "report=0" prevents it from printing a summary in verbose mode.
127Instead, the summary is delayed until all modules have been tested, and
128then "doctest.master.summarize()" forces the summary at the end.
129
130So this is very nice in practice: each module can be tested individually
131with almost no work beyond writing up docstring examples, and collections
132of modules can be tested too as a unit with no more work than the above.
133
134
135WHAT ABOUT EXCEPTIONS?
136
137No problem, as long as the only output generated by the example is the
138traceback itself. For example:
139
Tim Peters60e23f42001-02-14 00:43:21 +0000140 >>> [1, 2, 3].remove(42)
Tim Petersea4f9312001-02-13 20:54:42 +0000141 Traceback (most recent call last):
Tim Peters8a7d2d52001-01-16 07:10:57 +0000142 File "<stdin>", line 1, in ?
Tim Peters60e23f42001-02-14 00:43:21 +0000143 ValueError: list.remove(x): x not in list
Tim Peters8a7d2d52001-01-16 07:10:57 +0000144 >>>
145
146Note that only the exception type and value are compared (specifically,
147only the last line in the traceback).
148
149
150ADVANCED USAGE
151
152doctest.testmod() captures the testing policy I find most useful most
153often. You may want other policies.
154
155testmod() actually creates a local instance of class doctest.Tester, runs
156appropriate methods of that class, and merges the results into global
157Tester instance doctest.master.
158
159You can create your own instances of doctest.Tester, and so build your own
160policies, or even run methods of doctest.master directly. See
161doctest.Tester.__doc__ for details.
162
163
164SO WHAT DOES A DOCSTRING EXAMPLE LOOK LIKE ALREADY!?
165
166Oh ya. It's easy! In most cases a copy-and-paste of an interactive
167console session works fine -- just make sure the leading whitespace is
168rigidly consistent (you can mix tabs and spaces if you're too lazy to do it
169right, but doctest is not in the business of guessing what you think a tab
170means).
171
172 >>> # comments are ignored
173 >>> x = 12
174 >>> x
175 12
176 >>> if x == 13:
177 ... print "yes"
178 ... else:
179 ... print "no"
180 ... print "NO"
181 ... print "NO!!!"
182 ...
183 no
184 NO
185 NO!!!
186 >>>
187
188Any expected output must immediately follow the final ">>>" or "..." line
189containing the code, and the expected output (if any) extends to the next
190">>>" or all-whitespace line. That's it.
191
192Bummers:
193
194+ Expected output cannot contain an all-whitespace line, since such a line
195 is taken to signal the end of expected output.
196
197+ Output to stdout is captured, but not output to stderr (exception
198 tracebacks are captured via a different means).
199
200+ If you continue a line via backslashing in an interactive session, or for
201 any other reason use a backslash, you need to double the backslash in the
202 docstring version. This is simply because you're in a string, and so the
203 backslash must be escaped for it to survive intact. Like:
204
205>>> if "yes" == \\
206... "y" + \\
207... "es": # in the source code you'll see the doubled backslashes
208... print 'yes'
209yes
210
211The starting column doesn't matter:
212
213>>> assert "Easy!"
214 >>> import math
215 >>> math.floor(1.9)
216 1.0
217
218and as many leading whitespace characters are stripped from the expected
219output as appeared in the initial ">>>" line that triggered it.
220
221If you execute this very file, the examples above will be found and
222executed, leading to this output in verbose mode:
223
224Running doctest.__doc__
Tim Peters60e23f42001-02-14 00:43:21 +0000225Trying: [1, 2, 3].remove(42)
Tim Peters8a7d2d52001-01-16 07:10:57 +0000226Expecting:
Tim Petersea4f9312001-02-13 20:54:42 +0000227Traceback (most recent call last):
Tim Peters8a7d2d52001-01-16 07:10:57 +0000228 File "<stdin>", line 1, in ?
Tim Peters60e23f42001-02-14 00:43:21 +0000229ValueError: list.remove(x): x not in list
Tim Peters8a7d2d52001-01-16 07:10:57 +0000230ok
231Trying: x = 12
232Expecting: nothing
233ok
234Trying: x
235Expecting: 12
236ok
237Trying:
238if x == 13:
239 print "yes"
240else:
241 print "no"
242 print "NO"
243 print "NO!!!"
244Expecting:
245no
246NO
247NO!!!
248ok
249... and a bunch more like that, with this summary at the end:
250
2515 items had no tests:
252 doctest.Tester.__init__
253 doctest.Tester.run__test__
254 doctest.Tester.summarize
255 doctest.run_docstring_examples
256 doctest.testmod
25712 items passed all tests:
258 8 tests in doctest
259 6 tests in doctest.Tester
260 10 tests in doctest.Tester.merge
Tim Peters17111f32001-10-03 04:08:26 +0000261 14 tests in doctest.Tester.rundict
Tim Peters8a7d2d52001-01-16 07:10:57 +0000262 3 tests in doctest.Tester.rundoc
263 3 tests in doctest.Tester.runstring
264 2 tests in doctest.__test__._TestClass
265 2 tests in doctest.__test__._TestClass.__init__
266 2 tests in doctest.__test__._TestClass.get
267 1 tests in doctest.__test__._TestClass.square
268 2 tests in doctest.__test__.string
269 7 tests in doctest.is_private
Tim Peters17111f32001-10-03 04:08:26 +000027060 tests in 17 items.
27160 passed and 0 failed.
Tim Peters8a7d2d52001-01-16 07:10:57 +0000272Test passed.
273"""
274
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000275__all__ = [
276 'testmod',
277 'run_docstring_examples',
278 'is_private',
279 'Tester',
280]
Tim Peters8a7d2d52001-01-16 07:10:57 +0000281
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000282import __future__
Tim Peters8a7d2d52001-01-16 07:10:57 +0000283
Tim Peters8a7d2d52001-01-16 07:10:57 +0000284import re
285PS1 = ">>>"
286PS2 = "..."
287_isPS1 = re.compile(r"(\s*)" + re.escape(PS1)).match
288_isPS2 = re.compile(r"(\s*)" + re.escape(PS2)).match
289_isEmpty = re.compile(r"\s*$").match
290_isComment = re.compile(r"\s*#").match
291del re
292
Tim Peters7402f792001-10-02 03:53:41 +0000293from types import StringTypes as _StringTypes
294
295from inspect import isclass as _isclass
296from inspect import isfunction as _isfunction
297from inspect import ismodule as _ismodule
Tim Peters17111f32001-10-03 04:08:26 +0000298from inspect import classify_class_attrs as _classify_class_attrs
Tim Peters7402f792001-10-02 03:53:41 +0000299
Tim Peters8a7d2d52001-01-16 07:10:57 +0000300# Extract interactive examples from a string. Return a list of triples,
301# (source, outcome, lineno). "source" is the source code, and ends
302# with a newline iff the source spans more than one line. "outcome" is
303# the expected output if any, else an empty string. When not empty,
304# outcome always ends with a newline. "lineno" is the line number,
305# 0-based wrt the start of the string, of the first source line.
306
307def _extract_examples(s):
308 isPS1, isPS2 = _isPS1, _isPS2
309 isEmpty, isComment = _isEmpty, _isComment
310 examples = []
Eric S. Raymond630e69c2001-02-09 08:33:43 +0000311 lines = s.split("\n")
Tim Peters8a7d2d52001-01-16 07:10:57 +0000312 i, n = 0, len(lines)
313 while i < n:
314 line = lines[i]
315 i = i + 1
316 m = isPS1(line)
317 if m is None:
318 continue
319 j = m.end(0) # beyond the prompt
320 if isEmpty(line, j) or isComment(line, j):
321 # a bare prompt or comment -- not interesting
322 continue
323 lineno = i - 1
324 if line[j] != " ":
325 raise ValueError("line " + `lineno` + " of docstring lacks "
326 "blank after " + PS1 + ": " + line)
327 j = j + 1
328 blanks = m.group(1)
329 nblanks = len(blanks)
330 # suck up this and following PS2 lines
331 source = []
332 while 1:
333 source.append(line[j:])
334 line = lines[i]
335 m = isPS2(line)
336 if m:
337 if m.group(1) != blanks:
338 raise ValueError("inconsistent leading whitespace "
339 "in line " + `i` + " of docstring: " + line)
340 i = i + 1
341 else:
342 break
343 if len(source) == 1:
344 source = source[0]
345 else:
346 # get rid of useless null line from trailing empty "..."
347 if source[-1] == "":
348 del source[-1]
Eric S. Raymond630e69c2001-02-09 08:33:43 +0000349 source = "\n".join(source) + "\n"
Tim Peters8a7d2d52001-01-16 07:10:57 +0000350 # suck up response
351 if isPS1(line) or isEmpty(line):
352 expect = ""
353 else:
354 expect = []
355 while 1:
356 if line[:nblanks] != blanks:
357 raise ValueError("inconsistent leading whitespace "
358 "in line " + `i` + " of docstring: " + line)
359 expect.append(line[nblanks:])
360 i = i + 1
361 line = lines[i]
362 if isPS1(line) or isEmpty(line):
363 break
Eric S. Raymond630e69c2001-02-09 08:33:43 +0000364 expect = "\n".join(expect) + "\n"
Tim Peters8a7d2d52001-01-16 07:10:57 +0000365 examples.append( (source, expect, lineno) )
366 return examples
367
368# Capture stdout when running examples.
369
370class _SpoofOut:
371 def __init__(self):
372 self.clear()
373 def write(self, s):
374 self.buf.append(s)
375 def get(self):
Tim Petersf9bb4962001-02-14 06:35:35 +0000376 guts = "".join(self.buf)
377 # If anything at all was written, make sure there's a trailing
378 # newline. There's no way for the expected output to indicate
379 # that a trailing newline is missing.
380 if guts and not guts.endswith("\n"):
381 guts = guts + "\n"
Tim Petersc77db342001-10-23 02:21:52 +0000382 # Prevent softspace from screwing up the next test case, in
383 # case they used print with a trailing comma in an example.
384 if hasattr(self, "softspace"):
385 del self.softspace
Tim Petersf9bb4962001-02-14 06:35:35 +0000386 return guts
Tim Peters8a7d2d52001-01-16 07:10:57 +0000387 def clear(self):
388 self.buf = []
Tim Petersc77db342001-10-23 02:21:52 +0000389 if hasattr(self, "softspace"):
390 del self.softspace
Tim Peters8a7d2d52001-01-16 07:10:57 +0000391 def flush(self):
392 # JPython calls flush
393 pass
394
395# Display some tag-and-msg pairs nicely, keeping the tag and its msg
396# on the same line when that makes sense.
397
398def _tag_out(printer, *tag_msg_pairs):
399 for tag, msg in tag_msg_pairs:
400 printer(tag + ":")
401 msg_has_nl = msg[-1:] == "\n"
402 msg_has_two_nl = msg_has_nl and \
Eric S. Raymond630e69c2001-02-09 08:33:43 +0000403 msg.find("\n") < len(msg) - 1
Tim Peters8a7d2d52001-01-16 07:10:57 +0000404 if len(tag) + len(msg) < 76 and not msg_has_two_nl:
405 printer(" ")
406 else:
407 printer("\n")
408 printer(msg)
409 if not msg_has_nl:
410 printer("\n")
411
412# Run list of examples, in context globs. "out" can be used to display
413# stuff to "the real" stdout, and fakeout is an instance of _SpoofOut
414# that captures the examples' std output. Return (#failures, #tries).
415
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000416def _run_examples_inner(out, fakeout, examples, globs, verbose, name,
417 compileflags):
Tim Peters8a7d2d52001-01-16 07:10:57 +0000418 import sys, traceback
419 OK, BOOM, FAIL = range(3)
420 NADA = "nothing"
421 stderr = _SpoofOut()
422 failures = 0
423 for source, want, lineno in examples:
424 if verbose:
425 _tag_out(out, ("Trying", source),
426 ("Expecting", want or NADA))
427 fakeout.clear()
428 try:
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000429 exec compile(source, "<string>", "single",
430 compileflags, 1) in globs
Tim Peters8a7d2d52001-01-16 07:10:57 +0000431 got = fakeout.get()
432 state = OK
Tim Petersbcc2c122002-03-20 19:32:03 +0000433 except KeyboardInterrupt:
434 raise
Tim Peters8a7d2d52001-01-16 07:10:57 +0000435 except:
436 # See whether the exception was expected.
Tim Petersea4f9312001-02-13 20:54:42 +0000437 if want.find("Traceback (innermost last):\n") == 0 or \
438 want.find("Traceback (most recent call last):\n") == 0:
Tim Peters8a7d2d52001-01-16 07:10:57 +0000439 # Only compare exception type and value - the rest of
440 # the traceback isn't necessary.
Eric S. Raymond630e69c2001-02-09 08:33:43 +0000441 want = want.split('\n')[-2] + '\n'
Tim Peters77f2d502001-06-24 18:59:01 +0000442 exc_type, exc_val = sys.exc_info()[:2]
Tim Peters08bba952001-06-24 06:46:58 +0000443 got = traceback.format_exception_only(exc_type, exc_val)[-1]
Tim Peters8a7d2d52001-01-16 07:10:57 +0000444 state = OK
445 else:
446 # unexpected exception
447 stderr.clear()
448 traceback.print_exc(file=stderr)
449 state = BOOM
450
451 if state == OK:
452 if got == want:
453 if verbose:
454 out("ok\n")
455 continue
456 state = FAIL
457
458 assert state in (FAIL, BOOM)
459 failures = failures + 1
460 out("*" * 65 + "\n")
461 _tag_out(out, ("Failure in example", source))
462 out("from line #" + `lineno` + " of " + name + "\n")
463 if state == FAIL:
464 _tag_out(out, ("Expected", want or NADA), ("Got", got))
465 else:
466 assert state == BOOM
467 _tag_out(out, ("Exception raised", stderr.get()))
468
469 return failures, len(examples)
470
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000471# Get the future-flags associated with the future features that have been
472# imported into globs.
473
474def _extract_future_flags(globs):
475 flags = 0
476 for fname in __future__.all_feature_names:
477 feature = globs.get(fname, None)
478 if feature is getattr(__future__, fname):
479 flags |= feature.compiler_flag
480 return flags
481
Tim Petersd4ad59e2001-06-24 20:02:47 +0000482# Run list of examples, in a shallow copy of context (dict) globs.
483# Return (#failures, #tries).
Tim Peters8a7d2d52001-01-16 07:10:57 +0000484
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000485def _run_examples(examples, globs, verbose, name, compileflags):
Tim Peters8a7d2d52001-01-16 07:10:57 +0000486 import sys
487 saveout = sys.stdout
Tim Petersd4ad59e2001-06-24 20:02:47 +0000488 globs = globs.copy()
Tim Peters8a7d2d52001-01-16 07:10:57 +0000489 try:
490 sys.stdout = fakeout = _SpoofOut()
491 x = _run_examples_inner(saveout.write, fakeout, examples,
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000492 globs, verbose, name, compileflags)
Tim Peters8a7d2d52001-01-16 07:10:57 +0000493 finally:
494 sys.stdout = saveout
Tim Petersd4ad59e2001-06-24 20:02:47 +0000495 # While Python gc can clean up most cycles on its own, it doesn't
496 # chase frame objects. This is especially irksome when running
497 # generator tests that raise exceptions, because a named generator-
498 # iterator gets an entry in globs, and the generator-iterator
499 # object's frame's traceback info points back to globs. This is
Tim Petersfee69d02001-06-24 20:24:16 +0000500 # easy to break just by clearing the namespace. This can also
501 # help to break other kinds of cycles, and even for cycles that
502 # gc can break itself it's better to break them ASAP.
Tim Petersd4ad59e2001-06-24 20:02:47 +0000503 globs.clear()
Tim Peters8a7d2d52001-01-16 07:10:57 +0000504 return x
505
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000506def run_docstring_examples(f, globs, verbose=0, name="NoName",
507 compileflags=None):
Tim Peters8a7d2d52001-01-16 07:10:57 +0000508 """f, globs, verbose=0, name="NoName" -> run examples from f.__doc__.
509
Tim Petersd4ad59e2001-06-24 20:02:47 +0000510 Use (a shallow copy of) dict globs as the globals for execution.
Tim Peters8a7d2d52001-01-16 07:10:57 +0000511 Return (#failures, #tries).
512
513 If optional arg verbose is true, print stuff even if there are no
514 failures.
515 Use string name in failure msgs.
516 """
517
518 try:
519 doc = f.__doc__
520 if not doc:
521 # docstring empty or None
522 return 0, 0
523 # just in case CT invents a doc object that has to be forced
524 # to look like a string <0.9 wink>
525 doc = str(doc)
Tim Petersbcc2c122002-03-20 19:32:03 +0000526 except KeyboardInterrupt:
527 raise
Tim Peters8a7d2d52001-01-16 07:10:57 +0000528 except:
529 return 0, 0
530
531 e = _extract_examples(doc)
532 if not e:
533 return 0, 0
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000534 if compileflags is None:
535 compileflags = _extract_future_flags(globs)
536 return _run_examples(e, globs, verbose, name, compileflags)
Tim Peters8a7d2d52001-01-16 07:10:57 +0000537
538def is_private(prefix, base):
539 """prefix, base -> true iff name prefix + "." + base is "private".
540
541 Prefix may be an empty string, and base does not contain a period.
542 Prefix is ignored (although functions you write conforming to this
543 protocol may make use of it).
544 Return true iff base begins with an (at least one) underscore, but
545 does not both begin and end with (at least) two underscores.
546
547 >>> is_private("a.b", "my_func")
Guido van Rossum77f6a652002-04-03 22:41:51 +0000548 False
Tim Peters8a7d2d52001-01-16 07:10:57 +0000549 >>> is_private("____", "_my_func")
Guido van Rossum77f6a652002-04-03 22:41:51 +0000550 True
Tim Peters8a7d2d52001-01-16 07:10:57 +0000551 >>> is_private("someclass", "__init__")
Guido van Rossum77f6a652002-04-03 22:41:51 +0000552 False
Tim Peters8a7d2d52001-01-16 07:10:57 +0000553 >>> is_private("sometypo", "__init_")
Guido van Rossum77f6a652002-04-03 22:41:51 +0000554 True
Tim Peters8a7d2d52001-01-16 07:10:57 +0000555 >>> is_private("x.y.z", "_")
Guido van Rossum77f6a652002-04-03 22:41:51 +0000556 True
Tim Peters8a7d2d52001-01-16 07:10:57 +0000557 >>> is_private("_x.y.z", "__")
Guido van Rossum77f6a652002-04-03 22:41:51 +0000558 False
Tim Peters8a7d2d52001-01-16 07:10:57 +0000559 >>> is_private("", "") # senseless but consistent
Guido van Rossum77f6a652002-04-03 22:41:51 +0000560 False
Tim Peters8a7d2d52001-01-16 07:10:57 +0000561 """
562
563 return base[:1] == "_" and not base[:2] == "__" == base[-2:]
564
Tim Peters7402f792001-10-02 03:53:41 +0000565# Determine if a class of function was defined in the given module.
566
567def _from_module(module, object):
568 if _isfunction(object):
569 return module.__dict__ is object.func_globals
570 if _isclass(object):
571 return module.__name__ == object.__module__
572 raise ValueError("object must be a class or function")
573
Tim Peters8a7d2d52001-01-16 07:10:57 +0000574class Tester:
575 """Class Tester -- runs docstring examples and accumulates stats.
576
577In normal use, function doctest.testmod() hides all this from you,
578so use that if you can. Create your own instances of Tester to do
579fancier things.
580
581Methods:
582 runstring(s, name)
583 Search string s for examples to run; use name for logging.
584 Return (#failures, #tries).
585
586 rundoc(object, name=None)
587 Search object.__doc__ for examples to run; use name (or
588 object.__name__) for logging. Return (#failures, #tries).
589
Tim Peters7402f792001-10-02 03:53:41 +0000590 rundict(d, name, module=None)
Tim Peters8a7d2d52001-01-16 07:10:57 +0000591 Search for examples in docstrings in all of d.values(); use name
Tim Peters7402f792001-10-02 03:53:41 +0000592 for logging. Exclude functions and classes not defined in module
593 if specified. Return (#failures, #tries).
Tim Peters8a7d2d52001-01-16 07:10:57 +0000594
595 run__test__(d, name)
596 Treat dict d like module.__test__. Return (#failures, #tries).
597
598 summarize(verbose=None)
599 Display summary of testing results, to stdout. Return
600 (#failures, #tries).
601
602 merge(other)
603 Merge in the test results from Tester instance "other".
604
605>>> from doctest import Tester
606>>> t = Tester(globs={'x': 42}, verbose=0)
607>>> t.runstring(r'''
608... >>> x = x * 2
609... >>> print x
610... 42
611... ''', 'XYZ')
612*****************************************************************
613Failure in example: print x
614from line #2 of XYZ
615Expected: 42
616Got: 84
617(1, 2)
618>>> t.runstring(">>> x = x * 2\\n>>> print x\\n84\\n", 'example2')
619(0, 2)
620>>> t.summarize()
Guido van Rossum261d91a2001-03-18 17:05:58 +0000621*****************************************************************
Tim Peters8a7d2d52001-01-16 07:10:57 +00006221 items had failures:
623 1 of 2 in XYZ
624***Test Failed*** 1 failures.
625(1, 4)
626>>> t.summarize(verbose=1)
6271 items passed all tests:
628 2 tests in example2
Guido van Rossum261d91a2001-03-18 17:05:58 +0000629*****************************************************************
Tim Peters8a7d2d52001-01-16 07:10:57 +00006301 items had failures:
631 1 of 2 in XYZ
6324 tests in 2 items.
6333 passed and 1 failed.
634***Test Failed*** 1 failures.
635(1, 4)
636>>>
637"""
638
639 def __init__(self, mod=None, globs=None, verbose=None,
640 isprivate=None):
641 """mod=None, globs=None, verbose=None, isprivate=None
642
643See doctest.__doc__ for an overview.
644
645Optional keyword arg "mod" is a module, whose globals are used for
646executing examples. If not specified, globs must be specified.
647
648Optional keyword arg "globs" gives a dict to be used as the globals
649when executing examples; if not specified, use the globals from
650module mod.
651
652In either case, a copy of the dict is used for each docstring
653examined.
654
655Optional keyword arg "verbose" prints lots of stuff if true, only
656failures if false; by default, it's true iff "-v" is in sys.argv.
657
658Optional keyword arg "isprivate" specifies a function used to determine
659whether a name is private. The default function is doctest.is_private;
660see its docs for details.
661"""
662
663 if mod is None and globs is None:
664 raise TypeError("Tester.__init__: must specify mod or globs")
Tim Peters7402f792001-10-02 03:53:41 +0000665 if mod is not None and not _ismodule(mod):
Tim Peters8a7d2d52001-01-16 07:10:57 +0000666 raise TypeError("Tester.__init__: mod must be a module; " +
667 `mod`)
668 if globs is None:
669 globs = mod.__dict__
670 self.globs = globs
671
672 if verbose is None:
673 import sys
674 verbose = "-v" in sys.argv
675 self.verbose = verbose
676
677 if isprivate is None:
678 isprivate = is_private
679 self.isprivate = isprivate
680
681 self.name2ft = {} # map name to (#failures, #trials) pair
682
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000683 self.compileflags = _extract_future_flags(globs)
684
Tim Peters8a7d2d52001-01-16 07:10:57 +0000685 def runstring(self, s, name):
686 """
687 s, name -> search string s for examples to run, logging as name.
688
689 Use string name as the key for logging the outcome.
690 Return (#failures, #examples).
691
692 >>> t = Tester(globs={}, verbose=1)
693 >>> test = r'''
694 ... # just an example
695 ... >>> x = 1 + 2
696 ... >>> x
697 ... 3
698 ... '''
699 >>> t.runstring(test, "Example")
700 Running string Example
701 Trying: x = 1 + 2
702 Expecting: nothing
703 ok
704 Trying: x
705 Expecting: 3
706 ok
707 0 of 2 examples failed in string Example
708 (0, 2)
709 """
710
711 if self.verbose:
712 print "Running string", name
713 f = t = 0
714 e = _extract_examples(s)
715 if e:
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000716 f, t = _run_examples(e, self.globs, self.verbose, name,
717 self.compileflags)
Tim Peters8a7d2d52001-01-16 07:10:57 +0000718 if self.verbose:
719 print f, "of", t, "examples failed in string", name
720 self.__record_outcome(name, f, t)
721 return f, t
722
723 def rundoc(self, object, name=None):
724 """
725 object, name=None -> search object.__doc__ for examples to run.
726
727 Use optional string name as the key for logging the outcome;
728 by default use object.__name__.
729 Return (#failures, #examples).
730 If object is a class object, search recursively for method
731 docstrings too.
732 object.__doc__ is examined regardless of name, but if object is
733 a class, whether private names reached from object are searched
734 depends on the constructor's "isprivate" argument.
735
736 >>> t = Tester(globs={}, verbose=0)
737 >>> def _f():
738 ... '''Trivial docstring example.
739 ... >>> assert 2 == 2
740 ... '''
741 ... return 32
742 ...
743 >>> t.rundoc(_f) # expect 0 failures in 1 example
744 (0, 1)
745 """
746
747 if name is None:
748 try:
749 name = object.__name__
750 except AttributeError:
751 raise ValueError("Tester.rundoc: name must be given "
752 "when object.__name__ doesn't exist; " + `object`)
753 if self.verbose:
754 print "Running", name + ".__doc__"
Tim Peters4fd9e2f2001-08-18 00:05:50 +0000755 f, t = run_docstring_examples(object, self.globs, self.verbose, name,
756 self.compileflags)
Tim Peters8a7d2d52001-01-16 07:10:57 +0000757 if self.verbose:
758 print f, "of", t, "examples failed in", name + ".__doc__"
759 self.__record_outcome(name, f, t)
Tim Peters7402f792001-10-02 03:53:41 +0000760 if _isclass(object):
Tim Peters17111f32001-10-03 04:08:26 +0000761 # In 2.2, class and static methods complicate life. Build
762 # a dict "that works", by hook or by crook.
763 d = {}
764 for tag, kind, homecls, value in _classify_class_attrs(object):
765
766 if homecls is not object:
767 # Only look at names defined immediately by the class.
768 continue
769
770 elif self.isprivate(name, tag):
771 continue
772
773 elif kind == "method":
774 # value is already a function
775 d[tag] = value
776
777 elif kind == "static method":
778 # value isn't a function, but getattr reveals one
779 d[tag] = getattr(object, tag)
780
781 elif kind == "class method":
782 # Hmm. A classmethod object doesn't seem to reveal
783 # enough. But getattr turns it into a bound method,
784 # and from there .im_func retrieves the underlying
785 # function.
786 d[tag] = getattr(object, tag).im_func
787
788 elif kind == "property":
789 # The methods implementing the property have their
790 # own docstrings -- but the property may have one too.
791 if value.__doc__ is not None:
792 d[tag] = str(value.__doc__)
793
794 elif kind == "data":
795 # Grab nested classes.
796 if _isclass(value):
797 d[tag] = value
798
799 else:
800 raise ValueError("teach doctest about %r" % kind)
801
802 f2, t2 = self.run__test__(d, name)
803 f += f2
804 t += t2
805
Tim Peters8a7d2d52001-01-16 07:10:57 +0000806 return f, t
807
Tim Peters7402f792001-10-02 03:53:41 +0000808 def rundict(self, d, name, module=None):
Tim Peters8a7d2d52001-01-16 07:10:57 +0000809 """
Tim Peters7402f792001-10-02 03:53:41 +0000810 d, name, module=None -> search for docstring examples in d.values().
Tim Peters8a7d2d52001-01-16 07:10:57 +0000811
812 For k, v in d.items() such that v is a function or class,
813 do self.rundoc(v, name + "." + k). Whether this includes
814 objects with private names depends on the constructor's
Tim Peters7402f792001-10-02 03:53:41 +0000815 "isprivate" argument. If module is specified, functions and
816 classes that are not defined in module are excluded.
Tim Peters8a7d2d52001-01-16 07:10:57 +0000817 Return aggregate (#failures, #examples).
818
Tim Peters7402f792001-10-02 03:53:41 +0000819 Build and populate two modules with sample functions to test that
820 exclusion of external functions and classes works.
821
822 >>> import new
823 >>> m1 = new.module('_m1')
824 >>> m2 = new.module('_m2')
825 >>> test_data = \"""
Tim Peters4a9ac4a2001-10-02 22:47:08 +0000826 ... def _f():
Tim Peters7402f792001-10-02 03:53:41 +0000827 ... '''>>> assert 1 == 1
828 ... '''
829 ... def g():
Tim Peters8a7d2d52001-01-16 07:10:57 +0000830 ... '''>>> assert 2 != 1
831 ... '''
Tim Peters7402f792001-10-02 03:53:41 +0000832 ... class H:
833 ... '''>>> assert 2 > 1
834 ... '''
835 ... def bar(self):
836 ... '''>>> assert 1 < 2
837 ... '''
838 ... \"""
839 >>> exec test_data in m1.__dict__
840 >>> exec test_data in m2.__dict__
Tim Peters4a9ac4a2001-10-02 22:47:08 +0000841 >>> m1.__dict__.update({"f2": m2._f, "g2": m2.g, "h2": m2.H})
Tim Peters7402f792001-10-02 03:53:41 +0000842
843 Tests that objects outside m1 are excluded:
844
Tim Peters8a7d2d52001-01-16 07:10:57 +0000845 >>> t = Tester(globs={}, verbose=0)
Tim Peters4a9ac4a2001-10-02 22:47:08 +0000846 >>> t.rundict(m1.__dict__, "rundict_test", m1) # _f, f2 and g2 and h2 skipped
Tim Peters7402f792001-10-02 03:53:41 +0000847 (0, 3)
848
849 Again, but with a custom isprivate function allowing _f:
850
Tim Peters8a7d2d52001-01-16 07:10:57 +0000851 >>> t = Tester(globs={}, verbose=0, isprivate=lambda x,y: 0)
Tim Peters4a9ac4a2001-10-02 22:47:08 +0000852 >>> t.rundict(m1.__dict__, "rundict_test_pvt", m1) # Only f2, g2 and h2 skipped
Tim Peters7402f792001-10-02 03:53:41 +0000853 (0, 4)
854
855 And once more, not excluding stuff outside m1:
856
857 >>> t = Tester(globs={}, verbose=0, isprivate=lambda x,y: 0)
Tim Peters4a9ac4a2001-10-02 22:47:08 +0000858 >>> t.rundict(m1.__dict__, "rundict_test_pvt") # None are skipped.
Tim Peters7402f792001-10-02 03:53:41 +0000859 (0, 8)
Tim Peters4a9ac4a2001-10-02 22:47:08 +0000860
861 The exclusion of objects from outside the designated module is
862 meant to be invoked automagically by testmod.
863
864 >>> testmod(m1)
865 (0, 3)
866
Tim Peters8a7d2d52001-01-16 07:10:57 +0000867 """
868
869 if not hasattr(d, "items"):
870 raise TypeError("Tester.rundict: d must support .items(); " +
871 `d`)
872 f = t = 0
Tim Peters24a41912001-03-21 23:07:59 +0000873 # Run the tests by alpha order of names, for consistency in
874 # verbose-mode output.
875 names = d.keys()
876 names.sort()
877 for thisname in names:
878 value = d[thisname]
Tim Peters7402f792001-10-02 03:53:41 +0000879 if _isfunction(value) or _isclass(value):
880 if module and not _from_module(module, value):
881 continue
Tim Peters8a7d2d52001-01-16 07:10:57 +0000882 f2, t2 = self.__runone(value, name + "." + thisname)
883 f = f + f2
884 t = t + t2
885 return f, t
886
887 def run__test__(self, d, name):
888 """d, name -> Treat dict d like module.__test__.
889
890 Return (#failures, #tries).
891 See testmod.__doc__ for details.
892 """
893
894 failures = tries = 0
895 prefix = name + "."
896 savepvt = self.isprivate
897 try:
898 self.isprivate = lambda *args: 0
Tim Peters24a41912001-03-21 23:07:59 +0000899 # Run the tests by alpha order of names, for consistency in
900 # verbose-mode output.
901 keys = d.keys()
902 keys.sort()
903 for k in keys:
904 v = d[k]
Tim Peters8a7d2d52001-01-16 07:10:57 +0000905 thisname = prefix + k
Tim Peters7402f792001-10-02 03:53:41 +0000906 if type(v) in _StringTypes:
Tim Peters8a7d2d52001-01-16 07:10:57 +0000907 f, t = self.runstring(v, thisname)
Tim Peters7402f792001-10-02 03:53:41 +0000908 elif _isfunction(v) or _isclass(v):
Tim Peters8a7d2d52001-01-16 07:10:57 +0000909 f, t = self.rundoc(v, thisname)
910 else:
911 raise TypeError("Tester.run__test__: values in "
912 "dict must be strings, functions "
913 "or classes; " + `v`)
914 failures = failures + f
915 tries = tries + t
916 finally:
917 self.isprivate = savepvt
918 return failures, tries
919
920 def summarize(self, verbose=None):
921 """
922 verbose=None -> summarize results, return (#failures, #tests).
923
924 Print summary of test results to stdout.
925 Optional arg 'verbose' controls how wordy this is. By
926 default, use the verbose setting established by the
927 constructor.
928 """
929
930 if verbose is None:
931 verbose = self.verbose
932 notests = []
933 passed = []
934 failed = []
935 totalt = totalf = 0
936 for x in self.name2ft.items():
937 name, (f, t) = x
938 assert f <= t
939 totalt = totalt + t
940 totalf = totalf + f
941 if t == 0:
942 notests.append(name)
943 elif f == 0:
944 passed.append( (name, t) )
945 else:
946 failed.append(x)
947 if verbose:
948 if notests:
949 print len(notests), "items had no tests:"
950 notests.sort()
951 for thing in notests:
952 print " ", thing
953 if passed:
954 print len(passed), "items passed all tests:"
955 passed.sort()
956 for thing, count in passed:
957 print " %3d tests in %s" % (count, thing)
958 if failed:
Guido van Rossumaf00a462001-03-18 16:58:44 +0000959 print "*" * 65
Tim Peters8a7d2d52001-01-16 07:10:57 +0000960 print len(failed), "items had failures:"
961 failed.sort()
962 for thing, (f, t) in failed:
963 print " %3d of %3d in %s" % (f, t, thing)
964 if verbose:
965 print totalt, "tests in", len(self.name2ft), "items."
966 print totalt - totalf, "passed and", totalf, "failed."
967 if totalf:
968 print "***Test Failed***", totalf, "failures."
969 elif verbose:
970 print "Test passed."
971 return totalf, totalt
972
973 def merge(self, other):
974 """
975 other -> merge in test results from the other Tester instance.
976
977 If self and other both have a test result for something
978 with the same name, the (#failures, #tests) results are
979 summed, and a warning is printed to stdout.
980
981 >>> from doctest import Tester
982 >>> t1 = Tester(globs={}, verbose=0)
983 >>> t1.runstring('''
984 ... >>> x = 12
985 ... >>> print x
986 ... 12
987 ... ''', "t1example")
988 (0, 2)
989 >>>
990 >>> t2 = Tester(globs={}, verbose=0)
991 >>> t2.runstring('''
992 ... >>> x = 13
993 ... >>> print x
994 ... 13
995 ... ''', "t2example")
996 (0, 2)
997 >>> common = ">>> assert 1 + 2 == 3\\n"
998 >>> t1.runstring(common, "common")
999 (0, 1)
1000 >>> t2.runstring(common, "common")
1001 (0, 1)
1002 >>> t1.merge(t2)
1003 *** Tester.merge: 'common' in both testers; summing outcomes.
1004 >>> t1.summarize(1)
1005 3 items passed all tests:
1006 2 tests in common
1007 2 tests in t1example
1008 2 tests in t2example
1009 6 tests in 3 items.
1010 6 passed and 0 failed.
1011 Test passed.
1012 (0, 6)
1013 >>>
1014 """
1015
1016 d = self.name2ft
1017 for name, (f, t) in other.name2ft.items():
Raymond Hettinger54f02222002-06-01 14:18:47 +00001018 if name in d:
Tim Peters8a7d2d52001-01-16 07:10:57 +00001019 print "*** Tester.merge: '" + name + "' in both" \
1020 " testers; summing outcomes."
1021 f2, t2 = d[name]
1022 f = f + f2
1023 t = t + t2
1024 d[name] = f, t
1025
1026 def __record_outcome(self, name, f, t):
Raymond Hettinger54f02222002-06-01 14:18:47 +00001027 if name in self.name2ft:
Tim Peters8a7d2d52001-01-16 07:10:57 +00001028 print "*** Warning: '" + name + "' was tested before;", \
1029 "summing outcomes."
1030 f2, t2 = self.name2ft[name]
1031 f = f + f2
1032 t = t + t2
1033 self.name2ft[name] = f, t
1034
1035 def __runone(self, target, name):
1036 if "." in name:
Eric S. Raymond630e69c2001-02-09 08:33:43 +00001037 i = name.rindex(".")
Tim Peters8a7d2d52001-01-16 07:10:57 +00001038 prefix, base = name[:i], name[i+1:]
1039 else:
1040 prefix, base = "", base
1041 if self.isprivate(prefix, base):
1042 return 0, 0
1043 return self.rundoc(target, name)
1044
1045master = None
1046
Martin v. Löwis4581cfa2002-11-22 08:23:09 +00001047def testmod(m=None, name=None, globs=None, verbose=None, isprivate=None,
Tim Peters8a7d2d52001-01-16 07:10:57 +00001048 report=1):
Martin v. Löwis4581cfa2002-11-22 08:23:09 +00001049 """m=None, name=None, globs=None, verbose=None, isprivate=None, report=1
Tim Peters8a7d2d52001-01-16 07:10:57 +00001050
Martin v. Löwis4581cfa2002-11-22 08:23:09 +00001051 Test examples in docstrings in functions and classes reachable
1052 from module m (or the current module if m is not supplied), starting
1053 with m.__doc__. Private names are skipped.
Tim Peters8a7d2d52001-01-16 07:10:57 +00001054
1055 Also test examples reachable from dict m.__test__ if it exists and is
1056 not None. m.__dict__ maps names to functions, classes and strings;
1057 function and class docstrings are tested even if the name is private;
1058 strings are tested directly, as if they were docstrings.
1059
1060 Return (#failures, #tests).
1061
1062 See doctest.__doc__ for an overview.
1063
1064 Optional keyword arg "name" gives the name of the module; by default
1065 use m.__name__.
1066
1067 Optional keyword arg "globs" gives a dict to be used as the globals
1068 when executing examples; by default, use m.__dict__. A copy of this
1069 dict is actually used for each docstring, so that each docstring's
1070 examples start with a clean slate.
1071
1072 Optional keyword arg "verbose" prints lots of stuff if true, prints
1073 only failures if false; by default, it's true iff "-v" is in sys.argv.
1074
1075 Optional keyword arg "isprivate" specifies a function used to
1076 determine whether a name is private. The default function is
1077 doctest.is_private; see its docs for details.
1078
1079 Optional keyword arg "report" prints a summary at the end when true,
1080 else prints nothing at the end. In verbose mode, the summary is
1081 detailed, else very brief (in fact, empty if all tests passed).
1082
1083 Advanced tomfoolery: testmod runs methods of a local instance of
1084 class doctest.Tester, then merges the results into (or creates)
1085 global Tester instance doctest.master. Methods of doctest.master
1086 can be called directly too, if you want to do something unusual.
1087 Passing report=0 to testmod is especially useful then, to delay
1088 displaying a summary. Invoke doctest.master.summarize(verbose)
1089 when you're done fiddling.
1090 """
1091
1092 global master
1093
Martin v. Löwis4581cfa2002-11-22 08:23:09 +00001094 if m is None:
1095 import sys
1096 # DWA - m will still be None if this wasn't invoked from the command
1097 # line, in which case the following TypeError is about as good an error
1098 # as we should expect
1099 m = sys.modules.get('__main__')
1100
Tim Peters7402f792001-10-02 03:53:41 +00001101 if not _ismodule(m):
Tim Peters8a7d2d52001-01-16 07:10:57 +00001102 raise TypeError("testmod: module required; " + `m`)
1103 if name is None:
1104 name = m.__name__
1105 tester = Tester(m, globs=globs, verbose=verbose, isprivate=isprivate)
1106 failures, tries = tester.rundoc(m, name)
Tim Peters4a9ac4a2001-10-02 22:47:08 +00001107 f, t = tester.rundict(m.__dict__, name, m)
Tim Peters8a7d2d52001-01-16 07:10:57 +00001108 failures = failures + f
1109 tries = tries + t
1110 if hasattr(m, "__test__"):
1111 testdict = m.__test__
1112 if testdict:
1113 if not hasattr(testdict, "items"):
1114 raise TypeError("testmod: module.__test__ must support "
1115 ".items(); " + `testdict`)
1116 f, t = tester.run__test__(testdict, name + ".__test__")
1117 failures = failures + f
1118 tries = tries + t
1119 if report:
1120 tester.summarize()
1121 if master is None:
1122 master = tester
1123 else:
1124 master.merge(tester)
1125 return failures, tries
1126
1127class _TestClass:
1128 """
1129 A pointless class, for sanity-checking of docstring testing.
1130
1131 Methods:
1132 square()
1133 get()
1134
1135 >>> _TestClass(13).get() + _TestClass(-12).get()
1136 1
1137 >>> hex(_TestClass(13).square().get())
1138 '0xa9'
1139 """
1140
1141 def __init__(self, val):
1142 """val -> _TestClass object with associated value val.
1143
1144 >>> t = _TestClass(123)
1145 >>> print t.get()
1146 123
1147 """
1148
1149 self.val = val
1150
1151 def square(self):
1152 """square() -> square TestClass's associated value
1153
1154 >>> _TestClass(13).square().get()
1155 169
1156 """
1157
1158 self.val = self.val ** 2
1159 return self
1160
1161 def get(self):
1162 """get() -> return TestClass's associated value.
1163
1164 >>> x = _TestClass(-42)
1165 >>> print x.get()
1166 -42
1167 """
1168
1169 return self.val
1170
1171__test__ = {"_TestClass": _TestClass,
1172 "string": r"""
1173 Example of a string object, searched as-is.
1174 >>> x = 1; y = 2
1175 >>> x + y, x * y
1176 (3, 2)
1177 """
1178 }
1179
1180def _test():
1181 import doctest
1182 return doctest.testmod(doctest)
1183
1184if __name__ == "__main__":
1185 _test()