Turns out Neil didn't intend for *all* of his gen-branch work to get
committed.
tokenize.py: I like these changes, and have tested them extensively
without even realizing it, so I just updated the docstring and the docs.
tabnanny.py: Also liked this, but did a little code fiddling. I should
really rewrite this to *exploit* generators, but that's near the bottom
of my effort/benefit scale so doubt I'll get to it anytime soon (it
would be most useful as a non-trivial example of ideal use of generators;
but test_generators.py has already grown plenty of food-for-thought
examples).
inspect.py: I'm sure Ping intended for this to continue running even
under 1.5.2, so I reverted this to the last pre-gen-branch version. The
"bugfix" I checked in in-between was actually repairing a bug *introduced*
by the conversion to generators, so it's OK that the reverted version
doesn't reflect that checkin.
diff --git a/Lib/tabnanny.py b/Lib/tabnanny.py
index 8323a33..35e5cfa 100755
--- a/Lib/tabnanny.py
+++ b/Lib/tabnanny.py
@@ -14,6 +14,8 @@
import sys
import getopt
import tokenize
+if not hasattr(tokenize, 'NL'):
+ raise ValueError("tokenize.NL doesn't exist -- tokenize module too old")
__all__ = ["check"]
@@ -243,15 +245,11 @@
prefix = prefix + "s"
return prefix + " " + string.join(firsts, ', ')
-# Need Guido's enhancement
-assert hasattr(tokenize, 'NL'), "tokenize module too old"
-
-def process_tokens(tokens,
- INDENT=tokenize.INDENT,
- DEDENT=tokenize.DEDENT,
- NEWLINE=tokenize.NEWLINE,
- JUNK=(tokenize.COMMENT, tokenize.NL)):
-
+def process_tokens(tokens):
+ INDENT = tokenize.INDENT
+ DEDENT = tokenize.DEDENT
+ NEWLINE = tokenize.NEWLINE
+ JUNK = tokenize.COMMENT, tokenize.NL
indents = [Whitespace("")]
check_equal = 0