Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | **************************** |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 2 | What's New in Python 2.2 |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 3 | **************************** |
| 4 | |
| 5 | :Author: A.M. Kuchling |
| 6 | |
| 7 | .. |release| replace:: 1.02 |
| 8 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 9 | .. $Id: whatsnew22.tex 37315 2004-09-10 19:33:00Z akuchling $ |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 10 | |
| 11 | |
| 12 | Introduction |
| 13 | ============ |
| 14 | |
| 15 | This article explains the new features in Python 2.2.2, released on October 14, |
| 16 | 2002. Python 2.2.2 is a bugfix release of Python 2.2, originally released on |
| 17 | December 21, 2001. |
| 18 | |
| 19 | Python 2.2 can be thought of as the "cleanup release". There are some features |
| 20 | such as generators and iterators that are completely new, but most of the |
| 21 | changes, significant and far-reaching though they may be, are aimed at cleaning |
| 22 | up irregularities and dark corners of the language design. |
| 23 | |
| 24 | This article doesn't attempt to provide a complete specification of the new |
| 25 | features, but instead provides a convenient overview. For full details, you |
| 26 | should refer to the documentation for Python 2.2, such as the `Python Library |
| 27 | Reference <http://www.python.org/doc/2.2/lib/lib.html>`_ and the `Python |
| 28 | Reference Manual <http://www.python.org/doc/2.2/ref/ref.html>`_. If you want to |
| 29 | understand the complete implementation and design rationale for a change, refer |
| 30 | to the PEP for a particular new feature. |
| 31 | |
| 32 | |
| 33 | .. seealso:: |
| 34 | |
| 35 | http://www.unixreview.com/documents/s=1356/urm0109h/0109h.htm |
| 36 | "What's So Special About Python 2.2?" is also about the new 2.2 features, and |
| 37 | was written by Cameron Laird and Kathryn Soraiz. |
| 38 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 39 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 40 | |
| 41 | |
| 42 | PEPs 252 and 253: Type and Class Changes |
| 43 | ======================================== |
| 44 | |
| 45 | The largest and most far-reaching changes in Python 2.2 are to Python's model of |
| 46 | objects and classes. The changes should be backward compatible, so it's likely |
| 47 | that your code will continue to run unchanged, but the changes provide some |
| 48 | amazing new capabilities. Before beginning this, the longest and most |
| 49 | complicated section of this article, I'll provide an overview of the changes and |
| 50 | offer some comments. |
| 51 | |
| 52 | A long time ago I wrote a Web page (http://www.amk.ca/python/writing/warts.html) |
| 53 | listing flaws in Python's design. One of the most significant flaws was that |
| 54 | it's impossible to subclass Python types implemented in C. In particular, it's |
| 55 | not possible to subclass built-in types, so you can't just subclass, say, lists |
| 56 | in order to add a single useful method to them. The :mod:`UserList` module |
| 57 | provides a class that supports all of the methods of lists and that can be |
| 58 | subclassed further, but there's lots of C code that expects a regular Python |
| 59 | list and won't accept a :class:`UserList` instance. |
| 60 | |
| 61 | Python 2.2 fixes this, and in the process adds some exciting new capabilities. |
| 62 | A brief summary: |
| 63 | |
| 64 | * You can subclass built-in types such as lists and even integers, and your |
| 65 | subclasses should work in every place that requires the original type. |
| 66 | |
| 67 | * It's now possible to define static and class methods, in addition to the |
| 68 | instance methods available in previous versions of Python. |
| 69 | |
| 70 | * It's also possible to automatically call methods on accessing or setting an |
| 71 | instance attribute by using a new mechanism called :dfn:`properties`. Many uses |
| 72 | of :meth:`__getattr__` can be rewritten to use properties instead, making the |
| 73 | resulting code simpler and faster. As a small side benefit, attributes can now |
| 74 | have docstrings, too. |
| 75 | |
| 76 | * The list of legal attributes for an instance can be limited to a particular |
| 77 | set using :dfn:`slots`, making it possible to safeguard against typos and |
| 78 | perhaps make more optimizations possible in future versions of Python. |
| 79 | |
| 80 | Some users have voiced concern about all these changes. Sure, they say, the new |
| 81 | features are neat and lend themselves to all sorts of tricks that weren't |
| 82 | possible in previous versions of Python, but they also make the language more |
| 83 | complicated. Some people have said that they've always recommended Python for |
| 84 | its simplicity, and feel that its simplicity is being lost. |
| 85 | |
| 86 | Personally, I think there's no need to worry. Many of the new features are |
| 87 | quite esoteric, and you can write a lot of Python code without ever needed to be |
| 88 | aware of them. Writing a simple class is no more difficult than it ever was, so |
| 89 | you don't need to bother learning or teaching them unless they're actually |
| 90 | needed. Some very complicated tasks that were previously only possible from C |
| 91 | will now be possible in pure Python, and to my mind that's all for the better. |
| 92 | |
| 93 | I'm not going to attempt to cover every single corner case and small change that |
| 94 | were required to make the new features work. Instead this section will paint |
| 95 | only the broad strokes. See section :ref:`sect-rellinks`, "Related Links", for |
| 96 | further sources of information about Python 2.2's new object model. |
| 97 | |
| 98 | |
| 99 | Old and New Classes |
| 100 | ------------------- |
| 101 | |
| 102 | First, you should know that Python 2.2 really has two kinds of classes: classic |
| 103 | or old-style classes, and new-style classes. The old-style class model is |
| 104 | exactly the same as the class model in earlier versions of Python. All the new |
| 105 | features described in this section apply only to new-style classes. This |
| 106 | divergence isn't intended to last forever; eventually old-style classes will be |
| 107 | dropped, possibly in Python 3.0. |
| 108 | |
| 109 | So how do you define a new-style class? You do it by subclassing an existing |
| 110 | new-style class. Most of Python's built-in types, such as integers, lists, |
| 111 | dictionaries, and even files, are new-style classes now. A new-style class |
| 112 | named :class:`object`, the base class for all built-in types, has also been |
| 113 | added so if no built-in type is suitable, you can just subclass |
| 114 | :class:`object`:: |
| 115 | |
| 116 | class C(object): |
| 117 | def __init__ (self): |
| 118 | ... |
| 119 | ... |
| 120 | |
| 121 | This means that :keyword:`class` statements that don't have any base classes are |
| 122 | always classic classes in Python 2.2. (Actually you can also change this by |
| 123 | setting a module-level variable named :attr:`__metaclass__` --- see :pep:`253` |
| 124 | for the details --- but it's easier to just subclass :keyword:`object`.) |
| 125 | |
| 126 | The type objects for the built-in types are available as built-ins, named using |
| 127 | a clever trick. Python has always had built-in functions named :func:`int`, |
| 128 | :func:`float`, and :func:`str`. In 2.2, they aren't functions any more, but |
| 129 | type objects that behave as factories when called. :: |
| 130 | |
| 131 | >>> int |
| 132 | <type 'int'> |
| 133 | >>> int('123') |
| 134 | 123 |
| 135 | |
| 136 | To make the set of types complete, new type objects such as :func:`dict` and |
| 137 | :func:`file` have been added. Here's a more interesting example, adding a |
| 138 | :meth:`lock` method to file objects:: |
| 139 | |
| 140 | class LockableFile(file): |
| 141 | def lock (self, operation, length=0, start=0, whence=0): |
| 142 | import fcntl |
| 143 | return fcntl.lockf(self.fileno(), operation, |
| 144 | length, start, whence) |
| 145 | |
| 146 | The now-obsolete :mod:`posixfile` module contained a class that emulated all of |
| 147 | a file object's methods and also added a :meth:`lock` method, but this class |
| 148 | couldn't be passed to internal functions that expected a built-in file, |
| 149 | something which is possible with our new :class:`LockableFile`. |
| 150 | |
| 151 | |
| 152 | Descriptors |
| 153 | ----------- |
| 154 | |
| 155 | In previous versions of Python, there was no consistent way to discover what |
| 156 | attributes and methods were supported by an object. There were some informal |
| 157 | conventions, such as defining :attr:`__members__` and :attr:`__methods__` |
| 158 | attributes that were lists of names, but often the author of an extension type |
| 159 | or a class wouldn't bother to define them. You could fall back on inspecting |
| 160 | the :attr:`__dict__` of an object, but when class inheritance or an arbitrary |
| 161 | :meth:`__getattr__` hook were in use this could still be inaccurate. |
| 162 | |
| 163 | The one big idea underlying the new class model is that an API for describing |
| 164 | the attributes of an object using :dfn:`descriptors` has been formalized. |
| 165 | Descriptors specify the value of an attribute, stating whether it's a method or |
| 166 | a field. With the descriptor API, static methods and class methods become |
| 167 | possible, as well as more exotic constructs. |
| 168 | |
| 169 | Attribute descriptors are objects that live inside class objects, and have a few |
| 170 | attributes of their own: |
| 171 | |
| 172 | * :attr:`__name__` is the attribute's name. |
| 173 | |
| 174 | * :attr:`__doc__` is the attribute's docstring. |
| 175 | |
| 176 | * :meth:`__get__(object)` is a method that retrieves the attribute value from |
| 177 | *object*. |
| 178 | |
| 179 | * :meth:`__set__(object, value)` sets the attribute on *object* to *value*. |
| 180 | |
| 181 | * :meth:`__delete__(object, value)` deletes the *value* attribute of *object*. |
| 182 | |
| 183 | For example, when you write ``obj.x``, the steps that Python actually performs |
| 184 | are:: |
| 185 | |
| 186 | descriptor = obj.__class__.x |
| 187 | descriptor.__get__(obj) |
| 188 | |
| 189 | For methods, :meth:`descriptor.__get__` returns a temporary object that's |
| 190 | callable, and wraps up the instance and the method to be called on it. This is |
| 191 | also why static methods and class methods are now possible; they have |
| 192 | descriptors that wrap up just the method, or the method and the class. As a |
| 193 | brief explanation of these new kinds of methods, static methods aren't passed |
| 194 | the instance, and therefore resemble regular functions. Class methods are |
| 195 | passed the class of the object, but not the object itself. Static and class |
| 196 | methods are defined like this:: |
| 197 | |
| 198 | class C(object): |
| 199 | def f(arg1, arg2): |
| 200 | ... |
| 201 | f = staticmethod(f) |
| 202 | |
| 203 | def g(cls, arg1, arg2): |
| 204 | ... |
| 205 | g = classmethod(g) |
| 206 | |
| 207 | The :func:`staticmethod` function takes the function :func:`f`, and returns it |
| 208 | wrapped up in a descriptor so it can be stored in the class object. You might |
| 209 | expect there to be special syntax for creating such methods (``def static f``, |
| 210 | ``defstatic f()``, or something like that) but no such syntax has been defined |
| 211 | yet; that's been left for future versions of Python. |
| 212 | |
| 213 | More new features, such as slots and properties, are also implemented as new |
| 214 | kinds of descriptors, and it's not difficult to write a descriptor class that |
| 215 | does something novel. For example, it would be possible to write a descriptor |
| 216 | class that made it possible to write Eiffel-style preconditions and |
| 217 | postconditions for a method. A class that used this feature might be defined |
| 218 | like this:: |
| 219 | |
| 220 | from eiffel import eiffelmethod |
| 221 | |
| 222 | class C(object): |
| 223 | def f(self, arg1, arg2): |
| 224 | # The actual function |
| 225 | ... |
| 226 | def pre_f(self): |
| 227 | # Check preconditions |
| 228 | ... |
| 229 | def post_f(self): |
| 230 | # Check postconditions |
| 231 | ... |
| 232 | |
| 233 | f = eiffelmethod(f, pre_f, post_f) |
| 234 | |
| 235 | Note that a person using the new :func:`eiffelmethod` doesn't have to understand |
| 236 | anything about descriptors. This is why I think the new features don't increase |
| 237 | the basic complexity of the language. There will be a few wizards who need to |
| 238 | know about it in order to write :func:`eiffelmethod` or the ZODB or whatever, |
| 239 | but most users will just write code on top of the resulting libraries and ignore |
| 240 | the implementation details. |
| 241 | |
| 242 | |
| 243 | Multiple Inheritance: The Diamond Rule |
| 244 | -------------------------------------- |
| 245 | |
| 246 | Multiple inheritance has also been made more useful through changing the rules |
| 247 | under which names are resolved. Consider this set of classes (diagram taken |
| 248 | from :pep:`253` by Guido van Rossum):: |
| 249 | |
| 250 | class A: |
| 251 | ^ ^ def save(self): ... |
| 252 | / \ |
| 253 | / \ |
| 254 | / \ |
| 255 | / \ |
| 256 | class B class C: |
| 257 | ^ ^ def save(self): ... |
| 258 | \ / |
| 259 | \ / |
| 260 | \ / |
| 261 | \ / |
| 262 | class D |
| 263 | |
| 264 | The lookup rule for classic classes is simple but not very smart; the base |
| 265 | classes are searched depth-first, going from left to right. A reference to |
| 266 | :meth:`D.save` will search the classes :class:`D`, :class:`B`, and then |
| 267 | :class:`A`, where :meth:`save` would be found and returned. :meth:`C.save` |
| 268 | would never be found at all. This is bad, because if :class:`C`'s :meth:`save` |
| 269 | method is saving some internal state specific to :class:`C`, not calling it will |
| 270 | result in that state never getting saved. |
| 271 | |
| 272 | New-style classes follow a different algorithm that's a bit more complicated to |
| 273 | explain, but does the right thing in this situation. (Note that Python 2.3 |
| 274 | changes this algorithm to one that produces the same results in most cases, but |
| 275 | produces more useful results for really complicated inheritance graphs.) |
| 276 | |
| 277 | #. List all the base classes, following the classic lookup rule and include a |
| 278 | class multiple times if it's visited repeatedly. In the above example, the list |
| 279 | of visited classes is [:class:`D`, :class:`B`, :class:`A`, :class:`C`, |
| 280 | :class:`A`]. |
| 281 | |
| 282 | #. Scan the list for duplicated classes. If any are found, remove all but one |
| 283 | occurrence, leaving the *last* one in the list. In the above example, the list |
| 284 | becomes [:class:`D`, :class:`B`, :class:`C`, :class:`A`] after dropping |
| 285 | duplicates. |
| 286 | |
| 287 | Following this rule, referring to :meth:`D.save` will return :meth:`C.save`, |
| 288 | which is the behaviour we're after. This lookup rule is the same as the one |
| 289 | followed by Common Lisp. A new built-in function, :func:`super`, provides a way |
| 290 | to get at a class's superclasses without having to reimplement Python's |
| 291 | algorithm. The most commonly used form will be :func:`super(class, obj)`, which |
| 292 | returns a bound superclass object (not the actual class object). This form |
| 293 | will be used in methods to call a method in the superclass; for example, |
| 294 | :class:`D`'s :meth:`save` method would look like this:: |
| 295 | |
| 296 | class D (B,C): |
| 297 | def save (self): |
Georg Brandl | a1c6a1c | 2009-01-03 21:26:05 +0000 | [diff] [blame] | 298 | # Call superclass .save() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 299 | super(D, self).save() |
| 300 | # Save D's private information here |
| 301 | ... |
| 302 | |
| 303 | :func:`super` can also return unbound superclass objects when called as |
| 304 | :func:`super(class)` or :func:`super(class1, class2)`, but this probably won't |
| 305 | often be useful. |
| 306 | |
| 307 | |
| 308 | Attribute Access |
| 309 | ---------------- |
| 310 | |
| 311 | A fair number of sophisticated Python classes define hooks for attribute access |
| 312 | using :meth:`__getattr__`; most commonly this is done for convenience, to make |
| 313 | code more readable by automatically mapping an attribute access such as |
| 314 | ``obj.parent`` into a method call such as ``obj.get_parent``. Python 2.2 adds |
| 315 | some new ways of controlling attribute access. |
| 316 | |
| 317 | First, :meth:`__getattr__(attr_name)` is still supported by new-style classes, |
| 318 | and nothing about it has changed. As before, it will be called when an attempt |
| 319 | is made to access ``obj.foo`` and no attribute named ``foo`` is found in the |
| 320 | instance's dictionary. |
| 321 | |
| 322 | New-style classes also support a new method, |
| 323 | :meth:`__getattribute__(attr_name)`. The difference between the two methods is |
| 324 | that :meth:`__getattribute__` is *always* called whenever any attribute is |
| 325 | accessed, while the old :meth:`__getattr__` is only called if ``foo`` isn't |
| 326 | found in the instance's dictionary. |
| 327 | |
| 328 | However, Python 2.2's support for :dfn:`properties` will often be a simpler way |
| 329 | to trap attribute references. Writing a :meth:`__getattr__` method is |
| 330 | complicated because to avoid recursion you can't use regular attribute accesses |
| 331 | inside them, and instead have to mess around with the contents of |
| 332 | :attr:`__dict__`. :meth:`__getattr__` methods also end up being called by Python |
| 333 | when it checks for other methods such as :meth:`__repr__` or :meth:`__coerce__`, |
| 334 | and so have to be written with this in mind. Finally, calling a function on |
| 335 | every attribute access results in a sizable performance loss. |
| 336 | |
| 337 | :class:`property` is a new built-in type that packages up three functions that |
| 338 | get, set, or delete an attribute, and a docstring. For example, if you want to |
| 339 | define a :attr:`size` attribute that's computed, but also settable, you could |
| 340 | write:: |
| 341 | |
| 342 | class C(object): |
| 343 | def get_size (self): |
| 344 | result = ... computation ... |
| 345 | return result |
| 346 | def set_size (self, size): |
| 347 | ... compute something based on the size |
| 348 | and set internal state appropriately ... |
| 349 | |
| 350 | # Define a property. The 'delete this attribute' |
| 351 | # method is defined as None, so the attribute |
| 352 | # can't be deleted. |
| 353 | size = property(get_size, set_size, |
| 354 | None, |
| 355 | "Storage size of this instance") |
| 356 | |
| 357 | That is certainly clearer and easier to write than a pair of |
| 358 | :meth:`__getattr__`/:meth:`__setattr__` methods that check for the :attr:`size` |
| 359 | attribute and handle it specially while retrieving all other attributes from the |
| 360 | instance's :attr:`__dict__`. Accesses to :attr:`size` are also the only ones |
| 361 | which have to perform the work of calling a function, so references to other |
| 362 | attributes run at their usual speed. |
| 363 | |
| 364 | Finally, it's possible to constrain the list of attributes that can be |
| 365 | referenced on an object using the new :attr:`__slots__` class attribute. Python |
| 366 | objects are usually very dynamic; at any time it's possible to define a new |
| 367 | attribute on an instance by just doing ``obj.new_attr=1``. A new-style class |
| 368 | can define a class attribute named :attr:`__slots__` to limit the legal |
| 369 | attributes to a particular set of names. An example will make this clear:: |
| 370 | |
| 371 | >>> class C(object): |
| 372 | ... __slots__ = ('template', 'name') |
| 373 | ... |
| 374 | >>> obj = C() |
| 375 | >>> print obj.template |
| 376 | None |
| 377 | >>> obj.template = 'Test' |
| 378 | >>> print obj.template |
| 379 | Test |
| 380 | >>> obj.newattr = None |
| 381 | Traceback (most recent call last): |
| 382 | File "<stdin>", line 1, in ? |
| 383 | AttributeError: 'C' object has no attribute 'newattr' |
| 384 | |
| 385 | Note how you get an :exc:`AttributeError` on the attempt to assign to an |
| 386 | attribute not listed in :attr:`__slots__`. |
| 387 | |
| 388 | |
| 389 | .. _sect-rellinks: |
| 390 | |
| 391 | Related Links |
| 392 | ------------- |
| 393 | |
| 394 | This section has just been a quick overview of the new features, giving enough |
| 395 | of an explanation to start you programming, but many details have been |
| 396 | simplified or ignored. Where should you go to get a more complete picture? |
| 397 | |
| 398 | http://www.python.org/2.2/descrintro.html is a lengthy tutorial introduction to |
| 399 | the descriptor features, written by Guido van Rossum. If my description has |
| 400 | whetted your appetite, go read this tutorial next, because it goes into much |
| 401 | more detail about the new features while still remaining quite easy to read. |
| 402 | |
| 403 | Next, there are two relevant PEPs, :pep:`252` and :pep:`253`. :pep:`252` is |
| 404 | titled "Making Types Look More Like Classes", and covers the descriptor API. |
| 405 | :pep:`253` is titled "Subtyping Built-in Types", and describes the changes to |
| 406 | type objects that make it possible to subtype built-in objects. :pep:`253` is |
| 407 | the more complicated PEP of the two, and at a few points the necessary |
| 408 | explanations of types and meta-types may cause your head to explode. Both PEPs |
| 409 | were written and implemented by Guido van Rossum, with substantial assistance |
| 410 | from the rest of the Zope Corp. team. |
| 411 | |
| 412 | Finally, there's the ultimate authority: the source code. Most of the machinery |
| 413 | for the type handling is in :file:`Objects/typeobject.c`, but you should only |
| 414 | resort to it after all other avenues have been exhausted, including posting a |
| 415 | question to python-list or python-dev. |
| 416 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 417 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 418 | |
| 419 | |
| 420 | PEP 234: Iterators |
| 421 | ================== |
| 422 | |
| 423 | Another significant addition to 2.2 is an iteration interface at both the C and |
| 424 | Python levels. Objects can define how they can be looped over by callers. |
| 425 | |
| 426 | In Python versions up to 2.1, the usual way to make ``for item in obj`` work is |
| 427 | to define a :meth:`__getitem__` method that looks something like this:: |
| 428 | |
| 429 | def __getitem__(self, index): |
| 430 | return <next item> |
| 431 | |
| 432 | :meth:`__getitem__` is more properly used to define an indexing operation on an |
| 433 | object so that you can write ``obj[5]`` to retrieve the sixth element. It's a |
| 434 | bit misleading when you're using this only to support :keyword:`for` loops. |
| 435 | Consider some file-like object that wants to be looped over; the *index* |
| 436 | parameter is essentially meaningless, as the class probably assumes that a |
| 437 | series of :meth:`__getitem__` calls will be made with *index* incrementing by |
| 438 | one each time. In other words, the presence of the :meth:`__getitem__` method |
| 439 | doesn't mean that using ``file[5]`` to randomly access the sixth element will |
| 440 | work, though it really should. |
| 441 | |
| 442 | In Python 2.2, iteration can be implemented separately, and :meth:`__getitem__` |
| 443 | methods can be limited to classes that really do support random access. The |
| 444 | basic idea of iterators is simple. A new built-in function, :func:`iter(obj)` |
| 445 | or ``iter(C, sentinel)``, is used to get an iterator. :func:`iter(obj)` returns |
| 446 | an iterator for the object *obj*, while ``iter(C, sentinel)`` returns an |
| 447 | iterator that will invoke the callable object *C* until it returns *sentinel* to |
| 448 | signal that the iterator is done. |
| 449 | |
| 450 | Python classes can define an :meth:`__iter__` method, which should create and |
| 451 | return a new iterator for the object; if the object is its own iterator, this |
| 452 | method can just return ``self``. In particular, iterators will usually be their |
| 453 | own iterators. Extension types implemented in C can implement a :attr:`tp_iter` |
| 454 | function in order to return an iterator, and extension types that want to behave |
| 455 | as iterators can define a :attr:`tp_iternext` function. |
| 456 | |
| 457 | So, after all this, what do iterators actually do? They have one required |
| 458 | method, :meth:`next`, which takes no arguments and returns the next value. When |
| 459 | there are no more values to be returned, calling :meth:`next` should raise the |
| 460 | :exc:`StopIteration` exception. :: |
| 461 | |
| 462 | >>> L = [1,2,3] |
| 463 | >>> i = iter(L) |
| 464 | >>> print i |
| 465 | <iterator object at 0x8116870> |
| 466 | >>> i.next() |
| 467 | 1 |
| 468 | >>> i.next() |
| 469 | 2 |
| 470 | >>> i.next() |
| 471 | 3 |
| 472 | >>> i.next() |
| 473 | Traceback (most recent call last): |
| 474 | File "<stdin>", line 1, in ? |
| 475 | StopIteration |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 476 | >>> |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 477 | |
| 478 | In 2.2, Python's :keyword:`for` statement no longer expects a sequence; it |
| 479 | expects something for which :func:`iter` will return an iterator. For backward |
| 480 | compatibility and convenience, an iterator is automatically constructed for |
| 481 | sequences that don't implement :meth:`__iter__` or a :attr:`tp_iter` slot, so |
| 482 | ``for i in [1,2,3]`` will still work. Wherever the Python interpreter loops |
| 483 | over a sequence, it's been changed to use the iterator protocol. This means you |
| 484 | can do things like this:: |
| 485 | |
| 486 | >>> L = [1,2,3] |
| 487 | >>> i = iter(L) |
| 488 | >>> a,b,c = i |
| 489 | >>> a,b,c |
| 490 | (1, 2, 3) |
| 491 | |
| 492 | Iterator support has been added to some of Python's basic types. Calling |
| 493 | :func:`iter` on a dictionary will return an iterator which loops over its keys:: |
| 494 | |
| 495 | >>> m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6, |
| 496 | ... 'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12} |
| 497 | >>> for key in m: print key, m[key] |
| 498 | ... |
| 499 | Mar 3 |
| 500 | Feb 2 |
| 501 | Aug 8 |
| 502 | Sep 9 |
| 503 | May 5 |
| 504 | Jun 6 |
| 505 | Jul 7 |
| 506 | Jan 1 |
| 507 | Apr 4 |
| 508 | Nov 11 |
| 509 | Dec 12 |
| 510 | Oct 10 |
| 511 | |
| 512 | That's just the default behaviour. If you want to iterate over keys, values, or |
| 513 | key/value pairs, you can explicitly call the :meth:`iterkeys`, |
| 514 | :meth:`itervalues`, or :meth:`iteritems` methods to get an appropriate iterator. |
| 515 | In a minor related change, the :keyword:`in` operator now works on dictionaries, |
| 516 | so ``key in dict`` is now equivalent to ``dict.has_key(key)``. |
| 517 | |
| 518 | Files also provide an iterator, which calls the :meth:`readline` method until |
| 519 | there are no more lines in the file. This means you can now read each line of a |
| 520 | file using code like this:: |
| 521 | |
| 522 | for line in file: |
| 523 | # do something for each line |
| 524 | ... |
| 525 | |
| 526 | Note that you can only go forward in an iterator; there's no way to get the |
| 527 | previous element, reset the iterator, or make a copy of it. An iterator object |
| 528 | could provide such additional capabilities, but the iterator protocol only |
| 529 | requires a :meth:`next` method. |
| 530 | |
| 531 | |
| 532 | .. seealso:: |
| 533 | |
| 534 | :pep:`234` - Iterators |
| 535 | Written by Ka-Ping Yee and GvR; implemented by the Python Labs crew, mostly by |
| 536 | GvR and Tim Peters. |
| 537 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 538 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 539 | |
| 540 | |
| 541 | PEP 255: Simple Generators |
| 542 | ========================== |
| 543 | |
| 544 | Generators are another new feature, one that interacts with the introduction of |
| 545 | iterators. |
| 546 | |
| 547 | You're doubtless familiar with how function calls work in Python or C. When you |
| 548 | call a function, it gets a private namespace where its local variables are |
| 549 | created. When the function reaches a :keyword:`return` statement, the local |
| 550 | variables are destroyed and the resulting value is returned to the caller. A |
| 551 | later call to the same function will get a fresh new set of local variables. |
| 552 | But, what if the local variables weren't thrown away on exiting a function? |
| 553 | What if you could later resume the function where it left off? This is what |
| 554 | generators provide; they can be thought of as resumable functions. |
| 555 | |
| 556 | Here's the simplest example of a generator function:: |
| 557 | |
| 558 | def generate_ints(N): |
| 559 | for i in range(N): |
| 560 | yield i |
| 561 | |
| 562 | A new keyword, :keyword:`yield`, was introduced for generators. Any function |
| 563 | containing a :keyword:`yield` statement is a generator function; this is |
| 564 | detected by Python's bytecode compiler which compiles the function specially as |
| 565 | a result. Because a new keyword was introduced, generators must be explicitly |
| 566 | enabled in a module by including a ``from __future__ import generators`` |
| 567 | statement near the top of the module's source code. In Python 2.3 this |
| 568 | statement will become unnecessary. |
| 569 | |
| 570 | When you call a generator function, it doesn't return a single value; instead it |
| 571 | returns a generator object that supports the iterator protocol. On executing |
| 572 | the :keyword:`yield` statement, the generator outputs the value of ``i``, |
| 573 | similar to a :keyword:`return` statement. The big difference between |
| 574 | :keyword:`yield` and a :keyword:`return` statement is that on reaching a |
| 575 | :keyword:`yield` the generator's state of execution is suspended and local |
| 576 | variables are preserved. On the next call to the generator's ``next()`` method, |
| 577 | the function will resume executing immediately after the :keyword:`yield` |
| 578 | statement. (For complicated reasons, the :keyword:`yield` statement isn't |
| 579 | allowed inside the :keyword:`try` block of a :keyword:`try`...\ |
| 580 | :keyword:`finally` statement; read :pep:`255` for a full explanation of the |
| 581 | interaction between :keyword:`yield` and exceptions.) |
| 582 | |
| 583 | Here's a sample usage of the :func:`generate_ints` generator:: |
| 584 | |
| 585 | >>> gen = generate_ints(3) |
| 586 | >>> gen |
| 587 | <generator object at 0x8117f90> |
| 588 | >>> gen.next() |
| 589 | 0 |
| 590 | >>> gen.next() |
| 591 | 1 |
| 592 | >>> gen.next() |
| 593 | 2 |
| 594 | >>> gen.next() |
| 595 | Traceback (most recent call last): |
| 596 | File "<stdin>", line 1, in ? |
| 597 | File "<stdin>", line 2, in generate_ints |
| 598 | StopIteration |
| 599 | |
| 600 | You could equally write ``for i in generate_ints(5)``, or ``a,b,c = |
| 601 | generate_ints(3)``. |
| 602 | |
| 603 | Inside a generator function, the :keyword:`return` statement can only be used |
| 604 | without a value, and signals the end of the procession of values; afterwards the |
| 605 | generator cannot return any further values. :keyword:`return` with a value, such |
| 606 | as ``return 5``, is a syntax error inside a generator function. The end of the |
| 607 | generator's results can also be indicated by raising :exc:`StopIteration` |
| 608 | manually, or by just letting the flow of execution fall off the bottom of the |
| 609 | function. |
| 610 | |
| 611 | You could achieve the effect of generators manually by writing your own class |
| 612 | and storing all the local variables of the generator as instance variables. For |
| 613 | example, returning a list of integers could be done by setting ``self.count`` to |
| 614 | 0, and having the :meth:`next` method increment ``self.count`` and return it. |
| 615 | However, for a moderately complicated generator, writing a corresponding class |
| 616 | would be much messier. :file:`Lib/test/test_generators.py` contains a number of |
| 617 | more interesting examples. The simplest one implements an in-order traversal of |
| 618 | a tree using generators recursively. :: |
| 619 | |
| 620 | # A recursive generator that generates Tree leaves in in-order. |
| 621 | def inorder(t): |
| 622 | if t: |
| 623 | for x in inorder(t.left): |
| 624 | yield x |
| 625 | yield t.label |
| 626 | for x in inorder(t.right): |
| 627 | yield x |
| 628 | |
| 629 | Two other examples in :file:`Lib/test/test_generators.py` produce solutions for |
| 630 | the N-Queens problem (placing $N$ queens on an $NxN$ chess board so that no |
| 631 | queen threatens another) and the Knight's Tour (a route that takes a knight to |
| 632 | every square of an $NxN$ chessboard without visiting any square twice). |
| 633 | |
| 634 | The idea of generators comes from other programming languages, especially Icon |
| 635 | (http://www.cs.arizona.edu/icon/), where the idea of generators is central. In |
| 636 | Icon, every expression and function call behaves like a generator. One example |
| 637 | from "An Overview of the Icon Programming Language" at |
| 638 | http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks |
| 639 | like:: |
| 640 | |
| 641 | sentence := "Store it in the neighboring harbor" |
| 642 | if (i := find("or", sentence)) > 5 then write(i) |
| 643 | |
| 644 | In Icon the :func:`find` function returns the indexes at which the substring |
| 645 | "or" is found: 3, 23, 33. In the :keyword:`if` statement, ``i`` is first |
| 646 | assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon |
| 647 | retries it with the second value of 23. 23 is greater than 5, so the comparison |
| 648 | now succeeds, and the code prints the value 23 to the screen. |
| 649 | |
| 650 | Python doesn't go nearly as far as Icon in adopting generators as a central |
| 651 | concept. Generators are considered a new part of the core Python language, but |
| 652 | learning or using them isn't compulsory; if they don't solve any problems that |
| 653 | you have, feel free to ignore them. One novel feature of Python's interface as |
| 654 | compared to Icon's is that a generator's state is represented as a concrete |
| 655 | object (the iterator) that can be passed around to other functions or stored in |
| 656 | a data structure. |
| 657 | |
| 658 | |
| 659 | .. seealso:: |
| 660 | |
| 661 | :pep:`255` - Simple Generators |
| 662 | Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland. Implemented mostly |
| 663 | by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew. |
| 664 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 665 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 666 | |
| 667 | |
| 668 | PEP 237: Unifying Long Integers and Integers |
| 669 | ============================================ |
| 670 | |
| 671 | In recent versions, the distinction between regular integers, which are 32-bit |
| 672 | values on most machines, and long integers, which can be of arbitrary size, was |
| 673 | becoming an annoyance. For example, on platforms that support files larger than |
| 674 | ``2**32`` bytes, the :meth:`tell` method of file objects has to return a long |
| 675 | integer. However, there were various bits of Python that expected plain integers |
| 676 | and would raise an error if a long integer was provided instead. For example, |
| 677 | in Python 1.5, only regular integers could be used as a slice index, and |
| 678 | ``'abc'[1L:]`` would raise a :exc:`TypeError` exception with the message 'slice |
| 679 | index must be int'. |
| 680 | |
| 681 | Python 2.2 will shift values from short to long integers as required. The 'L' |
| 682 | suffix is no longer needed to indicate a long integer literal, as now the |
| 683 | compiler will choose the appropriate type. (Using the 'L' suffix will be |
| 684 | discouraged in future 2.x versions of Python, triggering a warning in Python |
| 685 | 2.4, and probably dropped in Python 3.0.) Many operations that used to raise an |
| 686 | :exc:`OverflowError` will now return a long integer as their result. For |
| 687 | example:: |
| 688 | |
| 689 | >>> 1234567890123 |
| 690 | 1234567890123L |
| 691 | >>> 2 ** 64 |
| 692 | 18446744073709551616L |
| 693 | |
| 694 | In most cases, integers and long integers will now be treated identically. You |
| 695 | can still distinguish them with the :func:`type` built-in function, but that's |
| 696 | rarely needed. |
| 697 | |
| 698 | |
| 699 | .. seealso:: |
| 700 | |
| 701 | :pep:`237` - Unifying Long Integers and Integers |
| 702 | Written by Moshe Zadka and Guido van Rossum. Implemented mostly by Guido van |
| 703 | Rossum. |
| 704 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 705 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 706 | |
| 707 | |
| 708 | PEP 238: Changing the Division Operator |
| 709 | ======================================= |
| 710 | |
| 711 | The most controversial change in Python 2.2 heralds the start of an effort to |
| 712 | fix an old design flaw that's been in Python from the beginning. Currently |
| 713 | Python's division operator, ``/``, behaves like C's division operator when |
| 714 | presented with two integer arguments: it returns an integer result that's |
| 715 | truncated down when there would be a fractional part. For example, ``3/2`` is |
| 716 | 1, not 1.5, and ``(-1)/2`` is -1, not -0.5. This means that the results of |
Benjamin Peterson | f10a79a | 2008-10-11 00:49:57 +0000 | [diff] [blame] | 717 | division can vary unexpectedly depending on the type of the two operands and |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 718 | because Python is dynamically typed, it can be difficult to determine the |
| 719 | possible types of the operands. |
| 720 | |
| 721 | (The controversy is over whether this is *really* a design flaw, and whether |
| 722 | it's worth breaking existing code to fix this. It's caused endless discussions |
| 723 | on python-dev, and in July 2001 erupted into an storm of acidly sarcastic |
| 724 | postings on :newsgroup:`comp.lang.python`. I won't argue for either side here |
| 725 | and will stick to describing what's implemented in 2.2. Read :pep:`238` for a |
| 726 | summary of arguments and counter-arguments.) |
| 727 | |
| 728 | Because this change might break code, it's being introduced very gradually. |
| 729 | Python 2.2 begins the transition, but the switch won't be complete until Python |
| 730 | 3.0. |
| 731 | |
| 732 | First, I'll borrow some terminology from :pep:`238`. "True division" is the |
| 733 | division that most non-programmers are familiar with: 3/2 is 1.5, 1/4 is 0.25, |
| 734 | and so forth. "Floor division" is what Python's ``/`` operator currently does |
| 735 | when given integer operands; the result is the floor of the value returned by |
| 736 | true division. "Classic division" is the current mixed behaviour of ``/``; it |
| 737 | returns the result of floor division when the operands are integers, and returns |
| 738 | the result of true division when one of the operands is a floating-point number. |
| 739 | |
| 740 | Here are the changes 2.2 introduces: |
| 741 | |
| 742 | * A new operator, ``//``, is the floor division operator. (Yes, we know it looks |
| 743 | like C++'s comment symbol.) ``//`` *always* performs floor division no matter |
| 744 | what the types of its operands are, so ``1 // 2`` is 0 and ``1.0 // 2.0`` is |
| 745 | also 0.0. |
| 746 | |
| 747 | ``//`` is always available in Python 2.2; you don't need to enable it using a |
| 748 | ``__future__`` statement. |
| 749 | |
| 750 | * By including a ``from __future__ import division`` in a module, the ``/`` |
| 751 | operator will be changed to return the result of true division, so ``1/2`` is |
| 752 | 0.5. Without the ``__future__`` statement, ``/`` still means classic division. |
| 753 | The default meaning of ``/`` will not change until Python 3.0. |
| 754 | |
| 755 | * Classes can define methods called :meth:`__truediv__` and :meth:`__floordiv__` |
| 756 | to overload the two division operators. At the C level, there are also slots in |
| 757 | the :ctype:`PyNumberMethods` structure so extension types can define the two |
| 758 | operators. |
| 759 | |
| 760 | * Python 2.2 supports some command-line arguments for testing whether code will |
| 761 | works with the changed division semantics. Running python with :option:`-Q |
| 762 | warn` will cause a warning to be issued whenever division is applied to two |
| 763 | integers. You can use this to find code that's affected by the change and fix |
| 764 | it. By default, Python 2.2 will simply perform classic division without a |
| 765 | warning; the warning will be turned on by default in Python 2.3. |
| 766 | |
| 767 | |
| 768 | .. seealso:: |
| 769 | |
| 770 | :pep:`238` - Changing the Division Operator |
| 771 | Written by Moshe Zadka and Guido van Rossum. Implemented by Guido van Rossum.. |
| 772 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 773 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 774 | |
| 775 | |
| 776 | Unicode Changes |
| 777 | =============== |
| 778 | |
| 779 | Python's Unicode support has been enhanced a bit in 2.2. Unicode strings are |
| 780 | usually stored as UCS-2, as 16-bit unsigned integers. Python 2.2 can also be |
| 781 | compiled to use UCS-4, 32-bit unsigned integers, as its internal encoding by |
| 782 | supplying :option:`--enable-unicode=ucs4` to the configure script. (It's also |
| 783 | possible to specify :option:`--disable-unicode` to completely disable Unicode |
| 784 | support.) |
| 785 | |
| 786 | When built to use UCS-4 (a "wide Python"), the interpreter can natively handle |
| 787 | Unicode characters from U+000000 to U+110000, so the range of legal values for |
| 788 | the :func:`unichr` function is expanded accordingly. Using an interpreter |
| 789 | compiled to use UCS-2 (a "narrow Python"), values greater than 65535 will still |
| 790 | cause :func:`unichr` to raise a :exc:`ValueError` exception. This is all |
| 791 | described in :pep:`261`, "Support for 'wide' Unicode characters"; consult it for |
| 792 | further details. |
| 793 | |
| 794 | Another change is simpler to explain. Since their introduction, Unicode strings |
| 795 | have supported an :meth:`encode` method to convert the string to a selected |
| 796 | encoding such as UTF-8 or Latin-1. A symmetric :meth:`decode([*encoding*])` |
| 797 | method has been added to 8-bit strings (though not to Unicode strings) in 2.2. |
| 798 | :meth:`decode` assumes that the string is in the specified encoding and decodes |
| 799 | it, returning whatever is returned by the codec. |
| 800 | |
| 801 | Using this new feature, codecs have been added for tasks not directly related to |
| 802 | Unicode. For example, codecs have been added for uu-encoding, MIME's base64 |
| 803 | encoding, and compression with the :mod:`zlib` module:: |
| 804 | |
| 805 | >>> s = """Here is a lengthy piece of redundant, overly verbose, |
| 806 | ... and repetitive text. |
| 807 | ... """ |
| 808 | >>> data = s.encode('zlib') |
| 809 | >>> data |
| 810 | 'x\x9c\r\xc9\xc1\r\x80 \x10\x04\xc0?Ul...' |
| 811 | >>> data.decode('zlib') |
| 812 | 'Here is a lengthy piece of redundant, overly verbose,\nand repetitive text.\n' |
| 813 | >>> print s.encode('uu') |
| 814 | begin 666 <data> |
| 815 | M2&5R92!I<R!A(&QE;F=T:'D@<&EE8V4@;V8@<F5D=6YD86YT+"!O=F5R;'D@ |
| 816 | >=F5R8F]S92P*86YD(')E<&5T:71I=F4@=&5X="X* |
| 817 | |
| 818 | end |
| 819 | >>> "sheesh".encode('rot-13') |
| 820 | 'furrfu' |
| 821 | |
| 822 | To convert a class instance to Unicode, a :meth:`__unicode__` method can be |
| 823 | defined by a class, analogous to :meth:`__str__`. |
| 824 | |
| 825 | :meth:`encode`, :meth:`decode`, and :meth:`__unicode__` were implemented by |
| 826 | Marc-André Lemburg. The changes to support using UCS-4 internally were |
| 827 | implemented by Fredrik Lundh and Martin von Löwis. |
| 828 | |
| 829 | |
| 830 | .. seealso:: |
| 831 | |
| 832 | :pep:`261` - Support for 'wide' Unicode characters |
| 833 | Written by Paul Prescod. |
| 834 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 835 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 836 | |
| 837 | |
| 838 | PEP 227: Nested Scopes |
| 839 | ====================== |
| 840 | |
| 841 | In Python 2.1, statically nested scopes were added as an optional feature, to be |
| 842 | enabled by a ``from __future__ import nested_scopes`` directive. In 2.2 nested |
| 843 | scopes no longer need to be specially enabled, and are now always present. The |
| 844 | rest of this section is a copy of the description of nested scopes from my |
| 845 | "What's New in Python 2.1" document; if you read it when 2.1 came out, you can |
| 846 | skip the rest of this section. |
| 847 | |
| 848 | The largest change introduced in Python 2.1, and made complete in 2.2, is to |
| 849 | Python's scoping rules. In Python 2.0, at any given time there are at most |
| 850 | three namespaces used to look up variable names: local, module-level, and the |
| 851 | built-in namespace. This often surprised people because it didn't match their |
| 852 | intuitive expectations. For example, a nested recursive function definition |
| 853 | doesn't work:: |
| 854 | |
| 855 | def f(): |
| 856 | ... |
| 857 | def g(value): |
| 858 | ... |
| 859 | return g(value-1) + 1 |
| 860 | ... |
| 861 | |
| 862 | The function :func:`g` will always raise a :exc:`NameError` exception, because |
| 863 | the binding of the name ``g`` isn't in either its local namespace or in the |
| 864 | module-level namespace. This isn't much of a problem in practice (how often do |
| 865 | you recursively define interior functions like this?), but this also made using |
| 866 | the :keyword:`lambda` statement clumsier, and this was a problem in practice. |
| 867 | In code which uses :keyword:`lambda` you can often find local variables being |
| 868 | copied by passing them as the default values of arguments. :: |
| 869 | |
| 870 | def find(self, name): |
| 871 | "Return list of any entries equal to 'name'" |
| 872 | L = filter(lambda x, name=name: x == name, |
| 873 | self.list_attribute) |
| 874 | return L |
| 875 | |
| 876 | The readability of Python code written in a strongly functional style suffers |
| 877 | greatly as a result. |
| 878 | |
| 879 | The most significant change to Python 2.2 is that static scoping has been added |
| 880 | to the language to fix this problem. As a first effect, the ``name=name`` |
| 881 | default argument is now unnecessary in the above example. Put simply, when a |
| 882 | given variable name is not assigned a value within a function (by an assignment, |
| 883 | or the :keyword:`def`, :keyword:`class`, or :keyword:`import` statements), |
| 884 | references to the variable will be looked up in the local namespace of the |
| 885 | enclosing scope. A more detailed explanation of the rules, and a dissection of |
| 886 | the implementation, can be found in the PEP. |
| 887 | |
| 888 | This change may cause some compatibility problems for code where the same |
| 889 | variable name is used both at the module level and as a local variable within a |
| 890 | function that contains further function definitions. This seems rather unlikely |
| 891 | though, since such code would have been pretty confusing to read in the first |
| 892 | place. |
| 893 | |
| 894 | One side effect of the change is that the ``from module import *`` and |
| 895 | :keyword:`exec` statements have been made illegal inside a function scope under |
| 896 | certain conditions. The Python reference manual has said all along that ``from |
| 897 | module import *`` is only legal at the top level of a module, but the CPython |
| 898 | interpreter has never enforced this before. As part of the implementation of |
| 899 | nested scopes, the compiler which turns Python source into bytecodes has to |
| 900 | generate different code to access variables in a containing scope. ``from |
| 901 | module import *`` and :keyword:`exec` make it impossible for the compiler to |
| 902 | figure this out, because they add names to the local namespace that are |
| 903 | unknowable at compile time. Therefore, if a function contains function |
| 904 | definitions or :keyword:`lambda` expressions with free variables, the compiler |
| 905 | will flag this by raising a :exc:`SyntaxError` exception. |
| 906 | |
| 907 | To make the preceding explanation a bit clearer, here's an example:: |
| 908 | |
| 909 | x = 1 |
| 910 | def f(): |
| 911 | # The next line is a syntax error |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 912 | exec 'x=2' |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 913 | def g(): |
| 914 | return x |
| 915 | |
| 916 | Line 4 containing the :keyword:`exec` statement is a syntax error, since |
| 917 | :keyword:`exec` would define a new local variable named ``x`` whose value should |
| 918 | be accessed by :func:`g`. |
| 919 | |
| 920 | This shouldn't be much of a limitation, since :keyword:`exec` is rarely used in |
| 921 | most Python code (and when it is used, it's often a sign of a poor design |
| 922 | anyway). |
| 923 | |
| 924 | |
| 925 | .. seealso:: |
| 926 | |
| 927 | :pep:`227` - Statically Nested Scopes |
| 928 | Written and implemented by Jeremy Hylton. |
| 929 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 930 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 931 | |
| 932 | |
| 933 | New and Improved Modules |
| 934 | ======================== |
| 935 | |
| 936 | * The :mod:`xmlrpclib` module was contributed to the standard library by Fredrik |
| 937 | Lundh, providing support for writing XML-RPC clients. XML-RPC is a simple |
| 938 | remote procedure call protocol built on top of HTTP and XML. For example, the |
| 939 | following snippet retrieves a list of RSS channels from the O'Reilly Network, |
| 940 | and then lists the recent headlines for one channel:: |
| 941 | |
| 942 | import xmlrpclib |
| 943 | s = xmlrpclib.Server( |
| 944 | 'http://www.oreillynet.com/meerkat/xml-rpc/server.php') |
| 945 | channels = s.meerkat.getChannels() |
| 946 | # channels is a list of dictionaries, like this: |
| 947 | # [{'id': 4, 'title': 'Freshmeat Daily News'} |
| 948 | # {'id': 190, 'title': '32Bits Online'}, |
| 949 | # {'id': 4549, 'title': '3DGamers'}, ... ] |
| 950 | |
| 951 | # Get the items for one channel |
| 952 | items = s.meerkat.getItems( {'channel': 4} ) |
| 953 | |
| 954 | # 'items' is another list of dictionaries, like this: |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 955 | # [{'link': 'http://freshmeat.net/releases/52719/', |
| 956 | # 'description': 'A utility which converts HTML to XSL FO.', |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 957 | # 'title': 'html2fo 0.3 (Default)'}, ... ] |
| 958 | |
| 959 | The :mod:`SimpleXMLRPCServer` module makes it easy to create straightforward |
| 960 | XML-RPC servers. See http://www.xmlrpc.com/ for more information about XML-RPC. |
| 961 | |
| 962 | * The new :mod:`hmac` module implements the HMAC algorithm described by |
| 963 | :rfc:`2104`. (Contributed by Gerhard Häring.) |
| 964 | |
| 965 | * Several functions that originally returned lengthy tuples now return pseudo- |
| 966 | sequences that still behave like tuples but also have mnemonic attributes such |
| 967 | as memberst_mtime or :attr:`tm_year`. The enhanced functions include |
| 968 | :func:`stat`, :func:`fstat`, :func:`statvfs`, and :func:`fstatvfs` in the |
| 969 | :mod:`os` module, and :func:`localtime`, :func:`gmtime`, and :func:`strptime` in |
| 970 | the :mod:`time` module. |
| 971 | |
| 972 | For example, to obtain a file's size using the old tuples, you'd end up writing |
| 973 | something like ``file_size = os.stat(filename)[stat.ST_SIZE]``, but now this can |
| 974 | be written more clearly as ``file_size = os.stat(filename).st_size``. |
| 975 | |
| 976 | The original patch for this feature was contributed by Nick Mathewson. |
| 977 | |
| 978 | * The Python profiler has been extensively reworked and various errors in its |
| 979 | output have been corrected. (Contributed by Fred L. Drake, Jr. and Tim Peters.) |
| 980 | |
| 981 | * The :mod:`socket` module can be compiled to support IPv6; specify the |
| 982 | :option:`--enable-ipv6` option to Python's configure script. (Contributed by |
| 983 | Jun-ichiro "itojun" Hagino.) |
| 984 | |
| 985 | * Two new format characters were added to the :mod:`struct` module for 64-bit |
| 986 | integers on platforms that support the C :ctype:`long long` type. ``q`` is for |
| 987 | a signed 64-bit integer, and ``Q`` is for an unsigned one. The value is |
| 988 | returned in Python's long integer type. (Contributed by Tim Peters.) |
| 989 | |
| 990 | * In the interpreter's interactive mode, there's a new built-in function |
| 991 | :func:`help` that uses the :mod:`pydoc` module introduced in Python 2.1 to |
| 992 | provide interactive help. ``help(object)`` displays any available help text |
| 993 | about *object*. :func:`help` with no argument puts you in an online help |
| 994 | utility, where you can enter the names of functions, classes, or modules to read |
| 995 | their help text. (Contributed by Guido van Rossum, using Ka-Ping Yee's |
| 996 | :mod:`pydoc` module.) |
| 997 | |
| 998 | * Various bugfixes and performance improvements have been made to the SRE engine |
| 999 | underlying the :mod:`re` module. For example, the :func:`re.sub` and |
| 1000 | :func:`re.split` functions have been rewritten in C. Another contributed patch |
| 1001 | speeds up certain Unicode character ranges by a factor of two, and a new |
| 1002 | :meth:`finditer` method that returns an iterator over all the non-overlapping |
| 1003 | matches in a given string. (SRE is maintained by Fredrik Lundh. The |
| 1004 | BIGCHARSET patch was contributed by Martin von Löwis.) |
| 1005 | |
| 1006 | * The :mod:`smtplib` module now supports :rfc:`2487`, "Secure SMTP over TLS", so |
| 1007 | it's now possible to encrypt the SMTP traffic between a Python program and the |
| 1008 | mail transport agent being handed a message. :mod:`smtplib` also supports SMTP |
| 1009 | authentication. (Contributed by Gerhard Häring.) |
| 1010 | |
| 1011 | * The :mod:`imaplib` module, maintained by Piers Lauder, has support for several |
| 1012 | new extensions: the NAMESPACE extension defined in :rfc:`2342`, SORT, GETACL and |
| 1013 | SETACL. (Contributed by Anthony Baxter and Michel Pelletier.) |
| 1014 | |
| 1015 | * The :mod:`rfc822` module's parsing of email addresses is now compliant with |
| 1016 | :rfc:`2822`, an update to :rfc:`822`. (The module's name is *not* going to be |
| 1017 | changed to ``rfc2822``.) A new package, :mod:`email`, has also been added for |
| 1018 | parsing and generating e-mail messages. (Contributed by Barry Warsaw, and |
| 1019 | arising out of his work on Mailman.) |
| 1020 | |
| 1021 | * The :mod:`difflib` module now contains a new :class:`Differ` class for |
| 1022 | producing human-readable lists of changes (a "delta") between two sequences of |
| 1023 | lines of text. There are also two generator functions, :func:`ndiff` and |
| 1024 | :func:`restore`, which respectively return a delta from two sequences, or one of |
| 1025 | the original sequences from a delta. (Grunt work contributed by David Goodger, |
| 1026 | from ndiff.py code by Tim Peters who then did the generatorization.) |
| 1027 | |
| 1028 | * New constants :const:`ascii_letters`, :const:`ascii_lowercase`, and |
| 1029 | :const:`ascii_uppercase` were added to the :mod:`string` module. There were |
| 1030 | several modules in the standard library that used :const:`string.letters` to |
| 1031 | mean the ranges A-Za-z, but that assumption is incorrect when locales are in |
| 1032 | use, because :const:`string.letters` varies depending on the set of legal |
| 1033 | characters defined by the current locale. The buggy modules have all been fixed |
| 1034 | to use :const:`ascii_letters` instead. (Reported by an unknown person; fixed by |
| 1035 | Fred L. Drake, Jr.) |
| 1036 | |
| 1037 | * The :mod:`mimetypes` module now makes it easier to use alternative MIME-type |
| 1038 | databases by the addition of a :class:`MimeTypes` class, which takes a list of |
| 1039 | filenames to be parsed. (Contributed by Fred L. Drake, Jr.) |
| 1040 | |
| 1041 | * A :class:`Timer` class was added to the :mod:`threading` module that allows |
| 1042 | scheduling an activity to happen at some future time. (Contributed by Itamar |
| 1043 | Shtull-Trauring.) |
| 1044 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 1045 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1046 | |
| 1047 | |
| 1048 | Interpreter Changes and Fixes |
| 1049 | ============================= |
| 1050 | |
| 1051 | Some of the changes only affect people who deal with the Python interpreter at |
| 1052 | the C level because they're writing Python extension modules, embedding the |
| 1053 | interpreter, or just hacking on the interpreter itself. If you only write Python |
| 1054 | code, none of the changes described here will affect you very much. |
| 1055 | |
| 1056 | * Profiling and tracing functions can now be implemented in C, which can operate |
| 1057 | at much higher speeds than Python-based functions and should reduce the overhead |
| 1058 | of profiling and tracing. This will be of interest to authors of development |
| 1059 | environments for Python. Two new C functions were added to Python's API, |
| 1060 | :cfunc:`PyEval_SetProfile` and :cfunc:`PyEval_SetTrace`. The existing |
| 1061 | :func:`sys.setprofile` and :func:`sys.settrace` functions still exist, and have |
| 1062 | simply been changed to use the new C-level interface. (Contributed by Fred L. |
| 1063 | Drake, Jr.) |
| 1064 | |
| 1065 | * Another low-level API, primarily of interest to implementors of Python |
| 1066 | debuggers and development tools, was added. :cfunc:`PyInterpreterState_Head` and |
| 1067 | :cfunc:`PyInterpreterState_Next` let a caller walk through all the existing |
| 1068 | interpreter objects; :cfunc:`PyInterpreterState_ThreadHead` and |
| 1069 | :cfunc:`PyThreadState_Next` allow looping over all the thread states for a given |
| 1070 | interpreter. (Contributed by David Beazley.) |
| 1071 | |
| 1072 | * The C-level interface to the garbage collector has been changed to make it |
| 1073 | easier to write extension types that support garbage collection and to debug |
| 1074 | misuses of the functions. Various functions have slightly different semantics, |
| 1075 | so a bunch of functions had to be renamed. Extensions that use the old API will |
| 1076 | still compile but will *not* participate in garbage collection, so updating them |
| 1077 | for 2.2 should be considered fairly high priority. |
| 1078 | |
| 1079 | To upgrade an extension module to the new API, perform the following steps: |
| 1080 | |
| 1081 | * Rename :cfunc:`Py_TPFLAGS_GC` to :cfunc:`PyTPFLAGS_HAVE_GC`. |
| 1082 | |
| 1083 | * Use :cfunc:`PyObject_GC_New` or :cfunc:`PyObject_GC_NewVar` to allocate |
| 1084 | objects, and :cfunc:`PyObject_GC_Del` to deallocate them. |
| 1085 | |
| 1086 | * Rename :cfunc:`PyObject_GC_Init` to :cfunc:`PyObject_GC_Track` and |
| 1087 | :cfunc:`PyObject_GC_Fini` to :cfunc:`PyObject_GC_UnTrack`. |
| 1088 | |
| 1089 | * Remove :cfunc:`PyGC_HEAD_SIZE` from object size calculations. |
| 1090 | |
| 1091 | * Remove calls to :cfunc:`PyObject_AS_GC` and :cfunc:`PyObject_FROM_GC`. |
| 1092 | |
| 1093 | * A new ``et`` format sequence was added to :cfunc:`PyArg_ParseTuple`; ``et`` |
| 1094 | takes both a parameter and an encoding name, and converts the parameter to the |
| 1095 | given encoding if the parameter turns out to be a Unicode string, or leaves it |
| 1096 | alone if it's an 8-bit string, assuming it to already be in the desired |
| 1097 | encoding. This differs from the ``es`` format character, which assumes that |
| 1098 | 8-bit strings are in Python's default ASCII encoding and converts them to the |
| 1099 | specified new encoding. (Contributed by M.-A. Lemburg, and used for the MBCS |
| 1100 | support on Windows described in the following section.) |
| 1101 | |
| 1102 | * A different argument parsing function, :cfunc:`PyArg_UnpackTuple`, has been |
| 1103 | added that's simpler and presumably faster. Instead of specifying a format |
| 1104 | string, the caller simply gives the minimum and maximum number of arguments |
| 1105 | expected, and a set of pointers to :ctype:`PyObject\*` variables that will be |
| 1106 | filled in with argument values. |
| 1107 | |
| 1108 | * Two new flags :const:`METH_NOARGS` and :const:`METH_O` are available in method |
| 1109 | definition tables to simplify implementation of methods with no arguments or a |
| 1110 | single untyped argument. Calling such methods is more efficient than calling a |
| 1111 | corresponding method that uses :const:`METH_VARARGS`. Also, the old |
| 1112 | :const:`METH_OLDARGS` style of writing C methods is now officially deprecated. |
| 1113 | |
| 1114 | * Two new wrapper functions, :cfunc:`PyOS_snprintf` and :cfunc:`PyOS_vsnprintf` |
| 1115 | were added to provide cross-platform implementations for the relatively new |
| 1116 | :cfunc:`snprintf` and :cfunc:`vsnprintf` C lib APIs. In contrast to the standard |
| 1117 | :cfunc:`sprintf` and :cfunc:`vsprintf` functions, the Python versions check the |
| 1118 | bounds of the buffer used to protect against buffer overruns. (Contributed by |
| 1119 | M.-A. Lemburg.) |
| 1120 | |
| 1121 | * The :cfunc:`_PyTuple_Resize` function has lost an unused parameter, so now it |
| 1122 | takes 2 parameters instead of 3. The third argument was never used, and can |
| 1123 | simply be discarded when porting code from earlier versions to Python 2.2. |
| 1124 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 1125 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1126 | |
| 1127 | |
| 1128 | Other Changes and Fixes |
| 1129 | ======================= |
| 1130 | |
| 1131 | As usual there were a bunch of other improvements and bugfixes scattered |
| 1132 | throughout the source tree. A search through the CVS change logs finds there |
| 1133 | were 527 patches applied and 683 bugs fixed between Python 2.1 and 2.2; 2.2.1 |
| 1134 | applied 139 patches and fixed 143 bugs; 2.2.2 applied 106 patches and fixed 82 |
| 1135 | bugs. These figures are likely to be underestimates. |
| 1136 | |
| 1137 | Some of the more notable changes are: |
| 1138 | |
| 1139 | * The code for the MacOS port for Python, maintained by Jack Jansen, is now kept |
| 1140 | in the main Python CVS tree, and many changes have been made to support MacOS X. |
| 1141 | |
| 1142 | The most significant change is the ability to build Python as a framework, |
| 1143 | enabled by supplying the :option:`--enable-framework` option to the configure |
| 1144 | script when compiling Python. According to Jack Jansen, "This installs a self- |
| 1145 | contained Python installation plus the OS X framework "glue" into |
| 1146 | :file:`/Library/Frameworks/Python.framework` (or another location of choice). |
| 1147 | For now there is little immediate added benefit to this (actually, there is the |
| 1148 | disadvantage that you have to change your PATH to be able to find Python), but |
| 1149 | it is the basis for creating a full-blown Python application, porting the |
| 1150 | MacPython IDE, possibly using Python as a standard OSA scripting language and |
| 1151 | much more." |
| 1152 | |
| 1153 | Most of the MacPython toolbox modules, which interface to MacOS APIs such as |
| 1154 | windowing, QuickTime, scripting, etc. have been ported to OS X, but they've been |
| 1155 | left commented out in :file:`setup.py`. People who want to experiment with |
| 1156 | these modules can uncomment them manually. |
| 1157 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 1158 | .. Jack's original comments: |
| 1159 | The main change is the possibility to build Python as a |
| 1160 | framework. This installs a self-contained Python installation plus the |
| 1161 | OSX framework "glue" into /Library/Frameworks/Python.framework (or |
| 1162 | another location of choice). For now there is little immedeate added |
| 1163 | benefit to this (actually, there is the disadvantage that you have to |
| 1164 | change your PATH to be able to find Python), but it is the basis for |
| 1165 | creating a fullblown Python application, porting the MacPython IDE, |
| 1166 | possibly using Python as a standard OSA scripting language and much |
| 1167 | more. You enable this with "configure --enable-framework". |
| 1168 | The other change is that most MacPython toolbox modules, which |
| 1169 | interface to all the MacOS APIs such as windowing, quicktime, |
| 1170 | scripting, etc. have been ported. Again, most of these are not of |
| 1171 | immedeate use, as they need a full application to be really useful, so |
| 1172 | they have been commented out in setup.py. People wanting to experiment |
| 1173 | can uncomment them. Gestalt and Internet Config modules are enabled by |
| 1174 | default. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1175 | |
| 1176 | * Keyword arguments passed to builtin functions that don't take them now cause a |
| 1177 | :exc:`TypeError` exception to be raised, with the message "*function* takes no |
| 1178 | keyword arguments". |
| 1179 | |
| 1180 | * Weak references, added in Python 2.1 as an extension module, are now part of |
| 1181 | the core because they're used in the implementation of new-style classes. The |
| 1182 | :exc:`ReferenceError` exception has therefore moved from the :mod:`weakref` |
| 1183 | module to become a built-in exception. |
| 1184 | |
| 1185 | * A new script, :file:`Tools/scripts/cleanfuture.py` by Tim Peters, |
| 1186 | automatically removes obsolete ``__future__`` statements from Python source |
| 1187 | code. |
| 1188 | |
| 1189 | * An additional *flags* argument has been added to the built-in function |
| 1190 | :func:`compile`, so the behaviour of ``__future__`` statements can now be |
| 1191 | correctly observed in simulated shells, such as those presented by IDLE and |
| 1192 | other development environments. This is described in :pep:`264`. (Contributed |
| 1193 | by Michael Hudson.) |
| 1194 | |
| 1195 | * The new license introduced with Python 1.6 wasn't GPL-compatible. This is |
| 1196 | fixed by some minor textual changes to the 2.2 license, so it's now legal to |
| 1197 | embed Python inside a GPLed program again. Note that Python itself is not |
| 1198 | GPLed, but instead is under a license that's essentially equivalent to the BSD |
| 1199 | license, same as it always was. The license changes were also applied to the |
| 1200 | Python 2.0.1 and 2.1.1 releases. |
| 1201 | |
| 1202 | * When presented with a Unicode filename on Windows, Python will now convert it |
| 1203 | to an MBCS encoded string, as used by the Microsoft file APIs. As MBCS is |
| 1204 | explicitly used by the file APIs, Python's choice of ASCII as the default |
| 1205 | encoding turns out to be an annoyance. On Unix, the locale's character set is |
| 1206 | used if :func:`locale.nl_langinfo(CODESET)` is available. (Windows support was |
| 1207 | contributed by Mark Hammond with assistance from Marc-André Lemburg. Unix |
| 1208 | support was added by Martin von Löwis.) |
| 1209 | |
| 1210 | * Large file support is now enabled on Windows. (Contributed by Tim Peters.) |
| 1211 | |
| 1212 | * The :file:`Tools/scripts/ftpmirror.py` script now parses a :file:`.netrc` |
| 1213 | file, if you have one. (Contributed by Mike Romberg.) |
| 1214 | |
| 1215 | * Some features of the object returned by the :func:`xrange` function are now |
| 1216 | deprecated, and trigger warnings when they're accessed; they'll disappear in |
| 1217 | Python 2.3. :class:`xrange` objects tried to pretend they were full sequence |
| 1218 | types by supporting slicing, sequence multiplication, and the :keyword:`in` |
| 1219 | operator, but these features were rarely used and therefore buggy. The |
| 1220 | :meth:`tolist` method and the :attr:`start`, :attr:`stop`, and :attr:`step` |
| 1221 | attributes are also being deprecated. At the C level, the fourth argument to |
| 1222 | the :cfunc:`PyRange_New` function, ``repeat``, has also been deprecated. |
| 1223 | |
| 1224 | * There were a bunch of patches to the dictionary implementation, mostly to fix |
| 1225 | potential core dumps if a dictionary contains objects that sneakily changed |
| 1226 | their hash value, or mutated the dictionary they were contained in. For a while |
| 1227 | python-dev fell into a gentle rhythm of Michael Hudson finding a case that |
| 1228 | dumped core, Tim Peters fixing the bug, Michael finding another case, and round |
| 1229 | and round it went. |
| 1230 | |
| 1231 | * On Windows, Python can now be compiled with Borland C thanks to a number of |
| 1232 | patches contributed by Stephen Hansen, though the result isn't fully functional |
| 1233 | yet. (But this *is* progress...) |
| 1234 | |
| 1235 | * Another Windows enhancement: Wise Solutions generously offered PythonLabs use |
| 1236 | of their InstallerMaster 8.1 system. Earlier PythonLabs Windows installers used |
| 1237 | Wise 5.0a, which was beginning to show its age. (Packaged up by Tim Peters.) |
| 1238 | |
| 1239 | * Files ending in ``.pyw`` can now be imported on Windows. ``.pyw`` is a |
| 1240 | Windows-only thing, used to indicate that a script needs to be run using |
| 1241 | PYTHONW.EXE instead of PYTHON.EXE in order to prevent a DOS console from popping |
| 1242 | up to display the output. This patch makes it possible to import such scripts, |
| 1243 | in case they're also usable as modules. (Implemented by David Bolen.) |
| 1244 | |
| 1245 | * On platforms where Python uses the C :cfunc:`dlopen` function to load |
| 1246 | extension modules, it's now possible to set the flags used by :cfunc:`dlopen` |
| 1247 | using the :func:`sys.getdlopenflags` and :func:`sys.setdlopenflags` functions. |
| 1248 | (Contributed by Bram Stolk.) |
| 1249 | |
| 1250 | * The :func:`pow` built-in function no longer supports 3 arguments when |
| 1251 | floating-point numbers are supplied. ``pow(x, y, z)`` returns ``(x**y) % z``, |
| 1252 | but this is never useful for floating point numbers, and the final result varies |
| 1253 | unpredictably depending on the platform. A call such as ``pow(2.0, 8.0, 7.0)`` |
| 1254 | will now raise a :exc:`TypeError` exception. |
| 1255 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 1256 | .. ====================================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1257 | |
| 1258 | |
| 1259 | Acknowledgements |
| 1260 | ================ |
| 1261 | |
| 1262 | The author would like to thank the following people for offering suggestions, |
| 1263 | corrections and assistance with various drafts of this article: Fred Bremmer, |
| 1264 | Keith Briggs, Andrew Dalke, Fred L. Drake, Jr., Carel Fellinger, David Goodger, |
| 1265 | Mark Hammond, Stephen Hansen, Michael Hudson, Jack Jansen, Marc-André Lemburg, |
| 1266 | Martin von Löwis, Fredrik Lundh, Michael McLay, Nick Mathewson, Paul Moore, |
| 1267 | Gustavo Niemeyer, Don O'Donnell, Joonas Paalasma, Tim Peters, Jens Quade, Tom |
| 1268 | Reinhardt, Neil Schemenauer, Guido van Rossum, Greg Ward, Edward Welbourne. |
| 1269 | |