| **************************** |
| What's New in Python 2.5 |
| **************************** |
| |
| :Author: A.M. Kuchling |
| |
| .. |release| replace:: 1.01 |
| |
| .. $Id: whatsnew25.tex 56611 2007-07-29 08:26:10Z georg.brandl $ |
| .. Fix XXX comments |
| |
| This article explains the new features in Python 2.5. The final release of |
| Python 2.5 is scheduled for August 2006; :pep:`356` describes the planned |
| release schedule. |
| |
| The changes in Python 2.5 are an interesting mix of language and library |
| improvements. The library enhancements will be more important to Python's user |
| community, I think, because several widely-useful packages were added. New |
| modules include ElementTree for XML processing (section :ref:`module-etree`), |
| the SQLite database module (section :ref:`module-sqlite`), and the :mod:`ctypes` |
| module for calling C functions (section :ref:`module-ctypes`). |
| |
| The language changes are of middling significance. Some pleasant new features |
| were added, but most of them aren't features that you'll use every day. |
| Conditional expressions were finally added to the language using a novel syntax; |
| see section :ref:`pep-308`. The new ':keyword:`with`' statement will make |
| writing cleanup code easier (section :ref:`pep-343`). Values can now be passed |
| into generators (section :ref:`pep-342`). Imports are now visible as either |
| absolute or relative (section :ref:`pep-328`). Some corner cases of exception |
| handling are handled better (section :ref:`pep-341`). All these improvements |
| are worthwhile, but they're improvements to one specific language feature or |
| another; none of them are broad modifications to Python's semantics. |
| |
| As well as the language and library additions, other improvements and bugfixes |
| were made throughout the source tree. A search through the SVN change logs |
| finds there were 353 patches applied and 458 bugs fixed between Python 2.4 and |
| 2.5. (Both figures are likely to be underestimates.) |
| |
| This article doesn't try to be a complete specification of the new features; |
| instead changes are briefly introduced using helpful examples. For full |
| details, you should always refer to the documentation for Python 2.5 at |
| http://docs.python.org. If you want to understand the complete implementation |
| and design rationale, refer to the PEP for a particular new feature. |
| |
| Comments, suggestions, and error reports for this document are welcome; please |
| e-mail them to the author or open a bug in the Python bug tracker. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-308: |
| |
| PEP 308: Conditional Expressions |
| ================================ |
| |
| For a long time, people have been requesting a way to write conditional |
| expressions, which are expressions that return value A or value B depending on |
| whether a Boolean value is true or false. A conditional expression lets you |
| write a single assignment statement that has the same effect as the following:: |
| |
| if condition: |
| x = true_value |
| else: |
| x = false_value |
| |
| There have been endless tedious discussions of syntax on both python-dev and |
| comp.lang.python. A vote was even held that found the majority of voters wanted |
| conditional expressions in some form, but there was no syntax that was preferred |
| by a clear majority. Candidates included C's ``cond ? true_v : false_v``, ``if |
| cond then true_v else false_v``, and 16 other variations. |
| |
| Guido van Rossum eventually chose a surprising syntax:: |
| |
| x = true_value if condition else false_value |
| |
| Evaluation is still lazy as in existing Boolean expressions, so the order of |
| evaluation jumps around a bit. The *condition* expression in the middle is |
| evaluated first, and the *true_value* expression is evaluated only if the |
| condition was true. Similarly, the *false_value* expression is only evaluated |
| when the condition is false. |
| |
| This syntax may seem strange and backwards; why does the condition go in the |
| *middle* of the expression, and not in the front as in C's ``c ? x : y``? The |
| decision was checked by applying the new syntax to the modules in the standard |
| library and seeing how the resulting code read. In many cases where a |
| conditional expression is used, one value seems to be the 'common case' and one |
| value is an 'exceptional case', used only on rarer occasions when the condition |
| isn't met. The conditional syntax makes this pattern a bit more obvious:: |
| |
| contents = ((doc + '\n') if doc else '') |
| |
| I read the above statement as meaning "here *contents* is usually assigned a |
| value of ``doc+'\n'``; sometimes *doc* is empty, in which special case an empty |
| string is returned." I doubt I will use conditional expressions very often |
| where there isn't a clear common and uncommon case. |
| |
| There was some discussion of whether the language should require surrounding |
| conditional expressions with parentheses. The decision was made to *not* |
| require parentheses in the Python language's grammar, but as a matter of style I |
| think you should always use them. Consider these two statements:: |
| |
| # First version -- no parens |
| level = 1 if logging else 0 |
| |
| # Second version -- with parens |
| level = (1 if logging else 0) |
| |
| In the first version, I think a reader's eye might group the statement into |
| 'level = 1', 'if logging', 'else 0', and think that the condition decides |
| whether the assignment to *level* is performed. The second version reads |
| better, in my opinion, because it makes it clear that the assignment is always |
| performed and the choice is being made between two values. |
| |
| Another reason for including the brackets: a few odd combinations of list |
| comprehensions and lambdas could look like incorrect conditional expressions. |
| See :pep:`308` for some examples. If you put parentheses around your |
| conditional expressions, you won't run into this case. |
| |
| |
| .. seealso:: |
| |
| :pep:`308` - Conditional Expressions |
| PEP written by Guido van Rossum and Raymond D. Hettinger; implemented by Thomas |
| Wouters. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-309: |
| |
| PEP 309: Partial Function Application |
| ===================================== |
| |
| The :mod:`functools` module is intended to contain tools for functional-style |
| programming. |
| |
| One useful tool in this module is the :func:`partial` function. For programs |
| written in a functional style, you'll sometimes want to construct variants of |
| existing functions that have some of the parameters filled in. Consider a |
| Python function ``f(a, b, c)``; you could create a new function ``g(b, c)`` that |
| was equivalent to ``f(1, b, c)``. This is called "partial function |
| application". |
| |
| :func:`partial` takes the arguments ``(function, arg1, arg2, ... kwarg1=value1, |
| kwarg2=value2)``. The resulting object is callable, so you can just call it to |
| invoke *function* with the filled-in arguments. |
| |
| Here's a small but realistic example:: |
| |
| import functools |
| |
| def log (message, subsystem): |
| "Write the contents of 'message' to the specified subsystem." |
| print '%s: %s' % (subsystem, message) |
| ... |
| |
| server_log = functools.partial(log, subsystem='server') |
| server_log('Unable to open socket') |
| |
| Here's another example, from a program that uses PyGTK. Here a context- |
| sensitive pop-up menu is being constructed dynamically. The callback provided |
| for the menu option is a partially applied version of the :meth:`open_item` |
| method, where the first argument has been provided. :: |
| |
| ... |
| class Application: |
| def open_item(self, path): |
| ... |
| def init (self): |
| open_func = functools.partial(self.open_item, item_path) |
| popup_menu.append( ("Open", open_func, 1) ) |
| |
| Another function in the :mod:`functools` module is the |
| :func:`update_wrapper(wrapper, wrapped)` function that helps you write well- |
| behaved decorators. :func:`update_wrapper` copies the name, module, and |
| docstring attribute to a wrapper function so that tracebacks inside the wrapped |
| function are easier to understand. For example, you might write:: |
| |
| def my_decorator(f): |
| def wrapper(*args, **kwds): |
| print 'Calling decorated function' |
| return f(*args, **kwds) |
| functools.update_wrapper(wrapper, f) |
| return wrapper |
| |
| :func:`wraps` is a decorator that can be used inside your own decorators to copy |
| the wrapped function's information. An alternate version of the previous |
| example would be:: |
| |
| def my_decorator(f): |
| @functools.wraps(f) |
| def wrapper(*args, **kwds): |
| print 'Calling decorated function' |
| return f(*args, **kwds) |
| return wrapper |
| |
| |
| .. seealso:: |
| |
| :pep:`309` - Partial Function Application |
| PEP proposed and written by Peter Harris; implemented by Hye-Shik Chang and Nick |
| Coghlan, with adaptations by Raymond Hettinger. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-314: |
| |
| PEP 314: Metadata for Python Software Packages v1.1 |
| =================================================== |
| |
| Some simple dependency support was added to Distutils. The :func:`setup` |
| function now has ``requires``, ``provides``, and ``obsoletes`` keyword |
| parameters. When you build a source distribution using the ``sdist`` command, |
| the dependency information will be recorded in the :file:`PKG-INFO` file. |
| |
| Another new keyword parameter is ``download_url``, which should be set to a URL |
| for the package's source code. This means it's now possible to look up an entry |
| in the package index, determine the dependencies for a package, and download the |
| required packages. :: |
| |
| VERSION = '1.0' |
| setup(name='PyPackage', |
| version=VERSION, |
| requires=['numarray', 'zlib (>=1.1.4)'], |
| obsoletes=['OldPackage'] |
| download_url=('http://www.example.com/pypackage/dist/pkg-%s.tar.gz' |
| % VERSION), |
| ) |
| |
| Another new enhancement to the Python package index at |
| http://cheeseshop.python.org is storing source and binary archives for a |
| package. The new :command:`upload` Distutils command will upload a package to |
| the repository. |
| |
| Before a package can be uploaded, you must be able to build a distribution using |
| the :command:`sdist` Distutils command. Once that works, you can run ``python |
| setup.py upload`` to add your package to the PyPI archive. Optionally you can |
| GPG-sign the package by supplying the :option:`--sign` and :option:`--identity` |
| options. |
| |
| Package uploading was implemented by Martin von Löwis and Richard Jones. |
| |
| |
| .. seealso:: |
| |
| :pep:`314` - Metadata for Python Software Packages v1.1 |
| PEP proposed and written by A.M. Kuchling, Richard Jones, and Fred Drake; |
| implemented by Richard Jones and Fred Drake. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-328: |
| |
| PEP 328: Absolute and Relative Imports |
| ====================================== |
| |
| The simpler part of PEP 328 was implemented in Python 2.4: parentheses could now |
| be used to enclose the names imported from a module using the ``from ... import |
| ...`` statement, making it easier to import many different names. |
| |
| The more complicated part has been implemented in Python 2.5: importing a module |
| can be specified to use absolute or package-relative imports. The plan is to |
| move toward making absolute imports the default in future versions of Python. |
| |
| Let's say you have a package directory like this:: |
| |
| pkg/ |
| pkg/__init__.py |
| pkg/main.py |
| pkg/string.py |
| |
| This defines a package named :mod:`pkg` containing the :mod:`pkg.main` and |
| :mod:`pkg.string` submodules. |
| |
| Consider the code in the :file:`main.py` module. What happens if it executes |
| the statement ``import string``? In Python 2.4 and earlier, it will first look |
| in the package's directory to perform a relative import, finds |
| :file:`pkg/string.py`, imports the contents of that file as the |
| :mod:`pkg.string` module, and that module is bound to the name ``string`` in the |
| :mod:`pkg.main` module's namespace. |
| |
| That's fine if :mod:`pkg.string` was what you wanted. But what if you wanted |
| Python's standard :mod:`string` module? There's no clean way to ignore |
| :mod:`pkg.string` and look for the standard module; generally you had to look at |
| the contents of ``sys.modules``, which is slightly unclean. Holger Krekel's |
| :mod:`py.std` package provides a tidier way to perform imports from the standard |
| library, ``import py ; py.std.string.join()``, but that package isn't available |
| on all Python installations. |
| |
| Reading code which relies on relative imports is also less clear, because a |
| reader may be confused about which module, :mod:`string` or :mod:`pkg.string`, |
| is intended to be used. Python users soon learned not to duplicate the names of |
| standard library modules in the names of their packages' submodules, but you |
| can't protect against having your submodule's name being used for a new module |
| added in a future version of Python. |
| |
| In Python 2.5, you can switch :keyword:`import`'s behaviour to absolute imports |
| using a ``from __future__ import absolute_import`` directive. This absolute- |
| import behaviour will become the default in a future version (probably Python |
| 2.7). Once absolute imports are the default, ``import string`` will always |
| find the standard library's version. It's suggested that users should begin |
| using absolute imports as much as possible, so it's preferable to begin writing |
| ``from pkg import string`` in your code. |
| |
| Relative imports are still possible by adding a leading period to the module |
| name when using the ``from ... import`` form:: |
| |
| # Import names from pkg.string |
| from .string import name1, name2 |
| # Import pkg.string |
| from . import string |
| |
| This imports the :mod:`string` module relative to the current package, so in |
| :mod:`pkg.main` this will import *name1* and *name2* from :mod:`pkg.string`. |
| Additional leading periods perform the relative import starting from the parent |
| of the current package. For example, code in the :mod:`A.B.C` module can do:: |
| |
| from . import D # Imports A.B.D |
| from .. import E # Imports A.E |
| from ..F import G # Imports A.F.G |
| |
| Leading periods cannot be used with the ``import modname`` form of the import |
| statement, only the ``from ... import`` form. |
| |
| |
| .. seealso:: |
| |
| :pep:`328` - Imports: Multi-Line and Absolute/Relative |
| PEP written by Aahz; implemented by Thomas Wouters. |
| |
| http://codespeak.net/py/current/doc/index.html |
| The py library by Holger Krekel, which contains the :mod:`py.std` package. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-338: |
| |
| PEP 338: Executing Modules as Scripts |
| ===================================== |
| |
| The :option:`-m` switch added in Python 2.4 to execute a module as a script |
| gained a few more abilities. Instead of being implemented in C code inside the |
| Python interpreter, the switch now uses an implementation in a new module, |
| :mod:`runpy`. |
| |
| The :mod:`runpy` module implements a more sophisticated import mechanism so that |
| it's now possible to run modules in a package such as :mod:`pychecker.checker`. |
| The module also supports alternative import mechanisms such as the |
| :mod:`zipimport` module. This means you can add a .zip archive's path to |
| ``sys.path`` and then use the :option:`-m` switch to execute code from the |
| archive. |
| |
| |
| .. seealso:: |
| |
| :pep:`338` - Executing modules as scripts |
| PEP written and implemented by Nick Coghlan. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-341: |
| |
| PEP 341: Unified try/except/finally |
| =================================== |
| |
| Until Python 2.5, the :keyword:`try` statement came in two flavours. You could |
| use a :keyword:`finally` block to ensure that code is always executed, or one or |
| more :keyword:`except` blocks to catch specific exceptions. You couldn't |
| combine both :keyword:`except` blocks and a :keyword:`finally` block, because |
| generating the right bytecode for the combined version was complicated and it |
| wasn't clear what the semantics of the combined statement should be. |
| |
| Guido van Rossum spent some time working with Java, which does support the |
| equivalent of combining :keyword:`except` blocks and a :keyword:`finally` block, |
| and this clarified what the statement should mean. In Python 2.5, you can now |
| write:: |
| |
| try: |
| block-1 ... |
| except Exception1: |
| handler-1 ... |
| except Exception2: |
| handler-2 ... |
| else: |
| else-block |
| finally: |
| final-block |
| |
| The code in *block-1* is executed. If the code raises an exception, the various |
| :keyword:`except` blocks are tested: if the exception is of class |
| :class:`Exception1`, *handler-1* is executed; otherwise if it's of class |
| :class:`Exception2`, *handler-2* is executed, and so forth. If no exception is |
| raised, the *else-block* is executed. |
| |
| No matter what happened previously, the *final-block* is executed once the code |
| block is complete and any raised exceptions handled. Even if there's an error in |
| an exception handler or the *else-block* and a new exception is raised, the code |
| in the *final-block* is still run. |
| |
| |
| .. seealso:: |
| |
| :pep:`341` - Unifying try-except and try-finally |
| PEP written by Georg Brandl; implementation by Thomas Lee. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-342: |
| |
| PEP 342: New Generator Features |
| =============================== |
| |
| Python 2.5 adds a simple way to pass values *into* a generator. As introduced in |
| Python 2.3, generators only produce output; once a generator's code was invoked |
| to create an iterator, there was no way to pass any new information into the |
| function when its execution is resumed. Sometimes the ability to pass in some |
| information would be useful. Hackish solutions to this include making the |
| generator's code look at a global variable and then changing the global |
| variable's value, or passing in some mutable object that callers then modify. |
| |
| To refresh your memory of basic generators, here's a simple example:: |
| |
| def counter (maximum): |
| i = 0 |
| while i < maximum: |
| yield i |
| i += 1 |
| |
| When you call ``counter(10)``, the result is an iterator that returns the values |
| from 0 up to 9. On encountering the :keyword:`yield` statement, the iterator |
| returns the provided value and suspends the function's execution, preserving the |
| local variables. Execution resumes on the following call to the iterator's |
| :meth:`next` method, picking up after the :keyword:`yield` statement. |
| |
| In Python 2.3, :keyword:`yield` was a statement; it didn't return any value. In |
| 2.5, :keyword:`yield` is now an expression, returning a value that can be |
| assigned to a variable or otherwise operated on:: |
| |
| val = (yield i) |
| |
| I recommend that you always put parentheses around a :keyword:`yield` expression |
| when you're doing something with the returned value, as in the above example. |
| The parentheses aren't always necessary, but it's easier to always add them |
| instead of having to remember when they're needed. |
| |
| (:pep:`342` explains the exact rules, which are that a :keyword:`yield`\ |
| -expression must always be parenthesized except when it occurs at the top-level |
| expression on the right-hand side of an assignment. This means you can write |
| ``val = yield i`` but have to use parentheses when there's an operation, as in |
| ``val = (yield i) + 12``.) |
| |
| Values are sent into a generator by calling its :meth:`send(value)` method. The |
| generator's code is then resumed and the :keyword:`yield` expression returns the |
| specified *value*. If the regular :meth:`next` method is called, the |
| :keyword:`yield` returns :const:`None`. |
| |
| Here's the previous example, modified to allow changing the value of the |
| internal counter. :: |
| |
| def counter (maximum): |
| i = 0 |
| while i < maximum: |
| val = (yield i) |
| # If value provided, change counter |
| if val is not None: |
| i = val |
| else: |
| i += 1 |
| |
| And here's an example of changing the counter:: |
| |
| >>> it = counter(10) |
| >>> print it.next() |
| 0 |
| >>> print it.next() |
| 1 |
| >>> print it.send(8) |
| 8 |
| >>> print it.next() |
| 9 |
| >>> print it.next() |
| Traceback (most recent call last): |
| File ``t.py'', line 15, in ? |
| print it.next() |
| StopIteration |
| |
| :keyword:`yield` will usually return :const:`None`, so you should always check |
| for this case. Don't just use its value in expressions unless you're sure that |
| the :meth:`send` method will be the only method used to resume your generator |
| function. |
| |
| In addition to :meth:`send`, there are two other new methods on generators: |
| |
| * :meth:`throw(type, value=None, traceback=None)` is used to raise an exception |
| inside the generator; the exception is raised by the :keyword:`yield` expression |
| where the generator's execution is paused. |
| |
| * :meth:`close` raises a new :exc:`GeneratorExit` exception inside the generator |
| to terminate the iteration. On receiving this exception, the generator's code |
| must either raise :exc:`GeneratorExit` or :exc:`StopIteration`. Catching the |
| :exc:`GeneratorExit` exception and returning a value is illegal and will trigger |
| a :exc:`RuntimeError`; if the function raises some other exception, that |
| exception is propagated to the caller. :meth:`close` will also be called by |
| Python's garbage collector when the generator is garbage-collected. |
| |
| If you need to run cleanup code when a :exc:`GeneratorExit` occurs, I suggest |
| using a ``try: ... finally:`` suite instead of catching :exc:`GeneratorExit`. |
| |
| The cumulative effect of these changes is to turn generators from one-way |
| producers of information into both producers and consumers. |
| |
| Generators also become *coroutines*, a more generalized form of subroutines. |
| Subroutines are entered at one point and exited at another point (the top of the |
| function, and a :keyword:`return` statement), but coroutines can be entered, |
| exited, and resumed at many different points (the :keyword:`yield` statements). |
| We'll have to figure out patterns for using coroutines effectively in Python. |
| |
| The addition of the :meth:`close` method has one side effect that isn't obvious. |
| :meth:`close` is called when a generator is garbage-collected, so this means the |
| generator's code gets one last chance to run before the generator is destroyed. |
| This last chance means that ``try...finally`` statements in generators can now |
| be guaranteed to work; the :keyword:`finally` clause will now always get a |
| chance to run. The syntactic restriction that you couldn't mix :keyword:`yield` |
| statements with a ``try...finally`` suite has therefore been removed. This |
| seems like a minor bit of language trivia, but using generators and |
| ``try...finally`` is actually necessary in order to implement the |
| :keyword:`with` statement described by PEP 343. I'll look at this new statement |
| in the following section. |
| |
| Another even more esoteric effect of this change: previously, the |
| :attr:`gi_frame` attribute of a generator was always a frame object. It's now |
| possible for :attr:`gi_frame` to be ``None`` once the generator has been |
| exhausted. |
| |
| |
| .. seealso:: |
| |
| :pep:`342` - Coroutines via Enhanced Generators |
| PEP written by Guido van Rossum and Phillip J. Eby; implemented by Phillip J. |
| Eby. Includes examples of some fancier uses of generators as coroutines. |
| |
| Earlier versions of these features were proposed in :pep:`288` by Raymond |
| Hettinger and :pep:`325` by Samuele Pedroni. |
| |
| http://en.wikipedia.org/wiki/Coroutine |
| The Wikipedia entry for coroutines. |
| |
| http://www.sidhe.org/~dan/blog/archives/000178.html |
| An explanation of coroutines from a Perl point of view, written by Dan Sugalski. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-343: |
| |
| PEP 343: The 'with' statement |
| ============================= |
| |
| The ':keyword:`with`' statement clarifies code that previously would use |
| ``try...finally`` blocks to ensure that clean-up code is executed. In this |
| section, I'll discuss the statement as it will commonly be used. In the next |
| section, I'll examine the implementation details and show how to write objects |
| for use with this statement. |
| |
| The ':keyword:`with`' statement is a new control-flow structure whose basic |
| structure is:: |
| |
| with expression [as variable]: |
| with-block |
| |
| The expression is evaluated, and it should result in an object that supports the |
| context management protocol (that is, has :meth:`__enter__` and :meth:`__exit__` |
| methods. |
| |
| The object's :meth:`__enter__` is called before *with-block* is executed and |
| therefore can run set-up code. It also may return a value that is bound to the |
| name *variable*, if given. (Note carefully that *variable* is *not* assigned |
| the result of *expression*.) |
| |
| After execution of the *with-block* is finished, the object's :meth:`__exit__` |
| method is called, even if the block raised an exception, and can therefore run |
| clean-up code. |
| |
| To enable the statement in Python 2.5, you need to add the following directive |
| to your module:: |
| |
| from __future__ import with_statement |
| |
| The statement will always be enabled in Python 2.6. |
| |
| Some standard Python objects now support the context management protocol and can |
| be used with the ':keyword:`with`' statement. File objects are one example:: |
| |
| with open('/etc/passwd', 'r') as f: |
| for line in f: |
| print line |
| ... more processing code ... |
| |
| After this statement has executed, the file object in *f* will have been |
| automatically closed, even if the :keyword:`for` loop raised an exception part- |
| way through the block. |
| |
| .. note:: |
| |
| In this case, *f* is the same object created by :func:`open`, because |
| :meth:`file.__enter__` returns *self*. |
| |
| The :mod:`threading` module's locks and condition variables also support the |
| ':keyword:`with`' statement:: |
| |
| lock = threading.Lock() |
| with lock: |
| # Critical section of code |
| ... |
| |
| The lock is acquired before the block is executed and always released once the |
| block is complete. |
| |
| The new :func:`localcontext` function in the :mod:`decimal` module makes it easy |
| to save and restore the current decimal context, which encapsulates the desired |
| precision and rounding characteristics for computations:: |
| |
| from decimal import Decimal, Context, localcontext |
| |
| # Displays with default precision of 28 digits |
| v = Decimal('578') |
| print v.sqrt() |
| |
| with localcontext(Context(prec=16)): |
| # All code in this block uses a precision of 16 digits. |
| # The original context is restored on exiting the block. |
| print v.sqrt() |
| |
| |
| .. _new-25-context-managers: |
| |
| Writing Context Managers |
| ------------------------ |
| |
| Under the hood, the ':keyword:`with`' statement is fairly complicated. Most |
| people will only use ':keyword:`with`' in company with existing objects and |
| don't need to know these details, so you can skip the rest of this section if |
| you like. Authors of new objects will need to understand the details of the |
| underlying implementation and should keep reading. |
| |
| A high-level explanation of the context management protocol is: |
| |
| * The expression is evaluated and should result in an object called a "context |
| manager". The context manager must have :meth:`__enter__` and :meth:`__exit__` |
| methods. |
| |
| * The context manager's :meth:`__enter__` method is called. The value returned |
| is assigned to *VAR*. If no ``'as VAR'`` clause is present, the value is simply |
| discarded. |
| |
| * The code in *BLOCK* is executed. |
| |
| * If *BLOCK* raises an exception, the :meth:`__exit__(type, value, traceback)` |
| is called with the exception details, the same values returned by |
| :func:`sys.exc_info`. The method's return value controls whether the exception |
| is re-raised: any false value re-raises the exception, and ``True`` will result |
| in suppressing it. You'll only rarely want to suppress the exception, because |
| if you do the author of the code containing the ':keyword:`with`' statement will |
| never realize anything went wrong. |
| |
| * If *BLOCK* didn't raise an exception, the :meth:`__exit__` method is still |
| called, but *type*, *value*, and *traceback* are all ``None``. |
| |
| Let's think through an example. I won't present detailed code but will only |
| sketch the methods necessary for a database that supports transactions. |
| |
| (For people unfamiliar with database terminology: a set of changes to the |
| database are grouped into a transaction. Transactions can be either committed, |
| meaning that all the changes are written into the database, or rolled back, |
| meaning that the changes are all discarded and the database is unchanged. See |
| any database textbook for more information.) |
| |
| Let's assume there's an object representing a database connection. Our goal will |
| be to let the user write code like this:: |
| |
| db_connection = DatabaseConnection() |
| with db_connection as cursor: |
| cursor.execute('insert into ...') |
| cursor.execute('delete from ...') |
| # ... more operations ... |
| |
| The transaction should be committed if the code in the block runs flawlessly or |
| rolled back if there's an exception. Here's the basic interface for |
| :class:`DatabaseConnection` that I'll assume:: |
| |
| class DatabaseConnection: |
| # Database interface |
| def cursor (self): |
| "Returns a cursor object and starts a new transaction" |
| def commit (self): |
| "Commits current transaction" |
| def rollback (self): |
| "Rolls back current transaction" |
| |
| The :meth:`__enter__` method is pretty easy, having only to start a new |
| transaction. For this application the resulting cursor object would be a useful |
| result, so the method will return it. The user can then add ``as cursor`` to |
| their ':keyword:`with`' statement to bind the cursor to a variable name. :: |
| |
| class DatabaseConnection: |
| ... |
| def __enter__ (self): |
| # Code to start a new transaction |
| cursor = self.cursor() |
| return cursor |
| |
| The :meth:`__exit__` method is the most complicated because it's where most of |
| the work has to be done. The method has to check if an exception occurred. If |
| there was no exception, the transaction is committed. The transaction is rolled |
| back if there was an exception. |
| |
| In the code below, execution will just fall off the end of the function, |
| returning the default value of ``None``. ``None`` is false, so the exception |
| will be re-raised automatically. If you wished, you could be more explicit and |
| add a :keyword:`return` statement at the marked location. :: |
| |
| class DatabaseConnection: |
| ... |
| def __exit__ (self, type, value, tb): |
| if tb is None: |
| # No exception, so commit |
| self.commit() |
| else: |
| # Exception occurred, so rollback. |
| self.rollback() |
| # return False |
| |
| |
| .. _module-contextlib: |
| |
| The contextlib module |
| --------------------- |
| |
| The new :mod:`contextlib` module provides some functions and a decorator that |
| are useful for writing objects for use with the ':keyword:`with`' statement. |
| |
| The decorator is called :func:`contextmanager`, and lets you write a single |
| generator function instead of defining a new class. The generator should yield |
| exactly one value. The code up to the :keyword:`yield` will be executed as the |
| :meth:`__enter__` method, and the value yielded will be the method's return |
| value that will get bound to the variable in the ':keyword:`with`' statement's |
| :keyword:`as` clause, if any. The code after the :keyword:`yield` will be |
| executed in the :meth:`__exit__` method. Any exception raised in the block will |
| be raised by the :keyword:`yield` statement. |
| |
| Our database example from the previous section could be written using this |
| decorator as:: |
| |
| from contextlib import contextmanager |
| |
| @contextmanager |
| def db_transaction (connection): |
| cursor = connection.cursor() |
| try: |
| yield cursor |
| except: |
| connection.rollback() |
| raise |
| else: |
| connection.commit() |
| |
| db = DatabaseConnection() |
| with db_transaction(db) as cursor: |
| ... |
| |
| The :mod:`contextlib` module also has a :func:`nested(mgr1, mgr2, ...)` function |
| that combines a number of context managers so you don't need to write nested |
| ':keyword:`with`' statements. In this example, the single ':keyword:`with`' |
| statement both starts a database transaction and acquires a thread lock:: |
| |
| lock = threading.Lock() |
| with nested (db_transaction(db), lock) as (cursor, locked): |
| ... |
| |
| Finally, the :func:`closing(object)` function returns *object* so that it can be |
| bound to a variable, and calls ``object.close`` at the end of the block. :: |
| |
| import urllib, sys |
| from contextlib import closing |
| |
| with closing(urllib.urlopen('http://www.yahoo.com')) as f: |
| for line in f: |
| sys.stdout.write(line) |
| |
| |
| .. seealso:: |
| |
| :pep:`343` - The "with" statement |
| PEP written by Guido van Rossum and Nick Coghlan; implemented by Mike Bland, |
| Guido van Rossum, and Neal Norwitz. The PEP shows the code generated for a |
| ':keyword:`with`' statement, which can be helpful in learning how the statement |
| works. |
| |
| The documentation for the :mod:`contextlib` module. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-352: |
| |
| PEP 352: Exceptions as New-Style Classes |
| ======================================== |
| |
| Exception classes can now be new-style classes, not just classic classes, and |
| the built-in :exc:`Exception` class and all the standard built-in exceptions |
| (:exc:`NameError`, :exc:`ValueError`, etc.) are now new-style classes. |
| |
| The inheritance hierarchy for exceptions has been rearranged a bit. In 2.5, the |
| inheritance relationships are:: |
| |
| BaseException # New in Python 2.5 |
| |- KeyboardInterrupt |
| |- SystemExit |
| |- Exception |
| |- (all other current built-in exceptions) |
| |
| This rearrangement was done because people often want to catch all exceptions |
| that indicate program errors. :exc:`KeyboardInterrupt` and :exc:`SystemExit` |
| aren't errors, though, and usually represent an explicit action such as the user |
| hitting Control-C or code calling :func:`sys.exit`. A bare ``except:`` will |
| catch all exceptions, so you commonly need to list :exc:`KeyboardInterrupt` and |
| :exc:`SystemExit` in order to re-raise them. The usual pattern is:: |
| |
| try: |
| ... |
| except (KeyboardInterrupt, SystemExit): |
| raise |
| except: |
| # Log error... |
| # Continue running program... |
| |
| In Python 2.5, you can now write ``except Exception`` to achieve the same |
| result, catching all the exceptions that usually indicate errors but leaving |
| :exc:`KeyboardInterrupt` and :exc:`SystemExit` alone. As in previous versions, |
| a bare ``except:`` still catches all exceptions. |
| |
| The goal for Python 3.0 is to require any class raised as an exception to derive |
| from :exc:`BaseException` or some descendant of :exc:`BaseException`, and future |
| releases in the Python 2.x series may begin to enforce this constraint. |
| Therefore, I suggest you begin making all your exception classes derive from |
| :exc:`Exception` now. It's been suggested that the bare ``except:`` form should |
| be removed in Python 3.0, but Guido van Rossum hasn't decided whether to do this |
| or not. |
| |
| Raising of strings as exceptions, as in the statement ``raise "Error |
| occurred"``, is deprecated in Python 2.5 and will trigger a warning. The aim is |
| to be able to remove the string-exception feature in a few releases. |
| |
| |
| .. seealso:: |
| |
| :pep:`352` - Required Superclass for Exceptions |
| PEP written by Brett Cannon and Guido van Rossum; implemented by Brett Cannon. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-353: |
| |
| PEP 353: Using ssize_t as the index type |
| ======================================== |
| |
| A wide-ranging change to Python's C API, using a new :ctype:`Py_ssize_t` type |
| definition instead of :ctype:`int`, will permit the interpreter to handle more |
| data on 64-bit platforms. This change doesn't affect Python's capacity on 32-bit |
| platforms. |
| |
| Various pieces of the Python interpreter used C's :ctype:`int` type to store |
| sizes or counts; for example, the number of items in a list or tuple were stored |
| in an :ctype:`int`. The C compilers for most 64-bit platforms still define |
| :ctype:`int` as a 32-bit type, so that meant that lists could only hold up to |
| ``2**31 - 1`` = 2147483647 items. (There are actually a few different |
| programming models that 64-bit C compilers can use -- see |
| http://www.unix.org/version2/whatsnew/lp64_wp.html for a discussion -- but the |
| most commonly available model leaves :ctype:`int` as 32 bits.) |
| |
| A limit of 2147483647 items doesn't really matter on a 32-bit platform because |
| you'll run out of memory before hitting the length limit. Each list item |
| requires space for a pointer, which is 4 bytes, plus space for a |
| :ctype:`PyObject` representing the item. 2147483647\*4 is already more bytes |
| than a 32-bit address space can contain. |
| |
| It's possible to address that much memory on a 64-bit platform, however. The |
| pointers for a list that size would only require 16 GiB of space, so it's not |
| unreasonable that Python programmers might construct lists that large. |
| Therefore, the Python interpreter had to be changed to use some type other than |
| :ctype:`int`, and this will be a 64-bit type on 64-bit platforms. The change |
| will cause incompatibilities on 64-bit machines, so it was deemed worth making |
| the transition now, while the number of 64-bit users is still relatively small. |
| (In 5 or 10 years, we may *all* be on 64-bit machines, and the transition would |
| be more painful then.) |
| |
| This change most strongly affects authors of C extension modules. Python |
| strings and container types such as lists and tuples now use |
| :ctype:`Py_ssize_t` to store their size. Functions such as |
| :cfunc:`PyList_Size` now return :ctype:`Py_ssize_t`. Code in extension modules |
| may therefore need to have some variables changed to :ctype:`Py_ssize_t`. |
| |
| The :cfunc:`PyArg_ParseTuple` and :cfunc:`Py_BuildValue` functions have a new |
| conversion code, ``n``, for :ctype:`Py_ssize_t`. :cfunc:`PyArg_ParseTuple`'s |
| ``s#`` and ``t#`` still output :ctype:`int` by default, but you can define the |
| macro :cmacro:`PY_SSIZE_T_CLEAN` before including :file:`Python.h` to make |
| them return :ctype:`Py_ssize_t`. |
| |
| :pep:`353` has a section on conversion guidelines that extension authors should |
| read to learn about supporting 64-bit platforms. |
| |
| |
| .. seealso:: |
| |
| :pep:`353` - Using ssize_t as the index type |
| PEP written and implemented by Martin von Löwis. |
| |
| .. ====================================================================== |
| |
| |
| .. _pep-357: |
| |
| PEP 357: The '__index__' method |
| =============================== |
| |
| The NumPy developers had a problem that could only be solved by adding a new |
| special method, :meth:`__index__`. When using slice notation, as in |
| ``[start:stop:step]``, the values of the *start*, *stop*, and *step* indexes |
| must all be either integers or long integers. NumPy defines a variety of |
| specialized integer types corresponding to unsigned and signed integers of 8, |
| 16, 32, and 64 bits, but there was no way to signal that these types could be |
| used as slice indexes. |
| |
| Slicing can't just use the existing :meth:`__int__` method because that method |
| is also used to implement coercion to integers. If slicing used |
| :meth:`__int__`, floating-point numbers would also become legal slice indexes |
| and that's clearly an undesirable behaviour. |
| |
| Instead, a new special method called :meth:`__index__` was added. It takes no |
| arguments and returns an integer giving the slice index to use. For example:: |
| |
| class C: |
| def __index__ (self): |
| return self.value |
| |
| The return value must be either a Python integer or long integer. The |
| interpreter will check that the type returned is correct, and raises a |
| :exc:`TypeError` if this requirement isn't met. |
| |
| A corresponding :attr:`nb_index` slot was added to the C-level |
| :ctype:`PyNumberMethods` structure to let C extensions implement this protocol. |
| :cfunc:`PyNumber_Index(obj)` can be used in extension code to call the |
| :meth:`__index__` function and retrieve its result. |
| |
| |
| .. seealso:: |
| |
| :pep:`357` - Allowing Any Object to be Used for Slicing |
| PEP written and implemented by Travis Oliphant. |
| |
| .. ====================================================================== |
| |
| |
| .. _other-lang: |
| |
| Other Language Changes |
| ====================== |
| |
| Here are all of the changes that Python 2.5 makes to the core Python language. |
| |
| * The :class:`dict` type has a new hook for letting subclasses provide a default |
| value when a key isn't contained in the dictionary. When a key isn't found, the |
| dictionary's :meth:`__missing__(key)` method will be called. This hook is used |
| to implement the new :class:`defaultdict` class in the :mod:`collections` |
| module. The following example defines a dictionary that returns zero for any |
| missing key:: |
| |
| class zerodict (dict): |
| def __missing__ (self, key): |
| return 0 |
| |
| d = zerodict({1:1, 2:2}) |
| print d[1], d[2] # Prints 1, 2 |
| print d[3], d[4] # Prints 0, 0 |
| |
| * Both 8-bit and Unicode strings have new :meth:`partition(sep)` and |
| :meth:`rpartition(sep)` methods that simplify a common use case. |
| |
| The :meth:`find(S)` method is often used to get an index which is then used to |
| slice the string and obtain the pieces that are before and after the separator. |
| :meth:`partition(sep)` condenses this pattern into a single method call that |
| returns a 3-tuple containing the substring before the separator, the separator |
| itself, and the substring after the separator. If the separator isn't found, |
| the first element of the tuple is the entire string and the other two elements |
| are empty. :meth:`rpartition(sep)` also returns a 3-tuple but starts searching |
| from the end of the string; the ``r`` stands for 'reverse'. |
| |
| Some examples:: |
| |
| >>> ('http://www.python.org').partition('://') |
| ('http', '://', 'www.python.org') |
| >>> ('file:/usr/share/doc/index.html').partition('://') |
| ('file:/usr/share/doc/index.html', '', '') |
| >>> (u'Subject: a quick question').partition(':') |
| (u'Subject', u':', u' a quick question') |
| >>> 'www.python.org'.rpartition('.') |
| ('www.python', '.', 'org') |
| >>> 'www.python.org'.rpartition(':') |
| ('', '', 'www.python.org') |
| |
| (Implemented by Fredrik Lundh following a suggestion by Raymond Hettinger.) |
| |
| * The :meth:`startswith` and :meth:`endswith` methods of string types now accept |
| tuples of strings to check for. :: |
| |
| def is_image_file (filename): |
| return filename.endswith(('.gif', '.jpg', '.tiff')) |
| |
| (Implemented by Georg Brandl following a suggestion by Tom Lynn.) |
| |
| .. RFE #1491485 |
| |
| * The :func:`min` and :func:`max` built-in functions gained a ``key`` keyword |
| parameter analogous to the ``key`` argument for :meth:`sort`. This parameter |
| supplies a function that takes a single argument and is called for every value |
| in the list; :func:`min`/:func:`max` will return the element with the |
| smallest/largest return value from this function. For example, to find the |
| longest string in a list, you can do:: |
| |
| L = ['medium', 'longest', 'short'] |
| # Prints 'longest' |
| print max(L, key=len) |
| # Prints 'short', because lexicographically 'short' has the largest value |
| print max(L) |
| |
| (Contributed by Steven Bethard and Raymond Hettinger.) |
| |
| * Two new built-in functions, :func:`any` and :func:`all`, evaluate whether an |
| iterator contains any true or false values. :func:`any` returns :const:`True` |
| if any value returned by the iterator is true; otherwise it will return |
| :const:`False`. :func:`all` returns :const:`True` only if all of the values |
| returned by the iterator evaluate as true. (Suggested by Guido van Rossum, and |
| implemented by Raymond Hettinger.) |
| |
| * The result of a class's :meth:`__hash__` method can now be either a long |
| integer or a regular integer. If a long integer is returned, the hash of that |
| value is taken. In earlier versions the hash value was required to be a |
| regular integer, but in 2.5 the :func:`id` built-in was changed to always |
| return non-negative numbers, and users often seem to use ``id(self)`` in |
| :meth:`__hash__` methods (though this is discouraged). |
| |
| .. Bug #1536021 |
| |
| * ASCII is now the default encoding for modules. It's now a syntax error if a |
| module contains string literals with 8-bit characters but doesn't have an |
| encoding declaration. In Python 2.4 this triggered a warning, not a syntax |
| error. See :pep:`263` for how to declare a module's encoding; for example, you |
| might add a line like this near the top of the source file:: |
| |
| # -*- coding: latin1 -*- |
| |
| * A new warning, :class:`UnicodeWarning`, is triggered when you attempt to |
| compare a Unicode string and an 8-bit string that can't be converted to Unicode |
| using the default ASCII encoding. The result of the comparison is false:: |
| |
| >>> chr(128) == unichr(128) # Can't convert chr(128) to Unicode |
| __main__:1: UnicodeWarning: Unicode equal comparison failed |
| to convert both arguments to Unicode - interpreting them |
| as being unequal |
| False |
| >>> chr(127) == unichr(127) # chr(127) can be converted |
| True |
| |
| Previously this would raise a :class:`UnicodeDecodeError` exception, but in 2.5 |
| this could result in puzzling problems when accessing a dictionary. If you |
| looked up ``unichr(128)`` and ``chr(128)`` was being used as a key, you'd get a |
| :class:`UnicodeDecodeError` exception. Other changes in 2.5 resulted in this |
| exception being raised instead of suppressed by the code in :file:`dictobject.c` |
| that implements dictionaries. |
| |
| Raising an exception for such a comparison is strictly correct, but the change |
| might have broken code, so instead :class:`UnicodeWarning` was introduced. |
| |
| (Implemented by Marc-André Lemburg.) |
| |
| * One error that Python programmers sometimes make is forgetting to include an |
| :file:`__init__.py` module in a package directory. Debugging this mistake can be |
| confusing, and usually requires running Python with the :option:`-v` switch to |
| log all the paths searched. In Python 2.5, a new :exc:`ImportWarning` warning is |
| triggered when an import would have picked up a directory as a package but no |
| :file:`__init__.py` was found. This warning is silently ignored by default; |
| provide the :option:`-Wd` option when running the Python executable to display |
| the warning message. (Implemented by Thomas Wouters.) |
| |
| * The list of base classes in a class definition can now be empty. As an |
| example, this is now legal:: |
| |
| class C(): |
| pass |
| |
| (Implemented by Brett Cannon.) |
| |
| .. ====================================================================== |
| |
| |
| .. _interactive: |
| |
| Interactive Interpreter Changes |
| ------------------------------- |
| |
| In the interactive interpreter, ``quit`` and ``exit`` have long been strings so |
| that new users get a somewhat helpful message when they try to quit:: |
| |
| >>> quit |
| 'Use Ctrl-D (i.e. EOF) to exit.' |
| |
| In Python 2.5, ``quit`` and ``exit`` are now objects that still produce string |
| representations of themselves, but are also callable. Newbies who try ``quit()`` |
| or ``exit()`` will now exit the interpreter as they expect. (Implemented by |
| Georg Brandl.) |
| |
| The Python executable now accepts the standard long options :option:`--help` |
| and :option:`--version`; on Windows, it also accepts the :option:`/?` option |
| for displaying a help message. (Implemented by Georg Brandl.) |
| |
| .. ====================================================================== |
| |
| |
| .. _opts: |
| |
| Optimizations |
| ------------- |
| |
| Several of the optimizations were developed at the NeedForSpeed sprint, an event |
| held in Reykjavik, Iceland, from May 21--28 2006. The sprint focused on speed |
| enhancements to the CPython implementation and was funded by EWT LLC with local |
| support from CCP Games. Those optimizations added at this sprint are specially |
| marked in the following list. |
| |
| * When they were introduced in Python 2.4, the built-in :class:`set` and |
| :class:`frozenset` types were built on top of Python's dictionary type. In 2.5 |
| the internal data structure has been customized for implementing sets, and as a |
| result sets will use a third less memory and are somewhat faster. (Implemented |
| by Raymond Hettinger.) |
| |
| * The speed of some Unicode operations, such as finding substrings, string |
| splitting, and character map encoding and decoding, has been improved. |
| (Substring search and splitting improvements were added by Fredrik Lundh and |
| Andrew Dalke at the NeedForSpeed sprint. Character maps were improved by Walter |
| Dörwald and Martin von Löwis.) |
| |
| .. Patch 1313939, 1359618 |
| |
| * The :func:`long(str, base)` function is now faster on long digit strings |
| because fewer intermediate results are calculated. The peak is for strings of |
| around 800--1000 digits where the function is 6 times faster. (Contributed by |
| Alan McIntyre and committed at the NeedForSpeed sprint.) |
| |
| .. Patch 1442927 |
| |
| * It's now illegal to mix iterating over a file with ``for line in file`` and |
| calling the file object's :meth:`read`/:meth:`readline`/:meth:`readlines` |
| methods. Iteration uses an internal buffer and the :meth:`read\*` methods |
| don't use that buffer. Instead they would return the data following the |
| buffer, causing the data to appear out of order. Mixing iteration and these |
| methods will now trigger a :exc:`ValueError` from the :meth:`read\*` method. |
| (Implemented by Thomas Wouters.) |
| |
| .. Patch 1397960 |
| |
| * The :mod:`struct` module now compiles structure format strings into an |
| internal representation and caches this representation, yielding a 20% speedup. |
| (Contributed by Bob Ippolito at the NeedForSpeed sprint.) |
| |
| * The :mod:`re` module got a 1 or 2% speedup by switching to Python's allocator |
| functions instead of the system's :cfunc:`malloc` and :cfunc:`free`. |
| (Contributed by Jack Diederich at the NeedForSpeed sprint.) |
| |
| * The code generator's peephole optimizer now performs simple constant folding |
| in expressions. If you write something like ``a = 2+3``, the code generator |
| will do the arithmetic and produce code corresponding to ``a = 5``. (Proposed |
| and implemented by Raymond Hettinger.) |
| |
| * Function calls are now faster because code objects now keep the most recently |
| finished frame (a "zombie frame") in an internal field of the code object, |
| reusing it the next time the code object is invoked. (Original patch by Michael |
| Hudson, modified by Armin Rigo and Richard Jones; committed at the NeedForSpeed |
| sprint.) Frame objects are also slightly smaller, which may improve cache |
| locality and reduce memory usage a bit. (Contributed by Neal Norwitz.) |
| |
| .. Patch 876206 |
| .. Patch 1337051 |
| |
| * Python's built-in exceptions are now new-style classes, a change that speeds |
| up instantiation considerably. Exception handling in Python 2.5 is therefore |
| about 30% faster than in 2.4. (Contributed by Richard Jones, Georg Brandl and |
| Sean Reifschneider at the NeedForSpeed sprint.) |
| |
| * Importing now caches the paths tried, recording whether they exist or not so |
| that the interpreter makes fewer :cfunc:`open` and :cfunc:`stat` calls on |
| startup. (Contributed by Martin von Löwis and Georg Brandl.) |
| |
| .. Patch 921466 |
| |
| .. ====================================================================== |
| |
| |
| .. _modules: |
| |
| New, Improved, and Removed Modules |
| ================================== |
| |
| The standard library received many enhancements and bug fixes in Python 2.5. |
| Here's a partial list of the most notable changes, sorted alphabetically by |
| module name. Consult the :file:`Misc/NEWS` file in the source tree for a more |
| complete list of changes, or look through the SVN logs for all the details. |
| |
| * The :mod:`audioop` module now supports the a-LAW encoding, and the code for |
| u-LAW encoding has been improved. (Contributed by Lars Immisch.) |
| |
| * The :mod:`codecs` module gained support for incremental codecs. The |
| :func:`codec.lookup` function now returns a :class:`CodecInfo` instance instead |
| of a tuple. :class:`CodecInfo` instances behave like a 4-tuple to preserve |
| backward compatibility but also have the attributes :attr:`encode`, |
| :attr:`decode`, :attr:`incrementalencoder`, :attr:`incrementaldecoder`, |
| :attr:`streamwriter`, and :attr:`streamreader`. Incremental codecs can receive |
| input and produce output in multiple chunks; the output is the same as if the |
| entire input was fed to the non-incremental codec. See the :mod:`codecs` module |
| documentation for details. (Designed and implemented by Walter Dörwald.) |
| |
| .. Patch 1436130 |
| |
| * The :mod:`collections` module gained a new type, :class:`defaultdict`, that |
| subclasses the standard :class:`dict` type. The new type mostly behaves like a |
| dictionary but constructs a default value when a key isn't present, |
| automatically adding it to the dictionary for the requested key value. |
| |
| The first argument to :class:`defaultdict`'s constructor is a factory function |
| that gets called whenever a key is requested but not found. This factory |
| function receives no arguments, so you can use built-in type constructors such |
| as :func:`list` or :func:`int`. For example, you can make an index of words |
| based on their initial letter like this:: |
| |
| words = """Nel mezzo del cammin di nostra vita |
| mi ritrovai per una selva oscura |
| che la diritta via era smarrita""".lower().split() |
| |
| index = defaultdict(list) |
| |
| for w in words: |
| init_letter = w[0] |
| index[init_letter].append(w) |
| |
| Printing ``index`` results in the following output:: |
| |
| defaultdict(<type 'list'>, {'c': ['cammin', 'che'], 'e': ['era'], |
| 'd': ['del', 'di', 'diritta'], 'm': ['mezzo', 'mi'], |
| 'l': ['la'], 'o': ['oscura'], 'n': ['nel', 'nostra'], |
| 'p': ['per'], 's': ['selva', 'smarrita'], |
| 'r': ['ritrovai'], 'u': ['una'], 'v': ['vita', 'via']} |
| |
| (Contributed by Guido van Rossum.) |
| |
| * The :class:`deque` double-ended queue type supplied by the :mod:`collections` |
| module now has a :meth:`remove(value)` method that removes the first occurrence |
| of *value* in the queue, raising :exc:`ValueError` if the value isn't found. |
| (Contributed by Raymond Hettinger.) |
| |
| * New module: The :mod:`contextlib` module contains helper functions for use |
| with the new ':keyword:`with`' statement. See section :ref:`module-contextlib` |
| for more about this module. |
| |
| * New module: The :mod:`cProfile` module is a C implementation of the existing |
| :mod:`profile` module that has much lower overhead. The module's interface is |
| the same as :mod:`profile`: you run ``cProfile.run('main()')`` to profile a |
| function, can save profile data to a file, etc. It's not yet known if the |
| Hotshot profiler, which is also written in C but doesn't match the |
| :mod:`profile` module's interface, will continue to be maintained in future |
| versions of Python. (Contributed by Armin Rigo.) |
| |
| Also, the :mod:`pstats` module for analyzing the data measured by the profiler |
| now supports directing the output to any file object by supplying a *stream* |
| argument to the :class:`Stats` constructor. (Contributed by Skip Montanaro.) |
| |
| * The :mod:`csv` module, which parses files in comma-separated value format, |
| received several enhancements and a number of bugfixes. You can now set the |
| maximum size in bytes of a field by calling the |
| :meth:`csv.field_size_limit(new_limit)` function; omitting the *new_limit* |
| argument will return the currently-set limit. The :class:`reader` class now has |
| a :attr:`line_num` attribute that counts the number of physical lines read from |
| the source; records can span multiple physical lines, so :attr:`line_num` is not |
| the same as the number of records read. |
| |
| The CSV parser is now stricter about multi-line quoted fields. Previously, if a |
| line ended within a quoted field without a terminating newline character, a |
| newline would be inserted into the returned field. This behavior caused problems |
| when reading files that contained carriage return characters within fields, so |
| the code was changed to return the field without inserting newlines. As a |
| consequence, if newlines embedded within fields are important, the input should |
| be split into lines in a manner that preserves the newline characters. |
| |
| (Contributed by Skip Montanaro and Andrew McNamara.) |
| |
| * The :class:`datetime` class in the :mod:`datetime` module now has a |
| :meth:`strptime(string, format)` method for parsing date strings, contributed |
| by Josh Spoerri. It uses the same format characters as :func:`time.strptime` and |
| :func:`time.strftime`:: |
| |
| from datetime import datetime |
| |
| ts = datetime.strptime('10:13:15 2006-03-07', |
| '%H:%M:%S %Y-%m-%d') |
| |
| * The :meth:`SequenceMatcher.get_matching_blocks` method in the :mod:`difflib` |
| module now guarantees to return a minimal list of blocks describing matching |
| subsequences. Previously, the algorithm would occasionally break a block of |
| matching elements into two list entries. (Enhancement by Tim Peters.) |
| |
| * The :mod:`doctest` module gained a ``SKIP`` option that keeps an example from |
| being executed at all. This is intended for code snippets that are usage |
| examples intended for the reader and aren't actually test cases. |
| |
| An *encoding* parameter was added to the :func:`testfile` function and the |
| :class:`DocFileSuite` class to specify the file's encoding. This makes it |
| easier to use non-ASCII characters in tests contained within a docstring. |
| (Contributed by Bjorn Tillenius.) |
| |
| .. Patch 1080727 |
| |
| * The :mod:`email` package has been updated to version 4.0. (Contributed by |
| Barry Warsaw.) |
| |
| .. XXX need to provide some more detail here |
| |
| * The :mod:`fileinput` module was made more flexible. Unicode filenames are now |
| supported, and a *mode* parameter that defaults to ``"r"`` was added to the |
| :func:`input` function to allow opening files in binary or universal-newline |
| mode. Another new parameter, *openhook*, lets you use a function other than |
| :func:`open` to open the input files. Once you're iterating over the set of |
| files, the :class:`FileInput` object's new :meth:`fileno` returns the file |
| descriptor for the currently opened file. (Contributed by Georg Brandl.) |
| |
| * In the :mod:`gc` module, the new :func:`get_count` function returns a 3-tuple |
| containing the current collection counts for the three GC generations. This is |
| accounting information for the garbage collector; when these counts reach a |
| specified threshold, a garbage collection sweep will be made. The existing |
| :func:`gc.collect` function now takes an optional *generation* argument of 0, 1, |
| or 2 to specify which generation to collect. (Contributed by Barry Warsaw.) |
| |
| * The :func:`nsmallest` and :func:`nlargest` functions in the :mod:`heapq` |
| module now support a ``key`` keyword parameter similar to the one provided by |
| the :func:`min`/:func:`max` functions and the :meth:`sort` methods. For |
| example:: |
| |
| >>> import heapq |
| >>> L = ["short", 'medium', 'longest', 'longer still'] |
| >>> heapq.nsmallest(2, L) # Return two lowest elements, lexicographically |
| ['longer still', 'longest'] |
| >>> heapq.nsmallest(2, L, key=len) # Return two shortest elements |
| ['short', 'medium'] |
| |
| (Contributed by Raymond Hettinger.) |
| |
| * The :func:`itertools.islice` function now accepts ``None`` for the start and |
| step arguments. This makes it more compatible with the attributes of slice |
| objects, so that you can now write the following:: |
| |
| s = slice(5) # Create slice object |
| itertools.islice(iterable, s.start, s.stop, s.step) |
| |
| (Contributed by Raymond Hettinger.) |
| |
| * The :func:`format` function in the :mod:`locale` module has been modified and |
| two new functions were added, :func:`format_string` and :func:`currency`. |
| |
| The :func:`format` function's *val* parameter could previously be a string as |
| long as no more than one %char specifier appeared; now the parameter must be |
| exactly one %char specifier with no surrounding text. An optional *monetary* |
| parameter was also added which, if ``True``, will use the locale's rules for |
| formatting currency in placing a separator between groups of three digits. |
| |
| To format strings with multiple %char specifiers, use the new |
| :func:`format_string` function that works like :func:`format` but also supports |
| mixing %char specifiers with arbitrary text. |
| |
| A new :func:`currency` function was also added that formats a number according |
| to the current locale's settings. |
| |
| (Contributed by Georg Brandl.) |
| |
| .. Patch 1180296 |
| |
| * The :mod:`mailbox` module underwent a massive rewrite to add the capability to |
| modify mailboxes in addition to reading them. A new set of classes that include |
| :class:`mbox`, :class:`MH`, and :class:`Maildir` are used to read mailboxes, and |
| have an :meth:`add(message)` method to add messages, :meth:`remove(key)` to |
| remove messages, and :meth:`lock`/:meth:`unlock` to lock/unlock the mailbox. |
| The following example converts a maildir-format mailbox into an mbox-format |
| one:: |
| |
| import mailbox |
| |
| # 'factory=None' uses email.Message.Message as the class representing |
| # individual messages. |
| src = mailbox.Maildir('maildir', factory=None) |
| dest = mailbox.mbox('/tmp/mbox') |
| |
| for msg in src: |
| dest.add(msg) |
| |
| (Contributed by Gregory K. Johnson. Funding was provided by Google's 2005 |
| Summer of Code.) |
| |
| * New module: the :mod:`msilib` module allows creating Microsoft Installer |
| :file:`.msi` files and CAB files. Some support for reading the :file:`.msi` |
| database is also included. (Contributed by Martin von Löwis.) |
| |
| * The :mod:`nis` module now supports accessing domains other than the system |
| default domain by supplying a *domain* argument to the :func:`nis.match` and |
| :func:`nis.maps` functions. (Contributed by Ben Bell.) |
| |
| * The :mod:`operator` module's :func:`itemgetter` and :func:`attrgetter` |
| functions now support multiple fields. A call such as |
| ``operator.attrgetter('a', 'b')`` will return a function that retrieves the |
| :attr:`a` and :attr:`b` attributes. Combining this new feature with the |
| :meth:`sort` method's ``key`` parameter lets you easily sort lists using |
| multiple fields. (Contributed by Raymond Hettinger.) |
| |
| * The :mod:`optparse` module was updated to version 1.5.1 of the Optik library. |
| The :class:`OptionParser` class gained an :attr:`epilog` attribute, a string |
| that will be printed after the help message, and a :meth:`destroy` method to |
| break reference cycles created by the object. (Contributed by Greg Ward.) |
| |
| * The :mod:`os` module underwent several changes. The :attr:`stat_float_times` |
| variable now defaults to true, meaning that :func:`os.stat` will now return time |
| values as floats. (This doesn't necessarily mean that :func:`os.stat` will |
| return times that are precise to fractions of a second; not all systems support |
| such precision.) |
| |
| Constants named :attr:`os.SEEK_SET`, :attr:`os.SEEK_CUR`, and |
| :attr:`os.SEEK_END` have been added; these are the parameters to the |
| :func:`os.lseek` function. Two new constants for locking are |
| :attr:`os.O_SHLOCK` and :attr:`os.O_EXLOCK`. |
| |
| Two new functions, :func:`wait3` and :func:`wait4`, were added. They're similar |
| the :func:`waitpid` function which waits for a child process to exit and returns |
| a tuple of the process ID and its exit status, but :func:`wait3` and |
| :func:`wait4` return additional information. :func:`wait3` doesn't take a |
| process ID as input, so it waits for any child process to exit and returns a |
| 3-tuple of *process-id*, *exit-status*, *resource-usage* as returned from the |
| :func:`resource.getrusage` function. :func:`wait4(pid)` does take a process ID. |
| (Contributed by Chad J. Schroeder.) |
| |
| On FreeBSD, the :func:`os.stat` function now returns times with nanosecond |
| resolution, and the returned object now has :attr:`st_gen` and |
| :attr:`st_birthtime`. The :attr:`st_flags` member is also available, if the |
| platform supports it. (Contributed by Antti Louko and Diego Pettenò.) |
| |
| .. (Patch 1180695, 1212117) |
| |
| * The Python debugger provided by the :mod:`pdb` module can now store lists of |
| commands to execute when a breakpoint is reached and execution stops. Once |
| breakpoint #1 has been created, enter ``commands 1`` and enter a series of |
| commands to be executed, finishing the list with ``end``. The command list can |
| include commands that resume execution, such as ``continue`` or ``next``. |
| (Contributed by Grégoire Dooms.) |
| |
| .. Patch 790710 |
| |
| * The :mod:`pickle` and :mod:`cPickle` modules no longer accept a return value |
| of ``None`` from the :meth:`__reduce__` method; the method must return a tuple |
| of arguments instead. The ability to return ``None`` was deprecated in Python |
| 2.4, so this completes the removal of the feature. |
| |
| * The :mod:`pkgutil` module, containing various utility functions for finding |
| packages, was enhanced to support PEP 302's import hooks and now also works for |
| packages stored in ZIP-format archives. (Contributed by Phillip J. Eby.) |
| |
| * The pybench benchmark suite by Marc-André Lemburg is now included in the |
| :file:`Tools/pybench` directory. The pybench suite is an improvement on the |
| commonly used :file:`pystone.py` program because pybench provides a more |
| detailed measurement of the interpreter's speed. It times particular operations |
| such as function calls, tuple slicing, method lookups, and numeric operations, |
| instead of performing many different operations and reducing the result to a |
| single number as :file:`pystone.py` does. |
| |
| * The :mod:`pyexpat` module now uses version 2.0 of the Expat parser. |
| (Contributed by Trent Mick.) |
| |
| * The :class:`Queue` class provided by the :mod:`Queue` module gained two new |
| methods. :meth:`join` blocks until all items in the queue have been retrieved |
| and all processing work on the items have been completed. Worker threads call |
| the other new method, :meth:`task_done`, to signal that processing for an item |
| has been completed. (Contributed by Raymond Hettinger.) |
| |
| * The old :mod:`regex` and :mod:`regsub` modules, which have been deprecated |
| ever since Python 2.0, have finally been deleted. Other deleted modules: |
| :mod:`statcache`, :mod:`tzparse`, :mod:`whrandom`. |
| |
| * Also deleted: the :file:`lib-old` directory, which includes ancient modules |
| such as :mod:`dircmp` and :mod:`ni`, was removed. :file:`lib-old` wasn't on the |
| default ``sys.path``, so unless your programs explicitly added the directory to |
| ``sys.path``, this removal shouldn't affect your code. |
| |
| * The :mod:`rlcompleter` module is no longer dependent on importing the |
| :mod:`readline` module and therefore now works on non-Unix platforms. (Patch |
| from Robert Kiendl.) |
| |
| .. Patch #1472854 |
| |
| * The :mod:`SimpleXMLRPCServer` and :mod:`DocXMLRPCServer` classes now have a |
| :attr:`rpc_paths` attribute that constrains XML-RPC operations to a limited set |
| of URL paths; the default is to allow only ``'/'`` and ``'/RPC2'``. Setting |
| :attr:`rpc_paths` to ``None`` or an empty tuple disables this path checking. |
| |
| .. Bug #1473048 |
| |
| * The :mod:`socket` module now supports :const:`AF_NETLINK` sockets on Linux, |
| thanks to a patch from Philippe Biondi. Netlink sockets are a Linux-specific |
| mechanism for communications between a user-space process and kernel code; an |
| introductory article about them is at http://www.linuxjournal.com/article/7356. |
| In Python code, netlink addresses are represented as a tuple of 2 integers, |
| ``(pid, group_mask)``. |
| |
| Two new methods on socket objects, :meth:`recv_into(buffer)` and |
| :meth:`recvfrom_into(buffer)`, store the received data in an object that |
| supports the buffer protocol instead of returning the data as a string. This |
| means you can put the data directly into an array or a memory-mapped file. |
| |
| Socket objects also gained :meth:`getfamily`, :meth:`gettype`, and |
| :meth:`getproto` accessor methods to retrieve the family, type, and protocol |
| values for the socket. |
| |
| * New module: the :mod:`spwd` module provides functions for accessing the shadow |
| password database on systems that support shadow passwords. |
| |
| * The :mod:`struct` is now faster because it compiles format strings into |
| :class:`Struct` objects with :meth:`pack` and :meth:`unpack` methods. This is |
| similar to how the :mod:`re` module lets you create compiled regular expression |
| objects. You can still use the module-level :func:`pack` and :func:`unpack` |
| functions; they'll create :class:`Struct` objects and cache them. Or you can |
| use :class:`Struct` instances directly:: |
| |
| s = struct.Struct('ih3s') |
| |
| data = s.pack(1972, 187, 'abc') |
| year, number, name = s.unpack(data) |
| |
| You can also pack and unpack data to and from buffer objects directly using the |
| :meth:`pack_into(buffer, offset, v1, v2, ...)` and :meth:`unpack_from(buffer, |
| offset)` methods. This lets you store data directly into an array or a memory- |
| mapped file. |
| |
| (:class:`Struct` objects were implemented by Bob Ippolito at the NeedForSpeed |
| sprint. Support for buffer objects was added by Martin Blais, also at the |
| NeedForSpeed sprint.) |
| |
| * The Python developers switched from CVS to Subversion during the 2.5 |
| development process. Information about the exact build version is available as |
| the ``sys.subversion`` variable, a 3-tuple of ``(interpreter-name, branch-name, |
| revision-range)``. For example, at the time of writing my copy of 2.5 was |
| reporting ``('CPython', 'trunk', '45313:45315')``. |
| |
| This information is also available to C extensions via the |
| :cfunc:`Py_GetBuildInfo` function that returns a string of build information |
| like this: ``"trunk:45355:45356M, Apr 13 2006, 07:42:19"``. (Contributed by |
| Barry Warsaw.) |
| |
| * Another new function, :func:`sys._current_frames`, returns the current stack |
| frames for all running threads as a dictionary mapping thread identifiers to the |
| topmost stack frame currently active in that thread at the time the function is |
| called. (Contributed by Tim Peters.) |
| |
| * The :class:`TarFile` class in the :mod:`tarfile` module now has an |
| :meth:`extractall` method that extracts all members from the archive into the |
| current working directory. It's also possible to set a different directory as |
| the extraction target, and to unpack only a subset of the archive's members. |
| |
| The compression used for a tarfile opened in stream mode can now be autodetected |
| using the mode ``'r|*'``. (Contributed by Lars Gustäbel.) |
| |
| .. patch 918101 |
| |
| * The :mod:`threading` module now lets you set the stack size used when new |
| threads are created. The :func:`stack_size([*size*])` function returns the |
| currently configured stack size, and supplying the optional *size* parameter |
| sets a new value. Not all platforms support changing the stack size, but |
| Windows, POSIX threading, and OS/2 all do. (Contributed by Andrew MacIntyre.) |
| |
| .. Patch 1454481 |
| |
| * The :mod:`unicodedata` module has been updated to use version 4.1.0 of the |
| Unicode character database. Version 3.2.0 is required by some specifications, |
| so it's still available as :attr:`unicodedata.ucd_3_2_0`. |
| |
| * New module: the :mod:`uuid` module generates universally unique identifiers |
| (UUIDs) according to :rfc:`4122`. The RFC defines several different UUID |
| versions that are generated from a starting string, from system properties, or |
| purely randomly. This module contains a :class:`UUID` class and functions |
| named :func:`uuid1`, :func:`uuid3`, :func:`uuid4`, and :func:`uuid5` to |
| generate different versions of UUID. (Version 2 UUIDs are not specified in |
| :rfc:`4122` and are not supported by this module.) :: |
| |
| >>> import uuid |
| >>> # make a UUID based on the host ID and current time |
| >>> uuid.uuid1() |
| UUID('a8098c1a-f86e-11da-bd1a-00112444be1e') |
| |
| >>> # make a UUID using an MD5 hash of a namespace UUID and a name |
| >>> uuid.uuid3(uuid.NAMESPACE_DNS, 'python.org') |
| UUID('6fa459ea-ee8a-3ca4-894e-db77e160355e') |
| |
| >>> # make a random UUID |
| >>> uuid.uuid4() |
| UUID('16fd2706-8baf-433b-82eb-8c7fada847da') |
| |
| >>> # make a UUID using a SHA-1 hash of a namespace UUID and a name |
| >>> uuid.uuid5(uuid.NAMESPACE_DNS, 'python.org') |
| UUID('886313e1-3b8a-5372-9b90-0c9aee199e5d') |
| |
| (Contributed by Ka-Ping Yee.) |
| |
| * The :mod:`weakref` module's :class:`WeakKeyDictionary` and |
| :class:`WeakValueDictionary` types gained new methods for iterating over the |
| weak references contained in the dictionary. :meth:`iterkeyrefs` and |
| :meth:`keyrefs` methods were added to :class:`WeakKeyDictionary`, and |
| :meth:`itervaluerefs` and :meth:`valuerefs` were added to |
| :class:`WeakValueDictionary`. (Contributed by Fred L. Drake, Jr.) |
| |
| * The :mod:`webbrowser` module received a number of enhancements. It's now |
| usable as a script with ``python -m webbrowser``, taking a URL as the argument; |
| there are a number of switches to control the behaviour (:option:`-n` for a new |
| browser window, :option:`-t` for a new tab). New module-level functions, |
| :func:`open_new` and :func:`open_new_tab`, were added to support this. The |
| module's :func:`open` function supports an additional feature, an *autoraise* |
| parameter that signals whether to raise the open window when possible. A number |
| of additional browsers were added to the supported list such as Firefox, Opera, |
| Konqueror, and elinks. (Contributed by Oleg Broytmann and Georg Brandl.) |
| |
| .. Patch #754022 |
| |
| * The :mod:`xmlrpclib` module now supports returning :class:`datetime` objects |
| for the XML-RPC date type. Supply ``use_datetime=True`` to the :func:`loads` |
| function or the :class:`Unmarshaller` class to enable this feature. (Contributed |
| by Skip Montanaro.) |
| |
| .. Patch 1120353 |
| |
| * The :mod:`zipfile` module now supports the ZIP64 version of the format, |
| meaning that a .zip archive can now be larger than 4 GiB and can contain |
| individual files larger than 4 GiB. (Contributed by Ronald Oussoren.) |
| |
| .. Patch 1446489 |
| |
| * The :mod:`zlib` module's :class:`Compress` and :class:`Decompress` objects now |
| support a :meth:`copy` method that makes a copy of the object's internal state |
| and returns a new :class:`Compress` or :class:`Decompress` object. |
| (Contributed by Chris AtLee.) |
| |
| .. Patch 1435422 |
| |
| .. ====================================================================== |
| |
| |
| .. _module-ctypes: |
| |
| The ctypes package |
| ------------------ |
| |
| The :mod:`ctypes` package, written by Thomas Heller, has been added to the |
| standard library. :mod:`ctypes` lets you call arbitrary functions in shared |
| libraries or DLLs. Long-time users may remember the :mod:`dl` module, which |
| provides functions for loading shared libraries and calling functions in them. |
| The :mod:`ctypes` package is much fancier. |
| |
| To load a shared library or DLL, you must create an instance of the |
| :class:`CDLL` class and provide the name or path of the shared library or DLL. |
| Once that's done, you can call arbitrary functions by accessing them as |
| attributes of the :class:`CDLL` object. :: |
| |
| import ctypes |
| |
| libc = ctypes.CDLL('libc.so.6') |
| result = libc.printf("Line of output\n") |
| |
| Type constructors for the various C types are provided: :func:`c_int`, |
| :func:`c_float`, :func:`c_double`, :func:`c_char_p` (equivalent to :ctype:`char |
| \*`), and so forth. Unlike Python's types, the C versions are all mutable; you |
| can assign to their :attr:`value` attribute to change the wrapped value. Python |
| integers and strings will be automatically converted to the corresponding C |
| types, but for other types you must call the correct type constructor. (And I |
| mean *must*; getting it wrong will often result in the interpreter crashing |
| with a segmentation fault.) |
| |
| You shouldn't use :func:`c_char_p` with a Python string when the C function will |
| be modifying the memory area, because Python strings are supposed to be |
| immutable; breaking this rule will cause puzzling bugs. When you need a |
| modifiable memory area, use :func:`create_string_buffer`:: |
| |
| s = "this is a string" |
| buf = ctypes.create_string_buffer(s) |
| libc.strfry(buf) |
| |
| C functions are assumed to return integers, but you can set the :attr:`restype` |
| attribute of the function object to change this:: |
| |
| >>> libc.atof('2.71828') |
| -1783957616 |
| >>> libc.atof.restype = ctypes.c_double |
| >>> libc.atof('2.71828') |
| 2.71828 |
| |
| :mod:`ctypes` also provides a wrapper for Python's C API as the |
| ``ctypes.pythonapi`` object. This object does *not* release the global |
| interpreter lock before calling a function, because the lock must be held when |
| calling into the interpreter's code. There's a :class:`py_object()` type |
| constructor that will create a :ctype:`PyObject \*` pointer. A simple usage:: |
| |
| import ctypes |
| |
| d = {} |
| ctypes.pythonapi.PyObject_SetItem(ctypes.py_object(d), |
| ctypes.py_object("abc"), ctypes.py_object(1)) |
| # d is now {'abc', 1}. |
| |
| Don't forget to use :class:`py_object()`; if it's omitted you end up with a |
| segmentation fault. |
| |
| :mod:`ctypes` has been around for a while, but people still write and |
| distribution hand-coded extension modules because you can't rely on |
| :mod:`ctypes` being present. Perhaps developers will begin to write Python |
| wrappers atop a library accessed through :mod:`ctypes` instead of extension |
| modules, now that :mod:`ctypes` is included with core Python. |
| |
| |
| .. seealso:: |
| |
| http://starship.python.net/crew/theller/ctypes/ |
| The ctypes web page, with a tutorial, reference, and FAQ. |
| |
| The documentation for the :mod:`ctypes` module. |
| |
| .. ====================================================================== |
| |
| |
| .. _module-etree: |
| |
| The ElementTree package |
| ----------------------- |
| |
| A subset of Fredrik Lundh's ElementTree library for processing XML has been |
| added to the standard library as :mod:`xml.etree`. The available modules are |
| :mod:`ElementTree`, :mod:`ElementPath`, and :mod:`ElementInclude` from |
| ElementTree 1.2.6. The :mod:`cElementTree` accelerator module is also |
| included. |
| |
| The rest of this section will provide a brief overview of using ElementTree. |
| Full documentation for ElementTree is available at http://effbot.org/zone |
| /element-index.htm. |
| |
| ElementTree represents an XML document as a tree of element nodes. The text |
| content of the document is stored as the :attr:`.text` and :attr:`.tail` |
| attributes of (This is one of the major differences between ElementTree and |
| the Document Object Model; in the DOM there are many different types of node, |
| including :class:`TextNode`.) |
| |
| The most commonly used parsing function is :func:`parse`, that takes either a |
| string (assumed to contain a filename) or a file-like object and returns an |
| :class:`ElementTree` instance:: |
| |
| from xml.etree import ElementTree as ET |
| |
| tree = ET.parse('ex-1.xml') |
| |
| feed = urllib.urlopen( |
| 'http://planet.python.org/rss10.xml') |
| tree = ET.parse(feed) |
| |
| Once you have an :class:`ElementTree` instance, you can call its :meth:`getroot` |
| method to get the root :class:`Element` node. |
| |
| There's also an :func:`XML` function that takes a string literal and returns an |
| :class:`Element` node (not an :class:`ElementTree`). This function provides a |
| tidy way to incorporate XML fragments, approaching the convenience of an XML |
| literal:: |
| |
| svg = ET.XML("""<svg width="10px" version="1.0"> |
| </svg>""") |
| svg.set('height', '320px') |
| svg.append(elem1) |
| |
| Each XML element supports some dictionary-like and some list-like access |
| methods. Dictionary-like operations are used to access attribute values, and |
| list-like operations are used to access child nodes. |
| |
| +-------------------------------+--------------------------------------------+ |
| | Operation | Result | |
| +===============================+============================================+ |
| | ``elem[n]`` | Returns n'th child element. | |
| +-------------------------------+--------------------------------------------+ |
| | ``elem[m:n]`` | Returns list of m'th through n'th child | |
| | | elements. | |
| +-------------------------------+--------------------------------------------+ |
| | ``len(elem)`` | Returns number of child elements. | |
| +-------------------------------+--------------------------------------------+ |
| | ``list(elem)`` | Returns list of child elements. | |
| +-------------------------------+--------------------------------------------+ |
| | ``elem.append(elem2)`` | Adds *elem2* as a child. | |
| +-------------------------------+--------------------------------------------+ |
| | ``elem.insert(index, elem2)`` | Inserts *elem2* at the specified location. | |
| +-------------------------------+--------------------------------------------+ |
| | ``del elem[n]`` | Deletes n'th child element. | |
| +-------------------------------+--------------------------------------------+ |
| | ``elem.keys()`` | Returns list of attribute names. | |
| +-------------------------------+--------------------------------------------+ |
| | ``elem.get(name)`` | Returns value of attribute *name*. | |
| +-------------------------------+--------------------------------------------+ |
| | ``elem.set(name, value)`` | Sets new value for attribute *name*. | |
| +-------------------------------+--------------------------------------------+ |
| | ``elem.attrib`` | Retrieves the dictionary containing | |
| | | attributes. | |
| +-------------------------------+--------------------------------------------+ |
| | ``del elem.attrib[name]`` | Deletes attribute *name*. | |
| +-------------------------------+--------------------------------------------+ |
| |
| Comments and processing instructions are also represented as :class:`Element` |
| nodes. To check if a node is a comment or processing instructions:: |
| |
| if elem.tag is ET.Comment: |
| ... |
| elif elem.tag is ET.ProcessingInstruction: |
| ... |
| |
| To generate XML output, you should call the :meth:`ElementTree.write` method. |
| Like :func:`parse`, it can take either a string or a file-like object:: |
| |
| # Encoding is US-ASCII |
| tree.write('output.xml') |
| |
| # Encoding is UTF-8 |
| f = open('output.xml', 'w') |
| tree.write(f, encoding='utf-8') |
| |
| (Caution: the default encoding used for output is ASCII. For general XML work, |
| where an element's name may contain arbitrary Unicode characters, ASCII isn't a |
| very useful encoding because it will raise an exception if an element's name |
| contains any characters with values greater than 127. Therefore, it's best to |
| specify a different encoding such as UTF-8 that can handle any Unicode |
| character.) |
| |
| This section is only a partial description of the ElementTree interfaces. Please |
| read the package's official documentation for more details. |
| |
| |
| .. seealso:: |
| |
| http://effbot.org/zone/element-index.htm |
| Official documentation for ElementTree. |
| |
| .. ====================================================================== |
| |
| |
| .. _module-hashlib: |
| |
| The hashlib package |
| ------------------- |
| |
| A new :mod:`hashlib` module, written by Gregory P. Smith, has been added to |
| replace the :mod:`md5` and :mod:`sha` modules. :mod:`hashlib` adds support for |
| additional secure hashes (SHA-224, SHA-256, SHA-384, and SHA-512). When |
| available, the module uses OpenSSL for fast platform optimized implementations |
| of algorithms. |
| |
| The old :mod:`md5` and :mod:`sha` modules still exist as wrappers around hashlib |
| to preserve backwards compatibility. The new module's interface is very close |
| to that of the old modules, but not identical. The most significant difference |
| is that the constructor functions for creating new hashing objects are named |
| differently. :: |
| |
| # Old versions |
| h = md5.md5() |
| h = md5.new() |
| |
| # New version |
| h = hashlib.md5() |
| |
| # Old versions |
| h = sha.sha() |
| h = sha.new() |
| |
| # New version |
| h = hashlib.sha1() |
| |
| # Hash that weren't previously available |
| h = hashlib.sha224() |
| h = hashlib.sha256() |
| h = hashlib.sha384() |
| h = hashlib.sha512() |
| |
| # Alternative form |
| h = hashlib.new('md5') # Provide algorithm as a string |
| |
| Once a hash object has been created, its methods are the same as before: |
| :meth:`update(string)` hashes the specified string into the current digest |
| state, :meth:`digest` and :meth:`hexdigest` return the digest value as a binary |
| string or a string of hex digits, and :meth:`copy` returns a new hashing object |
| with the same digest state. |
| |
| |
| .. seealso:: |
| |
| The documentation for the :mod:`hashlib` module. |
| |
| .. ====================================================================== |
| |
| |
| .. _module-sqlite: |
| |
| The sqlite3 package |
| ------------------- |
| |
| The pysqlite module (http://www.pysqlite.org), a wrapper for the SQLite embedded |
| database, has been added to the standard library under the package name |
| :mod:`sqlite3`. |
| |
| SQLite is a C library that provides a lightweight disk-based database that |
| doesn't require a separate server process and allows accessing the database |
| using a nonstandard variant of the SQL query language. Some applications can use |
| SQLite for internal data storage. It's also possible to prototype an |
| application using SQLite and then port the code to a larger database such as |
| PostgreSQL or Oracle. |
| |
| pysqlite was written by Gerhard Häring and provides a SQL interface compliant |
| with the DB-API 2.0 specification described by :pep:`249`. |
| |
| If you're compiling the Python source yourself, note that the source tree |
| doesn't include the SQLite code, only the wrapper module. You'll need to have |
| the SQLite libraries and headers installed before compiling Python, and the |
| build process will compile the module when the necessary headers are available. |
| |
| To use the module, you must first create a :class:`Connection` object that |
| represents the database. Here the data will be stored in the |
| :file:`/tmp/example` file:: |
| |
| conn = sqlite3.connect('/tmp/example') |
| |
| You can also supply the special name ``:memory:`` to create a database in RAM. |
| |
| Once you have a :class:`Connection`, you can create a :class:`Cursor` object |
| and call its :meth:`execute` method to perform SQL commands:: |
| |
| c = conn.cursor() |
| |
| # Create table |
| c.execute('''create table stocks |
| (date text, trans text, symbol text, |
| qty real, price real)''') |
| |
| # Insert a row of data |
| c.execute("""insert into stocks |
| values ('2006-01-05','BUY','RHAT',100,35.14)""") |
| |
| Usually your SQL operations will need to use values from Python variables. You |
| shouldn't assemble your query using Python's string operations because doing so |
| is insecure; it makes your program vulnerable to an SQL injection attack. |
| |
| Instead, use the DB-API's parameter substitution. Put ``?`` as a placeholder |
| wherever you want to use a value, and then provide a tuple of values as the |
| second argument to the cursor's :meth:`execute` method. (Other database modules |
| may use a different placeholder, such as ``%s`` or ``:1``.) For example:: |
| |
| # Never do this -- insecure! |
| symbol = 'IBM' |
| c.execute("... where symbol = '%s'" % symbol) |
| |
| # Do this instead |
| t = (symbol,) |
| c.execute('select * from stocks where symbol=?', t) |
| |
| # Larger example |
| for t in (('2006-03-28', 'BUY', 'IBM', 1000, 45.00), |
| ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00), |
| ('2006-04-06', 'SELL', 'IBM', 500, 53.00), |
| ): |
| c.execute('insert into stocks values (?,?,?,?,?)', t) |
| |
| To retrieve data after executing a SELECT statement, you can either treat the |
| cursor as an iterator, call the cursor's :meth:`fetchone` method to retrieve a |
| single matching row, or call :meth:`fetchall` to get a list of the matching |
| rows. |
| |
| This example uses the iterator form:: |
| |
| >>> c = conn.cursor() |
| >>> c.execute('select * from stocks order by price') |
| >>> for row in c: |
| ... print row |
| ... |
| (u'2006-01-05', u'BUY', u'RHAT', 100, 35.140000000000001) |
| (u'2006-03-28', u'BUY', u'IBM', 1000, 45.0) |
| (u'2006-04-06', u'SELL', u'IBM', 500, 53.0) |
| (u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0) |
| >>> |
| |
| For more information about the SQL dialect supported by SQLite, see |
| http://www.sqlite.org. |
| |
| |
| .. seealso:: |
| |
| http://www.pysqlite.org |
| The pysqlite web page. |
| |
| http://www.sqlite.org |
| The SQLite web page; the documentation describes the syntax and the available |
| data types for the supported SQL dialect. |
| |
| The documentation for the :mod:`sqlite3` module. |
| |
| :pep:`249` - Database API Specification 2.0 |
| PEP written by Marc-André Lemburg. |
| |
| .. ====================================================================== |
| |
| |
| .. _module-wsgiref: |
| |
| The wsgiref package |
| ------------------- |
| |
| The Web Server Gateway Interface (WSGI) v1.0 defines a standard interface |
| between web servers and Python web applications and is described in :pep:`333`. |
| The :mod:`wsgiref` package is a reference implementation of the WSGI |
| specification. |
| |
| .. XXX should this be in a PEP 333 section instead? |
| |
| The package includes a basic HTTP server that will run a WSGI application; this |
| server is useful for debugging but isn't intended for production use. Setting |
| up a server takes only a few lines of code:: |
| |
| from wsgiref import simple_server |
| |
| wsgi_app = ... |
| |
| host = '' |
| port = 8000 |
| httpd = simple_server.make_server(host, port, wsgi_app) |
| httpd.serve_forever() |
| |
| .. XXX discuss structure of WSGI applications? |
| .. XXX provide an example using Django or some other framework? |
| |
| |
| .. seealso:: |
| |
| http://www.wsgi.org |
| A central web site for WSGI-related resources. |
| |
| :pep:`333` - Python Web Server Gateway Interface v1.0 |
| PEP written by Phillip J. Eby. |
| |
| .. ====================================================================== |
| |
| |
| .. _build-api: |
| |
| Build and C API Changes |
| ======================= |
| |
| Changes to Python's build process and to the C API include: |
| |
| * The Python source tree was converted from CVS to Subversion, in a complex |
| migration procedure that was supervised and flawlessly carried out by Martin von |
| Löwis. The procedure was developed as :pep:`347`. |
| |
| * Coverity, a company that markets a source code analysis tool called Prevent, |
| provided the results of their examination of the Python source code. The |
| analysis found about 60 bugs that were quickly fixed. Many of the bugs were |
| refcounting problems, often occurring in error-handling code. See |
| http://scan.coverity.com for the statistics. |
| |
| * The largest change to the C API came from :pep:`353`, which modifies the |
| interpreter to use a :ctype:`Py_ssize_t` type definition instead of |
| :ctype:`int`. See the earlier section :ref:`pep-353` for a discussion of this |
| change. |
| |
| * The design of the bytecode compiler has changed a great deal, no longer |
| generating bytecode by traversing the parse tree. Instead the parse tree is |
| converted to an abstract syntax tree (or AST), and it is the abstract syntax |
| tree that's traversed to produce the bytecode. |
| |
| It's possible for Python code to obtain AST objects by using the |
| :func:`compile` built-in and specifying ``_ast.PyCF_ONLY_AST`` as the value of |
| the *flags* parameter:: |
| |
| from _ast import PyCF_ONLY_AST |
| ast = compile("""a=0 |
| for i in range(10): |
| a += i |
| """, "<string>", 'exec', PyCF_ONLY_AST) |
| |
| assignment = ast.body[0] |
| for_loop = ast.body[1] |
| |
| No official documentation has been written for the AST code yet, but :pep:`339` |
| discusses the design. To start learning about the code, read the definition of |
| the various AST nodes in :file:`Parser/Python.asdl`. A Python script reads this |
| file and generates a set of C structure definitions in |
| :file:`Include/Python-ast.h`. The :cfunc:`PyParser_ASTFromString` and |
| :cfunc:`PyParser_ASTFromFile`, defined in :file:`Include/pythonrun.h`, take |
| Python source as input and return the root of an AST representing the contents. |
| This AST can then be turned into a code object by :cfunc:`PyAST_Compile`. For |
| more information, read the source code, and then ask questions on python-dev. |
| |
| The AST code was developed under Jeremy Hylton's management, and implemented by |
| (in alphabetical order) Brett Cannon, Nick Coghlan, Grant Edwards, John |
| Ehresman, Kurt Kaiser, Neal Norwitz, Tim Peters, Armin Rigo, and Neil |
| Schemenauer, plus the participants in a number of AST sprints at conferences |
| such as PyCon. |
| |
| .. List of names taken from Jeremy's python-dev post at |
| .. http://mail.python.org/pipermail/python-dev/2005-October/057500.html |
| |
| * Evan Jones's patch to obmalloc, first described in a talk at PyCon DC 2005, |
| was applied. Python 2.4 allocated small objects in 256K-sized arenas, but never |
| freed arenas. With this patch, Python will free arenas when they're empty. The |
| net effect is that on some platforms, when you allocate many objects, Python's |
| memory usage may actually drop when you delete them and the memory may be |
| returned to the operating system. (Implemented by Evan Jones, and reworked by |
| Tim Peters.) |
| |
| Note that this change means extension modules must be more careful when |
| allocating memory. Python's API has many different functions for allocating |
| memory that are grouped into families. For example, :cfunc:`PyMem_Malloc`, |
| :cfunc:`PyMem_Realloc`, and :cfunc:`PyMem_Free` are one family that allocates |
| raw memory, while :cfunc:`PyObject_Malloc`, :cfunc:`PyObject_Realloc`, and |
| :cfunc:`PyObject_Free` are another family that's supposed to be used for |
| creating Python objects. |
| |
| Previously these different families all reduced to the platform's |
| :cfunc:`malloc` and :cfunc:`free` functions. This meant it didn't matter if |
| you got things wrong and allocated memory with the :cfunc:`PyMem` function but |
| freed it with the :cfunc:`PyObject` function. With 2.5's changes to obmalloc, |
| these families now do different things and mismatches will probably result in a |
| segfault. You should carefully test your C extension modules with Python 2.5. |
| |
| * The built-in set types now have an official C API. Call :cfunc:`PySet_New` |
| and :cfunc:`PyFrozenSet_New` to create a new set, :cfunc:`PySet_Add` and |
| :cfunc:`PySet_Discard` to add and remove elements, and :cfunc:`PySet_Contains` |
| and :cfunc:`PySet_Size` to examine the set's state. (Contributed by Raymond |
| Hettinger.) |
| |
| * C code can now obtain information about the exact revision of the Python |
| interpreter by calling the :cfunc:`Py_GetBuildInfo` function that returns a |
| string of build information like this: ``"trunk:45355:45356M, Apr 13 2006, |
| 07:42:19"``. (Contributed by Barry Warsaw.) |
| |
| * Two new macros can be used to indicate C functions that are local to the |
| current file so that a faster calling convention can be used. |
| :cfunc:`Py_LOCAL(type)` declares the function as returning a value of the |
| specified *type* and uses a fast-calling qualifier. |
| :cfunc:`Py_LOCAL_INLINE(type)` does the same thing and also requests the |
| function be inlined. If :cfunc:`PY_LOCAL_AGGRESSIVE` is defined before |
| :file:`python.h` is included, a set of more aggressive optimizations are enabled |
| for the module; you should benchmark the results to find out if these |
| optimizations actually make the code faster. (Contributed by Fredrik Lundh at |
| the NeedForSpeed sprint.) |
| |
| * :cfunc:`PyErr_NewException(name, base, dict)` can now accept a tuple of base |
| classes as its *base* argument. (Contributed by Georg Brandl.) |
| |
| * The :cfunc:`PyErr_Warn` function for issuing warnings is now deprecated in |
| favour of :cfunc:`PyErr_WarnEx(category, message, stacklevel)` which lets you |
| specify the number of stack frames separating this function and the caller. A |
| *stacklevel* of 1 is the function calling :cfunc:`PyErr_WarnEx`, 2 is the |
| function above that, and so forth. (Added by Neal Norwitz.) |
| |
| * The CPython interpreter is still written in C, but the code can now be |
| compiled with a C++ compiler without errors. (Implemented by Anthony Baxter, |
| Martin von Löwis, Skip Montanaro.) |
| |
| * The :cfunc:`PyRange_New` function was removed. It was never documented, never |
| used in the core code, and had dangerously lax error checking. In the unlikely |
| case that your extensions were using it, you can replace it by something like |
| the following:: |
| |
| range = PyObject_CallFunction((PyObject*) &PyRange_Type, "lll", |
| start, stop, step); |
| |
| .. ====================================================================== |
| |
| |
| .. _ports: |
| |
| Port-Specific Changes |
| --------------------- |
| |
| * MacOS X (10.3 and higher): dynamic loading of modules now uses the |
| :cfunc:`dlopen` function instead of MacOS-specific functions. |
| |
| * MacOS X: an :option:`--enable-universalsdk` switch was added to the |
| :program:`configure` script that compiles the interpreter as a universal binary |
| able to run on both PowerPC and Intel processors. (Contributed by Ronald |
| Oussoren; :issue:`2573`.) |
| |
| * Windows: :file:`.dll` is no longer supported as a filename extension for |
| extension modules. :file:`.pyd` is now the only filename extension that will be |
| searched for. |
| |
| .. ====================================================================== |
| |
| |
| .. _porting: |
| |
| Porting to Python 2.5 |
| ===================== |
| |
| This section lists previously described changes that may require changes to your |
| code: |
| |
| * ASCII is now the default encoding for modules. It's now a syntax error if a |
| module contains string literals with 8-bit characters but doesn't have an |
| encoding declaration. In Python 2.4 this triggered a warning, not a syntax |
| error. |
| |
| * Previously, the :attr:`gi_frame` attribute of a generator was always a frame |
| object. Because of the :pep:`342` changes described in section :ref:`pep-342`, |
| it's now possible for :attr:`gi_frame` to be ``None``. |
| |
| * A new warning, :class:`UnicodeWarning`, is triggered when you attempt to |
| compare a Unicode string and an 8-bit string that can't be converted to Unicode |
| using the default ASCII encoding. Previously such comparisons would raise a |
| :class:`UnicodeDecodeError` exception. |
| |
| * Library: the :mod:`csv` module is now stricter about multi-line quoted fields. |
| If your files contain newlines embedded within fields, the input should be split |
| into lines in a manner which preserves the newline characters. |
| |
| * Library: the :mod:`locale` module's :func:`format` function's would |
| previously accept any string as long as no more than one %char specifier |
| appeared. In Python 2.5, the argument must be exactly one %char specifier with |
| no surrounding text. |
| |
| * Library: The :mod:`pickle` and :mod:`cPickle` modules no longer accept a |
| return value of ``None`` from the :meth:`__reduce__` method; the method must |
| return a tuple of arguments instead. The modules also no longer accept the |
| deprecated *bin* keyword parameter. |
| |
| * Library: The :mod:`SimpleXMLRPCServer` and :mod:`DocXMLRPCServer` classes now |
| have a :attr:`rpc_paths` attribute that constrains XML-RPC operations to a |
| limited set of URL paths; the default is to allow only ``'/'`` and ``'/RPC2'``. |
| Setting :attr:`rpc_paths` to ``None`` or an empty tuple disables this path |
| checking. |
| |
| * C API: Many functions now use :ctype:`Py_ssize_t` instead of :ctype:`int` to |
| allow processing more data on 64-bit machines. Extension code may need to make |
| the same change to avoid warnings and to support 64-bit machines. See the |
| earlier section :ref:`pep-353` for a discussion of this change. |
| |
| * C API: The obmalloc changes mean that you must be careful to not mix usage |
| of the :cfunc:`PyMem_\*` and :cfunc:`PyObject_\*` families of functions. Memory |
| allocated with one family's :cfunc:`\*_Malloc` must be freed with the |
| corresponding family's :cfunc:`\*_Free` function. |
| |
| .. ====================================================================== |
| |
| |
| .. _acks: |
| |
| Acknowledgements |
| ================ |
| |
| The author would like to thank the following people for offering suggestions, |
| corrections and assistance with various drafts of this article: Georg Brandl, |
| Nick Coghlan, Phillip J. Eby, Lars Gustäbel, Raymond Hettinger, Ralf W. Grosse- |
| Kunstleve, Kent Johnson, Iain Lowe, Martin von Löwis, Fredrik Lundh, Andrew |
| McNamara, Skip Montanaro, Gustavo Niemeyer, Paul Prescod, James Pryor, Mike |
| Rovner, Scott Weikart, Barry Warsaw, Thomas Wouters. |
| |