blob: 5bef1ac4cc24919f44067e629cac58df4a8fea84 [file] [log] [blame]
Raymond Hettingerd73be672010-12-18 10:48:26 +00001****************************
Raymond Hettingerf558ddd2009-06-28 21:37:08 +00002 What's New In Python 3.2
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00003****************************
4
5:Author: Raymond Hettinger
6:Release: |release|
7:Date: |today|
8
9.. $Id$
10 Rules for maintenance:
11
12 * Anyone can add text to this document. Do not spend very much time
13 on the wording of your changes, because your text will probably
Raymond Hettinger6f04adc2010-12-04 22:56:25 +000014 get rewritten.
Raymond Hettinger6e6565b2009-06-28 20:56:11 +000015
16 * The maintainer will go through Misc/NEWS periodically and add
17 changes; it's therefore more important to add your changes to
18 Misc/NEWS than to this file.
19
20 * This is not a complete list of every single change; completeness
21 is the purpose of Misc/NEWS. Some changes I consider too small
22 or esoteric to include. If such a change is added to the text,
23 I'll just remove it. (This is another reason you shouldn't spend
24 too much time on writing your addition.)
25
26 * If you want to draw your new text to the attention of the
27 maintainer, add 'XXX' to the beginning of the paragraph or
28 section.
29
30 * It's OK to just add a fragmentary note about a change. For
31 example: "XXX Describe the transmogrify() function added to the
32 socket module." The maintainer will research the change and
33 write the necessary text.
34
35 * You can comment out your additions if you like, but it's not
36 necessary (especially when a final release is some months away).
37
38 * Credit the author of a patch or bugfix. Just the name is
Georg Brandlda0a2112010-09-05 11:28:33 +000039 sufficient; the e-mail address isn't necessary. It's helpful to
40 add the issue number:
Raymond Hettinger6e6565b2009-06-28 20:56:11 +000041
Éric Araujo4234ad42010-09-05 17:32:25 +000042 XXX Describe the transmogrify() function added to the socket
43 module.
44
45 (Contributed by P.Y. Developer; :issue:`12345`.)
Raymond Hettinger6e6565b2009-06-28 20:56:11 +000046
47 This saves the maintainer the effort of going through the SVN log
48 when researching a change.
49
Raymond Hettingerffad35e2010-12-14 21:12:03 +000050This article explains the new features in Python 3.2 as compared to 3.1. It
51focuses on a few highlights and gives a few examples. For full details, see the
52:source:`Misc/NEWS <Misc/NEWS>` file.
Raymond Hettinger2c1ecc32010-12-07 09:55:02 +000053
Raymond Hettinger6778fa92010-12-21 20:09:55 +000054.. seealso::
55
56 :pep:`392` - Python 3.2 Release Schedule
Raymond Hettinger6e6565b2009-06-28 20:56:11 +000057
Raymond Hettingerca904be2011-01-18 00:02:40 +000058
Martin v. Löwis932e49e2010-12-04 13:49:32 +000059PEP 384: Defining a Stable ABI
Martin v. Löwis4d0d4712010-12-03 20:14:31 +000060==============================
61
62In the past, extension modules built for one Python version were often
63not usable with other Python versions. Particularly on Windows, every
64feature release of Python required rebuilding all extension modules that
65one wanted to use. This requirement was the result of the free access to
66Python interpreter internals that extension modules could use.
67
68With Python 3.2, an alternative approach becomes available: extension
Raymond Hettinger6f04adc2010-12-04 22:56:25 +000069modules which restrict themselves to a limited API (by defining
Martin v. Löwis4d0d4712010-12-03 20:14:31 +000070Py_LIMITED_API) cannot use many of the internals, but are constrained
71to a set of API functions that are promised to be stable for several
72releases. As a consequence, extension modules built for 3.2 in that
73mode will also work with 3.3, 3.4, and so on. Extension modules that
74make use of details of memory structures can still be built, but will
75need to be recompiled for every feature release.
76
Raymond Hettinger6f04adc2010-12-04 22:56:25 +000077.. seealso::
78
Georg Brandl65b2eb92010-12-05 11:42:38 +000079 :pep:`384` - Defining a Stable ABI
Raymond Hettinger2c1ecc32010-12-07 09:55:02 +000080 PEP written by Martin von Löwis.
Raymond Hettinger6f04adc2010-12-04 22:56:25 +000081
Raymond Hettingerca904be2011-01-18 00:02:40 +000082
Raymond Hettingera5a35542010-12-05 00:39:18 +000083PEP 389: Argparse Command Line Parsing Module
84=============================================
85
86A new module for command line parsing, :mod:`argparse`, was introduced to
87overcome the limitations of :mod:`optparse` which did not provide support for
Raymond Hettinger677e10a2010-12-07 06:45:30 +000088positional arguments (not just options), subcommands, required options and other
Raymond Hettinger413abbc2010-12-05 07:06:47 +000089common patterns of specifying and validating options.
Raymond Hettingera5a35542010-12-05 00:39:18 +000090
Raymond Hettingerc8a16862011-01-18 09:01:34 +000091This module has already has widespread success in the community as a
Raymond Hettingerb1ff4022010-12-08 11:19:45 +000092third-party module. Being more fully featured than its predecessor, the
93:mod:`argparse` module is now the preferred module for command-line processing.
94The older module is still being kept available because of the substantial amount
95of legacy code that depends on it.
Raymond Hettingera5a35542010-12-05 00:39:18 +000096
Raymond Hettinger677e10a2010-12-07 06:45:30 +000097Here's an annotated example parser showing features like limiting results to a
98set of choices, specifying a *metavar* in the help screen, validating that one
Raymond Hettinger68f1e8d2010-12-07 09:24:30 +000099or more positional arguments is present, and making a required option::
Raymond Hettinger677e10a2010-12-07 06:45:30 +0000100
101 import argparse
102 parser = argparse.ArgumentParser(
103 description = 'Manage servers', # main description for help
104 epilog = 'Tested on Solaris and Linux') # displayed after help
105 parser.add_argument('action', # argument name
Raymond Hettinger92977092011-01-16 09:18:59 +0000106 choices = ['deploy', 'start', 'stop'], # three allowed values
Raymond Hettinger677e10a2010-12-07 06:45:30 +0000107 help = 'action on each target') # help msg
108 parser.add_argument('targets',
109 metavar = 'HOSTNAME', # var name used in help msg
Raymond Hettinger92977092011-01-16 09:18:59 +0000110 nargs = '+', # require one or more targets
Raymond Hettinger677e10a2010-12-07 06:45:30 +0000111 help = 'url for target machines') # help msg explanation
112 parser.add_argument('-u', '--user', # -u or --user option
Georg Brandl52a43b52011-01-16 09:11:45 +0000113 required = True, # make it a required argument
Raymond Hettinger677e10a2010-12-07 06:45:30 +0000114 help = 'login as user')
115
116Example of calling the parser on a command string::
117
118 >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
119 >>> result = parser.parse_args(cmd.split())
Raymond Hettinger677e10a2010-12-07 06:45:30 +0000120 >>> result.action
121 'deploy'
122 >>> result.targets
123 ['sneezy.example.com', 'sleepy.example.com']
124 >>> result.user
125 'skycaptain'
126
127Example of the parser's automatically generated help::
128
129 >>> parser.parse_args('-h'.split())
130
Raymond Hettinger3fcf0022010-12-08 01:13:53 +0000131 usage: manage_cloud.py [-h] -u USER
132 {deploy,start,stop} HOSTNAME [HOSTNAME ...]
Raymond Hettinger677e10a2010-12-07 06:45:30 +0000133
134 Manage servers
135
136 positional arguments:
137 {deploy,start,stop} action on each target
138 HOSTNAME url for target machines
139
140 optional arguments:
141 -h, --help show this help message and exit
142 -u USER, --user USER login as user
143
144 Tested on Solaris and Linux
145
Raymond Hettingerb1ff4022010-12-08 11:19:45 +0000146An especially nice :mod:`argparse` feature is the ability to define subparsers,
147each with their own argument patterns and help displays::
148
149 import argparse
150 parser = argparse.ArgumentParser(prog='HELM')
151 subparsers = parser.add_subparsers()
152
153 parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup
Raymond Hettingerbb9686f2010-12-16 00:53:05 +0000154 parser_l.add_argument('-m', '--missiles', action='store_true')
Raymond Hettingerb1ff4022010-12-08 11:19:45 +0000155 parser_l.add_argument('-t', '--torpedos', action='store_true')
156
Raymond Hettinger3094ed82010-12-18 09:41:32 +0000157 parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup
158 aliases=('steer', 'turn')) # equivalent names
Raymond Hettingerb1ff4022010-12-08 11:19:45 +0000159 parser_m.add_argument('-c', '--course', type=int, required=True)
160 parser_m.add_argument('-s', '--speed', type=int, default=0)
161
162 $ ./helm.py --help # top level help (launch and move)
163 $ ./helm.py launch --help # help for launch options
164 $ ./helm.py launch --missiles # set missiles=True and torpedos=False
Raymond Hettinger3094ed82010-12-18 09:41:32 +0000165 $ ./helm.py steer --course 180 --speed 5 # set movement parameters
Raymond Hettingera5a35542010-12-05 00:39:18 +0000166
167.. seealso::
168
169 :pep:`389` - New Command Line Parsing Module
170 PEP written by Steven Bethard.
171
Raymond Hettingerbe9994e2011-01-19 08:44:33 +0000172 :ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`.
Raymond Hettinger677e10a2010-12-07 06:45:30 +0000173
Raymond Hettinger6e6565b2009-06-28 20:56:11 +0000174
Éric Araujo4234ad42010-09-05 17:32:25 +0000175PEP 391: Dictionary Based Configuration for Logging
176====================================================
Raymond Hettingeref2335c2010-09-05 08:35:38 +0000177
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000178The :mod:`logging` module provided two kinds of configuration, one style with
179function calls for each option or another style driven by an external file saved
180in a :mod:`ConfigParser` format. Those options did not provide the flexibility
Georg Brandl9e75cad2010-09-06 06:45:47 +0000181to create configurations from JSON or YAML files, nor did they support
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000182incremental configuration, which is needed for specifying logger options from a
183command line.
Raymond Hettingeref2335c2010-09-05 08:35:38 +0000184
185To support a more flexible style, the module now offers
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000186:func:`logging.config.dictConfig` for specifying logging configuration with
187plain Python dictionaries. The configuration options include formatters,
188handlers, filters, and loggers. Here's a working example of a configuration
189dictionary::
Raymond Hettingeref2335c2010-09-05 08:35:38 +0000190
Georg Brandlda0a2112010-09-05 11:28:33 +0000191 {"version": 1,
192 "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
193 "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"},
194 },
195 "handlers": {"console": {
196 "class": "logging.StreamHandler",
197 "formatter": "brief",
198 "level": "INFO",
199 "stream": "ext://sys.stdout"},
200 "console_priority": {
201 "class": "logging.StreamHandler",
202 "formatter": "full",
203 "level": "ERROR",
204 "stream": "ext://sys.stderr"},
205 },
206 "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}
Raymond Hettingeref2335c2010-09-05 08:35:38 +0000207
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000208
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000209If that dictionary is stored in a file called :file:`conf.json`, it can be
210loaded and called with code like this::
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000211
Raymond Hettingerca904be2011-01-18 00:02:40 +0000212 >>> import json, logging.config
213 >>> with open('conf.json', 'rb') as f:
214 conf = json.load(f)
215 >>> logging.config.dictConfig(conf)
216 >>> logging.info("Transaction completed normally")
217 >>> logging.critical("Abnormal termination")
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000218
Raymond Hettingeref2335c2010-09-05 08:35:38 +0000219.. seealso::
220
221 :pep:`391` - Dictionary Based Configuration for Logging
222 PEP written by Vinay Sajip.
223
Raymond Hettingerca904be2011-01-18 00:02:40 +0000224
Georg Brandl97b20da2010-11-16 15:15:29 +0000225PEP 3148: The ``concurrent.futures`` module
226============================================
227
Raymond Hettingerc8a16862011-01-18 09:01:34 +0000228Code for creating and managing concurrency is being collected in a new top-level
Raymond Hettinger6f04adc2010-12-04 22:56:25 +0000229namespace, *concurrent*. Its first member is a *futures* package which provides
Raymond Hettingerc8a16862011-01-18 09:01:34 +0000230a uniform high-level interface for managing threads and processes.
Raymond Hettinger6f04adc2010-12-04 22:56:25 +0000231
232The design for :mod:`concurrent.futures` was inspired by
233*java.util.concurrent.package*. In that model, a running call and its result
Raymond Hettingerc136b042011-01-18 07:15:39 +0000234are represented by a :class:`~concurrent.futures.Future` object that abstracts
Raymond Hettinger6f04adc2010-12-04 22:56:25 +0000235features common to threads, processes, and remote procedure calls. That object
236supports status checks (running or done), timeouts, cancellations, adding
Raymond Hettinger24a09412010-12-08 06:50:02 +0000237callbacks, and access to results or exceptions.
Raymond Hettinger6f04adc2010-12-04 22:56:25 +0000238
239The primary offering of the new module is a pair of executor classes for
240launching and managing calls. The goal of the executors is to make it easier to
241use existing tools for making parallel calls. They save the effort needed to
242setup a pool of resources, launch the calls, create a results queue, add
243time-out handling, and limit the total number of threads, processes, or remote
Raymond Hettingerc269ae82010-12-05 01:01:52 +0000244procedure calls.
Raymond Hettinger6f04adc2010-12-04 22:56:25 +0000245
246Ideally, each application should share a single executor across multiple
247components so that process and thread limits can be centrally managed. This
248solves the design challenge that arises when each component has its own
249competing strategy for resource management.
250
Raymond Hettingerb1055192010-12-08 06:42:41 +0000251Both classes share a common interface with three methods:
252:meth:`~concurrent.futures.Executor.submit` for scheduling a callable and
253returning a :class:`~concurrent.futures.Future` object;
254:meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls
Raymond Hettinger83d80792010-12-08 06:48:33 +0000255at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing
Raymond Hettingerc8a16862011-01-18 09:01:34 +0000256resources. The class is a :term:`context manager` and can be used in a
Raymond Hettinger83d80792010-12-08 06:48:33 +0000257:keyword:`with` statement to assure that resources are automatically released
258when currently pending futures are done executing.
Raymond Hettinger6f04adc2010-12-04 22:56:25 +0000259
Raymond Hettingerb1055192010-12-08 06:42:41 +0000260A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a
Raymond Hettinger83d80792010-12-08 06:48:33 +0000261launch of four parallel threads for copying files::
Raymond Hettingerb1055192010-12-08 06:42:41 +0000262
263 import shutil
264 with ThreadPoolExecutor(max_workers=4) as e:
265 e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
266 e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
267 e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
268 e.submit(shutil.copy, 'src3.txt', 'dest4.txt')
269
Raymond Hettinger6f04adc2010-12-04 22:56:25 +0000270.. seealso::
271
Raymond Hettingera5a35542010-12-05 00:39:18 +0000272 :pep:`3148` - Futures -- Execute Computations Asynchronously
Andrew M. Kuchling42877fe2010-12-15 02:37:01 +0000273 PEP written by Brian Quinlan.
Georg Brandl97b20da2010-11-16 15:15:29 +0000274
Raymond Hettinger83d80792010-12-08 06:48:33 +0000275 :ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an
276 example using threads to fetch multiple web pages in parallel.
277
278 :ref:`Code for computing prime numbers in
279 parallel<processpoolexecutor-example>`, an example demonstrating
280 :class:`~concurrent.futures.ProcessPoolExecutor`.
281
282
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000283PEP 3147: PYC Repository Directories
284=====================================
285
David Malcolm778645a2010-12-07 00:32:04 +0000286Python's scheme for caching bytecode in *.pyc* files did not work well in
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000287environments with multiple Python interpreters. If one interpreter encountered
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000288a cached file created by another interpreter, it would recompile the source and
289overwrite the cached file, thus losing the benefits of caching.
290
291The issue of "pyc fights" has become more pronounced as it has become
Éric Araujo4234ad42010-09-05 17:32:25 +0000292commonplace for Linux distributions to ship with multiple versions of Python.
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000293These conflicts also arise with CPython alternatives such as Unladen Swallow.
294
295To solve this problem, Python's import machinery has been extended to use
Éric Araujo4234ad42010-09-05 17:32:25 +0000296distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and
297Unladen Swallow each competing for a file called "mymodule.pyc", they will now
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000298look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and
Éric Araujo4234ad42010-09-05 17:32:25 +0000299"mymodule.unladen10.pyc". And to prevent all of these new files from
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000300cluttering source directories, the *pyc* files are now collected in a
301"__pycache__" directory stored under the package directory.
302
303Aside from the filenames and target directories, the new scheme has a few
304aspects that are visible to the programmer:
305
Georg Brandlda0a2112010-09-05 11:28:33 +0000306* Imported modules now have a :attr:`__cached__` attribute which stores the name
307 of the actual file that was imported:
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000308
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000309 >>> import collections
310 >>> collections.__cached__
311 'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000312
313* The tag that is unique to each interpreter is accessible from the :mod:`imp`
Georg Brandlda0a2112010-09-05 11:28:33 +0000314 module:
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000315
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000316 >>> import imp
317 >>> imp.get_tag()
318 'cpython-32'
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000319
320* Scripts that try to deduce source filename from the imported file now need to
321 be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc"
322 filename. Instead, use the new functions in the :mod:`imp` module:
323
Georg Brandlda0a2112010-09-05 11:28:33 +0000324 >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc')
325 'c:/py32/lib/collections.py'
326 >>> imp.cache_from_source('c:/py32/lib/collections.py')
327 'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000328
329* The :mod:`py_compile` and :mod:`compileall` modules have been updated to
Raymond Hettinger50307b62011-01-24 01:18:30 +0000330 reflect the new naming convention and target directory. The command-line
331 invocation of *compileall* has new command-line options include ``-i`` for
332 specifying a list of files and directories to compile, and ``-b`` which causes
333 bytecode files to be written to their legacy location rather than
334 *__pycache__*.
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000335
Raymond Hettinger1dcc84e2011-01-17 21:55:40 +0000336* The :mod:`importlib.abc` module has been updated with new :term:`abstract base
Raymond Hettinger66352d22011-01-17 22:33:11 +0000337 classes <abstract base class>` for the loading bytecode files. The obsolete
338 ABCs, :class:`~importlib.abc.PyLoader` and
Raymond Hettinger1dcc84e2011-01-17 21:55:40 +0000339 :class:`~importlib.abc.PyPycLoader`, have been deprecated (instructions on how
Raymond Hettinger66352d22011-01-17 22:33:11 +0000340 to stay Python 3.1 compatible are included with the documentation).
Brett Cannon83a682d2011-01-16 21:02:09 +0000341
Raymond Hettingerf95b1992010-09-04 23:53:24 +0000342.. seealso::
343
344 :pep:`3147` - PYC Repository Directories
345 PEP written by Barry Warsaw.
346
Georg Brandlda0a2112010-09-05 11:28:33 +0000347
Georg Brandl3ad46752010-12-05 07:59:29 +0000348PEP 3149: ABI Version Tagged .so Files
349======================================
Georg Brandlf11c6c42010-09-03 22:20:58 +0000350
Raymond Hettingerebea6fa2010-09-05 00:27:25 +0000351The PYC repository directory allows multiple bytecode cache files to be
352co-located. This PEP implements a similar mechanism for shared object files by
353giving them a common directory and distinct names for each version.
Georg Brandlf11c6c42010-09-03 22:20:58 +0000354
Raymond Hettingerebea6fa2010-09-05 00:27:25 +0000355The common directory is "pyshared" and the file names are made distinct by
356identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the
357major and minor version numbers, and optional build flags (such as "d" for
Éric Araujo4234ad42010-09-05 17:32:25 +0000358debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo",
Raymond Hettingerebea6fa2010-09-05 00:27:25 +0000359you may see these files when the distribution package is installed::
360
361 /usr/share/pyshared/foo.cpython-32m.so
362 /usr/share/pyshared/foo.cpython-33md.so
363
364In Python itself, the tags are accessible from functions in the :mod:`sysconfig`
365module::
366
367 >>> import sysconfig
368 >>> sysconfig.get_config_var('SOABI') # find the version tag
369 'cpython-32mu'
370 >>> sysconfig.get_config_var('SO') # find the full filename extension
371 'cpython-32mu.so'
372
373.. seealso::
374
375 :pep:`3149` - ABI Version Tagged .so Files
376 PEP written by Barry Warsaw.
Raymond Hettinger6e6565b2009-06-28 20:56:11 +0000377
Raymond Hettingerca904be2011-01-18 00:02:40 +0000378
Raymond Hettinger3df46212011-01-06 02:01:26 +0000379PEP 3333: Python Web Server Gateway Interface v1.0.1
380=====================================================
381
382This informational PEP clarifies how bytes/text issues are to be handled by the
383WGSI protocol. The challenge is that string handling in Python 3 is most
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000384conveniently handled with the :class:`str` type even though the HTTP protocol
Raymond Hettinger3df46212011-01-06 02:01:26 +0000385is itself bytes oriented.
386
387The PEP differentiates so-called *native strings* that are used for
388request/response headers and metadata versus *byte strings* which are used for
389the bodies of requests and responses.
390
391The *native strings* are always of type :class:`str` but are restricted to code
Georg Brandl52a43b52011-01-16 09:11:45 +0000392points between *U+0000* through *U+00FF* which are translatable to bytes using
Raymond Hettinger32e8fea2011-01-07 21:04:30 +0000393*Latin-1* encoding. These strings are used for the keys and values in the
394environ dictionary and for response headers and statuses in the
395:func:`start_response` function. They must follow :rfc:`2616` with respect to
Raymond Hettinger3df46212011-01-06 02:01:26 +0000396encoding. That is, they must either be *ISO-8859-1* characters or use
397:rfc:`2047` MIME encoding.
398
Raymond Hettinger32e8fea2011-01-07 21:04:30 +0000399For developers porting WSGI applications from Python 2, here are the salient
400points:
401
402* If the app already used strings for headers in Python 2, no change is needed.
403
404* If instead, the app encoded output headers or decoded input headers, then the
405 headers will need to be re-encoded to Latin-1. For example, an output header
406 encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from
407 bytes to native strings using ``h.encode('utf-8').decode('latin-1')``.
408
409* Values yielded by an application or sent using the :meth:`write` method
Raymond Hettinger51e21072011-01-10 23:38:15 +0000410 must be byte strings. The :func:`start_response` function and environ
411 must use native strings. The two cannot be mixed.
Raymond Hettinger32e8fea2011-01-07 21:04:30 +0000412
413For server implementers writing CGI-to-WSGI pathways or other CGI-style
414protocols, the users must to be able access the environment using native strings
Raymond Hettingerc136b042011-01-18 07:15:39 +0000415even though the underlying platform may have a different convention. To bridge
Raymond Hettinger32e8fea2011-01-07 21:04:30 +0000416this gap, the :mod:`wsgiref` module has a new function,
417:func:`wsgiref.handlers.read_environ` for transcoding CGI variables from
418:attr:`os.environ` into native strings and returning a new dictionary.
Raymond Hettinger3df46212011-01-06 02:01:26 +0000419
420.. seealso::
421
422 :pep:`3333` - Python Web Server Gateway Interface v1.0.1
423 PEP written by Phillip Eby.
Raymond Hettinger6e6565b2009-06-28 20:56:11 +0000424
Raymond Hettingerca904be2011-01-18 00:02:40 +0000425
Raymond Hettinger6e6565b2009-06-28 20:56:11 +0000426Other Language Changes
427======================
428
429Some smaller changes made to the core Python language are:
430
Raymond Hettingere5e1a982010-12-05 08:35:21 +0000431* String formatting for :func:`format` and :meth:`str.format` gained new
432 capabilities for the format character **#**. Previously, for integers in
433 binary, octal, or hexadecimal, it caused the output to be prefixed with '0b',
434 '0o', or '0x' respectively. Now it can also handle floats, complex, and
435 Decimal, causing the output to always have a decimal point even when no digits
436 follow it.
Raymond Hettingere5e728b2010-12-05 06:35:16 +0000437
438 >>> format(20, '#o')
439 '0o24'
440 >>> format(12.34, '#5.0f')
441 ' 12.'
442
443 (Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.)
Raymond Hettinger43b5a852010-12-05 04:04:21 +0000444
Raymond Hettingerc269ae82010-12-05 01:01:52 +0000445* The interpreter can now be started with a quiet option, ``-q``, to suppress
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000446 the copyright and version information from being displayed in the interactive
447 mode. The option can be introspected using the :attr:`sys.flags` attribute::
Raymond Hettinger7d967712011-01-05 20:24:08 +0000448
449 $ python -q
450 >>> sys.flags
451 sys.flags(debug=0, division_warning=0, inspect=0, interactive=0,
452 optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
453 ignore_environment=0, verbose=0, bytes_warning=0, quiet=1)
Raymond Hettingerc269ae82010-12-05 01:01:52 +0000454
Raymond Hettinger51e21072011-01-10 23:38:15 +0000455 (Contributed by Marcin Wojdyr in :issue:`1772833`).
Raymond Hettingerc269ae82010-12-05 01:01:52 +0000456
Raymond Hettinger2169ee22011-01-05 22:27:49 +0000457* The :func:`hasattr` function works by calling :func:`getattr` and detecting
458 whether an exception is raised. This technique allows it to detect methods
459 created dynamically by :meth:`__getattr__` or :meth:`__getattribute__` which
Raymond Hettinger90a4b312011-01-06 02:08:30 +0000460 would otherwise be absent from the class dictionary. Formerly, *hasattr*
461 would catch any exception, possibly masking genuine errors. Now, *hasattr*
462 has been tightened to only catch :exc:`AttributeError` and let other
Raymond Hettinger03ca1a92011-01-20 04:12:37 +0000463 exceptions pass through::
464
465 >>> class A:
Raymond Hettinger2270d582011-01-20 09:04:39 +0000466 @property
467 def f(self):
468 return 1 // 0
Raymond Hettinger03ca1a92011-01-20 04:12:37 +0000469
470 >>> a = A()
471 >>> hasattr(a, 'f')
472 Traceback (most recent call last):
473 ...
474 ZeroDivisionError: integer division or modulo by zero
Raymond Hettinger1784ff02010-09-05 01:00:19 +0000475
Raymond Hettingera55ffbc2010-12-15 18:31:57 +0000476 (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.)
Raymond Hettinger1784ff02010-09-05 01:00:19 +0000477
Éric Araujo4234ad42010-09-05 17:32:25 +0000478* The :func:`str` of a float or complex number is now the same as its
Raymond Hettinger1784ff02010-09-05 01:00:19 +0000479 :func:`repr`. Previously, the :func:`str` form was shorter but that just
Éric Araujo4234ad42010-09-05 17:32:25 +0000480 caused confusion and is no longer needed now that the shortest possible
Georg Brandlda0a2112010-09-05 11:28:33 +0000481 :func:`repr` is displayed by default:
Raymond Hettingerbb734c62010-09-05 05:56:44 +0000482
Raymond Hettinger92ba2862010-09-06 01:16:46 +0000483 >>> repr(math.pi)
484 '3.141592653589793'
485 >>> str(math.pi)
486 '3.141592653589793'
Raymond Hettinger1784ff02010-09-05 01:00:19 +0000487
Georg Brandlda0a2112010-09-05 11:28:33 +0000488 (Proposed and implemented by Mark Dickinson; :issue:`9337`.)
Raymond Hettinger6e6565b2009-06-28 20:56:11 +0000489
Raymond Hettinger21ec4bc2010-12-10 01:09:01 +0000490* :class:`memoryview` objects now have a :meth:`~memoryview.release()` method
491 and they also now support the context manager protocol. This allows timely
492 release of any resources that were acquired when requesting a buffer from the
493 original object.
Antoine Pitroud3052002010-09-15 15:09:40 +0000494
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +0000495 >>> with memoryview(b'abcdefgh') as v:
Raymond Hettingerf1dae312011-01-21 03:00:00 +0000496 print(v.tolist())
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +0000497 [97, 98, 99, 100, 101, 102, 103, 104]
498
Antoine Pitroud3052002010-09-15 15:09:40 +0000499 (Added by Antoine Pitrou; :issue:`9757`.)
500
Amaury Forgeot d'Arcba117ef2010-09-10 21:39:53 +0000501* Previously it was illegal to delete a name from the local namespace if it
502 occurs as a free variable in a nested block::
503
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +0000504 def outer(x):
505 def inner():
506 return x
507 inner()
508 del x
Amaury Forgeot d'Arcba117ef2010-09-10 21:39:53 +0000509
510 This is now allowed. Remember that the target of an :keyword:`except` clause
511 is cleared, so this code which used to work with Python 2.6, raised a
512 :exc:`SyntaxError` with Python 3.1 and now works again::
513
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +0000514 def f():
515 def print_error():
516 print(e)
517 try:
518 something
519 except Exception as e:
520 print_error()
521 # implicit "del e" here
Amaury Forgeot d'Arcba117ef2010-09-10 21:39:53 +0000522
523 (See :issue:`4617`.)
524
Raymond Hettinger480ed782010-12-15 22:07:15 +0000525* The internal :c:type:`structsequence` tool now creates subclasses of tuple.
Raymond Hettinger51e21072011-01-10 23:38:15 +0000526 This means that C structures like those returned by :func:`os.stat`,
Raymond Hettingera275c982011-01-20 04:03:19 +0000527 :func:`time.gmtime`, and :attr:`sys.version_info` now work like a
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000528 :term:`named tuple` and now work with functions and methods that
Raymond Hettinger93c8cad2011-01-18 00:30:24 +0000529 expect a tuple as an argument. This is a big step forward in making the C
Raymond Hettingera275c982011-01-20 04:03:19 +0000530 structures as flexible as their pure Python counterparts:
531
532 >>> isinstance(sys.version_info, tuple)
533 True
534 >>> 'Version %d.%d.%d %s(%d)' % sys.version_info
535 'Version 3.2.0 final(0)'
Raymond Hettinger480ed782010-12-15 22:07:15 +0000536
537 (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented
538 by Benjamin Peterson in :issue:`8413`.)
539
Raymond Hettingerc8a16862011-01-18 09:01:34 +0000540* Warnings are now easier to control using the :envvar:`PYTHONWARNINGS`
Raymond Hettingera275c982011-01-20 04:03:19 +0000541 environment variable as an alternative to using ``-W`` at the command line::
542
543 $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'
Raymond Hettinger480ed782010-12-15 22:07:15 +0000544
545 (Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.)
546
Antoine Pitrou7d15a722010-11-05 22:13:55 +0000547* A new warning category, :exc:`ResourceWarning`, has been added. It is
Raymond Hettingerc269ae82010-12-05 01:01:52 +0000548 emitted when potential issues with resource consumption or cleanup
Raymond Hettinger93c8cad2011-01-18 00:30:24 +0000549 are detected. It is silenced by default in normal release builds but
Raymond Hettingerc269ae82010-12-05 01:01:52 +0000550 can be enabled through the means provided by the :mod:`warnings`
Antoine Pitrou7d15a722010-11-05 22:13:55 +0000551 module, or on the command line.
552
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +0000553 A :exc:`ResourceWarning` is issued at interpreter shutdown if the
Raymond Hettingerd0d59b12011-01-24 05:07:13 +0000554 :data:`gc.garbage` list isn't empty, and if :attr:`gc.DEBUG_UNCOLLECTABLE` is
555 set, all uncollectable objects are printed. This is meant to make the
556 programmer aware that their code contains object finalization issues.
Antoine Pitrou7d15a722010-11-05 22:13:55 +0000557
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +0000558 A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed
Antoine Pitrou7d15a722010-11-05 22:13:55 +0000559 without having been explicitly closed. While the deallocator for such
560 object ensures it closes the underlying operating system resource
561 (usually, a file descriptor), the delay in deallocating the object could
562 produce various issues, especially under Windows. Here is an example
563 of enabling the warning from the command line::
564
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000565 $ python -q -Wdefault
Antoine Pitrou7d15a722010-11-05 22:13:55 +0000566 >>> f = open("foo", "wb")
567 >>> del f
568 __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
Antoine Pitrou7d15a722010-11-05 22:13:55 +0000569
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +0000570 (Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.)
Antoine Pitrou7d15a722010-11-05 22:13:55 +0000571
Raymond Hettingera0266332010-12-07 08:52:41 +0000572* :class:`range` objects now support *index* and *count* methods. This is part
573 of an effort to make more objects fully implement the
574 :class:`collections.Sequence` :term:`abstract base class`. As a result, the
575 language will have a more uniform API. In addition, :class:`range` objects
Raymond Hettingerb9656292011-01-16 18:22:06 +0000576 now support slicing and negative indices, even with values larger than
577 :attr:`sys.maxsize`. This makes *range* more interoperable with lists::
Raymond Hettinger2ffa6712010-12-08 10:18:21 +0000578
579 >>> range(0, 100, 2).count(10)
580 1
581 >>> range(0, 100, 2).index(10)
582 5
583 >>> range(0, 100, 2)[5]
584 10
585 >>> range(0, 100, 2)[0:5]
586 range(0, 10, 2)
Raymond Hettingerdadf93c2010-12-05 02:56:21 +0000587
Raymond Hettingerb9656292011-01-16 18:22:06 +0000588 (Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky
589 in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.)
Nick Coghlan37ee8502010-12-03 14:26:13 +0000590
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +0000591* The :func:`callable` builtin function from Py2.x was resurrected. It provides
Raymond Hettingerb87ba262010-12-06 04:31:40 +0000592 a concise, readable alternative to using an :term:`abstract base class` in an
Raymond Hettinger792c0762010-12-09 16:41:54 +0000593 expression like ``isinstance(x, collections.Callable)``:
594
595 >>> callable(max)
596 True
597 >>> callable(20)
598 False
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +0000599
600 (See :issue:`10518`.)
Amaury Forgeot d'Arcba117ef2010-09-10 21:39:53 +0000601
Raymond Hettinger93c8cad2011-01-18 00:30:24 +0000602* Python's import mechanism can now load modules installed in directories with
Raymond Hettingera275c982011-01-20 04:03:19 +0000603 non-ASCII characters in the path name:
604
605 >>> import møøse.bites
Raymond Hettinger070ec702010-12-10 17:45:13 +0000606
607 (Required extensive work by Victor Stinner in :issue:`9425`.)
608
609
Raymond Hettinger6e6565b2009-06-28 20:56:11 +0000610New, Improved, and Deprecated Modules
611=====================================
612
Raymond Hettinger99db3fd2010-12-15 19:33:49 +0000613Python's standard library has undergone significant maintenance efforts and
614quality improvements.
Raymond Hettingere434b3b2010-12-15 19:20:01 +0000615
616The biggest news for Python 3.2 is that the :mod:`email` package and
Raymond Hettinger99db3fd2010-12-15 19:33:49 +0000617:mod:`nntplib` modules now work correctly with the bytes/text model in Python 3.
Raymond Hettingere434b3b2010-12-15 19:20:01 +0000618For the first time, there is correct handling of inputs with mixed encodings.
619
Raymond Hettinger6046e222010-12-16 00:21:08 +0000620Throughout the standard library, there has been more careful attention to
621encodings and text versus bytes issues. In particular, interactions with the
622operating system are now better able to pass non-ASCII data using the Windows
Raymond Hettingerba5512f2011-01-18 08:28:01 +0000623MBCS encoding, locale-aware encodings, or UTF-8.
Raymond Hettinger6046e222010-12-16 00:21:08 +0000624
Raymond Hettingere434b3b2010-12-15 19:20:01 +0000625Another significant win is the addition of substantially better support for
626*SSL* connections and security certificates.
627
Raymond Hettinger51e21072011-01-10 23:38:15 +0000628In addition, more classes now implement a :term:`context manager` to support
Raymond Hettingerc136b042011-01-18 07:15:39 +0000629convenient and reliable resource clean-up using a :keyword:`with` statement.
Raymond Hettingere434b3b2010-12-15 19:20:01 +0000630
Raymond Hettinger0358a172010-12-15 19:00:38 +0000631email
632-----
633
634The usability of the :mod:`email` package in Python 3 has been mostly fixed by
635the extensive efforts of R. David Murray. The problem was that emails are
636typically read and stored in the form of :class:`bytes` rather than :class:`str`
637text, and they may contain multiple encodings within a single email. So, the
638email package had to be extended to parse and generate email messages in bytes
639format.
640
641* New functions :func:`~email.message_from_bytes` and
642 :func:`~email.message_from_binary_file`, and new classes
643 :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser`
644 allow binary message data to be parsed into model objects.
645
646* Given bytes input to the model, :meth:`~email.message.Message.get_payload`
647 will by default decode a message body that has a
648 :mailheader:`Content-Transfer-Encoding` of *8bit* using the charset
649 specified in the MIME headers and return the resulting string.
650
651* Given bytes input to the model, :class:`~email.generator.Generator` will
652 convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of
653 *8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`.
Raymond Hettingerc08ea612011-01-08 10:32:31 +0000654
Raymond Hettingercf8a3822011-01-11 21:20:20 +0000655 Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded
656 using the *unknown-8bit* character set.
Raymond Hettinger0358a172010-12-15 19:00:38 +0000657
658* A new class :class:`~email.generator.BytesGenerator` produces bytes as output,
659 preserving any unchanged non-ASCII data that was present in the input used to
660 build the model, including message bodies with a
661 :mailheader:`Content-Transfer-Encoding` of *8bit*.
662
663* The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string
664 for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method,
665 and a new method, :meth:`~smtplib.SMTP.send_message` accepts a
666 :class:`~email.message.Message` object and can optionally obtain the
667 *from_addr* and *to_addrs* addresses directly from the object.
668
Raymond Hettinger0358a172010-12-15 19:00:38 +0000669(Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.)
670
Raymond Hettinger6046e222010-12-16 00:21:08 +0000671elementtree
672-----------
673
Georg Brandl5d53fdd2010-12-18 11:58:12 +0000674The :mod:`xml.etree.ElementTree` package and its :mod:`xml.etree.cElementTree`
Raymond Hettinger6046e222010-12-16 00:21:08 +0000675counterpart have been updated to version 1.3.
676
677Several new and useful functions and methods have been added:
678
679* :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document
680 from a sequence of fragments
681* :func:`xml.etree.ElementTree.register_namespace` for registering a global
682 namespace prefix
683* :func:`xml.etree.ElementTree.tostringlist` for string representation
684 including all sublists
685* :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero
686 or more elements
687* :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and
688 subelements
689* :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000690 an element and its subelements
Raymond Hettinger6046e222010-12-16 00:21:08 +0000691* :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element
692* :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype
693 declaration
694
695Two methods have been deprecated:
696
697* :meth:`xml.etree.ElementTree.getchildren` use ``list(elem)`` instead.
698* :meth:`xml.etree.ElementTree.getiterator` use ``Element.iter`` instead.
699
700For details of the update, see `Introducing ElementTree
701<http://effbot.org/zone/elementtree-13-intro.htm>`_ on Fredrik Lundh's website.
702
Antoine Pitrou12de8ac2010-12-16 13:33:56 +0000703(Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.)
Raymond Hettinger0358a172010-12-15 19:00:38 +0000704
Raymond Hettingerbba537b2010-12-15 18:20:19 +0000705functools
706---------
707
Éric Araujo4234ad42010-09-05 17:32:25 +0000708* The :mod:`functools` module includes a new decorator for caching function
Georg Brandlda0a2112010-09-05 11:28:33 +0000709 calls. :func:`functools.lru_cache` can save repeated queries to an external
710 resource whenever the results are expected to be the same.
Raymond Hettingeraed05eb2010-08-02 01:43:41 +0000711
Raymond Hettinger86f96132010-08-06 23:23:49 +0000712 For example, adding a caching decorator to a database query function can save
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +0000713 database accesses for popular searches:
Raymond Hettingeraed05eb2010-08-02 01:43:41 +0000714
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +0000715 >>> @functools.lru_cache(maxsize=300)
716 >>> def get_phone_number(name):
717 c = conn.cursor()
718 c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
719 return c.fetchone()[0]
Raymond Hettingeraed05eb2010-08-02 01:43:41 +0000720
Georg Brandlda0a2112010-09-05 11:28:33 +0000721 >>> for name in user_requests:
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +0000722 get_phone_number(name) # cached lookup
Raymond Hettinger7496b412010-11-30 19:15:45 +0000723
724 To help with choosing an effective cache size, the wrapped function is
725 instrumented for tracking cache statistics:
726
Raymond Hettinger5e20bab2010-11-30 07:13:04 +0000727 >>> get_phone_number.cache_info()
Raymond Hettinger7496b412010-11-30 19:15:45 +0000728 CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)
Raymond Hettingeraed05eb2010-08-02 01:43:41 +0000729
Raymond Hettingerf3098282010-08-15 03:30:45 +0000730 If the phonelist table gets updated, the outdated contents of the cache can be
Georg Brandlda0a2112010-09-05 11:28:33 +0000731 cleared with:
Raymond Hettingerf3098282010-08-15 03:30:45 +0000732
Georg Brandlda0a2112010-09-05 11:28:33 +0000733 >>> get_phone_number.cache_clear()
Raymond Hettingerf3098282010-08-15 03:30:45 +0000734
Raymond Hettinger6e353942010-12-04 23:42:12 +0000735 (Contributed by Raymond Hettinger and incorporating design ideas from
Raymond Hettingerb87ba262010-12-06 04:31:40 +0000736 Jim Baker, Miki Tebeka, and Nick Coghlan.)
Raymond Hettingeraed05eb2010-08-02 01:43:41 +0000737
Antoine Pitrou7d49bc92010-09-15 15:13:17 +0000738* The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute
739 pointing to the original callable function. This allows wrapped functions to
740 be introspected. It also copies :attr:`__annotations__` if defined. And now
741 it also gracefully skips over missing attributes such as :attr:`__doc__` which
Raymond Hettinger5eb63902010-12-09 23:43:34 +0000742 might not be defined for the wrapped callable.
Antoine Pitrou7d49bc92010-09-15 15:13:17 +0000743
Raymond Hettinger7a168d92011-01-21 04:59:00 +0000744 In the above example, the cache can be removed by recovering the original
745 function:
746
747 >>> get_phone_number = get_phone_number.__wrapped__ # uncached function
748
Antoine Pitrou7d49bc92010-09-15 15:13:17 +0000749 (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and
750 :issue:`8814`.)
751
Raymond Hettinger6046e222010-12-16 00:21:08 +0000752* To help write classes with rich comparison methods, a new decorator
753 :func:`functools.total_ordering` will use a existing equality and inequality
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000754 methods to fill in the remaining methods.
Raymond Hettinger6046e222010-12-16 00:21:08 +0000755
756 For example, supplying *__eq__* and *__lt__* will enable
757 :func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*::
758
759 @total_ordering
760 class Student:
761 def __eq__(self, other):
762 return ((self.lastname.lower(), self.firstname.lower()) ==
763 (other.lastname.lower(), other.firstname.lower()))
764 def __lt__(self, other):
765 return ((self.lastname.lower(), self.firstname.lower()) <
766 (other.lastname.lower(), other.firstname.lower()))
767
Raymond Hettinger2169ee22011-01-05 22:27:49 +0000768 With the *total_ordering* decorator, the remaining comparison methods
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000769 are filled in automatically.
Raymond Hettinger6046e222010-12-16 00:21:08 +0000770
Raymond Hettinger2169ee22011-01-05 22:27:49 +0000771 (Contributed by Raymond Hettinger.)
Raymond Hettingerf35a34c2010-12-22 09:11:54 +0000772
Raymond Hettinger4bea9782011-01-19 04:14:34 +0000773* To aid in porting programs from Python 2, the :func:`functools.cmp_to_key`
Raymond Hettingerbb9686f2010-12-16 00:53:05 +0000774 function converts an old-style comparison function to
Raymond Hettinger6046e222010-12-16 00:21:08 +0000775 modern :term:`key function`:
776
777 >>> # locale-aware sort order
778 >>> sorted(iterable, key=cmp_to_key(locale.strcoll))
779
780 For sorting examples and a brief sorting tutorial, see the `Sorting HowTo
781 <http://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial.
782
783 (Contributed by Raymond Hettinger.)
784
Raymond Hettingerbba537b2010-12-15 18:20:19 +0000785itertools
786---------
787
Raymond Hettinger673ccf22010-12-07 09:37:11 +0000788* The :mod:`itertools` module has a new :func:`~itertools.accumulate` function
Raymond Hettinger4bea9782011-01-19 04:14:34 +0000789 modeled on APL's *scan* operator and Numpy's *accumulate* function:
Raymond Hettinger6e353942010-12-04 23:42:12 +0000790
791 >>> list(accumulate(8, 2, 50))
792 [8, 10, 60]
793
794 >>> prob_dist = [0.1, 0.4, 0.2, 0.3]
795 >>> list(accumulate(prob_dist)) # cumulative probability distribution
796 [0.1, 0.5, 0.7, 1.0]
797
798 For an example using :func:`~itertools.accumulate`, see the :ref:`examples for
799 the random module <random-examples>`.
800
801 (Contributed by Raymond Hettinger and incorporating design suggestions
802 from Mark Dickinson.)
803
Raymond Hettingerbba537b2010-12-15 18:20:19 +0000804collections
805-----------
806
Raymond Hettinger792c0762010-12-09 16:41:54 +0000807* The :class:`collections.Counter` class now has two forms of in-place
808 subtraction, the existing *-=* operator for `saturating subtraction
809 <http://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new
810 :meth:`~collections.Counter.subtract` method for regular subtraction. The
811 former is suitable for `multisets <http://en.wikipedia.org/wiki/Multiset>`_
Raymond Hettingerffad35e2010-12-14 21:12:03 +0000812 which only have positive counts, and the latter is more suitable for use cases
Raymond Hettinger792c0762010-12-09 16:41:54 +0000813 that allow negative counts:
Alexander Belopolsky72572312010-12-08 21:21:56 +0000814
Raymond Hettinger792c0762010-12-09 16:41:54 +0000815 >>> tally = Counter(dogs=5, cat=3)
816 >>> tally -= Counter(dogs=2, cats=8) # saturating subtraction
817 >>> tally
818 Counter({'dogs': 3})
Alexander Belopolsky72572312010-12-08 21:21:56 +0000819
Raymond Hettinger792c0762010-12-09 16:41:54 +0000820 >>> tally = Counter(dogs=5, cats=3)
821 >>> tally.subtract(dogs=2, cats=8) # regular subtraction
822 >>> tally
823 Counter({'dogs': 3, 'cats': -5})
Alexander Belopolsky72572312010-12-08 21:21:56 +0000824
Raymond Hettinger792c0762010-12-09 16:41:54 +0000825 (Contributed by Raymond Hettinger.)
Alexander Belopolsky72572312010-12-08 21:21:56 +0000826
Raymond Hettingere0a96002010-12-15 17:54:13 +0000827* The :class:`collections.OrderedDict` class has a new method
828 :meth:`~collections.OrderedDict.move_to_end` which takes an existing key and
Raymond Hettinger23ab1012011-01-18 20:25:04 +0000829 moves it to either the first or last position in the ordered sequence.
830
831 The default is to move an item to the last position. This is equivalent of
832 renewing an entry with ``od[k] = od.pop(k)``.
833
834 A fast move-to-end operation is useful for resequencing entries. For example,
Raymond Hettinger00db6aa2011-01-20 09:47:04 +0000835 an ordered dictionary can be used to track order of access by aging entries
836 from the oldest to the most recently accessed.
Raymond Hettingere0a96002010-12-15 17:54:13 +0000837
838 >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e'])
839 >>> list(d)
840 ['a', 'b', 'X', 'd', 'e']
Raymond Hettinger23ab1012011-01-18 20:25:04 +0000841 >>> d.move_to_end('X')
Raymond Hettingere0a96002010-12-15 17:54:13 +0000842 >>> list(d)
843 ['a', 'b', 'd', 'e', 'X']
Raymond Hettingere0a96002010-12-15 17:54:13 +0000844
Raymond Hettinger6046e222010-12-16 00:21:08 +0000845 (Contributed by Raymond Hettinger.)
846
Raymond Hettinger7ec790d2011-01-18 00:19:30 +0000847* The :class:`collections.deque` class grew two new methods
848 :meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that
849 make them more substitutable for :class:`list` objects:
Raymond Hettinger6046e222010-12-16 00:21:08 +0000850
851 >>> d = deque('simsalabim')
852 >>> d.count('s')
853 2
854 >>> d.reverse()
855 >>> d
856 deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])
857
858 (Contributed by Raymond Hettinger.)
859
Raymond Hettinger5cee47f2011-01-11 19:59:46 +0000860threading
861---------
862
863The :mod:`threading` module has a new :class:`~threading.Barrier`
864synchronization class for making multiple threads wait until all of them have
865reached a common barrier point. Barriers are useful for making sure that a task
866with multiple preconditions does not run until all of the predecessor tasks are
867complete.
868
869Barriers can work with an arbitrary number of threads. This is a generalization
870of a `Rendezvous <http://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which
871is defined for only two threads.
872
Raymond Hettinger15b47c52011-01-17 21:05:07 +0000873Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects
874are suitable for use in loops. The separate *filling* and *draining* phases
Raymond Hettingere0f1f322011-01-18 21:14:27 +0000875assure that all threads get released (drained) before any one of them can loop
876back and re-enter the barrier. The barrier fully resets after each cycle.
Raymond Hettinger5cee47f2011-01-11 19:59:46 +0000877
Raymond Hettinger5cee47f2011-01-11 19:59:46 +0000878Example of using barriers::
879
880 def get_votes(site):
881 ballots = conduct_election(site)
882 all_polls_closed.wait() # do not count until all polls are closed
Raymond Hettinger97673652011-01-11 21:13:26 +0000883 totals = summarize(ballots)
884 publish(site, totals)
Raymond Hettinger5cee47f2011-01-11 19:59:46 +0000885
886 all_polls_closed = Barrier(len(sites))
Raymond Hettinger3a8ae5f2011-01-11 20:51:45 +0000887 for site in sites:
Raymond Hettinger5cee47f2011-01-11 19:59:46 +0000888 Thread(target=get_votes, args=(site,)).start()
889
890In this example, the barrier enforces a rule that votes cannot be counted at any
891polling site until all polls are closed. Notice how a solution with a barrier
892is similar to one with :meth:`threading.Thread.join`, but the threads stay alive
893and continue to do work (summarizing ballots) after the barrier point is
894crossed.
895
Raymond Hettinger2c3865b2011-01-18 22:58:33 +0000896If any of the predecessor tasks can hang or be delayed, a barrier can be created
897with an optional *timeout* parameter. Then if the timeout period elapses before
898all the predecessor tasks reach the barrier point, all waiting threads are
899released and a :exc:`~threading.BrokenBarrierError` exception is raised::
900
901 def get_votes(site):
902 ballots = conduct_election(site)
903 try:
904 all_polls_closed.wait(timeout = midnight - time.now())
David Malcolm49348642011-01-18 23:45:53 +0000905 except BrokenBarrierError:
Raymond Hettinger2c3865b2011-01-18 22:58:33 +0000906 lockbox = seal_ballots(ballots)
907 queue.put(lockbox)
908 else:
909 totals = summarize(ballots)
910 publish(site, totals)
911
912In this example, the barrier enforces a more robust rule. If some election
913sites do not finish before midnight, the barrier times-out and the ballots are
914sealed and deposited in a queue for later handling.
915
Raymond Hettinger5cee47f2011-01-11 19:59:46 +0000916See `Barrier Synchronization Patterns
Raymond Hettinger3a8ae5f2011-01-11 20:51:45 +0000917<http://parlab.eecs.berkeley.edu/wiki/_media/patterns/paraplop_g1_3.pdf>`_ for
918more examples of how barriers can be used in parallel computing. Also, there is
919a simple but thorough explanation of barriers in `The Little Book of Semaphores
920<http://greenteapress.com/semaphores/downey08semaphores.pdf>`_, *section 3.6*.
Raymond Hettinger5cee47f2011-01-11 19:59:46 +0000921
Raymond Hettinger3a8ae5f2011-01-11 20:51:45 +0000922(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in
923:issue:`8777`.)
Raymond Hettinger6655d112011-01-11 08:49:10 +0000924
Raymond Hettinger97673652011-01-11 21:13:26 +0000925datetime and time
926-----------------
Raymond Hettingerbba537b2010-12-15 18:20:19 +0000927
Raymond Hettinger792c0762010-12-09 16:41:54 +0000928* The :mod:`datetime` module has a new type :class:`~datetime.timezone` that
929 implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC
Raymond Hettingerde2e6182011-01-10 05:40:57 +0000930 offset and timezone name. This makes it easier to create timezone-aware
Raymond Hettinger792c0762010-12-09 16:41:54 +0000931 datetime objects:
Alexander Belopolsky72572312010-12-08 21:21:56 +0000932
Raymond Hettinger792c0762010-12-09 16:41:54 +0000933 >>> datetime.now(timezone.utc)
934 datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)
Alexander Belopolsky72572312010-12-08 21:21:56 +0000935
Raymond Hettinger792c0762010-12-09 16:41:54 +0000936 >>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")
937 datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)
Alexander Belopolsky72572312010-12-08 21:21:56 +0000938
Raymond Hettingerbba537b2010-12-15 18:20:19 +0000939* Also, :class:`~datetime.timedelta` objects can now be multiplied by
Raymond Hettinger792c0762010-12-09 16:41:54 +0000940 :class:`float` and divided by :class:`float` and :class:`int` objects.
Raymond Hettinger2169ee22011-01-05 22:27:49 +0000941 And :class:`~datetime.timedelta` objects can now divide one another.
Raymond Hettinger792c0762010-12-09 16:41:54 +0000942
Raymond Hettingerca904be2011-01-18 00:02:40 +0000943* The :meth:`datetime.date.strftime` method is no longer restricted to years
944 after 1900. The new supported year range is from 1000 to 9999 inclusive.
Alexander Belopolsky72572312010-12-08 21:21:56 +0000945
Raymond Hettingerf1dae312011-01-21 03:00:00 +0000946* Whenever a two-digit year is used in a time tuple, the interpretation has been
947 governed by :attr:`time.accept2dyear`. The default is *True* which means that
948 for a two-digit year, the century is guessed according to the POSIX rules
949 governing the ``%y`` strptime format.
Alexander Belopolsky9ee94de2011-01-20 19:51:31 +0000950
Raymond Hettingerf1dae312011-01-21 03:00:00 +0000951 Starting with Py3.2, use of the century guessing heuristic will emit a
952 :exc:`DeprecationWarning`. Instead, it is recommended that
953 :attr:`time.accept2dyear` be set to *False* so that large date ranges
954 can be used without guesswork:
955
Raymond Hettinger7a168d92011-01-21 04:59:00 +0000956 >>> warnings.resetwarnings() # remove the default warning filters
957 >>> time.accept2dyear = True # guess whether 11 means 11 or 2011
Raymond Hettingerf1dae312011-01-21 03:00:00 +0000958 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
Raymond Hettinger7a168d92011-01-21 04:59:00 +0000959 Warning (from warnings module):
960 ...
961 DeprecationWarning: Century info guessed for a 2-digit year.
Raymond Hettingerf1dae312011-01-21 03:00:00 +0000962 'Fri Jan 1 12:34:56 2011'
Raymond Hettinger7a168d92011-01-21 04:59:00 +0000963 >>> time.accept2dyear = False # use the full range of allowable dates
Raymond Hettingerf1dae312011-01-21 03:00:00 +0000964 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
965 'Fri Jan 1 12:34:56 11'
966
967 Several functions now have significantly expanded date ranges. When
968 :attr:`time.accept2dyear` is false, the :func:`time.asctime` function will
969 accept any year that fits in a C int, while the :func:`time.mktime` and
970 :func:`time.strftime` functions will accept the full range supported by the
971 corresponding operating system functions.
Alexander Belopolskybd96b062011-01-10 21:55:34 +0000972
Raymond Hettinger97673652011-01-11 21:13:26 +0000973(Contributed by Alexander Belopolsky and Victor Stinner.)
Alexander Belopolskybd96b062011-01-10 21:55:34 +0000974
Raymond Hettingera4cfb422011-01-25 02:35:58 +0000975math
976----
977
978The :mod:`math` module has been updated with five new functions inspired by the
979C99 standard.
980
981The :func:`~math.isfinite` function provides a reliable and fast way to detect
982special values. It returns *True* for regular numbers and *False* for *Nan* or
983*Infinity*:
984
985>>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))]
986[True, True, False, False]
987
988The :func:`~math.expm1` function computes ``e**x-1`` for small values of *x*
989without incuring the loss of precision that usually accompanies the subtraction
990of nearly equal quantities:
991
992>>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x
9930.013765762467652909
994
Raymond Hettingerf9b8a192011-01-25 05:53:27 +0000995The :func:`~math.erf` function computes a probability integral or `Gaussian
Raymond Hettingera4cfb422011-01-25 02:35:58 +0000996error function <http://en.wikipedia.org/wiki/Error_function>`_:
997
998>>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation
9990.682689492137086
1000
Raymond Hettinger2c639062011-01-25 02:38:59 +00001001The :func:`~math.gamma` function is a continuous extension of the factorial
1002function. See http://en.wikipedia.org/wiki/Gamma_function for details. Because
1003the function is related to factorials, it grows large even for small values of
1004*x*, so there is also a :func:`~math.lgamma` function for computing the natural
1005logarithm of the gamma function:
Raymond Hettingera4cfb422011-01-25 02:35:58 +00001006
1007>>> gamma(7.0) # six factorial
1008720.0
1009>>> lgamma(801.0) # log(800 factorial)
10104551.950730698041
1011
1012(Contributed by Mark Dickinson.)
1013
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001014abc
1015---
Antoine Pitrou7d49bc92010-09-15 15:13:17 +00001016
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001017The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and
1018:func:`~abc.abstractstaticmethod`.
Raymond Hettingera5a35542010-12-05 00:39:18 +00001019
Raymond Hettinger7ec790d2011-01-18 00:19:30 +00001020These tools make it possible to define an :term:`abstract base class` that
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001021requires a particular :func:`classmethod` or :func:`staticmethod` to be
Raymond Hettinger7ec790d2011-01-18 00:19:30 +00001022implemented::
1023
1024 class Temperature(metaclass=ABCMeta):
1025 @abc.abstractclassmethod
Raymond Hettingera80ab102011-01-24 18:19:01 +00001026 def from_fahrenheit(self, t):
Raymond Hettinger7ec790d2011-01-18 00:19:30 +00001027 ...
1028 @abc.abstractclassmethod
Raymond Hettingera80ab102011-01-24 18:19:01 +00001029 def from_celsius(self, t):
Raymond Hettinger7ec790d2011-01-18 00:19:30 +00001030 ...
Antoine Pitrou7d49bc92010-09-15 15:13:17 +00001031
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001032(Patch submitted by Daniel Urban; :issue:`5867`.)
Raymond Hettingerbcbd6962010-09-05 08:46:36 +00001033
Raymond Hettingerf4f0e6c2011-01-24 22:14:42 +00001034io
1035--
1036
1037The :class:`io.BytesIO` has a new method, :meth:`~io.BytesIO.getbuffer`, which
1038provides functionality similar to :func:`memoryview`. It creates an editable
1039view of the data without making a copy. The buffer's random access and support
1040for slice notation are well-suited to in-place editing::
1041
1042 import io
1043
1044 REC_LEN, LOC_START, LOC_LEN = 34, 7, 11
1045
1046 def change_location(buffer, record_number, location):
1047 start = record_number * REC_LEN + LOC_START
1048 buffer[start: start+LOC_LEN] = location
1049
1050 >>> byte_stream = io.BytesIO(
1051 b'G3805 storeroom Main chassis '
1052 b'X7899 shipping Reserve cog '
1053 b'L6988 receiving Primary sprocket'
1054 )
1055 >>> buffer = byte_stream.getbuffer()
1056 >>> change_location(buffer, 1, b'warehouse ')
1057 >>> change_location(buffer, 0, b'showroom ')
1058 >>> print(byte_stream.getvalue())
1059 b'G3805 showroom Main chassis ' ->
1060 b'X7899 warehouse Reserve cog ' ->
1061 b'L6988 receiving Primary sprocket'
1062
Raymond Hettinger98b140c2011-01-23 21:05:46 +00001063reprlib
1064-------
1065
1066When writing a :meth:`__repr__` method for a custom container, it is easy to
1067forget to handle the case where a member refers back to the container itself.
1068Python's builtin objects such as :class:`list` and :class:`set` handle
1069self-reference by displaying "..." in the recursive part of the representation
1070string.
1071
1072To help write such :meth:`__repr__` methods, the :mod:`reprlib` module has a new
Raymond Hettingercbc903b2011-01-23 21:13:27 +00001073decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to
Raymond Hettinger98b140c2011-01-23 21:05:46 +00001074:meth:`__repr__` and substituting a placeholder string instead:
1075
1076 >>> class MyList(list):
1077 @recursive_repr()
1078 def __repr__(self):
1079 return '<' + '|'.join(map(repr, self)) + '>'
1080
1081 >>> m = MyList('abc')
1082 >>> m.append(m)
1083 >>> m.append('x')
1084 >>> print(m)
1085 <'a'|'b'|'c'|...|'x'>
1086
Raymond Hettingercbc903b2011-01-23 21:13:27 +00001087(Contributed by Raymond Hettinger in :issue:`9826` and :issue:`9840`.)
Raymond Hettinger98b140c2011-01-23 21:05:46 +00001088
Raymond Hettinger50307b62011-01-24 01:18:30 +00001089csv
1090---
1091
1092The :mod:`csv` module now supports a new dialect, :class:`~csv.unix_dialect`,
1093which applies quoting for all fields and a traditional Unix style with ``'\n'`` as
1094the line terminator. The registered dialect name is ``unix``.
1095
1096The :class:`csv.DictWriter` has a new method,
1097:meth:`~csv.DictWriter.writeheader` for writing-out an initial row to document
1098the field names::
1099
1100 >>> import csv, sys
1101 >>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix')
1102 >>> w.writeheader()
1103 "name","dept"
1104 >>> w.writerows([
1105 {'name': 'tom', 'dept': 'accounting'},
1106 {'name': 'susan', 'dept': 'Salesl'}])
1107 "tom","accounting"
1108 "susan","sales"
1109
1110(New dialect suggested by Jay Talbot in :issue:`5975`, and the new method
1111suggested by Ed Abraham in :issue:`1537721`.)
1112
Raymond Hettinger480ed782010-12-15 22:07:15 +00001113contextlib
1114----------
1115
1116There is a new and slightly mind-blowing tool
1117:class:`~contextlib.ContextDecorator` that is helpful for creating a
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001118:term:`context manager` that does double duty as a function decorator.
Raymond Hettinger480ed782010-12-15 22:07:15 +00001119
1120As a convenience, this new functionality is used by
1121:func:`~contextlib.contextmanager` so that no extra effort is needed to support
1122both roles.
1123
1124The basic idea is that both context managers and function decorators can be used
1125for pre-action and post-action wrappers. Context managers wrap a group of
Raymond Hettingerc136b042011-01-18 07:15:39 +00001126statements using a :keyword:`with` statement, and function decorators wrap a
Raymond Hettinger480ed782010-12-15 22:07:15 +00001127group of statements enclosed in a function. So, occasionally there is a need to
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001128write a pre-action or post-action wrapper that can be used in either role.
Raymond Hettinger480ed782010-12-15 22:07:15 +00001129
1130For example, it is sometimes useful to wrap functions or groups of statements
1131with a logger that can track the time of entry and time of exit. Rather than
1132writing both a function decorator and a context manager for the task, the
1133:func:`~contextlib.contextmanager` provides both capabilities in a single
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001134definition::
Raymond Hettinger480ed782010-12-15 22:07:15 +00001135
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001136 import logging
1137 logging.basicConfig(level=logging.INFO)
1138 @contextmanager
1139 def track_entry_and_exit(name):
1140 logging.info('Entering: {}'.format(name))
1141 yield
1142 logging.info('Exiting: {}'.format(name))
Raymond Hettinger480ed782010-12-15 22:07:15 +00001143
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001144Formerly, this would have only been usable as a context manager::
Raymond Hettinger480ed782010-12-15 22:07:15 +00001145
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001146 with track_entry_and_exit('widget loader'):
1147 print('Some time consuming activity goes here')
1148 load_widget()
Raymond Hettinger480ed782010-12-15 22:07:15 +00001149
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001150Now, it can be used as a decorator as well::
Raymond Hettinger480ed782010-12-15 22:07:15 +00001151
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001152 @track_entry_and_exit('widget loader')
1153 def activity():
1154 print('Some time consuming activity goes here')
1155 load_widget()
Raymond Hettinger480ed782010-12-15 22:07:15 +00001156
1157Trying to fulfill two roles at once places some limitations on the technique.
1158Context managers normally have the flexibility to return an argument usable by
Raymond Hettingerc136b042011-01-18 07:15:39 +00001159a :keyword:`with` statement, but there is no parallel for function decorators.
Raymond Hettinger480ed782010-12-15 22:07:15 +00001160
Raymond Hettinger9743e4f2010-12-16 02:24:12 +00001161In the above example, there is not a clean way for the *track_entry_and_exit*
Raymond Hettinger388af4b2011-01-06 20:55:29 +00001162context manager to return a logging instance for use in the body of enclosed
1163statements.
Raymond Hettinger480ed782010-12-15 22:07:15 +00001164
1165(Contributed by Michael Foord in :issue:`9110`.)
1166
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001167decimal and fractions
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001168---------------------
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001169
1170Mark Dickinson crafted an elegant and efficient scheme for assuring that
1171different numeric datatypes will have the same hash value whenever their actual
1172values are equal (:issue:`8188`)::
1173
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001174 assert hash(Fraction(3, 2)) == hash(1.5) == \
1175 hash(Decimal("1.5")) == hash(complex(1.5, 0))
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001176
Raymond Hettingere7dfe742011-01-24 09:17:24 +00001177Some of the hashing details are exposed through a new attribute,
1178:attr:`sys.hash_info`, which describes the bit width of the hash value, the
1179prime modulus, the hash values for *infinity* and *nan*, and the multiplier
1180used for the imaginary part of a number:
1181
1182>>> sys.hash_info
1183sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003)
1184
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001185An early decision to limit the inter-operability of various numeric types has
Raymond Hettingerc8a16862011-01-18 09:01:34 +00001186been relaxed. It is still unsupported (and ill-advised) to have implicit
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001187mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')``
1188because the latter loses information in the process of constructing the binary
1189float. However, since existing floating point value can be converted losslessly
1190to either a decimal or rational representation, it makes sense to add them to
1191the constructor and to support mixed-type comparisons.
1192
Raymond Hettingerbb9686f2010-12-16 00:53:05 +00001193* The :class:`decimal.Decimal` constructor now accepts :class:`float` objects
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001194 directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float`
Raymond Hettinger6046e222010-12-16 00:21:08 +00001195 method (:issue:`8257`).
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001196
1197* Mixed type comparisons are now fully supported so that
1198 :class:`~decimal.Decimal` objects can be directly compared with :class:`float`
Raymond Hettinger6046e222010-12-16 00:21:08 +00001199 and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`).
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001200
1201Similar changes were made to :class:`fractions.Fraction` so that the
1202:meth:`~fractions.Fraction.from_float()` and :meth:`~fractions.Fraction.from_decimal`
Raymond Hettinger6046e222010-12-16 00:21:08 +00001203methods are no longer needed (:issue:`8294`):
1204
1205>>> Decimal(1.1)
1206Decimal('1.100000000000000088817841970012523233890533447265625')
1207>>> Fraction(1.1)
1208Fraction(2476979795053773, 2251799813685248)
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001209
1210Another useful change for the :mod:`decimal` module is that the
1211:attr:`Context.clamp` attribute is now public. This is useful in creating
1212contexts that correspond to the decimal interchange formats specified in IEEE
1213754 (see :issue:`8540`).
1214
Raymond Hettinger6046e222010-12-16 00:21:08 +00001215(Contributed by Mark Dickinson and Raymond Hettinger.)
Raymond Hettinger07a605b2010-12-15 22:35:03 +00001216
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001217ftp
1218---
Raymond Hettingerbcbd6962010-09-05 08:46:36 +00001219
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001220The :class:`ftplib.FTP` class now supports the context manager protocol to
1221unconditionally consume :exc:`socket.error` exceptions and to close the FTP
1222connection when done::
Giampaolo Rodolàbd576b72010-05-10 14:53:29 +00001223
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001224 >>> from ftplib import FTP
1225 >>> with FTP("ftp1.at.proftpd.org") as ftp:
1226 ... ftp.login()
1227 ... ftp.dir()
1228 ...
1229 '230 Anonymous login ok, restrictions apply.'
1230 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .
1231 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 ..
1232 dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS
1233 dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora
Georg Brandlda0a2112010-09-05 11:28:33 +00001234
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001235Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input`
1236also grew auto-closing context managers::
1237
1238 with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
1239 for line in f:
1240 process(line)
1241
1242(Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and
1243by Georg Brandl in :issue:`8046` and :issue:`1286`.)
Antoine Pitrou696e0352010-08-08 22:18:46 +00001244
Antoine Pitroubcba4342011-01-16 18:29:34 +00001245The :class:`~ftplib.FTP_TLS` class now accepts a *context* parameter, which is a
1246:class:`ssl.SSLContext` object allowing bundling SSL configuration options,
Raymond Hettinger4854d142011-01-17 21:29:58 +00001247certificates and private keys into a single (potentially long-lived) structure.
Antoine Pitroubcba4342011-01-16 18:29:34 +00001248
1249(Contributed by Giampaolo Rodolà; :issue:`8806`.)
1250
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001251popen
1252-----
1253
1254The :func:`os.popen` and :func:`subprocess.Popen` functions now support
Raymond Hettingerc136b042011-01-18 07:15:39 +00001255:keyword:`with` statements for auto-closing of the file descriptors.
Georg Brandl3ad46752010-12-05 07:59:29 +00001256
Raymond Hettinger6046e222010-12-16 00:21:08 +00001257gzip and zipfile
1258----------------
Antoine Pitroucd889af2010-10-06 21:13:56 +00001259
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001260:class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase`
1261:term:`abstract base class` (except for ``truncate()``). It also has a
1262:meth:`~gzip.GzipFile.peek` method and supports unseekable as well as
1263zero-padded file objects.
Raymond Hettingera0266332010-12-07 08:52:41 +00001264
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001265The :mod:`gzip` module also gains the :func:`~gzip.compress` and
1266:func:`~gzip.decompress` functions for easier in-memory compression and
Raymond Hettinger51e21072011-01-10 23:38:15 +00001267decompression. Keep in mind that text needs to be encoded as :class:`bytes`
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001268before compressing and decompressing:
Raymond Hettingera0266332010-12-07 08:52:41 +00001269
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001270>>> s = 'Three shall be the number thou shalt count, '
1271>>> s += 'and the number of the counting shall be three'
1272>>> b = s.encode() # convert to utf-8
1273>>> len(b)
127489
1275>>> c = gzip.compress(b)
1276>>> len(c)
127777
1278>>> gzip.decompress(c).decode()[:42] # decompress and convert to text
1279'Three shall be the number thou shalt count,'
Antoine Pitroucd889af2010-10-06 21:13:56 +00001280
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001281(Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir
1282Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and
1283:issue:`2846`.)
1284
Raymond Hettinger6046e222010-12-16 00:21:08 +00001285Also, the :class:`zipfile.ZipExtFile` class was reworked internally to represent
1286files stored inside an archive. The new implementation is significantly faster
1287and can be wrapped in a :class:`io.BufferedReader` object for more speedups. It
1288also solves an issue where interleaved calls to *read* and *readline* gave the
1289wrong results.
1290
Raymond Hettingerc8a16862011-01-18 09:01:34 +00001291(Patch submitted by Nir Aides in :issue:`7610`.)
Raymond Hettinger6046e222010-12-16 00:21:08 +00001292
Raymond Hettingerd0d59b12011-01-24 05:07:13 +00001293hashlib
1294-------
1295
1296The :mod:`hashlib` module has two new constant attributes listing the hashing
1297algorithms guaranteed to be present in all implementations and those available
1298on the current implementation:
1299
1300 >>> import hashlib
1301 >>> hashlib.algorithms_guaranteed
1302 {'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'}
1303 >>> hashlib.algorithms_available
1304 {'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256',
1305 'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2',
1306 'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224',
1307 'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'}
1308
1309(Suggested by Carl Chenet in :issue:`7418`.)
1310
Raymond Hettinger50307b62011-01-24 01:18:30 +00001311ast
1312---
1313
1314The :mod:`ast` module has a wonderful a general-purpose tool for safely
1315evaluating strings containing Python expressions using the Python literal
1316syntax. The :func:`ast.literal_eval` function serves as a secure alternative to
1317the builtin :func:`eval` function which is easily abused. Python 3.2 adds
1318:class:`bytes` and :class:`set` literals to the list of supported types:
1319strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.
1320
1321::
1322
1323 >>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}"
1324 >>> literal_eval(request)
1325 {'args': (2, 0.5), 'req': 3, 'func': 'pow'}
1326
1327 >>> request = "os.system('do something harmful')"
1328 >>> literal_eval(request)
1329 Traceback (most recent call last):
1330 ...
1331 ValueError: malformed node or string: <_ast.Call object at 0x101739a10>
1332
Raymond Hettinger2270d582011-01-20 09:04:39 +00001333os
1334--
1335
1336Different operating systems use various encodings for filenames and environment
1337variables. The :mod:`os` module provides two new functions,
1338:func:`~os.fsencode` and :func:`~os.fsdecode`, for encoding and decoding
1339filenames:
1340
Raymond Hettinger2e042d32011-01-21 09:18:19 +00001341>>> filename = 'Sehenswürdigkeiten'
Raymond Hettinger2270d582011-01-20 09:04:39 +00001342>>> os.fsencode(filename)
Raymond Hettinger2e042d32011-01-21 09:18:19 +00001343b'Sehensw\xc3\xbcrdigkeiten'
Raymond Hettinger2270d582011-01-20 09:04:39 +00001344
1345Some operating systems allow direct access to the unencoded bytes in the
1346environment. If so, the :attr:`os.supports_bytes_environ` constant will be
1347true.
1348
1349For direct access to unencoded environment variables (if available),
1350use the new :func:`os.getenvb` function or use :data:`os.environb`
1351which is a bytes version of :data:`os.environ`.
1352
1353
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001354shutil
1355------
1356
1357The :func:`shutil.copytree` function has two new options:
Antoine Pitroud67075e2010-07-31 22:48:02 +00001358
Antoine Pitrou121a0552011-01-16 18:16:52 +00001359* *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function
Raymond Hettingerc136b042011-01-18 07:15:39 +00001360 copies a file pointed to by a symlink, not the symlink itself. This option
1361 will silence the error raised if the file doesn't exist.
Antoine Pitroud67075e2010-07-31 22:48:02 +00001362
Antoine Pitrou121a0552011-01-16 18:16:52 +00001363* *copy_function*: is a callable that will be used to copy files.
1364 :func:`shutil.copy2` is used by default.
Antoine Pitroud67075e2010-07-31 22:48:02 +00001365
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001366(Contributed by Tarek Ziadé.)
Antoine Pitroud67075e2010-07-31 22:48:02 +00001367
Raymond Hettinger0929b1f2011-01-23 11:29:08 +00001368In addition, the :mod:`shutil` module now supports :ref:`archiving operations
1369<archiving-operations>` for zipfiles, uncompressed tarfiles, gzipped tarfiles,
1370and bzipped tarfiles. And there are functions for registering additional
1371archiving file formats (such as xz compressed tarfiles or custom formats).
1372
1373The principal functions are :func:`~shutil.make_archive` and
1374:func:`~shutil.unpack_archive`. By default, both operate on the current
1375directory (which can be set by :func:`os.chdir`) and on any sub-directories.
1376The archive filename needs to specified with a full pathname. The archiving
1377step is non-destructive (the original files are left unchanged).
1378
1379::
1380
1381 >>> import shutil, pprint
1382 >>> os.chdir('mydata') # change to the source directory
1383 >>> f = make_archive('/var/backup/mydata', 'zip') # archive the current directory
1384 >>> f # show the name of archive
1385 '/var/backup/mydata.zip'
1386 >>> os.chdir('tmp') # change to an unpacking
1387 >>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data
1388 >>> pprint.pprint(shutil.get_archive_formats()) # display known formats
1389 [('bztar', "bzip2'ed tar-file"),
1390 ('gztar', "gzip'ed tar-file"),
1391 ('tar', 'uncompressed tar file'),
1392 ('zip', 'ZIP file')]
1393 >>> shutil.register_archive_format( # register a new archive format
1394 name = 'xz',
1395 function = 'xz.compress',
1396 extra_args = [('level', 8)],
1397 description = 'xz compression'
1398 )
1399
1400(Contributed by Tarek Ziadé.)
1401
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001402sqlite3
1403-------
Antoine Pitroue43f9d02010-08-08 23:24:50 +00001404
Raymond Hettinger6046e222010-12-16 00:21:08 +00001405The :mod:`sqlite3` module was updated to version 2.6.0. It has two new capabilities.
Antoine Pitroue43f9d02010-08-08 23:24:50 +00001406
Raymond Hettinger0358a172010-12-15 19:00:38 +00001407* The :attr:`sqlite3.Connection.in_transit` attribute is true if there is an
1408 active transaction for uncommitted changes.
Antoine Pitroud67075e2010-07-31 22:48:02 +00001409
Raymond Hettinger0358a172010-12-15 19:00:38 +00001410* The :meth:`sqlite3.Connection.enable_load_extension` and
1411 :meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite
1412 extensions from ".so" files. One well-known extension is the fulltext-search
1413 extension distributed with SQLite.
Antoine Pitroud67075e2010-07-31 22:48:02 +00001414
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001415(Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.)
1416
Raymond Hettingera3b7a142011-01-24 05:26:00 +00001417html
1418----
1419
1420A new :mod:`html` module was introduced with only a single function,
1421:func:`~html.escape`, which is used for escaping reserved characters from HTML
1422markup:
1423
1424>>> import html
1425>>> html.escape('x > 2 && x < 7')
1426'x &gt; 2 &amp;&amp; x &lt; 7'
1427
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001428socket
1429------
1430
1431The :mod:`socket` module has two new improvements.
1432
1433* Socket objects now have a :meth:`~socket.socket.detach()` method which puts
1434 the socket into closed state without actually closing the underlying file
1435 descriptor. The latter can then be reused for other purposes.
1436 (Added by Antoine Pitrou; :issue:`8524`.)
1437
1438* :func:`socket.create_connection` now supports the context manager protocol
1439 to unconditionally consume :exc:`socket.error` exceptions and to close the
1440 socket when done.
1441 (Contributed by Giampaolo Rodolà; :issue:`9794`.)
1442
1443ssl
1444---
Antoine Pitroud67075e2010-07-31 22:48:02 +00001445
Raymond Hettinger4854d142011-01-17 21:29:58 +00001446The :mod:`ssl` module added a number of features to satisfy common requirements
1447for secure (encrypted, authenticated) internet connections:
Antoine Pitrou33da1d62011-01-16 18:16:09 +00001448
Raymond Hettinger4854d142011-01-17 21:29:58 +00001449* A new class, :class:`~ssl.SSLContext`, serves as a container for persistent
1450 SSL data, such as protocol settings, certificates, private keys, and various
1451 other options. It includes a :meth:`~ssl.SSLContext.wrap_socket` for creating
1452 an SSL socket from an SSL context.
Antoine Pitrou4f2a0a82010-07-31 18:08:33 +00001453
Raymond Hettinger4854d142011-01-17 21:29:58 +00001454* A new function, :func:`ssl.match_hostname`, supports server identity
1455 verification for higher-level protocols by implementing the rules of HTTPS
1456 (from :rfc:`2818`) which are also suitable for other protocols.
Antoine Pitrou0ee4c9f2010-10-08 16:46:17 +00001457
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001458* The :func:`ssl.wrap_socket` constructor function now takes a *ciphers*
Raymond Hettinger4854d142011-01-17 21:29:58 +00001459 argument. The *ciphers* string lists the allowed encryption algorithms using
1460 the format described in the `OpenSSL documentation
1461 <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__.
Antoine Pitrou4f2a0a82010-07-31 18:08:33 +00001462
Raymond Hettinger4854d142011-01-17 21:29:58 +00001463* When linked against recent versions of OpenSSL, the :mod:`ssl` module now
1464 supports the Server Name Indication extension to the TLS protocol, allowing
1465 multiple "virtual hosts" using different certificates on a single IP port.
1466 This extension is only supported in client mode, and is activated by passing
1467 the *server_hostname* argument to :meth:`ssl.SSLContext.wrap_socket`.
Antoine Pitrou7d15a722010-11-05 22:13:55 +00001468
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001469* Various options have been added to the :mod:`ssl` module, such as
Raymond Hettinger4854d142011-01-17 21:29:58 +00001470 :data:`~ssl.OP_NO_SSLv2` which disables the insecure and obsolete SSLv2
1471 protocol.
Antoine Pitrou4f2a0a82010-07-31 18:08:33 +00001472
Raymond Hettinger4854d142011-01-17 21:29:58 +00001473* The extension now loads all the OpenSSL ciphers and digest algorithms. If
1474 some SSL certificates cannot be verified, they are reported as an "unknown
1475 algorithm" error.
Antoine Pitrou4f2a0a82010-07-31 18:08:33 +00001476
Raymond Hettinger4854d142011-01-17 21:29:58 +00001477* The version of OpenSSL being used is now accessible using the module
1478 attributes :data:`ssl.OPENSSL_VERSION` (a string),
1479 :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and
1480 :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer).
1481
1482(Contributed by Antoine Pitrou in :issue:`8850`, :issue:`1589`, :issue:`8322`,
1483:issue:`5639`, :issue:`4870`, :issue:`8484`, and :issue:`8321`.)
Antoine Pitrou4f2a0a82010-07-31 18:08:33 +00001484
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001485nntp
1486----
Raymond Hettinger070ec702010-12-10 17:45:13 +00001487
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001488The :mod:`nntplib` module has a revamped implementation with better bytes and
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001489text semantics as well as more practical APIs. These improvements break
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001490compatibility with the nntplib version in Python 3.1, which was partly
1491dysfunctional in itself.
Raymond Hettinger070ec702010-12-10 17:45:13 +00001492
Antoine Pitrou33da1d62011-01-16 18:16:09 +00001493Support for secure connections through both implicit (using
1494:class:`nntplib.NNTP_SSL`) and explicit (using :meth:`nntplib.NNTP.starttls`)
1495TLS has also been added.
1496
1497(Contributed by Antoine Pitrou in :issue:`9360` and Andrew Vant in :issue:`1926`.)
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001498
1499certificates
1500------------
1501
1502:class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler`
1503and :func:`urllib.request.urlopen` now take optional arguments to allow for
1504server certificate checking against a set of Certificate Authorities,
1505as recommended in public uses of HTTPS.
1506
1507(Added by Antoine Pitrou, :issue:`9003`.)
1508
Antoine Pitrou2e8ec222011-01-16 18:41:36 +00001509imaplib
1510-------
1511
1512Support for explicit TLS on standard IMAP4 connections has been added through
1513the new :mod:`imaplib.IMAP4.starttls` method.
1514
1515(Contributed by Lorenzo M. Catucci and Antoine Pitrou, :issue:`4471`.)
1516
Raymond Hettinger399bf7b2011-01-24 10:11:12 +00001517.. XXX sys._xoptions http://bugs.python.org/issue10089
1518
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001519unittest
1520--------
Antoine Pitrouafb078d2010-11-05 22:18:28 +00001521
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001522The unittest module has a number of improvements supporting test discovery for
1523packages, easier experimentation at the interactive prompt, new testcase
1524methods, improved diagnostic messages for test failures, and better method
1525names.
1526
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001527* The command-line call ``python -m unittest`` can now accept file paths
Raymond Hettingera0266332010-12-07 08:52:41 +00001528 instead of module names for running specific tests (:issue:`10620`). The new
1529 test discovery can find tests within packages, locating any test importable
Raymond Hettingerc136b042011-01-18 07:15:39 +00001530 from the top-level directory. The top-level directory can be specified with
Raymond Hettingera0266332010-12-07 08:52:41 +00001531 the `-t` option, a pattern for matching files with ``-p``, and a directory to
1532 start discovery with ``-s``::
1533
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001534 $ python -m unittest discover -s my_proj_dir -p _test.py
Raymond Hettingera0266332010-12-07 08:52:41 +00001535
1536 (Contributed by Michael Foord.)
Antoine Pitroud3052002010-09-15 15:09:40 +00001537
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001538* Experimentation at the interactive prompt is now easier because the
1539 :class:`unittest.case.TestCase` class can now be instantiated without
1540 arguments:
1541
1542 >>> TestCase().assertEqual(pow(2, 3), 8)
1543
1544 (Contributed by Michael Foord.)
1545
Raymond Hettingerdc2f9b52010-12-05 07:02:45 +00001546* The :mod:`unittest` module has two new methods,
1547 :meth:`~unittest.TestCase.assertWarns` and
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001548 :meth:`~unittest.TestCase.assertWarnsRegex` to verify that a given warning type
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001549 is triggered by the code under test::
Antoine Pitroud3052002010-09-15 15:09:40 +00001550
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001551 with self.assertWarns(DeprecationWarning):
1552 legacy_function('XYZ')
Ezio Melotti2baf1a62010-11-22 12:56:58 +00001553
Antoine Pitroueec6dbf2011-01-16 18:21:12 +00001554 (Contributed by Antoine Pitrou, :issue:`9754`.)
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001555
Raymond Hettinger21ec4bc2010-12-10 01:09:01 +00001556 Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to
Raymond Hettingerffad35e2010-12-14 21:12:03 +00001557 compare two iterables to determine if their element counts are equal (whether
1558 the same elements are present with the same number of occurrences regardless
1559 of order)::
Raymond Hettingera0266332010-12-07 08:52:41 +00001560
1561 def test_anagram(self):
1562 self.assertCountEqual('algorithm', 'logarithm')
1563
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001564 (Contributed by Raymond Hettinger.)
1565
1566* A principal feature of the unittest module is an effort to produce meaningful
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001567 diagnostics when a test fails. When possible, the failure is recorded along
Raymond Hettingera0266332010-12-07 08:52:41 +00001568 with a diff of the output. This is especially helpful for analyzing log files
1569 of failed test runs. However, since diffs can sometime be voluminous, there is
1570 a new :attr:`~unittest.TestCase.maxDiff` attribute which sets maximum length of
1571 diffs.
1572
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001573* In addition, the method names in the module have undergone a number of clean-ups.
1574
1575 For example, :meth:`~unittest.TestCase.assertRegex` is the new name for
Raymond Hettingera0266332010-12-07 08:52:41 +00001576 :meth:`~unittest.TestCase.assertRegexpMatches` which was misnamed because the
Raymond Hettinger21ec4bc2010-12-10 01:09:01 +00001577 test uses :func:`re.search`, not :func:`re.match`. Other methods using
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001578 regular expressions are now named using short form "Regex" in preference to
1579 "Regexp" -- this matches the names used in other unittest implementations,
Raymond Hettinger21ec4bc2010-12-10 01:09:01 +00001580 matches Python's old name for the :mod:`re` module, and it has unambiguous
1581 camel-casing.
Raymond Hettingerdc2f9b52010-12-05 07:02:45 +00001582
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001583 (Contributed by Raymond Hettinger and implemented by Ezio Melotti.)
1584
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001585* To improve consistency, some long-standing method aliases are being
Raymond Hettingerdc2f9b52010-12-05 07:02:45 +00001586 deprecated in favor of the preferred names:
1587
Raymond Hettingerc1dfa2e2011-01-19 04:24:57 +00001588 =============================== ==============================
1589 Old Name Preferred Name
1590 =============================== ==============================
1591 :meth:`assert_` :meth:`.assertTrue`
1592 :meth:`assertEquals` :meth:`.assertEqual`
1593 :meth:`assertNotEquals` :meth:`.assertNotEqual`
1594 :meth:`assertAlmostEquals` :meth:`.assertAlmostEqual`
1595 :meth:`assertNotAlmostEquals` :meth:`.assertNotAlmostEqual`
1596 =============================== ==============================
Raymond Hettingerdc2f9b52010-12-05 07:02:45 +00001597
1598 Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected
Raymond Hettingerc1dfa2e2011-01-19 04:24:57 +00001599 to be removed in Python 3.3. Also see the :ref:`deprecated-aliases` section in
Raymond Hettingerdc2f9b52010-12-05 07:02:45 +00001600 the :mod:`unittest` documentation.
Ezio Melotti2baf1a62010-11-22 12:56:58 +00001601
1602 (Contributed by Ezio Melotti; :issue:`9424`.)
Antoine Pitroud3052002010-09-15 15:09:40 +00001603
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001604* The :meth:`~unittest.TestCase.assertDictContainsSubset` method was deprecated
Raymond Hettingerc8a16862011-01-18 09:01:34 +00001605 because it was misimplemented with the arguments in the wrong order. This
Raymond Hettinger2169ee22011-01-05 22:27:49 +00001606 created hard-to-debug optical illusions where tests like
1607 ``TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})`` would fail.
1608
1609 (Contributed by Raymond Hettinger.)
1610
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001611random
1612------
Raymond Hettingere0a96002010-12-15 17:54:13 +00001613
Raymond Hettinger0358a172010-12-15 19:00:38 +00001614The integer methods in the :mod:`random` module now do a better job of producing
Raymond Hettinger99db3fd2010-12-15 19:33:49 +00001615uniform distributions. Previously, they computed selections with
1616``int(n*random())`` which had a slight bias whenever *n* was not a power of two.
Raymond Hettingerc136b042011-01-18 07:15:39 +00001617Now, multiple selections are made from a range up to the next power of two and a
Raymond Hettinger99db3fd2010-12-15 19:33:49 +00001618selection is kept only when it falls within the range ``0 <= x < n``. The
1619functions and methods affected are :func:`~random.randrange`,
1620:func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and
1621:func:`~random.sample`.
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001622
1623(Contributed by Raymond Hettinger; :issue:`9025`.)
1624
1625poplib
1626------
Raymond Hettingere0a96002010-12-15 17:54:13 +00001627
Giampaolo Rodolà42382fe2010-08-17 16:09:53 +00001628* :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a
1629 :class:`ssl.SSLContext` object allowing bundling SSL configuration options,
1630 certificates and private keys into a single (potentially long-lived)
1631 structure.
1632
1633 (Contributed by Giampaolo Rodolà; :issue:`8807`.)
1634
Giampaolo Rodolà977c7072010-10-04 21:08:36 +00001635* :class:`asyncore.dispatcher` now provides a
1636 :meth:`~asyncore.dispatcher.handle_accepted()` method
1637 returning a `(sock, addr)` pair which is called when a connection has actually
1638 been established with a new remote endpoint. This is supposed to be used as a
1639 replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids
1640 the user to call :meth:`~asyncore.dispatcher.accept()` directly.
1641
1642 (Contributed by Giampaolo Rodolà; :issue:`6706`.)
Georg Brandlda0a2112010-09-05 11:28:33 +00001643
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001644tempfile
1645--------
Raymond Hettingera0266332010-12-07 08:52:41 +00001646
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001647The :mod:`tempfile` module has a new context manager,
1648:class:`~tempfile.TemporaryDirectory` which provides easy deterministic
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001649cleanup of temporary directories::
Nick Coghlan543af752010-10-24 11:23:25 +00001650
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001651 with tempfile.TemporaryDirectory() as tmpdirname:
1652 print('created temporary dir:', tmpdirname)
Nick Coghlan543af752010-10-24 11:23:25 +00001653
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001654(Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.)
Nick Coghlane0f04652010-11-21 03:44:04 +00001655
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001656inspect
1657-------
1658
Raymond Hettinger0358a172010-12-15 19:00:38 +00001659* The :mod:`inspect` module has a new function
1660 :func:`~inspect.getgeneratorstate` to easily identify the current state of a
Raymond Hettingera275c982011-01-20 04:03:19 +00001661 generator-iterator::
Raymond Hettinger4bea9782011-01-19 04:14:34 +00001662
Raymond Hettingera275c982011-01-20 04:03:19 +00001663 >>> from inspect import getgeneratorstate
Raymond Hettinger4bea9782011-01-19 04:14:34 +00001664 >>> def gen():
Raymond Hettingera275c982011-01-20 04:03:19 +00001665 yield 'demo'
Raymond Hettinger4bea9782011-01-19 04:14:34 +00001666 >>> g = gen()
Raymond Hettingera275c982011-01-20 04:03:19 +00001667 >>> getgeneratorstate(g)
Raymond Hettinger4bea9782011-01-19 04:14:34 +00001668 'GEN_CREATED'
1669 >>> next(g)
Raymond Hettingera275c982011-01-20 04:03:19 +00001670 'demo'
1671 >>> getgeneratorstate(g)
Raymond Hettinger4bea9782011-01-19 04:14:34 +00001672 'GEN_SUSPENDED'
Raymond Hettingera275c982011-01-20 04:03:19 +00001673 >>> next(g, None)
1674 >>> getgeneratorstate(g)
1675 'GEN_CLOSED'
Raymond Hettinger4bea9782011-01-19 04:14:34 +00001676
1677 (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.)
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001678
Raymond Hettingera55ffbc2010-12-15 18:31:57 +00001679* To support lookups without the possibility of activating a dynamic attribute,
1680 the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`.
Raymond Hettingerc8a16862011-01-18 09:01:34 +00001681 Unlike :func:`hasattr`, this is a true read-only search, guaranteed not to
Raymond Hettinger4bea9782011-01-19 04:14:34 +00001682 change state while it is searching::
1683
1684 >>> class A:
1685 @property
1686 def f(self):
1687 print('Running')
1688 return 10
1689
1690 >>> a = A()
1691 >>> getattr(a, 'f')
1692 Running
1693 10
1694 >>> inspect.getattr_static(a, 'f')
1695 <property object at 0x1022bd788>
1696
1697 (Contributed by Michael Foord.)
Nick Coghlane0f04652010-11-21 03:44:04 +00001698
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001699pydoc
1700-----
Nick Coghlan7bb30b72010-12-03 09:29:11 +00001701
Raymond Hettinger89c1cd12011-01-19 04:43:45 +00001702The :mod:`pydoc` module now provides a much-improved Web server interface, as
1703well as a new command-line option ``-b`` to automatically open a browser window
1704to display that server::
1705
1706 $ pydoc3.2 -b
Nick Coghlan7bb30b72010-12-03 09:29:11 +00001707
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001708(Contributed by Ron Adam; :issue:`2001`.)
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001709
Raymond Hettingeracff5952011-01-24 01:51:49 +00001710dis
1711---
1712
1713The :mod:`dis` module gained two new functions for inspecting code,
1714:func:`~dis.code_info` and :func:`~dis.show_code`. Both provide detailed code
1715object information for the supplied function, method, source code string or code
1716object. The former returns a string and the latter prints it::
1717
1718 >>> import dis, random
1719 >>> show_code(random.choice)
1720 Name: choice
1721 Filename: /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py
1722 Argument count: 2
1723 Kw-only arguments: 0
1724 Number of locals: 3
1725 Stack size: 11
1726 Flags: OPTIMIZED, NEWLOCALS, NOFREE
1727 Constants:
1728 0: 'Choose a random element from a non-empty sequence.'
1729 1: 'Cannot choose from an empty sequence'
1730 Names:
1731 0: _randbelow
1732 1: len
1733 2: ValueError
1734 3: IndexError
1735 Variable names:
1736 0: self
1737 1: seq
1738 2: i
1739
1740(Contributed by Nick Coghlan in :issue:`9147`.)
1741
1742dbm
1743---
1744
1745All database modules now support :meth:`get` and :meth:`setdefault` are now
1746available in all database modules
1747
1748(Suggested by Ray Allen in :issue:`9523`.)
1749
1750ctypes
1751------
1752
1753A new type, :class:`ctypes.c_ssize_t` represents the C :c:type:`ssize_t` datatype.
1754
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001755sysconfig
1756---------
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001757
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001758The new :mod:`sysconfig` module makes it straightforward to discover
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001759installation paths and configuration variables which vary across platforms and
1760installations.
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001761
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001762The module offers access simple access functions for platform and version
1763information:
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001764
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001765* :func:`~sysconfig.get_platform` returning values like *linux-i586* or
1766 *macosx-10.6-ppc*.
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001767* :func:`~sysconfig.get_python_version` returns a Python version string
1768 such as "3.2".
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001769
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001770It also provides access to the paths and variables corresponding to one of
1771seven named schemes used by :mod:`distutils`. Those include *posix_prefix*,
1772*posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*:
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001773
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001774* :func:`~sysconfig.get_paths` makes a dictionary containing installation paths
1775 for the current installation scheme.
1776* :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific
1777 variables.
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001778
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001779There is also a convenient command-line interface::
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001780
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001781 C:\Python32>python -m sysconfig
1782 Platform: "win32"
1783 Python version: "3.2"
1784 Current installation scheme: "nt"
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001785
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001786 Paths:
1787 data = "C:\Python32"
Łukasz Langa79a06ed2010-12-17 22:05:46 +00001788 include = "C:\Python32\Include"
1789 platinclude = "C:\Python32\Include"
1790 platlib = "C:\Python32\Lib\site-packages"
1791 platstdlib = "C:\Python32\Lib"
1792 purelib = "C:\Python32\Lib\site-packages"
1793 scripts = "C:\Python32\Scripts"
1794 stdlib = "C:\Python32\Lib"
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001795
1796 Variables:
1797 BINDIR = "C:\Python32"
Łukasz Langa79a06ed2010-12-17 22:05:46 +00001798 BINLIBDEST = "C:\Python32\Lib"
1799 EXE = ".exe"
1800 INCLUDEPY = "C:\Python32\Include"
1801 LIBDEST = "C:\Python32\Lib"
1802 SO = ".pyd"
1803 VERSION = "32"
1804 abiflags = ""
1805 base = "C:\Python32"
1806 exec_prefix = "C:\Python32"
1807 platbase = "C:\Python32"
1808 prefix = "C:\Python32"
1809 projectbase = "C:\Python32"
1810 py_version = "3.2"
1811 py_version_nodot = "32"
1812 py_version_short = "3.2"
1813 srcdir = "C:\Python32"
1814 userbase = "C:\Documents and Settings\Raymond\Application Data\Python"
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001815
1816pdb
1817---
1818
1819The :mod:`pdb` debugger module gained a number of usability improvements:
Raymond Hettingerb5d79332010-12-07 02:04:56 +00001820
Raymond Hettinger99db3fd2010-12-15 19:33:49 +00001821* :file:`pdb.py` now has a ``-c`` option that executes commands as given in a
1822 :file:`.pdbrc` script file.
1823* A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands
1824 that continue debugging.
1825* The :class:`Pdb` class constructor now accepts a *nosigint* argument.
Raymond Hettinger51e21072011-01-10 23:38:15 +00001826* New commands: ``l(list)``, ``ll(long list)`` and ``source`` for
Raymond Hettinger99db3fd2010-12-15 19:33:49 +00001827 listing source code.
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001828* New commands: ``display`` and ``undisplay`` for showing or hiding
Raymond Hettinger99db3fd2010-12-15 19:33:49 +00001829 the value of an expression if it has changed.
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001830* New command: ``interact`` for starting an interactive interpreter containing
Raymond Hettinger99db3fd2010-12-15 19:33:49 +00001831 the global and local names found in the current scope.
Raymond Hettingerde2e6182011-01-10 05:40:57 +00001832* Breakpoints can be cleared by breakpoint number.
Raymond Hettingerb5d79332010-12-07 02:04:56 +00001833
Georg Brandl101234b2010-12-18 11:53:25 +00001834(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)
1835
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001836configparser
1837------------
Raymond Hettinger3f9734c2010-12-07 01:47:52 +00001838
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001839The :mod:`configparser` module was modified to improve usability and
1840predictability of the default parser and its supported INI syntax. The old
1841:class:`ConfigParser` class was removed in favor of :class:`SafeConfigParser`
Raymond Hettinger04129742010-12-18 10:57:50 +00001842which has in turn been renamed to :class:`~configparser.ConfigParser`. Support
1843for inline comments is now turned off by default and section or option
1844duplicates are not allowed in a single configuration source.
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001845
1846Config parsers gained a new API based on the mapping protocol::
1847
Raymond Hettinger02dd70b2011-01-17 23:39:39 +00001848 >>> parser = ConfigParser()
1849 >>> parser.read_string("""
1850 [DEFAULT]
1851 location = upper left
1852 visible = yes
1853 editable = no
1854 color = blue
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001855
Raymond Hettinger02dd70b2011-01-17 23:39:39 +00001856 [main]
1857 title = Main Menu
1858 color = green
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001859
Raymond Hettinger02dd70b2011-01-17 23:39:39 +00001860 [options]
1861 title = Options
1862 """)
1863 >>> parser['main']['color']
1864 'green'
1865 >>> parser['main']['editable']
1866 'no'
1867 >>> section = parser['options']
1868 >>> section['title']
1869 'Options'
1870 >>> section['title'] = 'Options (editable: %(editable)s)'
1871 >>> section['title']
1872 'Options (editable: no)'
1873
1874The new API is implemented on top of the classical API, so custom parser
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001875subclasses should be able to use it without modifications.
1876
1877The INI file structure accepted by config parsers can now be customized. Users
Raymond Hettinger04129742010-12-18 10:57:50 +00001878can specify alternative option/value delimiters and comment prefixes, change the
Raymond Hettinger02dd70b2011-01-17 23:39:39 +00001879name of the *DEFAULT* section or switch the interpolation syntax.
1880
Raymond Hettingerba5512f2011-01-18 08:28:01 +00001881There is support for pluggable interpolation including an additional interpolation
Raymond Hettinger02dd70b2011-01-17 23:39:39 +00001882handler :class:`~configparser.ExtendedInterpolation`::
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001883
1884 >>> parser = ConfigParser(interpolation=ExtendedInterpolation())
1885 >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'},
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001886 'custom': {'prefix': '/usr/local'}})
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001887 >>> parser.read_string("""
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00001888 [buildout]
1889 parts =
1890 zope9
1891 instance
1892 find-links =
1893 ${buildout:directory}/downloads/dist
1894
1895 [zope9]
1896 recipe = plone.recipe.zope9install
1897 location = /opt/zope
1898
1899 [instance]
1900 recipe = plone.recipe.zope9instance
1901 zope9-location = ${zope9:location}
1902 zope-conf = ${custom:prefix}/etc/zope.conf
1903 """)
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001904 >>> parser['buildout']['find-links']
1905 '\n/home/ambv/zope9/downloads/dist'
1906 >>> parser['instance']['zope-conf']
1907 '/usr/local/etc/zope.conf'
1908 >>> instance = parser['instance']
1909 >>> instance['zope-conf']
1910 '/usr/local/etc/zope.conf'
1911 >>> instance['zope9-location']
1912 '/opt/zope'
1913
1914A number of smaller features were also introduced, like support for specifying
Raymond Hettinger04129742010-12-18 10:57:50 +00001915encoding in read operations, specifying fallback values for get-functions, or
1916reading directly from dictionaries and strings.
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00001917
1918(All changes contributed by Łukasz Langa.)
1919
Raymond Hettinger9a236b02011-01-24 09:01:27 +00001920urllib.parse
1921------------
1922
1923A number of usability improvements were made for the :mod:`urllib.parse` module.
1924
1925The :func:`~urllib.parse.urlparse` function now supports `IPv6
1926<http://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`:
1927
1928 >>> import urllib.parse
1929 >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/')
1930 ParseResult(scheme='http',
1931 netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]',
1932 path='/foo/',
1933 params='',
1934 query='',
1935 fragment='')
1936
1937The :func:`~urllib.parse.urldefrag` function now returns a :term:`named tuple`::
1938
1939 >>> r = urllib.parse.urldefrag('http://python.org/about/#target')
1940 >>> r
1941 DefragResult(url='http://python.org/about/', fragment='target')
1942 >>> r[0]
1943 'http://python.org/about/
1944 >>> r.fragment
1945 'target'
1946
1947And, the :func:`~urllib.parse.urlencode` function is now much more flexible,
1948accepting either a string or bytes type for the *query* argument. If it is a
1949string, then the *safe*, *encoding*, and *error* parameters are sent to
1950:func:`~urllib.parse.quote_plus` for encoding::
1951
1952 >>> urllib.parse.urlencode([
1953 ('type', 'telenovela'),
1954 ('name', '¿Dónde Está Elisa?')],
1955 encoding='latin-1')
1956 'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F'
1957
Georg Brandl009a6bd2011-01-24 19:59:08 +00001958As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse`
Raymond Hettinger9a236b02011-01-24 09:01:27 +00001959functions now accept ASCII-encoded byte strings as input, so long as they are
1960not mixed with regular strings. If ASCII-encoded byte strings are given as
1961parameters, the return types will also be an ASCII-encoded byte strings:
1962
1963 >>> urllib.parse.urlparse(b'http://www.python.org:80/about/')
1964 ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80',
1965 path=b'/about/', params=b'', query=b'', fragment=b'')
1966
1967(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in :issue:`2987`,
1968:issue:`5468`, and :issue:`9873`.)
1969
Raymond Hettinger3df46212011-01-06 02:01:26 +00001970
Antoine Pitroud42bc512009-11-10 23:18:31 +00001971Multi-threading
1972===============
1973
Georg Brandlda0a2112010-09-05 11:28:33 +00001974* The mechanism for serializing execution of concurrently running Python threads
Raymond Hettingerba5512f2011-01-18 08:28:01 +00001975 (generally known as the :term:`GIL` or :term:`Global Interpreter Lock`) has
1976 been rewritten. Among the objectives were more predictable switching
1977 intervals and reduced overhead due to lock contention and the number of
1978 ensuing system calls. The notion of a "check interval" to allow thread
1979 switches has been abandoned and replaced by an absolute duration expressed in
1980 seconds. This parameter is tunable through :func:`sys.setswitchinterval()`.
1981 It currently defaults to 5 milliseconds.
Antoine Pitroud42bc512009-11-10 23:18:31 +00001982
1983 Additional details about the implementation can be read from a `python-dev
1984 mailing-list message
1985 <http://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_
Georg Brandlda0a2112010-09-05 11:28:33 +00001986 (however, "priority requests" as exposed in this message have not been kept
1987 for inclusion).
Antoine Pitroud42bc512009-11-10 23:18:31 +00001988
Georg Brandl5e73a812010-04-22 07:02:51 +00001989 (Contributed by Antoine Pitrou.)
Antoine Pitroud42bc512009-11-10 23:18:31 +00001990
Georg Brandlda0a2112010-09-05 11:28:33 +00001991* Regular and recursive locks now accept an optional *timeout* argument to their
Raymond Hettingerba5512f2011-01-18 08:28:01 +00001992 :meth:`~threading.Lock.acquire` method. (Contributed by Antoine Pitrou;
1993 :issue:`7316`.)
Georg Brandlda0a2112010-09-05 11:28:33 +00001994
Raymond Hettingerbba537b2010-12-15 18:20:19 +00001995* Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout*
Georg Brandlda0a2112010-09-05 11:28:33 +00001996 argument. (Contributed by Torsten Landschoff; :issue:`850728`.)
Antoine Pitroue95a9ff2010-05-04 23:31:41 +00001997
Antoine Pitrou810023d2010-12-15 22:59:16 +00001998* Regular and recursive lock acquisitions can now be interrupted by signals on
Raymond Hettingerc136b042011-01-18 07:15:39 +00001999 platforms using Pthreads. This means that Python programs that deadlock while
Antoine Pitrou810023d2010-12-15 22:59:16 +00002000 acquiring locks can be successfully killed by repeatedly sending SIGINT to the
Georg Brandleebb2522010-12-18 12:01:15 +00002001 process (by pressing :kbd:`Ctrl+C` in most shells).
Antoine Pitrou810023d2010-12-15 22:59:16 +00002002 (Contributed by Reid Kleckner; :issue:`8844`.)
2003
Antoine Pitroud42bc512009-11-10 23:18:31 +00002004
Raymond Hettinger92ba2862010-09-06 01:16:46 +00002005Optimizations
2006=============
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002007
Raymond Hettinger92ba2862010-09-06 01:16:46 +00002008A number of small performance enhancements have been added:
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002009
Antoine Pitroud3052002010-09-15 15:09:40 +00002010* Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as
Raymond Hettinger92ba2862010-09-06 01:16:46 +00002011 being a test for membership in a set of constants. The optimizer recasts the
2012 :class:`set` as a :class:`frozenset` and stores the pre-built constant.
2013
2014 Now that the speed penalty is gone, it is practical to start writing
2015 membership tests using set-notation. This style is both semantically clear
2016 and operationally fast::
2017
2018 extension = name.rpartition('.')[2]
2019 if extension in {'xml', 'html', 'xhtml', 'css'}:
2020 handle(name)
2021
Raymond Hettingerc136b042011-01-18 07:15:39 +00002022 (Patch and additional tests contributed by Dave Malcolm; :issue:`6690`).
Raymond Hettinger92ba2862010-09-06 01:16:46 +00002023
Antoine Pitroud3052002010-09-15 15:09:40 +00002024* Serializing and unserializing data using the :mod:`pickle` module is now
Raymond Hettingerdadf93c2010-12-05 02:56:21 +00002025 several times faster.
2026
2027 (Contributed by Alexandre Vassalotti, Antoine Pitrou
Antoine Pitrouff150f22010-10-22 21:41:05 +00002028 and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.)
Antoine Pitroud3052002010-09-15 15:09:40 +00002029
Raymond Hettingerc269ae82010-12-05 01:01:52 +00002030* The `Timsort algorithm <http://en.wikipedia.org/wiki/Timsort>`_ used in
Raymond Hettingerffad35e2010-12-14 21:12:03 +00002031 :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory
Raymond Hettingerc269ae82010-12-05 01:01:52 +00002032 when called with a :term:`key function`. Previously, every element of
2033 a list was wrapped with a temporary object that remembered the key value
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002034 associated with each element. Now, two arrays of keys and values are
Raymond Hettingerc136b042011-01-18 07:15:39 +00002035 sorted in parallel. This saves the memory consumed by the sort wrappers,
2036 and it saves time lost to delegating comparisons.
Raymond Hettingerc269ae82010-12-05 01:01:52 +00002037
Raymond Hettingereb70b902011-01-10 21:26:49 +00002038 (Patch by Daniel Stutzbach in :issue:`9915`.)
Raymond Hettingerc269ae82010-12-05 01:01:52 +00002039
Raymond Hettingerdadf93c2010-12-05 02:56:21 +00002040* JSON decoding performance is improved and memory consumption is reduced
Raymond Hettinger413abbc2010-12-05 07:06:47 +00002041 whenever the same string is repeated for multiple keys. Also, JSON encoding
Raymond Hettingerdadf93c2010-12-05 02:56:21 +00002042 now uses the C speedups when the ``sort_keys`` argument is true.
2043
2044 (Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and
2045 Antoine Pitrou in :issue:`10314`.)
2046
Raymond Hettinger21ec4bc2010-12-10 01:09:01 +00002047* Recursive locks (created with the :func:`threading.RLock` API) now benefit
2048 from a C implementation which makes them as fast as regular locks, and between
2049 10x and 15x faster than their previous pure Python implementation.
2050
2051 (Contributed by Antoine Pitrou; :issue:`3001`.)
2052
Raymond Hettingerdadf93c2010-12-05 02:56:21 +00002053* The fast-search algorithm in stringlib is now used by the :meth:`split`,
2054 :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on
2055 :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the
2056 algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and
2057 :meth:`rpartition`.
2058
2059 (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.)
2060
Raymond Hettinger480ed782010-12-15 22:07:15 +00002061
2062* String to integer conversions now work two "digits" at a time, reducing the
2063 number of division and modulo operations.
2064
2065 (:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.)
2066
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +00002067There were several other minor optimizations. Set differencing now runs faster
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002068when one operand is much larger than the other (patch by Andress Bennetts in
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +00002069:issue:`8685`). The :meth:`array.repeat` method has a faster implementation
2070(:issue:`1569291` by Alexander Belopolsky). The :class:`BaseHTTPRequestHandler`
2071has more efficient buffering (:issue:`3709` by Andrew Schaaf). The
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002072multi-argument form of :func:`operator.attrgetter` function now runs slightly
Raymond Hettingerd8fae4e2010-12-05 05:39:54 +00002073faster (:issue:`10160` by Christos Georgiou). And :class:`ConfigParser` loads
2074multi-line arguments a bit faster (:issue:`7113` by Łukasz Langa).
2075
Antoine Pitroud3052002010-09-15 15:09:40 +00002076
Victor Stinner47ce9652010-10-29 00:57:35 +00002077Unicode
2078=======
Victor Stinner94908bb2010-08-18 21:23:25 +00002079
Raymond Hettinger2270d582011-01-20 09:04:39 +00002080Python has been updated to `Unicode 6.0.0
2081<http://unicode.org/versions/Unicode6.0.0/>`_. The update to the standard adds
2082over 2,000 new characters including `emoji <http://en.wikipedia.org/wiki/Emoji>`_
2083symbols which are important for mobile phones.
Alexander Belopolsky507e3f82010-12-02 00:05:57 +00002084
Raymond Hettinger2270d582011-01-20 09:04:39 +00002085In addition, the updated standard has altered the character properties for two
2086Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character
2087(U+19DA), making the former eligible for use in identifiers while disqualifying
2088the latter. For more information, see `Unicode Character Database Changes
2089<http://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_.
Alexander Belopolsky507e3f82010-12-02 00:05:57 +00002090
Alexander Belopolsky507e3f82010-12-02 00:05:57 +00002091
Raymond Hettinger2270d582011-01-20 09:04:39 +00002092Codecs
2093======
Raymond Hettingerc74d5182010-12-02 01:38:25 +00002094
Raymond Hettinger2270d582011-01-20 09:04:39 +00002095Support was added for *cp720* Arabic DOS encoding (:issue:`1616979`).
Alexander Belopolsky84cc0622010-12-08 21:38:46 +00002096
Raymond Hettinger2270d582011-01-20 09:04:39 +00002097MBCS encoding no longer ignores the error handler argument. In the default
2098strict mode, it raises an :exc:`UnicodeDecodeError` when it encounters an
2099undecodable byte sequence and an :exc:`UnicodeEncodeError` for an unencodable
2100character.
Alexander Belopolsky507e3f82010-12-02 00:05:57 +00002101
Raymond Hettinger2270d582011-01-20 09:04:39 +00002102The MBCS codec supports ``'strict'`` and ``'ignore'`` error handlers for
2103decoding, and ``'strict'`` and ``'replace'`` for encoding.
Victor Stinnere8d51452010-08-19 01:05:19 +00002104
Raymond Hettinger2270d582011-01-20 09:04:39 +00002105To emulate Python3.1 MBCS encoding, select the ``'ignore'`` handler for decoding
2106and the ``'replace'`` handler for encoding.
Victor Stinner47ce9652010-10-29 00:57:35 +00002107
Raymond Hettinger2e042d32011-01-21 09:18:19 +00002108On Mac OS X, Python decodes command line arguments with ``'utf-8'`` rather than
Raymond Hettinger2270d582011-01-20 09:04:39 +00002109the locale encoding.
Victor Stinner47ce9652010-10-29 00:57:35 +00002110
Raymond Hettinger00db6aa2011-01-20 09:47:04 +00002111By default, :mod:`tarfile` uses ``'utf-8'`` encoding on Windows (instead of
2112``'mbcs'``) and the ``'surrogateescape'`` error handler on all operating
2113systems.
Raymond Hettinger480ed782010-12-15 22:07:15 +00002114
Victor Stinner94908bb2010-08-18 21:23:25 +00002115
Raymond Hettinger1fa76822010-12-06 23:31:36 +00002116Documentation
2117=============
2118
2119The documentation continues to be improved.
2120
2121A table of quick links has been added to the top of lengthy sections such as
2122:ref:`built-in-funcs`. In the case of :mod:`itertools`, the links are
2123accompanied by tables of cheatsheet-style summaries to provide an overview and
2124memory jog without having to read all of the docs.
2125
Raymond Hettinger51e21072011-01-10 23:38:15 +00002126In some cases, the pure Python source code can be a helpful adjunct to the
2127documentation, so now many modules now feature quick links to the latest version
2128of the source code. For example, the :mod:`functools` module documentation has
2129a quick link at the top labeled: **Source code** :source:`Lib/functools.py`.
Raymond Hettinger1fa76822010-12-06 23:31:36 +00002130
2131The docs now contain more examples and recipes. In particular, :mod:`re` module
2132has an extensive section, :ref:`re-examples`. Likewise, the :mod:`itertools`
2133module continues to be updated with new :ref:`itertools-recipes`.
2134
Raymond Hettinger677e10a2010-12-07 06:45:30 +00002135The :mod:`datetime` module now has an auxiliary implementation in pure Python.
2136No functionality was changed. This just provides an easier-to-read
2137alternate implementation. (Contributed by Alexander Belopolsky.)
2138
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002139The unmaintained :file:`Demo` directory has been removed. Some demos were
2140integrated into the documentation, some were moved to the :file:`Tools/demo`
2141directory, and others were removed altogether. (Contributed by Georg Brandl.)
Raymond Hettinger2169ee22011-01-05 22:27:49 +00002142
Raymond Hettinger1fa76822010-12-06 23:31:36 +00002143
2144IDLE
2145====
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002146
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002147* The format menu now has an option to clean source files by stripping
Raymond Hettinger2169ee22011-01-05 22:27:49 +00002148 trailing whitespace.
2149
2150 (Contributed by Raymond Hettinger; :issue:`5150`.)
2151
2152* IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk.
2153
2154 (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; :issue:`6075`.)
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002155
Raymond Hettinger00db6aa2011-01-20 09:47:04 +00002156Code Repository
2157===============
2158
2159In addition to the existing Subversion code repository at http://svn.python.org
2160there is now a `Mercurial <http://mercurial.selenic.com/>`_ repository at
2161http://hg.python.org/ .
2162
2163After the 3.2 release, there are plans to switch to Mercurial as the primary
2164repository. This distributed version control system should make it easier for
2165members of the community to create and share external changesets. See
2166:pep:`385` for details.
2167
2168To learn to use the new version control system, see the `tutorial by Joel
2169Spolsky <http://hginit.com>`_ or the `guide to Mercurial workflows
2170<http://mercurial.selenic.com/guide/>`_.
2171
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002172
2173Build and C API Changes
2174=======================
2175
2176Changes to Python's build process and to the C API include:
2177
Raymond Hettinger2169ee22011-01-05 22:27:49 +00002178* The *idle*, *pydoc* and *2to3* scripts are now installed with a
2179 version-specific suffix on ``make altinstall`` (:issue:`10679`).
2180
Georg Brandlda0a2112010-09-05 11:28:33 +00002181* The C functions that access the Unicode Database now accept and return
2182 characters from the full Unicode range, even on narrow unicode builds
Raymond Hettinger1784ff02010-09-05 01:00:19 +00002183 (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference
Georg Brandlda0a2112010-09-05 11:28:33 +00002184 in Python is that :func:`unicodedata.numeric` now returns the correct value
2185 for large code points, and :func:`repr` may consider more characters as
2186 printable.
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002187
Raymond Hettinger1784ff02010-09-05 01:00:19 +00002188 (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.)
2189
Georg Brandlda0a2112010-09-05 11:28:33 +00002190* Computed gotos are now enabled by default on supported compilers (which are
Raymond Hettingerdb9044e2010-09-06 01:29:23 +00002191 detected by the configure script). They can still be disabled selectively by
Georg Brandlda0a2112010-09-05 11:28:33 +00002192 specifying ``--without-computed-gotos``.
Raymond Hettinger1784ff02010-09-05 01:00:19 +00002193
Georg Brandlda0a2112010-09-05 11:28:33 +00002194 (Contributed by Antoine Pitrou; :issue:`9203`.)
2195
Amaury Forgeot d'Arcfeb73072010-09-12 22:42:57 +00002196* The option ``--with-wctype-functions`` was removed. The built-in unicode
2197 database is now used for all functions.
2198
2199 (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.)
2200
Raymond Hettinger480ed782010-12-15 22:07:15 +00002201* Hash values are now values of a new type, :c:type:`Py_hash_t`, which is
2202 defined to be the same size as a pointer. Previously they were of type long,
2203 which on some 64-bit operating systems is still only 32 bits long. As a
2204 result of this fix, :class:`set` and :class:`dict` can now hold more than
2205 ``2**32`` entries on builds with 64-bit pointers (previously, they could grow
2206 to that size but their performance degraded catastrophically).
Skip Montanaro961aaf52010-10-17 22:22:24 +00002207
Raymond Hettinger480ed782010-12-15 22:07:15 +00002208 (Suggested by Raymond Hettinger and implemented by Benjamin Peterson;
2209 :issue:`9778`.)
2210
2211* A new macro :c:macro:`Py_VA_COPY` copies the state of the variable argument
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002212 list. It is equivalent to C99 *va_copy* but available on all Python platforms
Raymond Hettinger480ed782010-12-15 22:07:15 +00002213 (:issue:`2443`).
2214
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002215* A new C API function :c:func:`PySys_SetArgvEx` allows an embedded interpreter
2216 to set :attr:`sys.argv` without also modifying :attr:`sys.path`
Raymond Hettinger480ed782010-12-15 22:07:15 +00002217 (:issue:`5753`).
2218
2219* :c:macro:`PyEval_CallObject` is now only available in macro form. The
2220 function declaration, which was kept for backwards compatibility reasons, is
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002221 now removed -- the macro was introduced in 1997 (:issue:`8276`).
Raymond Hettinger480ed782010-12-15 22:07:15 +00002222
2223* The is a new function :c:func:`PyLong_AsLongLongAndOverflow` which
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002224 is analogous to :c:func:`PyLong_AsLongAndOverflow`. They both serve to
Raymond Hettinger480ed782010-12-15 22:07:15 +00002225 convert Python :class:`int` into a native fixed-width type while providing
2226 detection of cases where the conversion won't fit (:issue:`7767`).
2227
2228* The :c:func:`PyUnicode_CompareWithASCIIString` now returns *not equal*
Raymond Hettingerba5512f2011-01-18 08:28:01 +00002229 if the Python string is *NUL* terminated.
Raymond Hettinger480ed782010-12-15 22:07:15 +00002230
2231* There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is
2232 like :c:func:`PyErr_NewException` but allows a docstring to be specified.
2233 This lets C exceptions have the same self-documenting capabilities as
2234 their pure Python counterparts (:issue:`7033`).
2235
2236* When compiled with the ``--with-valgrind`` option, the pymalloc
2237 allocator will be automatically disabled when running under Valgrind. This
2238 gives improved memory leak detection when running under Valgrind, while taking
2239 advantage of pymalloc at other times (:issue:`2422`).
2240
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002241* Removed the ``O?`` format from the *PyArg_Parse* functions. The format is no
Raymond Hettinger480ed782010-12-15 22:07:15 +00002242 longer used and it had never been documented (:issue:`8837`).
2243
2244There were a number of other small changes to the C-API. See the
2245:file:`Misc/NEWS` file for a complete list.
Skip Montanaro961aaf52010-10-17 22:22:24 +00002246
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002247
Raymond Hettingerf558ddd2009-06-28 21:37:08 +00002248Porting to Python 3.2
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002249=====================
2250
Georg Brandlda0a2112010-09-05 11:28:33 +00002251This section lists previously described changes and other bugfixes that may
2252require changes to your code:
Raymond Hettinger6e6565b2009-06-28 20:56:11 +00002253
Raymond Hettingerd73be672010-12-18 10:48:26 +00002254* The :mod:`configparser` module has a number of clean-ups. The major change is
2255 to replace the old :class:`ConfigParser` class with long-standing preferred
2256 alternative :class:`SafeConfigParser`. In addition there are a number of
Raymond Hettingerc8a16862011-01-18 09:01:34 +00002257 smaller incompatibilities:
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00002258
Raymond Hettingerd73be672010-12-18 10:48:26 +00002259 * The interpolation syntax is now validated on
2260 :meth:`~configparser.ConfigParser.get` and
2261 :meth:`~configparser.ConfigParser.set` operations. In the default
2262 interpolation scheme, only two tokens with percent signs are valid: ``%(name)s``
2263 and ``%%``, the latter being an escaped percent sign.
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00002264
Raymond Hettingerd73be672010-12-18 10:48:26 +00002265 * The :meth:`~configparser.ConfigParser.set` and
2266 :meth:`~configparser.ConfigParser.add_section` methods now verify that
2267 values are actual strings. Formerly, unsupported types could be introduced
2268 unintentionally.
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00002269
Raymond Hettinger2b8861f2010-12-18 11:20:52 +00002270 * Duplicate sections or options from a single source now raise either
Raymond Hettingerd73be672010-12-18 10:48:26 +00002271 :exc:`~configparser.DuplicateSectionError` or
2272 :exc:`~configparser.DuplicateOptionError`. Formerly, duplicates would
2273 silently overwrite a previous entry.
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00002274
Raymond Hettingerd73be672010-12-18 10:48:26 +00002275 * Inline comments are now disabled by default so now the **;** character
2276 can be safely used in values.
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00002277
Raymond Hettingerd73be672010-12-18 10:48:26 +00002278 * Comments now can be indented. Consequently, for **;** or **#** to appear at
2279 the start of a line in multiline values, it has to be interpolated. This
Raymond Hettinger2b8861f2010-12-18 11:20:52 +00002280 keeps comment prefix characters in values from being mistaken as comments.
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00002281
Raymond Hettingerd73be672010-12-18 10:48:26 +00002282 * ``""`` is now a valid value and is no longer automatically converted to an
2283 empty string. For empty strings, use ``"option ="`` in a line.
Łukasz Langa2b38b6c2010-12-17 21:57:32 +00002284
Antoine Pitroucd889af2010-10-06 21:13:56 +00002285* The :mod:`nntplib` module was reworked extensively, meaning that its APIs
2286 are often incompatible with the 3.1 APIs.
2287
Raymond Hettinger1fa76822010-12-06 23:31:36 +00002288* :class:`bytearray` objects can no longer be used as filenames; instead,
2289 they should be converted to :class:`bytes`.
Victor Stinnerdcb24032010-04-22 12:08:36 +00002290
Raymond Hettinger399bf7b2011-01-24 10:11:12 +00002291* The :meth:`array.tostring` and :meth:`array.fromstring` have been renamed to
Raymond Hettinger50307b62011-01-24 01:18:30 +00002292 :meth:`array.tobytes` and :meth:`array.frombytes` for clarity. The old names
2293 have been deprecated. (See :issue:`8990`.)
2294
Raymond Hettinger51e21072011-01-10 23:38:15 +00002295* ``PyArg_Parse*()`` functions:
Victor Stinner3dcb5ac2010-06-08 22:54:19 +00002296
Victor Stinner25e8ec42010-06-25 00:02:38 +00002297 * "t#" format has been removed: use "s#" or "s*" instead
2298 * "w" and "w#" formats has been removed: use "w*" instead
2299
Georg Brandl60203b42010-10-06 10:11:56 +00002300* The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap
2301 opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used
Éric Araujo4234ad42010-09-05 17:32:25 +00002302 instead; the new type has a well-defined interface for passing typing safety
Georg Brandlda0a2112010-09-05 11:28:33 +00002303 information and a less complicated signature for calling a destructor.
Victor Stinner0cbec572010-09-12 20:32:57 +00002304
Raymond Hettingere0a96002010-12-15 17:54:13 +00002305* The :func:`sys.setfilesystemencoding` function was removed because
2306 it had a flawed design.
Raymond Hettinger3fcf0022010-12-08 01:13:53 +00002307
Raymond Hettingere0a96002010-12-15 17:54:13 +00002308* The :func:`random.seed` function and method now salt string seeds with an
2309 sha512 hash function. To access the previous version of *seed* in order to
2310 reproduce Python 3.1 sequences, set the *version* argument to *1*,
2311 ``random.seed(s, version=1)``.
Raymond Hettinger21ec4bc2010-12-10 01:09:01 +00002312
Raymond Hettinger522cc0a2010-12-10 01:19:15 +00002313* The previously deprecated :func:`string.maketrans` function has been removed
Raymond Hettinger51e21072011-01-10 23:38:15 +00002314 in favor of the static methods :meth:`bytes.maketrans` and
Raymond Hettinger522cc0a2010-12-10 01:19:15 +00002315 :meth:`bytearray.maketrans`. This change solves the confusion around which
2316 types were supported by the :mod:`string` module. Now, :class:`str`,
2317 :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and
2318 **translate** methods with intermediate translation tables of the appropriate
2319 type.
2320
2321 (Contributed by Georg Brandl; :issue:`5675`.)
2322
2323* The previously deprecated :func:`contextlib.nested` function has been removed
2324 in favor of a plain :keyword:`with` statement which can accept multiple
2325 context managers. The latter technique is faster (because it is built-in),
2326 and it does a better job finalizing multiple context managers when one of them
2327 raises an exception::
2328
Raymond Hettinger6f0d59b2011-01-17 23:10:55 +00002329 with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
2330 for line in infile:
2331 if '<critical>' in line:
2332 outfile.write(line)
Raymond Hettinger522cc0a2010-12-10 01:19:15 +00002333
2334 (Contributed by Georg Brandl and Mattias Brändström;
2335 `appspot issue 53094 <http://codereview.appspot.com/53094>`_.)
Victor Stinnerda9ec992010-12-28 13:26:42 +00002336
Raymond Hettinger2169ee22011-01-05 22:27:49 +00002337* :func:`struct.pack` now only allows bytes for the ``s`` string pack code.
2338 Formerly, it would accept text arguments and implicitly encode them to bytes
2339 using UTF-8. This was problematic because it made assumptions about the
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002340 correct encoding and because a variable-length encoding can fail when writing
Raymond Hettinger2169ee22011-01-05 22:27:49 +00002341 to fixed length segment of a structure.
Victor Stinnerda9ec992010-12-28 13:26:42 +00002342
Raymond Hettinger2169ee22011-01-05 22:27:49 +00002343 Code such as ``struct.pack('<6sHHBBB', 'GIF87a', x, y)`` should be rewritten
2344 with to use bytes instead of text, ``struct.pack('<6sHHBBB', b'GIF87a', x, y)``.
2345
Raymond Hettingerde2e6182011-01-10 05:40:57 +00002346 (Discovered by David Beazley and fixed by Victor Stinner; :issue:`10783`.)
Raymond Hettingere40808a2011-01-05 23:00:00 +00002347
2348* The :class:`xml.etree.ElementTree` class now raises an
2349 :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it
2350 raised a :exc:`xml.parsers.expat.ExpatError`.
2351
2352* The new, longer :func:`str` value on floats may break doctests which rely on
2353 the old output format.
Antoine Pitroubcba4342011-01-16 18:29:34 +00002354
2355* In :class:`subprocess.Popen`, the default value for *close_fds* is now
2356 ``True`` under Unix; under Windows, it is ``True`` if the three standard
2357 streams are set to ``None``, ``False`` otherwise. Previously, *close_fds*
2358 was always ``False`` by default, which produced difficult to solve bugs
2359 or race conditions when open file descriptors would leak into the child
2360 process.
2361
Antoine Pitrouf7fb7622011-01-16 18:34:09 +00002362* Support for legacy HTTP 0.9 has been removed from :mod:`urllib.request`
2363 and :mod:`http.client`. Such support is still present on the server side
2364 (in :mod:`http.server`).
2365
2366 (Contributed by Antoine Pitrou, :issue:`10711`.)
2367
Antoine Pitrou2e8ec222011-01-16 18:41:36 +00002368* SSL sockets in timeout mode now raise :exc:`socket.timeout` when a timeout
2369 occurs, rather than a generic :exc:`~ssl.SSLError`.
2370
2371 (Contributed by Antoine Pitrou, :issue:`10272`.)
Antoine Pitrouebeb9032011-01-16 18:45:17 +00002372
2373* The misleading functions :c:func:`PyEval_AcquireLock()` and
2374 :c:func:`PyEval_ReleaseLock()` have been officially deprecated. The
2375 thread-state aware APIs (such as :c:func:`PyEval_SaveThread()`
2376 and :c:func:`PyEval_RestoreThread()`) should be used instead.
Raymond Hettinger50307b62011-01-24 01:18:30 +00002377
2378* Due to security risks, :func:`asyncore.handle_accept` has been deprecated, and
2379 a new functions, :func:`asyncore.handle_accepted` was added to replace it.
2380
2381 (Contributed by Giampaolo Rodola in :issue:`6706`.)