blob: 66232919aafb057180f27eb593adc5142b999116 [file] [log] [blame]
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001\documentclass{howto}
2\usepackage{distutils}
3% $Id$
4
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +00005% Don't write extensive text for new sections; I'll do that.
6% Feel free to add commented-out reminders of things that need
7% to be covered. --amk
8
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +00009% XXX pydoc can display links to module docs -- but when?
10%
11
Fred Drakeed0fa3d2003-07-30 19:14:09 +000012\title{What's New in Python 2.4}
Andrew M. Kuchling2cc0c302004-09-10 12:38:36 +000013\release{0.4}
Fred Drakeed0fa3d2003-07-30 19:14:09 +000014\author{A.M.\ Kuchling}
Fred Drakeb914ef02004-01-02 06:57:50 +000015\authoraddress{
16 \strong{Python Software Foundation}\\
17 Email: \email{amk@amk.ca}
18}
Fred Drakeed0fa3d2003-07-30 19:14:09 +000019
20\begin{document}
21\maketitle
22\tableofcontents
23
Andrew M. Kuchling9fa544c2004-09-23 20:17:26 +000024This article explains the new features in Python 2.4 beta1, scheduled
25for release in mid-October. The final version of Python 2.4 is
Andrew M. Kuchling3294e9d2004-08-31 11:26:23 +000026expected to be released around December 2004.
Fred Drakeed0fa3d2003-07-30 19:14:09 +000027
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +000028Python 2.4 is a medium-sized release. It doesn't introduce as many
Andrew M. Kuchling3b790912004-07-04 16:39:40 +000029changes as the radical Python 2.2, but introduces more features than
30the conservative 2.3 release did. The most significant new language
Andrew M. Kuchling3294e9d2004-08-31 11:26:23 +000031features (as of this writing) are function decorators and generator
32expressions; most other changes are to the standard library.
Andrew M. Kuchlinga331e862004-09-10 13:05:22 +000033% XXX update these figures as we go
34According to the CVS change logs, there were 421 patches applied and
35413 bugs fixed between Python 2.3 and 2.4. Both figures are likely to
36be underestimates.
37
Fred Drakeed0fa3d2003-07-30 19:14:09 +000038
39This article doesn't attempt to provide a complete specification of
Andrew M. Kuchling3b790912004-07-04 16:39:40 +000040every single new feature, but instead provides a convenient overview.
41For full details, you should refer to the documentation for Python
422.4, such as the \citetitle[../lib/lib.html]{Python Library Reference}
43and the \citetitle[../ref/ref.html]{Python Reference Manual}. If you
44want to understand the complete implementation and design rationale,
45refer to the PEP for a particular new feature or to the module
46documentation.
Fred Drakeed0fa3d2003-07-30 19:14:09 +000047
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +000048
Raymond Hettinger7e0282f2003-11-24 07:14:54 +000049%======================================================================
50\section{PEP 218: Built-In Set Objects}
51
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +000052Python 2.3 introduced the \module{sets} module. C implementations of
53set data types have now been added to the Python core as two new
54built-in types, \function{set(\var{iterable})} and
55\function{frozenset(\var{iterable})}. They provide high speed
56operations for membership testing, for eliminating duplicates from
57sequences, and for mathematical operations like unions, intersections,
58differences, and symmetric differences.
Raymond Hettinger7e0282f2003-11-24 07:14:54 +000059
60\begin{verbatim}
61>>> a = set('abracadabra') # form a set from a string
62>>> 'z' in a # fast membership testing
63False
64>>> a # unique letters in a
65set(['a', 'r', 'b', 'c', 'd'])
66>>> ''.join(a) # convert back into a string
67'arbcd'
Raymond Hettingerd4462302003-11-26 17:52:45 +000068
Raymond Hettinger7e0282f2003-11-24 07:14:54 +000069>>> b = set('alacazam') # form a second set
70>>> a - b # letters in a but not in b
71set(['r', 'd', 'b'])
72>>> a | b # letters in either a or b
73set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
74>>> a & b # letters in both a and b
75set(['a', 'c'])
76>>> a ^ b # letters in a or b but not both
77set(['r', 'd', 'b', 'm', 'z', 'l'])
Raymond Hettingerd4462302003-11-26 17:52:45 +000078
Raymond Hettinger7e0282f2003-11-24 07:14:54 +000079>>> a.add('z') # add a new element
80>>> a.update('wxy') # add multiple new elements
81>>> a
82set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])
83>>> a.remove('x') # take one element out
84>>> a
85set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])
86\end{verbatim}
87
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +000088The \function{frozenset} type is an immutable version of \function{set}.
Raymond Hettinger7e0282f2003-11-24 07:14:54 +000089Since it is immutable and hashable, it may be used as a dictionary key or
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +000090as a member of another set.
Raymond Hettinger7e0282f2003-11-24 07:14:54 +000091
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +000092The \module{sets} module remains in the standard library, and may be
93useful if you wish to subclass the \class{Set} or \class{ImmutableSet}
94classes. There are currently no plans to deprecate the module.
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +000095
Raymond Hettinger7e0282f2003-11-24 07:14:54 +000096\begin{seealso}
97\seepep{218}{Adding a Built-In Set Object Type}{Originally proposed by
98Greg Wilson and ultimately implemented by Raymond Hettinger.}
99\end{seealso}
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000100
101%======================================================================
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +0000102\section{PEP 237: Unifying Long Integers and Integers}
103
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000104The lengthy transition process for this PEP, begun in Python 2.2,
Andrew M. Kuchlingd4be86c2004-07-04 01:44:04 +0000105takes another step forward in Python 2.4. In 2.3, certain integer
106operations that would behave differently after int/long unification
107triggered \exception{FutureWarning} warnings and returned values
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000108limited to 32 or 64 bits (depending on your platform). In 2.4, these
109expressions no longer produce a warning and instead produce a
110different result that's usually a long integer.
Andrew M. Kuchlingd4be86c2004-07-04 01:44:04 +0000111
112The problematic expressions are primarily left shifts and lengthy
Raymond Hettingerca1a7752004-07-12 13:00:45 +0000113hexadecimal and octal constants. For example,
114\code{2 \textless{}\textless{} 32} results
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000115in a warning in 2.3, evaluating to 0 on 32-bit platforms. In Python
1162.4, this expression now returns the correct answer, 8589934592.
Andrew M. Kuchlingd4be86c2004-07-04 01:44:04 +0000117
118\begin{seealso}
119\seepep{237}{Unifying Long Integers and Integers}{Original PEP
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000120written by Moshe Zadka and GvR. The changes for 2.4 were implemented by
Andrew M. Kuchlingd4be86c2004-07-04 01:44:04 +0000121Kalle Svensson.}
122\end{seealso}
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +0000123
124%======================================================================
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000125\section{PEP 289: Generator Expressions}
Raymond Hettinger354433a2004-05-19 08:20:33 +0000126
Andrew M. Kuchling38dc2a62004-08-07 13:24:12 +0000127The iterator feature introduced in Python 2.2 and the
128\module{itertools} module make it easier to write programs that loop
129through large data sets without having the entire data set in memory
130at one time. List comprehensions don't fit into this picture very
131well because they produce a Python list object containing all of the
132items, unavoidably pulling them all into memory. When trying to write
133a functionally-styled program, it would be natural to write something
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000134like:
Raymond Hettinger354433a2004-05-19 08:20:33 +0000135
136\begin{verbatim}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000137links = [link for link in get_all_links() if not link.followed]
138for link in links:
139 ...
Raymond Hettinger354433a2004-05-19 08:20:33 +0000140\end{verbatim}
141
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000142instead of
Raymond Hettinger354433a2004-05-19 08:20:33 +0000143
144\begin{verbatim}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000145for link in get_all_links():
146 if link.followed:
147 continue
148 ...
149\end{verbatim}
Raymond Hettinger354433a2004-05-19 08:20:33 +0000150
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000151The first form is more concise and perhaps more readable, but if
152you're dealing with a large number of link objects the second form
Andrew M. Kuchling38dc2a62004-08-07 13:24:12 +0000153would have to be used to avoid having all link objects in memory at
154the same time.
Raymond Hettinger354433a2004-05-19 08:20:33 +0000155
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000156Generator expressions work similarly to list comprehensions but don't
157materialize the entire list; instead they create a generator that will
158return elements one by one. The above example could be written as:
Raymond Hettinger354433a2004-05-19 08:20:33 +0000159
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000160\begin{verbatim}
161links = (link for link in get_all_links() if not link.followed)
162for link in links:
163 ...
164\end{verbatim}
Raymond Hettinger170a6222004-05-19 19:45:19 +0000165
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000166Generator expressions always have to be written inside parentheses, as
167in the above example. The parentheses signalling a function call also
168count, so if you want to create a iterator that will be immediately
169passed to a function you could write:
Raymond Hettinger170a6222004-05-19 19:45:19 +0000170
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000171\begin{verbatim}
172print sum(obj.count for obj in list_all_objects())
173\end{verbatim}
Raymond Hettinger170a6222004-05-19 19:45:19 +0000174
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000175Generator expressions differ from list comprehensions in various small
176ways. Most notably, the loop variable (\var{obj} in the above
177example) is not accessible outside of the generator expression. List
178comprehensions leave the variable assigned to its last value; future
179versions of Python will change this, making list comprehensions match
180generator expressions in this respect.
Raymond Hettinger354433a2004-05-19 08:20:33 +0000181
182\begin{seealso}
183\seepep{289}{Generator Expressions}{Proposed by Raymond Hettinger and
184implemented by Jiwon Seo with early efforts steered by Hye-Shik Chang.}
185\end{seealso}
186
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +0000187
188%======================================================================
189\section{PEP 292: Simpler String Substitutions}
190
191Some new classes in the standard library provide a
192alternative mechanism for substituting variables into strings that's
193better-suited for applications where untrained users need to edit templates.
194
195The usual way of substituting variables by name is the \code{\%}
196operator:
197
198\begin{verbatim}
199>>> '%(page)i: %(title)s' % {'page':2, 'title': 'The Best of Times'}
200'2: The Best of Times'
201\end{verbatim}
202
203When writing the template string, it can be easy to forget the
204\samp{i} or \samp{s} after the closing parenthesis. This isn't a big
205problem if the template is in a Python module, because you run the
206code, get an ``Unsupported format character'' \exception{ValueError},
207and fix the problem. However, consider an application such as Mailman
208where template strings or translations are being edited by users who
209aren't aware of the Python language; the syntax is complicated to
210explain to such users, and if they make a mistake, it's difficult to
211provide helpful feedback to them.
212
213PEP 292 adds a \class{Template} class to the \module{string} module
214that uses \samp{\$} to indicate a substitution. \class{Template} is a
215subclass of the built-in Unicode type, so the result is always a
216Unicode string:
217
218\begin{verbatim}
219>>> import string
220>>> t = string.Template('$page: $title')
Andrew M. Kuchlinga79ec222004-09-10 11:34:39 +0000221>>> t.substitute({'page':2, 'title': 'The Best of Times'})
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +0000222u'2: The Best of Times'
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +0000223\end{verbatim}
224
225% $ Terminate $-mode for Emacs
226
Andrew M. Kuchlinga79ec222004-09-10 11:34:39 +0000227If a key is missing from the dictionary, the \method{substitute} method
228will raise a \exception{KeyError}. There's also a \method{safe_substitute}
229method that ignores missing keys:
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +0000230
231\begin{verbatim}
232>>> t = string.SafeTemplate('$page: $title')
Andrew M. Kuchlinga79ec222004-09-10 11:34:39 +0000233>>> t.safe_substitute({'page':3})
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +0000234u'3: $title'
235\end{verbatim}
236
Andrew M. Kuchlinga331e862004-09-10 13:05:22 +0000237% $ Terminate math-mode for Emacs
238
239
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +0000240\begin{seealso}
241\seepep{292}{Simpler String Substitutions}{Written and implemented
242by Barry Warsaw.}
243\end{seealso}
244
245
Raymond Hettinger354433a2004-05-19 08:20:33 +0000246%======================================================================
Andrew M. Kuchlingd91fcbe2004-08-02 12:44:28 +0000247\section{PEP 318: Decorators for Functions, Methods and Classes}
248
Andrew M. Kuchling77a602f2004-08-02 13:48:18 +0000249Python 2.2 extended Python's object model by adding static methods and
250class methods, but it didn't extend Python's syntax to provide any new
251way of defining static or class methods. Instead, you had to write a
252\keyword{def} statement in the usual way, and pass the resulting
253method to a \function{staticmethod()} or \function{classmethod()}
254function that would wrap up the function as a method of the new type.
255Your code would look like this:
256
257\begin{verbatim}
258class C:
259 def meth (cls):
260 ...
261
262 meth = classmethod(meth) # Rebind name to wrapped-up class method
263\end{verbatim}
264
265If the method was very long, it would be easy to miss or forget the
266\function{classmethod()} invocation after the function body.
267
268The intention was always to add some syntax to make such definitions
269more readable, but at the time of 2.2's release a good syntax was not
270obvious. Years later, when Python 2.4 is coming out, a good syntax
271\emph{still} isn't obvious but users are asking for easier access to
272the feature, so a new syntactic feature has been added.
273
274The feature is called ``function decorators''. The name comes from
275the idea that \function{classmethod}, \function{staticmethod}, and
276friends are storing additional information on a function object; they're
277\emph{decorating} functions with more details.
278
Fred Drake3f5c6542004-08-06 03:34:20 +0000279The notation borrows from Java and uses the \character{@} character as an
Andrew M. Kuchling77a602f2004-08-02 13:48:18 +0000280indicator. Using the new syntax, the example above would be written:
281
282\begin{verbatim}
283class C:
284
285 @classmethod
286 def meth (cls):
287 ...
288
289\end{verbatim}
290
291The \code{@classmethod} is shorthand for the
Fred Drake3f5c6542004-08-06 03:34:20 +0000292\code{meth=classmethod(meth)} assignment. More generally, if you have
Andrew M. Kuchling77a602f2004-08-02 13:48:18 +0000293the following:
294
295\begin{verbatim}
296@A @B @C
297def f ():
298 ...
299\end{verbatim}
300
301It's equivalent to:
302
303\begin{verbatim}
304def f(): ...
Andrew M. Kuchlingcebdd3c2004-10-08 18:29:29 +0000305f = A(B(C(f)))
Andrew M. Kuchling77a602f2004-08-02 13:48:18 +0000306\end{verbatim}
307
308Decorators must come on the line before a function definition, and
309can't be on the same line, meaning that \code{@A def f(): ...} is
310illegal. You can only decorate function definitions, either at the
311module-level or inside a class; you can't decorate class definitions.
312
313A decorator is just a function that takes the function to be decorated
314as an argument and returns either the same function or some new
315callable thing. It's easy to write your own decorators. The
316following simple example just sets an attribute on the function
317object:
318
319\begin{verbatim}
320>>> def deco(func):
321... func.attr = 'decorated'
322... return func
323...
324>>> @deco
325... def f(): pass
326...
327>>> f
328<function f at 0x402ef0d4>
329>>> f.attr
330'decorated'
331>>>
332\end{verbatim}
333
334As a slightly more realistic example, the following decorator checks
335that the supplied argument is an integer:
336
337\begin{verbatim}
338def require_int (func):
339 def wrapper (arg):
340 assert isinstance(arg, int)
341 return func(arg)
342
343 return wrapper
344
345@require_int
346def p1 (arg):
347 print arg
348
349@require_int
350def p2(arg):
351 print arg*2
352\end{verbatim}
353
354An example in \pep{318} contains a fancier version of this idea that
355lets you specify the required type and check the returned type as
356well.
357
358Decorator functions can take arguments. If arguments are supplied,
359the decorator function is called with only those arguments and must
360return a new decorator function; this new function must take a single
361function and return a function, as previously described. In other
362words, \code{@A @B @C(args)} becomes:
363
364\begin{verbatim}
365def f(): ...
366_deco = C(args)
Andrew M. Kuchlingcebdd3c2004-10-08 18:29:29 +0000367f = A(B(_deco(f)))
Andrew M. Kuchling77a602f2004-08-02 13:48:18 +0000368\end{verbatim}
369
370Getting this right can be slightly brain-bending, but it's not too
371difficult.
372
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +0000373A small related change makes the \member{func_name} attribute of
374functions writable. This attribute is used to display function names
375in tracebacks, so decorators should change the name of any new
376function that's constructed and returned.
377
Andrew M. Kuchling77a602f2004-08-02 13:48:18 +0000378The new syntax was provisionally added in 2.4alpha2, and is subject to
Andrew M. Kuchling9fa544c2004-09-23 20:17:26 +0000379change during the 2.4beta release cycle depending on the Python
Andrew M. Kuchling77a602f2004-08-02 13:48:18 +0000380community's reaction. Post-2.4 versions of Python will preserve
381compatibility with whatever syntax is used in 2.4final.
Andrew M. Kuchlingd91fcbe2004-08-02 12:44:28 +0000382
383\begin{seealso}
384\seepep{318}{Decorators for Functions, Methods and Classes}{Written
Andrew M. Kuchling77a602f2004-08-02 13:48:18 +0000385by Kevin D. Smith, Jim Jewett, and Skip Montanaro. Several people
386wrote patches implementing function decorators, but the one that was
Fred Drakee72bd4d2004-08-02 21:50:26 +0000387actually checked in was patch \#979728, written by Mark Russell.}
Andrew M. Kuchlingd91fcbe2004-08-02 12:44:28 +0000388\end{seealso}
389
390%======================================================================
Andrew M. Kuchling1a420252003-11-08 15:58:49 +0000391\section{PEP 322: Reverse Iteration}
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000392
Fred Drake56fcc232004-05-06 02:55:35 +0000393A new built-in function, \function{reversed(\var{seq})}, takes a sequence
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000394and returns an iterator that loops over the elements of the sequence
Andrew M. Kuchling1a420252003-11-08 15:58:49 +0000395in reverse order.
396
397\begin{verbatim}
Raymond Hettingerbc3cba22003-11-12 16:39:30 +0000398>>> for i in reversed(xrange(1,4)):
Andrew M. Kuchling1a420252003-11-08 15:58:49 +0000399... print i
400...
4013
4022
4031
404\end{verbatim}
405
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000406Compared to extended slicing, such as \code{range(1,4)[::-1]},
407\function{reversed()} is easier to read, runs faster, and uses
408substantially less memory.
Raymond Hettingerbc3cba22003-11-12 16:39:30 +0000409
Andrew M. Kuchling1a420252003-11-08 15:58:49 +0000410Note that \function{reversed()} only accepts sequences, not arbitrary
Raymond Hettingerbc3cba22003-11-12 16:39:30 +0000411iterators. If you want to reverse an iterator, first convert it to
412a list with \function{list()}.
Andrew M. Kuchling1a420252003-11-08 15:58:49 +0000413
414\begin{verbatim}
Andrew M. Kuchling44a31e12004-01-01 18:33:34 +0000415>>> input= open('/etc/passwd', 'r')
416>>> for line in reversed(list(input)):
Andrew M. Kuchling1a420252003-11-08 15:58:49 +0000417... print line
418...
419root:*:0:0:System Administrator:/var/root:/bin/tcsh
420 ...
421\end{verbatim}
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000422
Andrew M. Kuchlingf7a6b672003-11-08 16:05:37 +0000423\begin{seealso}
424\seepep{322}{Reverse Iteration}{Written and implemented by Raymond Hettinger.}
425
426\end{seealso}
427
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000428
429%======================================================================
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000430\section{PEP 327: Decimal Data Type}
431
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000432Python has always supported floating-point (FP) numbers as a data
433type, based on the underlying C \ctype{double} type. However, while
434most programming languages provide a floating-point type, most people
435(even programmers) are unaware that computing with floating-point
436numbers entails certain unavoidable inaccuracies. The new decimal
437type provides a way to avoid these inaccuracies.
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000438
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000439\subsection{Why is Decimal needed?}
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000440
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000441The limitations arise from the representation used for floating-point numbers.
442FP numbers are made up of three components:
443
444\begin{itemize}
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000445\item The sign, which is positive or negative.
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000446\item The mantissa, which is a single-digit binary number
447followed by a fractional part. For example, \code{1.01} in base-2 notation
448is \code{1 + 0/2 + 1/4}, or 1.25 in decimal notation.
449\item The exponent, which tells where the decimal point is located in the number represented.
450\end{itemize}
451
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000452For example, the number 1.25 has positive sign, a mantissa value of
4531.01 (in binary), and an exponent of 0 (the decimal point doesn't need
454to be shifted). The number 5 has the same sign and mantissa, but the
455exponent is 2 because the mantissa is multiplied by 4 (2 to the power
456of the exponent 2).
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000457
458Modern systems usually provide floating-point support that conforms to
459a relevant standard called IEEE 754. C's \ctype{double} type is
460usually implemented as a 64-bit IEEE 754 number, which uses 52 bits of
461space for the mantissa. This means that numbers can only be specified
462to 52 bits of precision. If you're trying to represent numbers whose
463expansion repeats endlessly, the expansion is cut off after 52 bits.
464Unfortunately, most software needs to produce output in base 10, and
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000465base 10 often gives rise to such repeating decimals in the binary
466expansion. For example, 1.1 decimal is binary \code{1.0001100110011
467...}; .1 = 1/16 + 1/32 + 1/256 plus an infinite number of additional
468terms. IEEE 754 has to chop off that infinitely repeated decimal
469after 52 digits, so the representation is slightly inaccurate.
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000470
471Sometimes you can see this inaccuracy when the number is printed:
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000472\begin{verbatim}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000473>>> 1.1
4741.1000000000000001
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000475\end{verbatim}
476
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000477The inaccuracy isn't always visible when you print the number because
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000478the FP-to-decimal-string conversion is provided by the C library, and
479most C libraries try to produce sensible output. Even if it's not
480displayed, however, the inaccuracy is still there and subsequent
481operations can magnify the error.
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000482
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000483For many applications this doesn't matter. If I'm plotting points and
484displaying them on my monitor, the difference between 1.1 and
4851.1000000000000001 is too small to be visible. Reports often limit
486output to a certain number of decimal places, and if you round the
487number to two or three or even eight decimal places, the error is
488never apparent. However, for applications where it does matter,
489it's a lot of work to implement your own custom arithmetic routines.
490
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000491Hence, the \class{Decimal} type was created.
492
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000493\subsection{The \class{Decimal} type}
494
495A new module, \module{decimal}, was added to Python's standard library.
496It contains two classes, \class{Decimal} and \class{Context}.
497\class{Decimal} instances represent numbers, and
498\class{Context} instances are used to wrap up various settings such as the precision and default rounding mode.
499
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000500\class{Decimal} instances, like regular Python integers and FP
501numbers, are immutable; once they've been created, you can't change
502the value it represents. \class{Decimal} instances can be created
503from integers or strings:
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000504
505\begin{verbatim}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000506>>> import decimal
507>>> decimal.Decimal(1972)
508Decimal("1972")
509>>> decimal.Decimal("1.1")
510Decimal("1.1")
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000511\end{verbatim}
512
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000513You can also provide tuples containing the sign, the mantissa represented
514as a tuple of decimal digits, and the exponent:
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000515
516\begin{verbatim}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000517>>> decimal.Decimal((1, (1, 4, 7, 5), -2))
518Decimal("-14.75")
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000519\end{verbatim}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000520
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000521Cautionary note: the sign bit is a Boolean value, so 0 is positive and
5221 is negative.
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000523
Andrew M. Kuchlinge34c3bd2004-08-31 12:21:44 +0000524Converting from floating-point numbers poses a bit of a problem:
525should the FP number representing 1.1 turn into the decimal number for
526exactly 1.1, or for 1.1 plus whatever inaccuracies are introduced?
527The decision was to leave such a conversion out of the API. Instead,
528you should convert the floating-point number into a string using the
529desired precision and pass the string to the \class{Decimal}
530constructor:
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000531
532\begin{verbatim}
533>>> f = 1.1
534>>> decimal.Decimal(str(f))
535Decimal("1.1")
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000536>>> decimal.Decimal('%.12f' % f)
537Decimal("1.100000000000")
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000538\end{verbatim}
539
540Once you have \class{Decimal} instances, you can perform the usual
541mathematical operations on them. One limitation: exponentiation
542requires an integer exponent:
543
544\begin{verbatim}
545>>> a = decimal.Decimal('35.72')
546>>> b = decimal.Decimal('1.73')
547>>> a+b
548Decimal("37.45")
549>>> a-b
550Decimal("33.99")
551>>> a*b
552Decimal("61.7956")
553>>> a/b
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000554Decimal("20.64739884393063583815028902")
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000555>>> a ** 2
556Decimal("1275.9184")
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000557>>> a**b
558Traceback (most recent call last):
559 ...
560decimal.InvalidOperation: x ** (non-integer)
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000561\end{verbatim}
562
563You can combine \class{Decimal} instances with integers, but not with
564floating-point numbers:
565
566\begin{verbatim}
567>>> a + 4
568Decimal("39.72")
569>>> a + 4.5
570Traceback (most recent call last):
571 ...
572TypeError: You can interact Decimal only with int, long or Decimal data types.
573>>>
574\end{verbatim}
575
576\class{Decimal} numbers can be used with the \module{math} and
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000577\module{cmath} modules, but note that they'll be immediately converted to
578floating-point numbers before the operation is performed, resulting in
579a possible loss of precision and accuracy. You'll also get back a
580regular floating-point number and not a \class{Decimal}.
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000581
582\begin{verbatim}
583>>> import math, cmath
584>>> d = decimal.Decimal('123456789012.345')
585>>> math.sqrt(d)
586351364.18288201344
587>>> cmath.sqrt(-d)
588351364.18288201344j
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000589\end{verbatim}
590
591Instances also have a \method{sqrt()} method that returns a
592\class{Decimal}, but if you need other things such as trigonometric
593functions you'll have to implement them.
594
595\begin{verbatim}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000596>>> d.sqrt()
Raymond Hettingerca1a7752004-07-12 13:00:45 +0000597Decimal("351364.1828820134592177245001")
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000598\end{verbatim}
599
600
601\subsection{The \class{Context} type}
602
603Instances of the \class{Context} class encapsulate several settings for
604decimal operations:
605
606\begin{itemize}
607 \item \member{prec} is the precision, the number of decimal places.
608 \item \member{rounding} specifies the rounding mode. The \module{decimal}
609 module has constants for the various possibilities:
610 \constant{ROUND_DOWN}, \constant{ROUND_CEILING}, \constant{ROUND_HALF_EVEN}, and various others.
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000611 \item \member{traps} is a dictionary specifying what happens on
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000612encountering certain error conditions: either an exception is raised or
613a value is returned. Some examples of error conditions are
614division by zero, loss of precision, and overflow.
615\end{itemize}
616
617There's a thread-local default context available by calling
618\function{getcontext()}; you can change the properties of this context
619to alter the default precision, rounding, or trap handling.
620
621\begin{verbatim}
622>>> decimal.getcontext().prec
62328
624>>> decimal.Decimal(1) / decimal.Decimal(7)
Raymond Hettingerca1a7752004-07-12 13:00:45 +0000625Decimal("0.1428571428571428571428571429")
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000626>>> decimal.getcontext().prec = 9
627>>> decimal.Decimal(1) / decimal.Decimal(7)
Raymond Hettingerca1a7752004-07-12 13:00:45 +0000628Decimal("0.142857143")
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000629\end{verbatim}
630
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000631The default action for error conditions is selectable; the module can
632either return a special value such as infinity or not-a-number, or
633exceptions can be raised:
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000634
635\begin{verbatim}
636>>> decimal.Decimal(1) / decimal.Decimal(0)
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000637Traceback (most recent call last):
638 ...
639decimal.DivisionByZero: x / 0
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000640>>> decimal.getcontext().traps[decimal.DivisionByZero] = False
641>>> decimal.Decimal(1) / decimal.Decimal(0)
642Decimal("Infinity")
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000643>>>
644\end{verbatim}
645
646The \class{Context} instance also has various methods for formatting
647numbers such as \method{to_eng_string()} and \method{to_sci_string()}.
648
Andrew M. Kuchling0ad20f12004-07-21 13:00:06 +0000649For more information, see the documentation for the \module{decimal}
650module, which includes a quick-start tutorial and a reference.
651
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000652\begin{seealso}
653\seepep{327}{Decimal Data Type}{Written by Facundo Batista and implemented
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000654 by Facundo Batista, Eric Price, Raymond Hettinger, Aahz, and Tim Peters.}
655
Raymond Hettingerca1a7752004-07-12 13:00:45 +0000656\seeurl{http://research.microsoft.com/\textasciitilde hollasch/cgindex/coding/ieeefloat.html}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000657{A more detailed overview of the IEEE-754 representation.}
658
659\seeurl{http://www.lahey.com/float.htm}
660{The article uses Fortran code to illustrate many of the problems
661that floating-point inaccuracy can cause.}
662
663\seeurl{http://www2.hursley.ibm.com/decimal/}
664{A description of a decimal-based representation. This representation
665is being proposed as a standard, and underlies the new Python decimal
666type. Much of this material was written by Mike Cowlishaw, designer of the
Raymond Hettingerca1a7752004-07-12 13:00:45 +0000667Rexx language.}
Andrew M. Kuchlingc8f8a812004-07-04 01:26:42 +0000668
Raymond Hettinger0fff62f2004-07-01 11:52:15 +0000669\end{seealso}
670
671
672%======================================================================
Andrew M. Kuchling3294e9d2004-08-31 11:26:23 +0000673\section{PEP 328: Multi-line Imports}
674
675One language change is a small syntactic tweak aimed at making it
676easier to import many names from a module. In a
677\code{from \var{module} import \var{names}} statement,
678\var{names} is a sequence of names separated by commas. If the sequence is
679very long, you can either write multiple imports from the same module,
680or you can use backslashes to escape the line endings:
681
682\begin{verbatim}
683from SimpleXMLRPCServer import SimpleXMLRPCServer,\
684 SimpleXMLRPCRequestHandler,\
685 CGIXMLRPCRequestHandler,\
686 resolve_dotted_attribute
687\end{verbatim}
688
689The syntactic change simply allows putting the names within
690parentheses. Python ignores newlines within a parenthesized
691expression, so the backslashes are no longer needed:
692
693\begin{verbatim}
694from SimpleXMLRPCServer import (SimpleXMLRPCServer,
695 SimpleXMLRPCRequestHandler,
696 CGIXMLRPCRequestHandler,
697 resolve_dotted_attribute)
698\end{verbatim}
699
700The PEP also proposes that all \keyword{import} statements be
701absolute imports, with a leading \samp{.} character to indicate a
702relative import. This part of the PEP is not yet implemented.
703
704\begin{seealso}
Fred Drake410eb842004-09-01 04:05:08 +0000705\seepep{328}{Imports: Multi-Line and Absolute/Relative}
706 {Written by Aahz. Multi-line imports were implemented by
707 Dima Dorfman.}
708\end{seealso}
Andrew M. Kuchling3294e9d2004-08-31 11:26:23 +0000709
710
711%======================================================================
Andrew M. Kuchling65a33322004-07-21 12:41:38 +0000712\section{PEP 331: Locale-Independent Float/String Conversions}
713
714The \module{locale} modules lets Python software select various
715conversions and display conventions that are localized to a particular
716country or language. However, the module was careful to not change
717the numeric locale because various functions in Python's
718implementation required that the numeric locale remain set to the
719\code{'C'} locale. Often this was because the code was using the C library's
720\cfunction{atof()} function.
721
722Not setting the numeric locale caused trouble for extensions that used
723third-party C libraries, however, because they wouldn't have the
724correct locale set. The motivating example was GTK+, whose user
725interface widgets weren't displaying numbers in the current locale.
726
727The solution described in the PEP is to add three new functions to the
728Python API that perform ASCII-only conversions, ignoring the locale
729setting:
730
731\begin{itemize}
732 \item \cfunction{PyOS_ascii_strtod(\var{str}, \var{ptr})}
733and \cfunction{PyOS_ascii_atof(\var{str}, \var{ptr})}
734both convert a string to a C \ctype{double}.
735 \item \cfunction{PyOS_ascii_formatd(\var{buffer}, \var{buf_len}, \var{format}, \var{d})} converts a \ctype{double} to an ASCII string.
736\end{itemize}
737
738The code for these functions came from the GLib library
739(\url{http://developer.gnome.org/arch/gtk/glib.html}), whose
740developers kindly relicensed the relevant functions and donated them
741to the Python Software Foundation. The \module{locale} module
742can now change the numeric locale, letting extensions such as GTK+
743produce the correct results.
744
745\begin{seealso}
746\seepep{331}{Locale-Independent Float/String Conversions}{Written by Christian R. Reis, and implemented by Gustavo Carneiro.}
747\end{seealso}
748
749%======================================================================
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000750\section{Other Language Changes}
751
752Here are all of the changes that Python 2.4 makes to the core Python
753language.
754
755\begin{itemize}
Raymond Hettingerd4462302003-11-26 17:52:45 +0000756
Raymond Hettinger31017ae2004-03-04 08:25:44 +0000757\item The \method{dict.update()} method now accepts the same
758argument forms as the \class{dict} constructor. This includes any
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000759mapping, any iterable of key/value pairs, and keyword arguments.
760(Contributed by Raymond Hettinger.)
Raymond Hettinger31017ae2004-03-04 08:25:44 +0000761
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000762\item The string methods \method{ljust()}, \method{rjust()}, and
Andrew M. Kuchling67087562003-11-26 18:03:48 +0000763\method{center()} now take an optional argument for specifying a
Raymond Hettingerd4462302003-11-26 17:52:45 +0000764fill character other than a space.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000765(Contributed by Raymond Hettinger.)
Raymond Hettingerd4462302003-11-26 17:52:45 +0000766
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +0000767\item Strings also gained an \method{rsplit()} method that
Raymond Hettingered54d912003-12-31 01:59:18 +0000768works like the \method{split()} method but splits from the end of
Andrew M. Kuchling44a31e12004-01-01 18:33:34 +0000769the string.
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +0000770
771\begin{verbatim}
Raymond Hettinger7a6d2972004-02-13 19:00:07 +0000772>>> 'www.python.org'.split('.', 1)
773['www', 'python.org']
774'www.python.org'.rsplit('.', 1)
775['www.python', 'org']
776\end{verbatim}
Raymond Hettinger97ef8de2004-01-05 00:29:57 +0000777
Andrew M. Kuchling2fb4d512003-10-21 12:31:16 +0000778\item The \method{sort()} method of lists gained three keyword
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000779arguments: \var{cmp}, \var{key}, and \var{reverse}. These arguments
Andrew M. Kuchling2fb4d512003-10-21 12:31:16 +0000780make some common usages of \method{sort()} simpler. All are optional.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000781(Contributed by Raymond Hettinger.)
Andrew M. Kuchling2fb4d512003-10-21 12:31:16 +0000782
783\var{cmp} is the same as the previous single argument to
784\method{sort()}; if provided, the value should be a comparison
785function that takes two arguments and returns -1, 0, or +1 depending
786on how the arguments compare.
787
788\var{key} should be a single-argument function that takes a list
789element and returns a comparison key for the element. The list is
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000790then sorted using the comparison keys. The following example sorts a
791list case-insensitively:
Andrew M. Kuchling2fb4d512003-10-21 12:31:16 +0000792
793\begin{verbatim}
794>>> L = ['A', 'b', 'c', 'D']
795>>> L.sort() # Case-sensitive sort
796>>> L
797['A', 'D', 'b', 'c']
798>>> L.sort(key=lambda x: x.lower())
799>>> L
800['A', 'b', 'c', 'D']
801>>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
802>>> L
803['A', 'b', 'c', 'D']
804\end{verbatim}
805
806The last example, which uses the \var{cmp} parameter, is the old way
Raymond Hettingered54d912003-12-31 01:59:18 +0000807to perform a case-insensitive sort. It works but is slower than
Andrew M. Kuchling2fb4d512003-10-21 12:31:16 +0000808using a \var{key} parameter. Using \var{key} results in calling the
809\method{lower()} method once for each element in the list while using
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000810\var{cmp} will call it twice for each comparison.
Andrew M. Kuchling2fb4d512003-10-21 12:31:16 +0000811
Andrew M. Kuchling981a9182003-11-13 21:33:26 +0000812For simple key functions and comparison functions, it is often
813possible to avoid a \keyword{lambda} expression by using an unbound
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000814method instead. For example, the above case-insensitive sort is best
815coded as:
816
817\begin{verbatim}
818>>> L.sort(key=str.lower)
819>>> L
820['A', 'b', 'c', 'D']
821\end{verbatim}
822
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000823The \var{reverse} parameter should have a Boolean value. If the value
824is \constant{True}, the list will be sorted into reverse order.
825Instead of \code{L.sort(lambda x,y: cmp(x.score, y.score)) ;
826L.reverse()}, you can now write: \code{L.sort(key = lambda x: x.score,
827reverse=True)}.
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000828
Andrew M. Kuchling981a9182003-11-13 21:33:26 +0000829The results of sorting are now guaranteed to be stable. This means
830that two entries with equal keys will be returned in the same order as
831they were input. For example, you can sort a list of people by name,
832and then sort the list by age, resulting in a list sorted by age where
833people with the same age are in name-sorted order.
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000834
Fred Drake56fcc232004-05-06 02:55:35 +0000835\item There is a new built-in function
836\function{sorted(\var{iterable})} that works like the in-place
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000837\method{list.sort()} method but can be used in
Fred Drake56fcc232004-05-06 02:55:35 +0000838expressions. The differences are:
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000839 \begin{itemize}
Raymond Hettinger7d1dd042003-11-12 16:42:10 +0000840 \item the input may be any iterable;
841 \item a newly formed copy is sorted, leaving the original intact; and
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000842 \item the expression returns the new sorted copy
843 \end{itemize}
Andrew M. Kuchling1a420252003-11-08 15:58:49 +0000844
845\begin{verbatim}
846>>> L = [9,7,8,3,2,4,1,6,5]
Raymond Hettinger64958a12003-12-17 20:43:33 +0000847>>> [10+i for i in sorted(L)] # usable in a list comprehension
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000848[11, 12, 13, 14, 15, 16, 17, 18, 19]
Hye-Shik Chang2b052482004-07-17 13:53:48 +0000849>>> L # original is left unchanged
Andrew M. Kuchlinge3e1eca2004-07-26 18:52:48 +0000850[9,7,8,3,2,4,1,6,5]
851>>> sorted('Monty Python') # any iterable may be an input
852[' ', 'M', 'P', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y', 'y']
Raymond Hettingerd4462302003-11-26 17:52:45 +0000853
854>>> # List the contents of a dict sorted by key values
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000855>>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)
Raymond Hettinger64958a12003-12-17 20:43:33 +0000856>>> for k, v in sorted(colormap.iteritems()):
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000857... print k, v
858...
859black 4
860blue 2
861green 3
862red 1
863yellow 5
Andrew M. Kuchling1a420252003-11-08 15:58:49 +0000864\end{verbatim}
865
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000866(Contributed by Raymond Hettinger.)
867
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +0000868\item Integer operations will no longer trigger an \exception{OverflowWarning}.
869The \exception{OverflowWarning} warning will disappear in Python 2.5.
870
Andrew M. Kuchling5e3f9232004-10-07 12:00:33 +0000871\item The interpreter gained a new switch, \programopt{-m}, that
872takes a name, searches for the corresponding module on \code{sys.path},
873and runs the module as a script. For example,
874you can now run the Python profiler with \code{python -m profile}.
875(Contributed by Nick Coghlan.)
876
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +0000877\item The \function{eval(\var{expr}, \var{globals}, \var{locals})}
Andrew M. Kuchling1455f792004-08-02 12:09:58 +0000878and \function{execfile(\var{filename}, \var{globals}, \var{locals})}
879functions and the \keyword{exec} statement now accept any mapping type
880for the \var{locals} argument. Previously this had to be a regular
881Python dictionary. (Contributed by Raymond Hettinger.)
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +0000882
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000883\item The \function{zip()} built-in function and \function{itertools.izip()}
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000884 now return an empty list if called with no arguments.
885 Previously they raised a \exception{TypeError}
886 exception. This makes them more
Raymond Hettinger607c00f2003-11-12 16:27:50 +0000887 suitable for use with variable length argument lists:
888
889\begin{verbatim}
890>>> def transpose(array):
891... return zip(*array)
892...
893>>> transpose([(1,2,3), (4,5,6)])
894[(1, 4), (2, 5), (3, 6)]
895>>> transpose([])
896[]
897\end{verbatim}
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000898(Contributed by Raymond Hettinger.)
899
Andrew M. Kuchlingd91fcbe2004-08-02 12:44:28 +0000900\item Encountering a failure while importing a module no longer leaves
901a partially-initialized module object in \code{sys.modules}. The
902incomplete module object left behind would fool further imports of the
903same module into succeeding, leading to confusing errors.
904
Andrew M. Kuchling65a33322004-07-21 12:41:38 +0000905\item \constant{None} is now a constant; code that binds a new value to
906the name \samp{None} is now a syntax error.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000907(Contributed by Raymond Hettinger.)
Andrew M. Kuchling65a33322004-07-21 12:41:38 +0000908
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000909\end{itemize}
910
911
912%======================================================================
913\subsection{Optimizations}
914
915\begin{itemize}
916
Raymond Hettingerca1a7752004-07-12 13:00:45 +0000917\item The inner loops for list and tuple slicing
Andrew M. Kuchling65a33322004-07-21 12:41:38 +0000918 were optimized and now run about one-third faster. The inner loops
919 were also optimized for dictionaries, resulting in performance boosts for
920 \method{keys()}, \method{values()}, \method{items()},
921 \method{iterkeys()}, \method{itervalues()}, and \method{iteritems()}.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000922 (Contributed by Raymond Hettinger.)
Raymond Hettingerb7d05db2004-03-08 07:25:05 +0000923
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +0000924\item The machinery for growing and shrinking lists was optimized for
925 speed and for space efficiency. Appending and popping from lists now
926 runs faster due to more efficient code paths and less frequent use of
927 the underlying system \cfunction{realloc()}. List comprehensions
928 also benefit. \method{list.extend()} was also optimized and no
929 longer converts its argument into a temporary list before extending
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000930 the base list. (Contributed by Raymond Hettinger.)
Raymond Hettinger7a6d2972004-02-13 19:00:07 +0000931
Raymond Hettinger97ef8de2004-01-05 00:29:57 +0000932\item \function{list()}, \function{tuple()}, \function{map()},
933 \function{filter()}, and \function{zip()} now run several times
934 faster with non-sequence arguments that supply a \method{__len__()}
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000935 method. (Contributed by Raymond Hettinger.)
Raymond Hettinger97ef8de2004-01-05 00:29:57 +0000936
Raymond Hettinger23a0f4e2004-01-05 08:15:20 +0000937\item The methods \method{list.__getitem__()},
Raymond Hettinger97ef8de2004-01-05 00:29:57 +0000938 \method{dict.__getitem__()}, and \method{dict.__contains__()} are
939 are now implemented as \class{method_descriptor} objects rather
940 than \class{wrapper_descriptor} objects. This form of optimized
941 access doubles their performance and makes them more suitable for
Raymond Hettinger23a0f4e2004-01-05 08:15:20 +0000942 use as arguments to functionals:
943 \samp{map(mydict.__getitem__, keylist)}.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000944 (Contributed by Raymond Hettinger.)
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000945
Fred Draked6d35d92004-06-03 13:31:22 +0000946\item Added a new opcode, \code{LIST_APPEND}, that simplifies
Raymond Hettingerdd80f762004-03-07 07:31:06 +0000947 the generated bytecode for list comprehensions and speeds them up
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000948 by about a third. (Contributed by Raymond Hettinger.)
Raymond Hettingerdd80f762004-03-07 07:31:06 +0000949
Andrew M. Kuchling0c789562004-09-23 20:15:41 +0000950\item The peephole bytecode optimizer has been improved to
951produce shorter, faster bytecode; remarkably the resulting bytecode is
952more readable. (Enhanced by Raymond Hettinger.)
953
Andrew M. Kuchlingac642872004-08-07 13:13:31 +0000954\item String concatenations in statements of the form \code{s = s +
955"abc"} and \code{s += "abc"} are now performed more efficiently in
956certain circumstances. This optimization won't be present in other
957Python implementations such as Jython, so you shouldn't rely on it;
958using the \method{join()} method of strings is still recommended when
959you want to efficiently glue a large number of strings together.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +0000960(Contributed by Armin Rigo.)
Andrew M. Kuchlingac642872004-08-07 13:13:31 +0000961
Fred Drakeed0fa3d2003-07-30 19:14:09 +0000962\end{itemize}
963
964The net result of the 2.4 optimizations is that Python 2.4 runs the
965pystone benchmark around XX\% faster than Python 2.3 and YY\% faster
966than Python 2.2.
967
968
969%======================================================================
970\section{New, Improved, and Deprecated Modules}
971
972As usual, Python's standard library received a number of enhancements and
973bug fixes. Here's a partial list of the most notable changes, sorted
974alphabetically by module name. Consult the
975\file{Misc/NEWS} file in the source tree for a more
976complete list of changes, or look through the CVS logs for all the
977details.
978
979\begin{itemize}
980
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +0000981\item The \module{asyncore} module's \function{loop()} now has a
982 \var{count} parameter that lets you perform a limited number
983 of passes through the polling loop. The default is still to loop
984 forever.
985
Andrew M. Kuchlinga331e862004-09-10 13:05:22 +0000986\item The \module{base64} module now has more complete RFC 3548 support
987 for Base64, Base32, and Base16 encoding and decoding, including
988 optional case folding and optional alternative alphabets.
989 (Contributed by Barry Warsaw.)
Andrew M. Kuchling6aedcfc2003-10-21 12:48:23 +0000990
Raymond Hettinger0c410272004-01-05 10:13:35 +0000991\item The \module{bisect} module now has an underlying C implementation
992 for improved performance.
993 (Contributed by Dmitry Vasiliev.)
994
Andrew M. Kuchling5303a962004-01-18 15:55:51 +0000995\item The CJKCodecs collections of East Asian codecs, maintained
996by Hye-Shik Chang, was integrated into 2.4.
997The new encodings are:
998
999\begin{itemize}
Andrew M. Kuchling671c5062004-07-28 15:29:39 +00001000 \item Chinese (PRC): gb2312, gbk, gb18030, big5hkscs, hz
Andrew M. Kuchling5303a962004-01-18 15:55:51 +00001001 \item Chinese (ROC): big5, cp950
Andrew M. Kuchling671c5062004-07-28 15:29:39 +00001002 \item Japanese: cp932, euc-jis-2004, euc-jp,
Andrew M. Kuchling5303a962004-01-18 15:55:51 +00001003euc-jisx0213, iso-2022-jp, iso-2022-jp-1, iso-2022-jp-2,
Andrew M. Kuchling671c5062004-07-28 15:29:39 +00001004 iso-2022-jp-3, iso-2022-jp-ext, iso-2022-jp-2004,
1005 shift-jis, shift-jisx0213, shift-jis-2004
Andrew M. Kuchling5303a962004-01-18 15:55:51 +00001006 \item Korean: cp949, euc-kr, johab, iso-2022-kr
1007\end{itemize}
1008
Andrew M. Kuchling579b3e22004-10-05 20:23:34 +00001009\item The UTF-8 and UTF-16 codecs now cope better with receiving partial input.
1010Previously the \class{StreamReader} class would try to read more data,
1011which made it impossible to resume decoding from the stream. The
1012\method{read()} method will now return as much data as it can and future
1013calls will resume decoding where previous ones left off.
1014(Implemented by Walter D\"orwald.)
1015
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +00001016\item Some other new encodings were added: HP Roman8,
Andrew M. Kuchling579b3e22004-10-05 20:23:34 +00001017ISO_8859-11, ISO_8859-16, PCTP-154, and TIS-620.
Andrew M. Kuchlinge30c4d42004-08-07 13:58:02 +00001018
Andrew M. Kuchlingfd0e4942004-02-09 13:23:34 +00001019\item There is a new \module{collections} module for
1020 various specialized collection datatypes.
1021 Currently it contains just one type, \class{deque},
1022 a double-ended queue that supports efficiently adding and removing
1023 elements from either end.
Raymond Hettinger756b3f32004-01-29 06:37:52 +00001024
1025\begin{verbatim}
1026>>> from collections import deque
1027>>> d = deque('ghi') # make a new deque with three items
1028>>> d.append('j') # add a new entry to the right side
1029>>> d.appendleft('f') # add a new entry to the left side
1030>>> d # show the representation of the deque
1031deque(['f', 'g', 'h', 'i', 'j'])
1032>>> d.pop() # return and remove the rightmost item
1033'j'
1034>>> d.popleft() # return and remove the leftmost item
1035'f'
1036>>> list(d) # list the contents of the deque
1037['g', 'h', 'i']
1038>>> 'h' in d # search the deque
1039True
1040\end{verbatim}
1041
Andrew M. Kuchlingfd0e4942004-02-09 13:23:34 +00001042Several modules now take advantage of \class{collections.deque} for
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +00001043improved performance, such as the \module{Queue} and
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001044\module{threading} modules. (Contributed by Raymond Hettinger.)
Andrew M. Kuchling5303a962004-01-18 15:55:51 +00001045
Fred Drake9f15b5c2004-05-18 04:30:00 +00001046\item The \module{ConfigParser} classes have been enhanced slightly.
1047 The \method{read()} method now returns a list of the files that
1048 were successfully parsed, and the \method{set()} method raises
1049 \exception{TypeError} if passed a \var{value} argument that isn't a
1050 string.
1051
Andrew M. Kuchlinga331e862004-09-10 13:05:22 +00001052\item The \module{curses} module now supports the ncurses extension
1053 \function{use_default_colors()}. On platforms where the terminal
1054 supports transparency, this makes it possible to use a transparent
1055 background. (Contributed by J\"org Lehmann.)
1056
1057\item The \module{difflib} module now includes an \class{HtmlDiff} class
1058that creates an HTML table showing a side by side comparison
1059of two versions of a text. (Contributed by Dan Gass.)
1060
Andrew M. Kuchling579b3e22004-10-05 20:23:34 +00001061\item The \module{email} package was updated to version 3.0,
1062which dropped various deprecated APIs and removes support for Python
1063versions earlier than 2.3. The 3.0 version of the package uses a new
1064incremental parser for MIME message, available in the
1065\module{email.FeedParser} module. The new parser doesn't require
1066reading the entire message into memory, and doesn't throw exceptions
1067if a message is malformed; instead it records any problems as a
1068\member{defect} attribute of the message. (Developed by Anthony
1069Baxter, Barry Warsaw, Thomas Wouters, and others.)
Andrew M. Kuchlinga331e862004-09-10 13:05:22 +00001070
Raymond Hettinger607c00f2003-11-12 16:27:50 +00001071\item The \module{heapq} module has been converted to C. The resulting
Andrew M. Kuchlingfd0e4942004-02-09 13:23:34 +00001072 tenfold improvement in speed makes the module suitable for handling
Raymond Hettinger33ecffb2004-06-10 05:03:17 +00001073 high volumes of data. In addition, the module has two new functions
1074 \function{nlargest()} and \function{nsmallest()} that use heaps to
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +00001075 find the N largest or smallest values in a dataset without the
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001076 expense of a full sort. (Contributed by Raymond Hettinger.)
Andrew M. Kuchling1a420252003-11-08 15:58:49 +00001077
Andrew M. Kuchling0c789562004-09-23 20:15:41 +00001078\item The \module{httplib} module now contains constants for HTTP
1079status codes defined in various HTTP-related RFC documents. Constants
1080have names such as \constant{OK}, \constant{CREATED},
1081\constant{CONTINUE}, and \constant{MOVED_PERMANENTLY}; use pydoc to
1082get a full list. (Contributed by Andrew Eland.)
1083
Andrew M. Kuchlingce4bae62004-07-27 12:13:25 +00001084\item The \module{imaplib} module now supports IMAP's THREAD command
1085(contributed by Yves Dionne) and new \method{deleteacl()} and
1086\method{myrights()} methods (contributed by Arnaud Mazin).
Andrew M. Kuchlingdff9dbd2003-11-20 22:22:19 +00001087
Andrew M. Kuchlingad809552003-12-06 23:19:23 +00001088\item The \module{itertools} module gained a
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +00001089 \function{groupby(\var{iterable}\optional{, \var{func}})} function.
Andrew M. Kuchlingad809552003-12-06 23:19:23 +00001090 \var{iterable} returns a succession of elements, and the optional
1091 \var{func} is a function that takes an element and returns a key
1092 value; if omitted, the key is simply the element itself.
1093 \function{groupby()} then groups the elements into subsequences
1094 which have matching values of the key, and returns a series of 2-tuples
1095 containing the key value and an iterator over the subsequence.
1096
1097Here's an example. The \var{key} function simply returns whether a
1098number is even or odd, so the result of \function{groupby()} is to
1099return consecutive runs of odd or even numbers.
1100
1101\begin{verbatim}
1102>>> import itertools
1103>>> L = [2,4,6, 7,8,9,11, 12, 14]
1104>>> for key_val, it in itertools.groupby(L, lambda x: x % 2):
1105... print key_val, list(it)
1106...
11070 [2, 4, 6]
11081 [7]
11090 [8]
11101 [9, 11]
11110 [12, 14]
1112>>>
1113\end{verbatim}
1114
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +00001115\function{groupby()} is typically used with sorted input. The logic
1116for \function{groupby()} is similar to the \UNIX{} \code{uniq} filter
1117which makes it handy for eliminating, counting, or identifying
1118duplicate elements:
Raymond Hettingerfeb78c92003-12-12 13:13:47 +00001119
1120\begin{verbatim}
1121>>> word = 'abracadabra'
Raymond Hettingered54d912003-12-31 01:59:18 +00001122>>> letters = sorted(word) # Turn string into a sorted list of letters
Raymond Hettinger64958a12003-12-17 20:43:33 +00001123>>> letters
Andrew M. Kuchling4612bc52003-12-16 20:59:37 +00001124['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +00001125>>> for k, g in itertools.groupby(letters):
1126... print k, list(g)
1127...
1128a ['a', 'a', 'a', 'a', 'a']
1129b ['b', 'b']
1130c ['c']
1131d ['d']
1132r ['r', 'r']
1133>>> # List unique letters
1134>>> [k for k, g in groupby(letters)]
Raymond Hettingerfeb78c92003-12-12 13:13:47 +00001135['a', 'b', 'c', 'd', 'r']
Johannes Gijsbersd3452252004-09-11 16:50:06 +00001136>>> # Count letter occurrences
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +00001137>>> [(k, len(list(g))) for k, g in groupby(letters)]
Raymond Hettingerfeb78c92003-12-12 13:13:47 +00001138[('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]
Raymond Hettingerfeb78c92003-12-12 13:13:47 +00001139\end{verbatim}
1140
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001141(Contributed by Hye-Shik Chang.)
1142
Raymond Hettingered54d912003-12-31 01:59:18 +00001143\item \module{itertools} also gained a function named
1144\function{tee(\var{iterator}, \var{N})} that returns \var{N} independent
1145iterators that replicate \var{iterator}. If \var{N} is omitted, the
1146default is 2.
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +00001147
1148\begin{verbatim}
1149>>> L = [1,2,3]
1150>>> i1, i2 = itertools.tee(L)
1151>>> i1,i2
1152(<itertools.tee object at 0x402c2080>, <itertools.tee object at 0x402c2090>)
Raymond Hettingered54d912003-12-31 01:59:18 +00001153>>> list(i1) # Run the first iterator to exhaustion
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +00001154[1, 2, 3]
Raymond Hettingered54d912003-12-31 01:59:18 +00001155>>> list(i2) # Run the second iterator to exhaustion
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +00001156[1, 2, 3]
1157>\end{verbatim}
1158
1159Note that \function{tee()} has to keep copies of the values returned
Raymond Hettingered54d912003-12-31 01:59:18 +00001160by the iterator; in the worst case, it may need to keep all of them.
Andrew M. Kuchling44a31e12004-01-01 18:33:34 +00001161This should therefore be used carefully if the leading iterator
Raymond Hettingered54d912003-12-31 01:59:18 +00001162can run far ahead of the trailing iterator in a long stream of inputs.
Andrew M. Kuchling3bf85f12004-07-05 01:37:07 +00001163If the separation is large, then you might as well use
Raymond Hettingered54d912003-12-31 01:59:18 +00001164\function{list()} instead. When the iterators track closely with one
1165another, \function{tee()} is ideal. Possible applications include
1166bookmarking, windowing, or lookahead iterators.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001167(Contributed by Raymond Hettinger.)
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +00001168
Andrew M. Kuchling5785a132004-07-26 19:28:46 +00001169\item A number of functions were added to the \module{locale}
1170module, such as \function{bind_textdomain_codeset()} to specify a
1171particular encoding, and a family of \function{l*gettext()} functions
1172that return messages in the chosen encoding.
1173(Contributed by Gustavo Niemeyer.)
1174
Andrew M. Kuchling23406892004-07-15 11:44:42 +00001175\item The \module{logging} package's \function{basicConfig} function
1176gained some keyword arguments to simplify log configuration. The
1177default behavior is to log messages to standard error, but
1178various keyword arguments can be specified to log to a particular file,
1179change the logging format, or set the logging level. For example:
Andrew M. Kuchlingbcefe692004-07-07 13:01:53 +00001180
1181\begin{verbatim}
1182import logging
1183logging.basicConfig(filename = '/var/log/application.log',
1184 level=0, # Log all messages, including debugging,
1185 format='%(levelname):%(process):%(thread):%(message)')
1186\end{verbatim}
1187
Andrew M. Kuchling579b3e22004-10-05 20:23:34 +00001188Other additions to \module{logging} include a \method{log(\var{level},
1189\var{msg})} convenience method, and a
1190\class{TimedRotatingFileHandler} class that rotates its log files at
Andrew M. Kuchlingbcefe692004-07-07 13:01:53 +00001191a timed interval. The module already had \class{RotatingFileHandler},
1192which rotated logs once the file exceeded a certain size. Both
1193classes derive from a new \class{BaseRotatingHandler} class that can
1194be used to implement other rotating handlers.
1195
Andrew M. Kuchling579b3e22004-10-05 20:23:34 +00001196(Changes implemented by Vinay Sajip.)
1197
Andrew M. Kuchling0c789562004-09-23 20:15:41 +00001198\item The \module{marshal} module now shares interned strings on unpacking a
1199data structure. This may shrink the size of certain pickle strings,
1200but the primary effect is to make \file{.pyc} files significantly smaller.
1201(Contributed by Martin von Loewis.)
1202
Andrew M. Kuchling5785a132004-07-26 19:28:46 +00001203\item The \module{nntplib} module's \class{NNTP} class gained
1204\method{description()} and \method{descriptions()} methods to retrieve
1205newsgroup descriptions for a single group or for a range of groups.
1206(Contributed by J\"urgen A. Erhard.)
1207
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +00001208\item The \module{operator} module gained two new functions,
1209\function{attrgetter(\var{attr})} and \function{itemgetter(\var{index})}.
1210Both functions return callables that take a single argument and return
Raymond Hettingered54d912003-12-31 01:59:18 +00001211the corresponding attribute or item; these callables make excellent
Andrew M. Kuchlingbcefe692004-07-07 13:01:53 +00001212data extractors when used with \function{map()} or
1213\function{sorted()}. For example:
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +00001214
1215\begin{verbatim}
Raymond Hettingered54d912003-12-31 01:59:18 +00001216>>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)]
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +00001217>>> map(operator.itemgetter(0), L)
1218['c', 'd', 'a', 'b']
1219>>> map(operator.itemgetter(1), L)
Raymond Hettingered54d912003-12-31 01:59:18 +00001220[2, 1, 4, 3]
1221>>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item
1222[('d', 1), ('c', 2), ('b', 3), ('a', 4)]
Andrew M. Kuchling35f2b052003-12-18 13:28:13 +00001223\end{verbatim}
1224
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001225(Contributed by Raymond Hettinger.)
1226
Andrew M. Kuchlinge30c4d42004-08-07 13:58:02 +00001227\item The \module{optparse} module was updated. The module now passes
1228its messages through \function{gettext.gettext()}, making it possible
1229to internationalize Optik's help and error messages. Help messages
Fred Drake9bae19e2004-08-07 14:28:37 +00001230for options can now include the string \code{'\%default'}, which will
Andrew M. Kuchlinge30c4d42004-08-07 13:58:02 +00001231be replaced by the option's default value.
1232
Andrew M. Kuchlingcb7b3f32004-08-30 11:58:04 +00001233\item A new \function{urandom(\var{n})} function
1234was added to the \module{os} module, providing access to
1235platform-specific sources of randomness such as
Johannes Gijsbersed047482004-08-30 15:03:23 +00001236\file{/dev/urandom} on Linux or the Windows CryptoAPI. The
Andrew M. Kuchlingcb7b3f32004-08-30 11:58:04 +00001237function returns a string containing \var{n} bytes of random data.
1238(Contributed by Trevor Perrin.)
1239
1240\item Another new function: \function{os.path.lexists(\var{path})}
1241returns true if the file specified by \var{path} exists, whether or
1242not it's a symbolic link. This differs from the existing
1243\function{os.path.exists(\var{path})} function, which returns false if
1244\var{path} is a symlink that points to a destination that doesn't exist.
1245(Contributed by Beni Cherniavsky.)
1246
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +00001247\item A new \function{getsid()} function was added to the
1248\module{posix} module that underlies the \module{os} module.
1249(Contributed by J. Raynor.)
1250
1251\item The \module{poplib} module now supports POP over SSL.
1252
1253\item The \module{profile} module can now profile C extension functions.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001254% XXX more to say about this?
1255(Contributed by Nick Bastin.)
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +00001256
Andrew M. Kuchling6aedcfc2003-10-21 12:48:23 +00001257\item The \module{random} module has a new method called \method{getrandbits(N)}
Raymond Hettinger607c00f2003-11-12 16:27:50 +00001258 which returns an N-bit long integer. This method supports the existing
1259 \method{randrange()} method, making it possible to efficiently generate
Andrew M. Kuchling44a31e12004-01-01 18:33:34 +00001260 arbitrarily large random numbers.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001261 (Contributed by Raymond Hettinger.)
Andrew M. Kuchling6aedcfc2003-10-21 12:48:23 +00001262
1263\item The regular expression language accepted by the \module{re} module
1264 was extended with simple conditional expressions, written as
Andrew M. Kuchlingab778222004-08-31 12:07:43 +00001265 \regexp{(?(\var{group})\var{A}|\var{B})}. \var{group} is either a
1266 numeric group ID or a group name defined with \regexp{(?P<group>...)}
Andrew M. Kuchling6aedcfc2003-10-21 12:48:23 +00001267 earlier in the expression. If the specified group matched, the
1268 regular expression pattern \var{A} will be tested against the string; if
1269 the group didn't match, the pattern \var{B} will be used instead.
Raymond Hettinger874ebd52004-05-31 03:15:02 +00001270
Andrew M. Kuchlingab778222004-08-31 12:07:43 +00001271\item The \module{re} module is also no longer recursive, thanks
1272to a massive amount of work by Gustavo Niemeyer. In a recursive
1273regular expression engine, certain patterns result in a large amount
1274of C stack space being consumed, and it was possible to overflow the
1275stack. For example, if you matched a 30000-byte string of \samp{a}
1276characters against the expression \regexp{(a|b)+}, one stack frame was
1277consumed per character. Python 2.3 tried to check for stack overflow
1278and raise a \exception{RuntimeError} exception, but if you were
1279unlucky Python could dump core. Python 2.4's regular expression
1280engine can match this pattern without problems.
1281
Andrew M. Kuchling7f203b82004-08-09 14:48:28 +00001282\item A new \function{socketpair()} function was added to the
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +00001283\module{socket} module, returning a pair of connected sockets.
1284(Contributed by Dave Cole.)
Andrew M. Kuchling7f203b82004-08-09 14:48:28 +00001285
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +00001286\item The \function{sys.exitfunc()} function has been deprecated. Code
1287should be using the existing \module{atexit} module, which correctly
1288handles calling multiple exit functions. Eventually
1289\function{sys.exitfunc()} will become a purely internal interface,
1290accessed only by \module{atexit}.
1291
1292\item The \module{tarfile} module now generates GNU-format tar files
1293by default.
1294
Andrew M. Kuchling00457172004-07-15 11:52:40 +00001295\item The \module{threading} module now has an elegantly simple way to support
1296thread-local data. The module contains a \class{local} class whose
1297attribute values are local to different threads.
1298
1299\begin{verbatim}
1300import threading
1301
1302data = threading.local()
1303data.number = 42
1304data.url = ('www.python.org', 80)
1305\end{verbatim}
1306
1307Other threads can assign and retrieve their own values for the
1308\member{number} and \member{url} attributes. You can subclass
1309\class{local} to initialize attributes or to add methods.
1310(Contributed by Jim Fulton.)
1311
Andrew M. Kuchlinga331e862004-09-10 13:05:22 +00001312\item The \module{timeit} module now automatically disables periodic
1313 garbarge collection during the timing loop. This change makes
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001314 consecutive timings more comparable. (Contributed by Raymond Hettinger.)
Andrew M. Kuchlinga331e862004-09-10 13:05:22 +00001315
Raymond Hettinger874ebd52004-05-31 03:15:02 +00001316\item The \module{weakref} module now supports a wider variety of objects
1317 including Python functions, class instances, sets, frozensets, deques,
1318 arrays, files, sockets, and regular expression pattern objects.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001319 (Contributed by Raymond Hettinger.)
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +00001320
1321\item The \module{xmlrpclib} module now supports a multi-call extension for
Andrew M. Kuchling00457172004-07-15 11:52:40 +00001322transmitting multiple XML-RPC calls in a single HTTP operation.
Andrew M. Kuchling3d3db962004-08-31 13:57:02 +00001323
1324\item The \module{mpz}, \module{rotor}, and \module{xreadlines} modules have
1325been removed.
Andrew M. Kuchling69f31eb2003-08-13 23:11:04 +00001326
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001327\end{itemize}
1328
1329
1330%======================================================================
Raymond Hettingerca1a7752004-07-12 13:00:45 +00001331% whole new modules get described in subsections here
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001332
Martin v. Löwis2a6ba902004-05-31 18:22:40 +00001333\subsection{cookielib}
1334
1335The \module{cookielib} library supports client-side handling for HTTP
1336cookies, just as the \module{Cookie} provides server-side cookie
Andrew M. Kuchling71432f12004-07-05 01:40:07 +00001337support in CGI scripts. Cookies are stored in cookie jars; the library
Martin v. Löwis2a6ba902004-05-31 18:22:40 +00001338transparently stores cookies offered by the web server in the cookie
1339jar, and fetches the cookie from the jar when connecting to the
1340server. Similar to web browsers, policy objects control whether
1341cookies are accepted or not.
1342
1343In order to store cookies across sessions, two implementations of
1344cookie jars are provided: one that stores cookies in the Netscape
1345format, so applications can use the Mozilla or Lynx cookie jars, and
1346one that stores cookies in the same format as the Perl libwww libary.
1347
1348\module{urllib2} has been changed to interact with \module{cookielib}:
1349\class{HTTPCookieProcessor} manages a cookie jar that is used when
1350accessing URLs.
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001351
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +00001352\subsection{doctest}
1353
1354The \module{doctest} module underwent considerable refactoring thanks
1355to Edward Loper and Tim Peters.
1356
1357% XXX describe this
1358
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001359% ======================================================================
1360\section{Build and C API Changes}
1361
1362Changes to Python's build process and to the C API include:
1363
1364\begin{itemize}
1365
Andrew M. Kuchling6aedcfc2003-10-21 12:48:23 +00001366 \item Three new convenience macros were added for common return
1367 values from extension functions: \csimplemacro{Py_RETURN_NONE},
1368 \csimplemacro{Py_RETURN_TRUE}, and \csimplemacro{Py_RETURN_FALSE}.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001369 (Contributed by Brett Cannon.)
Andrew M. Kuchling6aedcfc2003-10-21 12:48:23 +00001370
Andrew M. Kuchling5785a132004-07-26 19:28:46 +00001371 \item Another new macro, \csimplemacro{Py_CLEAR(\var{obj})},
1372 decreases the reference count of \var{obj} and sets \var{obj} to the
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001373 null pointer. (Contributed by Jim Fulton.)
Andrew M. Kuchling5785a132004-07-26 19:28:46 +00001374
Fred Drakece3caf22004-02-12 18:13:12 +00001375 \item A new function, \cfunction{PyTuple_Pack(\var{N}, \var{obj1},
1376 \var{obj2}, ..., \var{objN})}, constructs tuples from a variable
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001377 length argument list of Python objects. (Contributed by Raymond Hettinger.)
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001378
Fred Drakece3caf22004-02-12 18:13:12 +00001379 \item A new function, \cfunction{PyDict_Contains(\var{d}, \var{k})},
1380 implements fast dictionary lookups without masking exceptions raised
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001381 during the look-up process. (Contributed by Raymond Hettinger.)
Raymond Hettingerd4462302003-11-26 17:52:45 +00001382
Andrew M. Kuchling0c789562004-09-23 20:15:41 +00001383 \item The \csimplemacro{Py_IS_NAN(\var{X})} macro returns 1 if
1384 its float or double argument \var{X} is a NaN.
1385 (Contributed by Tim Peters.)
1386
Andrew M. Kuchlinge30c4d42004-08-07 13:58:02 +00001387 \item A new function, \cfunction{PyArg_VaParseTupleAndKeywords()},
1388 is the same as \cfunction{PyArg_ParseTupleAndKeywords()} but takes a
1389 \ctype{va_list} instead of a number of arguments.
1390 (Contributed by Greg Chapman.)
1391
Fred Drakece3caf22004-02-12 18:13:12 +00001392 \item A new method flag, \constant{METH_COEXISTS}, allows a function
Andrew M. Kuchling71432f12004-07-05 01:40:07 +00001393 defined in slots to co-exist with a \ctype{PyCFunction} having the
1394 same name. This can halve the access time for a method such as
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001395 \method{set.__contains__()}. (Contributed by Raymond Hettinger.)
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +00001396
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +00001397 \item Python can now be built with additional profiling for the
1398 interpreter itself. This is intended for people developing on the
1399 Python core. Providing \longprogramopt{--enable-profiling} to the
1400 \program{configure} script will let you profile the interpreter with
1401 \program{gprof}, and providing the \longprogramopt{--with-tsc}
1402 switch enables profiling using the Pentium's Time-Stamp-Counter
1403 register. The switch is slightly misnamed, because the profiling
1404 feature also works on the PowerPC platform, though that processor
Raymond Hettinger468af712004-09-20 17:47:46 +00001405 architecture doesn't call that register a TSC.
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +00001406
Andrew M. Kuchlingd0b6d9d2004-07-04 15:35:00 +00001407 \item The \ctype{tracebackobject} type has been renamed to \ctype{PyTracebackObject}.
Raymond Hettinger97ef8de2004-01-05 00:29:57 +00001408
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001409\end{itemize}
1410
1411
1412%======================================================================
1413\subsection{Port-Specific Changes}
1414
Raymond Hettinger97ef8de2004-01-05 00:29:57 +00001415\begin{itemize}
1416
1417\item The Windows port now builds under MSVC++ 7.1 as well as version 6.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001418 (Contributed by Martin von Loewis.)
Raymond Hettinger97ef8de2004-01-05 00:29:57 +00001419
1420\end{itemize}
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001421
1422
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001423
1424%======================================================================
1425\section{Porting to Python 2.4}
1426
1427This section lists previously described changes that may require
1428changes to your code:
1429
1430\begin{itemize}
1431
Raymond Hettinger607c00f2003-11-12 16:27:50 +00001432\item The \function{zip()} built-in function and \function{itertools.izip()}
1433 now return an empty list instead of raising a \exception{TypeError}
1434 exception if called with no arguments.
Andrew M. Kuchling7642f7a2004-09-13 15:06:50 +00001435 (Contributed by Raymond Hettinger.)
Andrew M. Kuchling6aedcfc2003-10-21 12:48:23 +00001436
1437\item \function{dircache.listdir()} now passes exceptions to the caller
1438 instead of returning empty lists.
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001439
Andrew M. Kuchling71432f12004-07-05 01:40:07 +00001440\item \function{LexicalHandler.startDTD()} used to receive the public and
1441 system IDs in the wrong order. This has been corrected; applications
Fred Drake56fcc232004-05-06 02:55:35 +00001442 relying on the wrong order need to be fixed.
Martin v. Löwis456ab1d2004-05-06 01:54:36 +00001443
Andrew M. Kuchling71432f12004-07-05 01:40:07 +00001444\item \function{fcntl.ioctl} now warns if the \var{mutate}
1445 argument is omitted and relevant.
Martin v. Löwis77ca6c42004-06-03 12:47:26 +00001446
Andrew M. Kuchling87c98b22004-08-25 13:38:46 +00001447\item The \module{tarfile} module now generates GNU-format tar files
1448by default.
1449
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001450\end{itemize}
1451
1452
1453%======================================================================
1454\section{Acknowledgements \label{acks}}
1455
1456The author would like to thank the following people for offering
1457suggestions, corrections and assistance with various drafts of this
Andrew M. Kuchlingcebdd3c2004-10-08 18:29:29 +00001458article: Hye-Shik Chang, Michael Dyck, Raymond Hettinger, Hamish Lawson.
Fred Drakeed0fa3d2003-07-30 19:14:09 +00001459
1460\end{document}