Doc/whatsnew/whatsnew24.tex

\documentclass{howto}
\usepackage{distutils}
% $Id$

% Don't write extensive text for new sections; I'll do that.  
% Feel free to add commented-out reminders of things that need
% to be covered.  --amk

% XXX pydoc can display links to module docs -- but when?
% 

\title{What's New in Python 2.4}
\release{0.1}
\author{A.M.\ Kuchling}
\authoraddress{
	\strong{Python Software Foundation}\\
	Email: \email{amk@amk.ca}
}

\begin{document}
\maketitle
\tableofcontents

This article explains the new features in Python 2.4 alpha1, to be released in early July 2004  The final version of Python 2.4 
is expected to be around September 2004.

Python 2.4 is a middle-sized release.  It doesn't introduce as many
changes as the radical Python 2.2, but introduces more features than
the conservative 2.3 release did.  The most significant new language
feature (as of this writing) is the addition of generator expressions;
most of the changes are to the standard library.

This article doesn't attempt to provide a complete specification of
every single new feature, but instead provides a convenient overview.
For full details, you should refer to the documentation for Python
2.4, such as the \citetitle[../lib/lib.html]{Python Library Reference}
and the \citetitle[../ref/ref.html]{Python Reference Manual}.  If you
want to understand the complete implementation and design rationale,
refer to the PEP for a particular new feature or to the module
documentation.


%======================================================================
\section{PEP 218: Built-In Set Objects}

Two new built-in types, \function{set(\var{iterable})} and
\function{frozenset(\var{iterable})} provide high speed data types for
membership testing, for eliminating duplicates from sequences, and
for mathematical operations like unions, intersections, differences,
and symmetric differences.  

\begin{verbatim}
>>> a = set('abracadabra')              # form a set from a string
>>> 'z' in a                            # fast membership testing
False
>>> a                                   # unique letters in a
set(['a', 'r', 'b', 'c', 'd'])
>>> ''.join(a)                          # convert back into a string
'arbcd'

>>> b = set('alacazam')                 # form a second set
>>> a - b                               # letters in a but not in b
set(['r', 'd', 'b'])
>>> a | b                               # letters in either a or b
set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'])
>>> a & b                               # letters in both a and b
set(['a', 'c'])
>>> a ^ b                               # letters in a or b but not both
set(['r', 'd', 'b', 'm', 'z', 'l'])

>>> a.add('z')                          # add a new element
>>> a.update('wxy')                     # add multiple new elements
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'x', 'z'])       
>>> a.remove('x')                       # take one element out
>>> a
set(['a', 'c', 'b', 'd', 'r', 'w', 'y', 'z'])       
\end{verbatim}

The type \function{frozenset()} is an immutable version of \function{set()}.
Since it is immutable and hashable, it may be used as a dictionary key or
as a member of another set.  Accordingly, it does not have methods
like \method{add()} and \method{remove()} which could alter its contents.

% XXX what happens to the sets module?
% The current thinking is that the sets module will be left alone.
% That way, existing code will continue to run without alteration.
% Also, the module provides an autoconversion feature not supported by set()
% and frozenset().

\begin{seealso}
\seepep{218}{Adding a Built-In Set Object Type}{Originally proposed by
Greg Wilson and ultimately implemented by Raymond Hettinger.}
\end{seealso}

%======================================================================
\section{PEP 237: Unifying Long Integers and Integers}

The lengthy transition process for the PEP, begun with Python 2.2,
takes another step forward in Python 2.4.  In 2.3, certain integer
operations that would behave differently after int/long unification
triggered \exception{FutureWarning} warnings and returned values
limited to 32 or 64 bits.  In 2.4, these expressions no longer produce
a warning, but they now produce a different value that's a long
integer.  

The problematic expressions are primarily left shifts and lengthy
hexadecimal and octal constants.  For example, \code{2 << 32} is one
expression that results in a warning in 2.3, evaluating to 0 on 32-bit
platforms.  In Python 2.4, this expression now returns 8589934592.


\begin{seealso}
\seepep{237}{Unifying Long Integers and Integers}{Original PEP
written by Moshe Zadka and Gvr.  The changes for 2.4 were implemented by 
Kalle Svensson.}
\end{seealso}

%======================================================================
\section{PEP 289: Generator Expressions}

The iterator feature introduced in Python 2.2 makes it easier to write
programs that loop through large data sets without having the entire
data set in memory at one time.  Programmers can use iterators and the
\module{itertools} module to write code in a fairly functional style.

The fly in the ointment has been list comprehensions, because they
produce a Python list object containing all of the items, unavoidably
pulling them all into memory.  When trying to write a program using the functional approach, it would be natural to write something like:

\begin{verbatim}
links = [link for link in get_all_links() if not link.followed]
for link in links:
    ...
\end{verbatim}

instead of 

\begin{verbatim}
for link in get_all_links():
    if link.followed:
        continue
    ...
\end{verbatim}

The first form is more concise and perhaps more readable, but if
you're dealing with a large number of link objects the second form
would have to be used.

Generator expressions work similarly to list comprehensions but don't
materialize the entire list; instead they create a generator that will
return elements one by one.  The above example could be written as:

\begin{verbatim}
links = (link for link in get_all_links() if not link.followed)
for link in links:
    ...
\end{verbatim}

Generator expressions always have to be written inside parentheses, as
in the above example.  The parentheses signalling a function call also
count, so if you want to create a iterator that will be immediately
passed to a function you could write:

\begin{verbatim}
print sum(obj.count for obj in list_all_objects())
\end{verbatim}

There are some small differences from list comprehensions.  Most
notably, the loop variable (\var{obj} in the above example) is not
accessible outside of the generator expression.  List comprehensions
leave the variable assigned to its last value; future versions of
Python will change this, making list comprehensions match generator
expressions in this respect.

\begin{seealso}
\seepep{289}{Generator Expressions}{Proposed by Raymond Hettinger and
implemented by Jiwon Seo with early efforts steered by Hye-Shik Chang.}
\end{seealso}

%======================================================================
\section{PEP 322: Reverse Iteration}

A new built-in function, \function{reversed(\var{seq})}, takes a sequence
and returns an iterator that returns the elements of the sequence 
in reverse order.  

\begin{verbatim}
>>> for i in reversed(xrange(1,4)):
...    print i
... 
3
2
1
\end{verbatim}

Compared to extended slicing, \code{range(1,4)[::-1]}, \function{reversed()}
is easier to read, runs faster, and uses substantially less memory.

Note that \function{reversed()} only accepts sequences, not arbitrary
iterators.  If you want to reverse an iterator, first convert it to 
a list with \function{list()}.

\begin{verbatim}
>>> input= open('/etc/passwd', 'r')
>>> for line in reversed(list(input)):
...   print line
... 
root:*:0:0:System Administrator:/var/root:/bin/tcsh
  ...
\end{verbatim}

\begin{seealso}
\seepep{322}{Reverse Iteration}{Written and implemented by Raymond Hettinger.}

\end{seealso}


%======================================================================
\section{PEP 327: Decimal Data Type}

Python has always supported floating-point (FP) numbers as a data
type, based on the underlying C \ctype{double} type.  However, while
most programming languages provide a floating-point type, most people
(even programmers) are unaware that computing with floating-point
numbers entails certain unavoidable inaccuracies.  The new decimal
type provides a way to avoid these inaccuracies.

\subsection{Why is Decimal needed?}

The limitations arise from the representation used for floating-point numbers.
FP numbers are made up of three components:

\begin{itemize}
\item The sign, which is -1 or +1.
\item The mantissa, which is a single-digit binary number  
followed by a fractional part.  For example, \code{1.01} in base-2 notation
is \code{1 + 0/2 + 1/4}, or 1.25 in decimal notation.
\item The exponent, which tells where the decimal point is located in the number represented.  
\end{itemize}

For example, the number 1.25 has sign +1, mantissa 1.01 (in binary),
and exponent of 0 (the decimal point doesn't need to be shifted).  The
number 5 has the same sign and mantissa, but the exponent is 2
because the mantissa is multiplied by 4 (2 to the power of the exponent 2).

Modern systems usually provide floating-point support that conforms to
a relevant standard called IEEE 754.  C's \ctype{double} type is
usually implemented as a 64-bit IEEE 754 number, which uses 52 bits of
space for the mantissa.  This means that numbers can only be specified
to 52 bits of precision.  If you're trying to represent numbers whose
expansion repeats endlessly, the expansion is cut off after 52 bits.
Unfortunately, most software needs to produce output in base 10, and
base 10 often gives rise to such repeating decimals.  For example, 1.1
decimal is binary \code{1.0001100110011 ...}; .1 = 1/16 + 1/32 + 1/256
plus an infinite number of additional terms.  IEEE 754 has to chop off
that infinitely repeated decimal after 52 digits, so the
representation is slightly inaccurate.  

Sometimes you can see this inaccuracy when the number is printed:
\begin{verbatim}
>>> 1.1
1.1000000000000001
\end{verbatim}

The inaccuracy isn't always visible when you print the number because
the FP-to-decimal-string conversion is provided by the C library, and
most C libraries try to produce sensible output, but the inaccuracy is
still there and subsequent operations can magnify the error.

For many applications this doesn't matter.  If I'm plotting points and
displaying them on my monitor, the difference between 1.1 and
1.1000000000000001 is too small to be visible.  Reports often limit
output to a certain number of decimal places, and if you round the
number to two or three or even eight decimal places, the error is
never apparent.  However, for applications where it does matter, 
it's a lot of work to implement your own custom arithmetic routines.

\subsection{The \class{Decimal} type}

A new module, \module{decimal}, was added to Python's standard library.
It contains two classes, \class{Decimal} and \class{Context}.
\class{Decimal} instances represent numbers, and
\class{Context} instances are used to wrap up various settings such as the precision and default rounding mode.

\class{Decimal} instances, like regular Python integers and FP numbers, are immutable; once they've been created, you can't change the value it represents.  
\class{Decimal} instances can be created from integers or strings:

\begin{verbatim}
>>> import decimal
>>> decimal.Decimal(1972)
Decimal("1972")
>>> decimal.Decimal("1.1")
Decimal("1.1")
\end{verbatim}

You can also provide tuples containing the sign, mantissa represented 
as a tuple of decimal digits, and exponent:

\begin{verbatim}
>>> decimal.Decimal((1, (1, 4, 7, 5), -2))
Decimal("-14.75")
\end{verbatim}

Cautionary note: the sign bit is a Boolean value, so 0 is positive and 1 is negative.

Floating-point numbers posed a bit of a problem: should the FP number
representing 1.1 turn into the decimal number for exactly 1.1, or for
1.1 plus whatever inaccuracies are introduced?  The decision was to
leave such a conversion out of the API.  Instead, you should convert
the floating-point number into a string using the desired precision and 
pass the string to the \class{Decimal} constructor:

\begin{verbatim}
>>> f = 1.1
>>> decimal.Decimal(str(f))
Decimal("1.1")
>>> decimal.Decimal(repr(f))
Decimal("1.1000000000000001")
\end{verbatim}

Once you have \class{Decimal} instances, you can perform the usual
mathematical operations on them.  One limitation: exponentiation
requires an integer exponent:

\begin{verbatim}
>>> a = decimal.Decimal('35.72')
>>> b = decimal.Decimal('1.73')
>>> a+b
Decimal("37.45")
>>> a-b
Decimal("33.99")
>>> a*b
Decimal("61.7956")
>>> a/b
Decimal("20.6473988")
>>> a ** 2
Decimal("1275.9184")
>>> a ** b
Decimal("NaN")
\end{verbatim}

You can combine \class{Decimal} instances with integers, but not with
floating-point numbers:

\begin{verbatim}
>>> a + 4
Decimal("39.72")
>>> a + 4.5
Traceback (most recent call last):
  ...
TypeError: You can interact Decimal only with int, long or Decimal data types.
>>>
\end{verbatim}

\class{Decimal} numbers can be used with the \module{math} and
\module{cmath} modules, though you'll get back a regular
floating-point number and not a \class{Decimal}.  Instances also have a \method{sqrt()} method:

\begin{verbatim}
>>> import math, cmath
>>> d = decimal.Decimal('123456789012.345')
>>> math.sqrt(d)
351364.18288201344
>>> cmath.sqrt(-d)
351364.18288201344j
>>> d.sqrt()
Decimal(``351364.1828820134592177245001'')
\end{verbatim}


\subsection{The \class{Context} type}

Instances of the \class{Context} class encapsulate several settings for 
decimal operations:

\begin{itemize}
 \item \member{prec} is the precision, the number of decimal places.
 \item \member{rounding} specifies the rounding mode.  The \module{decimal}
       module has constants for the various possibilities:
       \constant{ROUND_DOWN}, \constant{ROUND_CEILING}, \constant{ROUND_HALF_EVEN}, and various others.
 \item \member{trap_enablers} is a dictionary specifying what happens on
encountering certain error conditions: either  an exception is raised or 
a value is returned.  Some examples of error conditions are
division by zero, loss of precision, and overflow.
\end{itemize}

There's a thread-local default context available by calling
\function{getcontext()}; you can change the properties of this context
to alter the default precision, rounding, or trap handling.

\begin{verbatim}
>>> decimal.getcontext().prec
28
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal(``0.1428571428571428571428571429'')
>>> decimal.getcontext().prec = 9 
>>> decimal.Decimal(1) / decimal.Decimal(7)
Decimal(``0.142857143'')
\end{verbatim}

The default action for error conditions is to return a special value
such as infinity or not-a-number, but you can request that exceptions
be raised:

\begin{verbatim}
>>> decimal.Decimal(1) / decimal.Decimal(0)
Decimal(``Infinity'')
>>> decimal.getcontext().trap_enablers[decimal.DivisionByZero] = True
>>> decimal.Decimal(1) / decimal.Decimal(0)
Traceback (most recent call last):
  ...
decimal.DivisionByZero: x / 0
>>> 
\end{verbatim}

The \class{Context} instance also has various methods for formatting 
numbers such as \method{to_eng_string()} and \method{to_sci_string()}.

       
\begin{seealso}
\seepep{327}{Decimal Data Type}{Written by Facundo Batista and implemented
  by Facundo Batista, Eric Price, Raymond Hettinger, Aahz, and Tim Peters.}

\seeurl{http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html}
{A more detailed overview of the IEEE-754 representation.}

\seeurl{http://www.lahey.com/float.htm}
{The article uses Fortran code to illustrate many of the problems
that floating-point inaccuracy can cause.}

\seeurl{http://www2.hursley.ibm.com/decimal/}
{A description of a decimal-based representation.  This representation
is being proposed as a standard, and underlies the new Python decimal
type.  Much of this material was written by Mike Cowlishaw, designer of the
REXX language.}

\end{seealso}      


%======================================================================
\section{Other Language Changes}

Here are all of the changes that Python 2.4 makes to the core Python
language.

\begin{itemize}

\item The \method{dict.update()} method now accepts the same
argument forms as the \class{dict} constructor.  This includes any
mapping, any iterable of key/value pairs, and keyword arguments.       

\item The string methods, \method{ljust()}, \method{rjust()}, and
\method{center()} now take an optional argument for specifying a
fill character other than a space.

\item Strings also gained an \method{rsplit()} method that
works like the \method{split()} method but splits from the end of
the string.  

\begin{verbatim}
>>> 'www.python.org'.split('.', 1)
['www', 'python.org']
'www.python.org'.rsplit('.', 1)
['www.python', 'org']        
\end{verbatim}      

\item The \method{sort()} method of lists gained three keyword
arguments, \var{cmp}, \var{key}, and \var{reverse}.  These arguments
make some common usages of \method{sort()} simpler.  All are optional.

\var{cmp} is the same as the previous single argument to
\method{sort()}; if provided, the value should be a comparison
function that takes two arguments and returns -1, 0, or +1 depending
on how the arguments compare.  

\var{key} should be a single-argument function that takes a list
element and returns a comparison key for the element.  The list is
then sorted using the comparison keys.  The following example sorts a
list case-insensitively:

\begin{verbatim}
>>> L = ['A', 'b', 'c', 'D']
>>> L.sort()                 # Case-sensitive sort
>>> L
['A', 'D', 'b', 'c']
>>> L.sort(key=lambda x: x.lower())
>>> L
['A', 'b', 'c', 'D']
>>> L.sort(cmp=lambda x,y: cmp(x.lower(), y.lower()))
>>> L
['A', 'b', 'c', 'D']
\end{verbatim}

The last example, which uses the \var{cmp} parameter, is the old way
to perform a case-insensitive sort.  It works but is slower than
using a \var{key} parameter.  Using \var{key} results in calling the
\method{lower()} method once for each element in the list while using
\var{cmp} will call the method twice for each comparison.

For simple key functions and comparison functions, it is often
possible to avoid a \keyword{lambda} expression by using an unbound
method instead.  For example, the above case-insensitive sort is best
coded as:

\begin{verbatim}
>>> L.sort(key=str.lower)
>>> L
['A', 'b', 'c', 'D']
\end{verbatim}       

The \var{reverse} parameter should have a Boolean value.  If the value is
\constant{True}, the list will be sorted into reverse order.  Instead
of \code{L.sort(lambda x,y: cmp(y.score, x.score))}, you can now write:
\code{L.sort(key = lambda x: x.score, reverse=True)}.

The results of sorting are now guaranteed to be stable.  This means
that two entries with equal keys will be returned in the same order as
they were input.  For example, you can sort a list of people by name,
and then sort the list by age, resulting in a list sorted by age where
people with the same age are in name-sorted order.

\item There is a new built-in function
\function{sorted(\var{iterable})} that works like the in-place
\method{list.sort()} method but has been made suitable for use in
expressions.  The differences are:
  \begin{itemize}
  \item the input may be any iterable;
  \item a newly formed copy is sorted, leaving the original intact; and
  \item the expression returns the new sorted copy
  \end{itemize}

\begin{verbatim}
>>> L = [9,7,8,3,2,4,1,6,5]
>>> [10+i for i in sorted(L)]       # usable in a list comprehension
[11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> L = [9,7,8,3,2,4,1,6,5]         # original is left unchanged
[9,7,8,3,2,4,1,6,5]   

>>> sorted('Monte Python')          # any iterable may be an input
[' ', 'M', 'P', 'e', 'h', 'n', 'n', 'o', 'o', 't', 't', 'y']

>>> # List the contents of a dict sorted by key values
>>> colormap = dict(red=1, blue=2, green=3, black=4, yellow=5)
>>> for k, v in sorted(colormap.iteritems()):
...     print k, v
...
black 4
blue 2
green 3
red 1
yellow 5

\end{verbatim}

\item The \function{eval(\var{expr}, \var{globals}, \var{locals})}
function now accepts any mapping type for the \var{locals} argument.
Previously this had to be a regular Python dictionary.

\item The \function{zip()} built-in function and \function{itertools.izip()}
  now return an empty list instead of raising a \exception{TypeError}
  exception if called with no arguments.  This makes them more
  suitable for use with variable length argument lists:

\begin{verbatim}
>>> def transpose(array):
...    return zip(*array)
...
>>> transpose([(1,2,3), (4,5,6)])
[(1, 4), (2, 5), (3, 6)]
>>> transpose([])
[]
\end{verbatim}

\end{itemize}


%======================================================================
\subsection{Optimizations}

\begin{itemize}

\item The inner loops for \class{list} and \class{tuple} slicing
 were optimized and now run about one-third faster.  The inner
 loops were also optimized for \class{dict} with performance
 boosts to \method{keys()}, \method{values()}, \method{items()},
\method{iterkeys()}, \method{itervalues()}, and \method{iteritems()}.

\item The machinery for growing and shrinking lists was optimized
 for speed and for space efficiency.  Small lists (under eight elements)
 never over-allocate by more than three elements.  Large lists do not
 over-allocate by more than 1/8th.  Appending and popping from lists
 now runs faster due to more efficient code paths and less frequent
 use of the underlying system realloc().  List comprehensions also
 benefit.  The amount of improvement varies between systems and shows
 the greatest improvement on systems with poor realloc() implementations.
 \method{list.extend()} was also optimized and no longer converts its
 argument into a temporary list prior to extending the base list.

\item \function{list()}, \function{tuple()}, \function{map()},
  \function{filter()}, and \function{zip()} now run several times
  faster with non-sequence arguments that supply a \method{__len__()}
  method.  Previously, the pre-sizing optimization only applied to
  sequence arguments.

\item The methods \method{list.__getitem__()},
  \method{dict.__getitem__()}, and \method{dict.__contains__()} are
  are now implemented as \class{method_descriptor} objects rather
  than \class{wrapper_descriptor} objects.  This form of optimized
  access doubles their performance and makes them more suitable for
  use as arguments to functionals:
  \samp{map(mydict.__getitem__, keylist)}.

\item Added a new opcode, \code{LIST_APPEND}, that simplifies
  the generated bytecode for list comprehensions and speeds them up
  by about a third.

\end{itemize}

The net result of the 2.4 optimizations is that Python 2.4 runs the
pystone benchmark around XX\% faster than Python 2.3 and YY\% faster
than Python 2.2.


%======================================================================
\section{New, Improved, and Deprecated Modules}

As usual, Python's standard library received a number of enhancements and
bug fixes.  Here's a partial list of the most notable changes, sorted
alphabetically by module name. Consult the
\file{Misc/NEWS} file in the source tree for a more
complete list of changes, or look through the CVS logs for all the
details.

\begin{itemize}

\item The \module{asyncore} module's \function{loop()} now has a 
   \var{count} parameter that lets you perform a limited number 
   of passes through the polling loop.  The default is still to loop
   forever.

\item The \module{curses} modules now supports the ncurses extension 
   \function{use_default_colors()}.  On platforms where the terminal
   supports transparency, this makes it possible to use a transparent
   background.  (Contributed by J\"org Lehmann.)

\item The \module{bisect} module now has an underlying C implementation
   for improved performance.
   (Contributed by Dmitry Vasiliev.)

\item The CJKCodecs collections of East Asian codecs, maintained
by Hye-Shik Chang, was integrated into 2.4.  
The new encodings are:

\begin{itemize}
 \item Chinese (PRC): gb2312, gbk, gb18030, hz
 \item Chinese (ROC): big5, cp950
 \item Japanese: cp932, shift-jis, shift-jisx0213, euc-jp,
euc-jisx0213, iso-2022-jp, iso-2022-jp-1, iso-2022-jp-2,
 iso-2022-jp-3, iso-2022-jp-ext
 \item Korean: cp949, euc-kr, johab, iso-2022-kr
\end{itemize} 

\item There is a new \module{collections} module for 
   various specialized collection datatypes.  
   Currently it contains just one type, \class{deque}, 
   a double-ended queue that supports efficiently adding and removing
   elements from either end.

\begin{verbatim}
>>> from collections import deque
>>> d = deque('ghi')        # make a new deque with three items
>>> d.append('j')           # add a new entry to the right side
>>> d.appendleft('f')       # add a new entry to the left side
>>> d                       # show the representation of the deque
deque(['f', 'g', 'h', 'i', 'j'])
>>> d.pop()                 # return and remove the rightmost item
'j'
>>> d.popleft()             # return and remove the leftmost item
'f'
>>> list(d)                 # list the contents of the deque
['g', 'h', 'i']
>>> 'h' in d                # search the deque
True  
\end{verbatim}

Several modules now take advantage of \class{collections.deque} for
improved performance:  \module{Queue}, \module{mutex}, \module{shlex}
\module{threading}, and \module{pydoc}.

\item The \module{ConfigParser} classes have been enhanced slightly.
   The \method{read()} method now returns a list of the files that
   were successfully parsed, and the \method{set()} method raises
   \exception{TypeError} if passed a \var{value} argument that isn't a
   string.

\item The \module{heapq} module has been converted to C.  The resulting
   tenfold improvement in speed makes the module suitable for handling
   high volumes of data.  In addition, the module has two new functions
   \function{nlargest()} and \function{nsmallest()} that use heaps to
   find the N largest or smallest values in a dataset without the
   expense of a full sort.

\item The \module{imaplib} module now supports IMAP's THREAD command.
(Contributed by Yves Dionne.)

\item The \module{itertools} module gained a
  \function{groupby(\var{iterable}\optional{, \var{func}})} function,
  inspired by the GROUP BY clause from SQL.
  \var{iterable} returns a succession of elements, and the optional
  \var{func} is a function that takes an element and returns a key
  value; if omitted, the key is simply the element itself.
  \function{groupby()} then groups the elements into subsequences
  which have matching values of the key, and returns a series of 2-tuples
  containing the key value and an iterator over the subsequence.
 
Here's an example.  The \var{key} function simply returns whether a
number is even or odd, so the result of \function{groupby()} is to
return consecutive runs of odd or even numbers.

\begin{verbatim}
>>> import itertools
>>> L = [2,4,6, 7,8,9,11, 12, 14]
>>> for key_val, it in itertools.groupby(L, lambda x: x % 2):
...    print key_val, list(it)
... 
0 [2, 4, 6]
1 [7]
0 [8]
1 [9, 11]
0 [12, 14]
>>> 
\end{verbatim}

Like its SQL counterpart, \function{groupby()} is typically used with
sorted input.  The logic for \function{groupby()} is similar to the
\UNIX{} \code{uniq} filter which makes it handy for eliminating,
counting, or identifying duplicate elements:

\begin{verbatim}
>>> word = 'abracadabra'
>>> letters = sorted(word)   # Turn string into a sorted list of letters
>>> letters 
['a', 'a', 'a', 'a', 'a', 'b', 'b', 'c', 'd', 'r', 'r']
>>> [k for k, g in groupby(letters)]                     # List unique letters
['a', 'b', 'c', 'd', 'r']
>>> [(k, len(list(g))) for k, g in groupby(letters)]     # Count letter occurences
[('a', 5), ('b', 2), ('c', 1), ('d', 1), ('r', 2)]
>>> [k for k, g in groupby(letters) if len(list(g)) > 1] # List duplicated letters
['a', 'b', 'r']
\end{verbatim}

\item \module{itertools} also gained a function named
\function{tee(\var{iterator}, \var{N})} that returns \var{N} independent
iterators that replicate \var{iterator}.  If \var{N} is omitted, the
default is 2.

\begin{verbatim}
>>> L = [1,2,3]
>>> i1, i2 = itertools.tee(L)
>>> i1,i2
(<itertools.tee object at 0x402c2080>, <itertools.tee object at 0x402c2090>)
>>> list(i1)               # Run the first iterator to exhaustion
[1, 2, 3]
>>> list(i2)               # Run the second iterator to exhaustion
[1, 2, 3]
>\end{verbatim}

Note that \function{tee()} has to keep copies of the values returned 
by the iterator; in the worst case, it may need to keep all of them.  
This should therefore be used carefully if the leading iterator
can run far ahead of the trailing iterator in a long stream of inputs.
If the separation is large, then it becomes preferable to use
\function{list()} instead.  When the iterators track closely with one
another, \function{tee()} is ideal.  Possible applications include
bookmarking, windowing, or lookahead iterators.

\item The \module{operator} module gained two new functions, 
\function{attrgetter(\var{attr})} and \function{itemgetter(\var{index})}.
Both functions return callables that take a single argument and return
the corresponding attribute or item; these callables make excellent
data extractors when used with \function{map()} or \function{sorted()}.
For example:

\begin{verbatim}
>>> L = [('c', 2), ('d', 1), ('a', 4), ('b', 3)]
>>> map(operator.itemgetter(0), L)
['c', 'd', 'a', 'b']
>>> map(operator.itemgetter(1), L)
[2, 1, 4, 3]
>>> sorted(L, key=operator.itemgetter(1)) # Sort list by second tuple item
[('d', 1), ('c', 2), ('b', 3), ('a', 4)]
\end{verbatim}

\item A new \function{getsid()} function was added to the
\module{posix} module that underlies the \module{os} module.
(Contributed by J. Raynor.)

\item The \module{poplib} module now supports POP over SSL.

\item The \module{profile} module can now profile C extension functions.
% XXX more to say about this? 

\item The \module{random} module has a new method called \method{getrandbits(N)} 
   which returns an N-bit long integer.  This method supports the existing
   \method{randrange()} method, making it possible to efficiently generate
   arbitrarily large random numbers.

\item The regular expression language accepted by the \module{re} module
   was extended with simple conditional expressions, written as
   \code{(?(\var{group})\var{A}|\var{B})}.  \var{group} is either a
   numeric group ID or a group name defined with \code{(?P<group>...)} 
   earlier in the expression.  If the specified group matched, the
   regular expression pattern \var{A} will be tested against the string; if
   the group didn't match, the pattern \var{B} will be used instead.

\item The \module{weakref} module now supports a wider variety of objects
   including Python functions, class instances, sets, frozensets, deques,
   arrays, files, sockets, and regular expression pattern objects.

\item The \module{xmlrpclib} module now supports a multi-call extension for 
tranmitting multiple XML-RPC calls in a single HTTP operation.
   
\end{itemize}


%======================================================================
% whole new modules get described in \subsections here

\subsection{cookielib}

The \module{cookielib} library supports client-side handling for HTTP
cookies, just as the \module{Cookie} provides server-side cookie
support in CGI scripts. This library manages cookies in a way similar
to web browsers. Cookies are stored in cookie jars; the library
transparently stores cookies offered by the web server in the cookie
jar, and fetches the cookie from the jar when connecting to the
server. Similar to web browsers, policy objects control whether
cookies are accepted or not.

In order to store cookies across sessions, two implementations of
cookie jars are provided: one that stores cookies in the Netscape
format, so applications can use the Mozilla or Lynx cookie jars, and
one that stores cookies in the same format as the Perl libwww libary.

\module{urllib2} has been changed to interact with \module{cookielib}:
\class{HTTPCookieProcessor} manages a cookie jar that is used when
accessing URLs.

% ======================================================================
\section{Build and C API Changes}

Changes to Python's build process and to the C API include:

\begin{itemize}

  \item Three new convenience macros were added for common return
  values from extension functions: \csimplemacro{Py_RETURN_NONE},
  \csimplemacro{Py_RETURN_TRUE}, and \csimplemacro{Py_RETURN_FALSE}.

  \item A new function, \cfunction{PyTuple_Pack(\var{N}, \var{obj1},
  \var{obj2}, ..., \var{objN})}, constructs tuples from a variable
  length argument list of Python objects.

  \item A new function, \cfunction{PyDict_Contains(\var{d}, \var{k})},
  implements fast dictionary lookups without masking exceptions raised
  during the look-up process.

  \item A new method flag, \constant{METH_COEXISTS}, allows a function
  defined in slots to co-exist with a PyCFunction having the same name.
  This can halve the access to time to a method such as
  \method{set.__contains__()}.

  \item Python can now be built with additional profiling for the interpreter
   itself, useful if you're working on the Python core.  
   Providing \longprogramopt{--enable-profiling} to the  
   \program{configure} script will let you profile the interpreter with 
   \program{gprof}, and providing the \longprogramopt{--with-tsc} switch 
   enables profiling using the Pentium's Time-Stamp-Counter.

  \item The \ctype{tracebackobject} type has been renamed to \ctype{PyTracebackObject}.

\end{itemize}


%======================================================================
\subsection{Port-Specific Changes}

\begin{itemize}

\item The Windows port now builds under MSVC++ 7.1 as well as version 6.

\end{itemize}


%======================================================================
\section{Other Changes and Fixes \label{section-other}}

As usual, there were a bunch of other improvements and bugfixes
scattered throughout the source tree.  A search through the CVS change
logs finds there were XXX patches applied and YYY bugs fixed between
Python 2.3 and 2.4.  Both figures are likely to be underestimates.

Some of the more notable changes are:

\begin{itemize}

\item The \module{timeit} module now automatically disables periodic
  garbarge collection during the timing loop.  This change makes
  consecutive timings more comparable.

\item The \module{base64} module now has more complete RFC 3548 support
  for Base64, Base32, and Base16 encoding and decoding, including
  optional case folding and optional alternative alphabets.
  (Contributed by Barry Warsaw.)

\end{itemize}


%======================================================================
\section{Porting to Python 2.4}

This section lists previously described changes that may require
changes to your code:

\begin{itemize}

\item The \function{zip()} built-in function and \function{itertools.izip()}
  now return  an empty list instead of raising a \exception{TypeError}
  exception if called with no arguments.

\item \function{dircache.listdir()} now passes exceptions to the caller
      instead of returning empty lists.

\item \function{LexicalHandler.startDTD()} used to receive public and
  system ID in the wrong order.  This has been corrected; applications
  relying on the wrong order need to be fixed.

\item \function{fcntl.ioctl} now warns if the mutate arg is omitted
 and relevant.

\end{itemize}


%======================================================================
\section{Acknowledgements \label{acks}}

The author would like to thank the following people for offering
suggestions, corrections and assistance with various drafts of this
article: Raymond Hettinger.

\end{document}