Fred Drake | 03e1031 | 2002-03-26 19:17:43 +0000 | [diff] [blame] | 1 | \documentclass{howto} |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 2 | % $Id$ |
| 3 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 4 | % TODO: |
| 5 | % Go through and get the contributor's name for all the various changes |
| 6 | |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 7 | \title{What's New in Python 2.3} |
Andrew M. Kuchling | 72b58e0 | 2002-05-29 17:30:34 +0000 | [diff] [blame] | 8 | \release{0.02} |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 9 | \author{A.M. Kuchling} |
| 10 | \authoraddress{\email{akuchlin@mems-exchange.org}} |
Fred Drake | 03e1031 | 2002-03-26 19:17:43 +0000 | [diff] [blame] | 11 | |
| 12 | \begin{document} |
| 13 | \maketitle |
| 14 | \tableofcontents |
| 15 | |
Andrew M. Kuchling | f70a0a8 | 2002-06-10 13:22:46 +0000 | [diff] [blame] | 16 | % Timeout sockets: |
| 17 | % Executive summary: after sock.settimeout(T), all methods of sock will |
| 18 | % block for at most T floating seconds and fail if they can't complete |
| 19 | % within that time. sock.settimeout(None) restores full blocking mode. |
| 20 | % |
| 21 | % Optik (or whatever it gets called) |
| 22 | % |
| 23 | % getopt.gnu_getopt |
| 24 | % |
Andrew M. Kuchling | 7f147a7 | 2002-06-10 18:58:19 +0000 | [diff] [blame] | 25 | % Docstrings now optional (with --without-doc-strings) |
| 26 | % |
Andrew M. Kuchling | f70a0a8 | 2002-06-10 13:22:46 +0000 | [diff] [blame] | 27 | |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 28 | %\section{Introduction \label{intro}} |
| 29 | |
| 30 | {\large This article is a draft, and is currently up to date for some |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 31 | random version of the CVS tree around May 26 2002. Please send any |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 32 | additions, comments or errata to the author.} |
| 33 | |
| 34 | This article explains the new features in Python 2.3. The tentative |
| 35 | release date of Python 2.3 is currently scheduled for August 30 2002. |
| 36 | |
| 37 | This article doesn't attempt to provide a complete specification of |
| 38 | the new features, but instead provides a convenient overview. For |
| 39 | full details, you should refer to the documentation for Python 2.3, |
| 40 | such as the |
| 41 | \citetitle[http://www.python.org/doc/2.3/lib/lib.html]{Python Library |
| 42 | Reference} and the |
| 43 | \citetitle[http://www.python.org/doc/2.3/ref/ref.html]{Python |
| 44 | Reference Manual}. If you want to understand the complete |
| 45 | implementation and design rationale for a change, refer to the PEP for |
| 46 | a particular new feature. |
Fred Drake | 03e1031 | 2002-03-26 19:17:43 +0000 | [diff] [blame] | 47 | |
| 48 | |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 49 | %====================================================================== |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 50 | \section{PEP 255: Simple Generators\label{section-generators}} |
Andrew M. Kuchling | f4dd65d | 2002-04-01 19:28:09 +0000 | [diff] [blame] | 51 | |
| 52 | In Python 2.2, generators were added as an optional feature, to be |
| 53 | enabled by a \code{from __future__ import generators} directive. In |
| 54 | 2.3 generators no longer need to be specially enabled, and are now |
| 55 | always present; this means that \keyword{yield} is now always a |
| 56 | keyword. The rest of this section is a copy of the description of |
| 57 | generators from the ``What's New in Python 2.2'' document; if you read |
| 58 | it when 2.2 came out, you can skip the rest of this section. |
| 59 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 60 | You're doubtless familiar with how function calls work in Python or C. |
| 61 | When you call a function, it gets a private namespace where its local |
Andrew M. Kuchling | f4dd65d | 2002-04-01 19:28:09 +0000 | [diff] [blame] | 62 | variables are created. When the function reaches a \keyword{return} |
| 63 | statement, the local variables are destroyed and the resulting value |
| 64 | is returned to the caller. A later call to the same function will get |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 65 | a fresh new set of local variables. But, what if the local variables |
Andrew M. Kuchling | f4dd65d | 2002-04-01 19:28:09 +0000 | [diff] [blame] | 66 | weren't thrown away on exiting a function? What if you could later |
| 67 | resume the function where it left off? This is what generators |
| 68 | provide; they can be thought of as resumable functions. |
| 69 | |
| 70 | Here's the simplest example of a generator function: |
| 71 | |
| 72 | \begin{verbatim} |
| 73 | def generate_ints(N): |
| 74 | for i in range(N): |
| 75 | yield i |
| 76 | \end{verbatim} |
| 77 | |
| 78 | A new keyword, \keyword{yield}, was introduced for generators. Any |
| 79 | function containing a \keyword{yield} statement is a generator |
| 80 | function; this is detected by Python's bytecode compiler which |
| 81 | compiles the function specially as a result. |
| 82 | |
| 83 | When you call a generator function, it doesn't return a single value; |
| 84 | instead it returns a generator object that supports the iterator |
| 85 | protocol. On executing the \keyword{yield} statement, the generator |
| 86 | outputs the value of \code{i}, similar to a \keyword{return} |
| 87 | statement. The big difference between \keyword{yield} and a |
| 88 | \keyword{return} statement is that on reaching a \keyword{yield} the |
| 89 | generator's state of execution is suspended and local variables are |
| 90 | preserved. On the next call to the generator's \code{.next()} method, |
| 91 | the function will resume executing immediately after the |
| 92 | \keyword{yield} statement. (For complicated reasons, the |
| 93 | \keyword{yield} statement isn't allowed inside the \keyword{try} block |
| 94 | of a \code{try...finally} statement; read \pep{255} for a full |
| 95 | explanation of the interaction between \keyword{yield} and |
| 96 | exceptions.) |
| 97 | |
| 98 | Here's a sample usage of the \function{generate_ints} generator: |
| 99 | |
| 100 | \begin{verbatim} |
| 101 | >>> gen = generate_ints(3) |
| 102 | >>> gen |
| 103 | <generator object at 0x8117f90> |
| 104 | >>> gen.next() |
| 105 | 0 |
| 106 | >>> gen.next() |
| 107 | 1 |
| 108 | >>> gen.next() |
| 109 | 2 |
| 110 | >>> gen.next() |
| 111 | Traceback (most recent call last): |
| 112 | File "<stdin>", line 1, in ? |
| 113 | File "<stdin>", line 2, in generate_ints |
| 114 | StopIteration |
| 115 | \end{verbatim} |
| 116 | |
| 117 | You could equally write \code{for i in generate_ints(5)}, or |
| 118 | \code{a,b,c = generate_ints(3)}. |
| 119 | |
| 120 | Inside a generator function, the \keyword{return} statement can only |
| 121 | be used without a value, and signals the end of the procession of |
| 122 | values; afterwards the generator cannot return any further values. |
| 123 | \keyword{return} with a value, such as \code{return 5}, is a syntax |
| 124 | error inside a generator function. The end of the generator's results |
| 125 | can also be indicated by raising \exception{StopIteration} manually, |
| 126 | or by just letting the flow of execution fall off the bottom of the |
| 127 | function. |
| 128 | |
| 129 | You could achieve the effect of generators manually by writing your |
| 130 | own class and storing all the local variables of the generator as |
| 131 | instance variables. For example, returning a list of integers could |
| 132 | be done by setting \code{self.count} to 0, and having the |
| 133 | \method{next()} method increment \code{self.count} and return it. |
| 134 | However, for a moderately complicated generator, writing a |
| 135 | corresponding class would be much messier. |
| 136 | \file{Lib/test/test_generators.py} contains a number of more |
| 137 | interesting examples. The simplest one implements an in-order |
| 138 | traversal of a tree using generators recursively. |
| 139 | |
| 140 | \begin{verbatim} |
| 141 | # A recursive generator that generates Tree leaves in in-order. |
| 142 | def inorder(t): |
| 143 | if t: |
| 144 | for x in inorder(t.left): |
| 145 | yield x |
| 146 | yield t.label |
| 147 | for x in inorder(t.right): |
| 148 | yield x |
| 149 | \end{verbatim} |
| 150 | |
| 151 | Two other examples in \file{Lib/test/test_generators.py} produce |
| 152 | solutions for the N-Queens problem (placing $N$ queens on an $NxN$ |
| 153 | chess board so that no queen threatens another) and the Knight's Tour |
| 154 | (a route that takes a knight to every square of an $NxN$ chessboard |
| 155 | without visiting any square twice). |
| 156 | |
| 157 | The idea of generators comes from other programming languages, |
| 158 | especially Icon (\url{http://www.cs.arizona.edu/icon/}), where the |
| 159 | idea of generators is central. In Icon, every |
| 160 | expression and function call behaves like a generator. One example |
| 161 | from ``An Overview of the Icon Programming Language'' at |
| 162 | \url{http://www.cs.arizona.edu/icon/docs/ipd266.htm} gives an idea of |
| 163 | what this looks like: |
| 164 | |
| 165 | \begin{verbatim} |
| 166 | sentence := "Store it in the neighboring harbor" |
| 167 | if (i := find("or", sentence)) > 5 then write(i) |
| 168 | \end{verbatim} |
| 169 | |
| 170 | In Icon the \function{find()} function returns the indexes at which the |
| 171 | substring ``or'' is found: 3, 23, 33. In the \keyword{if} statement, |
| 172 | \code{i} is first assigned a value of 3, but 3 is less than 5, so the |
| 173 | comparison fails, and Icon retries it with the second value of 23. 23 |
| 174 | is greater than 5, so the comparison now succeeds, and the code prints |
| 175 | the value 23 to the screen. |
| 176 | |
| 177 | Python doesn't go nearly as far as Icon in adopting generators as a |
| 178 | central concept. Generators are considered a new part of the core |
| 179 | Python language, but learning or using them isn't compulsory; if they |
| 180 | don't solve any problems that you have, feel free to ignore them. |
| 181 | One novel feature of Python's interface as compared to |
| 182 | Icon's is that a generator's state is represented as a concrete object |
| 183 | (the iterator) that can be passed around to other functions or stored |
| 184 | in a data structure. |
| 185 | |
| 186 | \begin{seealso} |
| 187 | |
| 188 | \seepep{255}{Simple Generators}{Written by Neil Schemenauer, Tim |
| 189 | Peters, Magnus Lie Hetland. Implemented mostly by Neil Schemenauer |
| 190 | and Tim Peters, with other fixes from the Python Labs crew.} |
| 191 | |
| 192 | \end{seealso} |
| 193 | |
| 194 | |
| 195 | %====================================================================== |
Andrew M. Kuchling | f367651 | 2002-04-15 02:27:55 +0000 | [diff] [blame] | 196 | \section{PEP 278: Universal Newline Support} |
| 197 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 198 | The three major operating systems used today are Microsoft Windows, |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 199 | Apple's Macintosh OS, and the various \UNIX\ derivatives. A minor |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 200 | irritation is that these three platforms all use different characters |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 201 | to mark the ends of lines in text files. \UNIX\ uses character 10, |
| 202 | the ASCII linefeed, while MacOS uses character 13, the ASCII carriage |
| 203 | return, and Windows uses a two-character sequence of a carriage return |
| 204 | plus a newline. |
Andrew M. Kuchling | f367651 | 2002-04-15 02:27:55 +0000 | [diff] [blame] | 205 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 206 | Python's file objects can now support end of line conventions other |
| 207 | than the one followed by the platform on which Python is running. |
| 208 | Opening a file with the mode \samp{U} or \samp{rU} will open a file |
| 209 | for reading in universal newline mode. All three line ending |
| 210 | conventions will be translated to a \samp{\e n} in the strings |
| 211 | returned by the various file methods such as \method{read()} and |
| 212 | \method{readline()}. |
Andrew M. Kuchling | f367651 | 2002-04-15 02:27:55 +0000 | [diff] [blame] | 213 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 214 | Universal newline support is also used when importing modules and when |
| 215 | executing a file with the \function{execfile()} function. This means |
| 216 | that Python modules can be shared between all three operating systems |
| 217 | without needing to convert the line-endings. |
| 218 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 219 | This feature can be disabled at compile-time by specifying |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 220 | \longprogramopt{without-universal-newlines} when running Python's |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 221 | \file{configure} script. |
Andrew M. Kuchling | f367651 | 2002-04-15 02:27:55 +0000 | [diff] [blame] | 222 | |
| 223 | \begin{seealso} |
| 224 | |
| 225 | \seepep{278}{Universal Newline Support}{Written |
| 226 | and implemented by Jack Jansen.} |
| 227 | |
| 228 | \end{seealso} |
| 229 | |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 230 | |
| 231 | %====================================================================== |
| 232 | \section{PEP 279: The \function{enumerate()} Built-in Function} |
| 233 | |
| 234 | A new built-in function, \function{enumerate()}, will make |
| 235 | certain loops a bit clearer. \code{enumerate(thing)}, where |
| 236 | \var{thing} is either an iterator or a sequence, returns a iterator |
| 237 | that will return \code{(0, \var{thing[0]})}, \code{(1, |
| 238 | \var{thing[1]})}, \code{(2, \var{thing[2]})}, and so forth. Fairly |
| 239 | often you'll see code to change every element of a list that looks |
| 240 | like this: |
| 241 | |
| 242 | \begin{verbatim} |
| 243 | for i in range(len(L)): |
| 244 | item = L[i] |
| 245 | # ... compute some result based on item ... |
| 246 | L[i] = result |
| 247 | \end{verbatim} |
| 248 | |
| 249 | This can be rewritten using \function{enumerate()} as: |
| 250 | |
| 251 | \begin{verbatim} |
| 252 | for i, item in enumerate(L): |
| 253 | # ... compute some result based on item ... |
| 254 | L[i] = result |
| 255 | \end{verbatim} |
| 256 | |
| 257 | |
| 258 | \begin{seealso} |
| 259 | |
| 260 | \seepep{279}{The enumerate() built-in function}{Written |
| 261 | by Raymond D. Hettinger.} |
| 262 | |
| 263 | \end{seealso} |
| 264 | |
| 265 | |
Andrew M. Kuchling | f367651 | 2002-04-15 02:27:55 +0000 | [diff] [blame] | 266 | %====================================================================== |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 267 | \section{PEP 285: The \class{bool} Type\label{section-bool}} |
| 268 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 269 | A Boolean type was added to Python 2.3. Two new constants were added |
| 270 | to the \module{__builtin__} module, \constant{True} and |
| 271 | \constant{False}. The type object for this new type is named |
| 272 | \class{bool}; the constructor for it takes any Python value and |
| 273 | converts it to \constant{True} or \constant{False}. |
| 274 | |
| 275 | \begin{verbatim} |
| 276 | >>> bool(1) |
| 277 | True |
| 278 | >>> bool(0) |
| 279 | False |
| 280 | >>> bool([]) |
| 281 | False |
| 282 | >>> bool( (1,) ) |
| 283 | True |
| 284 | \end{verbatim} |
| 285 | |
| 286 | Most of the standard library modules and built-in functions have been |
| 287 | changed to return Booleans. |
| 288 | |
| 289 | \begin{verbatim} |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 290 | >>> obj = [] |
| 291 | >>> hasattr(obj, 'append') |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 292 | True |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 293 | >>> isinstance(obj, list) |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 294 | True |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 295 | >>> isinstance(obj, tuple) |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 296 | False |
| 297 | \end{verbatim} |
| 298 | |
| 299 | Python's Booleans were added with the primary goal of making code |
| 300 | clearer. For example, if you're reading a function and encounter the |
| 301 | statement \code{return 1}, you might wonder whether the \samp{1} |
| 302 | represents a truth value, or whether it's an index, or whether it's a |
| 303 | coefficient that multiplies some other quantity. If the statement is |
| 304 | \code{return True}, however, the meaning of the return value is quite |
| 305 | clearly a truth value. |
| 306 | |
| 307 | Python's Booleans were not added for the sake of strict type-checking. |
Andrew M. Kuchling | a2a206b | 2002-05-24 21:08:58 +0000 | [diff] [blame] | 308 | A very strict language such as Pascal would also prevent you |
| 309 | performing arithmetic with Booleans, and would require that the |
| 310 | expression in an \keyword{if} statement always evaluate to a Boolean. |
| 311 | Python is not this strict, and it never will be. (\pep{285} |
| 312 | explicitly says so.) So you can still use any expression in an |
| 313 | \keyword{if}, even ones that evaluate to a list or tuple or some |
| 314 | random object, and the Boolean type is a subclass of the |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 315 | \class{int} class, so arithmetic using a Boolean still works. |
| 316 | |
| 317 | \begin{verbatim} |
| 318 | >>> True + 1 |
| 319 | 2 |
| 320 | >>> False + 1 |
| 321 | 1 |
| 322 | >>> False * 75 |
| 323 | 0 |
| 324 | >>> True * 75 |
| 325 | 75 |
| 326 | \end{verbatim} |
| 327 | |
| 328 | To sum up \constant{True} and \constant{False} in a sentence: they're |
| 329 | alternative ways to spell the integer values 1 and 0, with the single |
| 330 | difference that \function{str()} and \function{repr()} return the |
| 331 | strings \samp{True} and \samp{False} instead of \samp{1} and \samp{0}. |
Andrew M. Kuchling | 3a52ff6 | 2002-04-03 22:44:47 +0000 | [diff] [blame] | 332 | |
| 333 | \begin{seealso} |
| 334 | |
| 335 | \seepep{285}{Adding a bool type}{Written and implemented by GvR.} |
| 336 | |
| 337 | \end{seealso} |
| 338 | |
Michael W. Hudson | 5efaf7e | 2002-06-11 10:55:12 +0000 | [diff] [blame] | 339 | \section{Extended Slices\label{extended-slices}} |
| 340 | |
| 341 | Ever since Python 1.4 the slice syntax has supported a third |
| 342 | ``stride'' argument, but the builtin sequence types have not supported |
| 343 | this feature (it was initially included at the behest of the |
| 344 | developers of the Numerical Python package). This changes with Python |
| 345 | 2.3. |
| 346 | |
| 347 | % XXX examples, etc. |
Andrew M. Kuchling | 3a52ff6 | 2002-04-03 22:44:47 +0000 | [diff] [blame] | 348 | |
| 349 | %====================================================================== |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 350 | %\section{Other Language Changes} |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 351 | |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 352 | %Here are the changes that Python 2.3 makes to the core language. |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 353 | |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 354 | %\begin{itemize} |
| 355 | %\item The \keyword{yield} statement is now always a keyword, as |
| 356 | %described in section~\ref{section-generators}. |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 357 | |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 358 | %\item Two new constants, \constant{True} and \constant{False} were |
| 359 | %added along with the built-in \class{bool} type, as described in |
| 360 | %section~\ref{section-bool}. |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 361 | |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 362 | %\item |
| 363 | %\end{itemize} |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 364 | |
| 365 | |
Neal Norwitz | d68f517 | 2002-05-29 15:54:55 +0000 | [diff] [blame] | 366 | %\begin{PendingDeprecationWarning} |
| 367 | A new warning PendingDeprecationWarning was added to provide |
| 368 | direction on features which are in the process of being deprecated. |
| 369 | The warning will not be printed by default. To see the pending |
| 370 | deprecations, use -Walways::PendingDeprecationWarning:: on the command line |
| 371 | or warnings.filterwarnings(). |
| 372 | %\end{PendingDeprecationWarning} |
| 373 | |
| 374 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 375 | %====================================================================== |
| 376 | \section{Specialized Object Allocator (pymalloc)\label{section-pymalloc}} |
| 377 | |
| 378 | An experimental feature added to Python 2.1 was a specialized object |
| 379 | allocator called pymalloc, written by Vladimir Marangozov. Pymalloc |
| 380 | was intended to be faster than the system \function{malloc()} and have |
Michael W. Hudson | 497bdd6 | 2002-06-10 13:19:42 +0000 | [diff] [blame] | 381 | less memory overhead for typical allocation patterns of Python |
| 382 | programs. The allocator uses C's \function{malloc()} function to get |
| 383 | large pools of memory, and then fulfills smaller memory requests from |
| 384 | these pools. |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 385 | |
| 386 | In 2.1 and 2.2, pymalloc was an experimental feature and wasn't |
| 387 | enabled by default; you had to explicitly turn it on by providing the |
| 388 | \longprogramopt{with-pymalloc} option to the \program{configure} |
| 389 | script. In 2.3, pymalloc has had further enhancements and is now |
| 390 | enabled by default; you'll have to supply |
| 391 | \longprogramopt{without-pymalloc} to disable it. |
| 392 | |
| 393 | This change is transparent to code written in Python; however, |
| 394 | pymalloc may expose bugs in C extensions. Authors of C extension |
| 395 | modules should test their code with the object allocator enabled, |
| 396 | because some incorrect code may cause core dumps at runtime. There |
| 397 | are a bunch of memory allocation functions in Python's C API that have |
| 398 | previously been just aliases for the C library's \function{malloc()} |
| 399 | and \function{free()}, meaning that if you accidentally called |
| 400 | mismatched functions, the error wouldn't be noticeable. When the |
| 401 | object allocator is enabled, these functions aren't aliases of |
| 402 | \function{malloc()} and \function{free()} any more, and calling the |
Michael W. Hudson | 497bdd6 | 2002-06-10 13:19:42 +0000 | [diff] [blame] | 403 | wrong function to free memory may get you a core dump. For example, |
| 404 | if memory was allocated using \function{PyObject_Malloc()}, it has to |
| 405 | be freed using \function{PyObject_Free()}, not \function{free()}. A |
| 406 | few modules included with Python fell afoul of this and had to be |
| 407 | fixed; doubtless there are more third-party modules that will have the |
| 408 | same problem. |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 409 | |
| 410 | As part of this change, the confusing multiple interfaces for |
Michael W. Hudson | 497bdd6 | 2002-06-10 13:19:42 +0000 | [diff] [blame] | 411 | allocating memory have been consolidated down into two API families. |
| 412 | Memory allocated with one family must not be manipulated with |
| 413 | functions from the other family. |
| 414 | |
| 415 | There is another family of functions specifically for allocating |
| 416 | Python \emph{objects} (as opposed to memory). |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 417 | |
| 418 | \begin{itemize} |
Michael W. Hudson | 497bdd6 | 2002-06-10 13:19:42 +0000 | [diff] [blame] | 419 | \item To allocate and free an undistinguished chunk of memory use |
| 420 | the ``raw memory'' family: \cfunction{PyMem_Malloc()}, |
| 421 | \cfunction{PyMem_Realloc()}, and \cfunction{PyMem_Free()}. |
Andrew M. Kuchling | a2a206b | 2002-05-24 21:08:58 +0000 | [diff] [blame] | 422 | |
Michael W. Hudson | 497bdd6 | 2002-06-10 13:19:42 +0000 | [diff] [blame] | 423 | \item The ``object memory'' family is the interface to the pymalloc |
| 424 | facility described above and is biased towards a large number of |
| 425 | ``small'' allocations: \cfunction{PyObject_Malloc}, |
| 426 | \cfunction{PyObject_Realloc}, and \cfunction{PyObject_Free}. |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 427 | |
Michael W. Hudson | 497bdd6 | 2002-06-10 13:19:42 +0000 | [diff] [blame] | 428 | \item To allocate and free Python objects, use the ``object'' family |
| 429 | \cfunction{PyObject_New()}, \cfunction{PyObject_NewVar()}, and |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 430 | \cfunction{PyObject_Del()}. |
| 431 | \end{itemize} |
| 432 | |
| 433 | Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides |
| 434 | debugging features to catch memory overwrites and doubled frees in |
| 435 | both extension modules and in the interpreter itself. To enable this |
| 436 | support, turn on the Python interpreter's debugging code by running |
| 437 | \program{configure} with \longprogramopt{with-pydebug}. |
| 438 | |
Michael W. Hudson | 497bdd6 | 2002-06-10 13:19:42 +0000 | [diff] [blame] | 439 | To aid extension writers, a header file \file{Misc/pymemcompat.h} is |
| 440 | distributed with the source to Python 2.3 that allows Python |
| 441 | extensions to use the 2.3 interfaces to memory allocation and compile |
| 442 | against any version of Python since 1.5.2. (The idea is that you take |
| 443 | the file from Python's source distribution and bundle it with the |
Andrew M. Kuchling | f70a0a8 | 2002-06-10 13:22:46 +0000 | [diff] [blame] | 444 | source of your extension). |
Michael W. Hudson | 497bdd6 | 2002-06-10 13:19:42 +0000 | [diff] [blame] | 445 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 446 | \begin{seealso} |
| 447 | |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 448 | \seeurl{http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Objects/obmalloc.c} |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 449 | {For the full details of the pymalloc implementation, see |
| 450 | the comments at the top of the file \file{Objects/obmalloc.c} in the |
| 451 | Python source code. The above link points to the file within the |
| 452 | SourceForge CVS browser.} |
| 453 | |
| 454 | \end{seealso} |
| 455 | |
| 456 | %====================================================================== |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 457 | \section{New and Improved Modules} |
| 458 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 459 | As usual, Python's standard modules had a number of enhancements and |
| 460 | bug fixes. Here's a partial list; consult the \file{Misc/NEWS} file |
| 461 | in the source tree, or the CVS logs, for a more complete list. |
| 462 | |
| 463 | \begin{itemize} |
| 464 | |
| 465 | \item One minor but far-reaching change is that the names of extension |
| 466 | types defined by the modules included with Python now contain the |
| 467 | module and a \samp{.} in front of the type name. For example, in |
| 468 | Python 2.2, if you created a socket and printed its |
| 469 | \member{__class__}, you'd get this output: |
| 470 | |
| 471 | \begin{verbatim} |
| 472 | >>> s = socket.socket() |
| 473 | >>> s.__class__ |
| 474 | <type 'socket'> |
| 475 | \end{verbatim} |
| 476 | |
| 477 | In 2.3, you get this: |
| 478 | \begin{verbatim} |
| 479 | >>> s.__class__ |
| 480 | <type '_socket.socket'> |
| 481 | \end{verbatim} |
| 482 | |
| 483 | \item The \method{strip()}, \method{lstrip()}, and \method{rstrip()} |
| 484 | string methods now have an optional argument for specifying the |
| 485 | characters to strip. The default is still to remove all whitespace |
| 486 | characters: |
| 487 | |
| 488 | \begin{verbatim} |
| 489 | >>> ' abc '.strip() |
| 490 | 'abc' |
| 491 | >>> '><><abc<><><>'.strip('<>') |
| 492 | 'abc' |
| 493 | >>> '><><abc<><><>\n'.strip('<>') |
| 494 | 'abc<><><>\n' |
| 495 | >>> u'\u4000\u4001abc\u4000'.strip(u'\u4000') |
| 496 | u'\u4001abc' |
| 497 | >>> |
| 498 | \end{verbatim} |
| 499 | |
Neal Norwitz | 1f68fc7 | 2002-06-14 00:50:42 +0000 | [diff] [blame] | 500 | \item The \method{startswith()} and \method{endswith()} |
| 501 | string methods now have accept negative numbers for |
| 502 | start and end parameters. |
| 503 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 504 | \item Another new string method is \method{zfill()}, originally a |
| 505 | function in the \module{string} module. \method{zfill()} pads a |
| 506 | numeric string with zeros on the left until it's the specified width. |
| 507 | Note that the \code{\%} operator is still more flexible and powerful |
| 508 | than \method{zfill()}. |
| 509 | |
| 510 | \begin{verbatim} |
| 511 | >>> '45'.zfill(4) |
| 512 | '0045' |
| 513 | >>> '12345'.zfill(4) |
| 514 | '12345' |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 515 | >>> 'goofy'.zfill(6) |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 516 | '0goofy' |
| 517 | \end{verbatim} |
| 518 | |
Andrew M. Kuchling | eb91488 | 2002-06-10 15:53:05 +0000 | [diff] [blame] | 519 | \item Dictionaries have a new method, \method{pop(\var{key})}, that |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 520 | returns the value corresponding to \var{key} and removes that |
| 521 | key/value pair from the dictionary. \method{pop()} will raise a |
| 522 | \exception{KeyError} if the requsted key isn't present in the |
| 523 | dictionary: |
| 524 | |
| 525 | \begin{verbatim} |
| 526 | >>> d = {1:2} |
| 527 | >>> d |
| 528 | {1: 2} |
| 529 | >>> d.pop(4) |
| 530 | Traceback (most recent call last): |
Andrew M. Kuchling | 7f147a7 | 2002-06-10 18:58:19 +0000 | [diff] [blame] | 531 | File ``stdin'', line 1, in ? |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 532 | KeyError: 4 |
| 533 | >>> d.pop(1) |
| 534 | 2 |
| 535 | >>> d.pop(1) |
| 536 | Traceback (most recent call last): |
| 537 | File ``<stdin>'', line 1, in ? |
| 538 | KeyError: pop(): dictionary is empty |
| 539 | >>> d |
| 540 | {} |
| 541 | >>> |
| 542 | \end{verbatim} |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 543 | |
Andrew M. Kuchling | 72b58e0 | 2002-05-29 17:30:34 +0000 | [diff] [blame] | 544 | (Contributed by Raymond Hettinger.) |
| 545 | |
Andrew M. Kuchling | a2a206b | 2002-05-24 21:08:58 +0000 | [diff] [blame] | 546 | \item Two new functions in the \module{math} module, |
| 547 | \function{degrees(\var{rads})} and \function{radians(\var{degs})}, |
| 548 | convert between radians and degrees. Other functions in the |
| 549 | \module{math} module such as |
| 550 | \function{math.sin()} and \function{math.cos()} have always required |
| 551 | input values measured in radians. (Contributed by Raymond Hettinger.) |
| 552 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 553 | \item Two new functions, \function{killpg()} and \function{mknod()}, |
| 554 | were added to the \module{posix} module that underlies the \module{os} |
| 555 | module. |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 556 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 557 | \item Two new binary packagers were added to the Distutils. |
| 558 | \code{bdist_pkgtool} builds \file{.pkg} files to use with Solaris |
| 559 | \program{pkgtool}, and \code{bdist_sdux} builds \program{swinstall} |
| 560 | packages for use on HP-UX. (Contributed by Mark Alexander.) |
| 561 | |
| 562 | \item The \module{array} module now supports arrays of Unicode |
| 563 | characters using the \samp{u} format character. Arrays also |
| 564 | now support using the \code{+=} assignment operator to add another array's |
| 565 | contents, and the \code{*=} assignment operator to repeat an array. |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 566 | (Contributed by Jason Orendorff.) |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 567 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 568 | \item The \module{grp} module now returns enhanced tuples: |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 569 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 570 | \begin{verbatim} |
| 571 | >>> import grp |
| 572 | >>> g = grp.getgrnam('amk') |
| 573 | >>> g.gr_name, g.gr_gid |
| 574 | ('amk', 500) |
| 575 | \end{verbatim} |
Andrew M. Kuchling | f367651 | 2002-04-15 02:27:55 +0000 | [diff] [blame] | 576 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 577 | \item The \module{readline} module also gained a number of new |
| 578 | functions: \function{get_history_item()}, |
| 579 | \function{get_current_history_length()}, and \function{redisplay()}. |
Andrew M. Kuchling | 8e8af6e | 2002-04-15 14:05:59 +0000 | [diff] [blame] | 580 | |
Andrew M. Kuchling | 2b6edce | 2002-05-27 17:19:46 +0000 | [diff] [blame] | 581 | \item Support for more advanced POSIX signal handling was added |
| 582 | to the \module{signal} module by adding the \function{sigpending}, |
| 583 | \function{sigprocmask} and \function{sigsuspend} functions, where supported |
| 584 | by the platform. These functions make it possible to avoid some previously |
| 585 | unavoidable race conditions. |
Michael W. Hudson | 34f20ea | 2002-05-27 15:08:24 +0000 | [diff] [blame] | 586 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 587 | \end{itemize} |
| 588 | |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 589 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 590 | % ====================================================================== |
| 591 | \section{Build and C API Changes} |
| 592 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 593 | Changes to Python's build process, and to the C API, include: |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 594 | |
| 595 | \begin{itemize} |
| 596 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 597 | \item Python can now optionally be built as a shared library |
| 598 | (\file{libpython2.3.so}) by supplying \longprogramopt{enable-shared} |
Andrew M. Kuchling | fad2f59 | 2002-05-10 21:00:05 +0000 | [diff] [blame] | 599 | when running Python's \file{configure} script. (Contributed by Ondrej |
| 600 | Palkovsky.) |
Andrew M. Kuchling | f4dd65d | 2002-04-01 19:28:09 +0000 | [diff] [blame] | 601 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 602 | \item The \cfunction{PyArg_NoArgs()} macro is now deprecated, and code |
| 603 | that |
| 604 | uses it should be changed to use \code{PyArg_ParseTuple(args, "")} |
| 605 | instead. |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 606 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 607 | \item A new function, \cfunction{PyObject_DelItemString(\var{mapping}, |
| 608 | char *\var{key})} was added |
| 609 | as shorthand for |
| 610 | \code{PyObject_DelItem(\var{mapping}, PyString_New(\var{key})}. |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 611 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 612 | \item The source code for the Expat XML parser is now included with |
| 613 | the Python source, so the \module{pyexpat} module is no longer |
| 614 | dependent on having a system library containing Expat. |
| 615 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 616 | \item File objects now manage their internal string buffer |
| 617 | differently by increasing it exponentially when needed. |
| 618 | This results in the benchmark tests in \file{Lib/test/test_bufio.py} |
| 619 | speeding up from 57 seconds to 1.7 seconds, according to one |
| 620 | measurement. |
| 621 | |
Andrew M. Kuchling | 72b58e0 | 2002-05-29 17:30:34 +0000 | [diff] [blame] | 622 | \item It's now possible to define class and static methods for a C |
| 623 | extension type by setting either the \constant{METH_CLASS} or |
| 624 | \constant{METH_STATIC} flags in a method's \ctype{PyMethodDef} |
| 625 | structure. |
Andrew M. Kuchling | 45afd54 | 2002-04-02 14:25:25 +0000 | [diff] [blame] | 626 | |
Andrew M. Kuchling | 821013e | 2002-05-06 17:46:39 +0000 | [diff] [blame] | 627 | \end{itemize} |
| 628 | |
| 629 | \subsection{Port-Specific Changes} |
| 630 | |
Andrew M. Kuchling | 187b1d8 | 2002-05-29 19:20:57 +0000 | [diff] [blame] | 631 | Support for a port to IBM's OS/2 using the EMX runtime environment was |
| 632 | merged into the main Python source tree. EMX is a POSIX emulation |
| 633 | layer over the OS/2 system APIs. The Python port for EMX tries to |
| 634 | support all the POSIX-like capability exposed by the EMX runtime, and |
| 635 | mostly succeeds; \function{fork()} and \function{fcntl()} are |
| 636 | restricted by the limitations of the underlying emulation layer. The |
| 637 | standard OS/2 port, which uses IBM's Visual Age compiler, also gained |
| 638 | support for case-sensitive import semantics as part of the integration |
| 639 | of the EMX port into CVS. (Contributed by Andrew MacIntyre.) |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 640 | |
Andrew M. Kuchling | 72b58e0 | 2002-05-29 17:30:34 +0000 | [diff] [blame] | 641 | On MacOS, most toolbox modules have been weaklinked to improve |
| 642 | backward compatibility. This means that modules will no longer fail |
| 643 | to load if a single routine is missing on the curent OS version. |
Andrew M. Kuchling | 187b1d8 | 2002-05-29 19:20:57 +0000 | [diff] [blame] | 644 | Instead calling the missing routine will raise an exception. |
| 645 | (Contributed by Jack Jansen.) |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 646 | |
Andrew M. Kuchling | 187b1d8 | 2002-05-29 19:20:57 +0000 | [diff] [blame] | 647 | The RPM spec files, found in the \file{Misc/RPM/} directory in the |
| 648 | Python source distribution, were updated for 2.3. (Contributed by |
| 649 | Sean Reifschneider.) |
Fred Drake | 03e1031 | 2002-03-26 19:17:43 +0000 | [diff] [blame] | 650 | |
| 651 | |
| 652 | %====================================================================== |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 653 | \section{Other Changes and Fixes} |
| 654 | |
| 655 | Finally, there are various miscellaneous fixes: |
| 656 | |
| 657 | \begin{itemize} |
| 658 | |
| 659 | \item The tools used to build the documentation now work under Cygwin |
| 660 | as well as \UNIX. |
| 661 | |
| 662 | \end{itemize} |
| 663 | |
Andrew M. Kuchling | 187b1d8 | 2002-05-29 19:20:57 +0000 | [diff] [blame] | 664 | |
Andrew M. Kuchling | 517109b | 2002-05-07 21:01:16 +0000 | [diff] [blame] | 665 | %====================================================================== |
Fred Drake | 03e1031 | 2002-03-26 19:17:43 +0000 | [diff] [blame] | 666 | \section{Acknowledgements \label{acks}} |
| 667 | |
Andrew M. Kuchling | 03594bb | 2002-03-27 02:29:48 +0000 | [diff] [blame] | 668 | The author would like to thank the following people for offering |
| 669 | suggestions, corrections and assistance with various drafts of this |
Andrew M. Kuchling | 7f147a7 | 2002-06-10 18:58:19 +0000 | [diff] [blame] | 670 | article: Michael Chermside, Scott David Daniels, Fred~L. Drake, Jr., |
| 671 | Detlef Lannert, Andrew MacIntyre. |
Fred Drake | 03e1031 | 2002-03-26 19:17:43 +0000 | [diff] [blame] | 672 | |
| 673 | \end{document} |