blob: 7796e9e1e42dd9a3f28973729438a89f460a4c9b [file] [log] [blame]
Raymond Hettingerf3501602003-09-08 19:01:04 +00001\chapter{Extending Python with \C{} or \Cpp{} \label{intro}}
Fred Drakecc8f44b2001-08-20 19:30:29 +00002
3
4It is quite easy to add new built-in modules to Python, if you know
5how to program in C. Such \dfn{extension modules} can do two things
6that can't be done directly in Python: they can implement new built-in
7object types, and they can call C library functions and system calls.
8
9To support extensions, the Python API (Application Programmers
10Interface) defines a set of functions, macros and variables that
11provide access to most aspects of the Python run-time system. The
12Python API is incorporated in a C source file by including the header
13\code{"Python.h"}.
14
15The compilation of an extension module depends on its intended use as
16well as on your system setup; details are given in later chapters.
17
18
19\section{A Simple Example
20 \label{simpleExample}}
21
22Let's create an extension module called \samp{spam} (the favorite food
23of Monty Python fans...) and let's say we want to create a Python
24interface to the C library function \cfunction{system()}.\footnote{An
25interface for this function already exists in the standard module
Raymond Hettinger68804312005-01-01 00:28:46 +000026\module{os} --- it was chosen as a simple and straightforward example.}
Fred Drakecc8f44b2001-08-20 19:30:29 +000027This function takes a null-terminated character string as argument and
28returns an integer. We want this function to be callable from Python
29as follows:
30
31\begin{verbatim}
32>>> import spam
33>>> status = spam.system("ls -l")
34\end{verbatim}
35
36Begin by creating a file \file{spammodule.c}. (Historically, if a
37module is called \samp{spam}, the C file containing its implementation
38is called \file{spammodule.c}; if the module name is very long, like
39\samp{spammify}, the module name can be just \file{spammify.c}.)
40
41The first line of our file can be:
42
43\begin{verbatim}
44#include <Python.h>
45\end{verbatim}
46
47which pulls in the Python API (you can add a comment describing the
48purpose of the module and a copyright notice if you like).
Fred Drake34c43202004-03-31 07:45:46 +000049
50\begin{notice}[warning]
51 Since Python may define some pre-processor definitions which affect
52 the standard headers on some systems, you \emph{must} include
53 \file{Python.h} before any standard headers are included.
54\end{notice}
Fred Drakecc8f44b2001-08-20 19:30:29 +000055
Fred Drake396ca572001-09-06 16:30:30 +000056All user-visible symbols defined by \file{Python.h} have a prefix of
Fred Drakecc8f44b2001-08-20 19:30:29 +000057\samp{Py} or \samp{PY}, except those defined in standard header files.
58For convenience, and since they are used extensively by the Python
59interpreter, \code{"Python.h"} includes a few standard header files:
60\code{<stdio.h>}, \code{<string.h>}, \code{<errno.h>}, and
61\code{<stdlib.h>}. If the latter header file does not exist on your
62system, it declares the functions \cfunction{malloc()},
63\cfunction{free()} and \cfunction{realloc()} directly.
64
65The next thing we add to our module file is the C function that will
66be called when the Python expression \samp{spam.system(\var{string})}
67is evaluated (we'll see shortly how it ends up being called):
68
69\begin{verbatim}
70static PyObject *
Fred Drake723f94b2002-06-22 01:42:00 +000071spam_system(PyObject *self, PyObject *args)
Fred Drakecc8f44b2001-08-20 19:30:29 +000072{
Brett Cannon289e4cb2004-06-29 03:48:23 +000073 const char *command;
Fred Drakecc8f44b2001-08-20 19:30:29 +000074 int sts;
75
76 if (!PyArg_ParseTuple(args, "s", &command))
77 return NULL;
78 sts = system(command);
79 return Py_BuildValue("i", sts);
80}
81\end{verbatim}
82
83There is a straightforward translation from the argument list in
84Python (for example, the single expression \code{"ls -l"}) to the
85arguments passed to the C function. The C function always has two
86arguments, conventionally named \var{self} and \var{args}.
87
88The \var{self} argument is only used when the C function implements a
89built-in method, not a function. In the example, \var{self} will
90always be a \NULL{} pointer, since we are defining a function, not a
91method. (This is done so that the interpreter doesn't have to
92understand two different types of C functions.)
93
94The \var{args} argument will be a pointer to a Python tuple object
95containing the arguments. Each item of the tuple corresponds to an
96argument in the call's argument list. The arguments are Python
97objects --- in order to do anything with them in our C function we have
98to convert them to C values. The function \cfunction{PyArg_ParseTuple()}
99in the Python API checks the argument types and converts them to C
100values. It uses a template string to determine the required types of
101the arguments as well as the types of the C variables into which to
102store the converted values. More about this later.
103
104\cfunction{PyArg_ParseTuple()} returns true (nonzero) if all arguments have
105the right type and its components have been stored in the variables
106whose addresses are passed. It returns false (zero) if an invalid
107argument list was passed. In the latter case it also raises an
108appropriate exception so the calling function can return
109\NULL{} immediately (as we saw in the example).
110
111
112\section{Intermezzo: Errors and Exceptions
113 \label{errors}}
114
115An important convention throughout the Python interpreter is the
116following: when a function fails, it should set an exception condition
117and return an error value (usually a \NULL{} pointer). Exceptions
118are stored in a static global variable inside the interpreter; if this
119variable is \NULL{} no exception has occurred. A second global
120variable stores the ``associated value'' of the exception (the second
121argument to \keyword{raise}). A third variable contains the stack
122traceback in case the error originated in Python code. These three
Neal Norwitzac3625f2006-03-17 05:49:33 +0000123variables are the C equivalents of the result in Python of
124\method{sys.exc_info()} (see the section on module \module{sys} in the
Fred Drakecc8f44b2001-08-20 19:30:29 +0000125\citetitle[../lib/lib.html]{Python Library Reference}). It is
126important to know about them to understand how errors are passed
127around.
128
129The Python API defines a number of functions to set various types of
130exceptions.
131
132The most common one is \cfunction{PyErr_SetString()}. Its arguments
133are an exception object and a C string. The exception object is
134usually a predefined object like \cdata{PyExc_ZeroDivisionError}. The
135C string indicates the cause of the error and is converted to a
136Python string object and stored as the ``associated value'' of the
137exception.
138
139Another useful function is \cfunction{PyErr_SetFromErrno()}, which only
140takes an exception argument and constructs the associated value by
141inspection of the global variable \cdata{errno}. The most
142general function is \cfunction{PyErr_SetObject()}, which takes two object
143arguments, the exception and its associated value. You don't need to
144\cfunction{Py_INCREF()} the objects passed to any of these functions.
145
146You can test non-destructively whether an exception has been set with
147\cfunction{PyErr_Occurred()}. This returns the current exception object,
148or \NULL{} if no exception has occurred. You normally don't need
149to call \cfunction{PyErr_Occurred()} to see whether an error occurred in a
150function call, since you should be able to tell from the return value.
151
152When a function \var{f} that calls another function \var{g} detects
153that the latter fails, \var{f} should itself return an error value
154(usually \NULL{} or \code{-1}). It should \emph{not} call one of the
155\cfunction{PyErr_*()} functions --- one has already been called by \var{g}.
156\var{f}'s caller is then supposed to also return an error indication
157to \emph{its} caller, again \emph{without} calling \cfunction{PyErr_*()},
158and so on --- the most detailed cause of the error was already
159reported by the function that first detected it. Once the error
160reaches the Python interpreter's main loop, this aborts the currently
161executing Python code and tries to find an exception handler specified
162by the Python programmer.
163
164(There are situations where a module can actually give a more detailed
165error message by calling another \cfunction{PyErr_*()} function, and in
166such cases it is fine to do so. As a general rule, however, this is
167not necessary, and can cause information about the cause of the error
168to be lost: most operations can fail for a variety of reasons.)
169
170To ignore an exception set by a function call that failed, the exception
171condition must be cleared explicitly by calling \cfunction{PyErr_Clear()}.
172The only time C code should call \cfunction{PyErr_Clear()} is if it doesn't
173want to pass the error on to the interpreter but wants to handle it
174completely by itself (possibly by trying something else, or pretending
175nothing went wrong).
176
177Every failing \cfunction{malloc()} call must be turned into an
178exception --- the direct caller of \cfunction{malloc()} (or
179\cfunction{realloc()}) must call \cfunction{PyErr_NoMemory()} and
180return a failure indicator itself. All the object-creating functions
181(for example, \cfunction{PyInt_FromLong()}) already do this, so this
182note is only relevant to those who call \cfunction{malloc()} directly.
183
184Also note that, with the important exception of
185\cfunction{PyArg_ParseTuple()} and friends, functions that return an
186integer status usually return a positive value or zero for success and
187\code{-1} for failure, like \UNIX{} system calls.
188
189Finally, be careful to clean up garbage (by making
190\cfunction{Py_XDECREF()} or \cfunction{Py_DECREF()} calls for objects
191you have already created) when you return an error indicator!
192
193The choice of which exception to raise is entirely yours. There are
194predeclared C objects corresponding to all built-in Python exceptions,
195such as \cdata{PyExc_ZeroDivisionError}, which you can use directly.
196Of course, you should choose exceptions wisely --- don't use
197\cdata{PyExc_TypeError} to mean that a file couldn't be opened (that
198should probably be \cdata{PyExc_IOError}). If something's wrong with
199the argument list, the \cfunction{PyArg_ParseTuple()} function usually
200raises \cdata{PyExc_TypeError}. If you have an argument whose value
201must be in a particular range or must satisfy other conditions,
202\cdata{PyExc_ValueError} is appropriate.
203
204You can also define a new exception that is unique to your module.
205For this, you usually declare a static object variable at the
206beginning of your file:
207
208\begin{verbatim}
209static PyObject *SpamError;
210\end{verbatim}
211
212and initialize it in your module's initialization function
213(\cfunction{initspam()}) with an exception object (leaving out
214the error checking for now):
215
216\begin{verbatim}
Mark Hammond8235ea12002-07-19 06:55:41 +0000217PyMODINIT_FUNC
Fred Drakeef6373a2001-11-17 06:50:42 +0000218initspam(void)
Fred Drakecc8f44b2001-08-20 19:30:29 +0000219{
Fred Drake63e40a52002-04-12 19:08:31 +0000220 PyObject *m;
Fred Drakecc8f44b2001-08-20 19:30:29 +0000221
222 m = Py_InitModule("spam", SpamMethods);
Thomas Wouters89f507f2006-12-13 04:49:30 +0000223 if (m == NULL)
224 return;
Fred Drake63e40a52002-04-12 19:08:31 +0000225
Fred Drakecc8f44b2001-08-20 19:30:29 +0000226 SpamError = PyErr_NewException("spam.error", NULL, NULL);
Fred Drake63e40a52002-04-12 19:08:31 +0000227 Py_INCREF(SpamError);
228 PyModule_AddObject(m, "error", SpamError);
Fred Drakecc8f44b2001-08-20 19:30:29 +0000229}
230\end{verbatim}
231
232Note that the Python name for the exception object is
233\exception{spam.error}. The \cfunction{PyErr_NewException()} function
234may create a class with the base class being \exception{Exception}
235(unless another class is passed in instead of \NULL), described in the
236\citetitle[../lib/lib.html]{Python Library Reference} under ``Built-in
237Exceptions.''
238
239Note also that the \cdata{SpamError} variable retains a reference to
240the newly created exception class; this is intentional! Since the
241exception could be removed from the module by external code, an owned
242reference to the class is needed to ensure that it will not be
243discarded, causing \cdata{SpamError} to become a dangling pointer.
244Should it become a dangling pointer, C code which raises the exception
245could cause a core dump or other unintended side effects.
246
Brett Cannon555a9642004-06-26 23:10:32 +0000247We discuss the use of PyMODINIT_FUNC as a function return type later in this
248sample.
Fred Drakecc8f44b2001-08-20 19:30:29 +0000249
250\section{Back to the Example
251 \label{backToExample}}
252
253Going back to our example function, you should now be able to
254understand this statement:
255
256\begin{verbatim}
257 if (!PyArg_ParseTuple(args, "s", &command))
258 return NULL;
259\end{verbatim}
260
261It returns \NULL{} (the error indicator for functions returning
262object pointers) if an error is detected in the argument list, relying
263on the exception set by \cfunction{PyArg_ParseTuple()}. Otherwise the
264string value of the argument has been copied to the local variable
265\cdata{command}. This is a pointer assignment and you are not supposed
266to modify the string to which it points (so in Standard C, the variable
267\cdata{command} should properly be declared as \samp{const char
268*command}).
269
270The next statement is a call to the \UNIX{} function
271\cfunction{system()}, passing it the string we just got from
272\cfunction{PyArg_ParseTuple()}:
273
274\begin{verbatim}
275 sts = system(command);
276\end{verbatim}
277
278Our \function{spam.system()} function must return the value of
279\cdata{sts} as a Python object. This is done using the function
280\cfunction{Py_BuildValue()}, which is something like the inverse of
281\cfunction{PyArg_ParseTuple()}: it takes a format string and an
282arbitrary number of C values, and returns a new Python object.
283More info on \cfunction{Py_BuildValue()} is given later.
284
285\begin{verbatim}
286 return Py_BuildValue("i", sts);
287\end{verbatim}
288
289In this case, it will return an integer object. (Yes, even integers
290are objects on the heap in Python!)
291
292If you have a C function that returns no useful argument (a function
293returning \ctype{void}), the corresponding Python function must return
Brett Cannon634893d2004-06-27 04:28:00 +0000294\code{None}. You need this idiom to do so (which is implemented by the
295\csimplemacro{Py_RETURN_NONE} macro):
Fred Drakecc8f44b2001-08-20 19:30:29 +0000296
297\begin{verbatim}
298 Py_INCREF(Py_None);
299 return Py_None;
300\end{verbatim}
301
302\cdata{Py_None} is the C name for the special Python object
303\code{None}. It is a genuine Python object rather than a \NULL{}
304pointer, which means ``error'' in most contexts, as we have seen.
305
306
307\section{The Module's Method Table and Initialization Function
308 \label{methodTable}}
309
310I promised to show how \cfunction{spam_system()} is called from Python
311programs. First, we need to list its name and address in a ``method
312table'':
313
314\begin{verbatim}
315static PyMethodDef SpamMethods[] = {
316 ...
Fred Drakeef6373a2001-11-17 06:50:42 +0000317 {"system", spam_system, METH_VARARGS,
318 "Execute a shell command."},
Fred Drakecc8f44b2001-08-20 19:30:29 +0000319 ...
Fred Drakeef6373a2001-11-17 06:50:42 +0000320 {NULL, NULL, 0, NULL} /* Sentinel */
Fred Drakecc8f44b2001-08-20 19:30:29 +0000321};
322\end{verbatim}
323
324Note the third entry (\samp{METH_VARARGS}). This is a flag telling
325the interpreter the calling convention to be used for the C
326function. It should normally always be \samp{METH_VARARGS} or
327\samp{METH_VARARGS | METH_KEYWORDS}; a value of \code{0} means that an
328obsolete variant of \cfunction{PyArg_ParseTuple()} is used.
329
330When using only \samp{METH_VARARGS}, the function should expect
331the Python-level parameters to be passed in as a tuple acceptable for
332parsing via \cfunction{PyArg_ParseTuple()}; more information on this
333function is provided below.
334
335The \constant{METH_KEYWORDS} bit may be set in the third field if
336keyword arguments should be passed to the function. In this case, the
337C function should accept a third \samp{PyObject *} parameter which
338will be a dictionary of keywords. Use
339\cfunction{PyArg_ParseTupleAndKeywords()} to parse the arguments to
340such a function.
341
342The method table must be passed to the interpreter in the module's
343initialization function. The initialization function must be named
344\cfunction{init\var{name}()}, where \var{name} is the name of the
345module, and should be the only non-\keyword{static} item defined in
346the module file:
347
348\begin{verbatim}
Mark Hammond8235ea12002-07-19 06:55:41 +0000349PyMODINIT_FUNC
Fred Drakeef6373a2001-11-17 06:50:42 +0000350initspam(void)
Fred Drakecc8f44b2001-08-20 19:30:29 +0000351{
352 (void) Py_InitModule("spam", SpamMethods);
353}
354\end{verbatim}
355
Mark Hammond8235ea12002-07-19 06:55:41 +0000356Note that PyMODINIT_FUNC declares the function as \code{void} return type,
357declares any special linkage declarations required by the platform, and for
Raymond Hettingerf3501602003-09-08 19:01:04 +0000358\Cpp{} declares the function as \code{extern "C"}.
Fred Drakecc8f44b2001-08-20 19:30:29 +0000359
360When the Python program imports module \module{spam} for the first
361time, \cfunction{initspam()} is called. (See below for comments about
362embedding Python.) It calls
363\cfunction{Py_InitModule()}, which creates a ``module object'' (which
364is inserted in the dictionary \code{sys.modules} under the key
365\code{"spam"}), and inserts built-in function objects into the newly
366created module based upon the table (an array of \ctype{PyMethodDef}
367structures) that was passed as its second argument.
368\cfunction{Py_InitModule()} returns a pointer to the module object
Thomas Wouters89f507f2006-12-13 04:49:30 +0000369that it creates (which is unused here). It may abort with a fatal error
370for certain errors, or return \NULL{} if the module could not be
371initialized satisfactorily.
Fred Drakecc8f44b2001-08-20 19:30:29 +0000372
373When embedding Python, the \cfunction{initspam()} function is not
374called automatically unless there's an entry in the
375\cdata{_PyImport_Inittab} table. The easiest way to handle this is to
376statically initialize your statically-linked modules by directly
377calling \cfunction{initspam()} after the call to
Brett Cannon7706c2d2005-02-13 22:50:04 +0000378\cfunction{Py_Initialize()}:
Fred Drakecc8f44b2001-08-20 19:30:29 +0000379
380\begin{verbatim}
Fred Drake723f94b2002-06-22 01:42:00 +0000381int
382main(int argc, char *argv[])
Fred Drakecc8f44b2001-08-20 19:30:29 +0000383{
384 /* Pass argv[0] to the Python interpreter */
385 Py_SetProgramName(argv[0]);
386
387 /* Initialize the Python interpreter. Required. */
388 Py_Initialize();
389
390 /* Add a static module */
391 initspam();
392\end{verbatim}
393
394An example may be found in the file \file{Demo/embed/demo.c} in the
395Python source distribution.
396
Fred Drake0aa811c2001-10-20 04:24:09 +0000397\note{Removing entries from \code{sys.modules} or importing
Fred Drakecc8f44b2001-08-20 19:30:29 +0000398compiled modules into multiple interpreters within a process (or
399following a \cfunction{fork()} without an intervening
400\cfunction{exec()}) can create problems for some extension modules.
401Extension module authors should exercise caution when initializing
Guido van Rossume7ba4952007-06-06 23:52:48 +0000402internal data structures.}
Fred Drakecc8f44b2001-08-20 19:30:29 +0000403
404A more substantial example module is included in the Python source
405distribution as \file{Modules/xxmodule.c}. This file may be used as a
406template or simply read as an example. The \program{modulator.py}
407script included in the source distribution or Windows install provides
408a simple graphical user interface for declaring the functions and
409objects which a module should implement, and can generate a template
410which can be filled in. The script lives in the
411\file{Tools/modulator/} directory; see the \file{README} file there
412for more information.
413
414
415\section{Compilation and Linkage
416 \label{compilation}}
417
418There are two more things to do before you can use your new extension:
419compiling and linking it with the Python system. If you use dynamic
Fred Drake4e765552002-05-16 13:48:14 +0000420loading, the details may depend on the style of dynamic loading your
421system uses; see the chapters about building extension modules
422(chapter \ref{building}) and additional information that pertains only
423to building on Windows (chapter \ref{building-on-windows}) for more
424information about this.
Fred Drakecc8f44b2001-08-20 19:30:29 +0000425
426If you can't use dynamic loading, or if you want to make your module a
427permanent part of the Python interpreter, you will have to change the
428configuration setup and rebuild the interpreter. Luckily, this is
Fred Drake4e765552002-05-16 13:48:14 +0000429very simple on \UNIX: just place your file (\file{spammodule.c} for
430example) in the \file{Modules/} directory of an unpacked source
431distribution, add a line to the file \file{Modules/Setup.local}
432describing your file:
Fred Drakecc8f44b2001-08-20 19:30:29 +0000433
434\begin{verbatim}
435spam spammodule.o
436\end{verbatim}
437
438and rebuild the interpreter by running \program{make} in the toplevel
439directory. You can also run \program{make} in the \file{Modules/}
440subdirectory, but then you must first rebuild \file{Makefile}
441there by running `\program{make} Makefile'. (This is necessary each
442time you change the \file{Setup} file.)
443
444If your module requires additional libraries to link with, these can
445be listed on the line in the configuration file as well, for instance:
446
447\begin{verbatim}
448spam spammodule.o -lX11
449\end{verbatim}
450
451\section{Calling Python Functions from C
452 \label{callingPython}}
453
454So far we have concentrated on making C functions callable from
455Python. The reverse is also useful: calling Python functions from C.
456This is especially the case for libraries that support so-called
457``callback'' functions. If a C interface makes use of callbacks, the
458equivalent Python often needs to provide a callback mechanism to the
459Python programmer; the implementation will require calling the Python
460callback functions from a C callback. Other uses are also imaginable.
461
462Fortunately, the Python interpreter is easily called recursively, and
463there is a standard interface to call a Python function. (I won't
464dwell on how to call the Python parser with a particular string as
465input --- if you're interested, have a look at the implementation of
466the \programopt{-c} command line option in \file{Python/pythonmain.c}
467from the Python source code.)
468
469Calling a Python function is easy. First, the Python program must
470somehow pass you the Python function object. You should provide a
471function (or some other interface) to do this. When this function is
472called, save a pointer to the Python function object (be careful to
473\cfunction{Py_INCREF()} it!) in a global variable --- or wherever you
474see fit. For example, the following function might be part of a module
475definition:
476
477\begin{verbatim}
478static PyObject *my_callback = NULL;
479
480static PyObject *
Fred Drake723f94b2002-06-22 01:42:00 +0000481my_set_callback(PyObject *dummy, PyObject *args)
Fred Drakecc8f44b2001-08-20 19:30:29 +0000482{
483 PyObject *result = NULL;
484 PyObject *temp;
485
486 if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {
487 if (!PyCallable_Check(temp)) {
488 PyErr_SetString(PyExc_TypeError, "parameter must be callable");
489 return NULL;
490 }
491 Py_XINCREF(temp); /* Add a reference to new callback */
492 Py_XDECREF(my_callback); /* Dispose of previous callback */
493 my_callback = temp; /* Remember new callback */
494 /* Boilerplate to return "None" */
495 Py_INCREF(Py_None);
496 result = Py_None;
497 }
498 return result;
499}
500\end{verbatim}
501
502This function must be registered with the interpreter using the
503\constant{METH_VARARGS} flag; this is described in section
504\ref{methodTable}, ``The Module's Method Table and Initialization
505Function.'' The \cfunction{PyArg_ParseTuple()} function and its
Fred Drakec37b65e2001-11-28 07:26:15 +0000506arguments are documented in section~\ref{parseTuple}, ``Extracting
Fred Drakecc8f44b2001-08-20 19:30:29 +0000507Parameters in Extension Functions.''
508
509The macros \cfunction{Py_XINCREF()} and \cfunction{Py_XDECREF()}
510increment/decrement the reference count of an object and are safe in
511the presence of \NULL{} pointers (but note that \var{temp} will not be
Fred Drakec37b65e2001-11-28 07:26:15 +0000512\NULL{} in this context). More info on them in
513section~\ref{refcounts}, ``Reference Counts.''
Fred Drakecc8f44b2001-08-20 19:30:29 +0000514
515Later, when it is time to call the function, you call the C function
Fred Drake99181ac2001-11-29 05:02:34 +0000516\cfunction{PyEval_CallObject()}.\ttindex{PyEval_CallObject()} This
517function has two arguments, both pointers to arbitrary Python objects:
518the Python function, and the argument list. The argument list must
519always be a tuple object, whose length is the number of arguments. To
520call the Python function with no arguments, pass an empty tuple; to
521call it with one argument, pass a singleton tuple.
522\cfunction{Py_BuildValue()} returns a tuple when its format string
523consists of zero or more format codes between parentheses. For
524example:
Fred Drakecc8f44b2001-08-20 19:30:29 +0000525
526\begin{verbatim}
527 int arg;
528 PyObject *arglist;
529 PyObject *result;
530 ...
531 arg = 123;
532 ...
533 /* Time to call the callback */
534 arglist = Py_BuildValue("(i)", arg);
535 result = PyEval_CallObject(my_callback, arglist);
536 Py_DECREF(arglist);
537\end{verbatim}
538
539\cfunction{PyEval_CallObject()} returns a Python object pointer: this is
540the return value of the Python function. \cfunction{PyEval_CallObject()} is
541``reference-count-neutral'' with respect to its arguments. In the
542example a new tuple was created to serve as the argument list, which
543is \cfunction{Py_DECREF()}-ed immediately after the call.
544
545The return value of \cfunction{PyEval_CallObject()} is ``new'': either it
546is a brand new object, or it is an existing object whose reference
547count has been incremented. So, unless you want to save it in a
548global variable, you should somehow \cfunction{Py_DECREF()} the result,
549even (especially!) if you are not interested in its value.
550
551Before you do this, however, it is important to check that the return
Fred Drakec37b65e2001-11-28 07:26:15 +0000552value isn't \NULL. If it is, the Python function terminated by
Fred Drakecc8f44b2001-08-20 19:30:29 +0000553raising an exception. If the C code that called
554\cfunction{PyEval_CallObject()} is called from Python, it should now
555return an error indication to its Python caller, so the interpreter
556can print a stack trace, or the calling Python code can handle the
557exception. If this is not possible or desirable, the exception should
558be cleared by calling \cfunction{PyErr_Clear()}. For example:
559
560\begin{verbatim}
561 if (result == NULL)
562 return NULL; /* Pass error back */
563 ...use result...
564 Py_DECREF(result);
565\end{verbatim}
566
567Depending on the desired interface to the Python callback function,
568you may also have to provide an argument list to
569\cfunction{PyEval_CallObject()}. In some cases the argument list is
570also provided by the Python program, through the same interface that
571specified the callback function. It can then be saved and used in the
572same manner as the function object. In other cases, you may have to
573construct a new tuple to pass as the argument list. The simplest way
574to do this is to call \cfunction{Py_BuildValue()}. For example, if
575you want to pass an integral event code, you might use the following
576code:
577
578\begin{verbatim}
579 PyObject *arglist;
580 ...
581 arglist = Py_BuildValue("(l)", eventcode);
582 result = PyEval_CallObject(my_callback, arglist);
583 Py_DECREF(arglist);
584 if (result == NULL)
585 return NULL; /* Pass error back */
586 /* Here maybe use the result */
587 Py_DECREF(result);
588\end{verbatim}
589
590Note the placement of \samp{Py_DECREF(arglist)} immediately after the
591call, before the error check! Also note that strictly spoken this
592code is not complete: \cfunction{Py_BuildValue()} may run out of
593memory, and this should be checked.
594
595
596\section{Extracting Parameters in Extension Functions
597 \label{parseTuple}}
598
Fred Drakee9fba912002-03-28 22:36:56 +0000599\ttindex{PyArg_ParseTuple()}
600
Fred Drakecc8f44b2001-08-20 19:30:29 +0000601The \cfunction{PyArg_ParseTuple()} function is declared as follows:
602
603\begin{verbatim}
604int PyArg_ParseTuple(PyObject *arg, char *format, ...);
605\end{verbatim}
606
607The \var{arg} argument must be a tuple object containing an argument
608list passed from Python to a C function. The \var{format} argument
Fred Drake68304cc2002-04-05 23:01:14 +0000609must be a format string, whose syntax is explained in
610``\ulink{Parsing arguments and building
611values}{../api/arg-parsing.html}'' in the
612\citetitle[../api/api.html]{Python/C API Reference Manual}. The
Fred Drakecc8f44b2001-08-20 19:30:29 +0000613remaining arguments must be addresses of variables whose type is
Fred Drake68304cc2002-04-05 23:01:14 +0000614determined by the format string.
Fred Drakecc8f44b2001-08-20 19:30:29 +0000615
616Note that while \cfunction{PyArg_ParseTuple()} checks that the Python
617arguments have the required types, it cannot check the validity of the
618addresses of C variables passed to the call: if you make mistakes
619there, your code will probably crash or at least overwrite random bits
620in memory. So be careful!
621
Fred Drakecc8f44b2001-08-20 19:30:29 +0000622Note that any Python object references which are provided to the
623caller are \emph{borrowed} references; do not decrement their
624reference count!
625
Fred Drakecc8f44b2001-08-20 19:30:29 +0000626Some example calls:
627
628\begin{verbatim}
629 int ok;
630 int i, j;
631 long k, l;
Brett Cannon289e4cb2004-06-29 03:48:23 +0000632 const char *s;
Fred Drakecc8f44b2001-08-20 19:30:29 +0000633 int size;
634
635 ok = PyArg_ParseTuple(args, ""); /* No arguments */
636 /* Python call: f() */
637\end{verbatim}
638
639\begin{verbatim}
640 ok = PyArg_ParseTuple(args, "s", &s); /* A string */
641 /* Possible Python call: f('whoops!') */
642\end{verbatim}
643
644\begin{verbatim}
645 ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */
646 /* Possible Python call: f(1, 2, 'three') */
647\end{verbatim}
648
649\begin{verbatim}
650 ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size);
651 /* A pair of ints and a string, whose size is also returned */
652 /* Possible Python call: f((1, 2), 'three') */
653\end{verbatim}
654
655\begin{verbatim}
656 {
Brett Cannon289e4cb2004-06-29 03:48:23 +0000657 const char *file;
658 const char *mode = "r";
Fred Drakecc8f44b2001-08-20 19:30:29 +0000659 int bufsize = 0;
660 ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize);
661 /* A string, and optionally another string and an integer */
662 /* Possible Python calls:
663 f('spam')
664 f('spam', 'w')
665 f('spam', 'wb', 100000) */
666 }
667\end{verbatim}
668
669\begin{verbatim}
670 {
671 int left, top, right, bottom, h, v;
672 ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)",
673 &left, &top, &right, &bottom, &h, &v);
674 /* A rectangle and a point */
675 /* Possible Python call:
676 f(((0, 0), (400, 300)), (10, 10)) */
677 }
678\end{verbatim}
679
680\begin{verbatim}
681 {
682 Py_complex c;
683 ok = PyArg_ParseTuple(args, "D:myfunction", &c);
684 /* a complex, also providing a function name for errors */
685 /* Possible Python call: myfunction(1+2j) */
686 }
687\end{verbatim}
688
689
690\section{Keyword Parameters for Extension Functions
691 \label{parseTupleAndKeywords}}
692
Fred Drakee9fba912002-03-28 22:36:56 +0000693\ttindex{PyArg_ParseTupleAndKeywords()}
694
Fred Drakecc8f44b2001-08-20 19:30:29 +0000695The \cfunction{PyArg_ParseTupleAndKeywords()} function is declared as
696follows:
697
698\begin{verbatim}
699int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
Fred Drake723f94b2002-06-22 01:42:00 +0000700 char *format, char *kwlist[], ...);
Fred Drakecc8f44b2001-08-20 19:30:29 +0000701\end{verbatim}
702
703The \var{arg} and \var{format} parameters are identical to those of the
704\cfunction{PyArg_ParseTuple()} function. The \var{kwdict} parameter
705is the dictionary of keywords received as the third parameter from the
Fred Drakec37b65e2001-11-28 07:26:15 +0000706Python runtime. The \var{kwlist} parameter is a \NULL-terminated
Fred Drakecc8f44b2001-08-20 19:30:29 +0000707list of strings which identify the parameters; the names are matched
708with the type information from \var{format} from left to right. On
709success, \cfunction{PyArg_ParseTupleAndKeywords()} returns true,
710otherwise it returns false and raises an appropriate exception.
711
Fred Drake0aa811c2001-10-20 04:24:09 +0000712\note{Nested tuples cannot be parsed when using keyword
Fred Drakecc8f44b2001-08-20 19:30:29 +0000713arguments! Keyword parameters passed in which are not present in the
Fred Drake0aa811c2001-10-20 04:24:09 +0000714\var{kwlist} will cause \exception{TypeError} to be raised.}
Fred Drakecc8f44b2001-08-20 19:30:29 +0000715
716Here is an example module which uses keywords, based on an example by
717Geoff Philbrick (\email{philbrick@hks.com}):%
718\index{Philbrick, Geoff}
719
720\begin{verbatim}
Fred Drakecc8f44b2001-08-20 19:30:29 +0000721#include "Python.h"
722
723static PyObject *
Fred Drake723f94b2002-06-22 01:42:00 +0000724keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds)
Fred Drakecc8f44b2001-08-20 19:30:29 +0000725{
726 int voltage;
727 char *state = "a stiff";
728 char *action = "voom";
729 char *type = "Norwegian Blue";
730
731 static char *kwlist[] = {"voltage", "state", "action", "type", NULL};
732
733 if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
734 &voltage, &state, &action, &type))
735 return NULL;
736
737 printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
738 action, voltage);
739 printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);
740
741 Py_INCREF(Py_None);
742
743 return Py_None;
744}
745
746static PyMethodDef keywdarg_methods[] = {
747 /* The cast of the function is necessary since PyCFunction values
748 * only take two PyObject* parameters, and keywdarg_parrot() takes
749 * three.
750 */
Fred Drake31f84832002-03-28 20:19:23 +0000751 {"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS,
Fred Drakeef6373a2001-11-17 06:50:42 +0000752 "Print a lovely skit to standard output."},
753 {NULL, NULL, 0, NULL} /* sentinel */
Fred Drakecc8f44b2001-08-20 19:30:29 +0000754};
Fred Drake31f84832002-03-28 20:19:23 +0000755\end{verbatim}
Fred Drakecc8f44b2001-08-20 19:30:29 +0000756
Fred Drake31f84832002-03-28 20:19:23 +0000757\begin{verbatim}
Fred Drakecc8f44b2001-08-20 19:30:29 +0000758void
Fred Drakeef6373a2001-11-17 06:50:42 +0000759initkeywdarg(void)
Fred Drakecc8f44b2001-08-20 19:30:29 +0000760{
761 /* Create the module and add the functions */
762 Py_InitModule("keywdarg", keywdarg_methods);
763}
764\end{verbatim}
765
766
767\section{Building Arbitrary Values
768 \label{buildValue}}
769
770This function is the counterpart to \cfunction{PyArg_ParseTuple()}. It is
771declared as follows:
772
773\begin{verbatim}
774PyObject *Py_BuildValue(char *format, ...);
775\end{verbatim}
776
777It recognizes a set of format units similar to the ones recognized by
778\cfunction{PyArg_ParseTuple()}, but the arguments (which are input to the
779function, not output) must not be pointers, just values. It returns a
780new Python object, suitable for returning from a C function called
781from Python.
782
783One difference with \cfunction{PyArg_ParseTuple()}: while the latter
784requires its first argument to be a tuple (since Python argument lists
785are always represented as tuples internally),
786\cfunction{Py_BuildValue()} does not always build a tuple. It builds
787a tuple only if its format string contains two or more format units.
788If the format string is empty, it returns \code{None}; if it contains
789exactly one format unit, it returns whatever object is described by
790that format unit. To force it to return a tuple of size 0 or one,
791parenthesize the format string.
792
Fred Drakecc8f44b2001-08-20 19:30:29 +0000793Examples (to the left the call, to the right the resulting Python value):
794
795\begin{verbatim}
796 Py_BuildValue("") None
797 Py_BuildValue("i", 123) 123
798 Py_BuildValue("iii", 123, 456, 789) (123, 456, 789)
799 Py_BuildValue("s", "hello") 'hello'
Walter Dörwald612344f2007-05-04 19:28:21 +0000800 Py_BuildValue("y", "hello") b'hello'
Fred Drakecc8f44b2001-08-20 19:30:29 +0000801 Py_BuildValue("ss", "hello", "world") ('hello', 'world')
802 Py_BuildValue("s#", "hello", 4) 'hell'
Walter Dörwald612344f2007-05-04 19:28:21 +0000803 Py_BuildValue("y#", "hello", 4) b'hell'
Fred Drakecc8f44b2001-08-20 19:30:29 +0000804 Py_BuildValue("()") ()
805 Py_BuildValue("(i)", 123) (123,)
806 Py_BuildValue("(ii)", 123, 456) (123, 456)
807 Py_BuildValue("(i,i)", 123, 456) (123, 456)
808 Py_BuildValue("[i,i]", 123, 456) [123, 456]
809 Py_BuildValue("{s:i,s:i}",
810 "abc", 123, "def", 456) {'abc': 123, 'def': 456}
811 Py_BuildValue("((ii)(ii)) (ii)",
812 1, 2, 3, 4, 5, 6) (((1, 2), (3, 4)), (5, 6))
813\end{verbatim}
814
815
816\section{Reference Counts
817 \label{refcounts}}
818
Fred Drakec37b65e2001-11-28 07:26:15 +0000819In languages like C or \Cpp, the programmer is responsible for
Fred Drakecc8f44b2001-08-20 19:30:29 +0000820dynamic allocation and deallocation of memory on the heap. In C,
821this is done using the functions \cfunction{malloc()} and
Fred Drakec37b65e2001-11-28 07:26:15 +0000822\cfunction{free()}. In \Cpp, the operators \keyword{new} and
Michael W. Hudson241c2e92003-02-06 18:38:11 +0000823\keyword{delete} are used with essentially the same meaning and
Michael W. Hudsonff1f1942003-11-07 11:45:34 +0000824we'll restrict the following discussion to the C case.
Fred Drakecc8f44b2001-08-20 19:30:29 +0000825
826Every block of memory allocated with \cfunction{malloc()} should
827eventually be returned to the pool of available memory by exactly one
828call to \cfunction{free()}. It is important to call
829\cfunction{free()} at the right time. If a block's address is
830forgotten but \cfunction{free()} is not called for it, the memory it
831occupies cannot be reused until the program terminates. This is
832called a \dfn{memory leak}. On the other hand, if a program calls
833\cfunction{free()} for a block and then continues to use the block, it
834creates a conflict with re-use of the block through another
835\cfunction{malloc()} call. This is called \dfn{using freed memory}.
836It has the same bad consequences as referencing uninitialized data ---
837core dumps, wrong results, mysterious crashes.
838
839Common causes of memory leaks are unusual paths through the code. For
840instance, a function may allocate a block of memory, do some
841calculation, and then free the block again. Now a change in the
842requirements for the function may add a test to the calculation that
843detects an error condition and can return prematurely from the
844function. It's easy to forget to free the allocated memory block when
845taking this premature exit, especially when it is added later to the
846code. Such leaks, once introduced, often go undetected for a long
847time: the error exit is taken only in a small fraction of all calls,
848and most modern machines have plenty of virtual memory, so the leak
849only becomes apparent in a long-running process that uses the leaking
850function frequently. Therefore, it's important to prevent leaks from
851happening by having a coding convention or strategy that minimizes
852this kind of errors.
853
854Since Python makes heavy use of \cfunction{malloc()} and
855\cfunction{free()}, it needs a strategy to avoid memory leaks as well
856as the use of freed memory. The chosen method is called
857\dfn{reference counting}. The principle is simple: every object
858contains a counter, which is incremented when a reference to the
859object is stored somewhere, and which is decremented when a reference
860to it is deleted. When the counter reaches zero, the last reference
861to the object has been deleted and the object is freed.
862
863An alternative strategy is called \dfn{automatic garbage collection}.
864(Sometimes, reference counting is also referred to as a garbage
865collection strategy, hence my use of ``automatic'' to distinguish the
866two.) The big advantage of automatic garbage collection is that the
867user doesn't need to call \cfunction{free()} explicitly. (Another claimed
868advantage is an improvement in speed or memory usage --- this is no
869hard fact however.) The disadvantage is that for C, there is no
870truly portable automatic garbage collector, while reference counting
871can be implemented portably (as long as the functions \cfunction{malloc()}
872and \cfunction{free()} are available --- which the C Standard guarantees).
873Maybe some day a sufficiently portable automatic garbage collector
874will be available for C. Until then, we'll have to live with
875reference counts.
876
Fred Drake024e6472001-12-07 17:30:40 +0000877While Python uses the traditional reference counting implementation,
878it also offers a cycle detector that works to detect reference
879cycles. This allows applications to not worry about creating direct
880or indirect circular references; these are the weakness of garbage
881collection implemented using only reference counting. Reference
882cycles consist of objects which contain (possibly indirect) references
Tim Peters874c4f02001-12-07 17:51:41 +0000883to themselves, so that each object in the cycle has a reference count
Fred Drake024e6472001-12-07 17:30:40 +0000884which is non-zero. Typical reference counting implementations are not
Tim Peters874c4f02001-12-07 17:51:41 +0000885able to reclaim the memory belonging to any objects in a reference
Fred Drake024e6472001-12-07 17:30:40 +0000886cycle, or referenced from the objects in the cycle, even though there
887are no further references to the cycle itself.
888
889The cycle detector is able to detect garbage cycles and can reclaim
890them so long as there are no finalizers implemented in Python
891(\method{__del__()} methods). When there are such finalizers, the
892detector exposes the cycles through the \ulink{\module{gc}
Guido van Rossum44b3f762001-12-07 17:57:56 +0000893module}{../lib/module-gc.html} (specifically, the \code{garbage}
894variable in that module). The \module{gc} module also exposes a way
895to run the detector (the \function{collect()} function), as well as
Fred Drake024e6472001-12-07 17:30:40 +0000896configuration interfaces and the ability to disable the detector at
897runtime. The cycle detector is considered an optional component;
Guido van Rossum44b3f762001-12-07 17:57:56 +0000898though it is included by default, it can be disabled at build time
Fred Drake024e6472001-12-07 17:30:40 +0000899using the \longprogramopt{without-cycle-gc} option to the
900\program{configure} script on \UNIX{} platforms (including Mac OS X)
901or by removing the definition of \code{WITH_CYCLE_GC} in the
902\file{pyconfig.h} header on other platforms. If the cycle detector is
903disabled in this way, the \module{gc} module will not be available.
904
905
Fred Drakecc8f44b2001-08-20 19:30:29 +0000906\subsection{Reference Counting in Python
907 \label{refcountsInPython}}
908
909There are two macros, \code{Py_INCREF(x)} and \code{Py_DECREF(x)},
910which handle the incrementing and decrementing of the reference count.
911\cfunction{Py_DECREF()} also frees the object when the count reaches zero.
912For flexibility, it doesn't call \cfunction{free()} directly --- rather, it
913makes a call through a function pointer in the object's \dfn{type
914object}. For this purpose (and others), every object also contains a
915pointer to its type object.
916
917The big question now remains: when to use \code{Py_INCREF(x)} and
918\code{Py_DECREF(x)}? Let's first introduce some terms. Nobody
919``owns'' an object; however, you can \dfn{own a reference} to an
920object. An object's reference count is now defined as the number of
921owned references to it. The owner of a reference is responsible for
922calling \cfunction{Py_DECREF()} when the reference is no longer
923needed. Ownership of a reference can be transferred. There are three
924ways to dispose of an owned reference: pass it on, store it, or call
925\cfunction{Py_DECREF()}. Forgetting to dispose of an owned reference
926creates a memory leak.
927
928It is also possible to \dfn{borrow}\footnote{The metaphor of
929``borrowing'' a reference is not completely correct: the owner still
930has a copy of the reference.} a reference to an object. The borrower
931of a reference should not call \cfunction{Py_DECREF()}. The borrower must
932not hold on to the object longer than the owner from which it was
933borrowed. Using a borrowed reference after the owner has disposed of
934it risks using freed memory and should be avoided
935completely.\footnote{Checking that the reference count is at least 1
936\strong{does not work} --- the reference count itself could be in
937freed memory and may thus be reused for another object!}
938
939The advantage of borrowing over owning a reference is that you don't
940need to take care of disposing of the reference on all possible paths
941through the code --- in other words, with a borrowed reference you
942don't run the risk of leaking when a premature exit is taken. The
943disadvantage of borrowing over leaking is that there are some subtle
944situations where in seemingly correct code a borrowed reference can be
945used after the owner from which it was borrowed has in fact disposed
946of it.
947
948A borrowed reference can be changed into an owned reference by calling
949\cfunction{Py_INCREF()}. This does not affect the status of the owner from
950which the reference was borrowed --- it creates a new owned reference,
951and gives full owner responsibilities (the new owner must
952dispose of the reference properly, as well as the previous owner).
953
954
955\subsection{Ownership Rules
956 \label{ownershipRules}}
957
958Whenever an object reference is passed into or out of a function, it
959is part of the function's interface specification whether ownership is
960transferred with the reference or not.
961
962Most functions that return a reference to an object pass on ownership
963with the reference. In particular, all functions whose function it is
964to create a new object, such as \cfunction{PyInt_FromLong()} and
Fred Drake92024d12001-11-29 07:16:19 +0000965\cfunction{Py_BuildValue()}, pass ownership to the receiver. Even if
966the object is not actually new, you still receive ownership of a new
967reference to that object. For instance, \cfunction{PyInt_FromLong()}
968maintains a cache of popular values and can return a reference to a
969cached item.
Fred Drakecc8f44b2001-08-20 19:30:29 +0000970
971Many functions that extract objects from other objects also transfer
972ownership with the reference, for instance
973\cfunction{PyObject_GetAttrString()}. The picture is less clear, here,
974however, since a few common routines are exceptions:
975\cfunction{PyTuple_GetItem()}, \cfunction{PyList_GetItem()},
976\cfunction{PyDict_GetItem()}, and \cfunction{PyDict_GetItemString()}
977all return references that you borrow from the tuple, list or
978dictionary.
979
980The function \cfunction{PyImport_AddModule()} also returns a borrowed
981reference, even though it may actually create the object it returns:
982this is possible because an owned reference to the object is stored in
983\code{sys.modules}.
984
985When you pass an object reference into another function, in general,
986the function borrows the reference from you --- if it needs to store
987it, it will use \cfunction{Py_INCREF()} to become an independent
988owner. There are exactly two important exceptions to this rule:
989\cfunction{PyTuple_SetItem()} and \cfunction{PyList_SetItem()}. These
990functions take over ownership of the item passed to them --- even if
991they fail! (Note that \cfunction{PyDict_SetItem()} and friends don't
992take over ownership --- they are ``normal.'')
993
994When a C function is called from Python, it borrows references to its
995arguments from the caller. The caller owns a reference to the object,
996so the borrowed reference's lifetime is guaranteed until the function
997returns. Only when such a borrowed reference must be stored or passed
998on, it must be turned into an owned reference by calling
999\cfunction{Py_INCREF()}.
1000
1001The object reference returned from a C function that is called from
Raymond Hettinger68804312005-01-01 00:28:46 +00001002Python must be an owned reference --- ownership is transferred from
1003the function to its caller.
Fred Drakecc8f44b2001-08-20 19:30:29 +00001004
1005
1006\subsection{Thin Ice
1007 \label{thinIce}}
1008
1009There are a few situations where seemingly harmless use of a borrowed
1010reference can lead to problems. These all have to do with implicit
1011invocations of the interpreter, which can cause the owner of a
1012reference to dispose of it.
1013
1014The first and most important case to know about is using
1015\cfunction{Py_DECREF()} on an unrelated object while borrowing a
1016reference to a list item. For instance:
1017
1018\begin{verbatim}
Fred Drake723f94b2002-06-22 01:42:00 +00001019void
1020bug(PyObject *list)
1021{
Fred Drakecc8f44b2001-08-20 19:30:29 +00001022 PyObject *item = PyList_GetItem(list, 0);
1023
1024 PyList_SetItem(list, 1, PyInt_FromLong(0L));
1025 PyObject_Print(item, stdout, 0); /* BUG! */
1026}
1027\end{verbatim}
1028
1029This function first borrows a reference to \code{list[0]}, then
1030replaces \code{list[1]} with the value \code{0}, and finally prints
1031the borrowed reference. Looks harmless, right? But it's not!
1032
1033Let's follow the control flow into \cfunction{PyList_SetItem()}. The list
1034owns references to all its items, so when item 1 is replaced, it has
1035to dispose of the original item 1. Now let's suppose the original
1036item 1 was an instance of a user-defined class, and let's further
1037suppose that the class defined a \method{__del__()} method. If this
1038class instance has a reference count of 1, disposing of it will call
1039its \method{__del__()} method.
1040
1041Since it is written in Python, the \method{__del__()} method can execute
1042arbitrary Python code. Could it perhaps do something to invalidate
1043the reference to \code{item} in \cfunction{bug()}? You bet! Assuming
1044that the list passed into \cfunction{bug()} is accessible to the
1045\method{__del__()} method, it could execute a statement to the effect of
1046\samp{del list[0]}, and assuming this was the last reference to that
1047object, it would free the memory associated with it, thereby
1048invalidating \code{item}.
1049
1050The solution, once you know the source of the problem, is easy:
1051temporarily increment the reference count. The correct version of the
1052function reads:
1053
1054\begin{verbatim}
Fred Drake723f94b2002-06-22 01:42:00 +00001055void
1056no_bug(PyObject *list)
1057{
Fred Drakecc8f44b2001-08-20 19:30:29 +00001058 PyObject *item = PyList_GetItem(list, 0);
1059
1060 Py_INCREF(item);
1061 PyList_SetItem(list, 1, PyInt_FromLong(0L));
1062 PyObject_Print(item, stdout, 0);
1063 Py_DECREF(item);
1064}
1065\end{verbatim}
1066
1067This is a true story. An older version of Python contained variants
1068of this bug and someone spent a considerable amount of time in a C
1069debugger to figure out why his \method{__del__()} methods would fail...
1070
1071The second case of problems with a borrowed reference is a variant
1072involving threads. Normally, multiple threads in the Python
1073interpreter can't get in each other's way, because there is a global
1074lock protecting Python's entire object space. However, it is possible
1075to temporarily release this lock using the macro
Fred Drake375e3022002-04-09 21:09:42 +00001076\csimplemacro{Py_BEGIN_ALLOW_THREADS}, and to re-acquire it using
1077\csimplemacro{Py_END_ALLOW_THREADS}. This is common around blocking
1078I/O calls, to let other threads use the processor while waiting for
1079the I/O to complete. Obviously, the following function has the same
1080problem as the previous one:
Fred Drakecc8f44b2001-08-20 19:30:29 +00001081
1082\begin{verbatim}
Fred Drake723f94b2002-06-22 01:42:00 +00001083void
1084bug(PyObject *list)
1085{
Fred Drakecc8f44b2001-08-20 19:30:29 +00001086 PyObject *item = PyList_GetItem(list, 0);
1087 Py_BEGIN_ALLOW_THREADS
1088 ...some blocking I/O call...
1089 Py_END_ALLOW_THREADS
1090 PyObject_Print(item, stdout, 0); /* BUG! */
1091}
1092\end{verbatim}
1093
1094
1095\subsection{NULL Pointers
1096 \label{nullPointers}}
1097
1098In general, functions that take object references as arguments do not
1099expect you to pass them \NULL{} pointers, and will dump core (or
1100cause later core dumps) if you do so. Functions that return object
1101references generally return \NULL{} only to indicate that an
1102exception occurred. The reason for not testing for \NULL{}
1103arguments is that functions often pass the objects they receive on to
Fred Drakec37b65e2001-11-28 07:26:15 +00001104other function --- if each function were to test for \NULL,
Fred Drakecc8f44b2001-08-20 19:30:29 +00001105there would be a lot of redundant tests and the code would run more
1106slowly.
1107
1108It is better to test for \NULL{} only at the ``source:'' when a
1109pointer that may be \NULL{} is received, for example, from
1110\cfunction{malloc()} or from a function that may raise an exception.
1111
1112The macros \cfunction{Py_INCREF()} and \cfunction{Py_DECREF()}
1113do not check for \NULL{} pointers --- however, their variants
1114\cfunction{Py_XINCREF()} and \cfunction{Py_XDECREF()} do.
1115
1116The macros for checking for a particular object type
1117(\code{Py\var{type}_Check()}) don't check for \NULL{} pointers ---
1118again, there is much code that calls several of these in a row to test
1119an object against various different expected types, and this would
1120generate redundant tests. There are no variants with \NULL{}
1121checking.
1122
1123The C function calling mechanism guarantees that the argument list
1124passed to C functions (\code{args} in the examples) is never
1125\NULL{} --- in fact it guarantees that it is always a tuple.\footnote{
1126These guarantees don't hold when you use the ``old'' style
1127calling convention --- this is still found in much existing code.}
1128
1129It is a severe error to ever let a \NULL{} pointer ``escape'' to
1130the Python user.
1131
1132% Frank Stajano:
1133% A pedagogically buggy example, along the lines of the previous listing,
1134% would be helpful here -- showing in more concrete terms what sort of
1135% actions could cause the problem. I can't very well imagine it from the
1136% description.
1137
1138
Fred Drakec37b65e2001-11-28 07:26:15 +00001139\section{Writing Extensions in \Cpp
Fred Drakecc8f44b2001-08-20 19:30:29 +00001140 \label{cplusplus}}
1141
Fred Drakec37b65e2001-11-28 07:26:15 +00001142It is possible to write extension modules in \Cpp. Some restrictions
Fred Drakecc8f44b2001-08-20 19:30:29 +00001143apply. If the main program (the Python interpreter) is compiled and
1144linked by the C compiler, global or static objects with constructors
1145cannot be used. This is not a problem if the main program is linked
1146by the \Cpp{} compiler. Functions that will be called by the
Raymond Hettinger68804312005-01-01 00:28:46 +00001147Python interpreter (in particular, module initialization functions)
Fred Drakecc8f44b2001-08-20 19:30:29 +00001148have to be declared using \code{extern "C"}.
1149It is unnecessary to enclose the Python header files in
1150\code{extern "C" \{...\}} --- they use this form already if the symbol
1151\samp{__cplusplus} is defined (all recent \Cpp{} compilers define this
1152symbol).
1153
1154
1155\section{Providing a C API for an Extension Module
1156 \label{using-cobjects}}
1157\sectionauthor{Konrad Hinsen}{hinsen@cnrs-orleans.fr}
1158
1159Many extension modules just provide new functions and types to be
1160used from Python, but sometimes the code in an extension module can
1161be useful for other extension modules. For example, an extension
1162module could implement a type ``collection'' which works like lists
1163without order. Just like the standard Python list type has a C API
1164which permits extension modules to create and manipulate lists, this
1165new collection type should have a set of C functions for direct
1166manipulation from other extension modules.
1167
1168At first sight this seems easy: just write the functions (without
1169declaring them \keyword{static}, of course), provide an appropriate
1170header file, and document the C API. And in fact this would work if
1171all extension modules were always linked statically with the Python
1172interpreter. When modules are used as shared libraries, however, the
1173symbols defined in one module may not be visible to another module.
1174The details of visibility depend on the operating system; some systems
1175use one global namespace for the Python interpreter and all extension
1176modules (Windows, for example), whereas others require an explicit
1177list of imported symbols at module link time (AIX is one example), or
1178offer a choice of different strategies (most Unices). And even if
1179symbols are globally visible, the module whose functions one wishes to
1180call might not have been loaded yet!
1181
1182Portability therefore requires not to make any assumptions about
1183symbol visibility. This means that all symbols in extension modules
1184should be declared \keyword{static}, except for the module's
1185initialization function, in order to avoid name clashes with other
1186extension modules (as discussed in section~\ref{methodTable}). And it
1187means that symbols that \emph{should} be accessible from other
1188extension modules must be exported in a different way.
1189
1190Python provides a special mechanism to pass C-level information
1191(pointers) from one extension module to another one: CObjects.
1192A CObject is a Python data type which stores a pointer (\ctype{void
1193*}). CObjects can only be created and accessed via their C API, but
1194they can be passed around like any other Python object. In particular,
1195they can be assigned to a name in an extension module's namespace.
1196Other extension modules can then import this module, retrieve the
1197value of this name, and then retrieve the pointer from the CObject.
1198
1199There are many ways in which CObjects can be used to export the C API
1200of an extension module. Each name could get its own CObject, or all C
1201API pointers could be stored in an array whose address is published in
1202a CObject. And the various tasks of storing and retrieving the pointers
1203can be distributed in different ways between the module providing the
1204code and the client modules.
1205
1206The following example demonstrates an approach that puts most of the
1207burden on the writer of the exporting module, which is appropriate
1208for commonly used library modules. It stores all C API pointers
1209(just one in the example!) in an array of \ctype{void} pointers which
1210becomes the value of a CObject. The header file corresponding to
1211the module provides a macro that takes care of importing the module
1212and retrieving its C API pointers; client modules only have to call
1213this macro before accessing the C API.
1214
1215The exporting module is a modification of the \module{spam} module from
1216section~\ref{simpleExample}. The function \function{spam.system()}
1217does not call the C library function \cfunction{system()} directly,
1218but a function \cfunction{PySpam_System()}, which would of course do
1219something more complicated in reality (such as adding ``spam'' to
1220every command). This function \cfunction{PySpam_System()} is also
1221exported to other extension modules.
1222
1223The function \cfunction{PySpam_System()} is a plain C function,
1224declared \keyword{static} like everything else:
1225
1226\begin{verbatim}
1227static int
Brett Cannon289e4cb2004-06-29 03:48:23 +00001228PySpam_System(const char *command)
Fred Drakecc8f44b2001-08-20 19:30:29 +00001229{
1230 return system(command);
1231}
1232\end{verbatim}
1233
1234The function \cfunction{spam_system()} is modified in a trivial way:
1235
1236\begin{verbatim}
1237static PyObject *
Fred Drake723f94b2002-06-22 01:42:00 +00001238spam_system(PyObject *self, PyObject *args)
Fred Drakecc8f44b2001-08-20 19:30:29 +00001239{
Brett Cannon289e4cb2004-06-29 03:48:23 +00001240 const char *command;
Fred Drakecc8f44b2001-08-20 19:30:29 +00001241 int sts;
1242
1243 if (!PyArg_ParseTuple(args, "s", &command))
1244 return NULL;
1245 sts = PySpam_System(command);
1246 return Py_BuildValue("i", sts);
1247}
1248\end{verbatim}
1249
1250In the beginning of the module, right after the line
1251
1252\begin{verbatim}
1253#include "Python.h"
1254\end{verbatim}
1255
1256two more lines must be added:
1257
1258\begin{verbatim}
1259#define SPAM_MODULE
1260#include "spammodule.h"
1261\end{verbatim}
1262
1263The \code{\#define} is used to tell the header file that it is being
1264included in the exporting module, not a client module. Finally,
1265the module's initialization function must take care of initializing
1266the C API pointer array:
1267
1268\begin{verbatim}
Mark Hammond8235ea12002-07-19 06:55:41 +00001269PyMODINIT_FUNC
Fred Drakeef6373a2001-11-17 06:50:42 +00001270initspam(void)
Fred Drakecc8f44b2001-08-20 19:30:29 +00001271{
1272 PyObject *m;
1273 static void *PySpam_API[PySpam_API_pointers];
1274 PyObject *c_api_object;
1275
1276 m = Py_InitModule("spam", SpamMethods);
Thomas Wouters89f507f2006-12-13 04:49:30 +00001277 if (m == NULL)
1278 return;
Fred Drakecc8f44b2001-08-20 19:30:29 +00001279
1280 /* Initialize the C API pointer array */
1281 PySpam_API[PySpam_System_NUM] = (void *)PySpam_System;
1282
1283 /* Create a CObject containing the API pointer array's address */
1284 c_api_object = PyCObject_FromVoidPtr((void *)PySpam_API, NULL);
1285
Fred Drake63e40a52002-04-12 19:08:31 +00001286 if (c_api_object != NULL)
1287 PyModule_AddObject(m, "_C_API", c_api_object);
Fred Drakecc8f44b2001-08-20 19:30:29 +00001288}
1289\end{verbatim}
1290
Fred Drakeef6373a2001-11-17 06:50:42 +00001291Note that \code{PySpam_API} is declared \keyword{static}; otherwise
1292the pointer array would disappear when \function{initspam()} terminates!
Fred Drakecc8f44b2001-08-20 19:30:29 +00001293
1294The bulk of the work is in the header file \file{spammodule.h},
1295which looks like this:
1296
1297\begin{verbatim}
1298#ifndef Py_SPAMMODULE_H
1299#define Py_SPAMMODULE_H
1300#ifdef __cplusplus
1301extern "C" {
1302#endif
1303
1304/* Header file for spammodule */
1305
1306/* C API functions */
1307#define PySpam_System_NUM 0
1308#define PySpam_System_RETURN int
Georg Brandlb6c1bb82005-06-05 10:56:59 +00001309#define PySpam_System_PROTO (const char *command)
Fred Drakecc8f44b2001-08-20 19:30:29 +00001310
1311/* Total number of C API pointers */
1312#define PySpam_API_pointers 1
1313
1314
1315#ifdef SPAM_MODULE
1316/* This section is used when compiling spammodule.c */
1317
1318static PySpam_System_RETURN PySpam_System PySpam_System_PROTO;
1319
1320#else
1321/* This section is used in modules that use spammodule's API */
1322
1323static void **PySpam_API;
1324
1325#define PySpam_System \
1326 (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM])
1327
Fred Drake63e40a52002-04-12 19:08:31 +00001328/* Return -1 and set exception on error, 0 on success. */
1329static int
1330import_spam(void)
1331{
1332 PyObject *module = PyImport_ImportModule("spam");
1333
1334 if (module != NULL) {
1335 PyObject *c_api_object = PyObject_GetAttrString(module, "_C_API");
1336 if (c_api_object == NULL)
1337 return -1;
1338 if (PyCObject_Check(c_api_object))
1339 PySpam_API = (void **)PyCObject_AsVoidPtr(c_api_object);
1340 Py_DECREF(c_api_object);
1341 }
1342 return 0;
Fred Drakecc8f44b2001-08-20 19:30:29 +00001343}
1344
1345#endif
1346
1347#ifdef __cplusplus
1348}
1349#endif
1350
Raymond Hettingerf9c2eda2003-05-20 05:31:16 +00001351#endif /* !defined(Py_SPAMMODULE_H) */
Fred Drakecc8f44b2001-08-20 19:30:29 +00001352\end{verbatim}
1353
1354All that a client module must do in order to have access to the
1355function \cfunction{PySpam_System()} is to call the function (or
1356rather macro) \cfunction{import_spam()} in its initialization
1357function:
1358
1359\begin{verbatim}
Mark Hammond8235ea12002-07-19 06:55:41 +00001360PyMODINIT_FUNC
Fred Drakeef6373a2001-11-17 06:50:42 +00001361initclient(void)
Fred Drakecc8f44b2001-08-20 19:30:29 +00001362{
1363 PyObject *m;
1364
Thomas Wouters89f507f2006-12-13 04:49:30 +00001365 m = Py_InitModule("client", ClientMethods);
1366 if (m == NULL)
1367 return;
Fred Drake63e40a52002-04-12 19:08:31 +00001368 if (import_spam() < 0)
1369 return;
1370 /* additional initialization can happen here */
Fred Drakecc8f44b2001-08-20 19:30:29 +00001371}
1372\end{verbatim}
1373
1374The main disadvantage of this approach is that the file
1375\file{spammodule.h} is rather complicated. However, the
1376basic structure is the same for each function that is
1377exported, so it has to be learned only once.
1378
1379Finally it should be mentioned that CObjects offer additional
1380functionality, which is especially useful for memory allocation and
1381deallocation of the pointer stored in a CObject. The details
1382are described in the \citetitle[../api/api.html]{Python/C API
Fred Drake63e40a52002-04-12 19:08:31 +00001383Reference Manual} in the section
1384``\ulink{CObjects}{../api/cObjects.html}'' and in the implementation
1385of CObjects (files \file{Include/cobject.h} and
Fred Drakecc8f44b2001-08-20 19:30:29 +00001386\file{Objects/cobject.c} in the Python source code distribution).