| \chapter{Extending Python with C or \Cpp \label{intro}} |
| |
| |
| It is quite easy to add new built-in modules to Python, if you know |
| how to program in C. Such \dfn{extension modules} can do two things |
| that can't be done directly in Python: they can implement new built-in |
| object types, and they can call C library functions and system calls. |
| |
| To support extensions, the Python API (Application Programmers |
| Interface) defines a set of functions, macros and variables that |
| provide access to most aspects of the Python run-time system. The |
| Python API is incorporated in a C source file by including the header |
| \code{"Python.h"}. |
| |
| The compilation of an extension module depends on its intended use as |
| well as on your system setup; details are given in later chapters. |
| |
| |
| \section{A Simple Example |
| \label{simpleExample}} |
| |
| Let's create an extension module called \samp{spam} (the favorite food |
| of Monty Python fans...) and let's say we want to create a Python |
| interface to the C library function \cfunction{system()}.\footnote{An |
| interface for this function already exists in the standard module |
| \module{os} --- it was chosen as a simple and straightfoward example.} |
| This function takes a null-terminated character string as argument and |
| returns an integer. We want this function to be callable from Python |
| as follows: |
| |
| \begin{verbatim} |
| >>> import spam |
| >>> status = spam.system("ls -l") |
| \end{verbatim} |
| |
| Begin by creating a file \file{spammodule.c}. (Historically, if a |
| module is called \samp{spam}, the C file containing its implementation |
| is called \file{spammodule.c}; if the module name is very long, like |
| \samp{spammify}, the module name can be just \file{spammify.c}.) |
| |
| The first line of our file can be: |
| |
| \begin{verbatim} |
| #include <Python.h> |
| \end{verbatim} |
| |
| which pulls in the Python API (you can add a comment describing the |
| purpose of the module and a copyright notice if you like). |
| Since Python may define some pre-processor definitions which affect |
| the standard headers on some systems, you must include \file{Python.h} |
| before any standard headers are included. |
| |
| All user-visible symbols defined by \file{Python.h} have a prefix of |
| \samp{Py} or \samp{PY}, except those defined in standard header files. |
| For convenience, and since they are used extensively by the Python |
| interpreter, \code{"Python.h"} includes a few standard header files: |
| \code{<stdio.h>}, \code{<string.h>}, \code{<errno.h>}, and |
| \code{<stdlib.h>}. If the latter header file does not exist on your |
| system, it declares the functions \cfunction{malloc()}, |
| \cfunction{free()} and \cfunction{realloc()} directly. |
| |
| The next thing we add to our module file is the C function that will |
| be called when the Python expression \samp{spam.system(\var{string})} |
| is evaluated (we'll see shortly how it ends up being called): |
| |
| \begin{verbatim} |
| static PyObject * |
| spam_system(PyObject *self, PyObject *args) |
| { |
| char *command; |
| int sts; |
| |
| if (!PyArg_ParseTuple(args, "s", &command)) |
| return NULL; |
| sts = system(command); |
| return Py_BuildValue("i", sts); |
| } |
| \end{verbatim} |
| |
| There is a straightforward translation from the argument list in |
| Python (for example, the single expression \code{"ls -l"}) to the |
| arguments passed to the C function. The C function always has two |
| arguments, conventionally named \var{self} and \var{args}. |
| |
| The \var{self} argument is only used when the C function implements a |
| built-in method, not a function. In the example, \var{self} will |
| always be a \NULL{} pointer, since we are defining a function, not a |
| method. (This is done so that the interpreter doesn't have to |
| understand two different types of C functions.) |
| |
| The \var{args} argument will be a pointer to a Python tuple object |
| containing the arguments. Each item of the tuple corresponds to an |
| argument in the call's argument list. The arguments are Python |
| objects --- in order to do anything with them in our C function we have |
| to convert them to C values. The function \cfunction{PyArg_ParseTuple()} |
| in the Python API checks the argument types and converts them to C |
| values. It uses a template string to determine the required types of |
| the arguments as well as the types of the C variables into which to |
| store the converted values. More about this later. |
| |
| \cfunction{PyArg_ParseTuple()} returns true (nonzero) if all arguments have |
| the right type and its components have been stored in the variables |
| whose addresses are passed. It returns false (zero) if an invalid |
| argument list was passed. In the latter case it also raises an |
| appropriate exception so the calling function can return |
| \NULL{} immediately (as we saw in the example). |
| |
| |
| \section{Intermezzo: Errors and Exceptions |
| \label{errors}} |
| |
| An important convention throughout the Python interpreter is the |
| following: when a function fails, it should set an exception condition |
| and return an error value (usually a \NULL{} pointer). Exceptions |
| are stored in a static global variable inside the interpreter; if this |
| variable is \NULL{} no exception has occurred. A second global |
| variable stores the ``associated value'' of the exception (the second |
| argument to \keyword{raise}). A third variable contains the stack |
| traceback in case the error originated in Python code. These three |
| variables are the C equivalents of the Python variables |
| \code{sys.exc_type}, \code{sys.exc_value} and \code{sys.exc_traceback} (see |
| the section on module \module{sys} in the |
| \citetitle[../lib/lib.html]{Python Library Reference}). It is |
| important to know about them to understand how errors are passed |
| around. |
| |
| The Python API defines a number of functions to set various types of |
| exceptions. |
| |
| The most common one is \cfunction{PyErr_SetString()}. Its arguments |
| are an exception object and a C string. The exception object is |
| usually a predefined object like \cdata{PyExc_ZeroDivisionError}. The |
| C string indicates the cause of the error and is converted to a |
| Python string object and stored as the ``associated value'' of the |
| exception. |
| |
| Another useful function is \cfunction{PyErr_SetFromErrno()}, which only |
| takes an exception argument and constructs the associated value by |
| inspection of the global variable \cdata{errno}. The most |
| general function is \cfunction{PyErr_SetObject()}, which takes two object |
| arguments, the exception and its associated value. You don't need to |
| \cfunction{Py_INCREF()} the objects passed to any of these functions. |
| |
| You can test non-destructively whether an exception has been set with |
| \cfunction{PyErr_Occurred()}. This returns the current exception object, |
| or \NULL{} if no exception has occurred. You normally don't need |
| to call \cfunction{PyErr_Occurred()} to see whether an error occurred in a |
| function call, since you should be able to tell from the return value. |
| |
| When a function \var{f} that calls another function \var{g} detects |
| that the latter fails, \var{f} should itself return an error value |
| (usually \NULL{} or \code{-1}). It should \emph{not} call one of the |
| \cfunction{PyErr_*()} functions --- one has already been called by \var{g}. |
| \var{f}'s caller is then supposed to also return an error indication |
| to \emph{its} caller, again \emph{without} calling \cfunction{PyErr_*()}, |
| and so on --- the most detailed cause of the error was already |
| reported by the function that first detected it. Once the error |
| reaches the Python interpreter's main loop, this aborts the currently |
| executing Python code and tries to find an exception handler specified |
| by the Python programmer. |
| |
| (There are situations where a module can actually give a more detailed |
| error message by calling another \cfunction{PyErr_*()} function, and in |
| such cases it is fine to do so. As a general rule, however, this is |
| not necessary, and can cause information about the cause of the error |
| to be lost: most operations can fail for a variety of reasons.) |
| |
| To ignore an exception set by a function call that failed, the exception |
| condition must be cleared explicitly by calling \cfunction{PyErr_Clear()}. |
| The only time C code should call \cfunction{PyErr_Clear()} is if it doesn't |
| want to pass the error on to the interpreter but wants to handle it |
| completely by itself (possibly by trying something else, or pretending |
| nothing went wrong). |
| |
| Every failing \cfunction{malloc()} call must be turned into an |
| exception --- the direct caller of \cfunction{malloc()} (or |
| \cfunction{realloc()}) must call \cfunction{PyErr_NoMemory()} and |
| return a failure indicator itself. All the object-creating functions |
| (for example, \cfunction{PyInt_FromLong()}) already do this, so this |
| note is only relevant to those who call \cfunction{malloc()} directly. |
| |
| Also note that, with the important exception of |
| \cfunction{PyArg_ParseTuple()} and friends, functions that return an |
| integer status usually return a positive value or zero for success and |
| \code{-1} for failure, like \UNIX{} system calls. |
| |
| Finally, be careful to clean up garbage (by making |
| \cfunction{Py_XDECREF()} or \cfunction{Py_DECREF()} calls for objects |
| you have already created) when you return an error indicator! |
| |
| The choice of which exception to raise is entirely yours. There are |
| predeclared C objects corresponding to all built-in Python exceptions, |
| such as \cdata{PyExc_ZeroDivisionError}, which you can use directly. |
| Of course, you should choose exceptions wisely --- don't use |
| \cdata{PyExc_TypeError} to mean that a file couldn't be opened (that |
| should probably be \cdata{PyExc_IOError}). If something's wrong with |
| the argument list, the \cfunction{PyArg_ParseTuple()} function usually |
| raises \cdata{PyExc_TypeError}. If you have an argument whose value |
| must be in a particular range or must satisfy other conditions, |
| \cdata{PyExc_ValueError} is appropriate. |
| |
| You can also define a new exception that is unique to your module. |
| For this, you usually declare a static object variable at the |
| beginning of your file: |
| |
| \begin{verbatim} |
| static PyObject *SpamError; |
| \end{verbatim} |
| |
| and initialize it in your module's initialization function |
| (\cfunction{initspam()}) with an exception object (leaving out |
| the error checking for now): |
| |
| \begin{verbatim} |
| PyMODINIT_FUNC |
| initspam(void) |
| { |
| PyObject *m; |
| |
| m = Py_InitModule("spam", SpamMethods); |
| |
| SpamError = PyErr_NewException("spam.error", NULL, NULL); |
| Py_INCREF(SpamError); |
| PyModule_AddObject(m, "error", SpamError); |
| } |
| \end{verbatim} |
| |
| Note that the Python name for the exception object is |
| \exception{spam.error}. The \cfunction{PyErr_NewException()} function |
| may create a class with the base class being \exception{Exception} |
| (unless another class is passed in instead of \NULL), described in the |
| \citetitle[../lib/lib.html]{Python Library Reference} under ``Built-in |
| Exceptions.'' |
| |
| Note also that the \cdata{SpamError} variable retains a reference to |
| the newly created exception class; this is intentional! Since the |
| exception could be removed from the module by external code, an owned |
| reference to the class is needed to ensure that it will not be |
| discarded, causing \cdata{SpamError} to become a dangling pointer. |
| Should it become a dangling pointer, C code which raises the exception |
| could cause a core dump or other unintended side effects. |
| |
| We discuss the use of PyMODINIT_FUNC later in this sample. |
| |
| \section{Back to the Example |
| \label{backToExample}} |
| |
| Going back to our example function, you should now be able to |
| understand this statement: |
| |
| \begin{verbatim} |
| if (!PyArg_ParseTuple(args, "s", &command)) |
| return NULL; |
| \end{verbatim} |
| |
| It returns \NULL{} (the error indicator for functions returning |
| object pointers) if an error is detected in the argument list, relying |
| on the exception set by \cfunction{PyArg_ParseTuple()}. Otherwise the |
| string value of the argument has been copied to the local variable |
| \cdata{command}. This is a pointer assignment and you are not supposed |
| to modify the string to which it points (so in Standard C, the variable |
| \cdata{command} should properly be declared as \samp{const char |
| *command}). |
| |
| The next statement is a call to the \UNIX{} function |
| \cfunction{system()}, passing it the string we just got from |
| \cfunction{PyArg_ParseTuple()}: |
| |
| \begin{verbatim} |
| sts = system(command); |
| \end{verbatim} |
| |
| Our \function{spam.system()} function must return the value of |
| \cdata{sts} as a Python object. This is done using the function |
| \cfunction{Py_BuildValue()}, which is something like the inverse of |
| \cfunction{PyArg_ParseTuple()}: it takes a format string and an |
| arbitrary number of C values, and returns a new Python object. |
| More info on \cfunction{Py_BuildValue()} is given later. |
| |
| \begin{verbatim} |
| return Py_BuildValue("i", sts); |
| \end{verbatim} |
| |
| In this case, it will return an integer object. (Yes, even integers |
| are objects on the heap in Python!) |
| |
| If you have a C function that returns no useful argument (a function |
| returning \ctype{void}), the corresponding Python function must return |
| \code{None}. You need this idiom to do so: |
| |
| \begin{verbatim} |
| Py_INCREF(Py_None); |
| return Py_None; |
| \end{verbatim} |
| |
| \cdata{Py_None} is the C name for the special Python object |
| \code{None}. It is a genuine Python object rather than a \NULL{} |
| pointer, which means ``error'' in most contexts, as we have seen. |
| |
| |
| \section{The Module's Method Table and Initialization Function |
| \label{methodTable}} |
| |
| I promised to show how \cfunction{spam_system()} is called from Python |
| programs. First, we need to list its name and address in a ``method |
| table'': |
| |
| \begin{verbatim} |
| static PyMethodDef SpamMethods[] = { |
| ... |
| {"system", spam_system, METH_VARARGS, |
| "Execute a shell command."}, |
| ... |
| {NULL, NULL, 0, NULL} /* Sentinel */ |
| }; |
| \end{verbatim} |
| |
| Note the third entry (\samp{METH_VARARGS}). This is a flag telling |
| the interpreter the calling convention to be used for the C |
| function. It should normally always be \samp{METH_VARARGS} or |
| \samp{METH_VARARGS | METH_KEYWORDS}; a value of \code{0} means that an |
| obsolete variant of \cfunction{PyArg_ParseTuple()} is used. |
| |
| When using only \samp{METH_VARARGS}, the function should expect |
| the Python-level parameters to be passed in as a tuple acceptable for |
| parsing via \cfunction{PyArg_ParseTuple()}; more information on this |
| function is provided below. |
| |
| The \constant{METH_KEYWORDS} bit may be set in the third field if |
| keyword arguments should be passed to the function. In this case, the |
| C function should accept a third \samp{PyObject *} parameter which |
| will be a dictionary of keywords. Use |
| \cfunction{PyArg_ParseTupleAndKeywords()} to parse the arguments to |
| such a function. |
| |
| The method table must be passed to the interpreter in the module's |
| initialization function. The initialization function must be named |
| \cfunction{init\var{name}()}, where \var{name} is the name of the |
| module, and should be the only non-\keyword{static} item defined in |
| the module file: |
| |
| \begin{verbatim} |
| PyMODINIT_FUNC |
| initspam(void) |
| { |
| (void) Py_InitModule("spam", SpamMethods); |
| } |
| \end{verbatim} |
| |
| Note that PyMODINIT_FUNC declares the function as \code{void} return type, |
| declares any special linkage declarations required by the platform, and for |
| \Cpp declares the function as \code{extern "C"}. |
| |
| When the Python program imports module \module{spam} for the first |
| time, \cfunction{initspam()} is called. (See below for comments about |
| embedding Python.) It calls |
| \cfunction{Py_InitModule()}, which creates a ``module object'' (which |
| is inserted in the dictionary \code{sys.modules} under the key |
| \code{"spam"}), and inserts built-in function objects into the newly |
| created module based upon the table (an array of \ctype{PyMethodDef} |
| structures) that was passed as its second argument. |
| \cfunction{Py_InitModule()} returns a pointer to the module object |
| that it creates (which is unused here). It aborts with a fatal error |
| if the module could not be initialized satisfactorily, so the caller |
| doesn't need to check for errors. |
| |
| When embedding Python, the \cfunction{initspam()} function is not |
| called automatically unless there's an entry in the |
| \cdata{_PyImport_Inittab} table. The easiest way to handle this is to |
| statically initialize your statically-linked modules by directly |
| calling \cfunction{initspam()} after the call to |
| \cfunction{Py_Initialize()} or \cfunction{PyMac_Initialize()}: |
| |
| \begin{verbatim} |
| int |
| main(int argc, char *argv[]) |
| { |
| /* Pass argv[0] to the Python interpreter */ |
| Py_SetProgramName(argv[0]); |
| |
| /* Initialize the Python interpreter. Required. */ |
| Py_Initialize(); |
| |
| /* Add a static module */ |
| initspam(); |
| \end{verbatim} |
| |
| An example may be found in the file \file{Demo/embed/demo.c} in the |
| Python source distribution. |
| |
| \note{Removing entries from \code{sys.modules} or importing |
| compiled modules into multiple interpreters within a process (or |
| following a \cfunction{fork()} without an intervening |
| \cfunction{exec()}) can create problems for some extension modules. |
| Extension module authors should exercise caution when initializing |
| internal data structures. |
| Note also that the \function{reload()} function can be used with |
| extension modules, and will call the module initialization function |
| (\cfunction{initspam()} in the example), but will not load the module |
| again if it was loaded from a dynamically loadable object file |
| (\file{.so} on \UNIX, \file{.dll} on Windows).} |
| |
| A more substantial example module is included in the Python source |
| distribution as \file{Modules/xxmodule.c}. This file may be used as a |
| template or simply read as an example. The \program{modulator.py} |
| script included in the source distribution or Windows install provides |
| a simple graphical user interface for declaring the functions and |
| objects which a module should implement, and can generate a template |
| which can be filled in. The script lives in the |
| \file{Tools/modulator/} directory; see the \file{README} file there |
| for more information. |
| |
| |
| \section{Compilation and Linkage |
| \label{compilation}} |
| |
| There are two more things to do before you can use your new extension: |
| compiling and linking it with the Python system. If you use dynamic |
| loading, the details may depend on the style of dynamic loading your |
| system uses; see the chapters about building extension modules |
| (chapter \ref{building}) and additional information that pertains only |
| to building on Windows (chapter \ref{building-on-windows}) for more |
| information about this. |
| % XXX Add information about Mac OS |
| |
| If you can't use dynamic loading, or if you want to make your module a |
| permanent part of the Python interpreter, you will have to change the |
| configuration setup and rebuild the interpreter. Luckily, this is |
| very simple on \UNIX: just place your file (\file{spammodule.c} for |
| example) in the \file{Modules/} directory of an unpacked source |
| distribution, add a line to the file \file{Modules/Setup.local} |
| describing your file: |
| |
| \begin{verbatim} |
| spam spammodule.o |
| \end{verbatim} |
| |
| and rebuild the interpreter by running \program{make} in the toplevel |
| directory. You can also run \program{make} in the \file{Modules/} |
| subdirectory, but then you must first rebuild \file{Makefile} |
| there by running `\program{make} Makefile'. (This is necessary each |
| time you change the \file{Setup} file.) |
| |
| If your module requires additional libraries to link with, these can |
| be listed on the line in the configuration file as well, for instance: |
| |
| \begin{verbatim} |
| spam spammodule.o -lX11 |
| \end{verbatim} |
| |
| \section{Calling Python Functions from C |
| \label{callingPython}} |
| |
| So far we have concentrated on making C functions callable from |
| Python. The reverse is also useful: calling Python functions from C. |
| This is especially the case for libraries that support so-called |
| ``callback'' functions. If a C interface makes use of callbacks, the |
| equivalent Python often needs to provide a callback mechanism to the |
| Python programmer; the implementation will require calling the Python |
| callback functions from a C callback. Other uses are also imaginable. |
| |
| Fortunately, the Python interpreter is easily called recursively, and |
| there is a standard interface to call a Python function. (I won't |
| dwell on how to call the Python parser with a particular string as |
| input --- if you're interested, have a look at the implementation of |
| the \programopt{-c} command line option in \file{Python/pythonmain.c} |
| from the Python source code.) |
| |
| Calling a Python function is easy. First, the Python program must |
| somehow pass you the Python function object. You should provide a |
| function (or some other interface) to do this. When this function is |
| called, save a pointer to the Python function object (be careful to |
| \cfunction{Py_INCREF()} it!) in a global variable --- or wherever you |
| see fit. For example, the following function might be part of a module |
| definition: |
| |
| \begin{verbatim} |
| static PyObject *my_callback = NULL; |
| |
| static PyObject * |
| my_set_callback(PyObject *dummy, PyObject *args) |
| { |
| PyObject *result = NULL; |
| PyObject *temp; |
| |
| if (PyArg_ParseTuple(args, "O:set_callback", &temp)) { |
| if (!PyCallable_Check(temp)) { |
| PyErr_SetString(PyExc_TypeError, "parameter must be callable"); |
| return NULL; |
| } |
| Py_XINCREF(temp); /* Add a reference to new callback */ |
| Py_XDECREF(my_callback); /* Dispose of previous callback */ |
| my_callback = temp; /* Remember new callback */ |
| /* Boilerplate to return "None" */ |
| Py_INCREF(Py_None); |
| result = Py_None; |
| } |
| return result; |
| } |
| \end{verbatim} |
| |
| This function must be registered with the interpreter using the |
| \constant{METH_VARARGS} flag; this is described in section |
| \ref{methodTable}, ``The Module's Method Table and Initialization |
| Function.'' The \cfunction{PyArg_ParseTuple()} function and its |
| arguments are documented in section~\ref{parseTuple}, ``Extracting |
| Parameters in Extension Functions.'' |
| |
| The macros \cfunction{Py_XINCREF()} and \cfunction{Py_XDECREF()} |
| increment/decrement the reference count of an object and are safe in |
| the presence of \NULL{} pointers (but note that \var{temp} will not be |
| \NULL{} in this context). More info on them in |
| section~\ref{refcounts}, ``Reference Counts.'' |
| |
| Later, when it is time to call the function, you call the C function |
| \cfunction{PyEval_CallObject()}.\ttindex{PyEval_CallObject()} This |
| function has two arguments, both pointers to arbitrary Python objects: |
| the Python function, and the argument list. The argument list must |
| always be a tuple object, whose length is the number of arguments. To |
| call the Python function with no arguments, pass an empty tuple; to |
| call it with one argument, pass a singleton tuple. |
| \cfunction{Py_BuildValue()} returns a tuple when its format string |
| consists of zero or more format codes between parentheses. For |
| example: |
| |
| \begin{verbatim} |
| int arg; |
| PyObject *arglist; |
| PyObject *result; |
| ... |
| arg = 123; |
| ... |
| /* Time to call the callback */ |
| arglist = Py_BuildValue("(i)", arg); |
| result = PyEval_CallObject(my_callback, arglist); |
| Py_DECREF(arglist); |
| \end{verbatim} |
| |
| \cfunction{PyEval_CallObject()} returns a Python object pointer: this is |
| the return value of the Python function. \cfunction{PyEval_CallObject()} is |
| ``reference-count-neutral'' with respect to its arguments. In the |
| example a new tuple was created to serve as the argument list, which |
| is \cfunction{Py_DECREF()}-ed immediately after the call. |
| |
| The return value of \cfunction{PyEval_CallObject()} is ``new'': either it |
| is a brand new object, or it is an existing object whose reference |
| count has been incremented. So, unless you want to save it in a |
| global variable, you should somehow \cfunction{Py_DECREF()} the result, |
| even (especially!) if you are not interested in its value. |
| |
| Before you do this, however, it is important to check that the return |
| value isn't \NULL. If it is, the Python function terminated by |
| raising an exception. If the C code that called |
| \cfunction{PyEval_CallObject()} is called from Python, it should now |
| return an error indication to its Python caller, so the interpreter |
| can print a stack trace, or the calling Python code can handle the |
| exception. If this is not possible or desirable, the exception should |
| be cleared by calling \cfunction{PyErr_Clear()}. For example: |
| |
| \begin{verbatim} |
| if (result == NULL) |
| return NULL; /* Pass error back */ |
| ...use result... |
| Py_DECREF(result); |
| \end{verbatim} |
| |
| Depending on the desired interface to the Python callback function, |
| you may also have to provide an argument list to |
| \cfunction{PyEval_CallObject()}. In some cases the argument list is |
| also provided by the Python program, through the same interface that |
| specified the callback function. It can then be saved and used in the |
| same manner as the function object. In other cases, you may have to |
| construct a new tuple to pass as the argument list. The simplest way |
| to do this is to call \cfunction{Py_BuildValue()}. For example, if |
| you want to pass an integral event code, you might use the following |
| code: |
| |
| \begin{verbatim} |
| PyObject *arglist; |
| ... |
| arglist = Py_BuildValue("(l)", eventcode); |
| result = PyEval_CallObject(my_callback, arglist); |
| Py_DECREF(arglist); |
| if (result == NULL) |
| return NULL; /* Pass error back */ |
| /* Here maybe use the result */ |
| Py_DECREF(result); |
| \end{verbatim} |
| |
| Note the placement of \samp{Py_DECREF(arglist)} immediately after the |
| call, before the error check! Also note that strictly spoken this |
| code is not complete: \cfunction{Py_BuildValue()} may run out of |
| memory, and this should be checked. |
| |
| |
| \section{Extracting Parameters in Extension Functions |
| \label{parseTuple}} |
| |
| \ttindex{PyArg_ParseTuple()} |
| |
| The \cfunction{PyArg_ParseTuple()} function is declared as follows: |
| |
| \begin{verbatim} |
| int PyArg_ParseTuple(PyObject *arg, char *format, ...); |
| \end{verbatim} |
| |
| The \var{arg} argument must be a tuple object containing an argument |
| list passed from Python to a C function. The \var{format} argument |
| must be a format string, whose syntax is explained in |
| ``\ulink{Parsing arguments and building |
| values}{../api/arg-parsing.html}'' in the |
| \citetitle[../api/api.html]{Python/C API Reference Manual}. The |
| remaining arguments must be addresses of variables whose type is |
| determined by the format string. |
| |
| Note that while \cfunction{PyArg_ParseTuple()} checks that the Python |
| arguments have the required types, it cannot check the validity of the |
| addresses of C variables passed to the call: if you make mistakes |
| there, your code will probably crash or at least overwrite random bits |
| in memory. So be careful! |
| |
| Note that any Python object references which are provided to the |
| caller are \emph{borrowed} references; do not decrement their |
| reference count! |
| |
| Some example calls: |
| |
| \begin{verbatim} |
| int ok; |
| int i, j; |
| long k, l; |
| char *s; |
| int size; |
| |
| ok = PyArg_ParseTuple(args, ""); /* No arguments */ |
| /* Python call: f() */ |
| \end{verbatim} |
| |
| \begin{verbatim} |
| ok = PyArg_ParseTuple(args, "s", &s); /* A string */ |
| /* Possible Python call: f('whoops!') */ |
| \end{verbatim} |
| |
| \begin{verbatim} |
| ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */ |
| /* Possible Python call: f(1, 2, 'three') */ |
| \end{verbatim} |
| |
| \begin{verbatim} |
| ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size); |
| /* A pair of ints and a string, whose size is also returned */ |
| /* Possible Python call: f((1, 2), 'three') */ |
| \end{verbatim} |
| |
| \begin{verbatim} |
| { |
| char *file; |
| char *mode = "r"; |
| int bufsize = 0; |
| ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize); |
| /* A string, and optionally another string and an integer */ |
| /* Possible Python calls: |
| f('spam') |
| f('spam', 'w') |
| f('spam', 'wb', 100000) */ |
| } |
| \end{verbatim} |
| |
| \begin{verbatim} |
| { |
| int left, top, right, bottom, h, v; |
| ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)", |
| &left, &top, &right, &bottom, &h, &v); |
| /* A rectangle and a point */ |
| /* Possible Python call: |
| f(((0, 0), (400, 300)), (10, 10)) */ |
| } |
| \end{verbatim} |
| |
| \begin{verbatim} |
| { |
| Py_complex c; |
| ok = PyArg_ParseTuple(args, "D:myfunction", &c); |
| /* a complex, also providing a function name for errors */ |
| /* Possible Python call: myfunction(1+2j) */ |
| } |
| \end{verbatim} |
| |
| |
| \section{Keyword Parameters for Extension Functions |
| \label{parseTupleAndKeywords}} |
| |
| \ttindex{PyArg_ParseTupleAndKeywords()} |
| |
| The \cfunction{PyArg_ParseTupleAndKeywords()} function is declared as |
| follows: |
| |
| \begin{verbatim} |
| int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict, |
| char *format, char *kwlist[], ...); |
| \end{verbatim} |
| |
| The \var{arg} and \var{format} parameters are identical to those of the |
| \cfunction{PyArg_ParseTuple()} function. The \var{kwdict} parameter |
| is the dictionary of keywords received as the third parameter from the |
| Python runtime. The \var{kwlist} parameter is a \NULL-terminated |
| list of strings which identify the parameters; the names are matched |
| with the type information from \var{format} from left to right. On |
| success, \cfunction{PyArg_ParseTupleAndKeywords()} returns true, |
| otherwise it returns false and raises an appropriate exception. |
| |
| \note{Nested tuples cannot be parsed when using keyword |
| arguments! Keyword parameters passed in which are not present in the |
| \var{kwlist} will cause \exception{TypeError} to be raised.} |
| |
| Here is an example module which uses keywords, based on an example by |
| Geoff Philbrick (\email{philbrick@hks.com}):% |
| \index{Philbrick, Geoff} |
| |
| \begin{verbatim} |
| #include "Python.h" |
| |
| static PyObject * |
| keywdarg_parrot(PyObject *self, PyObject *args, PyObject *keywds) |
| { |
| int voltage; |
| char *state = "a stiff"; |
| char *action = "voom"; |
| char *type = "Norwegian Blue"; |
| |
| static char *kwlist[] = {"voltage", "state", "action", "type", NULL}; |
| |
| if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist, |
| &voltage, &state, &action, &type)) |
| return NULL; |
| |
| printf("-- This parrot wouldn't %s if you put %i Volts through it.\n", |
| action, voltage); |
| printf("-- Lovely plumage, the %s -- It's %s!\n", type, state); |
| |
| Py_INCREF(Py_None); |
| |
| return Py_None; |
| } |
| |
| static PyMethodDef keywdarg_methods[] = { |
| /* The cast of the function is necessary since PyCFunction values |
| * only take two PyObject* parameters, and keywdarg_parrot() takes |
| * three. |
| */ |
| {"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS | METH_KEYWORDS, |
| "Print a lovely skit to standard output."}, |
| {NULL, NULL, 0, NULL} /* sentinel */ |
| }; |
| \end{verbatim} |
| |
| \begin{verbatim} |
| void |
| initkeywdarg(void) |
| { |
| /* Create the module and add the functions */ |
| Py_InitModule("keywdarg", keywdarg_methods); |
| } |
| \end{verbatim} |
| |
| |
| \section{Building Arbitrary Values |
| \label{buildValue}} |
| |
| This function is the counterpart to \cfunction{PyArg_ParseTuple()}. It is |
| declared as follows: |
| |
| \begin{verbatim} |
| PyObject *Py_BuildValue(char *format, ...); |
| \end{verbatim} |
| |
| It recognizes a set of format units similar to the ones recognized by |
| \cfunction{PyArg_ParseTuple()}, but the arguments (which are input to the |
| function, not output) must not be pointers, just values. It returns a |
| new Python object, suitable for returning from a C function called |
| from Python. |
| |
| One difference with \cfunction{PyArg_ParseTuple()}: while the latter |
| requires its first argument to be a tuple (since Python argument lists |
| are always represented as tuples internally), |
| \cfunction{Py_BuildValue()} does not always build a tuple. It builds |
| a tuple only if its format string contains two or more format units. |
| If the format string is empty, it returns \code{None}; if it contains |
| exactly one format unit, it returns whatever object is described by |
| that format unit. To force it to return a tuple of size 0 or one, |
| parenthesize the format string. |
| |
| Examples (to the left the call, to the right the resulting Python value): |
| |
| \begin{verbatim} |
| Py_BuildValue("") None |
| Py_BuildValue("i", 123) 123 |
| Py_BuildValue("iii", 123, 456, 789) (123, 456, 789) |
| Py_BuildValue("s", "hello") 'hello' |
| Py_BuildValue("ss", "hello", "world") ('hello', 'world') |
| Py_BuildValue("s#", "hello", 4) 'hell' |
| Py_BuildValue("()") () |
| Py_BuildValue("(i)", 123) (123,) |
| Py_BuildValue("(ii)", 123, 456) (123, 456) |
| Py_BuildValue("(i,i)", 123, 456) (123, 456) |
| Py_BuildValue("[i,i]", 123, 456) [123, 456] |
| Py_BuildValue("{s:i,s:i}", |
| "abc", 123, "def", 456) {'abc': 123, 'def': 456} |
| Py_BuildValue("((ii)(ii)) (ii)", |
| 1, 2, 3, 4, 5, 6) (((1, 2), (3, 4)), (5, 6)) |
| \end{verbatim} |
| |
| |
| \section{Reference Counts |
| \label{refcounts}} |
| |
| In languages like C or \Cpp, the programmer is responsible for |
| dynamic allocation and deallocation of memory on the heap. In C, |
| this is done using the functions \cfunction{malloc()} and |
| \cfunction{free()}. In \Cpp, the operators \keyword{new} and |
| \keyword{delete} are used with essentially the same meaning and |
| we'll restrict the following discussion to the latter. |
| |
| Every block of memory allocated with \cfunction{malloc()} should |
| eventually be returned to the pool of available memory by exactly one |
| call to \cfunction{free()}. It is important to call |
| \cfunction{free()} at the right time. If a block's address is |
| forgotten but \cfunction{free()} is not called for it, the memory it |
| occupies cannot be reused until the program terminates. This is |
| called a \dfn{memory leak}. On the other hand, if a program calls |
| \cfunction{free()} for a block and then continues to use the block, it |
| creates a conflict with re-use of the block through another |
| \cfunction{malloc()} call. This is called \dfn{using freed memory}. |
| It has the same bad consequences as referencing uninitialized data --- |
| core dumps, wrong results, mysterious crashes. |
| |
| Common causes of memory leaks are unusual paths through the code. For |
| instance, a function may allocate a block of memory, do some |
| calculation, and then free the block again. Now a change in the |
| requirements for the function may add a test to the calculation that |
| detects an error condition and can return prematurely from the |
| function. It's easy to forget to free the allocated memory block when |
| taking this premature exit, especially when it is added later to the |
| code. Such leaks, once introduced, often go undetected for a long |
| time: the error exit is taken only in a small fraction of all calls, |
| and most modern machines have plenty of virtual memory, so the leak |
| only becomes apparent in a long-running process that uses the leaking |
| function frequently. Therefore, it's important to prevent leaks from |
| happening by having a coding convention or strategy that minimizes |
| this kind of errors. |
| |
| Since Python makes heavy use of \cfunction{malloc()} and |
| \cfunction{free()}, it needs a strategy to avoid memory leaks as well |
| as the use of freed memory. The chosen method is called |
| \dfn{reference counting}. The principle is simple: every object |
| contains a counter, which is incremented when a reference to the |
| object is stored somewhere, and which is decremented when a reference |
| to it is deleted. When the counter reaches zero, the last reference |
| to the object has been deleted and the object is freed. |
| |
| An alternative strategy is called \dfn{automatic garbage collection}. |
| (Sometimes, reference counting is also referred to as a garbage |
| collection strategy, hence my use of ``automatic'' to distinguish the |
| two.) The big advantage of automatic garbage collection is that the |
| user doesn't need to call \cfunction{free()} explicitly. (Another claimed |
| advantage is an improvement in speed or memory usage --- this is no |
| hard fact however.) The disadvantage is that for C, there is no |
| truly portable automatic garbage collector, while reference counting |
| can be implemented portably (as long as the functions \cfunction{malloc()} |
| and \cfunction{free()} are available --- which the C Standard guarantees). |
| Maybe some day a sufficiently portable automatic garbage collector |
| will be available for C. Until then, we'll have to live with |
| reference counts. |
| |
| While Python uses the traditional reference counting implementation, |
| it also offers a cycle detector that works to detect reference |
| cycles. This allows applications to not worry about creating direct |
| or indirect circular references; these are the weakness of garbage |
| collection implemented using only reference counting. Reference |
| cycles consist of objects which contain (possibly indirect) references |
| to themselves, so that each object in the cycle has a reference count |
| which is non-zero. Typical reference counting implementations are not |
| able to reclaim the memory belonging to any objects in a reference |
| cycle, or referenced from the objects in the cycle, even though there |
| are no further references to the cycle itself. |
| |
| The cycle detector is able to detect garbage cycles and can reclaim |
| them so long as there are no finalizers implemented in Python |
| (\method{__del__()} methods). When there are such finalizers, the |
| detector exposes the cycles through the \ulink{\module{gc} |
| module}{../lib/module-gc.html} (specifically, the \code{garbage} |
| variable in that module). The \module{gc} module also exposes a way |
| to run the detector (the \function{collect()} function), as well as |
| configuration interfaces and the ability to disable the detector at |
| runtime. The cycle detector is considered an optional component; |
| though it is included by default, it can be disabled at build time |
| using the \longprogramopt{without-cycle-gc} option to the |
| \program{configure} script on \UNIX{} platforms (including Mac OS X) |
| or by removing the definition of \code{WITH_CYCLE_GC} in the |
| \file{pyconfig.h} header on other platforms. If the cycle detector is |
| disabled in this way, the \module{gc} module will not be available. |
| |
| |
| \subsection{Reference Counting in Python |
| \label{refcountsInPython}} |
| |
| There are two macros, \code{Py_INCREF(x)} and \code{Py_DECREF(x)}, |
| which handle the incrementing and decrementing of the reference count. |
| \cfunction{Py_DECREF()} also frees the object when the count reaches zero. |
| For flexibility, it doesn't call \cfunction{free()} directly --- rather, it |
| makes a call through a function pointer in the object's \dfn{type |
| object}. For this purpose (and others), every object also contains a |
| pointer to its type object. |
| |
| The big question now remains: when to use \code{Py_INCREF(x)} and |
| \code{Py_DECREF(x)}? Let's first introduce some terms. Nobody |
| ``owns'' an object; however, you can \dfn{own a reference} to an |
| object. An object's reference count is now defined as the number of |
| owned references to it. The owner of a reference is responsible for |
| calling \cfunction{Py_DECREF()} when the reference is no longer |
| needed. Ownership of a reference can be transferred. There are three |
| ways to dispose of an owned reference: pass it on, store it, or call |
| \cfunction{Py_DECREF()}. Forgetting to dispose of an owned reference |
| creates a memory leak. |
| |
| It is also possible to \dfn{borrow}\footnote{The metaphor of |
| ``borrowing'' a reference is not completely correct: the owner still |
| has a copy of the reference.} a reference to an object. The borrower |
| of a reference should not call \cfunction{Py_DECREF()}. The borrower must |
| not hold on to the object longer than the owner from which it was |
| borrowed. Using a borrowed reference after the owner has disposed of |
| it risks using freed memory and should be avoided |
| completely.\footnote{Checking that the reference count is at least 1 |
| \strong{does not work} --- the reference count itself could be in |
| freed memory and may thus be reused for another object!} |
| |
| The advantage of borrowing over owning a reference is that you don't |
| need to take care of disposing of the reference on all possible paths |
| through the code --- in other words, with a borrowed reference you |
| don't run the risk of leaking when a premature exit is taken. The |
| disadvantage of borrowing over leaking is that there are some subtle |
| situations where in seemingly correct code a borrowed reference can be |
| used after the owner from which it was borrowed has in fact disposed |
| of it. |
| |
| A borrowed reference can be changed into an owned reference by calling |
| \cfunction{Py_INCREF()}. This does not affect the status of the owner from |
| which the reference was borrowed --- it creates a new owned reference, |
| and gives full owner responsibilities (the new owner must |
| dispose of the reference properly, as well as the previous owner). |
| |
| |
| \subsection{Ownership Rules |
| \label{ownershipRules}} |
| |
| Whenever an object reference is passed into or out of a function, it |
| is part of the function's interface specification whether ownership is |
| transferred with the reference or not. |
| |
| Most functions that return a reference to an object pass on ownership |
| with the reference. In particular, all functions whose function it is |
| to create a new object, such as \cfunction{PyInt_FromLong()} and |
| \cfunction{Py_BuildValue()}, pass ownership to the receiver. Even if |
| the object is not actually new, you still receive ownership of a new |
| reference to that object. For instance, \cfunction{PyInt_FromLong()} |
| maintains a cache of popular values and can return a reference to a |
| cached item. |
| |
| Many functions that extract objects from other objects also transfer |
| ownership with the reference, for instance |
| \cfunction{PyObject_GetAttrString()}. The picture is less clear, here, |
| however, since a few common routines are exceptions: |
| \cfunction{PyTuple_GetItem()}, \cfunction{PyList_GetItem()}, |
| \cfunction{PyDict_GetItem()}, and \cfunction{PyDict_GetItemString()} |
| all return references that you borrow from the tuple, list or |
| dictionary. |
| |
| The function \cfunction{PyImport_AddModule()} also returns a borrowed |
| reference, even though it may actually create the object it returns: |
| this is possible because an owned reference to the object is stored in |
| \code{sys.modules}. |
| |
| When you pass an object reference into another function, in general, |
| the function borrows the reference from you --- if it needs to store |
| it, it will use \cfunction{Py_INCREF()} to become an independent |
| owner. There are exactly two important exceptions to this rule: |
| \cfunction{PyTuple_SetItem()} and \cfunction{PyList_SetItem()}. These |
| functions take over ownership of the item passed to them --- even if |
| they fail! (Note that \cfunction{PyDict_SetItem()} and friends don't |
| take over ownership --- they are ``normal.'') |
| |
| When a C function is called from Python, it borrows references to its |
| arguments from the caller. The caller owns a reference to the object, |
| so the borrowed reference's lifetime is guaranteed until the function |
| returns. Only when such a borrowed reference must be stored or passed |
| on, it must be turned into an owned reference by calling |
| \cfunction{Py_INCREF()}. |
| |
| The object reference returned from a C function that is called from |
| Python must be an owned reference --- ownership is tranferred from the |
| function to its caller. |
| |
| |
| \subsection{Thin Ice |
| \label{thinIce}} |
| |
| There are a few situations where seemingly harmless use of a borrowed |
| reference can lead to problems. These all have to do with implicit |
| invocations of the interpreter, which can cause the owner of a |
| reference to dispose of it. |
| |
| The first and most important case to know about is using |
| \cfunction{Py_DECREF()} on an unrelated object while borrowing a |
| reference to a list item. For instance: |
| |
| \begin{verbatim} |
| void |
| bug(PyObject *list) |
| { |
| PyObject *item = PyList_GetItem(list, 0); |
| |
| PyList_SetItem(list, 1, PyInt_FromLong(0L)); |
| PyObject_Print(item, stdout, 0); /* BUG! */ |
| } |
| \end{verbatim} |
| |
| This function first borrows a reference to \code{list[0]}, then |
| replaces \code{list[1]} with the value \code{0}, and finally prints |
| the borrowed reference. Looks harmless, right? But it's not! |
| |
| Let's follow the control flow into \cfunction{PyList_SetItem()}. The list |
| owns references to all its items, so when item 1 is replaced, it has |
| to dispose of the original item 1. Now let's suppose the original |
| item 1 was an instance of a user-defined class, and let's further |
| suppose that the class defined a \method{__del__()} method. If this |
| class instance has a reference count of 1, disposing of it will call |
| its \method{__del__()} method. |
| |
| Since it is written in Python, the \method{__del__()} method can execute |
| arbitrary Python code. Could it perhaps do something to invalidate |
| the reference to \code{item} in \cfunction{bug()}? You bet! Assuming |
| that the list passed into \cfunction{bug()} is accessible to the |
| \method{__del__()} method, it could execute a statement to the effect of |
| \samp{del list[0]}, and assuming this was the last reference to that |
| object, it would free the memory associated with it, thereby |
| invalidating \code{item}. |
| |
| The solution, once you know the source of the problem, is easy: |
| temporarily increment the reference count. The correct version of the |
| function reads: |
| |
| \begin{verbatim} |
| void |
| no_bug(PyObject *list) |
| { |
| PyObject *item = PyList_GetItem(list, 0); |
| |
| Py_INCREF(item); |
| PyList_SetItem(list, 1, PyInt_FromLong(0L)); |
| PyObject_Print(item, stdout, 0); |
| Py_DECREF(item); |
| } |
| \end{verbatim} |
| |
| This is a true story. An older version of Python contained variants |
| of this bug and someone spent a considerable amount of time in a C |
| debugger to figure out why his \method{__del__()} methods would fail... |
| |
| The second case of problems with a borrowed reference is a variant |
| involving threads. Normally, multiple threads in the Python |
| interpreter can't get in each other's way, because there is a global |
| lock protecting Python's entire object space. However, it is possible |
| to temporarily release this lock using the macro |
| \csimplemacro{Py_BEGIN_ALLOW_THREADS}, and to re-acquire it using |
| \csimplemacro{Py_END_ALLOW_THREADS}. This is common around blocking |
| I/O calls, to let other threads use the processor while waiting for |
| the I/O to complete. Obviously, the following function has the same |
| problem as the previous one: |
| |
| \begin{verbatim} |
| void |
| bug(PyObject *list) |
| { |
| PyObject *item = PyList_GetItem(list, 0); |
| Py_BEGIN_ALLOW_THREADS |
| ...some blocking I/O call... |
| Py_END_ALLOW_THREADS |
| PyObject_Print(item, stdout, 0); /* BUG! */ |
| } |
| \end{verbatim} |
| |
| |
| \subsection{NULL Pointers |
| \label{nullPointers}} |
| |
| In general, functions that take object references as arguments do not |
| expect you to pass them \NULL{} pointers, and will dump core (or |
| cause later core dumps) if you do so. Functions that return object |
| references generally return \NULL{} only to indicate that an |
| exception occurred. The reason for not testing for \NULL{} |
| arguments is that functions often pass the objects they receive on to |
| other function --- if each function were to test for \NULL, |
| there would be a lot of redundant tests and the code would run more |
| slowly. |
| |
| It is better to test for \NULL{} only at the ``source:'' when a |
| pointer that may be \NULL{} is received, for example, from |
| \cfunction{malloc()} or from a function that may raise an exception. |
| |
| The macros \cfunction{Py_INCREF()} and \cfunction{Py_DECREF()} |
| do not check for \NULL{} pointers --- however, their variants |
| \cfunction{Py_XINCREF()} and \cfunction{Py_XDECREF()} do. |
| |
| The macros for checking for a particular object type |
| (\code{Py\var{type}_Check()}) don't check for \NULL{} pointers --- |
| again, there is much code that calls several of these in a row to test |
| an object against various different expected types, and this would |
| generate redundant tests. There are no variants with \NULL{} |
| checking. |
| |
| The C function calling mechanism guarantees that the argument list |
| passed to C functions (\code{args} in the examples) is never |
| \NULL{} --- in fact it guarantees that it is always a tuple.\footnote{ |
| These guarantees don't hold when you use the ``old'' style |
| calling convention --- this is still found in much existing code.} |
| |
| It is a severe error to ever let a \NULL{} pointer ``escape'' to |
| the Python user. |
| |
| % Frank Stajano: |
| % A pedagogically buggy example, along the lines of the previous listing, |
| % would be helpful here -- showing in more concrete terms what sort of |
| % actions could cause the problem. I can't very well imagine it from the |
| % description. |
| |
| |
| \section{Writing Extensions in \Cpp |
| \label{cplusplus}} |
| |
| It is possible to write extension modules in \Cpp. Some restrictions |
| apply. If the main program (the Python interpreter) is compiled and |
| linked by the C compiler, global or static objects with constructors |
| cannot be used. This is not a problem if the main program is linked |
| by the \Cpp{} compiler. Functions that will be called by the |
| Python interpreter (in particular, module initalization functions) |
| have to be declared using \code{extern "C"}. |
| It is unnecessary to enclose the Python header files in |
| \code{extern "C" \{...\}} --- they use this form already if the symbol |
| \samp{__cplusplus} is defined (all recent \Cpp{} compilers define this |
| symbol). |
| |
| |
| \section{Providing a C API for an Extension Module |
| \label{using-cobjects}} |
| \sectionauthor{Konrad Hinsen}{hinsen@cnrs-orleans.fr} |
| |
| Many extension modules just provide new functions and types to be |
| used from Python, but sometimes the code in an extension module can |
| be useful for other extension modules. For example, an extension |
| module could implement a type ``collection'' which works like lists |
| without order. Just like the standard Python list type has a C API |
| which permits extension modules to create and manipulate lists, this |
| new collection type should have a set of C functions for direct |
| manipulation from other extension modules. |
| |
| At first sight this seems easy: just write the functions (without |
| declaring them \keyword{static}, of course), provide an appropriate |
| header file, and document the C API. And in fact this would work if |
| all extension modules were always linked statically with the Python |
| interpreter. When modules are used as shared libraries, however, the |
| symbols defined in one module may not be visible to another module. |
| The details of visibility depend on the operating system; some systems |
| use one global namespace for the Python interpreter and all extension |
| modules (Windows, for example), whereas others require an explicit |
| list of imported symbols at module link time (AIX is one example), or |
| offer a choice of different strategies (most Unices). And even if |
| symbols are globally visible, the module whose functions one wishes to |
| call might not have been loaded yet! |
| |
| Portability therefore requires not to make any assumptions about |
| symbol visibility. This means that all symbols in extension modules |
| should be declared \keyword{static}, except for the module's |
| initialization function, in order to avoid name clashes with other |
| extension modules (as discussed in section~\ref{methodTable}). And it |
| means that symbols that \emph{should} be accessible from other |
| extension modules must be exported in a different way. |
| |
| Python provides a special mechanism to pass C-level information |
| (pointers) from one extension module to another one: CObjects. |
| A CObject is a Python data type which stores a pointer (\ctype{void |
| *}). CObjects can only be created and accessed via their C API, but |
| they can be passed around like any other Python object. In particular, |
| they can be assigned to a name in an extension module's namespace. |
| Other extension modules can then import this module, retrieve the |
| value of this name, and then retrieve the pointer from the CObject. |
| |
| There are many ways in which CObjects can be used to export the C API |
| of an extension module. Each name could get its own CObject, or all C |
| API pointers could be stored in an array whose address is published in |
| a CObject. And the various tasks of storing and retrieving the pointers |
| can be distributed in different ways between the module providing the |
| code and the client modules. |
| |
| The following example demonstrates an approach that puts most of the |
| burden on the writer of the exporting module, which is appropriate |
| for commonly used library modules. It stores all C API pointers |
| (just one in the example!) in an array of \ctype{void} pointers which |
| becomes the value of a CObject. The header file corresponding to |
| the module provides a macro that takes care of importing the module |
| and retrieving its C API pointers; client modules only have to call |
| this macro before accessing the C API. |
| |
| The exporting module is a modification of the \module{spam} module from |
| section~\ref{simpleExample}. The function \function{spam.system()} |
| does not call the C library function \cfunction{system()} directly, |
| but a function \cfunction{PySpam_System()}, which would of course do |
| something more complicated in reality (such as adding ``spam'' to |
| every command). This function \cfunction{PySpam_System()} is also |
| exported to other extension modules. |
| |
| The function \cfunction{PySpam_System()} is a plain C function, |
| declared \keyword{static} like everything else: |
| |
| \begin{verbatim} |
| static int |
| PySpam_System(char *command) |
| { |
| return system(command); |
| } |
| \end{verbatim} |
| |
| The function \cfunction{spam_system()} is modified in a trivial way: |
| |
| \begin{verbatim} |
| static PyObject * |
| spam_system(PyObject *self, PyObject *args) |
| { |
| char *command; |
| int sts; |
| |
| if (!PyArg_ParseTuple(args, "s", &command)) |
| return NULL; |
| sts = PySpam_System(command); |
| return Py_BuildValue("i", sts); |
| } |
| \end{verbatim} |
| |
| In the beginning of the module, right after the line |
| |
| \begin{verbatim} |
| #include "Python.h" |
| \end{verbatim} |
| |
| two more lines must be added: |
| |
| \begin{verbatim} |
| #define SPAM_MODULE |
| #include "spammodule.h" |
| \end{verbatim} |
| |
| The \code{\#define} is used to tell the header file that it is being |
| included in the exporting module, not a client module. Finally, |
| the module's initialization function must take care of initializing |
| the C API pointer array: |
| |
| \begin{verbatim} |
| PyMODINIT_FUNC |
| initspam(void) |
| { |
| PyObject *m; |
| static void *PySpam_API[PySpam_API_pointers]; |
| PyObject *c_api_object; |
| |
| m = Py_InitModule("spam", SpamMethods); |
| |
| /* Initialize the C API pointer array */ |
| PySpam_API[PySpam_System_NUM] = (void *)PySpam_System; |
| |
| /* Create a CObject containing the API pointer array's address */ |
| c_api_object = PyCObject_FromVoidPtr((void *)PySpam_API, NULL); |
| |
| if (c_api_object != NULL) |
| PyModule_AddObject(m, "_C_API", c_api_object); |
| } |
| \end{verbatim} |
| |
| Note that \code{PySpam_API} is declared \keyword{static}; otherwise |
| the pointer array would disappear when \function{initspam()} terminates! |
| |
| The bulk of the work is in the header file \file{spammodule.h}, |
| which looks like this: |
| |
| \begin{verbatim} |
| #ifndef Py_SPAMMODULE_H |
| #define Py_SPAMMODULE_H |
| #ifdef __cplusplus |
| extern "C" { |
| #endif |
| |
| /* Header file for spammodule */ |
| |
| /* C API functions */ |
| #define PySpam_System_NUM 0 |
| #define PySpam_System_RETURN int |
| #define PySpam_System_PROTO (char *command) |
| |
| /* Total number of C API pointers */ |
| #define PySpam_API_pointers 1 |
| |
| |
| #ifdef SPAM_MODULE |
| /* This section is used when compiling spammodule.c */ |
| |
| static PySpam_System_RETURN PySpam_System PySpam_System_PROTO; |
| |
| #else |
| /* This section is used in modules that use spammodule's API */ |
| |
| static void **PySpam_API; |
| |
| #define PySpam_System \ |
| (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM]) |
| |
| /* Return -1 and set exception on error, 0 on success. */ |
| static int |
| import_spam(void) |
| { |
| PyObject *module = PyImport_ImportModule("spam"); |
| |
| if (module != NULL) { |
| PyObject *c_api_object = PyObject_GetAttrString(module, "_C_API"); |
| if (c_api_object == NULL) |
| return -1; |
| if (PyCObject_Check(c_api_object)) |
| PySpam_API = (void **)PyCObject_AsVoidPtr(c_api_object); |
| Py_DECREF(c_api_object); |
| } |
| return 0; |
| } |
| |
| #endif |
| |
| #ifdef __cplusplus |
| } |
| #endif |
| |
| #endif /* !defined(Py_SPAMMODULE_H) */ |
| \end{verbatim} |
| |
| All that a client module must do in order to have access to the |
| function \cfunction{PySpam_System()} is to call the function (or |
| rather macro) \cfunction{import_spam()} in its initialization |
| function: |
| |
| \begin{verbatim} |
| PyMODINIT_FUNC |
| initclient(void) |
| { |
| PyObject *m; |
| |
| Py_InitModule("client", ClientMethods); |
| if (import_spam() < 0) |
| return; |
| /* additional initialization can happen here */ |
| } |
| \end{verbatim} |
| |
| The main disadvantage of this approach is that the file |
| \file{spammodule.h} is rather complicated. However, the |
| basic structure is the same for each function that is |
| exported, so it has to be learned only once. |
| |
| Finally it should be mentioned that CObjects offer additional |
| functionality, which is especially useful for memory allocation and |
| deallocation of the pointer stored in a CObject. The details |
| are described in the \citetitle[../api/api.html]{Python/C API |
| Reference Manual} in the section |
| ``\ulink{CObjects}{../api/cObjects.html}'' and in the implementation |
| of CObjects (files \file{Include/cobject.h} and |
| \file{Objects/cobject.c} in the Python source code distribution). |