| ======================= |
| Extending/Embedding FAQ |
| ======================= |
| |
| .. contents:: |
| |
| .. highlight:: c |
| |
| |
| Can I create my own functions in C? |
| ----------------------------------- |
| |
| Yes, you can create built-in modules containing functions, variables, exceptions |
| and even new types in C. This is explained in the document |
| :ref:`extending-index`. |
| |
| Most intermediate or advanced Python books will also cover this topic. |
| |
| |
| Can I create my own functions in C++? |
| ------------------------------------- |
| |
| Yes, using the C compatibility features found in C++. Place ``extern "C" { |
| ... }`` around the Python include files and put ``extern "C"`` before each |
| function that is going to be called by the Python interpreter. Global or static |
| C++ objects with constructors are probably not a good idea. |
| |
| |
| Writing C is hard; are there any alternatives? |
| ---------------------------------------------- |
| |
| There are a number of alternatives to writing your own C extensions, depending |
| on what you're trying to do. |
| |
| .. XXX make sure these all work; mention Cython |
| |
| If you need more speed, `Psyco <http://psyco.sourceforge.net/>`_ generates x86 |
| assembly code from Python bytecode. You can use Psyco to compile the most |
| time-critical functions in your code, and gain a significant improvement with |
| very little effort, as long as you're running on a machine with an |
| x86-compatible processor. |
| |
| `Pyrex <http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ is a compiler |
| that accepts a slightly modified form of Python and generates the corresponding |
| C code. Pyrex makes it possible to write an extension without having to learn |
| Python's C API. |
| |
| If you need to interface to some C or C++ library for which no Python extension |
| currently exists, you can try wrapping the library's data types and functions |
| with a tool such as `SWIG <http://www.swig.org>`_. `SIP |
| <http://www.riverbankcomputing.co.uk/software/sip/>`__, `CXX |
| <http://cxx.sourceforge.net/>`_ `Boost |
| <http://www.boost.org/libs/python/doc/index.html>`_, or `Weave |
| <http://www.scipy.org/Weave>`_ are also alternatives for wrapping |
| C++ libraries. |
| |
| |
| How can I execute arbitrary Python statements from C? |
| ----------------------------------------------------- |
| |
| The highest-level function to do this is :cfunc:`PyRun_SimpleString` which takes |
| a single string argument to be executed in the context of the module |
| ``__main__`` and returns 0 for success and -1 when an exception occurred |
| (including ``SyntaxError``). If you want more control, use |
| :cfunc:`PyRun_String`; see the source for :cfunc:`PyRun_SimpleString` in |
| ``Python/pythonrun.c``. |
| |
| |
| How can I evaluate an arbitrary Python expression from C? |
| --------------------------------------------------------- |
| |
| Call the function :cfunc:`PyRun_String` from the previous question with the |
| start symbol :cdata:`Py_eval_input`; it parses an expression, evaluates it and |
| returns its value. |
| |
| |
| How do I extract C values from a Python object? |
| ----------------------------------------------- |
| |
| That depends on the object's type. If it's a tuple, :cfunc:`PyTuple_Size` |
| returns its length and :cfunc:`PyTuple_GetItem` returns the item at a specified |
| index. Lists have similar functions, :cfunc:`PyListSize` and |
| :cfunc:`PyList_GetItem`. |
| |
| For strings, :cfunc:`PyString_Size` returns its length and |
| :cfunc:`PyString_AsString` a pointer to its value. Note that Python strings may |
| contain null bytes so C's :cfunc:`strlen` should not be used. |
| |
| To test the type of an object, first make sure it isn't *NULL*, and then use |
| :cfunc:`PyString_Check`, :cfunc:`PyTuple_Check`, :cfunc:`PyList_Check`, etc. |
| |
| There is also a high-level API to Python objects which is provided by the |
| so-called 'abstract' interface -- read ``Include/abstract.h`` for further |
| details. It allows interfacing with any kind of Python sequence using calls |
| like :cfunc:`PySequence_Length`, :cfunc:`PySequence_GetItem`, etc.) as well as |
| many other useful protocols. |
| |
| |
| How do I use Py_BuildValue() to create a tuple of arbitrary length? |
| ------------------------------------------------------------------- |
| |
| You can't. Use ``t = PyTuple_New(n)`` instead, and fill it with objects using |
| ``PyTuple_SetItem(t, i, o)`` -- note that this "eats" a reference count of |
| ``o``, so you have to :cfunc:`Py_INCREF` it. Lists have similar functions |
| ``PyList_New(n)`` and ``PyList_SetItem(l, i, o)``. Note that you *must* set all |
| the tuple items to some value before you pass the tuple to Python code -- |
| ``PyTuple_New(n)`` initializes them to NULL, which isn't a valid Python value. |
| |
| |
| How do I call an object's method from C? |
| ---------------------------------------- |
| |
| The :cfunc:`PyObject_CallMethod` function can be used to call an arbitrary |
| method of an object. The parameters are the object, the name of the method to |
| call, a format string like that used with :cfunc:`Py_BuildValue`, and the |
| argument values:: |
| |
| PyObject * |
| PyObject_CallMethod(PyObject *object, char *method_name, |
| char *arg_format, ...); |
| |
| This works for any object that has methods -- whether built-in or user-defined. |
| You are responsible for eventually :cfunc:`Py_DECREF`\ 'ing the return value. |
| |
| To call, e.g., a file object's "seek" method with arguments 10, 0 (assuming the |
| file object pointer is "f"):: |
| |
| res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0); |
| if (res == NULL) { |
| ... an exception occurred ... |
| } |
| else { |
| Py_DECREF(res); |
| } |
| |
| Note that since :cfunc:`PyObject_CallObject` *always* wants a tuple for the |
| argument list, to call a function without arguments, pass "()" for the format, |
| and to call a function with one argument, surround the argument in parentheses, |
| e.g. "(i)". |
| |
| |
| How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)? |
| ---------------------------------------------------------------------------------------- |
| |
| In Python code, define an object that supports the ``write()`` method. Assign |
| this object to :data:`sys.stdout` and :data:`sys.stderr`. Call print_error, or |
| just allow the standard traceback mechanism to work. Then, the output will go |
| wherever your ``write()`` method sends it. |
| |
| The easiest way to do this is to use the StringIO class in the standard library. |
| |
| Sample code and use for catching stdout: |
| |
| >>> class StdoutCatcher: |
| ... def __init__(self): |
| ... self.data = '' |
| ... def write(self, stuff): |
| ... self.data = self.data + stuff |
| ... |
| >>> import sys |
| >>> sys.stdout = StdoutCatcher() |
| >>> print 'foo' |
| >>> print 'hello world!' |
| >>> sys.stderr.write(sys.stdout.data) |
| foo |
| hello world! |
| |
| |
| How do I access a module written in Python from C? |
| -------------------------------------------------- |
| |
| You can get a pointer to the module object as follows:: |
| |
| module = PyImport_ImportModule("<modulename>"); |
| |
| If the module hasn't been imported yet (i.e. it is not yet present in |
| :data:`sys.modules`), this initializes the module; otherwise it simply returns |
| the value of ``sys.modules["<modulename>"]``. Note that it doesn't enter the |
| module into any namespace -- it only ensures it has been initialized and is |
| stored in :data:`sys.modules`. |
| |
| You can then access the module's attributes (i.e. any name defined in the |
| module) as follows:: |
| |
| attr = PyObject_GetAttrString(module, "<attrname>"); |
| |
| Calling :cfunc:`PyObject_SetAttrString` to assign to variables in the module |
| also works. |
| |
| |
| How do I interface to C++ objects from Python? |
| ---------------------------------------------- |
| |
| Depending on your requirements, there are many approaches. To do this manually, |
| begin by reading :ref:`the "Extending and Embedding" document |
| <extending-index>`. Realize that for the Python run-time system, there isn't a |
| whole lot of difference between C and C++ -- so the strategy of building a new |
| Python type around a C structure (pointer) type will also work for C++ objects. |
| |
| For C++ libraries, you can look at `SIP |
| <http://www.riverbankcomputing.co.uk/software/sip/>`_, `CXX |
| <http://cxx.sourceforge.net/>`_, `Boost |
| <http://www.boost.org/libs/python/doc/index.html>`_, `Weave |
| <http://www.scipy.org/site_content/weave>`_ or `SWIG <http://www.swig.org>`_ |
| |
| |
| I added a module using the Setup file and the make fails; why? |
| -------------------------------------------------------------- |
| |
| Setup must end in a newline, if there is no newline there, the build process |
| fails. (Fixing this requires some ugly shell script hackery, and this bug is so |
| minor that it doesn't seem worth the effort.) |
| |
| |
| How do I debug an extension? |
| ---------------------------- |
| |
| When using GDB with dynamically loaded extensions, you can't set a breakpoint in |
| your extension until your extension is loaded. |
| |
| In your ``.gdbinit`` file (or interactively), add the command:: |
| |
| br _PyImport_LoadDynamicModule |
| |
| Then, when you run GDB:: |
| |
| $ gdb /local/bin/python |
| gdb) run myscript.py |
| gdb) continue # repeat until your extension is loaded |
| gdb) finish # so that your extension is loaded |
| gdb) br myfunction.c:50 |
| gdb) continue |
| |
| I want to compile a Python module on my Linux system, but some files are missing. Why? |
| -------------------------------------------------------------------------------------- |
| |
| Most packaged versions of Python don't include the |
| :file:`/usr/lib/python2.{x}/config/` directory, which contains various files |
| required for compiling Python extensions. |
| |
| For Red Hat, install the python-devel RPM to get the necessary files. |
| |
| For Debian, run ``apt-get install python-dev``. |
| |
| |
| What does "SystemError: _PyImport_FixupExtension: module yourmodule not loaded" mean? |
| ------------------------------------------------------------------------------------- |
| |
| This means that you have created an extension module named "yourmodule", but |
| your module init function does not initialize with that name. |
| |
| Every module init function will have a line similar to:: |
| |
| module = Py_InitModule("yourmodule", yourmodule_functions); |
| |
| If the string passed to this function is not the same name as your extension |
| module, the :exc:`SystemError` exception will be raised. |
| |
| |
| How do I tell "incomplete input" from "invalid input"? |
| ------------------------------------------------------ |
| |
| Sometimes you want to emulate the Python interactive interpreter's behavior, |
| where it gives you a continuation prompt when the input is incomplete (e.g. you |
| typed the start of an "if" statement or you didn't close your parentheses or |
| triple string quotes), but it gives you a syntax error message immediately when |
| the input is invalid. |
| |
| In Python you can use the :mod:`codeop` module, which approximates the parser's |
| behavior sufficiently. IDLE uses this, for example. |
| |
| The easiest way to do it in C is to call :cfunc:`PyRun_InteractiveLoop` (perhaps |
| in a separate thread) and let the Python interpreter handle the input for |
| you. You can also set the :cfunc:`PyOS_ReadlineFunctionPointer` to point at your |
| custom input function. See ``Modules/readline.c`` and ``Parser/myreadline.c`` |
| for more hints. |
| |
| However sometimes you have to run the embedded Python interpreter in the same |
| thread as your rest application and you can't allow the |
| :cfunc:`PyRun_InteractiveLoop` to stop while waiting for user input. The one |
| solution then is to call :cfunc:`PyParser_ParseString` and test for ``e.error`` |
| equal to ``E_EOF``, which means the input is incomplete). Here's a sample code |
| fragment, untested, inspired by code from Alex Farber:: |
| |
| #include <Python.h> |
| #include <node.h> |
| #include <errcode.h> |
| #include <grammar.h> |
| #include <parsetok.h> |
| #include <compile.h> |
| |
| int testcomplete(char *code) |
| /* code should end in \n */ |
| /* return -1 for error, 0 for incomplete, 1 for complete */ |
| { |
| node *n; |
| perrdetail e; |
| |
| n = PyParser_ParseString(code, &_PyParser_Grammar, |
| Py_file_input, &e); |
| if (n == NULL) { |
| if (e.error == E_EOF) |
| return 0; |
| return -1; |
| } |
| |
| PyNode_Free(n); |
| return 1; |
| } |
| |
| Another solution is trying to compile the received string with |
| :cfunc:`Py_CompileString`. If it compiles without errors, try to execute the |
| returned code object by calling :cfunc:`PyEval_EvalCode`. Otherwise save the |
| input for later. If the compilation fails, find out if it's an error or just |
| more input is required - by extracting the message string from the exception |
| tuple and comparing it to the string "unexpected EOF while parsing". Here is a |
| complete example using the GNU readline library (you may want to ignore |
| **SIGINT** while calling readline()):: |
| |
| #include <stdio.h> |
| #include <readline.h> |
| |
| #include <Python.h> |
| #include <object.h> |
| #include <compile.h> |
| #include <eval.h> |
| |
| int main (int argc, char* argv[]) |
| { |
| int i, j, done = 0; /* lengths of line, code */ |
| char ps1[] = ">>> "; |
| char ps2[] = "... "; |
| char *prompt = ps1; |
| char *msg, *line, *code = NULL; |
| PyObject *src, *glb, *loc; |
| PyObject *exc, *val, *trb, *obj, *dum; |
| |
| Py_Initialize (); |
| loc = PyDict_New (); |
| glb = PyDict_New (); |
| PyDict_SetItemString (glb, "__builtins__", PyEval_GetBuiltins ()); |
| |
| while (!done) |
| { |
| line = readline (prompt); |
| |
| if (NULL == line) /* CTRL-D pressed */ |
| { |
| done = 1; |
| } |
| else |
| { |
| i = strlen (line); |
| |
| if (i > 0) |
| add_history (line); /* save non-empty lines */ |
| |
| if (NULL == code) /* nothing in code yet */ |
| j = 0; |
| else |
| j = strlen (code); |
| |
| code = realloc (code, i + j + 2); |
| if (NULL == code) /* out of memory */ |
| exit (1); |
| |
| if (0 == j) /* code was empty, so */ |
| code[0] = '\0'; /* keep strncat happy */ |
| |
| strncat (code, line, i); /* append line to code */ |
| code[i + j] = '\n'; /* append '\n' to code */ |
| code[i + j + 1] = '\0'; |
| |
| src = Py_CompileString (code, "<stdin>", Py_single_input); |
| |
| if (NULL != src) /* compiled just fine - */ |
| { |
| if (ps1 == prompt || /* ">>> " or */ |
| '\n' == code[i + j - 1]) /* "... " and double '\n' */ |
| { /* so execute it */ |
| dum = PyEval_EvalCode ((PyCodeObject *)src, glb, loc); |
| Py_XDECREF (dum); |
| Py_XDECREF (src); |
| free (code); |
| code = NULL; |
| if (PyErr_Occurred ()) |
| PyErr_Print (); |
| prompt = ps1; |
| } |
| } /* syntax error or E_EOF? */ |
| else if (PyErr_ExceptionMatches (PyExc_SyntaxError)) |
| { |
| PyErr_Fetch (&exc, &val, &trb); /* clears exception! */ |
| |
| if (PyArg_ParseTuple (val, "sO", &msg, &obj) && |
| !strcmp (msg, "unexpected EOF while parsing")) /* E_EOF */ |
| { |
| Py_XDECREF (exc); |
| Py_XDECREF (val); |
| Py_XDECREF (trb); |
| prompt = ps2; |
| } |
| else /* some other syntax error */ |
| { |
| PyErr_Restore (exc, val, trb); |
| PyErr_Print (); |
| free (code); |
| code = NULL; |
| prompt = ps1; |
| } |
| } |
| else /* some non-syntax error */ |
| { |
| PyErr_Print (); |
| free (code); |
| code = NULL; |
| prompt = ps1; |
| } |
| |
| free (line); |
| } |
| } |
| |
| Py_XDECREF(glb); |
| Py_XDECREF(loc); |
| Py_Finalize(); |
| exit(0); |
| } |
| |
| |
| How do I find undefined g++ symbols __builtin_new or __pure_virtual? |
| -------------------------------------------------------------------- |
| |
| To dynamically load g++ extension modules, you must recompile Python, relink it |
| using g++ (change LINKCC in the Python Modules Makefile), and link your |
| extension module using g++ (e.g., ``g++ -shared -o mymodule.so mymodule.o``). |
| |
| |
| Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)? |
| ---------------------------------------------------------------------------------------------------------------- |
| |
| In Python 2.2, you can inherit from builtin classes such as :class:`int`, |
| :class:`list`, :class:`dict`, etc. |
| |
| The Boost Python Library (BPL, http://www.boost.org/libs/python/doc/index.html) |
| provides a way of doing this from C++ (i.e. you can inherit from an extension |
| class written in C++ using the BPL). |
| |
| |
| When importing module X, why do I get "undefined symbol: PyUnicodeUCS2*"? |
| ------------------------------------------------------------------------- |
| |
| You are using a version of Python that uses a 4-byte representation for Unicode |
| characters, but some C extension module you are importing was compiled using a |
| Python that uses a 2-byte representation for Unicode characters (the default). |
| |
| If instead the name of the undefined symbol starts with ``PyUnicodeUCS4``, the |
| problem is the reverse: Python was built using 2-byte Unicode characters, and |
| the extension module was compiled using a Python with 4-byte Unicode characters. |
| |
| This can easily occur when using pre-built extension packages. RedHat Linux |
| 7.x, in particular, provided a "python2" binary that is compiled with 4-byte |
| Unicode. This only causes the link failure if the extension uses any of the |
| ``PyUnicode_*()`` functions. It is also a problem if an extension uses any of |
| the Unicode-related format specifiers for :cfunc:`Py_BuildValue` (or similar) or |
| parameter specifications for :cfunc:`PyArg_ParseTuple`. |
| |
| You can check the size of the Unicode character a Python interpreter is using by |
| checking the value of sys.maxunicode: |
| |
| >>> import sys |
| >>> if sys.maxunicode > 65535: |
| ... print 'UCS4 build' |
| ... else: |
| ... print 'UCS2 build' |
| |
| The only way to solve this problem is to use extension modules compiled with a |
| Python binary built using the same size for Unicode characters. |
| |
| |
| |