blob: ccd8b285b2fc24c79474652b68050ff56a006dab [file] [log] [blame]
Georg Brandl6728c5a2009-10-11 18:31:23 +00001=======================
2Extending/Embedding FAQ
3=======================
4
Georg Brandl44ea77b2013-03-28 13:28:44 +01005.. only:: html
6
7 .. contents::
Georg Brandl6728c5a2009-10-11 18:31:23 +00008
9.. highlight:: c
10
11
12Can I create my own functions in C?
13-----------------------------------
14
15Yes, you can create built-in modules containing functions, variables, exceptions
16and even new types in C. This is explained in the document
17:ref:`extending-index`.
18
19Most intermediate or advanced Python books will also cover this topic.
20
21
22Can I create my own functions in C++?
23-------------------------------------
24
25Yes, using the C compatibility features found in C++. Place ``extern "C" {
26... }`` around the Python include files and put ``extern "C"`` before each
27function that is going to be called by the Python interpreter. Global or static
28C++ objects with constructors are probably not a good idea.
29
30
Georg Brandle0289a32010-08-01 21:44:38 +000031.. _c-wrapper-software:
32
Georg Brandl6728c5a2009-10-11 18:31:23 +000033Writing C is hard; are there any alternatives?
34----------------------------------------------
35
36There are a number of alternatives to writing your own C extensions, depending
37on what you're trying to do.
38
39.. XXX make sure these all work; mention Cython
40
41If you need more speed, `Psyco <http://psyco.sourceforge.net/>`_ generates x86
42assembly code from Python bytecode. You can use Psyco to compile the most
43time-critical functions in your code, and gain a significant improvement with
44very little effort, as long as you're running on a machine with an
45x86-compatible processor.
46
Georg Brandlfa55a312014-10-29 09:24:54 +010047`Cython <http://cython.org>`_ and its relative `Pyrex
48<http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ are compilers
49that accept a slightly modified form of Python and generate the corresponding
Georg Brandl6728c5a2009-10-11 18:31:23 +000050C code. Pyrex makes it possible to write an extension without having to learn
51Python's C API.
52
53If you need to interface to some C or C++ library for which no Python extension
54currently exists, you can try wrapping the library's data types and functions
55with a tool such as `SWIG <http://www.swig.org>`_. `SIP
Georg Brandl0f5d6c02014-10-29 10:57:37 +010056<http://www.riverbankcomputing.co.uk/software/sip/intro>`__, `CXX
Georg Brandl6728c5a2009-10-11 18:31:23 +000057<http://cxx.sourceforge.net/>`_ `Boost
58<http://www.boost.org/libs/python/doc/index.html>`_, or `Weave
Georg Brandlfa55a312014-10-29 09:24:54 +010059<http://docs.scipy.org/doc/scipy-dev/reference/tutorial/weave.html>`_ are also
60alternatives for wrapping C++ libraries.
Georg Brandl6728c5a2009-10-11 18:31:23 +000061
62
63How can I execute arbitrary Python statements from C?
64-----------------------------------------------------
65
Sandro Tosi98ed08f2012-01-14 16:42:02 +010066The highest-level function to do this is :c:func:`PyRun_SimpleString` which takes
Georg Brandl6728c5a2009-10-11 18:31:23 +000067a single string argument to be executed in the context of the module
68``__main__`` and returns 0 for success and -1 when an exception occurred
69(including ``SyntaxError``). If you want more control, use
Sandro Tosi98ed08f2012-01-14 16:42:02 +010070:c:func:`PyRun_String`; see the source for :c:func:`PyRun_SimpleString` in
Georg Brandl6728c5a2009-10-11 18:31:23 +000071``Python/pythonrun.c``.
72
73
74How can I evaluate an arbitrary Python expression from C?
75---------------------------------------------------------
76
Sandro Tosi98ed08f2012-01-14 16:42:02 +010077Call the function :c:func:`PyRun_String` from the previous question with the
78start symbol :c:data:`Py_eval_input`; it parses an expression, evaluates it and
Georg Brandl6728c5a2009-10-11 18:31:23 +000079returns its value.
80
81
82How do I extract C values from a Python object?
83-----------------------------------------------
84
Sandro Tosi98ed08f2012-01-14 16:42:02 +010085That depends on the object's type. If it's a tuple, :c:func:`PyTuple_Size`
86returns its length and :c:func:`PyTuple_GetItem` returns the item at a specified
87index. Lists have similar functions, :c:func:`PyListSize` and
88:c:func:`PyList_GetItem`.
Georg Brandl6728c5a2009-10-11 18:31:23 +000089
Sandro Tosi98ed08f2012-01-14 16:42:02 +010090For strings, :c:func:`PyString_Size` returns its length and
91:c:func:`PyString_AsString` a pointer to its value. Note that Python strings may
92contain null bytes so C's :c:func:`strlen` should not be used.
Georg Brandl6728c5a2009-10-11 18:31:23 +000093
94To test the type of an object, first make sure it isn't *NULL*, and then use
Sandro Tosi98ed08f2012-01-14 16:42:02 +010095:c:func:`PyString_Check`, :c:func:`PyTuple_Check`, :c:func:`PyList_Check`, etc.
Georg Brandl6728c5a2009-10-11 18:31:23 +000096
97There is also a high-level API to Python objects which is provided by the
98so-called 'abstract' interface -- read ``Include/abstract.h`` for further
99details. It allows interfacing with any kind of Python sequence using calls
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100100like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc.) as well as
Georg Brandl6728c5a2009-10-11 18:31:23 +0000101many other useful protocols.
102
103
104How do I use Py_BuildValue() to create a tuple of arbitrary length?
105-------------------------------------------------------------------
106
107You can't. Use ``t = PyTuple_New(n)`` instead, and fill it with objects using
108``PyTuple_SetItem(t, i, o)`` -- note that this "eats" a reference count of
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100109``o``, so you have to :c:func:`Py_INCREF` it. Lists have similar functions
Georg Brandl6728c5a2009-10-11 18:31:23 +0000110``PyList_New(n)`` and ``PyList_SetItem(l, i, o)``. Note that you *must* set all
111the tuple items to some value before you pass the tuple to Python code --
112``PyTuple_New(n)`` initializes them to NULL, which isn't a valid Python value.
113
114
115How do I call an object's method from C?
116----------------------------------------
117
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100118The :c:func:`PyObject_CallMethod` function can be used to call an arbitrary
Georg Brandl6728c5a2009-10-11 18:31:23 +0000119method of an object. The parameters are the object, the name of the method to
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100120call, a format string like that used with :c:func:`Py_BuildValue`, and the
Georg Brandl6728c5a2009-10-11 18:31:23 +0000121argument values::
122
123 PyObject *
124 PyObject_CallMethod(PyObject *object, char *method_name,
125 char *arg_format, ...);
126
127This works for any object that has methods -- whether built-in or user-defined.
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100128You are responsible for eventually :c:func:`Py_DECREF`\ 'ing the return value.
Georg Brandl6728c5a2009-10-11 18:31:23 +0000129
130To call, e.g., a file object's "seek" method with arguments 10, 0 (assuming the
131file object pointer is "f")::
132
133 res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0);
134 if (res == NULL) {
135 ... an exception occurred ...
136 }
137 else {
138 Py_DECREF(res);
139 }
140
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100141Note that since :c:func:`PyObject_CallObject` *always* wants a tuple for the
Georg Brandl6728c5a2009-10-11 18:31:23 +0000142argument list, to call a function without arguments, pass "()" for the format,
143and to call a function with one argument, surround the argument in parentheses,
144e.g. "(i)".
145
146
147How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)?
148----------------------------------------------------------------------------------------
149
150In Python code, define an object that supports the ``write()`` method. Assign
151this object to :data:`sys.stdout` and :data:`sys.stderr`. Call print_error, or
152just allow the standard traceback mechanism to work. Then, the output will go
153wherever your ``write()`` method sends it.
154
155The easiest way to do this is to use the StringIO class in the standard library.
156
157Sample code and use for catching stdout:
158
159 >>> class StdoutCatcher:
160 ... def __init__(self):
161 ... self.data = ''
162 ... def write(self, stuff):
163 ... self.data = self.data + stuff
164 ...
165 >>> import sys
166 >>> sys.stdout = StdoutCatcher()
167 >>> print 'foo'
168 >>> print 'hello world!'
169 >>> sys.stderr.write(sys.stdout.data)
170 foo
171 hello world!
172
173
174How do I access a module written in Python from C?
175--------------------------------------------------
176
177You can get a pointer to the module object as follows::
178
179 module = PyImport_ImportModule("<modulename>");
180
181If the module hasn't been imported yet (i.e. it is not yet present in
182:data:`sys.modules`), this initializes the module; otherwise it simply returns
183the value of ``sys.modules["<modulename>"]``. Note that it doesn't enter the
184module into any namespace -- it only ensures it has been initialized and is
185stored in :data:`sys.modules`.
186
187You can then access the module's attributes (i.e. any name defined in the
188module) as follows::
189
190 attr = PyObject_GetAttrString(module, "<attrname>");
191
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100192Calling :c:func:`PyObject_SetAttrString` to assign to variables in the module
Georg Brandl6728c5a2009-10-11 18:31:23 +0000193also works.
194
195
196How do I interface to C++ objects from Python?
197----------------------------------------------
198
199Depending on your requirements, there are many approaches. To do this manually,
200begin by reading :ref:`the "Extending and Embedding" document
201<extending-index>`. Realize that for the Python run-time system, there isn't a
202whole lot of difference between C and C++ -- so the strategy of building a new
203Python type around a C structure (pointer) type will also work for C++ objects.
204
Georg Brandle0289a32010-08-01 21:44:38 +0000205For C++ libraries, see :ref:`c-wrapper-software`.
Georg Brandl6728c5a2009-10-11 18:31:23 +0000206
207
208I added a module using the Setup file and the make fails; why?
209--------------------------------------------------------------
210
211Setup must end in a newline, if there is no newline there, the build process
212fails. (Fixing this requires some ugly shell script hackery, and this bug is so
213minor that it doesn't seem worth the effort.)
214
215
216How do I debug an extension?
217----------------------------
218
219When using GDB with dynamically loaded extensions, you can't set a breakpoint in
220your extension until your extension is loaded.
221
222In your ``.gdbinit`` file (or interactively), add the command::
223
224 br _PyImport_LoadDynamicModule
225
226Then, when you run GDB::
227
228 $ gdb /local/bin/python
229 gdb) run myscript.py
230 gdb) continue # repeat until your extension is loaded
231 gdb) finish # so that your extension is loaded
232 gdb) br myfunction.c:50
233 gdb) continue
234
235I want to compile a Python module on my Linux system, but some files are missing. Why?
236--------------------------------------------------------------------------------------
237
238Most packaged versions of Python don't include the
239:file:`/usr/lib/python2.{x}/config/` directory, which contains various files
240required for compiling Python extensions.
241
242For Red Hat, install the python-devel RPM to get the necessary files.
243
244For Debian, run ``apt-get install python-dev``.
245
246
247What does "SystemError: _PyImport_FixupExtension: module yourmodule not loaded" mean?
248-------------------------------------------------------------------------------------
249
250This means that you have created an extension module named "yourmodule", but
251your module init function does not initialize with that name.
252
253Every module init function will have a line similar to::
254
255 module = Py_InitModule("yourmodule", yourmodule_functions);
256
257If the string passed to this function is not the same name as your extension
258module, the :exc:`SystemError` exception will be raised.
259
260
261How do I tell "incomplete input" from "invalid input"?
262------------------------------------------------------
263
264Sometimes you want to emulate the Python interactive interpreter's behavior,
265where it gives you a continuation prompt when the input is incomplete (e.g. you
266typed the start of an "if" statement or you didn't close your parentheses or
267triple string quotes), but it gives you a syntax error message immediately when
268the input is invalid.
269
270In Python you can use the :mod:`codeop` module, which approximates the parser's
271behavior sufficiently. IDLE uses this, for example.
272
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100273The easiest way to do it in C is to call :c:func:`PyRun_InteractiveLoop` (perhaps
Georg Brandl6728c5a2009-10-11 18:31:23 +0000274in a separate thread) and let the Python interpreter handle the input for
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100275you. You can also set the :c:func:`PyOS_ReadlineFunctionPointer` to point at your
Georg Brandl6728c5a2009-10-11 18:31:23 +0000276custom input function. See ``Modules/readline.c`` and ``Parser/myreadline.c``
277for more hints.
278
279However sometimes you have to run the embedded Python interpreter in the same
280thread as your rest application and you can't allow the
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100281:c:func:`PyRun_InteractiveLoop` to stop while waiting for user input. The one
282solution then is to call :c:func:`PyParser_ParseString` and test for ``e.error``
Georg Brandl6728c5a2009-10-11 18:31:23 +0000283equal to ``E_EOF``, which means the input is incomplete). Here's a sample code
284fragment, untested, inspired by code from Alex Farber::
285
286 #include <Python.h>
287 #include <node.h>
288 #include <errcode.h>
289 #include <grammar.h>
290 #include <parsetok.h>
291 #include <compile.h>
292
293 int testcomplete(char *code)
294 /* code should end in \n */
295 /* return -1 for error, 0 for incomplete, 1 for complete */
296 {
297 node *n;
298 perrdetail e;
299
300 n = PyParser_ParseString(code, &_PyParser_Grammar,
301 Py_file_input, &e);
302 if (n == NULL) {
303 if (e.error == E_EOF)
304 return 0;
305 return -1;
306 }
307
308 PyNode_Free(n);
309 return 1;
310 }
311
312Another solution is trying to compile the received string with
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100313:c:func:`Py_CompileString`. If it compiles without errors, try to execute the
314returned code object by calling :c:func:`PyEval_EvalCode`. Otherwise save the
Georg Brandl6728c5a2009-10-11 18:31:23 +0000315input for later. If the compilation fails, find out if it's an error or just
316more input is required - by extracting the message string from the exception
317tuple and comparing it to the string "unexpected EOF while parsing". Here is a
318complete example using the GNU readline library (you may want to ignore
319**SIGINT** while calling readline())::
320
321 #include <stdio.h>
322 #include <readline.h>
323
324 #include <Python.h>
325 #include <object.h>
326 #include <compile.h>
327 #include <eval.h>
328
329 int main (int argc, char* argv[])
330 {
331 int i, j, done = 0; /* lengths of line, code */
332 char ps1[] = ">>> ";
333 char ps2[] = "... ";
334 char *prompt = ps1;
335 char *msg, *line, *code = NULL;
336 PyObject *src, *glb, *loc;
337 PyObject *exc, *val, *trb, *obj, *dum;
338
339 Py_Initialize ();
340 loc = PyDict_New ();
341 glb = PyDict_New ();
342 PyDict_SetItemString (glb, "__builtins__", PyEval_GetBuiltins ());
343
344 while (!done)
345 {
346 line = readline (prompt);
347
348 if (NULL == line) /* CTRL-D pressed */
349 {
350 done = 1;
351 }
352 else
353 {
354 i = strlen (line);
355
356 if (i > 0)
357 add_history (line); /* save non-empty lines */
358
359 if (NULL == code) /* nothing in code yet */
360 j = 0;
361 else
362 j = strlen (code);
363
364 code = realloc (code, i + j + 2);
365 if (NULL == code) /* out of memory */
366 exit (1);
367
368 if (0 == j) /* code was empty, so */
369 code[0] = '\0'; /* keep strncat happy */
370
371 strncat (code, line, i); /* append line to code */
372 code[i + j] = '\n'; /* append '\n' to code */
373 code[i + j + 1] = '\0';
374
375 src = Py_CompileString (code, "<stdin>", Py_single_input);
376
377 if (NULL != src) /* compiled just fine - */
378 {
379 if (ps1 == prompt || /* ">>> " or */
380 '\n' == code[i + j - 1]) /* "... " and double '\n' */
381 { /* so execute it */
382 dum = PyEval_EvalCode ((PyCodeObject *)src, glb, loc);
383 Py_XDECREF (dum);
384 Py_XDECREF (src);
385 free (code);
386 code = NULL;
387 if (PyErr_Occurred ())
388 PyErr_Print ();
389 prompt = ps1;
390 }
391 } /* syntax error or E_EOF? */
392 else if (PyErr_ExceptionMatches (PyExc_SyntaxError))
393 {
394 PyErr_Fetch (&exc, &val, &trb); /* clears exception! */
395
396 if (PyArg_ParseTuple (val, "sO", &msg, &obj) &&
397 !strcmp (msg, "unexpected EOF while parsing")) /* E_EOF */
398 {
399 Py_XDECREF (exc);
400 Py_XDECREF (val);
401 Py_XDECREF (trb);
402 prompt = ps2;
403 }
404 else /* some other syntax error */
405 {
406 PyErr_Restore (exc, val, trb);
407 PyErr_Print ();
408 free (code);
409 code = NULL;
410 prompt = ps1;
411 }
412 }
413 else /* some non-syntax error */
414 {
415 PyErr_Print ();
416 free (code);
417 code = NULL;
418 prompt = ps1;
419 }
420
421 free (line);
422 }
423 }
424
425 Py_XDECREF(glb);
426 Py_XDECREF(loc);
427 Py_Finalize();
428 exit(0);
429 }
430
431
432How do I find undefined g++ symbols __builtin_new or __pure_virtual?
433--------------------------------------------------------------------
434
435To dynamically load g++ extension modules, you must recompile Python, relink it
Ezio Melotti062d2b52009-12-19 22:41:49 +0000436using g++ (change LINKCC in the Python Modules Makefile), and link your
Georg Brandl6728c5a2009-10-11 18:31:23 +0000437extension module using g++ (e.g., ``g++ -shared -o mymodule.so mymodule.o``).
438
439
440Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)?
441----------------------------------------------------------------------------------------------------------------
442
Georg Brandl6f82cd32010-02-06 18:44:44 +0000443In Python 2.2, you can inherit from built-in classes such as :class:`int`,
Georg Brandl6728c5a2009-10-11 18:31:23 +0000444:class:`list`, :class:`dict`, etc.
445
446The Boost Python Library (BPL, http://www.boost.org/libs/python/doc/index.html)
447provides a way of doing this from C++ (i.e. you can inherit from an extension
448class written in C++ using the BPL).
449
450
451When importing module X, why do I get "undefined symbol: PyUnicodeUCS2*"?
452-------------------------------------------------------------------------
453
454You are using a version of Python that uses a 4-byte representation for Unicode
455characters, but some C extension module you are importing was compiled using a
456Python that uses a 2-byte representation for Unicode characters (the default).
457
458If instead the name of the undefined symbol starts with ``PyUnicodeUCS4``, the
459problem is the reverse: Python was built using 2-byte Unicode characters, and
460the extension module was compiled using a Python with 4-byte Unicode characters.
461
462This can easily occur when using pre-built extension packages. RedHat Linux
4637.x, in particular, provided a "python2" binary that is compiled with 4-byte
464Unicode. This only causes the link failure if the extension uses any of the
465``PyUnicode_*()`` functions. It is also a problem if an extension uses any of
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100466the Unicode-related format specifiers for :c:func:`Py_BuildValue` (or similar) or
467parameter specifications for :c:func:`PyArg_ParseTuple`.
Georg Brandl6728c5a2009-10-11 18:31:23 +0000468
469You can check the size of the Unicode character a Python interpreter is using by
470checking the value of sys.maxunicode:
471
472 >>> import sys
473 >>> if sys.maxunicode > 65535:
474 ... print 'UCS4 build'
475 ... else:
476 ... print 'UCS2 build'
477
478The only way to solve this problem is to use extension modules compiled with a
479Python binary built using the same size for Unicode characters.
480
481
482