blob: 4f3cabf01ab8b4ac09eddad8414c7b01f85aae4f [file] [log] [blame]
Georg Brandl6728c5a2009-10-11 18:31:23 +00001=======================
2Extending/Embedding FAQ
3=======================
4
Georg Brandl44ea77b2013-03-28 13:28:44 +01005.. only:: html
6
7 .. contents::
Georg Brandl6728c5a2009-10-11 18:31:23 +00008
9.. highlight:: c
10
11
12Can I create my own functions in C?
13-----------------------------------
14
15Yes, you can create built-in modules containing functions, variables, exceptions
16and even new types in C. This is explained in the document
17:ref:`extending-index`.
18
19Most intermediate or advanced Python books will also cover this topic.
20
21
22Can I create my own functions in C++?
23-------------------------------------
24
25Yes, using the C compatibility features found in C++. Place ``extern "C" {
26... }`` around the Python include files and put ``extern "C"`` before each
27function that is going to be called by the Python interpreter. Global or static
28C++ objects with constructors are probably not a good idea.
29
30
Georg Brandle0289a32010-08-01 21:44:38 +000031.. _c-wrapper-software:
32
Georg Brandl6728c5a2009-10-11 18:31:23 +000033Writing C is hard; are there any alternatives?
34----------------------------------------------
35
36There are a number of alternatives to writing your own C extensions, depending
37on what you're trying to do.
38
39.. XXX make sure these all work; mention Cython
40
41If you need more speed, `Psyco <http://psyco.sourceforge.net/>`_ generates x86
42assembly code from Python bytecode. You can use Psyco to compile the most
43time-critical functions in your code, and gain a significant improvement with
44very little effort, as long as you're running on a machine with an
45x86-compatible processor.
46
47`Pyrex <http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/>`_ is a compiler
48that accepts a slightly modified form of Python and generates the corresponding
49C code. Pyrex makes it possible to write an extension without having to learn
50Python's C API.
51
52If you need to interface to some C or C++ library for which no Python extension
53currently exists, you can try wrapping the library's data types and functions
54with a tool such as `SWIG <http://www.swig.org>`_. `SIP
Georg Brandl4ebf8072009-10-22 07:56:56 +000055<http://www.riverbankcomputing.co.uk/software/sip/>`__, `CXX
Georg Brandl6728c5a2009-10-11 18:31:23 +000056<http://cxx.sourceforge.net/>`_ `Boost
57<http://www.boost.org/libs/python/doc/index.html>`_, or `Weave
Ezio Melotti425aa2e2010-04-05 12:51:45 +000058<http://www.scipy.org/Weave>`_ are also alternatives for wrapping
Georg Brandl6728c5a2009-10-11 18:31:23 +000059C++ libraries.
60
61
62How can I execute arbitrary Python statements from C?
63-----------------------------------------------------
64
Sandro Tosi98ed08f2012-01-14 16:42:02 +010065The highest-level function to do this is :c:func:`PyRun_SimpleString` which takes
Georg Brandl6728c5a2009-10-11 18:31:23 +000066a single string argument to be executed in the context of the module
67``__main__`` and returns 0 for success and -1 when an exception occurred
68(including ``SyntaxError``). If you want more control, use
Sandro Tosi98ed08f2012-01-14 16:42:02 +010069:c:func:`PyRun_String`; see the source for :c:func:`PyRun_SimpleString` in
Georg Brandl6728c5a2009-10-11 18:31:23 +000070``Python/pythonrun.c``.
71
72
73How can I evaluate an arbitrary Python expression from C?
74---------------------------------------------------------
75
Sandro Tosi98ed08f2012-01-14 16:42:02 +010076Call the function :c:func:`PyRun_String` from the previous question with the
77start symbol :c:data:`Py_eval_input`; it parses an expression, evaluates it and
Georg Brandl6728c5a2009-10-11 18:31:23 +000078returns its value.
79
80
81How do I extract C values from a Python object?
82-----------------------------------------------
83
Sandro Tosi98ed08f2012-01-14 16:42:02 +010084That depends on the object's type. If it's a tuple, :c:func:`PyTuple_Size`
85returns its length and :c:func:`PyTuple_GetItem` returns the item at a specified
86index. Lists have similar functions, :c:func:`PyListSize` and
87:c:func:`PyList_GetItem`.
Georg Brandl6728c5a2009-10-11 18:31:23 +000088
Sandro Tosi98ed08f2012-01-14 16:42:02 +010089For strings, :c:func:`PyString_Size` returns its length and
90:c:func:`PyString_AsString` a pointer to its value. Note that Python strings may
91contain null bytes so C's :c:func:`strlen` should not be used.
Georg Brandl6728c5a2009-10-11 18:31:23 +000092
93To test the type of an object, first make sure it isn't *NULL*, and then use
Sandro Tosi98ed08f2012-01-14 16:42:02 +010094:c:func:`PyString_Check`, :c:func:`PyTuple_Check`, :c:func:`PyList_Check`, etc.
Georg Brandl6728c5a2009-10-11 18:31:23 +000095
96There is also a high-level API to Python objects which is provided by the
97so-called 'abstract' interface -- read ``Include/abstract.h`` for further
98details. It allows interfacing with any kind of Python sequence using calls
Sandro Tosi98ed08f2012-01-14 16:42:02 +010099like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc.) as well as
Georg Brandl6728c5a2009-10-11 18:31:23 +0000100many other useful protocols.
101
102
103How do I use Py_BuildValue() to create a tuple of arbitrary length?
104-------------------------------------------------------------------
105
106You can't. Use ``t = PyTuple_New(n)`` instead, and fill it with objects using
107``PyTuple_SetItem(t, i, o)`` -- note that this "eats" a reference count of
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100108``o``, so you have to :c:func:`Py_INCREF` it. Lists have similar functions
Georg Brandl6728c5a2009-10-11 18:31:23 +0000109``PyList_New(n)`` and ``PyList_SetItem(l, i, o)``. Note that you *must* set all
110the tuple items to some value before you pass the tuple to Python code --
111``PyTuple_New(n)`` initializes them to NULL, which isn't a valid Python value.
112
113
114How do I call an object's method from C?
115----------------------------------------
116
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100117The :c:func:`PyObject_CallMethod` function can be used to call an arbitrary
Georg Brandl6728c5a2009-10-11 18:31:23 +0000118method of an object. The parameters are the object, the name of the method to
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100119call, a format string like that used with :c:func:`Py_BuildValue`, and the
Georg Brandl6728c5a2009-10-11 18:31:23 +0000120argument values::
121
122 PyObject *
123 PyObject_CallMethod(PyObject *object, char *method_name,
124 char *arg_format, ...);
125
126This works for any object that has methods -- whether built-in or user-defined.
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100127You are responsible for eventually :c:func:`Py_DECREF`\ 'ing the return value.
Georg Brandl6728c5a2009-10-11 18:31:23 +0000128
129To call, e.g., a file object's "seek" method with arguments 10, 0 (assuming the
130file object pointer is "f")::
131
132 res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0);
133 if (res == NULL) {
134 ... an exception occurred ...
135 }
136 else {
137 Py_DECREF(res);
138 }
139
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100140Note that since :c:func:`PyObject_CallObject` *always* wants a tuple for the
Georg Brandl6728c5a2009-10-11 18:31:23 +0000141argument list, to call a function without arguments, pass "()" for the format,
142and to call a function with one argument, surround the argument in parentheses,
143e.g. "(i)".
144
145
146How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)?
147----------------------------------------------------------------------------------------
148
149In Python code, define an object that supports the ``write()`` method. Assign
150this object to :data:`sys.stdout` and :data:`sys.stderr`. Call print_error, or
151just allow the standard traceback mechanism to work. Then, the output will go
152wherever your ``write()`` method sends it.
153
154The easiest way to do this is to use the StringIO class in the standard library.
155
156Sample code and use for catching stdout:
157
158 >>> class StdoutCatcher:
159 ... def __init__(self):
160 ... self.data = ''
161 ... def write(self, stuff):
162 ... self.data = self.data + stuff
163 ...
164 >>> import sys
165 >>> sys.stdout = StdoutCatcher()
166 >>> print 'foo'
167 >>> print 'hello world!'
168 >>> sys.stderr.write(sys.stdout.data)
169 foo
170 hello world!
171
172
173How do I access a module written in Python from C?
174--------------------------------------------------
175
176You can get a pointer to the module object as follows::
177
178 module = PyImport_ImportModule("<modulename>");
179
180If the module hasn't been imported yet (i.e. it is not yet present in
181:data:`sys.modules`), this initializes the module; otherwise it simply returns
182the value of ``sys.modules["<modulename>"]``. Note that it doesn't enter the
183module into any namespace -- it only ensures it has been initialized and is
184stored in :data:`sys.modules`.
185
186You can then access the module's attributes (i.e. any name defined in the
187module) as follows::
188
189 attr = PyObject_GetAttrString(module, "<attrname>");
190
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100191Calling :c:func:`PyObject_SetAttrString` to assign to variables in the module
Georg Brandl6728c5a2009-10-11 18:31:23 +0000192also works.
193
194
195How do I interface to C++ objects from Python?
196----------------------------------------------
197
198Depending on your requirements, there are many approaches. To do this manually,
199begin by reading :ref:`the "Extending and Embedding" document
200<extending-index>`. Realize that for the Python run-time system, there isn't a
201whole lot of difference between C and C++ -- so the strategy of building a new
202Python type around a C structure (pointer) type will also work for C++ objects.
203
Georg Brandle0289a32010-08-01 21:44:38 +0000204For C++ libraries, see :ref:`c-wrapper-software`.
Georg Brandl6728c5a2009-10-11 18:31:23 +0000205
206
207I added a module using the Setup file and the make fails; why?
208--------------------------------------------------------------
209
210Setup must end in a newline, if there is no newline there, the build process
211fails. (Fixing this requires some ugly shell script hackery, and this bug is so
212minor that it doesn't seem worth the effort.)
213
214
215How do I debug an extension?
216----------------------------
217
218When using GDB with dynamically loaded extensions, you can't set a breakpoint in
219your extension until your extension is loaded.
220
221In your ``.gdbinit`` file (or interactively), add the command::
222
223 br _PyImport_LoadDynamicModule
224
225Then, when you run GDB::
226
227 $ gdb /local/bin/python
228 gdb) run myscript.py
229 gdb) continue # repeat until your extension is loaded
230 gdb) finish # so that your extension is loaded
231 gdb) br myfunction.c:50
232 gdb) continue
233
234I want to compile a Python module on my Linux system, but some files are missing. Why?
235--------------------------------------------------------------------------------------
236
237Most packaged versions of Python don't include the
238:file:`/usr/lib/python2.{x}/config/` directory, which contains various files
239required for compiling Python extensions.
240
241For Red Hat, install the python-devel RPM to get the necessary files.
242
243For Debian, run ``apt-get install python-dev``.
244
245
246What does "SystemError: _PyImport_FixupExtension: module yourmodule not loaded" mean?
247-------------------------------------------------------------------------------------
248
249This means that you have created an extension module named "yourmodule", but
250your module init function does not initialize with that name.
251
252Every module init function will have a line similar to::
253
254 module = Py_InitModule("yourmodule", yourmodule_functions);
255
256If the string passed to this function is not the same name as your extension
257module, the :exc:`SystemError` exception will be raised.
258
259
260How do I tell "incomplete input" from "invalid input"?
261------------------------------------------------------
262
263Sometimes you want to emulate the Python interactive interpreter's behavior,
264where it gives you a continuation prompt when the input is incomplete (e.g. you
265typed the start of an "if" statement or you didn't close your parentheses or
266triple string quotes), but it gives you a syntax error message immediately when
267the input is invalid.
268
269In Python you can use the :mod:`codeop` module, which approximates the parser's
270behavior sufficiently. IDLE uses this, for example.
271
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100272The easiest way to do it in C is to call :c:func:`PyRun_InteractiveLoop` (perhaps
Georg Brandl6728c5a2009-10-11 18:31:23 +0000273in a separate thread) and let the Python interpreter handle the input for
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100274you. You can also set the :c:func:`PyOS_ReadlineFunctionPointer` to point at your
Georg Brandl6728c5a2009-10-11 18:31:23 +0000275custom input function. See ``Modules/readline.c`` and ``Parser/myreadline.c``
276for more hints.
277
278However sometimes you have to run the embedded Python interpreter in the same
279thread as your rest application and you can't allow the
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100280:c:func:`PyRun_InteractiveLoop` to stop while waiting for user input. The one
281solution then is to call :c:func:`PyParser_ParseString` and test for ``e.error``
Georg Brandl6728c5a2009-10-11 18:31:23 +0000282equal to ``E_EOF``, which means the input is incomplete). Here's a sample code
283fragment, untested, inspired by code from Alex Farber::
284
285 #include <Python.h>
286 #include <node.h>
287 #include <errcode.h>
288 #include <grammar.h>
289 #include <parsetok.h>
290 #include <compile.h>
291
292 int testcomplete(char *code)
293 /* code should end in \n */
294 /* return -1 for error, 0 for incomplete, 1 for complete */
295 {
296 node *n;
297 perrdetail e;
298
299 n = PyParser_ParseString(code, &_PyParser_Grammar,
300 Py_file_input, &e);
301 if (n == NULL) {
302 if (e.error == E_EOF)
303 return 0;
304 return -1;
305 }
306
307 PyNode_Free(n);
308 return 1;
309 }
310
311Another solution is trying to compile the received string with
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100312:c:func:`Py_CompileString`. If it compiles without errors, try to execute the
313returned code object by calling :c:func:`PyEval_EvalCode`. Otherwise save the
Georg Brandl6728c5a2009-10-11 18:31:23 +0000314input for later. If the compilation fails, find out if it's an error or just
315more input is required - by extracting the message string from the exception
316tuple and comparing it to the string "unexpected EOF while parsing". Here is a
317complete example using the GNU readline library (you may want to ignore
318**SIGINT** while calling readline())::
319
320 #include <stdio.h>
321 #include <readline.h>
322
323 #include <Python.h>
324 #include <object.h>
325 #include <compile.h>
326 #include <eval.h>
327
328 int main (int argc, char* argv[])
329 {
330 int i, j, done = 0; /* lengths of line, code */
331 char ps1[] = ">>> ";
332 char ps2[] = "... ";
333 char *prompt = ps1;
334 char *msg, *line, *code = NULL;
335 PyObject *src, *glb, *loc;
336 PyObject *exc, *val, *trb, *obj, *dum;
337
338 Py_Initialize ();
339 loc = PyDict_New ();
340 glb = PyDict_New ();
341 PyDict_SetItemString (glb, "__builtins__", PyEval_GetBuiltins ());
342
343 while (!done)
344 {
345 line = readline (prompt);
346
347 if (NULL == line) /* CTRL-D pressed */
348 {
349 done = 1;
350 }
351 else
352 {
353 i = strlen (line);
354
355 if (i > 0)
356 add_history (line); /* save non-empty lines */
357
358 if (NULL == code) /* nothing in code yet */
359 j = 0;
360 else
361 j = strlen (code);
362
363 code = realloc (code, i + j + 2);
364 if (NULL == code) /* out of memory */
365 exit (1);
366
367 if (0 == j) /* code was empty, so */
368 code[0] = '\0'; /* keep strncat happy */
369
370 strncat (code, line, i); /* append line to code */
371 code[i + j] = '\n'; /* append '\n' to code */
372 code[i + j + 1] = '\0';
373
374 src = Py_CompileString (code, "<stdin>", Py_single_input);
375
376 if (NULL != src) /* compiled just fine - */
377 {
378 if (ps1 == prompt || /* ">>> " or */
379 '\n' == code[i + j - 1]) /* "... " and double '\n' */
380 { /* so execute it */
381 dum = PyEval_EvalCode ((PyCodeObject *)src, glb, loc);
382 Py_XDECREF (dum);
383 Py_XDECREF (src);
384 free (code);
385 code = NULL;
386 if (PyErr_Occurred ())
387 PyErr_Print ();
388 prompt = ps1;
389 }
390 } /* syntax error or E_EOF? */
391 else if (PyErr_ExceptionMatches (PyExc_SyntaxError))
392 {
393 PyErr_Fetch (&exc, &val, &trb); /* clears exception! */
394
395 if (PyArg_ParseTuple (val, "sO", &msg, &obj) &&
396 !strcmp (msg, "unexpected EOF while parsing")) /* E_EOF */
397 {
398 Py_XDECREF (exc);
399 Py_XDECREF (val);
400 Py_XDECREF (trb);
401 prompt = ps2;
402 }
403 else /* some other syntax error */
404 {
405 PyErr_Restore (exc, val, trb);
406 PyErr_Print ();
407 free (code);
408 code = NULL;
409 prompt = ps1;
410 }
411 }
412 else /* some non-syntax error */
413 {
414 PyErr_Print ();
415 free (code);
416 code = NULL;
417 prompt = ps1;
418 }
419
420 free (line);
421 }
422 }
423
424 Py_XDECREF(glb);
425 Py_XDECREF(loc);
426 Py_Finalize();
427 exit(0);
428 }
429
430
431How do I find undefined g++ symbols __builtin_new or __pure_virtual?
432--------------------------------------------------------------------
433
434To dynamically load g++ extension modules, you must recompile Python, relink it
Ezio Melotti062d2b52009-12-19 22:41:49 +0000435using g++ (change LINKCC in the Python Modules Makefile), and link your
Georg Brandl6728c5a2009-10-11 18:31:23 +0000436extension module using g++ (e.g., ``g++ -shared -o mymodule.so mymodule.o``).
437
438
439Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)?
440----------------------------------------------------------------------------------------------------------------
441
Georg Brandl6f82cd32010-02-06 18:44:44 +0000442In Python 2.2, you can inherit from built-in classes such as :class:`int`,
Georg Brandl6728c5a2009-10-11 18:31:23 +0000443:class:`list`, :class:`dict`, etc.
444
445The Boost Python Library (BPL, http://www.boost.org/libs/python/doc/index.html)
446provides a way of doing this from C++ (i.e. you can inherit from an extension
447class written in C++ using the BPL).
448
449
450When importing module X, why do I get "undefined symbol: PyUnicodeUCS2*"?
451-------------------------------------------------------------------------
452
453You are using a version of Python that uses a 4-byte representation for Unicode
454characters, but some C extension module you are importing was compiled using a
455Python that uses a 2-byte representation for Unicode characters (the default).
456
457If instead the name of the undefined symbol starts with ``PyUnicodeUCS4``, the
458problem is the reverse: Python was built using 2-byte Unicode characters, and
459the extension module was compiled using a Python with 4-byte Unicode characters.
460
461This can easily occur when using pre-built extension packages. RedHat Linux
4627.x, in particular, provided a "python2" binary that is compiled with 4-byte
463Unicode. This only causes the link failure if the extension uses any of the
464``PyUnicode_*()`` functions. It is also a problem if an extension uses any of
Sandro Tosi98ed08f2012-01-14 16:42:02 +0100465the Unicode-related format specifiers for :c:func:`Py_BuildValue` (or similar) or
466parameter specifications for :c:func:`PyArg_ParseTuple`.
Georg Brandl6728c5a2009-10-11 18:31:23 +0000467
468You can check the size of the Unicode character a Python interpreter is using by
469checking the value of sys.maxunicode:
470
471 >>> import sys
472 >>> if sys.maxunicode > 65535:
473 ... print 'UCS4 build'
474 ... else:
475 ... print 'UCS2 build'
476
477The only way to solve this problem is to use extension modules compiled with a
478Python binary built using the same size for Unicode characters.
479
480
481