Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | .. highlightlang:: c |
| 2 | |
| 3 | |
| 4 | .. _embedding: |
| 5 | |
| 6 | *************************************** |
| 7 | Embedding Python in Another Application |
| 8 | *************************************** |
| 9 | |
| 10 | The previous chapters discussed how to extend Python, that is, how to extend the |
| 11 | functionality of Python by attaching a library of C functions to it. It is also |
| 12 | possible to do it the other way around: enrich your C/C++ application by |
| 13 | embedding Python in it. Embedding provides your application with the ability to |
| 14 | implement some of the functionality of your application in Python rather than C |
| 15 | or C++. This can be used for many purposes; one example would be to allow users |
| 16 | to tailor the application to their needs by writing some scripts in Python. You |
| 17 | can also use it yourself if some of the functionality can be written in Python |
| 18 | more easily. |
| 19 | |
| 20 | Embedding Python is similar to extending it, but not quite. The difference is |
| 21 | that when you extend Python, the main program of the application is still the |
| 22 | Python interpreter, while if you embed Python, the main program may have nothing |
| 23 | to do with Python --- instead, some parts of the application occasionally call |
| 24 | the Python interpreter to run some Python code. |
| 25 | |
| 26 | So if you are embedding Python, you are providing your own main program. One of |
| 27 | the things this main program has to do is initialize the Python interpreter. At |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 28 | the very least, you have to call the function :c:func:`Py_Initialize`. There are |
Georg Brandl | c575c90 | 2008-09-13 17:46:05 +0000 | [diff] [blame] | 29 | optional calls to pass command line arguments to Python. Then later you can |
| 30 | call the interpreter from any part of the application. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 31 | |
| 32 | There are several different ways to call the interpreter: you can pass a string |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 33 | containing Python statements to :c:func:`PyRun_SimpleString`, or you can pass a |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 34 | stdio file pointer and a file name (for identification in error messages only) |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 35 | to :c:func:`PyRun_SimpleFile`. You can also call the lower-level operations |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 36 | described in the previous chapters to construct and use Python objects. |
| 37 | |
| 38 | A simple demo of embedding Python can be found in the directory |
| 39 | :file:`Demo/embed/` of the source distribution. |
| 40 | |
| 41 | |
| 42 | .. seealso:: |
| 43 | |
| 44 | :ref:`c-api-index` |
| 45 | The details of Python's C interface are given in this manual. A great deal of |
| 46 | necessary information can be found here. |
| 47 | |
| 48 | |
| 49 | .. _high-level-embedding: |
| 50 | |
| 51 | Very High Level Embedding |
| 52 | ========================= |
| 53 | |
| 54 | The simplest form of embedding Python is the use of the very high level |
| 55 | interface. This interface is intended to execute a Python script without needing |
| 56 | to interact with the application directly. This can for example be used to |
| 57 | perform some operation on a file. :: |
| 58 | |
| 59 | #include <Python.h> |
| 60 | |
| 61 | int |
| 62 | main(int argc, char *argv[]) |
| 63 | { |
| 64 | Py_Initialize(); |
| 65 | PyRun_SimpleString("from time import time,ctime\n" |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 66 | "print('Today is', ctime(time()))\n"); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 67 | Py_Finalize(); |
| 68 | return 0; |
| 69 | } |
| 70 | |
| 71 | The above code first initializes the Python interpreter with |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 72 | :c:func:`Py_Initialize`, followed by the execution of a hard-coded Python script |
| 73 | that print the date and time. Afterwards, the :c:func:`Py_Finalize` call shuts |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 74 | the interpreter down, followed by the end of the program. In a real program, |
| 75 | you may want to get the Python script from another source, perhaps a text-editor |
| 76 | routine, a file, or a database. Getting the Python code from a file can better |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 77 | be done by using the :c:func:`PyRun_SimpleFile` function, which saves you the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 78 | trouble of allocating memory space and loading the file contents. |
| 79 | |
| 80 | |
| 81 | .. _lower-level-embedding: |
| 82 | |
| 83 | Beyond Very High Level Embedding: An overview |
| 84 | ============================================= |
| 85 | |
| 86 | The high level interface gives you the ability to execute arbitrary pieces of |
| 87 | Python code from your application, but exchanging data values is quite |
| 88 | cumbersome to say the least. If you want that, you should use lower level calls. |
| 89 | At the cost of having to write more C code, you can achieve almost anything. |
| 90 | |
| 91 | It should be noted that extending Python and embedding Python is quite the same |
| 92 | activity, despite the different intent. Most topics discussed in the previous |
| 93 | chapters are still valid. To show this, consider what the extension code from |
| 94 | Python to C really does: |
| 95 | |
| 96 | #. Convert data values from Python to C, |
| 97 | |
| 98 | #. Perform a function call to a C routine using the converted values, and |
| 99 | |
| 100 | #. Convert the data values from the call from C to Python. |
| 101 | |
| 102 | When embedding Python, the interface code does: |
| 103 | |
| 104 | #. Convert data values from C to Python, |
| 105 | |
| 106 | #. Perform a function call to a Python interface routine using the converted |
| 107 | values, and |
| 108 | |
| 109 | #. Convert the data values from the call from Python to C. |
| 110 | |
| 111 | As you can see, the data conversion steps are simply swapped to accommodate the |
| 112 | different direction of the cross-language transfer. The only difference is the |
| 113 | routine that you call between both data conversions. When extending, you call a |
| 114 | C routine, when embedding, you call a Python routine. |
| 115 | |
| 116 | This chapter will not discuss how to convert data from Python to C and vice |
| 117 | versa. Also, proper use of references and dealing with errors is assumed to be |
| 118 | understood. Since these aspects do not differ from extending the interpreter, |
| 119 | you can refer to earlier chapters for the required information. |
| 120 | |
| 121 | |
| 122 | .. _pure-embedding: |
| 123 | |
| 124 | Pure Embedding |
| 125 | ============== |
| 126 | |
| 127 | The first program aims to execute a function in a Python script. Like in the |
| 128 | section about the very high level interface, the Python interpreter does not |
| 129 | directly interact with the application (but that will change in the next |
| 130 | section). |
| 131 | |
| 132 | The code to run a function defined in a Python script is: |
| 133 | |
| 134 | .. literalinclude:: ../includes/run-func.c |
| 135 | |
| 136 | |
| 137 | This code loads a Python script using ``argv[1]``, and calls the function named |
| 138 | in ``argv[2]``. Its integer arguments are the other values of the ``argv`` |
| 139 | array. If you compile and link this program (let's call the finished executable |
| 140 | :program:`call`), and use it to execute a Python script, such as:: |
| 141 | |
| 142 | def multiply(a,b): |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 143 | print("Will compute", a, "times", b) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 144 | c = 0 |
| 145 | for i in range(0, a): |
| 146 | c = c + b |
| 147 | return c |
| 148 | |
| 149 | then the result should be:: |
| 150 | |
| 151 | $ call multiply multiply 3 2 |
| 152 | Will compute 3 times 2 |
| 153 | Result of call: 6 |
| 154 | |
| 155 | Although the program is quite large for its functionality, most of the code is |
| 156 | for data conversion between Python and C, and for error reporting. The |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 157 | interesting part with respect to embedding Python starts with :: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 158 | |
| 159 | Py_Initialize(); |
| 160 | pName = PyString_FromString(argv[1]); |
| 161 | /* Error checking of pName left out */ |
| 162 | pModule = PyImport_Import(pName); |
| 163 | |
| 164 | After initializing the interpreter, the script is loaded using |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 165 | :c:func:`PyImport_Import`. This routine needs a Python string as its argument, |
| 166 | which is constructed using the :c:func:`PyString_FromString` data conversion |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 167 | routine. :: |
| 168 | |
| 169 | pFunc = PyObject_GetAttrString(pModule, argv[2]); |
| 170 | /* pFunc is a new reference */ |
| 171 | |
| 172 | if (pFunc && PyCallable_Check(pFunc)) { |
| 173 | ... |
| 174 | } |
| 175 | Py_XDECREF(pFunc); |
| 176 | |
| 177 | Once the script is loaded, the name we're looking for is retrieved using |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 178 | :c:func:`PyObject_GetAttrString`. If the name exists, and the object returned is |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 179 | callable, you can safely assume that it is a function. The program then |
| 180 | proceeds by constructing a tuple of arguments as normal. The call to the Python |
| 181 | function is then made with:: |
| 182 | |
| 183 | pValue = PyObject_CallObject(pFunc, pArgs); |
| 184 | |
| 185 | Upon return of the function, ``pValue`` is either *NULL* or it contains a |
| 186 | reference to the return value of the function. Be sure to release the reference |
| 187 | after examining the value. |
| 188 | |
| 189 | |
| 190 | .. _extending-with-embedding: |
| 191 | |
| 192 | Extending Embedded Python |
| 193 | ========================= |
| 194 | |
| 195 | Until now, the embedded Python interpreter had no access to functionality from |
| 196 | the application itself. The Python API allows this by extending the embedded |
| 197 | interpreter. That is, the embedded interpreter gets extended with routines |
| 198 | provided by the application. While it sounds complex, it is not so bad. Simply |
| 199 | forget for a while that the application starts the Python interpreter. Instead, |
| 200 | consider the application to be a set of subroutines, and write some glue code |
| 201 | that gives Python access to those routines, just like you would write a normal |
| 202 | Python extension. For example:: |
| 203 | |
| 204 | static int numargs=0; |
| 205 | |
| 206 | /* Return the number of arguments of the application command line */ |
| 207 | static PyObject* |
| 208 | emb_numargs(PyObject *self, PyObject *args) |
| 209 | { |
| 210 | if(!PyArg_ParseTuple(args, ":numargs")) |
| 211 | return NULL; |
Georg Brandl | c877a7c | 2010-11-26 11:55:48 +0000 | [diff] [blame] | 212 | return PyLong_FromLong(numargs); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 213 | } |
| 214 | |
| 215 | static PyMethodDef EmbMethods[] = { |
| 216 | {"numargs", emb_numargs, METH_VARARGS, |
| 217 | "Return the number of arguments received by the process."}, |
| 218 | {NULL, NULL, 0, NULL} |
| 219 | }; |
| 220 | |
Georg Brandl | 05b482c | 2008-12-07 22:45:56 +0000 | [diff] [blame] | 221 | static PyModuleDef EmbModule = { |
| 222 | PyModuleDef_HEAD_INIT, "emb", NULL, -1, EmbMethods, |
| 223 | NULL, NULL, NULL, NULL |
| 224 | }; |
| 225 | |
Georg Brandl | acc68cc | 2008-12-09 23:48:44 +0000 | [diff] [blame] | 226 | static PyObject* |
| 227 | PyInit_emb(void) |
| 228 | { |
| 229 | return PyModule_Create(&EmbModule); |
| 230 | } |
| 231 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 232 | Insert the above code just above the :c:func:`main` function. Also, insert the |
| 233 | following two statements before the call to :c:func:`Py_Initialize`:: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 234 | |
| 235 | numargs = argc; |
Georg Brandl | acc68cc | 2008-12-09 23:48:44 +0000 | [diff] [blame] | 236 | PyImport_AppendInittab("emb", &PyInit_emb); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 237 | |
| 238 | These two lines initialize the ``numargs`` variable, and make the |
| 239 | :func:`emb.numargs` function accessible to the embedded Python interpreter. |
| 240 | With these extensions, the Python script can do things like :: |
| 241 | |
| 242 | import emb |
Georg Brandl | 6911e3c | 2007-09-04 07:15:32 +0000 | [diff] [blame] | 243 | print("Number of arguments", emb.numargs()) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 244 | |
| 245 | In a real application, the methods will expose an API of the application to |
| 246 | Python. |
| 247 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 248 | .. TODO: threads, code examples do not really behave well if errors happen |
| 249 | (what to watch out for) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 250 | |
| 251 | |
| 252 | .. _embeddingincplusplus: |
| 253 | |
| 254 | Embedding Python in C++ |
| 255 | ======================= |
| 256 | |
| 257 | It is also possible to embed Python in a C++ program; precisely how this is done |
| 258 | will depend on the details of the C++ system used; in general you will need to |
| 259 | write the main program in C++, and use the C++ compiler to compile and link your |
| 260 | program. There is no need to recompile Python itself using C++. |
| 261 | |
| 262 | |
| 263 | .. _link-reqs: |
| 264 | |
| 265 | Linking Requirements |
| 266 | ==================== |
| 267 | |
| 268 | While the :program:`configure` script shipped with the Python sources will |
| 269 | correctly build Python to export the symbols needed by dynamically linked |
| 270 | extensions, this is not automatically inherited by applications which embed the |
| 271 | Python library statically, at least on Unix. This is an issue when the |
| 272 | application is linked to the static runtime library (:file:`libpython.a`) and |
| 273 | needs to load dynamic extensions (implemented as :file:`.so` files). |
| 274 | |
| 275 | The problem is that some entry points are defined by the Python runtime solely |
| 276 | for extension modules to use. If the embedding application does not use any of |
| 277 | these entry points, some linkers will not include those entries in the symbol |
| 278 | table of the finished executable. Some additional options are needed to inform |
| 279 | the linker not to remove these symbols. |
| 280 | |
| 281 | Determining the right options to use for any given platform can be quite |
| 282 | difficult, but fortunately the Python configuration already has those values. |
| 283 | To retrieve them from an installed Python interpreter, start an interactive |
| 284 | interpreter and have a short session like this:: |
| 285 | |
| 286 | >>> import distutils.sysconfig |
| 287 | >>> distutils.sysconfig.get_config_var('LINKFORSHARED') |
| 288 | '-Xlinker -export-dynamic' |
| 289 | |
| 290 | .. index:: module: distutils.sysconfig |
| 291 | |
| 292 | The contents of the string presented will be the options that should be used. |
| 293 | If the string is empty, there's no need to add any additional options. The |
| 294 | :const:`LINKFORSHARED` definition corresponds to the variable of the same name |
| 295 | in Python's top-level :file:`Makefile`. |
| 296 | |