blob: bba1e566b8e6872cb22ce67702031808ff2238b1 [file] [log] [blame]
Dean Moldovan6d2411f2017-04-22 23:24:13 +02001Embedding the interpreter
2#########################
3
4While pybind11 is mainly focused on extending Python using C++, it's also
5possible to do the reverse: embed the Python interpreter into a C++ program.
6All of the other documentation pages still apply here, so refer to them for
7general pybind11 usage. This section will cover a few extra things required
8for embedding.
9
10Getting started
11===============
12
13A basic executable with an embedded interpreter can be created with just a few
14lines of CMake and the ``pybind11::embed`` target, as shown below. For more
15information, see :doc:`/compiling`.
16
17.. code-block:: cmake
18
19 cmake_minimum_required(VERSION 3.0)
20 project(example)
21
22 find_package(pybind11 REQUIRED) # or `add_subdirectory(pybind11)`
23
Dean Moldovan8f6c1292017-05-31 13:48:39 +020024 add_executable(example main.cpp)
Dean Moldovan6d2411f2017-04-22 23:24:13 +020025 target_link_libraries(example PRIVATE pybind11::embed)
26
27The essential structure of the ``main.cpp`` file looks like this:
28
29.. code-block:: cpp
30
31 #include <pybind11/embed.h> // everything needed for embedding
32 namespace py = pybind11;
33
34 int main() {
35 py::scoped_interpreter guard{}; // start the interpreter and keep it alive
36
37 py::print("Hello, World!"); // use the Python API
38 }
39
40The interpreter must be initialized before using any Python API, which includes
41all the functions and classes in pybind11. The RAII guard class `scoped_interpreter`
42takes care of the interpreter lifetime. After the guard is destroyed, the interpreter
43shuts down and clears its memory. No Python functions can be called after this.
44
45Executing Python code
46=====================
47
Dean Moldovan8f6c1292017-05-31 13:48:39 +020048There are a few different ways to run Python code. One option is to use `eval`,
Dean Moldovan6d2411f2017-04-22 23:24:13 +020049`exec` or `eval_file`, as explained in :ref:`eval`. Here is a quick example in
50the context of an executable with an embedded interpreter:
51
52.. code-block:: cpp
53
54 #include <pybind11/embed.h>
55 namespace py = pybind11;
56
57 int main() {
58 py::scoped_interpreter guard{};
59
60 py::exec(R"(
61 kwargs = dict(name="World", number=42)
62 message = "Hello, {name}! The answer is {number}".format(**kwargs)
63 print(message)
64 )");
65 }
66
67Alternatively, similar results can be achieved using pybind11's API (see
68:doc:`/advanced/pycpp/index` for more details).
69
70.. code-block:: cpp
71
72 #include <pybind11/embed.h>
73 namespace py = pybind11;
74 using namespace py::literals;
75
76 int main() {
77 py::scoped_interpreter guard{};
78
79 auto kwargs = py::dict("name"_a="World", "number"_a=42);
80 auto message = "Hello, {name}! The answer is {number}"_s.format(**kwargs);
81 py::print(message);
82 }
83
84The two approaches can also be combined:
85
86.. code-block:: cpp
87
88 #include <pybind11/embed.h>
89 #include <iostream>
90
91 namespace py = pybind11;
92 using namespace py::literals;
93
94 int main() {
95 py::scoped_interpreter guard{};
96
97 auto locals = py::dict("name"_a="World", "number"_a=42);
98 py::exec(R"(
99 message = "Hello, {name}! The answer is {number}".format(**locals())
100 )", py::globals(), locals);
101
102 auto message = locals["message"].cast<std::string>();
103 std::cout << message;
104 }
105
106Importing modules
107=================
108
109Python modules can be imported using `module::import()`:
110
111.. code-block:: cpp
112
113 py::module sys = py::module::import("sys");
114 py::print(sys.attr("path"))
115
116For convenience, the current working directory is included in ``sys.path`` when
117embedding the interpreter. This makes it easy to import local Python files:
118
119.. code-block:: python
120
121 """calc.py located in the working directory"""
122
123 def add(i, j):
124 return i + j
125
126
127.. code-block:: cpp
128
129 py::module calc = py::module::import("calc");
130 py::object result = calc.attr("add")(1, 2);
131 int n = result.cast<int>();
132 assert(n == 3);
133
134
135Adding embedded modules
136=======================
137
138Embedded binary modules can be added using the `PYBIND11_EMBEDDED_MODULE` macro.
139Note that the definition must be placed at global scope. They can be imported
140like any other module.
141
142.. code-block:: cpp
143
144 #include <pybind11/embed.h>
145 namespace py = pybind11;
146
147 PYBIND11_EMBEDDED_MODULE(fast_calc, m) {
148 // `m` is a `py::module` which is used to bind functions and classes
149 m.def("add", [](int i, int j) {
150 return i + j;
151 });
152 }
153
154 int main() {
155 py::scoped_interpreter guard{};
156
157 auto fast_calc = py::module::import("fast_calc");
158 auto result = fast_calc.attr("add")(1, 2).cast<int>();
159 assert(result == 3);
160 }
161
162Unlike extension modules where only a single binary module can be created, on
163the embedded side an unlimited number of modules can be added using multiple
164`PYBIND11_EMBEDDED_MODULE` definitions (as long as they have unique names).
165
166These modules are added to Python's list of builtins, so they can also be
167imported in pure Python files loaded by the interpreter. Everything interacts
168naturally:
169
170.. code-block:: python
171
172 """py_module.py located in the working directory"""
173 import cpp_module
174
175 a = cpp_module.a
176 b = a + 1
177
178
179.. code-block:: cpp
180
181 #include <pybind11/embed.h>
182 namespace py = pybind11;
183
184 PYBIND11_EMBEDDED_MODULE(cpp_module, m) {
185 m.attr("a") = 1
186 }
187
188 int main() {
189 py::scoped_interpreter guard{};
190
191 auto py_module = py::module::import("py_module");
192
193 auto locals = py::dict("fmt"_a="{} + {} = {}", **py_module.attr("__dict__"));
194 assert(locals["a"].cast<int>() == 1);
195 assert(locals["b"].cast<int>() == 2);
196
197 py::exec(R"(
198 c = a + b
199 message = fmt.format(a, b, c)
200 )", py::globals(), locals);
201
202 assert(locals["c"].cast<int>() == 3);
203 assert(locals["message"].cast<std::string>() == "1 + 2 = 3");
204 }
205
206
207Interpreter lifetime
208====================
209
210The Python interpreter shuts down when `scoped_interpreter` is destroyed. After
211this, creating a new instance will restart the interpreter. Alternatively, the
212`initialize_interpreter` / `finalize_interpreter` pair of functions can be used
213to directly set the state at any time.
214
215Modules created with pybind11 can be safely re-initialized after the interpreter
216has been restarted. However, this may not apply to third-party extension modules.
217The issue is that Python itself cannot completely unload extension modules and
218there are several caveats with regard to interpreter restarting. In short, not
219all memory may be freed, either due to Python reference cycles or user-created
220global data. All the details can be found in the CPython documentation.
221
222.. warning::
223
224 Creating two concurrent `scoped_interpreter` guards is a fatal error. So is
225 calling `initialize_interpreter` for a second time after the interpreter
226 has already been initialized.
227
228 Do not use the raw CPython API functions ``Py_Initialize`` and
229 ``Py_Finalize`` as these do not properly handle the lifetime of
230 pybind11's internal data.
231
232
233Sub-interpreter support
234=======================
235
236Creating multiple copies of `scoped_interpreter` is not possible because it
237represents the main Python interpreter. Sub-interpreters are something different
238and they do permit the existence of multiple interpreters. This is an advanced
239feature of the CPython API and should be handled with care. pybind11 does not
240currently offer a C++ interface for sub-interpreters, so refer to the CPython
241documentation for all the details regarding this feature.
242
243We'll just mention a couple of caveats the sub-interpreters support in pybind11:
244
245 1. Sub-interpreters will not receive independent copies of embedded modules.
246 Instead, these are shared and modifications in one interpreter may be
247 reflected in another.
248
249 2. Managing multiple threads, multiple interpreters and the GIL can be
250 challenging and there are several caveats here, even within the pure
251 CPython API (please refer to the Python docs for details). As for
252 pybind11, keep in mind that `gil_scoped_release` and `gil_scoped_acquire`
253 do not take sub-interpreters into account.