blob: 1ef7fdb67226e6c085d777f785d3d59eeededc19 [file] [log] [blame]
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001.. _advanced:
2
3Advanced topics
4###############
5
Wenzel Jakob93296692015-10-13 23:21:54 +02006For brevity, the rest of this chapter assumes that the following two lines are
7present:
8
9.. code-block:: cpp
10
Wenzel Jakob8f4eb002015-10-15 18:13:33 +020011 #include <pybind11/pybind11.h>
Wenzel Jakob93296692015-10-13 23:21:54 +020012
Wenzel Jakob10e62e12015-10-15 22:46:07 +020013 namespace py = pybind11;
Wenzel Jakob93296692015-10-13 23:21:54 +020014
Wenzel Jakobde3ad072016-02-02 11:38:21 +010015Exporting constants and mutable objects
16=======================================
17
18To expose a C++ constant, use the ``attr`` function to register it in a module
19as shown below. The ``int_`` class is one of many small wrapper objects defined
20in ``pybind11/pytypes.h``. General objects (including integers) can also be
21converted using the function ``cast``.
22
23.. code-block:: cpp
24
25 PYBIND11_PLUGIN(example) {
26 py::module m("example", "pybind11 example plugin");
27 m.attr("MY_CONSTANT") = py::int_(123);
28 m.attr("MY_CONSTANT_2") = py::cast(new MyObject());
29 }
30
Wenzel Jakob28f98aa2015-10-13 02:57:16 +020031Operator overloading
32====================
33
Wenzel Jakob93296692015-10-13 23:21:54 +020034Suppose that we're given the following ``Vector2`` class with a vector addition
35and scalar multiplication operation, all implemented using overloaded operators
36in C++.
37
38.. code-block:: cpp
39
40 class Vector2 {
41 public:
42 Vector2(float x, float y) : x(x), y(y) { }
43
Wenzel Jakob93296692015-10-13 23:21:54 +020044 Vector2 operator+(const Vector2 &v) const { return Vector2(x + v.x, y + v.y); }
45 Vector2 operator*(float value) const { return Vector2(x * value, y * value); }
46 Vector2& operator+=(const Vector2 &v) { x += v.x; y += v.y; return *this; }
47 Vector2& operator*=(float v) { x *= v; y *= v; return *this; }
48
Wenzel Jakobf64feaf2016-04-28 14:33:45 +020049 friend Vector2 operator*(float f, const Vector2 &v) {
50 return Vector2(f * v.x, f * v.y);
51 }
Wenzel Jakob93296692015-10-13 23:21:54 +020052
Wenzel Jakobf64feaf2016-04-28 14:33:45 +020053 std::string toString() const {
54 return "[" + std::to_string(x) + ", " + std::to_string(y) + "]";
55 }
Wenzel Jakob93296692015-10-13 23:21:54 +020056 private:
57 float x, y;
58 };
59
60The following snippet shows how the above operators can be conveniently exposed
61to Python.
62
63.. code-block:: cpp
64
Wenzel Jakob8f4eb002015-10-15 18:13:33 +020065 #include <pybind11/operators.h>
Wenzel Jakob93296692015-10-13 23:21:54 +020066
Wenzel Jakobb1b71402015-10-18 16:48:30 +020067 PYBIND11_PLUGIN(example) {
Wenzel Jakob8f4eb002015-10-15 18:13:33 +020068 py::module m("example", "pybind11 example plugin");
Wenzel Jakob93296692015-10-13 23:21:54 +020069
70 py::class_<Vector2>(m, "Vector2")
71 .def(py::init<float, float>())
72 .def(py::self + py::self)
73 .def(py::self += py::self)
74 .def(py::self *= float())
75 .def(float() * py::self)
76 .def("__repr__", &Vector2::toString);
77
78 return m.ptr();
79 }
80
81Note that a line like
82
83.. code-block:: cpp
84
85 .def(py::self * float())
86
87is really just short hand notation for
88
89.. code-block:: cpp
90
91 .def("__mul__", [](const Vector2 &a, float b) {
92 return a * b;
93 })
94
95This can be useful for exposing additional operators that don't exist on the
96C++ side, or to perform other types of customization.
97
98.. note::
99
100 To use the more convenient ``py::self`` notation, the additional
Wenzel Jakob8f4eb002015-10-15 18:13:33 +0200101 header file :file:`pybind11/operators.h` must be included.
Wenzel Jakob93296692015-10-13 23:21:54 +0200102
103.. seealso::
104
105 The file :file:`example/example3.cpp` contains a complete example that
106 demonstrates how to work with overloaded operators in more detail.
107
108Callbacks and passing anonymous functions
109=========================================
110
111The C++11 standard brought lambda functions and the generic polymorphic
112function wrapper ``std::function<>`` to the C++ programming language, which
113enable powerful new ways of working with functions. Lambda functions come in
114two flavors: stateless lambda function resemble classic function pointers that
115link to an anonymous piece of code, while stateful lambda functions
116additionally depend on captured variables that are stored in an anonymous
117*lambda closure object*.
118
119Here is a simple example of a C++ function that takes an arbitrary function
120(stateful or stateless) with signature ``int -> int`` as an argument and runs
121it with the value 10.
122
123.. code-block:: cpp
124
125 int func_arg(const std::function<int(int)> &f) {
126 return f(10);
127 }
128
129The example below is more involved: it takes a function of signature ``int -> int``
130and returns another function of the same kind. The return value is a stateful
131lambda function, which stores the value ``f`` in the capture object and adds 1 to
132its return value upon execution.
133
134.. code-block:: cpp
135
136 std::function<int(int)> func_ret(const std::function<int(int)> &f) {
137 return [f](int i) {
138 return f(i) + 1;
139 };
140 }
141
Wenzel Jakob8f4eb002015-10-15 18:13:33 +0200142After including the extra header file :file:`pybind11/functional.h`, it is almost
Wenzel Jakob93296692015-10-13 23:21:54 +0200143trivial to generate binding code for both of these functions.
144
145.. code-block:: cpp
146
Wenzel Jakob8f4eb002015-10-15 18:13:33 +0200147 #include <pybind11/functional.h>
Wenzel Jakob93296692015-10-13 23:21:54 +0200148
Wenzel Jakobb1b71402015-10-18 16:48:30 +0200149 PYBIND11_PLUGIN(example) {
Wenzel Jakob8f4eb002015-10-15 18:13:33 +0200150 py::module m("example", "pybind11 example plugin");
Wenzel Jakob93296692015-10-13 23:21:54 +0200151
152 m.def("func_arg", &func_arg);
153 m.def("func_ret", &func_ret);
154
155 return m.ptr();
156 }
157
158The following interactive session shows how to call them from Python.
159
Wenzel Jakob99279f72016-06-03 11:19:29 +0200160.. code-block:: pycon
Wenzel Jakob93296692015-10-13 23:21:54 +0200161
162 $ python
163 >>> import example
164 >>> def square(i):
165 ... return i * i
166 ...
167 >>> example.func_arg(square)
168 100L
169 >>> square_plus_1 = example.func_ret(square)
170 >>> square_plus_1(4)
171 17L
172 >>>
173
174.. note::
175
176 This functionality is very useful when generating bindings for callbacks in
177 C++ libraries (e.g. a graphical user interface library).
178
179 The file :file:`example/example5.cpp` contains a complete example that
180 demonstrates how to work with callbacks and anonymous functions in more detail.
181
Wenzel Jakoba4175d62015-11-17 08:30:34 +0100182.. warning::
183
184 Keep in mind that passing a function from C++ to Python (or vice versa)
185 will instantiate a piece of wrapper code that translates function
186 invocations between the two languages. Copying the same function back and
187 forth between Python and C++ many times in a row will cause these wrappers
188 to accumulate, which can decrease performance.
189
Wenzel Jakob28f98aa2015-10-13 02:57:16 +0200190Overriding virtual functions in Python
191======================================
192
Wenzel Jakob93296692015-10-13 23:21:54 +0200193Suppose that a C++ class or interface has a virtual function that we'd like to
194to override from within Python (we'll focus on the class ``Animal``; ``Dog`` is
195given as a specific example of how one would do this with traditional C++
196code).
197
198.. code-block:: cpp
199
200 class Animal {
201 public:
202 virtual ~Animal() { }
203 virtual std::string go(int n_times) = 0;
204 };
205
206 class Dog : public Animal {
207 public:
208 std::string go(int n_times) {
209 std::string result;
210 for (int i=0; i<n_times; ++i)
211 result += "woof! ";
212 return result;
213 }
214 };
215
216Let's also suppose that we are given a plain function which calls the
217function ``go()`` on an arbitrary ``Animal`` instance.
218
219.. code-block:: cpp
220
221 std::string call_go(Animal *animal) {
222 return animal->go(3);
223 }
224
225Normally, the binding code for these classes would look as follows:
226
227.. code-block:: cpp
228
Wenzel Jakobb1b71402015-10-18 16:48:30 +0200229 PYBIND11_PLUGIN(example) {
Wenzel Jakob8f4eb002015-10-15 18:13:33 +0200230 py::module m("example", "pybind11 example plugin");
Wenzel Jakob93296692015-10-13 23:21:54 +0200231
232 py::class_<Animal> animal(m, "Animal");
233 animal
234 .def("go", &Animal::go);
235
236 py::class_<Dog>(m, "Dog", animal)
237 .def(py::init<>());
238
239 m.def("call_go", &call_go);
240
241 return m.ptr();
242 }
243
244However, these bindings are impossible to extend: ``Animal`` is not
245constructible, and we clearly require some kind of "trampoline" that
246redirects virtual calls back to Python.
247
248Defining a new type of ``Animal`` from within Python is possible but requires a
249helper class that is defined as follows:
250
251.. code-block:: cpp
252
253 class PyAnimal : public Animal {
254 public:
255 /* Inherit the constructors */
256 using Animal::Animal;
257
258 /* Trampoline (need one for each virtual function) */
259 std::string go(int n_times) {
Wenzel Jakobb1b71402015-10-18 16:48:30 +0200260 PYBIND11_OVERLOAD_PURE(
Wenzel Jakob93296692015-10-13 23:21:54 +0200261 std::string, /* Return type */
262 Animal, /* Parent class */
263 go, /* Name of function */
264 n_times /* Argument(s) */
265 );
266 }
267 };
268
Wenzel Jakobb1b71402015-10-18 16:48:30 +0200269The macro :func:`PYBIND11_OVERLOAD_PURE` should be used for pure virtual
270functions, and :func:`PYBIND11_OVERLOAD` should be used for functions which have
Wenzel Jakob1e3be732016-05-24 23:42:05 +0200271a default implementation.
272
273There are also two alternate macros :func:`PYBIND11_OVERLOAD_PURE_NAME` and
274:func:`PYBIND11_OVERLOAD_NAME` which take a string-valued name argument
275after the *Name of the function* slot. This is useful when the C++ and Python
276versions of the function have different names, e.g. ``operator()`` vs ``__call__``.
277
278The binding code also needs a few minor adaptations (highlighted):
Wenzel Jakob93296692015-10-13 23:21:54 +0200279
280.. code-block:: cpp
281 :emphasize-lines: 4,6,7
282
Wenzel Jakobb1b71402015-10-18 16:48:30 +0200283 PYBIND11_PLUGIN(example) {
Wenzel Jakob8f4eb002015-10-15 18:13:33 +0200284 py::module m("example", "pybind11 example plugin");
Wenzel Jakob93296692015-10-13 23:21:54 +0200285
Wenzel Jakob86d825f2016-05-26 13:19:27 +0200286 py::class_<Animal, std::unique_ptr<Animal>, PyAnimal /* <--- trampoline*/> animal(m, "Animal");
Wenzel Jakob93296692015-10-13 23:21:54 +0200287 animal
Wenzel Jakob93296692015-10-13 23:21:54 +0200288 .def(py::init<>())
289 .def("go", &Animal::go);
290
291 py::class_<Dog>(m, "Dog", animal)
292 .def(py::init<>());
293
294 m.def("call_go", &call_go);
295
296 return m.ptr();
297 }
298
Wenzel Jakob86d825f2016-05-26 13:19:27 +0200299Importantly, pybind11 is made aware of the trampoline trampoline helper class
300by specifying it as the *third* template argument to :class:`class_`. The
301second argument with the unique pointer is simply the default holder type used
302by pybind11. Following this, we are able to define a constructor as usual.
Wenzel Jakob93296692015-10-13 23:21:54 +0200303
304The Python session below shows how to override ``Animal::go`` and invoke it via
305a virtual method call.
306
Wenzel Jakob99279f72016-06-03 11:19:29 +0200307.. code-block:: pycon
Wenzel Jakob93296692015-10-13 23:21:54 +0200308
309 >>> from example import *
310 >>> d = Dog()
311 >>> call_go(d)
312 u'woof! woof! woof! '
313 >>> class Cat(Animal):
314 ... def go(self, n_times):
315 ... return "meow! " * n_times
316 ...
317 >>> c = Cat()
318 >>> call_go(c)
319 u'meow! meow! meow! '
320
Wenzel Jakob9bb97c12016-06-03 11:19:41 +0200321Please take a look at the :ref:`macro_notes` before using this feature.
Wenzel Jakobbd986fe2016-05-21 10:48:30 +0200322
Wenzel Jakob93296692015-10-13 23:21:54 +0200323.. seealso::
324
325 The file :file:`example/example12.cpp` contains a complete example that
326 demonstrates how to override virtual functions using pybind11 in more
327 detail.
328
Wenzel Jakobecdd8682015-12-07 18:17:58 +0100329
Wenzel Jakob9bb97c12016-06-03 11:19:41 +0200330.. _macro_notes:
331
332General notes regarding convenience macros
333==========================================
334
335pybind11 provides a few convenience macros such as
336:func:`PYBIND11_MAKE_OPAQUE` and :func:`PYBIND11_DECLARE_HOLDER_TYPE`, and
337``PYBIND11_OVERLOAD_*``. Since these are "just" macros that are evaluated
338in the preprocessor (which has no concept of types), they *will* get confused
339by commas in a template argument such as ``PYBIND11_OVERLOAD(MyReturnValue<T1,
340T2>, myFunc)``. In this case, the preprocessor assumes that the comma indicates
341the beginnning of the next parameter. Use a ``typedef`` to bind the template to
342another name and use it in the macro to avoid this problem.
343
344
Wenzel Jakobecdd8682015-12-07 18:17:58 +0100345Global Interpreter Lock (GIL)
346=============================
347
348The classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be
349used to acquire and release the global interpreter lock in the body of a C++
350function call. In this way, long-running C++ code can be parallelized using
351multiple Python threads. Taking the previous section as an example, this could
352be realized as follows (important changes highlighted):
353
354.. code-block:: cpp
355 :emphasize-lines: 8,9,33,34
356
357 class PyAnimal : public Animal {
358 public:
359 /* Inherit the constructors */
360 using Animal::Animal;
361
362 /* Trampoline (need one for each virtual function) */
363 std::string go(int n_times) {
364 /* Acquire GIL before calling Python code */
Wenzel Jakoba4caa852015-12-14 12:39:02 +0100365 py::gil_scoped_acquire acquire;
Wenzel Jakobecdd8682015-12-07 18:17:58 +0100366
367 PYBIND11_OVERLOAD_PURE(
368 std::string, /* Return type */
369 Animal, /* Parent class */
370 go, /* Name of function */
371 n_times /* Argument(s) */
372 );
373 }
374 };
375
376 PYBIND11_PLUGIN(example) {
377 py::module m("example", "pybind11 example plugin");
378
Wenzel Jakob86d825f2016-05-26 13:19:27 +0200379 py::class_<Animal, std::unique_ptr<Animal>, PyAnimal> animal(m, "Animal");
Wenzel Jakobecdd8682015-12-07 18:17:58 +0100380 animal
Wenzel Jakobecdd8682015-12-07 18:17:58 +0100381 .def(py::init<>())
382 .def("go", &Animal::go);
383
384 py::class_<Dog>(m, "Dog", animal)
385 .def(py::init<>());
386
387 m.def("call_go", [](Animal *animal) -> std::string {
388 /* Release GIL before calling into (potentially long-running) C++ code */
Wenzel Jakoba4caa852015-12-14 12:39:02 +0100389 py::gil_scoped_release release;
Wenzel Jakobecdd8682015-12-07 18:17:58 +0100390 return call_go(animal);
391 });
392
393 return m.ptr();
394 }
395
Wenzel Jakob93296692015-10-13 23:21:54 +0200396Passing STL data structures
Wenzel Jakob28f98aa2015-10-13 02:57:16 +0200397===========================
398
Wenzel Jakob8f4eb002015-10-15 18:13:33 +0200399When including the additional header file :file:`pybind11/stl.h`, conversions
Wenzel Jakob978e3762016-04-07 18:00:41 +0200400between ``std::vector<>``, ``std::list<>``, ``std::set<>``, and ``std::map<>``
401and the Python ``list``, ``set`` and ``dict`` data structures are automatically
402enabled. The types ``std::pair<>`` and ``std::tuple<>`` are already supported
403out of the box with just the core :file:`pybind11/pybind11.h` header.
Wenzel Jakob93296692015-10-13 23:21:54 +0200404
405.. note::
406
Wenzel Jakob44db04f2015-12-14 12:40:45 +0100407 Arbitrary nesting of any of these types is supported.
Wenzel Jakob93296692015-10-13 23:21:54 +0200408
409.. seealso::
410
411 The file :file:`example/example2.cpp` contains a complete example that
412 demonstrates how to pass STL data types in more detail.
413
Wenzel Jakobb2825952016-04-13 23:33:00 +0200414Binding sequence data types, iterators, the slicing protocol, etc.
415==================================================================
Wenzel Jakob93296692015-10-13 23:21:54 +0200416
417Please refer to the supplemental example for details.
418
419.. seealso::
420
421 The file :file:`example/example6.cpp` contains a complete example that
422 shows how to bind a sequence data type, including length queries
423 (``__len__``), iterators (``__iter__``), the slicing protocol and other
424 kinds of useful operations.
425
Wenzel Jakob28f98aa2015-10-13 02:57:16 +0200426Return value policies
427=====================
428
Wenzel Jakob93296692015-10-13 23:21:54 +0200429Python and C++ use wildly different ways of managing the memory and lifetime of
430objects managed by them. This can lead to issues when creating bindings for
431functions that return a non-trivial type. Just by looking at the type
432information, it is not clear whether Python should take charge of the returned
433value and eventually free its resources, or if this is handled on the C++ side.
434For this reason, pybind11 provides a several `return value policy` annotations
435that can be passed to the :func:`module::def` and :func:`class_::def`
Wenzel Jakob61d67f02015-12-14 12:53:06 +0100436functions. The default policy is :enum:`return_value_policy::automatic`.
Wenzel Jakob28f98aa2015-10-13 02:57:16 +0200437
Wenzel Jakobf64feaf2016-04-28 14:33:45 +0200438.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}|
439
Wenzel Jakobf7b58742016-04-25 23:04:27 +0200440+--------------------------------------------------+----------------------------------------------------------------------------+
441| Return value policy | Description |
442+==================================================+============================================================================+
443| :enum:`return_value_policy::automatic` | This is the default return value policy, which falls back to the policy |
444| | :enum:`return_value_policy::take_ownership` when the return value is a |
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200445| | pointer. Otherwise, it uses :enum:`return_value::move` or |
446| | :enum:`return_value::copy` for rvalue and lvalue references, respectively. |
Wenzel Jakobf7b58742016-04-25 23:04:27 +0200447| | See below for a description of what all of these different policies do. |
448+--------------------------------------------------+----------------------------------------------------------------------------+
449| :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the |
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200450| | return value is a pointer. You probably won't need to use this. |
Wenzel Jakobf7b58742016-04-25 23:04:27 +0200451+--------------------------------------------------+----------------------------------------------------------------------------+
452| :enum:`return_value_policy::take_ownership` | Reference an existing object (i.e. do not create a new copy) and take |
453| | ownership. Python will call the destructor and delete operator when the |
454| | object's reference count reaches zero. Undefined behavior ensues when the |
455| | C++ side does the same.. |
456+--------------------------------------------------+----------------------------------------------------------------------------+
457| :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python. |
458| | This policy is comparably safe because the lifetimes of the two instances |
459| | are decoupled. |
460+--------------------------------------------------+----------------------------------------------------------------------------+
461| :enum:`return_value_policy::move` | Use ``std::move`` to move the return value contents into a new instance |
462| | that will be owned by Python. This policy is comparably safe because the |
463| | lifetimes of the two instances (move source and destination) are decoupled.|
464+--------------------------------------------------+----------------------------------------------------------------------------+
465| :enum:`return_value_policy::reference` | Reference an existing object, but do not take ownership. The C++ side is |
466| | responsible for managing the object's lifetime and deallocating it when |
467| | it is no longer used. Warning: undefined behavior will ensue when the C++ |
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200468| | side deletes an object that is still referenced and used by Python. |
Wenzel Jakobf7b58742016-04-25 23:04:27 +0200469+--------------------------------------------------+----------------------------------------------------------------------------+
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200470| :enum:`return_value_policy::reference_internal` | This policy only applies to methods and properties. It references the |
471| | object without taking ownership similar to the above |
472| | :enum:`return_value_policy::reference` policy. In contrast to that policy, |
473| | the function or property's implicit ``this`` argument (called the *parent*)|
474| | is considered to be the the owner of the return value (the *child*). |
475| | pybind11 then couples the lifetime of the parent to the child via a |
476| | reference relationship that ensures that the parent cannot be garbage |
477| | collected while Python is still using the child. More advanced variations |
478| | of this scheme are also possible using combinations of |
479| | :enum:`return_value_policy::reference` and the :class:`keep_alive` call |
480| | policy described next. |
Wenzel Jakobf7b58742016-04-25 23:04:27 +0200481+--------------------------------------------------+----------------------------------------------------------------------------+
Wenzel Jakob93296692015-10-13 23:21:54 +0200482
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200483The following example snippet shows a use case of the
Wenzel Jakob93296692015-10-13 23:21:54 +0200484:enum:`return_value_policy::reference_internal` policy.
485
486.. code-block:: cpp
487
488 class Example {
489 public:
490 Internal &get_internal() { return internal; }
491 private:
492 Internal internal;
493 };
494
Wenzel Jakobb1b71402015-10-18 16:48:30 +0200495 PYBIND11_PLUGIN(example) {
Wenzel Jakob8f4eb002015-10-15 18:13:33 +0200496 py::module m("example", "pybind11 example plugin");
Wenzel Jakob93296692015-10-13 23:21:54 +0200497
498 py::class_<Example>(m, "Example")
499 .def(py::init<>())
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200500 .def("get_internal", &Example::get_internal, "Return the internal data",
501 py::return_value_policy::reference_internal);
Wenzel Jakob93296692015-10-13 23:21:54 +0200502
503 return m.ptr();
504 }
505
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200506.. warning::
507
508 Code with invalid call policies might access unitialized memory or free
509 data structures multiple times, which can lead to hard-to-debug
510 non-determinism and segmentation faults, hence it is worth spending the
511 time to understand all the different options in the table above.
512
513.. note::
514
515 The next section on :ref:`call_policies` discusses *call policies* that can be
516 specified *in addition* to a return value policy from the list above. Call
517 policies indicate reference relationships that can involve both return values
518 and parameters of functions.
519
520.. note::
521
522 As an alternative to elaborate call policies and lifetime management logic,
523 consider using smart pointers (see the section on :ref:`smart_pointers` for
524 details). Smart pointers can tell whether an object is still referenced from
525 C++ or Python, which generally eliminates the kinds of inconsistencies that
526 can lead to crashes or undefined behavior. For functions returning smart
527 pointers, it is not necessary to specify a return value policy.
Wenzel Jakob5f218b32016-01-17 22:36:39 +0100528
Wenzel Jakobf7b58742016-04-25 23:04:27 +0200529.. _call_policies:
530
Wenzel Jakob5f218b32016-01-17 22:36:39 +0100531Additional call policies
532========================
533
534In addition to the above return value policies, further `call policies` can be
535specified to indicate dependencies between parameters. There is currently just
536one policy named ``keep_alive<Nurse, Patient>``, which indicates that the
537argument with index ``Patient`` should be kept alive at least until the
538argument with index ``Nurse`` is freed by the garbage collector; argument
Wenzel Jakob8e93df82016-05-01 02:36:58 +0200539indices start at one, while zero refers to the return value. For methods, index
540one refers to the implicit ``this`` pointer, while regular arguments begin at
541index two. Arbitrarily many call policies can be specified.
Wenzel Jakob5f218b32016-01-17 22:36:39 +0100542
Wenzel Jakob8e93df82016-05-01 02:36:58 +0200543Consider the following example: the binding code for a list append operation
544that ties the lifetime of the newly added element to the underlying container
545might be declared as follows:
Wenzel Jakob5f218b32016-01-17 22:36:39 +0100546
547.. code-block:: cpp
548
549 py::class_<List>(m, "List")
550 .def("append", &List::append, py::keep_alive<1, 2>());
551
552.. note::
553
554 ``keep_alive`` is analogous to the ``with_custodian_and_ward`` (if Nurse,
555 Patient != 0) and ``with_custodian_and_ward_postcall`` (if Nurse/Patient ==
556 0) policies from Boost.Python.
557
Wenzel Jakob61587162016-01-18 22:38:52 +0100558.. seealso::
559
560 The file :file:`example/example13.cpp` contains a complete example that
561 demonstrates using :class:`keep_alive` in more detail.
562
Wenzel Jakob93296692015-10-13 23:21:54 +0200563Implicit type conversions
564=========================
565
566Suppose that instances of two types ``A`` and ``B`` are used in a project, and
Wenzel Jakob8e93df82016-05-01 02:36:58 +0200567that an ``A`` can easily be converted into an instance of type ``B`` (examples of this
Wenzel Jakob93296692015-10-13 23:21:54 +0200568could be a fixed and an arbitrary precision number type).
569
570.. code-block:: cpp
571
572 py::class_<A>(m, "A")
573 /// ... members ...
574
575 py::class_<B>(m, "B")
576 .def(py::init<A>())
577 /// ... members ...
578
579 m.def("func",
580 [](const B &) { /* .... */ }
581 );
582
583To invoke the function ``func`` using a variable ``a`` containing an ``A``
584instance, we'd have to write ``func(B(a))`` in Python. On the other hand, C++
585will automatically apply an implicit type conversion, which makes it possible
586to directly write ``func(a)``.
587
588In this situation (i.e. where ``B`` has a constructor that converts from
589``A``), the following statement enables similar implicit conversions on the
590Python side:
591
592.. code-block:: cpp
593
594 py::implicitly_convertible<A, B>();
595
Wenzel Jakob9f0dfce2016-04-06 17:38:18 +0200596Unique pointers
597===============
598
599Given a class ``Example`` with Python bindings, it's possible to return
600instances wrapped in C++11 unique pointers, like so
601
602.. code-block:: cpp
603
604 std::unique_ptr<Example> create_example() { return std::unique_ptr<Example>(new Example()); }
605
606.. code-block:: cpp
607
608 m.def("create_example", &create_example);
609
610In other words, there is nothing special that needs to be done. While returning
611unique pointers in this way is allowed, it is *illegal* to use them as function
612arguments. For instance, the following function signature cannot be processed
613by pybind11.
614
615.. code-block:: cpp
616
617 void do_something_with_example(std::unique_ptr<Example> ex) { ... }
618
619The above signature would imply that Python needs to give up ownership of an
620object that is passed to this function, which is generally not possible (for
621instance, the object might be referenced elsewhere).
622
Wenzel Jakobf7b58742016-04-25 23:04:27 +0200623.. _smart_pointers:
624
Wenzel Jakob93296692015-10-13 23:21:54 +0200625Smart pointers
626==============
627
Wenzel Jakob9f0dfce2016-04-06 17:38:18 +0200628This section explains how to pass values that are wrapped in "smart" pointer
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200629types with internal reference counting. For the simpler C++11 unique pointers,
630refer to the previous section.
Wenzel Jakob9f0dfce2016-04-06 17:38:18 +0200631
Wenzel Jakobe84f5572016-04-26 23:19:19 +0200632The binding generator for classes, :class:`class_`, takes an optional second
Wenzel Jakob93296692015-10-13 23:21:54 +0200633template type, which denotes a special *holder* type that is used to manage
634references to the object. When wrapping a type named ``Type``, the default
635value of this template parameter is ``std::unique_ptr<Type>``, which means that
636the object is deallocated when Python's reference count goes to zero.
637
Wenzel Jakob1853b652015-10-18 15:38:50 +0200638It is possible to switch to other types of reference counting wrappers or smart
639pointers, which is useful in codebases that rely on them. For instance, the
640following snippet causes ``std::shared_ptr`` to be used instead.
Wenzel Jakob93296692015-10-13 23:21:54 +0200641
642.. code-block:: cpp
643
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100644 py::class_<Example, std::shared_ptr<Example> /* <- holder type */> obj(m, "Example");
Wenzel Jakob5ef12192015-12-15 17:07:35 +0100645
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100646Note that any particular class can only be associated with a single holder type.
Wenzel Jakob93296692015-10-13 23:21:54 +0200647
Wenzel Jakob1853b652015-10-18 15:38:50 +0200648To enable transparent conversions for functions that take shared pointers as an
Wenzel Jakob5ef12192015-12-15 17:07:35 +0100649argument or that return them, a macro invocation similar to the following must
Wenzel Jakob1853b652015-10-18 15:38:50 +0200650be declared at the top level before any binding code:
651
652.. code-block:: cpp
653
Wenzel Jakobb1b71402015-10-18 16:48:30 +0200654 PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr<T>);
Wenzel Jakob1853b652015-10-18 15:38:50 +0200655
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100656.. note::
Wenzel Jakob61d67f02015-12-14 12:53:06 +0100657
658 The first argument of :func:`PYBIND11_DECLARE_HOLDER_TYPE` should be a
659 placeholder name that is used as a template parameter of the second
660 argument. Thus, feel free to use any identifier, but use it consistently on
661 both sides; also, don't use the name of a type that already exists in your
662 codebase.
663
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100664One potential stumbling block when using holder types is that they need to be
665applied consistently. Can you guess what's broken about the following binding
666code?
Wenzel Jakob6e213c92015-11-24 23:05:58 +0100667
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100668.. code-block:: cpp
Wenzel Jakob6e213c92015-11-24 23:05:58 +0100669
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100670 class Child { };
Wenzel Jakob5ef12192015-12-15 17:07:35 +0100671
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100672 class Parent {
673 public:
674 Parent() : child(std::make_shared<Child>()) { }
675 Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */
676 private:
677 std::shared_ptr<Child> child;
678 };
Wenzel Jakob5ef12192015-12-15 17:07:35 +0100679
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100680 PYBIND11_PLUGIN(example) {
681 py::module m("example");
Wenzel Jakob5ef12192015-12-15 17:07:35 +0100682
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100683 py::class_<Child, std::shared_ptr<Child>>(m, "Child");
684
685 py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
686 .def(py::init<>())
687 .def("get_child", &Parent::get_child);
688
689 return m.ptr();
690 }
691
692The following Python code will cause undefined behavior (and likely a
693segmentation fault).
694
695.. code-block:: python
696
697 from example import Parent
698 print(Parent().get_child())
699
700The problem is that ``Parent::get_child()`` returns a pointer to an instance of
701``Child``, but the fact that this instance is already managed by
702``std::shared_ptr<...>`` is lost when passing raw pointers. In this case,
703pybind11 will create a second independent ``std::shared_ptr<...>`` that also
704claims ownership of the pointer. In the end, the object will be freed **twice**
705since these shared pointers have no way of knowing about each other.
706
707There are two ways to resolve this issue:
708
7091. For types that are managed by a smart pointer class, never use raw pointers
710 in function arguments or return values. In other words: always consistently
711 wrap pointers into their designated holder types (such as
712 ``std::shared_ptr<...>``). In this case, the signature of ``get_child()``
713 should be modified as follows:
714
715.. code-block:: cpp
716
717 std::shared_ptr<Child> get_child() { return child; }
718
7192. Adjust the definition of ``Child`` by specifying
720 ``std::enable_shared_from_this<T>`` (see cppreference_ for details) as a
721 base class. This adds a small bit of information to ``Child`` that allows
722 pybind11 to realize that there is already an existing
723 ``std::shared_ptr<...>`` and communicate with it. In this case, the
724 declaration of ``Child`` should look as follows:
Wenzel Jakob5ef12192015-12-15 17:07:35 +0100725
Wenzel Jakob6e213c92015-11-24 23:05:58 +0100726.. _cppreference: http://en.cppreference.com/w/cpp/memory/enable_shared_from_this
727
Wenzel Jakobb2c2c792016-01-17 22:36:40 +0100728.. code-block:: cpp
729
730 class Child : public std::enable_shared_from_this<Child> { };
731
Wenzel Jakob9bb97c12016-06-03 11:19:41 +0200732
733Please take a look at the :ref:`macro_notes` before using this feature.
734
Wenzel Jakob5ef12192015-12-15 17:07:35 +0100735.. seealso::
736
737 The file :file:`example/example8.cpp` contains a complete example that
738 demonstrates how to work with custom reference-counting holder types in
739 more detail.
740
Wenzel Jakob93296692015-10-13 23:21:54 +0200741.. _custom_constructors:
742
743Custom constructors
744===================
745
746The syntax for binding constructors was previously introduced, but it only
747works when a constructor with the given parameters actually exists on the C++
748side. To extend this to more general cases, let's take a look at what actually
749happens under the hood: the following statement
750
751.. code-block:: cpp
752
753 py::class_<Example>(m, "Example")
754 .def(py::init<int>());
755
756is short hand notation for
757
758.. code-block:: cpp
759
760 py::class_<Example>(m, "Example")
761 .def("__init__",
762 [](Example &instance, int arg) {
763 new (&instance) Example(arg);
764 }
765 );
766
767In other words, :func:`init` creates an anonymous function that invokes an
768in-place constructor. Memory allocation etc. is already take care of beforehand
769within pybind11.
770
771Catching and throwing exceptions
772================================
773
774When C++ code invoked from Python throws an ``std::exception``, it is
775automatically converted into a Python ``Exception``. pybind11 defines multiple
776special exception classes that will map to different types of Python
777exceptions:
778
Wenzel Jakobf64feaf2016-04-28 14:33:45 +0200779.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}|
780
Wenzel Jakob978e3762016-04-07 18:00:41 +0200781+--------------------------------------+------------------------------+
782| C++ exception type | Python exception type |
783+======================================+==============================+
784| :class:`std::exception` | ``RuntimeError`` |
785+--------------------------------------+------------------------------+
786| :class:`std::bad_alloc` | ``MemoryError`` |
787+--------------------------------------+------------------------------+
788| :class:`std::domain_error` | ``ValueError`` |
789+--------------------------------------+------------------------------+
790| :class:`std::invalid_argument` | ``ValueError`` |
791+--------------------------------------+------------------------------+
792| :class:`std::length_error` | ``ValueError`` |
793+--------------------------------------+------------------------------+
794| :class:`std::out_of_range` | ``ValueError`` |
795+--------------------------------------+------------------------------+
796| :class:`std::range_error` | ``ValueError`` |
797+--------------------------------------+------------------------------+
798| :class:`pybind11::stop_iteration` | ``StopIteration`` (used to |
799| | implement custom iterators) |
800+--------------------------------------+------------------------------+
801| :class:`pybind11::index_error` | ``IndexError`` (used to |
802| | indicate out of bounds |
803| | accesses in ``__getitem__``, |
804| | ``__setitem__``, etc.) |
805+--------------------------------------+------------------------------+
Sergey Lyskova95bde12016-05-08 19:31:55 -0400806| :class:`pybind11::value_error` | ``ValueError`` (used to |
807| | indicate wrong value passed |
808| | in ``container.remove(...)`` |
809+--------------------------------------+------------------------------+
Wenzel Jakob978e3762016-04-07 18:00:41 +0200810| :class:`pybind11::error_already_set` | Indicates that the Python |
811| | exception flag has already |
812| | been initialized |
813+--------------------------------------+------------------------------+
Wenzel Jakob93296692015-10-13 23:21:54 +0200814
815When a Python function invoked from C++ throws an exception, it is converted
816into a C++ exception of type :class:`error_already_set` whose string payload
817contains a textual summary.
818
819There is also a special exception :class:`cast_error` that is thrown by
820:func:`handle::call` when the input arguments cannot be converted to Python
821objects.
Wenzel Jakob28f98aa2015-10-13 02:57:16 +0200822
Wenzel Jakob9e0a0562016-05-05 20:33:54 +0200823.. _opaque:
824
825Treating STL data structures as opaque objects
826==============================================
827
828pybind11 heavily relies on a template matching mechanism to convert parameters
829and return values that are constructed from STL data types such as vectors,
830linked lists, hash tables, etc. This even works in a recursive manner, for
831instance to deal with lists of hash maps of pairs of elementary and custom
832types, etc.
833
834However, a fundamental limitation of this approach is that internal conversions
835between Python and C++ types involve a copy operation that prevents
836pass-by-reference semantics. What does this mean?
837
838Suppose we bind the following function
839
840.. code-block:: cpp
841
842 void append_1(std::vector<int> &v) {
843 v.push_back(1);
844 }
845
846and call it from Python, the following happens:
847
Wenzel Jakob99279f72016-06-03 11:19:29 +0200848.. code-block:: pycon
Wenzel Jakob9e0a0562016-05-05 20:33:54 +0200849
850 >>> v = [5, 6]
851 >>> append_1(v)
852 >>> print(v)
853 [5, 6]
854
855As you can see, when passing STL data structures by reference, modifications
856are not propagated back the Python side. A similar situation arises when
857exposing STL data structures using the ``def_readwrite`` or ``def_readonly``
858functions:
859
860.. code-block:: cpp
861
862 /* ... definition ... */
863
864 class MyClass {
865 std::vector<int> contents;
866 };
867
868 /* ... binding code ... */
869
870 py::class_<MyClass>(m, "MyClass")
871 .def(py::init<>)
872 .def_readwrite("contents", &MyClass::contents);
873
874In this case, properties can be read and written in their entirety. However, an
875``append`` operaton involving such a list type has no effect:
876
Wenzel Jakob99279f72016-06-03 11:19:29 +0200877.. code-block:: pycon
Wenzel Jakob9e0a0562016-05-05 20:33:54 +0200878
879 >>> m = MyClass()
880 >>> m.contents = [5, 6]
881 >>> print(m.contents)
882 [5, 6]
883 >>> m.contents.append(7)
884 >>> print(m.contents)
885 [5, 6]
886
887To deal with both of the above situations, pybind11 provides a macro named
888``PYBIND11_MAKE_OPAQUE(T)`` that disables the template-based conversion
889machinery of types, thus rendering them *opaque*. The contents of opaque
890objects are never inspected or extracted, hence they can be passed by
891reference. For instance, to turn ``std::vector<int>`` into an opaque type, add
892the declaration
893
894.. code-block:: cpp
895
896 PYBIND11_MAKE_OPAQUE(std::vector<int>);
897
898before any binding code (e.g. invocations to ``class_::def()``, etc.). This
899macro must be specified at the top level, since instantiates a partial template
900overload. If your binding code consists of multiple compilation units, it must
901be present in every file preceding any usage of ``std::vector<int>``. Opaque
902types must also have a corresponding ``class_`` declaration to associate them
903with a name in Python, and to define a set of available operations:
904
905.. code-block:: cpp
906
907 py::class_<std::vector<int>>(m, "IntVector")
908 .def(py::init<>())
909 .def("clear", &std::vector<int>::clear)
910 .def("pop_back", &std::vector<int>::pop_back)
911 .def("__len__", [](const std::vector<int> &v) { return v.size(); })
912 .def("__iter__", [](std::vector<int> &v) {
913 return py::make_iterator(v.begin(), v.end());
914 }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */
915 // ....
916
Wenzel Jakob9bb97c12016-06-03 11:19:41 +0200917Please take a look at the :ref:`macro_notes` before using this feature.
Wenzel Jakob9e0a0562016-05-05 20:33:54 +0200918
919.. seealso::
920
921 The file :file:`example/example14.cpp` contains a complete example that
922 demonstrates how to create and expose opaque types using pybind11 in more
923 detail.
924
925.. _eigen:
926
927Transparent conversion of dense and sparse Eigen data types
928===========================================================
929
930Eigen [#f1]_ is C++ header-based library for dense and sparse linear algebra. Due to
931its popularity and widespread adoption, pybind11 provides transparent
932conversion support between Eigen and Scientific Python linear algebra data types.
933
934Specifically, when including the optional header file :file:`pybind11/eigen.h`,
Wenzel Jakob178c8a82016-05-10 15:59:01 +0100935pybind11 will automatically and transparently convert
Wenzel Jakob9e0a0562016-05-05 20:33:54 +0200936
9371. Static and dynamic Eigen dense vectors and matrices to instances of
938 ``numpy.ndarray`` (and vice versa).
939
9401. Eigen sparse vectors and matrices to instances of
941 ``scipy.sparse.csr_matrix``/``scipy.sparse.csc_matrix`` (and vice versa).
942
943This makes it possible to bind most kinds of functions that rely on these types.
944One major caveat are functions that take Eigen matrices *by reference* and modify
945them somehow, in which case the information won't be propagated to the caller.
946
947.. code-block:: cpp
948
949 /* The Python bindings of this function won't replicate
950 the intended effect of modifying the function argument */
951 void scale_by_2(Eigen::Vector3f &v) {
952 v *= 2;
953 }
954
955To see why this is, refer to the section on :ref:`opaque` (although that
956section specifically covers STL data types, the underlying issue is the same).
957The next two sections discuss an efficient alternative for exposing the
958underlying native Eigen types as opaque objects in a way that still integrates
959with NumPy and SciPy.
960
961.. [#f1] http://eigen.tuxfamily.org
962
963.. seealso::
964
965 The file :file:`example/eigen.cpp` contains a complete example that
966 shows how to pass Eigen sparse and dense data types in more detail.
967
Wenzel Jakob28f98aa2015-10-13 02:57:16 +0200968Buffer protocol
969===============
970
971Python supports an extremely general and convenient approach for exchanging
Wenzel Jakob9e0a0562016-05-05 20:33:54 +0200972data between plugin libraries. Types can expose a buffer view [#f2]_, which
973provides fast direct access to the raw internal data representation. Suppose we
974want to bind the following simplistic Matrix class:
Wenzel Jakob28f98aa2015-10-13 02:57:16 +0200975
976.. code-block:: cpp
977
978 class Matrix {
979 public:
980 Matrix(size_t rows, size_t cols) : m_rows(rows), m_cols(cols) {
981 m_data = new float[rows*cols];
982 }
983 float *data() { return m_data; }
984 size_t rows() const { return m_rows; }
985 size_t cols() const { return m_cols; }
986 private:
987 size_t m_rows, m_cols;
988 float *m_data;
989 };
990
991The following binding code exposes the ``Matrix`` contents as a buffer object,
Wenzel Jakob8e93df82016-05-01 02:36:58 +0200992making it possible to cast Matrices into NumPy arrays. It is even possible to
Wenzel Jakob28f98aa2015-10-13 02:57:16 +0200993completely avoid copy operations with Python expressions like
994``np.array(matrix_instance, copy = False)``.
995
996.. code-block:: cpp
997
998 py::class_<Matrix>(m, "Matrix")
999 .def_buffer([](Matrix &m) -> py::buffer_info {
1000 return py::buffer_info(
Wenzel Jakob876eeab2016-05-04 22:22:48 +02001001 m.data(), /* Pointer to buffer */
1002 sizeof(float), /* Size of one scalar */
1003 py::format_descriptor<float>::value, /* Python struct-style format descriptor */
1004 2, /* Number of dimensions */
1005 { m.rows(), m.cols() }, /* Buffer dimensions */
1006 { sizeof(float) * m.rows(), /* Strides (in bytes) for each index */
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001007 sizeof(float) }
1008 );
1009 });
1010
1011The snippet above binds a lambda function, which can create ``py::buffer_info``
1012description records on demand describing a given matrix. The contents of
1013``py::buffer_info`` mirror the Python buffer protocol specification.
1014
1015.. code-block:: cpp
1016
1017 struct buffer_info {
1018 void *ptr;
1019 size_t itemsize;
1020 std::string format;
1021 int ndim;
1022 std::vector<size_t> shape;
1023 std::vector<size_t> strides;
1024 };
1025
1026To create a C++ function that can take a Python buffer object as an argument,
1027simply use the type ``py::buffer`` as one of its arguments. Buffers can exist
1028in a great variety of configurations, hence some safety checks are usually
1029necessary in the function body. Below, you can see an basic example on how to
1030define a custom constructor for the Eigen double precision matrix
1031(``Eigen::MatrixXd``) type, which supports initialization from compatible
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001032buffer objects (e.g. a NumPy matrix).
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001033
1034.. code-block:: cpp
1035
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001036 /* Bind MatrixXd (or some other Eigen type) to Python */
1037 typedef Eigen::MatrixXd Matrix;
1038
1039 typedef Matrix::Scalar Scalar;
1040 constexpr bool rowMajor = Matrix::Flags & Eigen::RowMajorBit;
1041
1042 py::class_<Matrix>(m, "Matrix")
1043 .def("__init__", [](Matrix &m, py::buffer b) {
Wenzel Jakobe7628532016-05-05 10:04:44 +02001044 typedef Eigen::Stride<Eigen::Dynamic, Eigen::Dynamic> Strides;
Wenzel Jakobe7628532016-05-05 10:04:44 +02001045
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001046 /* Request a buffer descriptor from Python */
1047 py::buffer_info info = b.request();
1048
1049 /* Some sanity checks ... */
Wenzel Jakobe7628532016-05-05 10:04:44 +02001050 if (info.format != py::format_descriptor<Scalar>::value)
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001051 throw std::runtime_error("Incompatible format: expected a double array!");
1052
1053 if (info.ndim != 2)
1054 throw std::runtime_error("Incompatible buffer dimension!");
1055
Wenzel Jakobe7628532016-05-05 10:04:44 +02001056 auto strides = Strides(
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001057 info.strides[rowMajor ? 0 : 1] / sizeof(Scalar),
1058 info.strides[rowMajor ? 1 : 0] / sizeof(Scalar));
Wenzel Jakobe7628532016-05-05 10:04:44 +02001059
1060 auto map = Eigen::Map<Matrix, 0, Strides>(
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001061 static_cat<Scalar *>(info.ptr), info.shape[0], info.shape[1], strides);
Wenzel Jakobe7628532016-05-05 10:04:44 +02001062
1063 new (&m) Matrix(map);
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001064 });
1065
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001066For reference, the ``def_buffer()`` call for this Eigen data type should look
1067as follows:
1068
1069.. code-block:: cpp
1070
1071 .def_buffer([](Matrix &m) -> py::buffer_info {
1072 return py::buffer_info(
1073 m.data(), /* Pointer to buffer */
1074 sizeof(Scalar), /* Size of one scalar */
1075 /* Python struct-style format descriptor */
1076 py::format_descriptor<Scalar>::value,
1077 /* Number of dimensions */
1078 2,
1079 /* Buffer dimensions */
1080 { (size_t) m.rows(),
1081 (size_t) m.cols() },
1082 /* Strides (in bytes) for each index */
1083 { sizeof(Scalar) * (rowMajor ? m.cols() : 1),
1084 sizeof(Scalar) * (rowMajor ? 1 : m.rows()) }
1085 );
1086 })
1087
1088For a much easier approach of binding Eigen types (although with some
1089limitations), refer to the section on :ref:`eigen`.
1090
Wenzel Jakob93296692015-10-13 23:21:54 +02001091.. seealso::
1092
1093 The file :file:`example/example7.cpp` contains a complete example that
1094 demonstrates using the buffer protocol with pybind11 in more detail.
1095
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001096.. [#f2] http://docs.python.org/3/c-api/buffer.html
Wenzel Jakob978e3762016-04-07 18:00:41 +02001097
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001098NumPy support
1099=============
1100
1101By exchanging ``py::buffer`` with ``py::array`` in the above snippet, we can
1102restrict the function so that it only accepts NumPy arrays (rather than any
Wenzel Jakob978e3762016-04-07 18:00:41 +02001103type of Python object satisfying the buffer protocol).
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001104
1105In many situations, we want to define a function which only accepts a NumPy
Wenzel Jakob93296692015-10-13 23:21:54 +02001106array of a certain data type. This is possible via the ``py::array_t<T>``
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001107template. For instance, the following function requires the argument to be a
Wenzel Jakobf1032df2016-05-05 10:00:00 +02001108NumPy array containing double precision values.
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001109
1110.. code-block:: cpp
1111
Wenzel Jakob93296692015-10-13 23:21:54 +02001112 void f(py::array_t<double> array);
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001113
Wenzel Jakobf1032df2016-05-05 10:00:00 +02001114When it is invoked with a different type (e.g. an integer or a list of
1115integers), the binding code will attempt to cast the input into a NumPy array
1116of the requested type. Note that this feature requires the
1117:file:``pybind11/numpy.h`` header to be included.
1118
1119Data in NumPy arrays is not guaranteed to packed in a dense manner;
1120furthermore, entries can be separated by arbitrary column and row strides.
1121Sometimes, it can be useful to require a function to only accept dense arrays
1122using either the C (row-major) or Fortran (column-major) ordering. This can be
1123accomplished via a second template argument with values ``py::array::c_style``
1124or ``py::array::f_style``.
1125
1126.. code-block:: cpp
1127
Wenzel Jakobb47a9de2016-05-19 16:02:09 +02001128 void f(py::array_t<double, py::array::c_style | py::array::forcecast> array);
Wenzel Jakobf1032df2016-05-05 10:00:00 +02001129
Wenzel Jakobb47a9de2016-05-19 16:02:09 +02001130The ``py::array::forcecast`` argument is the default value of the second
1131template paramenter, and it ensures that non-conforming arguments are converted
1132into an array satisfying the specified requirements instead of trying the next
1133function overload.
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001134
1135Vectorizing functions
1136=====================
1137
1138Suppose we want to bind a function with the following signature to Python so
1139that it can process arbitrary NumPy array arguments (vectors, matrices, general
1140N-D arrays) in addition to its normal arguments:
1141
1142.. code-block:: cpp
1143
1144 double my_func(int x, float y, double z);
1145
Wenzel Jakob8f4eb002015-10-15 18:13:33 +02001146After including the ``pybind11/numpy.h`` header, this is extremely simple:
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001147
1148.. code-block:: cpp
1149
1150 m.def("vectorized_func", py::vectorize(my_func));
1151
1152Invoking the function like below causes 4 calls to be made to ``my_func`` with
Wenzel Jakob8e93df82016-05-01 02:36:58 +02001153each of the array elements. The significant advantage of this compared to
Wenzel Jakob978e3762016-04-07 18:00:41 +02001154solutions like ``numpy.vectorize()`` is that the loop over the elements runs
1155entirely on the C++ side and can be crunched down into a tight, optimized loop
1156by the compiler. The result is returned as a NumPy array of type
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001157``numpy.dtype.float64``.
1158
Wenzel Jakob99279f72016-06-03 11:19:29 +02001159.. code-block:: pycon
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001160
1161 >>> x = np.array([[1, 3],[5, 7]])
1162 >>> y = np.array([[2, 4],[6, 8]])
1163 >>> z = 3
1164 >>> result = vectorized_func(x, y, z)
1165
1166The scalar argument ``z`` is transparently replicated 4 times. The input
1167arrays ``x`` and ``y`` are automatically converted into the right types (they
1168are of type ``numpy.dtype.int64`` but need to be ``numpy.dtype.int32`` and
1169``numpy.dtype.float32``, respectively)
1170
Wenzel Jakob8e93df82016-05-01 02:36:58 +02001171Sometimes we might want to explicitly exclude an argument from the vectorization
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001172because it makes little sense to wrap it in a NumPy array. For instance,
1173suppose the function signature was
1174
1175.. code-block:: cpp
1176
1177 double my_func(int x, float y, my_custom_type *z);
1178
1179This can be done with a stateful Lambda closure:
1180
1181.. code-block:: cpp
1182
1183 // Vectorize a lambda function with a capture object (e.g. to exclude some arguments from the vectorization)
1184 m.def("vectorized_func",
Wenzel Jakob93296692015-10-13 23:21:54 +02001185 [](py::array_t<int> x, py::array_t<float> y, my_custom_type *z) {
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001186 auto stateful_closure = [z](int x, float y) { return my_func(x, y, z); };
1187 return py::vectorize(stateful_closure)(x, y);
1188 }
1189 );
1190
Wenzel Jakob61587162016-01-18 22:38:52 +01001191In cases where the computation is too complicated to be reduced to
1192``vectorize``, it will be necessary to create and access the buffer contents
1193manually. The following snippet contains a complete example that shows how this
1194works (the code is somewhat contrived, since it could have been done more
1195simply using ``vectorize``).
1196
1197.. code-block:: cpp
1198
1199 #include <pybind11/pybind11.h>
1200 #include <pybind11/numpy.h>
1201
1202 namespace py = pybind11;
1203
1204 py::array_t<double> add_arrays(py::array_t<double> input1, py::array_t<double> input2) {
1205 auto buf1 = input1.request(), buf2 = input2.request();
1206
1207 if (buf1.ndim != 1 || buf2.ndim != 1)
1208 throw std::runtime_error("Number of dimensions must be one");
1209
1210 if (buf1.shape[0] != buf2.shape[0])
1211 throw std::runtime_error("Input shapes must match");
1212
1213 auto result = py::array(py::buffer_info(
1214 nullptr, /* Pointer to data (nullptr -> ask NumPy to allocate!) */
1215 sizeof(double), /* Size of one item */
Nils Wernerf7048f22016-05-19 11:17:17 +02001216 py::format_descriptor<double>::value(), /* Buffer format */
Wenzel Jakob61587162016-01-18 22:38:52 +01001217 buf1.ndim, /* How many dimensions? */
1218 { buf1.shape[0] }, /* Number of elements for each dimension */
1219 { sizeof(double) } /* Strides for each dimension */
1220 ));
1221
1222 auto buf3 = result.request();
1223
1224 double *ptr1 = (double *) buf1.ptr,
1225 *ptr2 = (double *) buf2.ptr,
1226 *ptr3 = (double *) buf3.ptr;
1227
1228 for (size_t idx = 0; idx < buf1.shape[0]; idx++)
1229 ptr3[idx] = ptr1[idx] + ptr2[idx];
1230
1231 return result;
1232 }
1233
1234 PYBIND11_PLUGIN(test) {
1235 py::module m("test");
1236 m.def("add_arrays", &add_arrays, "Add two NumPy arrays");
1237 return m.ptr();
1238 }
1239
Wenzel Jakob93296692015-10-13 23:21:54 +02001240.. seealso::
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001241
Wenzel Jakob93296692015-10-13 23:21:54 +02001242 The file :file:`example/example10.cpp` contains a complete example that
1243 demonstrates using :func:`vectorize` in more detail.
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001244
Wenzel Jakob93296692015-10-13 23:21:54 +02001245Functions taking Python objects as arguments
1246============================================
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001247
Wenzel Jakob93296692015-10-13 23:21:54 +02001248pybind11 exposes all major Python types using thin C++ wrapper classes. These
1249wrapper classes can also be used as parameters of functions in bindings, which
1250makes it possible to directly work with native Python types on the C++ side.
1251For instance, the following statement iterates over a Python ``dict``:
Wenzel Jakob28f98aa2015-10-13 02:57:16 +02001252
Wenzel Jakob93296692015-10-13 23:21:54 +02001253.. code-block:: cpp
1254
1255 void print_dict(py::dict dict) {
1256 /* Easily interact with Python types */
1257 for (auto item : dict)
1258 std::cout << "key=" << item.first << ", "
1259 << "value=" << item.second << std::endl;
1260 }
1261
1262Available types include :class:`handle`, :class:`object`, :class:`bool_`,
Wenzel Jakob27e8e102016-01-17 22:36:37 +01001263:class:`int_`, :class:`float_`, :class:`str`, :class:`bytes`, :class:`tuple`,
Wenzel Jakobf64feaf2016-04-28 14:33:45 +02001264:class:`list`, :class:`dict`, :class:`slice`, :class:`none`, :class:`capsule`,
1265:class:`iterable`, :class:`iterator`, :class:`function`, :class:`buffer`,
1266:class:`array`, and :class:`array_t`.
Wenzel Jakob93296692015-10-13 23:21:54 +02001267
Wenzel Jakob436b7312015-10-20 01:04:30 +02001268In this kind of mixed code, it is often necessary to convert arbitrary C++
1269types to Python, which can be done using :func:`cast`:
1270
1271.. code-block:: cpp
1272
1273 MyClass *cls = ..;
1274 py::object obj = py::cast(cls);
1275
1276The reverse direction uses the following syntax:
1277
1278.. code-block:: cpp
1279
1280 py::object obj = ...;
1281 MyClass *cls = obj.cast<MyClass *>();
1282
1283When conversion fails, both directions throw the exception :class:`cast_error`.
Wenzel Jakob178c8a82016-05-10 15:59:01 +01001284It is also possible to call python functions via ``operator()``.
1285
1286.. code-block:: cpp
1287
1288 py::function f = <...>;
1289 py::object result_py = f(1234, "hello", some_instance);
1290 MyClass &result = result_py.cast<MyClass>();
1291
1292The special ``f(*args)`` and ``f(*args, **kwargs)`` syntax is also supported to
1293supply arbitrary argument and keyword lists, although these cannot be mixed
1294with other parameters.
1295
1296.. code-block:: cpp
1297
1298 py::function f = <...>;
1299 py::tuple args = py::make_tuple(1234);
1300 py::dict kwargs;
1301 kwargs["y"] = py::cast(5678);
1302 py::object result = f(*args, **kwargs);
Wenzel Jakob436b7312015-10-20 01:04:30 +02001303
Wenzel Jakob93296692015-10-13 23:21:54 +02001304.. seealso::
1305
1306 The file :file:`example/example2.cpp` contains a complete example that
Wenzel Jakob178c8a82016-05-10 15:59:01 +01001307 demonstrates passing native Python types in more detail. The file
1308 :file:`example/example11.cpp` discusses usage of ``args`` and ``kwargs``.
Wenzel Jakob2ac50442016-01-17 22:36:35 +01001309
1310Default arguments revisited
1311===========================
1312
1313The section on :ref:`default_args` previously discussed basic usage of default
1314arguments using pybind11. One noteworthy aspect of their implementation is that
1315default arguments are converted to Python objects right at declaration time.
1316Consider the following example:
1317
1318.. code-block:: cpp
1319
1320 py::class_<MyClass>("MyClass")
1321 .def("myFunction", py::arg("arg") = SomeType(123));
1322
1323In this case, pybind11 must already be set up to deal with values of the type
1324``SomeType`` (via a prior instantiation of ``py::class_<SomeType>``), or an
1325exception will be thrown.
1326
1327Another aspect worth highlighting is that the "preview" of the default argument
1328in the function signature is generated using the object's ``__repr__`` method.
1329If not available, the signature may not be very helpful, e.g.:
1330
Wenzel Jakob99279f72016-06-03 11:19:29 +02001331.. code-block:: pycon
Wenzel Jakob2ac50442016-01-17 22:36:35 +01001332
1333 FUNCTIONS
1334 ...
1335 | myFunction(...)
Wenzel Jakob48548ea2016-01-17 22:36:44 +01001336 | Signature : (MyClass, arg : SomeType = <SomeType object at 0x101b7b080>) -> NoneType
Wenzel Jakob2ac50442016-01-17 22:36:35 +01001337 ...
1338
1339The first way of addressing this is by defining ``SomeType.__repr__``.
1340Alternatively, it is possible to specify the human-readable preview of the
1341default argument manually using the ``arg_t`` notation:
1342
1343.. code-block:: cpp
1344
1345 py::class_<MyClass>("MyClass")
1346 .def("myFunction", py::arg_t<SomeType>("arg", SomeType(123), "SomeType(123)"));
1347
Wenzel Jakobc769fce2016-03-03 12:03:30 +01001348Sometimes it may be necessary to pass a null pointer value as a default
1349argument. In this case, remember to cast it to the underlying type in question,
1350like so:
1351
1352.. code-block:: cpp
1353
1354 py::class_<MyClass>("MyClass")
1355 .def("myFunction", py::arg("arg") = (SomeType *) nullptr);
1356
Wenzel Jakob178c8a82016-05-10 15:59:01 +01001357Binding functions that accept arbitrary numbers of arguments and keywords arguments
1358===================================================================================
1359
1360Python provides a useful mechanism to define functions that accept arbitrary
1361numbers of arguments and keyword arguments:
1362
1363.. code-block:: cpp
1364
1365 def generic(*args, **kwargs):
1366 # .. do something with args and kwargs
1367
1368Such functions can also be created using pybind11:
1369
1370.. code-block:: cpp
1371
1372 void generic(py::args args, py::kwargs kwargs) {
1373 /// .. do something with args
1374 if (kwargs)
1375 /// .. do something with kwargs
1376 }
1377
1378 /// Binding code
1379 m.def("generic", &generic);
1380
1381(See ``example/example11.cpp``). The class ``py::args`` derives from
1382``py::list`` and ``py::kwargs`` derives from ``py::dict`` Note that the
1383``kwargs`` argument is invalid if no keyword arguments were actually provided.
1384Please refer to the other examples for details on how to iterate over these,
1385and on how to cast their entries into C++ objects.
1386
Wenzel Jakob2dfbade2016-01-17 22:36:37 +01001387Partitioning code over multiple extension modules
1388=================================================
1389
Wenzel Jakob90d2f5e2016-04-11 14:30:11 +02001390It's straightforward to split binding code over multiple extension modules,
1391while referencing types that are declared elsewhere. Everything "just" works
1392without any special precautions. One exception to this rule occurs when
1393extending a type declared in another extension module. Recall the basic example
1394from Section :ref:`inheritance`.
Wenzel Jakob2dfbade2016-01-17 22:36:37 +01001395
1396.. code-block:: cpp
1397
1398 py::class_<Pet> pet(m, "Pet");
1399 pet.def(py::init<const std::string &>())
1400 .def_readwrite("name", &Pet::name);
1401
1402 py::class_<Dog>(m, "Dog", pet /* <- specify parent */)
1403 .def(py::init<const std::string &>())
1404 .def("bark", &Dog::bark);
1405
1406Suppose now that ``Pet`` bindings are defined in a module named ``basic``,
1407whereas the ``Dog`` bindings are defined somewhere else. The challenge is of
1408course that the variable ``pet`` is not available anymore though it is needed
1409to indicate the inheritance relationship to the constructor of ``class_<Dog>``.
1410However, it can be acquired as follows:
1411
1412.. code-block:: cpp
1413
1414 py::object pet = (py::object) py::module::import("basic").attr("Pet");
1415
1416 py::class_<Dog>(m, "Dog", pet)
1417 .def(py::init<const std::string &>())
1418 .def("bark", &Dog::bark);
1419
Wenzel Jakob8d862b32016-03-06 13:37:22 +01001420Alternatively, we can rely on the ``base`` tag, which performs an automated
1421lookup of the corresponding Python type. However, this also requires invoking
1422the ``import`` function once to ensure that the pybind11 binding code of the
1423module ``basic`` has been executed.
1424
Wenzel Jakob8d862b32016-03-06 13:37:22 +01001425.. code-block:: cpp
1426
1427 py::module::import("basic");
1428
1429 py::class_<Dog>(m, "Dog", py::base<Pet>())
1430 .def(py::init<const std::string &>())
1431 .def("bark", &Dog::bark);
Wenzel Jakobeda978e2016-03-15 15:05:40 +01001432
Wenzel Jakob978e3762016-04-07 18:00:41 +02001433Naturally, both methods will fail when there are cyclic dependencies.
1434
Wenzel Jakob90d2f5e2016-04-11 14:30:11 +02001435Note that compiling code which has its default symbol visibility set to
1436*hidden* (e.g. via the command line flag ``-fvisibility=hidden`` on GCC/Clang) can interfere with the
1437ability to access types defined in another extension module. Workarounds
1438include changing the global symbol visibility (not recommended, because it will
1439lead unnecessarily large binaries) or manually exporting types that are
1440accessed by multiple extension modules:
1441
1442.. code-block:: cpp
1443
1444 #ifdef _WIN32
1445 # define EXPORT_TYPE __declspec(dllexport)
1446 #else
1447 # define EXPORT_TYPE __attribute__ ((visibility("default")))
1448 #endif
1449
1450 class EXPORT_TYPE Dog : public Animal {
1451 ...
1452 };
1453
1454
Wenzel Jakob1c329aa2016-04-13 02:37:36 +02001455Pickling support
1456================
1457
1458Python's ``pickle`` module provides a powerful facility to serialize and
1459de-serialize a Python object graph into a binary data stream. To pickle and
Wenzel Jakob3d0e6ff2016-04-13 11:48:10 +02001460unpickle C++ classes using pybind11, two additional functions must be provided.
Wenzel Jakob1c329aa2016-04-13 02:37:36 +02001461Suppose the class in question has the following signature:
1462
1463.. code-block:: cpp
1464
1465 class Pickleable {
1466 public:
1467 Pickleable(const std::string &value) : m_value(value) { }
1468 const std::string &value() const { return m_value; }
1469
1470 void setExtra(int extra) { m_extra = extra; }
1471 int extra() const { return m_extra; }
1472 private:
1473 std::string m_value;
1474 int m_extra = 0;
1475 };
1476
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001477The binding code including the requisite ``__setstate__`` and ``__getstate__`` methods [#f3]_
Wenzel Jakob1c329aa2016-04-13 02:37:36 +02001478looks as follows:
1479
1480.. code-block:: cpp
1481
1482 py::class_<Pickleable>(m, "Pickleable")
1483 .def(py::init<std::string>())
1484 .def("value", &Pickleable::value)
1485 .def("extra", &Pickleable::extra)
1486 .def("setExtra", &Pickleable::setExtra)
1487 .def("__getstate__", [](const Pickleable &p) {
1488 /* Return a tuple that fully encodes the state of the object */
1489 return py::make_tuple(p.value(), p.extra());
1490 })
1491 .def("__setstate__", [](Pickleable &p, py::tuple t) {
1492 if (t.size() != 2)
1493 throw std::runtime_error("Invalid state!");
1494
Wenzel Jakobd40885a2016-04-13 13:30:05 +02001495 /* Invoke the in-place constructor. Note that this is needed even
1496 when the object just has a trivial default constructor */
Wenzel Jakob1c329aa2016-04-13 02:37:36 +02001497 new (&p) Pickleable(t[0].cast<std::string>());
1498
1499 /* Assign any additional state */
1500 p.setExtra(t[1].cast<int>());
1501 });
1502
1503An instance can now be pickled as follows:
1504
1505.. code-block:: python
1506
1507 try:
1508 import cPickle as pickle # Use cPickle on Python 2.7
1509 except ImportError:
1510 import pickle
1511
1512 p = Pickleable("test_value")
1513 p.setExtra(15)
Wenzel Jakob81e09752016-04-30 23:13:03 +02001514 data = pickle.dumps(p, 2)
Wenzel Jakob1c329aa2016-04-13 02:37:36 +02001515
Wenzel Jakob81e09752016-04-30 23:13:03 +02001516Note that only the cPickle module is supported on Python 2.7. The second
1517argument to ``dumps`` is also crucial: it selects the pickle protocol version
15182, since the older version 1 is not supported. Newer versions are also fine—for
1519instance, specify ``-1`` to always use the latest available version. Beware:
1520failure to follow these instructions will cause important pybind11 memory
1521allocation routines to be skipped during unpickling, which will likely lead to
1522memory corruption and/or segmentation faults.
Wenzel Jakob1c329aa2016-04-13 02:37:36 +02001523
1524.. seealso::
1525
1526 The file :file:`example/example15.cpp` contains a complete example that
1527 demonstrates how to pickle and unpickle types using pybind11 in more detail.
1528
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001529.. [#f3] http://docs.python.org/3/library/pickle.html#pickling-class-instances
Wenzel Jakobef7a9b92016-04-13 18:41:59 +02001530
1531Generating documentation using Sphinx
1532=====================================
1533
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001534Sphinx [#f4]_ has the ability to inspect the signatures and documentation
Wenzel Jakobef7a9b92016-04-13 18:41:59 +02001535strings in pybind11-based extension modules to automatically generate beautiful
Wenzel Jakobca8dc082016-06-03 14:24:17 +02001536documentation in a variety formats. The python_example repository [#f5]_ contains a
Wenzel Jakobef7a9b92016-04-13 18:41:59 +02001537simple example repository which uses this approach.
1538
1539There are two potential gotchas when using this approach: first, make sure that
1540the resulting strings do not contain any :kbd:`TAB` characters, which break the
1541docstring parsing routines. You may want to use C++11 raw string literals,
1542which are convenient for multi-line comments. Conveniently, any excess
1543indentation will be automatically be removed by Sphinx. However, for this to
1544work, it is important that all lines are indented consistently, i.e.:
1545
1546.. code-block:: cpp
1547
1548 // ok
1549 m.def("foo", &foo, R"mydelimiter(
1550 The foo function
1551
1552 Parameters
1553 ----------
1554 )mydelimiter");
1555
1556 // *not ok*
1557 m.def("foo", &foo, R"mydelimiter(The foo function
1558
1559 Parameters
1560 ----------
1561 )mydelimiter");
1562
Wenzel Jakob9e0a0562016-05-05 20:33:54 +02001563.. [#f4] http://www.sphinx-doc.org
Wenzel Jakobca8dc082016-06-03 14:24:17 +02001564.. [#f5] http://github.com/pybind/python_example