Wenzel Jakob | de623a7 | 2016-03-09 21:11:19 +0100 | [diff] [blame] | 1 | Frequently asked questions |
| 2 | ########################## |
| 3 | |
Wenzel Jakob | c62360d | 2016-05-03 14:32:47 +0200 | [diff] [blame] | 4 | "ImportError: dynamic module does not define init function" |
| 5 | =========================================================== |
Wenzel Jakob | 50ed361 | 2016-04-11 17:38:25 +0200 | [diff] [blame] | 6 | |
| 7 | 1. Make sure that the name specified in ``pybind::module`` and |
| 8 | ``PYBIND11_PLUGIN`` is consistent and identical to the filename of the |
| 9 | extension library. The latter should not contain any extra prefixes (e.g. |
| 10 | ``test.so`` instead of ``libtest.so``). |
| 11 | |
| 12 | 2. If the above did not fix your issue, then you are likely using an |
| 13 | incompatible version of Python (for instance, the extension library was |
| 14 | compiled against Python 2, while the interpreter is running on top of some |
Wenzel Jakob | c62360d | 2016-05-03 14:32:47 +0200 | [diff] [blame] | 15 | version of Python 3, or vice versa) |
| 16 | |
| 17 | "Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``" |
| 18 | ======================================================================== |
| 19 | |
| 20 | See item 2 of the first answer. |
| 21 | |
| 22 | The Python interpreter immediately crashes when importing my module |
| 23 | =================================================================== |
| 24 | |
| 25 | See item 2 of the first answer. |
Wenzel Jakob | 50ed361 | 2016-04-11 17:38:25 +0200 | [diff] [blame] | 26 | |
Wenzel Jakob | a439cca | 2016-05-17 10:47:52 +0200 | [diff] [blame] | 27 | CMake doesn't detect the right Python version, or it finds mismatched interpreter and library versions |
| 28 | ====================================================================================================== |
| 29 | |
| 30 | The Python detection logic of CMake is flawed and can sometimes fail to find |
| 31 | the desired Python version, or it chooses mismatched interpreter and library |
| 32 | versions. A longer discussion is available on the pybind11 issue tracker |
| 33 | [#f1]_, though this is ultimately not a pybind11 issue. |
| 34 | |
| 35 | To force the build system to choose a particular version, delete CMakeCache.txt |
| 36 | and then invoke CMake as follows: |
| 37 | |
| 38 | .. code-block:: bash |
| 39 | |
| 40 | cmake -DPYTHON_EXECUTABLE:FILEPATH=<...> \ |
| 41 | -DPYTHON_LIBRARY:FILEPATH=<...> \ |
| 42 | -DPYTHON_INCLUDE_DIR:PATH=<...> . |
| 43 | |
| 44 | .. [#f1] http://github.com/pybind/pybind11/issues/99 |
| 45 | |
Wenzel Jakob | de623a7 | 2016-03-09 21:11:19 +0100 | [diff] [blame] | 46 | Limitations involving reference arguments |
| 47 | ========================================= |
| 48 | |
| 49 | In C++, it's fairly common to pass arguments using mutable references or |
| 50 | mutable pointers, which allows both read and write access to the value |
| 51 | supplied by the caller. This is sometimes done for efficiency reasons, or to |
| 52 | realize functions that have multiple return values. Here are two very basic |
| 53 | examples: |
| 54 | |
| 55 | .. code-block:: cpp |
| 56 | |
| 57 | void increment(int &i) { i++; } |
| 58 | void increment_ptr(int *i) { (*i)++; } |
| 59 | |
| 60 | In Python, all arguments are passed by reference, so there is no general |
| 61 | issue in binding such code from Python. |
| 62 | |
| 63 | However, certain basic Python types (like ``str``, ``int``, ``bool``, |
| 64 | ``float``, etc.) are **immutable**. This means that the following attempt |
| 65 | to port the function to Python doesn't have the same effect on the value |
| 66 | provided by the caller -- in fact, it does nothing at all. |
| 67 | |
| 68 | .. code-block:: python |
| 69 | |
| 70 | def increment(i): |
| 71 | i += 1 # nope.. |
| 72 | |
Wenzel Jakob | 4a48afb | 2016-03-09 21:31:21 +0100 | [diff] [blame] | 73 | pybind11 is also affected by such language-level conventions, which means that |
| 74 | binding ``increment`` or ``increment_ptr`` will also create Python functions |
| 75 | that don't modify their arguments. |
Wenzel Jakob | de623a7 | 2016-03-09 21:11:19 +0100 | [diff] [blame] | 76 | |
Wenzel Jakob | 4a48afb | 2016-03-09 21:31:21 +0100 | [diff] [blame] | 77 | Although inconvenient, one workaround is to encapsulate the immutable types in |
Wenzel Jakob | 2e03a58 | 2016-04-14 11:27:15 +0200 | [diff] [blame] | 78 | a custom type that does allow modifications. |
Wenzel Jakob | 4a48afb | 2016-03-09 21:31:21 +0100 | [diff] [blame] | 79 | |
| 80 | An other alternative involves binding a small wrapper lambda function that |
| 81 | returns a tuple with all output arguments (see the remainder of the |
| 82 | documentation for examples on binding lambda functions). An example: |
| 83 | |
| 84 | .. code-block:: cpp |
| 85 | |
| 86 | int foo(int &i) { i++; return 123; } |
| 87 | |
| 88 | and the binding code |
| 89 | |
| 90 | .. code-block:: cpp |
| 91 | |
| 92 | m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); }); |
| 93 | |
Wenzel Jakob | 2e03a58 | 2016-04-14 11:27:15 +0200 | [diff] [blame] | 94 | |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 95 | How can I reduce the build time? |
| 96 | ================================ |
| 97 | |
| 98 | It's good practice to split binding code over multiple files, as is done in |
| 99 | the included file :file:`example/example.cpp`. |
| 100 | |
| 101 | .. code-block:: cpp |
| 102 | |
| 103 | void init_ex1(py::module &); |
| 104 | void init_ex2(py::module &); |
| 105 | /* ... */ |
| 106 | |
| 107 | PYBIND11_PLUGIN(example) { |
| 108 | py::module m("example", "pybind example plugin"); |
| 109 | |
| 110 | init_ex1(m); |
| 111 | init_ex2(m); |
| 112 | |
| 113 | /* ... */ |
| 114 | |
| 115 | return m.ptr(); |
| 116 | } |
| 117 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 118 | The various ``init_ex`` functions should be contained in separate files that |
| 119 | can be compiled independently from another. Following this approach will |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 120 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 121 | 1. reduce memory requirements per compilation unit. |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 122 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 123 | 2. enable parallel builds (if desired). |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 124 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 125 | 3. allow for faster incremental builds. For instance, when a single class |
| 126 | definiton is changed, only a subset of the binding code will generally need |
| 127 | to be recompiled. |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 128 | |
| 129 | How can I create smaller binaries? |
| 130 | ================================== |
| 131 | |
| 132 | To do its job, pybind11 extensively relies on a programming technique known as |
| 133 | *template metaprogramming*, which is a way of performing computation at compile |
| 134 | time using type information. Template metaprogamming usually instantiates code |
| 135 | involving significant numbers of deeply nested types that are either completely |
| 136 | removed or reduced to just a few instrutions during the compiler's optimization |
| 137 | phase. However, due to the nested nature of these types, the resulting symbol |
| 138 | names in the compiled extension library can be extremely long. For instance, |
| 139 | the included test suite contains the following symbol: |
| 140 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 141 | .. only:: html |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 142 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 143 | .. code-block:: none |
| 144 | |
| 145 | __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ |
| 146 | |
| 147 | .. only:: not html |
| 148 | |
| 149 | .. code-block:: cpp |
| 150 | |
| 151 | __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 152 | |
| 153 | which is the mangled form of the following function type: |
| 154 | |
| 155 | .. code-block:: cpp |
| 156 | |
| 157 | pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28]) |
| 158 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 159 | The memory needed to store just the mangled name of this function (196 bytes) |
| 160 | is larger than the actual piece of code (111 bytes) it represents! On the other |
| 161 | hand, it's silly to even give this function a name -- after all, it's just a |
| 162 | tiny cog in a bigger piece of machinery that is not exposed to the outside |
| 163 | world. So we'll generally only want to export symbols for those functions which |
| 164 | are actually called from the outside. |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 165 | |
| 166 | This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC |
| 167 | and Clang, which sets the default symbol visibility to *hidden*. It's best to |
| 168 | do this only for release builds, since the symbol names can be helpful in |
| 169 | debugging sessions. On Visual Studio, symbols are already hidden by default, so |
| 170 | nothing needs to be done there. Needless to say, this has a tremendous impact |
| 171 | on the final binary size of the resulting extension library. |
| 172 | |
| 173 | Another aspect that can require a fair bit of code are function signature |
| 174 | descriptions. pybind11 automatically generates human-readable function |
| 175 | signatures for docstrings, e.g.: |
| 176 | |
| 177 | .. code-block:: none |
| 178 | |
| 179 | | __init__(...) |
| 180 | | __init__(*args, **kwargs) |
| 181 | | Overloaded function. |
| 182 | | |
| 183 | | 1. __init__(example.Example1) -> NoneType |
| 184 | | |
| 185 | | Docstring for overload #1 goes here |
| 186 | | |
| 187 | | 2. __init__(example.Example1, int) -> NoneType |
| 188 | | |
| 189 | | Docstring for overload #2 goes here |
| 190 | | |
| 191 | | 3. __init__(example.Example1, example.Example1) -> NoneType |
| 192 | | |
| 193 | | Docstring for overload #3 goes here |
| 194 | |
| 195 | |
| 196 | In C++11 mode, these are generated at run time using string concatenation, |
| 197 | which can amount to 10-20% of the size of the resulting binary. If you can, |
| 198 | enable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which |
| 199 | case signatures are efficiently pre-generated at compile time. Unfortunately, |
| 200 | Visual Studio's C++14 support (``constexpr``) is not good enough as of April |
| 201 | 2016, so it always uses the more expensive run-time approach. |
Wenzel Jakob | c62360d | 2016-05-03 14:32:47 +0200 | [diff] [blame] | 202 | |
| 203 | Working with ancient Visual Studio 2009 builds on Windows |
| 204 | ========================================================= |
| 205 | |
| 206 | The official Windows distributions of Python are compiled using truly |
| 207 | ancient versions of Visual Studio that lack good C++11 support. Some users |
| 208 | implicitly assume that it would be impossible to load a plugin built with |
| 209 | Visual Studio 2015 into a Python distribution that was compiled using Visual |
| 210 | Studio 2009. However, no such issue exists: it's perfectly legitimate to |
| 211 | interface DLLs that are built with different compilers and/or C libraries. |
| 212 | Common gotchas to watch out for involve not ``free()``-ing memory region |
| 213 | that that were ``malloc()``-ed in another shared library, using data |
| 214 | structures with incompatible ABIs, and so on. pybind11 is very careful not |
| 215 | to make these types of mistakes. |