Wenzel Jakob | de623a7 | 2016-03-09 21:11:19 +0100 | [diff] [blame] | 1 | Frequently asked questions |
| 2 | ########################## |
| 3 | |
Wenzel Jakob | c62360d | 2016-05-03 14:32:47 +0200 | [diff] [blame] | 4 | "ImportError: dynamic module does not define init function" |
| 5 | =========================================================== |
Wenzel Jakob | 50ed361 | 2016-04-11 17:38:25 +0200 | [diff] [blame] | 6 | |
| 7 | 1. Make sure that the name specified in ``pybind::module`` and |
| 8 | ``PYBIND11_PLUGIN`` is consistent and identical to the filename of the |
| 9 | extension library. The latter should not contain any extra prefixes (e.g. |
| 10 | ``test.so`` instead of ``libtest.so``). |
| 11 | |
| 12 | 2. If the above did not fix your issue, then you are likely using an |
| 13 | incompatible version of Python (for instance, the extension library was |
| 14 | compiled against Python 2, while the interpreter is running on top of some |
Wenzel Jakob | c62360d | 2016-05-03 14:32:47 +0200 | [diff] [blame] | 15 | version of Python 3, or vice versa) |
| 16 | |
| 17 | "Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``" |
| 18 | ======================================================================== |
| 19 | |
| 20 | See item 2 of the first answer. |
| 21 | |
Wenzel Jakob | 70f5a4d | 2016-09-05 17:19:18 +0900 | [diff] [blame] | 22 | "SystemError: dynamic module not initialized properly" |
| 23 | ====================================================== |
| 24 | |
| 25 | See item 2 of the first answer. |
| 26 | |
Wenzel Jakob | c62360d | 2016-05-03 14:32:47 +0200 | [diff] [blame] | 27 | The Python interpreter immediately crashes when importing my module |
| 28 | =================================================================== |
| 29 | |
| 30 | See item 2 of the first answer. |
Wenzel Jakob | 50ed361 | 2016-04-11 17:38:25 +0200 | [diff] [blame] | 31 | |
Wenzel Jakob | f600c1d | 2016-06-03 14:47:54 +0200 | [diff] [blame] | 32 | CMake doesn't detect the right Python version |
| 33 | ============================================= |
Wenzel Jakob | a439cca | 2016-05-17 10:47:52 +0200 | [diff] [blame] | 34 | |
Wenzel Jakob | f600c1d | 2016-06-03 14:47:54 +0200 | [diff] [blame] | 35 | The CMake-based build system will try to automatically detect the installed |
| 36 | version of Python and link against that. When this fails, or when there are |
| 37 | multiple versions of Python and it finds the wrong one, delete |
| 38 | ``CMakeCache.txt`` and then invoke CMake as follows: |
Wenzel Jakob | a439cca | 2016-05-17 10:47:52 +0200 | [diff] [blame] | 39 | |
| 40 | .. code-block:: bash |
| 41 | |
Wenzel Jakob | f600c1d | 2016-06-03 14:47:54 +0200 | [diff] [blame] | 42 | cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> . |
Wenzel Jakob | a439cca | 2016-05-17 10:47:52 +0200 | [diff] [blame] | 43 | |
Wenzel Jakob | de623a7 | 2016-03-09 21:11:19 +0100 | [diff] [blame] | 44 | Limitations involving reference arguments |
| 45 | ========================================= |
| 46 | |
| 47 | In C++, it's fairly common to pass arguments using mutable references or |
| 48 | mutable pointers, which allows both read and write access to the value |
| 49 | supplied by the caller. This is sometimes done for efficiency reasons, or to |
| 50 | realize functions that have multiple return values. Here are two very basic |
| 51 | examples: |
| 52 | |
| 53 | .. code-block:: cpp |
| 54 | |
| 55 | void increment(int &i) { i++; } |
| 56 | void increment_ptr(int *i) { (*i)++; } |
| 57 | |
| 58 | In Python, all arguments are passed by reference, so there is no general |
| 59 | issue in binding such code from Python. |
| 60 | |
| 61 | However, certain basic Python types (like ``str``, ``int``, ``bool``, |
| 62 | ``float``, etc.) are **immutable**. This means that the following attempt |
| 63 | to port the function to Python doesn't have the same effect on the value |
| 64 | provided by the caller -- in fact, it does nothing at all. |
| 65 | |
| 66 | .. code-block:: python |
| 67 | |
| 68 | def increment(i): |
| 69 | i += 1 # nope.. |
| 70 | |
Wenzel Jakob | 4a48afb | 2016-03-09 21:31:21 +0100 | [diff] [blame] | 71 | pybind11 is also affected by such language-level conventions, which means that |
| 72 | binding ``increment`` or ``increment_ptr`` will also create Python functions |
| 73 | that don't modify their arguments. |
Wenzel Jakob | de623a7 | 2016-03-09 21:11:19 +0100 | [diff] [blame] | 74 | |
Wenzel Jakob | 4a48afb | 2016-03-09 21:31:21 +0100 | [diff] [blame] | 75 | Although inconvenient, one workaround is to encapsulate the immutable types in |
Wenzel Jakob | 2e03a58 | 2016-04-14 11:27:15 +0200 | [diff] [blame] | 76 | a custom type that does allow modifications. |
Wenzel Jakob | 4a48afb | 2016-03-09 21:31:21 +0100 | [diff] [blame] | 77 | |
| 78 | An other alternative involves binding a small wrapper lambda function that |
| 79 | returns a tuple with all output arguments (see the remainder of the |
| 80 | documentation for examples on binding lambda functions). An example: |
| 81 | |
| 82 | .. code-block:: cpp |
| 83 | |
| 84 | int foo(int &i) { i++; return 123; } |
| 85 | |
| 86 | and the binding code |
| 87 | |
| 88 | .. code-block:: cpp |
| 89 | |
| 90 | m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); }); |
| 91 | |
Wenzel Jakob | 2e03a58 | 2016-04-14 11:27:15 +0200 | [diff] [blame] | 92 | |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 93 | How can I reduce the build time? |
| 94 | ================================ |
| 95 | |
Jason Rhinelander | fb7c9fd | 2016-10-22 12:54:33 -0400 | [diff] [blame] | 96 | It's good practice to split binding code over multiple files, as in the |
| 97 | following example: |
| 98 | |
| 99 | :file:`example.cpp`: |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 100 | |
| 101 | .. code-block:: cpp |
| 102 | |
| 103 | void init_ex1(py::module &); |
| 104 | void init_ex2(py::module &); |
| 105 | /* ... */ |
| 106 | |
| 107 | PYBIND11_PLUGIN(example) { |
| 108 | py::module m("example", "pybind example plugin"); |
| 109 | |
| 110 | init_ex1(m); |
| 111 | init_ex2(m); |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 112 | /* ... */ |
| 113 | |
| 114 | return m.ptr(); |
| 115 | } |
| 116 | |
Jason Rhinelander | fb7c9fd | 2016-10-22 12:54:33 -0400 | [diff] [blame] | 117 | :file:`ex1.cpp`: |
| 118 | |
| 119 | .. code-block:: cpp |
| 120 | |
| 121 | void init_ex1(py::module &m) { |
| 122 | m.def("add", [](int a, int b) { return a + b; }); |
| 123 | } |
| 124 | |
| 125 | :file:`ex2.cpp`: |
| 126 | |
| 127 | .. code-block:: cpp |
| 128 | |
| 129 | void init_ex1(py::module &m) { |
| 130 | m.def("sub", [](int a, int b) { return a - b; }); |
| 131 | } |
| 132 | |
| 133 | :command:`python`: |
| 134 | |
| 135 | .. code-block:: pycon |
| 136 | |
| 137 | >>> import example |
| 138 | >>> example.add(1, 2) |
| 139 | 3 |
| 140 | >>> example.sub(1, 1) |
| 141 | 0 |
| 142 | |
| 143 | As shown above, the various ``init_ex`` functions should be contained in |
| 144 | separate files that can be compiled independently from one another, and then |
| 145 | linked together into the same final shared object. Following this approach |
| 146 | will: |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 147 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 148 | 1. reduce memory requirements per compilation unit. |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 149 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 150 | 2. enable parallel builds (if desired). |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 151 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 152 | 3. allow for faster incremental builds. For instance, when a single class |
Jason Rhinelander | 20ef626 | 2016-09-21 13:39:02 -0400 | [diff] [blame] | 153 | definition is changed, only a subset of the binding code will generally need |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 154 | to be recompiled. |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 155 | |
Wenzel Jakob | 2fb5f1d | 2016-11-16 17:36:54 +0100 | [diff] [blame] | 156 | "recursive template instantiation exceeded maximum depth of 256" |
| 157 | ================================================================ |
| 158 | |
| 159 | If you receive an error about excessive recursive template evaluation, try |
| 160 | specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The |
| 161 | culprit is generally the generation of function signatures at compile time |
| 162 | using C++14 template metaprogramming. |
| 163 | |
| 164 | |
Lori A. Burns | 5cafc99 | 2016-12-13 10:55:38 -0500 | [diff] [blame] | 165 | .. _`faq:symhidden`: |
| 166 | |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 167 | How can I create smaller binaries? |
| 168 | ================================== |
| 169 | |
| 170 | To do its job, pybind11 extensively relies on a programming technique known as |
| 171 | *template metaprogramming*, which is a way of performing computation at compile |
| 172 | time using type information. Template metaprogamming usually instantiates code |
| 173 | involving significant numbers of deeply nested types that are either completely |
Jason Rhinelander | 20ef626 | 2016-09-21 13:39:02 -0400 | [diff] [blame] | 174 | removed or reduced to just a few instructions during the compiler's optimization |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 175 | phase. However, due to the nested nature of these types, the resulting symbol |
| 176 | names in the compiled extension library can be extremely long. For instance, |
| 177 | the included test suite contains the following symbol: |
| 178 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 179 | .. only:: html |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 180 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 181 | .. code-block:: none |
| 182 | |
Wenzel Jakob | fe34241 | 2016-09-06 13:02:29 +0900 | [diff] [blame] | 183 | __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 184 | |
| 185 | .. only:: not html |
| 186 | |
| 187 | .. code-block:: cpp |
| 188 | |
| 189 | __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 190 | |
| 191 | which is the mangled form of the following function type: |
| 192 | |
| 193 | .. code-block:: cpp |
| 194 | |
| 195 | pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28]) |
| 196 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 197 | The memory needed to store just the mangled name of this function (196 bytes) |
| 198 | is larger than the actual piece of code (111 bytes) it represents! On the other |
| 199 | hand, it's silly to even give this function a name -- after all, it's just a |
| 200 | tiny cog in a bigger piece of machinery that is not exposed to the outside |
| 201 | world. So we'll generally only want to export symbols for those functions which |
| 202 | are actually called from the outside. |
Wenzel Jakob | c79dbe4 | 2016-04-17 21:54:31 +0200 | [diff] [blame] | 203 | |
| 204 | This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC |
| 205 | and Clang, which sets the default symbol visibility to *hidden*. It's best to |
| 206 | do this only for release builds, since the symbol names can be helpful in |
| 207 | debugging sessions. On Visual Studio, symbols are already hidden by default, so |
| 208 | nothing needs to be done there. Needless to say, this has a tremendous impact |
| 209 | on the final binary size of the resulting extension library. |
| 210 | |
| 211 | Another aspect that can require a fair bit of code are function signature |
| 212 | descriptions. pybind11 automatically generates human-readable function |
| 213 | signatures for docstrings, e.g.: |
| 214 | |
| 215 | .. code-block:: none |
| 216 | |
| 217 | | __init__(...) |
| 218 | | __init__(*args, **kwargs) |
| 219 | | Overloaded function. |
| 220 | | |
| 221 | | 1. __init__(example.Example1) -> NoneType |
| 222 | | |
| 223 | | Docstring for overload #1 goes here |
| 224 | | |
| 225 | | 2. __init__(example.Example1, int) -> NoneType |
| 226 | | |
| 227 | | Docstring for overload #2 goes here |
| 228 | | |
| 229 | | 3. __init__(example.Example1, example.Example1) -> NoneType |
| 230 | | |
| 231 | | Docstring for overload #3 goes here |
| 232 | |
| 233 | |
| 234 | In C++11 mode, these are generated at run time using string concatenation, |
| 235 | which can amount to 10-20% of the size of the resulting binary. If you can, |
| 236 | enable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which |
| 237 | case signatures are efficiently pre-generated at compile time. Unfortunately, |
| 238 | Visual Studio's C++14 support (``constexpr``) is not good enough as of April |
| 239 | 2016, so it always uses the more expensive run-time approach. |
Wenzel Jakob | c62360d | 2016-05-03 14:32:47 +0200 | [diff] [blame] | 240 | |
| 241 | Working with ancient Visual Studio 2009 builds on Windows |
| 242 | ========================================================= |
| 243 | |
| 244 | The official Windows distributions of Python are compiled using truly |
| 245 | ancient versions of Visual Studio that lack good C++11 support. Some users |
| 246 | implicitly assume that it would be impossible to load a plugin built with |
| 247 | Visual Studio 2015 into a Python distribution that was compiled using Visual |
| 248 | Studio 2009. However, no such issue exists: it's perfectly legitimate to |
| 249 | interface DLLs that are built with different compilers and/or C libraries. |
| 250 | Common gotchas to watch out for involve not ``free()``-ing memory region |
| 251 | that that were ``malloc()``-ed in another shared library, using data |
| 252 | structures with incompatible ABIs, and so on. pybind11 is very careful not |
| 253 | to make these types of mistakes. |