Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 1 | Benchmark |
| 2 | ========= |
| 3 | |
| 4 | The following is the result of a synthetic benchmark comparing both compilation |
Wenzel Jakob | 192eb88 | 2016-08-19 09:38:14 +0200 | [diff] [blame] | 5 | time and module size of pybind11 against Boost.Python. A detailed report about a |
| 6 | Boost.Python to pybind11 conversion of a real project is available here: [#f1]_. |
| 7 | |
| 8 | .. [#f1] http://graylab.jhu.edu/RosettaCon2016/PyRosetta-4.pdf |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 9 | |
Wenzel Jakob | 66c9a40 | 2016-01-17 22:36:36 +0100 | [diff] [blame] | 10 | Setup |
| 11 | ----- |
| 12 | |
| 13 | A python script (see the ``docs/benchmark.py`` file) was used to generate a set |
| 14 | of files with dummy classes whose count increases for each successive benchmark |
| 15 | (between 1 and 2048 classes in powers of two). Each class has four methods with |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 16 | a randomly generated signature with a return value and four arguments. (There |
| 17 | was no particular reason for this setup other than the desire to generate many |
| 18 | unique function signatures whose count could be controlled in a simple way.) |
| 19 | |
| 20 | Here is an example of the binding code for one class: |
| 21 | |
| 22 | .. code-block:: cpp |
| 23 | |
| 24 | ... |
| 25 | class cl034 { |
| 26 | public: |
| 27 | cl279 *fn_000(cl084 *, cl057 *, cl065 *, cl042 *); |
| 28 | cl025 *fn_001(cl098 *, cl262 *, cl414 *, cl121 *); |
| 29 | cl085 *fn_002(cl445 *, cl297 *, cl145 *, cl421 *); |
| 30 | cl470 *fn_003(cl200 *, cl323 *, cl332 *, cl492 *); |
| 31 | }; |
| 32 | ... |
| 33 | |
Dean Moldovan | 443ab59 | 2017-04-24 01:51:44 +0200 | [diff] [blame] | 34 | PYBIND11_MODULE(example, m) { |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 35 | ... |
| 36 | py::class_<cl034>(m, "cl034") |
| 37 | .def("fn_000", &cl034::fn_000) |
| 38 | .def("fn_001", &cl034::fn_001) |
| 39 | .def("fn_002", &cl034::fn_002) |
| 40 | .def("fn_003", &cl034::fn_003) |
| 41 | ... |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 42 | } |
| 43 | |
| 44 | The Boost.Python version looks almost identical except that a return value |
| 45 | policy had to be specified as an argument to ``def()``. For both libraries, |
| 46 | compilation was done with |
| 47 | |
| 48 | .. code-block:: bash |
| 49 | |
Wenzel Jakob | 66c9a40 | 2016-01-17 22:36:36 +0100 | [diff] [blame] | 50 | Apple LLVM version 7.0.2 (clang-700.1.81) |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 51 | |
| 52 | and the following compilation flags |
| 53 | |
| 54 | .. code-block:: bash |
| 55 | |
Wenzel Jakob | 66c9a40 | 2016-01-17 22:36:36 +0100 | [diff] [blame] | 56 | g++ -Os -shared -rdynamic -undefined dynamic_lookup -fvisibility=hidden -std=c++14 |
| 57 | |
| 58 | Compilation time |
| 59 | ---------------- |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 60 | |
| 61 | The following log-log plot shows how the compilation time grows for an |
Wenzel Jakob | 66c9a40 | 2016-01-17 22:36:36 +0100 | [diff] [blame] | 62 | increasing number of class and function declarations. pybind11 includes many |
| 63 | fewer headers, which initially leads to shorter compilation times, but the |
| 64 | performance is ultimately fairly similar (pybind11 is 19.8 seconds faster for |
| 65 | the largest largest file with 2048 classes and a total of 8192 methods -- a |
| 66 | modest **1.2x** speedup relative to Boost.Python, which required 116.35 |
| 67 | seconds). |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 68 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 69 | .. only:: not latex |
| 70 | |
| 71 | .. image:: pybind11_vs_boost_python1.svg |
| 72 | |
| 73 | .. only:: latex |
| 74 | |
| 75 | .. image:: pybind11_vs_boost_python1.png |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 76 | |
Wenzel Jakob | 66c9a40 | 2016-01-17 22:36:36 +0100 | [diff] [blame] | 77 | Module size |
| 78 | ----------- |
| 79 | |
| 80 | Differences between the two libraries become much more pronounced when |
| 81 | considering the file size of the generated Python plugin: for the largest file, |
| 82 | the binary generated by Boost.Python required 16.8 MiB, which was **2.17 |
| 83 | times** / **9.1 megabytes** larger than the output generated by pybind11. For |
| 84 | very small inputs, Boost.Python has an edge in the plot below -- however, note |
| 85 | that it stores many definitions in an external library, whose size was not |
| 86 | included here, hence the comparison is slightly shifted in Boost.Python's |
| 87 | favor. |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 88 | |
Wenzel Jakob | f64feaf | 2016-04-28 14:33:45 +0200 | [diff] [blame] | 89 | .. only:: not latex |
| 90 | |
| 91 | .. image:: pybind11_vs_boost_python2.svg |
| 92 | |
| 93 | .. only:: latex |
| 94 | |
| 95 | .. image:: pybind11_vs_boost_python2.png |
| 96 | |
Wenzel Jakob | 5cd3311 | 2015-10-20 00:58:59 +0200 | [diff] [blame] | 97 | |