Improve documentation of Python and C++ exceptions (#2408)
The main change is to treat error_already_set as a separate category
of exception that arises in different circumstances and needs to be
handled differently. The asymmetry between Python and C++ exceptions
is further emphasized.
diff --git a/.gitignore b/.gitignore
index 47e010c..5613b36 100644
--- a/.gitignore
+++ b/.gitignore
@@ -38,3 +38,4 @@
pybind11Config*.cmake
pybind11Targets.cmake
/*env*
+/.vscode
diff --git a/docs/advanced/exceptions.rst b/docs/advanced/exceptions.rst
index 7206310..b7d3601 100644
--- a/docs/advanced/exceptions.rst
+++ b/docs/advanced/exceptions.rst
@@ -1,18 +1,24 @@
Exceptions
##########
-Built-in exception translation
-==============================
+Built-in C++ to Python exception translation
+============================================
-When C++ code invoked from Python throws an ``std::exception``, it is
-automatically converted into a Python ``Exception``. pybind11 defines multiple
-special exception classes that will map to different types of Python
-exceptions:
+When Python calls C++ code through pybind11, pybind11 provides a C++ exception handler
+that will trap C++ exceptions, translate them to the corresponding Python exception,
+and raise them so that Python code can handle them.
+
+pybind11 defines translations for ``std::exception`` and its standard
+subclasses, and several special exception classes that translate to specific
+Python exceptions. Note that these are not actually Python exceptions, so they
+cannot be examined using the Python C API. Instead, they are pure C++ objects
+that pybind11 will translate the corresponding Python exception when they arrive
+at its exception handler.
.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}|
+--------------------------------------+--------------------------------------+
-| C++ exception type | Python exception type |
+| Exception thrown by C++ | Translated to Python exception type |
+======================================+======================================+
| :class:`std::exception` | ``RuntimeError`` |
+--------------------------------------+--------------------------------------+
@@ -46,22 +52,11 @@
| | ``__setitem__`` in dict-like |
| | objects, etc.) |
+--------------------------------------+--------------------------------------+
-| :class:`pybind11::error_already_set` | Indicates that the Python exception |
-| | flag has already been set via Python |
-| | API calls from C++ code; this C++ |
-| | exception is used to propagate such |
-| | a Python exception back to Python. |
-+--------------------------------------+--------------------------------------+
-When a Python function invoked from C++ throws an exception, pybind11 will convert
-it into a C++ exception of type :class:`error_already_set` whose string payload
-contains a textual summary. If you call the Python C-API directly, and it
-returns an error, you should ``throw py::error_already_set();``, which allows
-pybind11 to deal with the exception and pass it back to the Python interpreter.
-(Another option is to call ``PyErr_Clear`` in the
-`Python C-API <https://docs.python.org/3/c-api/exceptions.html#c.PyErr_Clear>`_
-to clear the error. The Python error must be thrown or cleared, or Python/pybind11
-will be left in an invalid state.)
+Exception translation is not bidirectional. That is, *catching* the C++
+exceptions defined above above will not trap exceptions that originate from
+Python. For that, catch :class:`pybind11::error_already_set`. See :ref:`below
+<handling_python_exceptions_cpp>` for further details.
There is also a special exception :class:`cast_error` that is thrown by
:func:`handle::call` when the input arguments cannot be converted to Python
@@ -106,7 +101,6 @@
to make this a static declaration when using it inside a lambda expression
without requiring capturing).
-
The following example demonstrates this for a hypothetical exception classes
``MyCustomException`` and ``OtherException``: the first is translated to a
custom python exception ``MyCustomError``, while the second is translated to a
@@ -140,7 +134,7 @@
.. note::
- You must call either ``PyErr_SetString`` or a custom exception's call
+ Call either ``PyErr_SetString`` or a custom exception's call
operator (``exc(string)``) for every exception caught in a custom exception
translator. Failure to do so will cause Python to crash with ``SystemError:
error return without exception set``.
@@ -149,6 +143,103 @@
may be explicitly (re-)thrown to delegate it to the other,
previously-declared existing exception translators.
+.. _handling_python_exceptions_cpp:
+
+Handling exceptions from Python in C++
+======================================
+
+When C++ calls Python functions, such as in a callback function or when
+manipulating Python objects, and Python raises an ``Exception``, pybind11
+converts the Python exception into a C++ exception of type
+:class:`pybind11::error_already_set` whose payload contains a C++ string textual
+summary and the actual Python exception. ``error_already_set`` is used to
+propagate Python exception back to Python (or possibly, handle them in C++).
+
+.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}|
+
++--------------------------------------+--------------------------------------+
+| Exception raised in Python | Thrown as C++ exception type |
++======================================+======================================+
+| Any Python ``Exception`` | :class:`pybind11::error_already_set` |
++--------------------------------------+--------------------------------------+
+
+For example:
+
+.. code-block:: cpp
+
+ try {
+ // open("missing.txt", "r")
+ auto file = py::module::import("io").attr("open")("missing.txt", "r");
+ auto text = file.attr("read")();
+ file.attr("close")();
+ } catch (py::error_already_set &e) {
+ if (e.matches(PyExc_FileNotFoundError)) {
+ py::print("missing.txt not found");
+ } else if (e.match(PyExc_PermissionError)) {
+ py::print("missing.txt found but not accessible");
+ } else {
+ throw;
+ }
+ }
+
+Note that C++ to Python exception translation does not apply here, since that is
+a method for translating C++ exceptions to Python, not vice versa. The error raised
+from Python is always ``error_already_set``.
+
+This example illustrates this behavior:
+
+.. code-block:: cpp
+
+ try {
+ py::eval("raise ValueError('The Ring')");
+ } catch (py::value_error &boromir) {
+ // Boromir never gets the ring
+ assert(false);
+ } catch (py::error_already_set &frodo) {
+ // Frodo gets the ring
+ py::print("I will take the ring");
+ }
+
+ try {
+ // py::value_error is a request for pybind11 to raise a Python exception
+ throw py::value_error("The ball");
+ } catch (py::error_already_set &cat) {
+ // cat won't catch the ball since
+ // py::value_error is not a Python exception
+ assert(false);
+ } catch (py::value_error &dog) {
+ // dog will catch the ball
+ py::print("Run Spot run");
+ throw; // Throw it again (pybind11 will raise ValueError)
+ }
+
+Handling errors from the Python C API
+=====================================
+
+Where possible, use :ref:`pybind11 wrappers <wrappers>` instead of calling
+the Python C API directly. When calling the Python C API directly, in
+addition to manually managing reference counts, one must follow the pybind11
+error protocol, which is outlined here.
+
+After calling the Python C API, if Python returns an error,
+``throw py::error_already_set();``, which allows pybind11 to deal with the
+exception and pass it back to the Python interpreter. This includes calls to
+the error setting functions such as ``PyErr_SetString``.
+
+.. code-block:: cpp
+
+ PyErr_SetString(PyExc_TypeError, "C API type error demo");
+ throw py::error_already_set();
+
+ // But it would be easier to simply...
+ throw py::type_error("pybind11 wrapper type error");
+
+Alternately, to ignore the error, call `PyErr_Clear
+<https://docs.python.org/3/c-api/exceptions.html#c.PyErr_Clear>`_.
+
+Any Python error must be thrown or cleared, or Python/pybind11 will be left in
+an invalid state.
+
.. _unraisable_exceptions:
Handling unraisable exceptions
@@ -156,20 +247,24 @@
If a Python function invoked from a C++ destructor or any function marked
``noexcept(true)`` (collectively, "noexcept functions") throws an exception, there
-is no way to propagate the exception, as such functions may not throw at
-run-time.
+is no way to propagate the exception, as such functions may not throw.
+Should they throw or fail to catch any exceptions in their call graph,
+the C++ runtime calls ``std::terminate()`` to abort immediately.
-Neither Python nor C++ allow exceptions raised in a noexcept function to propagate. In
-Python, an exception raised in a class's ``__del__`` method is logged as an
-unraisable error. In Python 3.8+, a system hook is triggered and an auditing
-event is logged. In C++, ``std::terminate()`` is called to abort immediately.
+Similarly, Python exceptions raised in a class's ``__del__`` method do not
+propagate, but are logged by Python as an unraisable error. In Python 3.8+, a
+`system hook is triggered
+<https://docs.python.org/3/library/sys.html#sys.unraisablehook>`_
+and an auditing event is logged.
Any noexcept function should have a try-catch block that traps
-class:`error_already_set` (or any other exception that can occur). Note that pybind11
-wrappers around Python exceptions such as :class:`pybind11::value_error` are *not*
-Python exceptions; they are C++ exceptions that pybind11 catches and converts to
-Python exceptions. Noexcept functions cannot propagate these exceptions either.
-You can convert them to Python exceptions and then discard as unraisable.
+class:`error_already_set` (or any other exception that can occur). Note that
+pybind11 wrappers around Python exceptions such as
+:class:`pybind11::value_error` are *not* Python exceptions; they are C++
+exceptions that pybind11 catches and converts to Python exceptions. Noexcept
+functions cannot propagate these exceptions either. A useful approach is to
+convert them to Python exceptions and then ``discard_as_unraisable`` as shown
+below.
.. code-block:: cpp
@@ -183,8 +278,6 @@
eas.discard_as_unraisable(__func__);
} catch (const std::exception &e) {
// Log and discard C++ exceptions.
- // (We cannot use discard_as_unraisable, since we have a generic C++
- // exception, not an exception that originated from Python.)
third_party::log(e);
}
}
diff --git a/docs/advanced/pycpp/object.rst b/docs/advanced/pycpp/object.rst
index 19a226a..07525d0 100644
--- a/docs/advanced/pycpp/object.rst
+++ b/docs/advanced/pycpp/object.rst
@@ -1,6 +1,8 @@
Python types
############
+.. _wrappers:
+
Available wrappers
==================
@@ -168,3 +170,11 @@
Python functions from C++, including keywords arguments and unpacking.
.. _PEP448: https://www.python.org/dev/peps/pep-0448/
+
+Handling exceptions
+===================
+
+Python exceptions from wrapper classes will be thrown as a ``py::error_already_set``.
+See :ref:`Handling exceptions from Python in C++
+<handling_python_exceptions_cpp>` for more information on handling exceptions
+raised when calling C++ wrapper classes.