blob: 63231808b4ad924b73ca73e8943a84571153cfa9 [file] [log] [blame]
Georg Brandl116aa622007-08-15 14:28:22 +00001.. highlightlang:: c
2
3
4.. _api-intro:
5
6************
7Introduction
8************
9
10The Application Programmer's Interface to Python gives C and C++ programmers
11access to the Python interpreter at a variety of levels. The API is equally
12usable from C++, but for brevity it is generally referred to as the Python/C
13API. There are two fundamentally different reasons for using the Python/C API.
14The first reason is to write *extension modules* for specific purposes; these
15are C modules that extend the Python interpreter. This is probably the most
16common use. The second reason is to use Python as a component in a larger
17application; this technique is generally referred to as :dfn:`embedding` Python
18in an application.
19
Barry Warsawb2e57942017-09-14 18:13:16 -070020Writing an extension module is a relatively well-understood process, where a
21"cookbook" approach works well. There are several tools that automate the
22process to some extent. While people have embedded Python in other
23applications since its early existence, the process of embedding Python is
24less straightforward than writing an extension.
Georg Brandl116aa622007-08-15 14:28:22 +000025
26Many API functions are useful independent of whether you're embedding or
27extending Python; moreover, most applications that embed Python will need to
28provide a custom extension as well, so it's probably a good idea to become
29familiar with writing an extension before attempting to embed Python in a real
30application.
31
32
Barry Warsawb2e57942017-09-14 18:13:16 -070033Coding standards
34================
35
36If you're writing C code for inclusion in CPython, you **must** follow the
37guidelines and standards defined in :PEP:`7`. These guidelines apply
38regardless of the version of Python you are contributing to. Following these
39conventions is not necessary for your own third party extension modules,
40unless you eventually expect to contribute them to Python.
41
42
Georg Brandl116aa622007-08-15 14:28:22 +000043.. _api-includes:
44
45Include Files
46=============
47
48All function, type and macro definitions needed to use the Python/C API are
49included in your code by the following line::
50
51 #include "Python.h"
52
53This implies inclusion of the following standard headers: ``<stdio.h>``,
Georg Brandl4f13d612010-11-23 18:14:57 +000054``<string.h>``, ``<errno.h>``, ``<limits.h>``, ``<assert.h>`` and ``<stdlib.h>``
55(if available).
Georg Brandl116aa622007-08-15 14:28:22 +000056
Georg Brandle720c0a2009-04-27 16:20:50 +000057.. note::
Georg Brandl116aa622007-08-15 14:28:22 +000058
59 Since Python may define some pre-processor definitions which affect the standard
60 headers on some systems, you *must* include :file:`Python.h` before any standard
61 headers are included.
62
63All user visible names defined by Python.h (except those defined by the included
64standard headers) have one of the prefixes ``Py`` or ``_Py``. Names beginning
65with ``_Py`` are for internal use by the Python implementation and should not be
66used by extension writers. Structure member names do not have a reserved prefix.
67
68**Important:** user code should never define names that begin with ``Py`` or
69``_Py``. This confuses the reader, and jeopardizes the portability of the user
70code to future Python versions, which may define additional names beginning with
71one of these prefixes.
72
73The header files are typically installed with Python. On Unix, these are
74located in the directories :file:`{prefix}/include/pythonversion/` and
75:file:`{exec_prefix}/include/pythonversion/`, where :envvar:`prefix` and
76:envvar:`exec_prefix` are defined by the corresponding parameters to Python's
Serhiy Storchaka885bdc42016-02-11 13:10:36 +020077:program:`configure` script and *version* is
78``'%d.%d' % sys.version_info[:2]``. On Windows, the headers are installed
79in :file:`{prefix}/include`, where :envvar:`prefix` is the installation
80directory specified to the installer.
Georg Brandl116aa622007-08-15 14:28:22 +000081
82To include the headers, place both directories (if different) on your compiler's
83search path for includes. Do *not* place the parent directories on the search
84path and then use ``#include <pythonX.Y/Python.h>``; this will break on
85multi-platform builds since the platform independent headers under
86:envvar:`prefix` include the platform specific headers from
87:envvar:`exec_prefix`.
88
89C++ users should note that though the API is defined entirely using C, the
90header files do properly declare the entry points to be ``extern "C"``, so there
91is no need to do anything special to use the API from C++.
92
93
Barry Warsawb2e57942017-09-14 18:13:16 -070094Useful macros
95=============
96
97Several useful macros are defined in the Python header files. Many are
98defined closer to where they are useful (e.g. :c:macro:`Py_RETURN_NONE`).
99Others of a more general utility are defined here. This is not necessarily a
100complete listing.
101
102.. c:macro:: Py_UNREACHABLE()
103
104 Use this when you have a code path that you do not expect to be reached.
105 For example, in the ``default:`` clause in a ``switch`` statement for which
106 all possible values are covered in ``case`` statements. Use this in places
107 where you might be tempted to put an ``assert(0)`` or ``abort()`` call.
108
Petr Viktorin8bf288e2017-11-08 14:11:16 +0100109 .. versionadded:: 3.7
110
Barry Warsawb2e57942017-09-14 18:13:16 -0700111.. c:macro:: Py_ABS(x)
112
113 Return the absolute value of ``x``.
114
Victor Stinner54cc0c02017-11-08 06:06:24 -0800115 .. versionadded:: 3.3
116
Barry Warsawb2e57942017-09-14 18:13:16 -0700117.. c:macro:: Py_MIN(x, y)
118
119 Return the minimum value between ``x`` and ``y``.
120
Victor Stinner54cc0c02017-11-08 06:06:24 -0800121 .. versionadded:: 3.3
122
Barry Warsawb2e57942017-09-14 18:13:16 -0700123.. c:macro:: Py_MAX(x, y)
124
125 Return the maximum value between ``x`` and ``y``.
126
Victor Stinner54cc0c02017-11-08 06:06:24 -0800127 .. versionadded:: 3.3
128
Barry Warsawb2e57942017-09-14 18:13:16 -0700129.. c:macro:: Py_STRINGIFY(x)
130
131 Convert ``x`` to a C string. E.g. ``Py_STRINGIFY(123)`` returns
132 ``"123"``.
133
Victor Stinner54cc0c02017-11-08 06:06:24 -0800134 .. versionadded:: 3.4
135
Barry Warsawb2e57942017-09-14 18:13:16 -0700136.. c:macro:: Py_MEMBER_SIZE(type, member)
137
138 Return the size of a structure (``type``) ``member`` in bytes.
139
Victor Stinner54cc0c02017-11-08 06:06:24 -0800140 .. versionadded:: 3.6
141
Barry Warsawb2e57942017-09-14 18:13:16 -0700142.. c:macro:: Py_CHARMASK(c)
143
144 Argument must be a character or an integer in the range [-128, 127] or [0,
145 255]. This macro returns ``c`` cast to an ``unsigned char``.
146
Barry Warsawa51b90a2017-10-06 09:53:48 -0400147.. c:macro:: Py_GETENV(s)
148
149 Like ``getenv(s)``, but returns *NULL* if :option:`-E` was passed on the
150 command line (i.e. if ``Py_IgnoreEnvironmentFlag`` is set).
151
Barry Warsawb2e57942017-09-14 18:13:16 -0700152
Georg Brandl116aa622007-08-15 14:28:22 +0000153.. _api-objects:
154
155Objects, Types and Reference Counts
156===================================
157
158.. index:: object: type
159
160Most Python/C API functions have one or more arguments as well as a return value
Georg Brandl60203b42010-10-06 10:11:56 +0000161of type :c:type:`PyObject\*`. This type is a pointer to an opaque data type
Georg Brandl116aa622007-08-15 14:28:22 +0000162representing an arbitrary Python object. Since all Python object types are
163treated the same way by the Python language in most situations (e.g.,
164assignments, scope rules, and argument passing), it is only fitting that they
165should be represented by a single C type. Almost all Python objects live on the
166heap: you never declare an automatic or static variable of type
Georg Brandl60203b42010-10-06 10:11:56 +0000167:c:type:`PyObject`, only pointer variables of type :c:type:`PyObject\*` can be
Georg Brandl116aa622007-08-15 14:28:22 +0000168declared. The sole exception are the type objects; since these must never be
Georg Brandl60203b42010-10-06 10:11:56 +0000169deallocated, they are typically static :c:type:`PyTypeObject` objects.
Georg Brandl116aa622007-08-15 14:28:22 +0000170
171All Python objects (even Python integers) have a :dfn:`type` and a
172:dfn:`reference count`. An object's type determines what kind of object it is
173(e.g., an integer, a list, or a user-defined function; there are many more as
174explained in :ref:`types`). For each of the well-known types there is a macro
175to check whether an object is of that type; for instance, ``PyList_Check(a)`` is
176true if (and only if) the object pointed to by *a* is a Python list.
177
178
179.. _api-refcounts:
180
181Reference Counts
182----------------
183
184The reference count is important because today's computers have a finite (and
185often severely limited) memory size; it counts how many different places there
186are that have a reference to an object. Such a place could be another object,
187or a global (or static) C variable, or a local variable in some C function.
188When an object's reference count becomes zero, the object is deallocated. If
189it contains references to other objects, their reference count is decremented.
190Those other objects may be deallocated in turn, if this decrement makes their
191reference count become zero, and so on. (There's an obvious problem with
192objects that reference each other here; for now, the solution is "don't do
193that.")
194
195.. index::
196 single: Py_INCREF()
197 single: Py_DECREF()
198
199Reference counts are always manipulated explicitly. The normal way is to use
Georg Brandl60203b42010-10-06 10:11:56 +0000200the macro :c:func:`Py_INCREF` to increment an object's reference count by one,
201and :c:func:`Py_DECREF` to decrement it by one. The :c:func:`Py_DECREF` macro
Georg Brandl116aa622007-08-15 14:28:22 +0000202is considerably more complex than the incref one, since it must check whether
203the reference count becomes zero and then cause the object's deallocator to be
204called. The deallocator is a function pointer contained in the object's type
205structure. The type-specific deallocator takes care of decrementing the
206reference counts for other objects contained in the object if this is a compound
207object type, such as a list, as well as performing any additional finalization
208that's needed. There's no chance that the reference count can overflow; at
209least as many bits are used to hold the reference count as there are distinct
Christian Heimesdd15f6c2008-03-16 00:07:10 +0000210memory locations in virtual memory (assuming ``sizeof(Py_ssize_t) >= sizeof(void*)``).
Georg Brandl116aa622007-08-15 14:28:22 +0000211Thus, the reference count increment is a simple operation.
212
213It is not necessary to increment an object's reference count for every local
214variable that contains a pointer to an object. In theory, the object's
215reference count goes up by one when the variable is made to point to it and it
216goes down by one when the variable goes out of scope. However, these two
217cancel each other out, so at the end the reference count hasn't changed. The
218only real reason to use the reference count is to prevent the object from being
219deallocated as long as our variable is pointing to it. If we know that there
220is at least one other reference to the object that lives at least as long as
221our variable, there is no need to increment the reference count temporarily.
222An important situation where this arises is in objects that are passed as
223arguments to C functions in an extension module that are called from Python;
224the call mechanism guarantees to hold a reference to every argument for the
225duration of the call.
226
227However, a common pitfall is to extract an object from a list and hold on to it
228for a while without incrementing its reference count. Some other operation might
229conceivably remove the object from the list, decrementing its reference count
230and possible deallocating it. The real danger is that innocent-looking
231operations may invoke arbitrary Python code which could do this; there is a code
Georg Brandl60203b42010-10-06 10:11:56 +0000232path which allows control to flow back to the user from a :c:func:`Py_DECREF`, so
Georg Brandl116aa622007-08-15 14:28:22 +0000233almost any operation is potentially dangerous.
234
235A safe approach is to always use the generic operations (functions whose name
236begins with ``PyObject_``, ``PyNumber_``, ``PySequence_`` or ``PyMapping_``).
237These operations always increment the reference count of the object they return.
Georg Brandl60203b42010-10-06 10:11:56 +0000238This leaves the caller with the responsibility to call :c:func:`Py_DECREF` when
Georg Brandl116aa622007-08-15 14:28:22 +0000239they are done with the result; this soon becomes second nature.
240
241
242.. _api-refcountdetails:
243
244Reference Count Details
245^^^^^^^^^^^^^^^^^^^^^^^
246
247The reference count behavior of functions in the Python/C API is best explained
248in terms of *ownership of references*. Ownership pertains to references, never
249to objects (objects are not owned: they are always shared). "Owning a
250reference" means being responsible for calling Py_DECREF on it when the
251reference is no longer needed. Ownership can also be transferred, meaning that
252the code that receives ownership of the reference then becomes responsible for
Georg Brandl60203b42010-10-06 10:11:56 +0000253eventually decref'ing it by calling :c:func:`Py_DECREF` or :c:func:`Py_XDECREF`
Georg Brandl116aa622007-08-15 14:28:22 +0000254when it's no longer needed---or passing on this responsibility (usually to its
255caller). When a function passes ownership of a reference on to its caller, the
256caller is said to receive a *new* reference. When no ownership is transferred,
257the caller is said to *borrow* the reference. Nothing needs to be done for a
258borrowed reference.
259
Benjamin Petersonad3d5c22009-02-26 03:38:59 +0000260Conversely, when a calling function passes in a reference to an object, there
Georg Brandl116aa622007-08-15 14:28:22 +0000261are two possibilities: the function *steals* a reference to the object, or it
262does not. *Stealing a reference* means that when you pass a reference to a
263function, that function assumes that it now owns that reference, and you are not
264responsible for it any longer.
265
266.. index::
267 single: PyList_SetItem()
268 single: PyTuple_SetItem()
269
270Few functions steal references; the two notable exceptions are
Georg Brandl60203b42010-10-06 10:11:56 +0000271:c:func:`PyList_SetItem` and :c:func:`PyTuple_SetItem`, which steal a reference
Georg Brandl116aa622007-08-15 14:28:22 +0000272to the item (but not to the tuple or list into which the item is put!). These
273functions were designed to steal a reference because of a common idiom for
274populating a tuple or list with newly created objects; for example, the code to
275create the tuple ``(1, 2, "three")`` could look like this (forgetting about
276error handling for the moment; a better way to code this is shown below)::
277
278 PyObject *t;
279
280 t = PyTuple_New(3);
Georg Brandld019fe22007-12-08 18:58:51 +0000281 PyTuple_SetItem(t, 0, PyLong_FromLong(1L));
282 PyTuple_SetItem(t, 1, PyLong_FromLong(2L));
Gregory P. Smith4b52ae82013-03-22 13:43:30 -0700283 PyTuple_SetItem(t, 2, PyUnicode_FromString("three"));
Georg Brandl116aa622007-08-15 14:28:22 +0000284
Georg Brandl60203b42010-10-06 10:11:56 +0000285Here, :c:func:`PyLong_FromLong` returns a new reference which is immediately
286stolen by :c:func:`PyTuple_SetItem`. When you want to keep using an object
287although the reference to it will be stolen, use :c:func:`Py_INCREF` to grab
Georg Brandl116aa622007-08-15 14:28:22 +0000288another reference before calling the reference-stealing function.
289
Georg Brandl60203b42010-10-06 10:11:56 +0000290Incidentally, :c:func:`PyTuple_SetItem` is the *only* way to set tuple items;
291:c:func:`PySequence_SetItem` and :c:func:`PyObject_SetItem` refuse to do this
Georg Brandl116aa622007-08-15 14:28:22 +0000292since tuples are an immutable data type. You should only use
Georg Brandl60203b42010-10-06 10:11:56 +0000293:c:func:`PyTuple_SetItem` for tuples that you are creating yourself.
Georg Brandl116aa622007-08-15 14:28:22 +0000294
Georg Brandl60203b42010-10-06 10:11:56 +0000295Equivalent code for populating a list can be written using :c:func:`PyList_New`
296and :c:func:`PyList_SetItem`.
Georg Brandl116aa622007-08-15 14:28:22 +0000297
298However, in practice, you will rarely use these ways of creating and populating
Georg Brandl60203b42010-10-06 10:11:56 +0000299a tuple or list. There's a generic function, :c:func:`Py_BuildValue`, that can
Georg Brandl116aa622007-08-15 14:28:22 +0000300create most common objects from C values, directed by a :dfn:`format string`.
301For example, the above two blocks of code could be replaced by the following
302(which also takes care of the error checking)::
303
304 PyObject *tuple, *list;
305
306 tuple = Py_BuildValue("(iis)", 1, 2, "three");
307 list = Py_BuildValue("[iis]", 1, 2, "three");
308
Georg Brandl60203b42010-10-06 10:11:56 +0000309It is much more common to use :c:func:`PyObject_SetItem` and friends with items
Georg Brandl116aa622007-08-15 14:28:22 +0000310whose references you are only borrowing, like arguments that were passed in to
311the function you are writing. In that case, their behaviour regarding reference
312counts is much saner, since you don't have to increment a reference count so you
313can give a reference away ("have it be stolen"). For example, this function
314sets all items of a list (actually, any mutable sequence) to a given item::
315
316 int
317 set_all(PyObject *target, PyObject *item)
318 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100319 Py_ssize_t i, n;
Georg Brandl116aa622007-08-15 14:28:22 +0000320
321 n = PyObject_Length(target);
322 if (n < 0)
323 return -1;
324 for (i = 0; i < n; i++) {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100325 PyObject *index = PyLong_FromSsize_t(i);
Georg Brandl116aa622007-08-15 14:28:22 +0000326 if (!index)
327 return -1;
Antoine Pitrou04707c02012-01-27 14:07:29 +0100328 if (PyObject_SetItem(target, index, item) < 0) {
329 Py_DECREF(index);
Georg Brandl116aa622007-08-15 14:28:22 +0000330 return -1;
Antoine Pitrou04707c02012-01-27 14:07:29 +0100331 }
Georg Brandl116aa622007-08-15 14:28:22 +0000332 Py_DECREF(index);
333 }
334 return 0;
335 }
336
337.. index:: single: set_all()
338
339The situation is slightly different for function return values. While passing
340a reference to most functions does not change your ownership responsibilities
341for that reference, many functions that return a reference to an object give
342you ownership of the reference. The reason is simple: in many cases, the
343returned object is created on the fly, and the reference you get is the only
344reference to the object. Therefore, the generic functions that return object
Georg Brandl60203b42010-10-06 10:11:56 +0000345references, like :c:func:`PyObject_GetItem` and :c:func:`PySequence_GetItem`,
Georg Brandl116aa622007-08-15 14:28:22 +0000346always return a new reference (the caller becomes the owner of the reference).
347
348It is important to realize that whether you own a reference returned by a
349function depends on which function you call only --- *the plumage* (the type of
350the object passed as an argument to the function) *doesn't enter into it!*
Georg Brandl60203b42010-10-06 10:11:56 +0000351Thus, if you extract an item from a list using :c:func:`PyList_GetItem`, you
Georg Brandl116aa622007-08-15 14:28:22 +0000352don't own the reference --- but if you obtain the same item from the same list
Georg Brandl60203b42010-10-06 10:11:56 +0000353using :c:func:`PySequence_GetItem` (which happens to take exactly the same
Georg Brandl116aa622007-08-15 14:28:22 +0000354arguments), you do own a reference to the returned object.
355
356.. index::
357 single: PyList_GetItem()
358 single: PySequence_GetItem()
359
360Here is an example of how you could write a function that computes the sum of
Georg Brandl60203b42010-10-06 10:11:56 +0000361the items in a list of integers; once using :c:func:`PyList_GetItem`, and once
362using :c:func:`PySequence_GetItem`. ::
Georg Brandl116aa622007-08-15 14:28:22 +0000363
364 long
365 sum_list(PyObject *list)
366 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100367 Py_ssize_t i, n;
368 long total = 0, value;
Georg Brandl116aa622007-08-15 14:28:22 +0000369 PyObject *item;
370
371 n = PyList_Size(list);
372 if (n < 0)
373 return -1; /* Not a list */
374 for (i = 0; i < n; i++) {
375 item = PyList_GetItem(list, i); /* Can't fail */
Georg Brandld019fe22007-12-08 18:58:51 +0000376 if (!PyLong_Check(item)) continue; /* Skip non-integers */
Antoine Pitrou04707c02012-01-27 14:07:29 +0100377 value = PyLong_AsLong(item);
378 if (value == -1 && PyErr_Occurred())
379 /* Integer too big to fit in a C long, bail out */
380 return -1;
381 total += value;
Georg Brandl116aa622007-08-15 14:28:22 +0000382 }
383 return total;
384 }
385
386.. index:: single: sum_list()
387
388::
389
390 long
391 sum_sequence(PyObject *sequence)
392 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100393 Py_ssize_t i, n;
394 long total = 0, value;
Georg Brandl116aa622007-08-15 14:28:22 +0000395 PyObject *item;
396 n = PySequence_Length(sequence);
397 if (n < 0)
398 return -1; /* Has no length */
399 for (i = 0; i < n; i++) {
400 item = PySequence_GetItem(sequence, i);
401 if (item == NULL)
402 return -1; /* Not a sequence, or other failure */
Antoine Pitrou04707c02012-01-27 14:07:29 +0100403 if (PyLong_Check(item)) {
404 value = PyLong_AsLong(item);
405 Py_DECREF(item);
406 if (value == -1 && PyErr_Occurred())
407 /* Integer too big to fit in a C long, bail out */
408 return -1;
409 total += value;
410 }
411 else {
412 Py_DECREF(item); /* Discard reference ownership */
413 }
Georg Brandl116aa622007-08-15 14:28:22 +0000414 }
415 return total;
416 }
417
418.. index:: single: sum_sequence()
419
420
421.. _api-types:
422
423Types
424-----
425
426There are few other data types that play a significant role in the Python/C
Georg Brandl60203b42010-10-06 10:11:56 +0000427API; most are simple C types such as :c:type:`int`, :c:type:`long`,
428:c:type:`double` and :c:type:`char\*`. A few structure types are used to
Georg Brandl116aa622007-08-15 14:28:22 +0000429describe static tables used to list the functions exported by a module or the
430data attributes of a new object type, and another is used to describe the value
431of a complex number. These will be discussed together with the functions that
432use them.
433
434
435.. _api-exceptions:
436
437Exceptions
438==========
439
440The Python programmer only needs to deal with exceptions if specific error
441handling is required; unhandled exceptions are automatically propagated to the
442caller, then to the caller's caller, and so on, until they reach the top-level
443interpreter, where they are reported to the user accompanied by a stack
444traceback.
445
446.. index:: single: PyErr_Occurred()
447
Georg Brandldd909db2010-10-17 06:32:59 +0000448For C programmers, however, error checking always has to be explicit. All
449functions in the Python/C API can raise exceptions, unless an explicit claim is
450made otherwise in a function's documentation. In general, when a function
451encounters an error, it sets an exception, discards any object references that
452it owns, and returns an error indicator. If not documented otherwise, this
453indicator is either *NULL* or ``-1``, depending on the function's return type.
454A few functions return a Boolean true/false result, with false indicating an
455error. Very few functions return no explicit error indicator or have an
456ambiguous return value, and require explicit testing for errors with
457:c:func:`PyErr_Occurred`. These exceptions are always explicitly documented.
Georg Brandl116aa622007-08-15 14:28:22 +0000458
459.. index::
460 single: PyErr_SetString()
461 single: PyErr_Clear()
462
463Exception state is maintained in per-thread storage (this is equivalent to
464using global storage in an unthreaded application). A thread can be in one of
465two states: an exception has occurred, or not. The function
Georg Brandl60203b42010-10-06 10:11:56 +0000466:c:func:`PyErr_Occurred` can be used to check for this: it returns a borrowed
Georg Brandl116aa622007-08-15 14:28:22 +0000467reference to the exception type object when an exception has occurred, and
468*NULL* otherwise. There are a number of functions to set the exception state:
Georg Brandl60203b42010-10-06 10:11:56 +0000469:c:func:`PyErr_SetString` is the most common (though not the most general)
470function to set the exception state, and :c:func:`PyErr_Clear` clears the
Georg Brandl116aa622007-08-15 14:28:22 +0000471exception state.
472
473The full exception state consists of three objects (all of which can be
474*NULL*): the exception type, the corresponding exception value, and the
475traceback. These have the same meanings as the Python result of
476``sys.exc_info()``; however, they are not the same: the Python objects represent
477the last exception being handled by a Python :keyword:`try` ...
478:keyword:`except` statement, while the C level exception state only exists while
479an exception is being passed on between C functions until it reaches the Python
480bytecode interpreter's main loop, which takes care of transferring it to
481``sys.exc_info()`` and friends.
482
483.. index:: single: exc_info() (in module sys)
484
485Note that starting with Python 1.5, the preferred, thread-safe way to access the
486exception state from Python code is to call the function :func:`sys.exc_info`,
487which returns the per-thread exception state for Python code. Also, the
488semantics of both ways to access the exception state have changed so that a
489function which catches an exception will save and restore its thread's exception
490state so as to preserve the exception state of its caller. This prevents common
491bugs in exception handling code caused by an innocent-looking function
492overwriting the exception being handled; it also reduces the often unwanted
493lifetime extension for objects that are referenced by the stack frames in the
494traceback.
495
496As a general principle, a function that calls another function to perform some
497task should check whether the called function raised an exception, and if so,
498pass the exception state on to its caller. It should discard any object
499references that it owns, and return an error indicator, but it should *not* set
500another exception --- that would overwrite the exception that was just raised,
501and lose important information about the exact cause of the error.
502
503.. index:: single: sum_sequence()
504
505A simple example of detecting exceptions and passing them on is shown in the
Terry Jan Reedy65e69b32013-03-11 17:23:46 -0400506:c:func:`sum_sequence` example above. It so happens that this example doesn't
Georg Brandl116aa622007-08-15 14:28:22 +0000507need to clean up any owned references when it detects an error. The following
508example function shows some error cleanup. First, to remind you why you like
509Python, we show the equivalent Python code::
510
511 def incr_item(dict, key):
512 try:
513 item = dict[key]
514 except KeyError:
515 item = 0
516 dict[key] = item + 1
517
518.. index:: single: incr_item()
519
520Here is the corresponding C code, in all its glory::
521
522 int
523 incr_item(PyObject *dict, PyObject *key)
524 {
525 /* Objects all initialized to NULL for Py_XDECREF */
526 PyObject *item = NULL, *const_one = NULL, *incremented_item = NULL;
527 int rv = -1; /* Return value initialized to -1 (failure) */
528
529 item = PyObject_GetItem(dict, key);
530 if (item == NULL) {
531 /* Handle KeyError only: */
532 if (!PyErr_ExceptionMatches(PyExc_KeyError))
533 goto error;
534
535 /* Clear the error and use zero: */
536 PyErr_Clear();
Georg Brandld019fe22007-12-08 18:58:51 +0000537 item = PyLong_FromLong(0L);
Georg Brandl116aa622007-08-15 14:28:22 +0000538 if (item == NULL)
539 goto error;
540 }
Georg Brandld019fe22007-12-08 18:58:51 +0000541 const_one = PyLong_FromLong(1L);
Georg Brandl116aa622007-08-15 14:28:22 +0000542 if (const_one == NULL)
543 goto error;
544
545 incremented_item = PyNumber_Add(item, const_one);
546 if (incremented_item == NULL)
547 goto error;
548
549 if (PyObject_SetItem(dict, key, incremented_item) < 0)
550 goto error;
551 rv = 0; /* Success */
552 /* Continue with cleanup code */
553
554 error:
555 /* Cleanup code, shared by success and failure path */
556
557 /* Use Py_XDECREF() to ignore NULL references */
558 Py_XDECREF(item);
559 Py_XDECREF(const_one);
560 Py_XDECREF(incremented_item);
561
562 return rv; /* -1 for error, 0 for success */
563 }
564
565.. index:: single: incr_item()
566
567.. index::
568 single: PyErr_ExceptionMatches()
569 single: PyErr_Clear()
570 single: Py_XDECREF()
571
Christian Heimes5b5e81c2007-12-31 16:14:33 +0000572This example represents an endorsed use of the ``goto`` statement in C!
Georg Brandl60203b42010-10-06 10:11:56 +0000573It illustrates the use of :c:func:`PyErr_ExceptionMatches` and
574:c:func:`PyErr_Clear` to handle specific exceptions, and the use of
575:c:func:`Py_XDECREF` to dispose of owned references that may be *NULL* (note the
576``'X'`` in the name; :c:func:`Py_DECREF` would crash when confronted with a
Georg Brandl116aa622007-08-15 14:28:22 +0000577*NULL* reference). It is important that the variables used to hold owned
578references are initialized to *NULL* for this to work; likewise, the proposed
579return value is initialized to ``-1`` (failure) and only set to success after
580the final call made is successful.
581
582
583.. _api-embedding:
584
585Embedding Python
586================
587
588The one important task that only embedders (as opposed to extension writers) of
589the Python interpreter have to worry about is the initialization, and possibly
590the finalization, of the Python interpreter. Most functionality of the
591interpreter can only be used after the interpreter has been initialized.
592
593.. index::
594 single: Py_Initialize()
Georg Brandl1a3284e2007-12-02 09:40:06 +0000595 module: builtins
Georg Brandl116aa622007-08-15 14:28:22 +0000596 module: __main__
597 module: sys
Georg Brandl116aa622007-08-15 14:28:22 +0000598 triple: module; search; path
599 single: path (in module sys)
600
Georg Brandl60203b42010-10-06 10:11:56 +0000601The basic initialization function is :c:func:`Py_Initialize`. This initializes
Georg Brandl116aa622007-08-15 14:28:22 +0000602the table of loaded modules, and creates the fundamental modules
Éric Araujo8b8f2ec2011-03-26 07:22:01 +0100603:mod:`builtins`, :mod:`__main__`, and :mod:`sys`. It also
Georg Brandl116aa622007-08-15 14:28:22 +0000604initializes the module search path (``sys.path``).
605
Benjamin Peterson2ebf8ce2010-06-27 21:48:35 +0000606.. index:: single: PySys_SetArgvEx()
Georg Brandl116aa622007-08-15 14:28:22 +0000607
Georg Brandl60203b42010-10-06 10:11:56 +0000608:c:func:`Py_Initialize` does not set the "script argument list" (``sys.argv``).
Benjamin Peterson2ebf8ce2010-06-27 21:48:35 +0000609If this variable is needed by Python code that will be executed later, it must
610be set explicitly with a call to ``PySys_SetArgvEx(argc, argv, updatepath)``
Georg Brandl60203b42010-10-06 10:11:56 +0000611after the call to :c:func:`Py_Initialize`.
Georg Brandl116aa622007-08-15 14:28:22 +0000612
613On most systems (in particular, on Unix and Windows, although the details are
Georg Brandl60203b42010-10-06 10:11:56 +0000614slightly different), :c:func:`Py_Initialize` calculates the module search path
Georg Brandl116aa622007-08-15 14:28:22 +0000615based upon its best guess for the location of the standard Python interpreter
616executable, assuming that the Python library is found in a fixed location
617relative to the Python interpreter executable. In particular, it looks for a
618directory named :file:`lib/python{X.Y}` relative to the parent directory
619where the executable named :file:`python` is found on the shell command search
620path (the environment variable :envvar:`PATH`).
621
622For instance, if the Python executable is found in
623:file:`/usr/local/bin/python`, it will assume that the libraries are in
624:file:`/usr/local/lib/python{X.Y}`. (In fact, this particular path is also
625the "fallback" location, used when no executable file named :file:`python` is
626found along :envvar:`PATH`.) The user can override this behavior by setting the
627environment variable :envvar:`PYTHONHOME`, or insert additional directories in
628front of the standard path by setting :envvar:`PYTHONPATH`.
629
630.. index::
631 single: Py_SetProgramName()
632 single: Py_GetPath()
633 single: Py_GetPrefix()
634 single: Py_GetExecPrefix()
635 single: Py_GetProgramFullPath()
636
637The embedding application can steer the search by calling
Georg Brandl60203b42010-10-06 10:11:56 +0000638``Py_SetProgramName(file)`` *before* calling :c:func:`Py_Initialize`. Note that
Georg Brandl116aa622007-08-15 14:28:22 +0000639:envvar:`PYTHONHOME` still overrides this and :envvar:`PYTHONPATH` is still
640inserted in front of the standard path. An application that requires total
Georg Brandl60203b42010-10-06 10:11:56 +0000641control has to provide its own implementation of :c:func:`Py_GetPath`,
642:c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, and
643:c:func:`Py_GetProgramFullPath` (all defined in :file:`Modules/getpath.c`).
Georg Brandl116aa622007-08-15 14:28:22 +0000644
645.. index:: single: Py_IsInitialized()
646
647Sometimes, it is desirable to "uninitialize" Python. For instance, the
648application may want to start over (make another call to
Georg Brandl60203b42010-10-06 10:11:56 +0000649:c:func:`Py_Initialize`) or the application is simply done with its use of
Georg Brandl116aa622007-08-15 14:28:22 +0000650Python and wants to free memory allocated by Python. This can be accomplished
Martin Panterb4ce1fc2015-11-30 03:18:29 +0000651by calling :c:func:`Py_FinalizeEx`. The function :c:func:`Py_IsInitialized` returns
Georg Brandl116aa622007-08-15 14:28:22 +0000652true if Python is currently in the initialized state. More information about
Martin Panterb4ce1fc2015-11-30 03:18:29 +0000653these functions is given in a later chapter. Notice that :c:func:`Py_FinalizeEx`
Georg Brandl116aa622007-08-15 14:28:22 +0000654does *not* free all memory allocated by the Python interpreter, e.g. memory
655allocated by extension modules currently cannot be released.
656
657
658.. _api-debugging:
659
660Debugging Builds
661================
662
663Python can be built with several macros to enable extra checks of the
664interpreter and extension modules. These checks tend to add a large amount of
665overhead to the runtime so they are not enabled by default.
666
667A full list of the various types of debugging builds is in the file
668:file:`Misc/SpecialBuilds.txt` in the Python source distribution. Builds are
669available that support tracing of reference counts, debugging the memory
670allocator, or low-level profiling of the main interpreter loop. Only the most
671frequently-used builds will be described in the remainder of this section.
672
Georg Brandl60203b42010-10-06 10:11:56 +0000673Compiling the interpreter with the :c:macro:`Py_DEBUG` macro defined produces
674what is generally meant by "a debug build" of Python. :c:macro:`Py_DEBUG` is
Éric Araujod2f8cec2011-06-08 05:29:39 +0200675enabled in the Unix build by adding ``--with-pydebug`` to the
676:file:`./configure` command. It is also implied by the presence of the
Georg Brandl60203b42010-10-06 10:11:56 +0000677not-Python-specific :c:macro:`_DEBUG` macro. When :c:macro:`Py_DEBUG` is enabled
Georg Brandl116aa622007-08-15 14:28:22 +0000678in the Unix build, compiler optimization is disabled.
679
680In addition to the reference count debugging described below, the following
681extra checks are performed:
682
683* Extra checks are added to the object allocator.
684
685* Extra checks are added to the parser and compiler.
686
687* Downcasts from wide types to narrow types are checked for loss of information.
688
689* A number of assertions are added to the dictionary and set implementations.
690 In addition, the set object acquires a :meth:`test_c_api` method.
691
692* Sanity checks of the input arguments are added to frame creation.
693
Mark Dickinsonbf5c6a92009-01-17 10:21:23 +0000694* The storage for ints is initialized with a known invalid pattern to catch
Georg Brandl116aa622007-08-15 14:28:22 +0000695 reference to uninitialized digits.
696
697* Low-level tracing and extra exception checking are added to the runtime
698 virtual machine.
699
700* Extra checks are added to the memory arena implementation.
701
702* Extra debugging is added to the thread module.
703
704There may be additional checks not mentioned here.
705
Georg Brandl60203b42010-10-06 10:11:56 +0000706Defining :c:macro:`Py_TRACE_REFS` enables reference tracing. When defined, a
Georg Brandl116aa622007-08-15 14:28:22 +0000707circular doubly linked list of active objects is maintained by adding two extra
Georg Brandl60203b42010-10-06 10:11:56 +0000708fields to every :c:type:`PyObject`. Total allocations are tracked as well. Upon
Georg Brandl116aa622007-08-15 14:28:22 +0000709exit, all existing references are printed. (In interactive mode this happens
Georg Brandl60203b42010-10-06 10:11:56 +0000710after every statement run by the interpreter.) Implied by :c:macro:`Py_DEBUG`.
Georg Brandl116aa622007-08-15 14:28:22 +0000711
712Please refer to :file:`Misc/SpecialBuilds.txt` in the Python source distribution
713for more detailed information.
714