blob: e654c4c012651ebefe52b3c0c568061de4a15eca [file] [log] [blame]
Georg Brandl116aa622007-08-15 14:28:22 +00001.. highlightlang:: c
2
3
4.. _api-intro:
5
6************
7Introduction
8************
9
10The Application Programmer's Interface to Python gives C and C++ programmers
11access to the Python interpreter at a variety of levels. The API is equally
12usable from C++, but for brevity it is generally referred to as the Python/C
13API. There are two fundamentally different reasons for using the Python/C API.
14The first reason is to write *extension modules* for specific purposes; these
15are C modules that extend the Python interpreter. This is probably the most
16common use. The second reason is to use Python as a component in a larger
17application; this technique is generally referred to as :dfn:`embedding` Python
18in an application.
19
Barry Warsawb2e57942017-09-14 18:13:16 -070020Writing an extension module is a relatively well-understood process, where a
21"cookbook" approach works well. There are several tools that automate the
22process to some extent. While people have embedded Python in other
23applications since its early existence, the process of embedding Python is
24less straightforward than writing an extension.
Georg Brandl116aa622007-08-15 14:28:22 +000025
26Many API functions are useful independent of whether you're embedding or
27extending Python; moreover, most applications that embed Python will need to
28provide a custom extension as well, so it's probably a good idea to become
29familiar with writing an extension before attempting to embed Python in a real
30application.
31
32
Barry Warsawb2e57942017-09-14 18:13:16 -070033Coding standards
34================
35
36If you're writing C code for inclusion in CPython, you **must** follow the
37guidelines and standards defined in :PEP:`7`. These guidelines apply
38regardless of the version of Python you are contributing to. Following these
39conventions is not necessary for your own third party extension modules,
40unless you eventually expect to contribute them to Python.
41
42
Georg Brandl116aa622007-08-15 14:28:22 +000043.. _api-includes:
44
45Include Files
46=============
47
48All function, type and macro definitions needed to use the Python/C API are
49included in your code by the following line::
50
51 #include "Python.h"
52
53This implies inclusion of the following standard headers: ``<stdio.h>``,
Georg Brandl4f13d612010-11-23 18:14:57 +000054``<string.h>``, ``<errno.h>``, ``<limits.h>``, ``<assert.h>`` and ``<stdlib.h>``
55(if available).
Georg Brandl116aa622007-08-15 14:28:22 +000056
Georg Brandle720c0a2009-04-27 16:20:50 +000057.. note::
Georg Brandl116aa622007-08-15 14:28:22 +000058
59 Since Python may define some pre-processor definitions which affect the standard
60 headers on some systems, you *must* include :file:`Python.h` before any standard
61 headers are included.
62
63All user visible names defined by Python.h (except those defined by the included
64standard headers) have one of the prefixes ``Py`` or ``_Py``. Names beginning
65with ``_Py`` are for internal use by the Python implementation and should not be
66used by extension writers. Structure member names do not have a reserved prefix.
67
68**Important:** user code should never define names that begin with ``Py`` or
69``_Py``. This confuses the reader, and jeopardizes the portability of the user
70code to future Python versions, which may define additional names beginning with
71one of these prefixes.
72
73The header files are typically installed with Python. On Unix, these are
74located in the directories :file:`{prefix}/include/pythonversion/` and
75:file:`{exec_prefix}/include/pythonversion/`, where :envvar:`prefix` and
76:envvar:`exec_prefix` are defined by the corresponding parameters to Python's
Serhiy Storchaka885bdc42016-02-11 13:10:36 +020077:program:`configure` script and *version* is
78``'%d.%d' % sys.version_info[:2]``. On Windows, the headers are installed
79in :file:`{prefix}/include`, where :envvar:`prefix` is the installation
80directory specified to the installer.
Georg Brandl116aa622007-08-15 14:28:22 +000081
82To include the headers, place both directories (if different) on your compiler's
83search path for includes. Do *not* place the parent directories on the search
84path and then use ``#include <pythonX.Y/Python.h>``; this will break on
85multi-platform builds since the platform independent headers under
86:envvar:`prefix` include the platform specific headers from
87:envvar:`exec_prefix`.
88
89C++ users should note that though the API is defined entirely using C, the
90header files do properly declare the entry points to be ``extern "C"``, so there
91is no need to do anything special to use the API from C++.
92
93
Barry Warsawb2e57942017-09-14 18:13:16 -070094Useful macros
95=============
96
97Several useful macros are defined in the Python header files. Many are
98defined closer to where they are useful (e.g. :c:macro:`Py_RETURN_NONE`).
99Others of a more general utility are defined here. This is not necessarily a
100complete listing.
101
102.. c:macro:: Py_UNREACHABLE()
103
104 Use this when you have a code path that you do not expect to be reached.
105 For example, in the ``default:`` clause in a ``switch`` statement for which
106 all possible values are covered in ``case`` statements. Use this in places
107 where you might be tempted to put an ``assert(0)`` or ``abort()`` call.
108
109.. c:macro:: Py_ABS(x)
110
111 Return the absolute value of ``x``.
112
113.. c:macro:: Py_MIN(x, y)
114
115 Return the minimum value between ``x`` and ``y``.
116
117.. c:macro:: Py_MAX(x, y)
118
119 Return the maximum value between ``x`` and ``y``.
120
121.. c:macro:: Py_STRINGIFY(x)
122
123 Convert ``x`` to a C string. E.g. ``Py_STRINGIFY(123)`` returns
124 ``"123"``.
125
126.. c:macro:: Py_MEMBER_SIZE(type, member)
127
128 Return the size of a structure (``type``) ``member`` in bytes.
129
130.. c:macro:: Py_CHARMASK(c)
131
132 Argument must be a character or an integer in the range [-128, 127] or [0,
133 255]. This macro returns ``c`` cast to an ``unsigned char``.
134
Barry Warsawa51b90a2017-10-06 09:53:48 -0400135.. c:macro:: Py_GETENV(s)
136
137 Like ``getenv(s)``, but returns *NULL* if :option:`-E` was passed on the
138 command line (i.e. if ``Py_IgnoreEnvironmentFlag`` is set).
139
Barry Warsawb2e57942017-09-14 18:13:16 -0700140
Georg Brandl116aa622007-08-15 14:28:22 +0000141.. _api-objects:
142
143Objects, Types and Reference Counts
144===================================
145
146.. index:: object: type
147
148Most Python/C API functions have one or more arguments as well as a return value
Georg Brandl60203b42010-10-06 10:11:56 +0000149of type :c:type:`PyObject\*`. This type is a pointer to an opaque data type
Georg Brandl116aa622007-08-15 14:28:22 +0000150representing an arbitrary Python object. Since all Python object types are
151treated the same way by the Python language in most situations (e.g.,
152assignments, scope rules, and argument passing), it is only fitting that they
153should be represented by a single C type. Almost all Python objects live on the
154heap: you never declare an automatic or static variable of type
Georg Brandl60203b42010-10-06 10:11:56 +0000155:c:type:`PyObject`, only pointer variables of type :c:type:`PyObject\*` can be
Georg Brandl116aa622007-08-15 14:28:22 +0000156declared. The sole exception are the type objects; since these must never be
Georg Brandl60203b42010-10-06 10:11:56 +0000157deallocated, they are typically static :c:type:`PyTypeObject` objects.
Georg Brandl116aa622007-08-15 14:28:22 +0000158
159All Python objects (even Python integers) have a :dfn:`type` and a
160:dfn:`reference count`. An object's type determines what kind of object it is
161(e.g., an integer, a list, or a user-defined function; there are many more as
162explained in :ref:`types`). For each of the well-known types there is a macro
163to check whether an object is of that type; for instance, ``PyList_Check(a)`` is
164true if (and only if) the object pointed to by *a* is a Python list.
165
166
167.. _api-refcounts:
168
169Reference Counts
170----------------
171
172The reference count is important because today's computers have a finite (and
173often severely limited) memory size; it counts how many different places there
174are that have a reference to an object. Such a place could be another object,
175or a global (or static) C variable, or a local variable in some C function.
176When an object's reference count becomes zero, the object is deallocated. If
177it contains references to other objects, their reference count is decremented.
178Those other objects may be deallocated in turn, if this decrement makes their
179reference count become zero, and so on. (There's an obvious problem with
180objects that reference each other here; for now, the solution is "don't do
181that.")
182
183.. index::
184 single: Py_INCREF()
185 single: Py_DECREF()
186
187Reference counts are always manipulated explicitly. The normal way is to use
Georg Brandl60203b42010-10-06 10:11:56 +0000188the macro :c:func:`Py_INCREF` to increment an object's reference count by one,
189and :c:func:`Py_DECREF` to decrement it by one. The :c:func:`Py_DECREF` macro
Georg Brandl116aa622007-08-15 14:28:22 +0000190is considerably more complex than the incref one, since it must check whether
191the reference count becomes zero and then cause the object's deallocator to be
192called. The deallocator is a function pointer contained in the object's type
193structure. The type-specific deallocator takes care of decrementing the
194reference counts for other objects contained in the object if this is a compound
195object type, such as a list, as well as performing any additional finalization
196that's needed. There's no chance that the reference count can overflow; at
197least as many bits are used to hold the reference count as there are distinct
Christian Heimesdd15f6c2008-03-16 00:07:10 +0000198memory locations in virtual memory (assuming ``sizeof(Py_ssize_t) >= sizeof(void*)``).
Georg Brandl116aa622007-08-15 14:28:22 +0000199Thus, the reference count increment is a simple operation.
200
201It is not necessary to increment an object's reference count for every local
202variable that contains a pointer to an object. In theory, the object's
203reference count goes up by one when the variable is made to point to it and it
204goes down by one when the variable goes out of scope. However, these two
205cancel each other out, so at the end the reference count hasn't changed. The
206only real reason to use the reference count is to prevent the object from being
207deallocated as long as our variable is pointing to it. If we know that there
208is at least one other reference to the object that lives at least as long as
209our variable, there is no need to increment the reference count temporarily.
210An important situation where this arises is in objects that are passed as
211arguments to C functions in an extension module that are called from Python;
212the call mechanism guarantees to hold a reference to every argument for the
213duration of the call.
214
215However, a common pitfall is to extract an object from a list and hold on to it
216for a while without incrementing its reference count. Some other operation might
217conceivably remove the object from the list, decrementing its reference count
218and possible deallocating it. The real danger is that innocent-looking
219operations may invoke arbitrary Python code which could do this; there is a code
Georg Brandl60203b42010-10-06 10:11:56 +0000220path which allows control to flow back to the user from a :c:func:`Py_DECREF`, so
Georg Brandl116aa622007-08-15 14:28:22 +0000221almost any operation is potentially dangerous.
222
223A safe approach is to always use the generic operations (functions whose name
224begins with ``PyObject_``, ``PyNumber_``, ``PySequence_`` or ``PyMapping_``).
225These operations always increment the reference count of the object they return.
Georg Brandl60203b42010-10-06 10:11:56 +0000226This leaves the caller with the responsibility to call :c:func:`Py_DECREF` when
Georg Brandl116aa622007-08-15 14:28:22 +0000227they are done with the result; this soon becomes second nature.
228
229
230.. _api-refcountdetails:
231
232Reference Count Details
233^^^^^^^^^^^^^^^^^^^^^^^
234
235The reference count behavior of functions in the Python/C API is best explained
236in terms of *ownership of references*. Ownership pertains to references, never
237to objects (objects are not owned: they are always shared). "Owning a
238reference" means being responsible for calling Py_DECREF on it when the
239reference is no longer needed. Ownership can also be transferred, meaning that
240the code that receives ownership of the reference then becomes responsible for
Georg Brandl60203b42010-10-06 10:11:56 +0000241eventually decref'ing it by calling :c:func:`Py_DECREF` or :c:func:`Py_XDECREF`
Georg Brandl116aa622007-08-15 14:28:22 +0000242when it's no longer needed---or passing on this responsibility (usually to its
243caller). When a function passes ownership of a reference on to its caller, the
244caller is said to receive a *new* reference. When no ownership is transferred,
245the caller is said to *borrow* the reference. Nothing needs to be done for a
246borrowed reference.
247
Benjamin Petersonad3d5c22009-02-26 03:38:59 +0000248Conversely, when a calling function passes in a reference to an object, there
Georg Brandl116aa622007-08-15 14:28:22 +0000249are two possibilities: the function *steals* a reference to the object, or it
250does not. *Stealing a reference* means that when you pass a reference to a
251function, that function assumes that it now owns that reference, and you are not
252responsible for it any longer.
253
254.. index::
255 single: PyList_SetItem()
256 single: PyTuple_SetItem()
257
258Few functions steal references; the two notable exceptions are
Georg Brandl60203b42010-10-06 10:11:56 +0000259:c:func:`PyList_SetItem` and :c:func:`PyTuple_SetItem`, which steal a reference
Georg Brandl116aa622007-08-15 14:28:22 +0000260to the item (but not to the tuple or list into which the item is put!). These
261functions were designed to steal a reference because of a common idiom for
262populating a tuple or list with newly created objects; for example, the code to
263create the tuple ``(1, 2, "three")`` could look like this (forgetting about
264error handling for the moment; a better way to code this is shown below)::
265
266 PyObject *t;
267
268 t = PyTuple_New(3);
Georg Brandld019fe22007-12-08 18:58:51 +0000269 PyTuple_SetItem(t, 0, PyLong_FromLong(1L));
270 PyTuple_SetItem(t, 1, PyLong_FromLong(2L));
Gregory P. Smith4b52ae82013-03-22 13:43:30 -0700271 PyTuple_SetItem(t, 2, PyUnicode_FromString("three"));
Georg Brandl116aa622007-08-15 14:28:22 +0000272
Georg Brandl60203b42010-10-06 10:11:56 +0000273Here, :c:func:`PyLong_FromLong` returns a new reference which is immediately
274stolen by :c:func:`PyTuple_SetItem`. When you want to keep using an object
275although the reference to it will be stolen, use :c:func:`Py_INCREF` to grab
Georg Brandl116aa622007-08-15 14:28:22 +0000276another reference before calling the reference-stealing function.
277
Georg Brandl60203b42010-10-06 10:11:56 +0000278Incidentally, :c:func:`PyTuple_SetItem` is the *only* way to set tuple items;
279:c:func:`PySequence_SetItem` and :c:func:`PyObject_SetItem` refuse to do this
Georg Brandl116aa622007-08-15 14:28:22 +0000280since tuples are an immutable data type. You should only use
Georg Brandl60203b42010-10-06 10:11:56 +0000281:c:func:`PyTuple_SetItem` for tuples that you are creating yourself.
Georg Brandl116aa622007-08-15 14:28:22 +0000282
Georg Brandl60203b42010-10-06 10:11:56 +0000283Equivalent code for populating a list can be written using :c:func:`PyList_New`
284and :c:func:`PyList_SetItem`.
Georg Brandl116aa622007-08-15 14:28:22 +0000285
286However, in practice, you will rarely use these ways of creating and populating
Georg Brandl60203b42010-10-06 10:11:56 +0000287a tuple or list. There's a generic function, :c:func:`Py_BuildValue`, that can
Georg Brandl116aa622007-08-15 14:28:22 +0000288create most common objects from C values, directed by a :dfn:`format string`.
289For example, the above two blocks of code could be replaced by the following
290(which also takes care of the error checking)::
291
292 PyObject *tuple, *list;
293
294 tuple = Py_BuildValue("(iis)", 1, 2, "three");
295 list = Py_BuildValue("[iis]", 1, 2, "three");
296
Georg Brandl60203b42010-10-06 10:11:56 +0000297It is much more common to use :c:func:`PyObject_SetItem` and friends with items
Georg Brandl116aa622007-08-15 14:28:22 +0000298whose references you are only borrowing, like arguments that were passed in to
299the function you are writing. In that case, their behaviour regarding reference
300counts is much saner, since you don't have to increment a reference count so you
301can give a reference away ("have it be stolen"). For example, this function
302sets all items of a list (actually, any mutable sequence) to a given item::
303
304 int
305 set_all(PyObject *target, PyObject *item)
306 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100307 Py_ssize_t i, n;
Georg Brandl116aa622007-08-15 14:28:22 +0000308
309 n = PyObject_Length(target);
310 if (n < 0)
311 return -1;
312 for (i = 0; i < n; i++) {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100313 PyObject *index = PyLong_FromSsize_t(i);
Georg Brandl116aa622007-08-15 14:28:22 +0000314 if (!index)
315 return -1;
Antoine Pitrou04707c02012-01-27 14:07:29 +0100316 if (PyObject_SetItem(target, index, item) < 0) {
317 Py_DECREF(index);
Georg Brandl116aa622007-08-15 14:28:22 +0000318 return -1;
Antoine Pitrou04707c02012-01-27 14:07:29 +0100319 }
Georg Brandl116aa622007-08-15 14:28:22 +0000320 Py_DECREF(index);
321 }
322 return 0;
323 }
324
325.. index:: single: set_all()
326
327The situation is slightly different for function return values. While passing
328a reference to most functions does not change your ownership responsibilities
329for that reference, many functions that return a reference to an object give
330you ownership of the reference. The reason is simple: in many cases, the
331returned object is created on the fly, and the reference you get is the only
332reference to the object. Therefore, the generic functions that return object
Georg Brandl60203b42010-10-06 10:11:56 +0000333references, like :c:func:`PyObject_GetItem` and :c:func:`PySequence_GetItem`,
Georg Brandl116aa622007-08-15 14:28:22 +0000334always return a new reference (the caller becomes the owner of the reference).
335
336It is important to realize that whether you own a reference returned by a
337function depends on which function you call only --- *the plumage* (the type of
338the object passed as an argument to the function) *doesn't enter into it!*
Georg Brandl60203b42010-10-06 10:11:56 +0000339Thus, if you extract an item from a list using :c:func:`PyList_GetItem`, you
Georg Brandl116aa622007-08-15 14:28:22 +0000340don't own the reference --- but if you obtain the same item from the same list
Georg Brandl60203b42010-10-06 10:11:56 +0000341using :c:func:`PySequence_GetItem` (which happens to take exactly the same
Georg Brandl116aa622007-08-15 14:28:22 +0000342arguments), you do own a reference to the returned object.
343
344.. index::
345 single: PyList_GetItem()
346 single: PySequence_GetItem()
347
348Here is an example of how you could write a function that computes the sum of
Georg Brandl60203b42010-10-06 10:11:56 +0000349the items in a list of integers; once using :c:func:`PyList_GetItem`, and once
350using :c:func:`PySequence_GetItem`. ::
Georg Brandl116aa622007-08-15 14:28:22 +0000351
352 long
353 sum_list(PyObject *list)
354 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100355 Py_ssize_t i, n;
356 long total = 0, value;
Georg Brandl116aa622007-08-15 14:28:22 +0000357 PyObject *item;
358
359 n = PyList_Size(list);
360 if (n < 0)
361 return -1; /* Not a list */
362 for (i = 0; i < n; i++) {
363 item = PyList_GetItem(list, i); /* Can't fail */
Georg Brandld019fe22007-12-08 18:58:51 +0000364 if (!PyLong_Check(item)) continue; /* Skip non-integers */
Antoine Pitrou04707c02012-01-27 14:07:29 +0100365 value = PyLong_AsLong(item);
366 if (value == -1 && PyErr_Occurred())
367 /* Integer too big to fit in a C long, bail out */
368 return -1;
369 total += value;
Georg Brandl116aa622007-08-15 14:28:22 +0000370 }
371 return total;
372 }
373
374.. index:: single: sum_list()
375
376::
377
378 long
379 sum_sequence(PyObject *sequence)
380 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100381 Py_ssize_t i, n;
382 long total = 0, value;
Georg Brandl116aa622007-08-15 14:28:22 +0000383 PyObject *item;
384 n = PySequence_Length(sequence);
385 if (n < 0)
386 return -1; /* Has no length */
387 for (i = 0; i < n; i++) {
388 item = PySequence_GetItem(sequence, i);
389 if (item == NULL)
390 return -1; /* Not a sequence, or other failure */
Antoine Pitrou04707c02012-01-27 14:07:29 +0100391 if (PyLong_Check(item)) {
392 value = PyLong_AsLong(item);
393 Py_DECREF(item);
394 if (value == -1 && PyErr_Occurred())
395 /* Integer too big to fit in a C long, bail out */
396 return -1;
397 total += value;
398 }
399 else {
400 Py_DECREF(item); /* Discard reference ownership */
401 }
Georg Brandl116aa622007-08-15 14:28:22 +0000402 }
403 return total;
404 }
405
406.. index:: single: sum_sequence()
407
408
409.. _api-types:
410
411Types
412-----
413
414There are few other data types that play a significant role in the Python/C
Georg Brandl60203b42010-10-06 10:11:56 +0000415API; most are simple C types such as :c:type:`int`, :c:type:`long`,
416:c:type:`double` and :c:type:`char\*`. A few structure types are used to
Georg Brandl116aa622007-08-15 14:28:22 +0000417describe static tables used to list the functions exported by a module or the
418data attributes of a new object type, and another is used to describe the value
419of a complex number. These will be discussed together with the functions that
420use them.
421
422
423.. _api-exceptions:
424
425Exceptions
426==========
427
428The Python programmer only needs to deal with exceptions if specific error
429handling is required; unhandled exceptions are automatically propagated to the
430caller, then to the caller's caller, and so on, until they reach the top-level
431interpreter, where they are reported to the user accompanied by a stack
432traceback.
433
434.. index:: single: PyErr_Occurred()
435
Georg Brandldd909db2010-10-17 06:32:59 +0000436For C programmers, however, error checking always has to be explicit. All
437functions in the Python/C API can raise exceptions, unless an explicit claim is
438made otherwise in a function's documentation. In general, when a function
439encounters an error, it sets an exception, discards any object references that
440it owns, and returns an error indicator. If not documented otherwise, this
441indicator is either *NULL* or ``-1``, depending on the function's return type.
442A few functions return a Boolean true/false result, with false indicating an
443error. Very few functions return no explicit error indicator or have an
444ambiguous return value, and require explicit testing for errors with
445:c:func:`PyErr_Occurred`. These exceptions are always explicitly documented.
Georg Brandl116aa622007-08-15 14:28:22 +0000446
447.. index::
448 single: PyErr_SetString()
449 single: PyErr_Clear()
450
451Exception state is maintained in per-thread storage (this is equivalent to
452using global storage in an unthreaded application). A thread can be in one of
453two states: an exception has occurred, or not. The function
Georg Brandl60203b42010-10-06 10:11:56 +0000454:c:func:`PyErr_Occurred` can be used to check for this: it returns a borrowed
Georg Brandl116aa622007-08-15 14:28:22 +0000455reference to the exception type object when an exception has occurred, and
456*NULL* otherwise. There are a number of functions to set the exception state:
Georg Brandl60203b42010-10-06 10:11:56 +0000457:c:func:`PyErr_SetString` is the most common (though not the most general)
458function to set the exception state, and :c:func:`PyErr_Clear` clears the
Georg Brandl116aa622007-08-15 14:28:22 +0000459exception state.
460
461The full exception state consists of three objects (all of which can be
462*NULL*): the exception type, the corresponding exception value, and the
463traceback. These have the same meanings as the Python result of
464``sys.exc_info()``; however, they are not the same: the Python objects represent
465the last exception being handled by a Python :keyword:`try` ...
466:keyword:`except` statement, while the C level exception state only exists while
467an exception is being passed on between C functions until it reaches the Python
468bytecode interpreter's main loop, which takes care of transferring it to
469``sys.exc_info()`` and friends.
470
471.. index:: single: exc_info() (in module sys)
472
473Note that starting with Python 1.5, the preferred, thread-safe way to access the
474exception state from Python code is to call the function :func:`sys.exc_info`,
475which returns the per-thread exception state for Python code. Also, the
476semantics of both ways to access the exception state have changed so that a
477function which catches an exception will save and restore its thread's exception
478state so as to preserve the exception state of its caller. This prevents common
479bugs in exception handling code caused by an innocent-looking function
480overwriting the exception being handled; it also reduces the often unwanted
481lifetime extension for objects that are referenced by the stack frames in the
482traceback.
483
484As a general principle, a function that calls another function to perform some
485task should check whether the called function raised an exception, and if so,
486pass the exception state on to its caller. It should discard any object
487references that it owns, and return an error indicator, but it should *not* set
488another exception --- that would overwrite the exception that was just raised,
489and lose important information about the exact cause of the error.
490
491.. index:: single: sum_sequence()
492
493A simple example of detecting exceptions and passing them on is shown in the
Terry Jan Reedy65e69b32013-03-11 17:23:46 -0400494:c:func:`sum_sequence` example above. It so happens that this example doesn't
Georg Brandl116aa622007-08-15 14:28:22 +0000495need to clean up any owned references when it detects an error. The following
496example function shows some error cleanup. First, to remind you why you like
497Python, we show the equivalent Python code::
498
499 def incr_item(dict, key):
500 try:
501 item = dict[key]
502 except KeyError:
503 item = 0
504 dict[key] = item + 1
505
506.. index:: single: incr_item()
507
508Here is the corresponding C code, in all its glory::
509
510 int
511 incr_item(PyObject *dict, PyObject *key)
512 {
513 /* Objects all initialized to NULL for Py_XDECREF */
514 PyObject *item = NULL, *const_one = NULL, *incremented_item = NULL;
515 int rv = -1; /* Return value initialized to -1 (failure) */
516
517 item = PyObject_GetItem(dict, key);
518 if (item == NULL) {
519 /* Handle KeyError only: */
520 if (!PyErr_ExceptionMatches(PyExc_KeyError))
521 goto error;
522
523 /* Clear the error and use zero: */
524 PyErr_Clear();
Georg Brandld019fe22007-12-08 18:58:51 +0000525 item = PyLong_FromLong(0L);
Georg Brandl116aa622007-08-15 14:28:22 +0000526 if (item == NULL)
527 goto error;
528 }
Georg Brandld019fe22007-12-08 18:58:51 +0000529 const_one = PyLong_FromLong(1L);
Georg Brandl116aa622007-08-15 14:28:22 +0000530 if (const_one == NULL)
531 goto error;
532
533 incremented_item = PyNumber_Add(item, const_one);
534 if (incremented_item == NULL)
535 goto error;
536
537 if (PyObject_SetItem(dict, key, incremented_item) < 0)
538 goto error;
539 rv = 0; /* Success */
540 /* Continue with cleanup code */
541
542 error:
543 /* Cleanup code, shared by success and failure path */
544
545 /* Use Py_XDECREF() to ignore NULL references */
546 Py_XDECREF(item);
547 Py_XDECREF(const_one);
548 Py_XDECREF(incremented_item);
549
550 return rv; /* -1 for error, 0 for success */
551 }
552
553.. index:: single: incr_item()
554
555.. index::
556 single: PyErr_ExceptionMatches()
557 single: PyErr_Clear()
558 single: Py_XDECREF()
559
Christian Heimes5b5e81c2007-12-31 16:14:33 +0000560This example represents an endorsed use of the ``goto`` statement in C!
Georg Brandl60203b42010-10-06 10:11:56 +0000561It illustrates the use of :c:func:`PyErr_ExceptionMatches` and
562:c:func:`PyErr_Clear` to handle specific exceptions, and the use of
563:c:func:`Py_XDECREF` to dispose of owned references that may be *NULL* (note the
564``'X'`` in the name; :c:func:`Py_DECREF` would crash when confronted with a
Georg Brandl116aa622007-08-15 14:28:22 +0000565*NULL* reference). It is important that the variables used to hold owned
566references are initialized to *NULL* for this to work; likewise, the proposed
567return value is initialized to ``-1`` (failure) and only set to success after
568the final call made is successful.
569
570
571.. _api-embedding:
572
573Embedding Python
574================
575
576The one important task that only embedders (as opposed to extension writers) of
577the Python interpreter have to worry about is the initialization, and possibly
578the finalization, of the Python interpreter. Most functionality of the
579interpreter can only be used after the interpreter has been initialized.
580
581.. index::
582 single: Py_Initialize()
Georg Brandl1a3284e2007-12-02 09:40:06 +0000583 module: builtins
Georg Brandl116aa622007-08-15 14:28:22 +0000584 module: __main__
585 module: sys
Georg Brandl116aa622007-08-15 14:28:22 +0000586 triple: module; search; path
587 single: path (in module sys)
588
Georg Brandl60203b42010-10-06 10:11:56 +0000589The basic initialization function is :c:func:`Py_Initialize`. This initializes
Georg Brandl116aa622007-08-15 14:28:22 +0000590the table of loaded modules, and creates the fundamental modules
Éric Araujo8b8f2ec2011-03-26 07:22:01 +0100591:mod:`builtins`, :mod:`__main__`, and :mod:`sys`. It also
Georg Brandl116aa622007-08-15 14:28:22 +0000592initializes the module search path (``sys.path``).
593
Benjamin Peterson2ebf8ce2010-06-27 21:48:35 +0000594.. index:: single: PySys_SetArgvEx()
Georg Brandl116aa622007-08-15 14:28:22 +0000595
Georg Brandl60203b42010-10-06 10:11:56 +0000596:c:func:`Py_Initialize` does not set the "script argument list" (``sys.argv``).
Benjamin Peterson2ebf8ce2010-06-27 21:48:35 +0000597If this variable is needed by Python code that will be executed later, it must
598be set explicitly with a call to ``PySys_SetArgvEx(argc, argv, updatepath)``
Georg Brandl60203b42010-10-06 10:11:56 +0000599after the call to :c:func:`Py_Initialize`.
Georg Brandl116aa622007-08-15 14:28:22 +0000600
601On most systems (in particular, on Unix and Windows, although the details are
Georg Brandl60203b42010-10-06 10:11:56 +0000602slightly different), :c:func:`Py_Initialize` calculates the module search path
Georg Brandl116aa622007-08-15 14:28:22 +0000603based upon its best guess for the location of the standard Python interpreter
604executable, assuming that the Python library is found in a fixed location
605relative to the Python interpreter executable. In particular, it looks for a
606directory named :file:`lib/python{X.Y}` relative to the parent directory
607where the executable named :file:`python` is found on the shell command search
608path (the environment variable :envvar:`PATH`).
609
610For instance, if the Python executable is found in
611:file:`/usr/local/bin/python`, it will assume that the libraries are in
612:file:`/usr/local/lib/python{X.Y}`. (In fact, this particular path is also
613the "fallback" location, used when no executable file named :file:`python` is
614found along :envvar:`PATH`.) The user can override this behavior by setting the
615environment variable :envvar:`PYTHONHOME`, or insert additional directories in
616front of the standard path by setting :envvar:`PYTHONPATH`.
617
618.. index::
619 single: Py_SetProgramName()
620 single: Py_GetPath()
621 single: Py_GetPrefix()
622 single: Py_GetExecPrefix()
623 single: Py_GetProgramFullPath()
624
625The embedding application can steer the search by calling
Georg Brandl60203b42010-10-06 10:11:56 +0000626``Py_SetProgramName(file)`` *before* calling :c:func:`Py_Initialize`. Note that
Georg Brandl116aa622007-08-15 14:28:22 +0000627:envvar:`PYTHONHOME` still overrides this and :envvar:`PYTHONPATH` is still
628inserted in front of the standard path. An application that requires total
Georg Brandl60203b42010-10-06 10:11:56 +0000629control has to provide its own implementation of :c:func:`Py_GetPath`,
630:c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, and
631:c:func:`Py_GetProgramFullPath` (all defined in :file:`Modules/getpath.c`).
Georg Brandl116aa622007-08-15 14:28:22 +0000632
633.. index:: single: Py_IsInitialized()
634
635Sometimes, it is desirable to "uninitialize" Python. For instance, the
636application may want to start over (make another call to
Georg Brandl60203b42010-10-06 10:11:56 +0000637:c:func:`Py_Initialize`) or the application is simply done with its use of
Georg Brandl116aa622007-08-15 14:28:22 +0000638Python and wants to free memory allocated by Python. This can be accomplished
Martin Panterb4ce1fc2015-11-30 03:18:29 +0000639by calling :c:func:`Py_FinalizeEx`. The function :c:func:`Py_IsInitialized` returns
Georg Brandl116aa622007-08-15 14:28:22 +0000640true if Python is currently in the initialized state. More information about
Martin Panterb4ce1fc2015-11-30 03:18:29 +0000641these functions is given in a later chapter. Notice that :c:func:`Py_FinalizeEx`
Georg Brandl116aa622007-08-15 14:28:22 +0000642does *not* free all memory allocated by the Python interpreter, e.g. memory
643allocated by extension modules currently cannot be released.
644
645
646.. _api-debugging:
647
648Debugging Builds
649================
650
651Python can be built with several macros to enable extra checks of the
652interpreter and extension modules. These checks tend to add a large amount of
653overhead to the runtime so they are not enabled by default.
654
655A full list of the various types of debugging builds is in the file
656:file:`Misc/SpecialBuilds.txt` in the Python source distribution. Builds are
657available that support tracing of reference counts, debugging the memory
658allocator, or low-level profiling of the main interpreter loop. Only the most
659frequently-used builds will be described in the remainder of this section.
660
Georg Brandl60203b42010-10-06 10:11:56 +0000661Compiling the interpreter with the :c:macro:`Py_DEBUG` macro defined produces
662what is generally meant by "a debug build" of Python. :c:macro:`Py_DEBUG` is
Éric Araujod2f8cec2011-06-08 05:29:39 +0200663enabled in the Unix build by adding ``--with-pydebug`` to the
664:file:`./configure` command. It is also implied by the presence of the
Georg Brandl60203b42010-10-06 10:11:56 +0000665not-Python-specific :c:macro:`_DEBUG` macro. When :c:macro:`Py_DEBUG` is enabled
Georg Brandl116aa622007-08-15 14:28:22 +0000666in the Unix build, compiler optimization is disabled.
667
668In addition to the reference count debugging described below, the following
669extra checks are performed:
670
671* Extra checks are added to the object allocator.
672
673* Extra checks are added to the parser and compiler.
674
675* Downcasts from wide types to narrow types are checked for loss of information.
676
677* A number of assertions are added to the dictionary and set implementations.
678 In addition, the set object acquires a :meth:`test_c_api` method.
679
680* Sanity checks of the input arguments are added to frame creation.
681
Mark Dickinsonbf5c6a92009-01-17 10:21:23 +0000682* The storage for ints is initialized with a known invalid pattern to catch
Georg Brandl116aa622007-08-15 14:28:22 +0000683 reference to uninitialized digits.
684
685* Low-level tracing and extra exception checking are added to the runtime
686 virtual machine.
687
688* Extra checks are added to the memory arena implementation.
689
690* Extra debugging is added to the thread module.
691
692There may be additional checks not mentioned here.
693
Georg Brandl60203b42010-10-06 10:11:56 +0000694Defining :c:macro:`Py_TRACE_REFS` enables reference tracing. When defined, a
Georg Brandl116aa622007-08-15 14:28:22 +0000695circular doubly linked list of active objects is maintained by adding two extra
Georg Brandl60203b42010-10-06 10:11:56 +0000696fields to every :c:type:`PyObject`. Total allocations are tracked as well. Upon
Georg Brandl116aa622007-08-15 14:28:22 +0000697exit, all existing references are printed. (In interactive mode this happens
Georg Brandl60203b42010-10-06 10:11:56 +0000698after every statement run by the interpreter.) Implied by :c:macro:`Py_DEBUG`.
Georg Brandl116aa622007-08-15 14:28:22 +0000699
700Please refer to :file:`Misc/SpecialBuilds.txt` in the Python source distribution
701for more detailed information.
702