blob: 74681d2c8518892dd13324281d82e1e82b59b9e7 [file] [log] [blame]
Georg Brandl116aa622007-08-15 14:28:22 +00001.. highlightlang:: c
2
3
4.. _api-intro:
5
6************
7Introduction
8************
9
10The Application Programmer's Interface to Python gives C and C++ programmers
11access to the Python interpreter at a variety of levels. The API is equally
12usable from C++, but for brevity it is generally referred to as the Python/C
13API. There are two fundamentally different reasons for using the Python/C API.
14The first reason is to write *extension modules* for specific purposes; these
15are C modules that extend the Python interpreter. This is probably the most
16common use. The second reason is to use Python as a component in a larger
17application; this technique is generally referred to as :dfn:`embedding` Python
18in an application.
19
20Writing an extension module is a relatively well-understood process, where a
21"cookbook" approach works well. There are several tools that automate the
22process to some extent. While people have embedded Python in other
23applications since its early existence, the process of embedding Python is less
24straightforward than writing an extension.
25
26Many API functions are useful independent of whether you're embedding or
27extending Python; moreover, most applications that embed Python will need to
28provide a custom extension as well, so it's probably a good idea to become
29familiar with writing an extension before attempting to embed Python in a real
30application.
31
32
33.. _api-includes:
34
35Include Files
36=============
37
38All function, type and macro definitions needed to use the Python/C API are
39included in your code by the following line::
40
41 #include "Python.h"
42
43This implies inclusion of the following standard headers: ``<stdio.h>``,
Georg Brandl4f13d612010-11-23 18:14:57 +000044``<string.h>``, ``<errno.h>``, ``<limits.h>``, ``<assert.h>`` and ``<stdlib.h>``
45(if available).
Georg Brandl116aa622007-08-15 14:28:22 +000046
Georg Brandle720c0a2009-04-27 16:20:50 +000047.. note::
Georg Brandl116aa622007-08-15 14:28:22 +000048
49 Since Python may define some pre-processor definitions which affect the standard
50 headers on some systems, you *must* include :file:`Python.h` before any standard
51 headers are included.
52
53All user visible names defined by Python.h (except those defined by the included
54standard headers) have one of the prefixes ``Py`` or ``_Py``. Names beginning
55with ``_Py`` are for internal use by the Python implementation and should not be
56used by extension writers. Structure member names do not have a reserved prefix.
57
58**Important:** user code should never define names that begin with ``Py`` or
59``_Py``. This confuses the reader, and jeopardizes the portability of the user
60code to future Python versions, which may define additional names beginning with
61one of these prefixes.
62
63The header files are typically installed with Python. On Unix, these are
64located in the directories :file:`{prefix}/include/pythonversion/` and
65:file:`{exec_prefix}/include/pythonversion/`, where :envvar:`prefix` and
66:envvar:`exec_prefix` are defined by the corresponding parameters to Python's
Serhiy Storchaka885bdc42016-02-11 13:10:36 +020067:program:`configure` script and *version* is
68``'%d.%d' % sys.version_info[:2]``. On Windows, the headers are installed
69in :file:`{prefix}/include`, where :envvar:`prefix` is the installation
70directory specified to the installer.
Georg Brandl116aa622007-08-15 14:28:22 +000071
72To include the headers, place both directories (if different) on your compiler's
73search path for includes. Do *not* place the parent directories on the search
74path and then use ``#include <pythonX.Y/Python.h>``; this will break on
75multi-platform builds since the platform independent headers under
76:envvar:`prefix` include the platform specific headers from
77:envvar:`exec_prefix`.
78
79C++ users should note that though the API is defined entirely using C, the
80header files do properly declare the entry points to be ``extern "C"``, so there
81is no need to do anything special to use the API from C++.
82
83
84.. _api-objects:
85
86Objects, Types and Reference Counts
87===================================
88
89.. index:: object: type
90
91Most Python/C API functions have one or more arguments as well as a return value
Georg Brandl60203b42010-10-06 10:11:56 +000092of type :c:type:`PyObject\*`. This type is a pointer to an opaque data type
Georg Brandl116aa622007-08-15 14:28:22 +000093representing an arbitrary Python object. Since all Python object types are
94treated the same way by the Python language in most situations (e.g.,
95assignments, scope rules, and argument passing), it is only fitting that they
96should be represented by a single C type. Almost all Python objects live on the
97heap: you never declare an automatic or static variable of type
Georg Brandl60203b42010-10-06 10:11:56 +000098:c:type:`PyObject`, only pointer variables of type :c:type:`PyObject\*` can be
Georg Brandl116aa622007-08-15 14:28:22 +000099declared. The sole exception are the type objects; since these must never be
Georg Brandl60203b42010-10-06 10:11:56 +0000100deallocated, they are typically static :c:type:`PyTypeObject` objects.
Georg Brandl116aa622007-08-15 14:28:22 +0000101
102All Python objects (even Python integers) have a :dfn:`type` and a
103:dfn:`reference count`. An object's type determines what kind of object it is
104(e.g., an integer, a list, or a user-defined function; there are many more as
105explained in :ref:`types`). For each of the well-known types there is a macro
106to check whether an object is of that type; for instance, ``PyList_Check(a)`` is
107true if (and only if) the object pointed to by *a* is a Python list.
108
109
110.. _api-refcounts:
111
112Reference Counts
113----------------
114
115The reference count is important because today's computers have a finite (and
116often severely limited) memory size; it counts how many different places there
117are that have a reference to an object. Such a place could be another object,
118or a global (or static) C variable, or a local variable in some C function.
119When an object's reference count becomes zero, the object is deallocated. If
120it contains references to other objects, their reference count is decremented.
121Those other objects may be deallocated in turn, if this decrement makes their
122reference count become zero, and so on. (There's an obvious problem with
123objects that reference each other here; for now, the solution is "don't do
124that.")
125
126.. index::
127 single: Py_INCREF()
128 single: Py_DECREF()
129
130Reference counts are always manipulated explicitly. The normal way is to use
Georg Brandl60203b42010-10-06 10:11:56 +0000131the macro :c:func:`Py_INCREF` to increment an object's reference count by one,
132and :c:func:`Py_DECREF` to decrement it by one. The :c:func:`Py_DECREF` macro
Georg Brandl116aa622007-08-15 14:28:22 +0000133is considerably more complex than the incref one, since it must check whether
134the reference count becomes zero and then cause the object's deallocator to be
135called. The deallocator is a function pointer contained in the object's type
136structure. The type-specific deallocator takes care of decrementing the
137reference counts for other objects contained in the object if this is a compound
138object type, such as a list, as well as performing any additional finalization
139that's needed. There's no chance that the reference count can overflow; at
140least as many bits are used to hold the reference count as there are distinct
Christian Heimesdd15f6c2008-03-16 00:07:10 +0000141memory locations in virtual memory (assuming ``sizeof(Py_ssize_t) >= sizeof(void*)``).
Georg Brandl116aa622007-08-15 14:28:22 +0000142Thus, the reference count increment is a simple operation.
143
144It is not necessary to increment an object's reference count for every local
145variable that contains a pointer to an object. In theory, the object's
146reference count goes up by one when the variable is made to point to it and it
147goes down by one when the variable goes out of scope. However, these two
148cancel each other out, so at the end the reference count hasn't changed. The
149only real reason to use the reference count is to prevent the object from being
150deallocated as long as our variable is pointing to it. If we know that there
151is at least one other reference to the object that lives at least as long as
152our variable, there is no need to increment the reference count temporarily.
153An important situation where this arises is in objects that are passed as
154arguments to C functions in an extension module that are called from Python;
155the call mechanism guarantees to hold a reference to every argument for the
156duration of the call.
157
158However, a common pitfall is to extract an object from a list and hold on to it
159for a while without incrementing its reference count. Some other operation might
160conceivably remove the object from the list, decrementing its reference count
161and possible deallocating it. The real danger is that innocent-looking
162operations may invoke arbitrary Python code which could do this; there is a code
Georg Brandl60203b42010-10-06 10:11:56 +0000163path which allows control to flow back to the user from a :c:func:`Py_DECREF`, so
Georg Brandl116aa622007-08-15 14:28:22 +0000164almost any operation is potentially dangerous.
165
166A safe approach is to always use the generic operations (functions whose name
167begins with ``PyObject_``, ``PyNumber_``, ``PySequence_`` or ``PyMapping_``).
168These operations always increment the reference count of the object they return.
Georg Brandl60203b42010-10-06 10:11:56 +0000169This leaves the caller with the responsibility to call :c:func:`Py_DECREF` when
Georg Brandl116aa622007-08-15 14:28:22 +0000170they are done with the result; this soon becomes second nature.
171
172
173.. _api-refcountdetails:
174
175Reference Count Details
176^^^^^^^^^^^^^^^^^^^^^^^
177
178The reference count behavior of functions in the Python/C API is best explained
179in terms of *ownership of references*. Ownership pertains to references, never
180to objects (objects are not owned: they are always shared). "Owning a
181reference" means being responsible for calling Py_DECREF on it when the
182reference is no longer needed. Ownership can also be transferred, meaning that
183the code that receives ownership of the reference then becomes responsible for
Georg Brandl60203b42010-10-06 10:11:56 +0000184eventually decref'ing it by calling :c:func:`Py_DECREF` or :c:func:`Py_XDECREF`
Georg Brandl116aa622007-08-15 14:28:22 +0000185when it's no longer needed---or passing on this responsibility (usually to its
186caller). When a function passes ownership of a reference on to its caller, the
187caller is said to receive a *new* reference. When no ownership is transferred,
188the caller is said to *borrow* the reference. Nothing needs to be done for a
189borrowed reference.
190
Benjamin Petersonad3d5c22009-02-26 03:38:59 +0000191Conversely, when a calling function passes in a reference to an object, there
Georg Brandl116aa622007-08-15 14:28:22 +0000192are two possibilities: the function *steals* a reference to the object, or it
193does not. *Stealing a reference* means that when you pass a reference to a
194function, that function assumes that it now owns that reference, and you are not
195responsible for it any longer.
196
197.. index::
198 single: PyList_SetItem()
199 single: PyTuple_SetItem()
200
201Few functions steal references; the two notable exceptions are
Georg Brandl60203b42010-10-06 10:11:56 +0000202:c:func:`PyList_SetItem` and :c:func:`PyTuple_SetItem`, which steal a reference
Georg Brandl116aa622007-08-15 14:28:22 +0000203to the item (but not to the tuple or list into which the item is put!). These
204functions were designed to steal a reference because of a common idiom for
205populating a tuple or list with newly created objects; for example, the code to
206create the tuple ``(1, 2, "three")`` could look like this (forgetting about
207error handling for the moment; a better way to code this is shown below)::
208
209 PyObject *t;
210
211 t = PyTuple_New(3);
Georg Brandld019fe22007-12-08 18:58:51 +0000212 PyTuple_SetItem(t, 0, PyLong_FromLong(1L));
213 PyTuple_SetItem(t, 1, PyLong_FromLong(2L));
Gregory P. Smith4b52ae82013-03-22 13:43:30 -0700214 PyTuple_SetItem(t, 2, PyUnicode_FromString("three"));
Georg Brandl116aa622007-08-15 14:28:22 +0000215
Georg Brandl60203b42010-10-06 10:11:56 +0000216Here, :c:func:`PyLong_FromLong` returns a new reference which is immediately
217stolen by :c:func:`PyTuple_SetItem`. When you want to keep using an object
218although the reference to it will be stolen, use :c:func:`Py_INCREF` to grab
Georg Brandl116aa622007-08-15 14:28:22 +0000219another reference before calling the reference-stealing function.
220
Georg Brandl60203b42010-10-06 10:11:56 +0000221Incidentally, :c:func:`PyTuple_SetItem` is the *only* way to set tuple items;
222:c:func:`PySequence_SetItem` and :c:func:`PyObject_SetItem` refuse to do this
Georg Brandl116aa622007-08-15 14:28:22 +0000223since tuples are an immutable data type. You should only use
Georg Brandl60203b42010-10-06 10:11:56 +0000224:c:func:`PyTuple_SetItem` for tuples that you are creating yourself.
Georg Brandl116aa622007-08-15 14:28:22 +0000225
Georg Brandl60203b42010-10-06 10:11:56 +0000226Equivalent code for populating a list can be written using :c:func:`PyList_New`
227and :c:func:`PyList_SetItem`.
Georg Brandl116aa622007-08-15 14:28:22 +0000228
229However, in practice, you will rarely use these ways of creating and populating
Georg Brandl60203b42010-10-06 10:11:56 +0000230a tuple or list. There's a generic function, :c:func:`Py_BuildValue`, that can
Georg Brandl116aa622007-08-15 14:28:22 +0000231create most common objects from C values, directed by a :dfn:`format string`.
232For example, the above two blocks of code could be replaced by the following
233(which also takes care of the error checking)::
234
235 PyObject *tuple, *list;
236
237 tuple = Py_BuildValue("(iis)", 1, 2, "three");
238 list = Py_BuildValue("[iis]", 1, 2, "three");
239
Georg Brandl60203b42010-10-06 10:11:56 +0000240It is much more common to use :c:func:`PyObject_SetItem` and friends with items
Georg Brandl116aa622007-08-15 14:28:22 +0000241whose references you are only borrowing, like arguments that were passed in to
242the function you are writing. In that case, their behaviour regarding reference
243counts is much saner, since you don't have to increment a reference count so you
244can give a reference away ("have it be stolen"). For example, this function
245sets all items of a list (actually, any mutable sequence) to a given item::
246
247 int
248 set_all(PyObject *target, PyObject *item)
249 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100250 Py_ssize_t i, n;
Georg Brandl116aa622007-08-15 14:28:22 +0000251
252 n = PyObject_Length(target);
253 if (n < 0)
254 return -1;
255 for (i = 0; i < n; i++) {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100256 PyObject *index = PyLong_FromSsize_t(i);
Georg Brandl116aa622007-08-15 14:28:22 +0000257 if (!index)
258 return -1;
Antoine Pitrou04707c02012-01-27 14:07:29 +0100259 if (PyObject_SetItem(target, index, item) < 0) {
260 Py_DECREF(index);
Georg Brandl116aa622007-08-15 14:28:22 +0000261 return -1;
Antoine Pitrou04707c02012-01-27 14:07:29 +0100262 }
Georg Brandl116aa622007-08-15 14:28:22 +0000263 Py_DECREF(index);
264 }
265 return 0;
266 }
267
268.. index:: single: set_all()
269
270The situation is slightly different for function return values. While passing
271a reference to most functions does not change your ownership responsibilities
272for that reference, many functions that return a reference to an object give
273you ownership of the reference. The reason is simple: in many cases, the
274returned object is created on the fly, and the reference you get is the only
275reference to the object. Therefore, the generic functions that return object
Georg Brandl60203b42010-10-06 10:11:56 +0000276references, like :c:func:`PyObject_GetItem` and :c:func:`PySequence_GetItem`,
Georg Brandl116aa622007-08-15 14:28:22 +0000277always return a new reference (the caller becomes the owner of the reference).
278
279It is important to realize that whether you own a reference returned by a
280function depends on which function you call only --- *the plumage* (the type of
281the object passed as an argument to the function) *doesn't enter into it!*
Georg Brandl60203b42010-10-06 10:11:56 +0000282Thus, if you extract an item from a list using :c:func:`PyList_GetItem`, you
Georg Brandl116aa622007-08-15 14:28:22 +0000283don't own the reference --- but if you obtain the same item from the same list
Georg Brandl60203b42010-10-06 10:11:56 +0000284using :c:func:`PySequence_GetItem` (which happens to take exactly the same
Georg Brandl116aa622007-08-15 14:28:22 +0000285arguments), you do own a reference to the returned object.
286
287.. index::
288 single: PyList_GetItem()
289 single: PySequence_GetItem()
290
291Here is an example of how you could write a function that computes the sum of
Georg Brandl60203b42010-10-06 10:11:56 +0000292the items in a list of integers; once using :c:func:`PyList_GetItem`, and once
293using :c:func:`PySequence_GetItem`. ::
Georg Brandl116aa622007-08-15 14:28:22 +0000294
295 long
296 sum_list(PyObject *list)
297 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100298 Py_ssize_t i, n;
299 long total = 0, value;
Georg Brandl116aa622007-08-15 14:28:22 +0000300 PyObject *item;
301
302 n = PyList_Size(list);
303 if (n < 0)
304 return -1; /* Not a list */
305 for (i = 0; i < n; i++) {
306 item = PyList_GetItem(list, i); /* Can't fail */
Georg Brandld019fe22007-12-08 18:58:51 +0000307 if (!PyLong_Check(item)) continue; /* Skip non-integers */
Antoine Pitrou04707c02012-01-27 14:07:29 +0100308 value = PyLong_AsLong(item);
309 if (value == -1 && PyErr_Occurred())
310 /* Integer too big to fit in a C long, bail out */
311 return -1;
312 total += value;
Georg Brandl116aa622007-08-15 14:28:22 +0000313 }
314 return total;
315 }
316
317.. index:: single: sum_list()
318
319::
320
321 long
322 sum_sequence(PyObject *sequence)
323 {
Antoine Pitrou04707c02012-01-27 14:07:29 +0100324 Py_ssize_t i, n;
325 long total = 0, value;
Georg Brandl116aa622007-08-15 14:28:22 +0000326 PyObject *item;
327 n = PySequence_Length(sequence);
328 if (n < 0)
329 return -1; /* Has no length */
330 for (i = 0; i < n; i++) {
331 item = PySequence_GetItem(sequence, i);
332 if (item == NULL)
333 return -1; /* Not a sequence, or other failure */
Antoine Pitrou04707c02012-01-27 14:07:29 +0100334 if (PyLong_Check(item)) {
335 value = PyLong_AsLong(item);
336 Py_DECREF(item);
337 if (value == -1 && PyErr_Occurred())
338 /* Integer too big to fit in a C long, bail out */
339 return -1;
340 total += value;
341 }
342 else {
343 Py_DECREF(item); /* Discard reference ownership */
344 }
Georg Brandl116aa622007-08-15 14:28:22 +0000345 }
346 return total;
347 }
348
349.. index:: single: sum_sequence()
350
351
352.. _api-types:
353
354Types
355-----
356
357There are few other data types that play a significant role in the Python/C
Georg Brandl60203b42010-10-06 10:11:56 +0000358API; most are simple C types such as :c:type:`int`, :c:type:`long`,
359:c:type:`double` and :c:type:`char\*`. A few structure types are used to
Georg Brandl116aa622007-08-15 14:28:22 +0000360describe static tables used to list the functions exported by a module or the
361data attributes of a new object type, and another is used to describe the value
362of a complex number. These will be discussed together with the functions that
363use them.
364
365
366.. _api-exceptions:
367
368Exceptions
369==========
370
371The Python programmer only needs to deal with exceptions if specific error
372handling is required; unhandled exceptions are automatically propagated to the
373caller, then to the caller's caller, and so on, until they reach the top-level
374interpreter, where they are reported to the user accompanied by a stack
375traceback.
376
377.. index:: single: PyErr_Occurred()
378
Georg Brandldd909db2010-10-17 06:32:59 +0000379For C programmers, however, error checking always has to be explicit. All
380functions in the Python/C API can raise exceptions, unless an explicit claim is
381made otherwise in a function's documentation. In general, when a function
382encounters an error, it sets an exception, discards any object references that
383it owns, and returns an error indicator. If not documented otherwise, this
384indicator is either *NULL* or ``-1``, depending on the function's return type.
385A few functions return a Boolean true/false result, with false indicating an
386error. Very few functions return no explicit error indicator or have an
387ambiguous return value, and require explicit testing for errors with
388:c:func:`PyErr_Occurred`. These exceptions are always explicitly documented.
Georg Brandl116aa622007-08-15 14:28:22 +0000389
390.. index::
391 single: PyErr_SetString()
392 single: PyErr_Clear()
393
394Exception state is maintained in per-thread storage (this is equivalent to
395using global storage in an unthreaded application). A thread can be in one of
396two states: an exception has occurred, or not. The function
Georg Brandl60203b42010-10-06 10:11:56 +0000397:c:func:`PyErr_Occurred` can be used to check for this: it returns a borrowed
Georg Brandl116aa622007-08-15 14:28:22 +0000398reference to the exception type object when an exception has occurred, and
399*NULL* otherwise. There are a number of functions to set the exception state:
Georg Brandl60203b42010-10-06 10:11:56 +0000400:c:func:`PyErr_SetString` is the most common (though not the most general)
401function to set the exception state, and :c:func:`PyErr_Clear` clears the
Georg Brandl116aa622007-08-15 14:28:22 +0000402exception state.
403
404The full exception state consists of three objects (all of which can be
405*NULL*): the exception type, the corresponding exception value, and the
406traceback. These have the same meanings as the Python result of
407``sys.exc_info()``; however, they are not the same: the Python objects represent
408the last exception being handled by a Python :keyword:`try` ...
409:keyword:`except` statement, while the C level exception state only exists while
410an exception is being passed on between C functions until it reaches the Python
411bytecode interpreter's main loop, which takes care of transferring it to
412``sys.exc_info()`` and friends.
413
414.. index:: single: exc_info() (in module sys)
415
416Note that starting with Python 1.5, the preferred, thread-safe way to access the
417exception state from Python code is to call the function :func:`sys.exc_info`,
418which returns the per-thread exception state for Python code. Also, the
419semantics of both ways to access the exception state have changed so that a
420function which catches an exception will save and restore its thread's exception
421state so as to preserve the exception state of its caller. This prevents common
422bugs in exception handling code caused by an innocent-looking function
423overwriting the exception being handled; it also reduces the often unwanted
424lifetime extension for objects that are referenced by the stack frames in the
425traceback.
426
427As a general principle, a function that calls another function to perform some
428task should check whether the called function raised an exception, and if so,
429pass the exception state on to its caller. It should discard any object
430references that it owns, and return an error indicator, but it should *not* set
431another exception --- that would overwrite the exception that was just raised,
432and lose important information about the exact cause of the error.
433
434.. index:: single: sum_sequence()
435
436A simple example of detecting exceptions and passing them on is shown in the
Terry Jan Reedy65e69b32013-03-11 17:23:46 -0400437:c:func:`sum_sequence` example above. It so happens that this example doesn't
Georg Brandl116aa622007-08-15 14:28:22 +0000438need to clean up any owned references when it detects an error. The following
439example function shows some error cleanup. First, to remind you why you like
440Python, we show the equivalent Python code::
441
442 def incr_item(dict, key):
443 try:
444 item = dict[key]
445 except KeyError:
446 item = 0
447 dict[key] = item + 1
448
449.. index:: single: incr_item()
450
451Here is the corresponding C code, in all its glory::
452
453 int
454 incr_item(PyObject *dict, PyObject *key)
455 {
456 /* Objects all initialized to NULL for Py_XDECREF */
457 PyObject *item = NULL, *const_one = NULL, *incremented_item = NULL;
458 int rv = -1; /* Return value initialized to -1 (failure) */
459
460 item = PyObject_GetItem(dict, key);
461 if (item == NULL) {
462 /* Handle KeyError only: */
463 if (!PyErr_ExceptionMatches(PyExc_KeyError))
464 goto error;
465
466 /* Clear the error and use zero: */
467 PyErr_Clear();
Georg Brandld019fe22007-12-08 18:58:51 +0000468 item = PyLong_FromLong(0L);
Georg Brandl116aa622007-08-15 14:28:22 +0000469 if (item == NULL)
470 goto error;
471 }
Georg Brandld019fe22007-12-08 18:58:51 +0000472 const_one = PyLong_FromLong(1L);
Georg Brandl116aa622007-08-15 14:28:22 +0000473 if (const_one == NULL)
474 goto error;
475
476 incremented_item = PyNumber_Add(item, const_one);
477 if (incremented_item == NULL)
478 goto error;
479
480 if (PyObject_SetItem(dict, key, incremented_item) < 0)
481 goto error;
482 rv = 0; /* Success */
483 /* Continue with cleanup code */
484
485 error:
486 /* Cleanup code, shared by success and failure path */
487
488 /* Use Py_XDECREF() to ignore NULL references */
489 Py_XDECREF(item);
490 Py_XDECREF(const_one);
491 Py_XDECREF(incremented_item);
492
493 return rv; /* -1 for error, 0 for success */
494 }
495
496.. index:: single: incr_item()
497
498.. index::
499 single: PyErr_ExceptionMatches()
500 single: PyErr_Clear()
501 single: Py_XDECREF()
502
Christian Heimes5b5e81c2007-12-31 16:14:33 +0000503This example represents an endorsed use of the ``goto`` statement in C!
Georg Brandl60203b42010-10-06 10:11:56 +0000504It illustrates the use of :c:func:`PyErr_ExceptionMatches` and
505:c:func:`PyErr_Clear` to handle specific exceptions, and the use of
506:c:func:`Py_XDECREF` to dispose of owned references that may be *NULL* (note the
507``'X'`` in the name; :c:func:`Py_DECREF` would crash when confronted with a
Georg Brandl116aa622007-08-15 14:28:22 +0000508*NULL* reference). It is important that the variables used to hold owned
509references are initialized to *NULL* for this to work; likewise, the proposed
510return value is initialized to ``-1`` (failure) and only set to success after
511the final call made is successful.
512
513
514.. _api-embedding:
515
516Embedding Python
517================
518
519The one important task that only embedders (as opposed to extension writers) of
520the Python interpreter have to worry about is the initialization, and possibly
521the finalization, of the Python interpreter. Most functionality of the
522interpreter can only be used after the interpreter has been initialized.
523
524.. index::
525 single: Py_Initialize()
Georg Brandl1a3284e2007-12-02 09:40:06 +0000526 module: builtins
Georg Brandl116aa622007-08-15 14:28:22 +0000527 module: __main__
528 module: sys
Georg Brandl116aa622007-08-15 14:28:22 +0000529 triple: module; search; path
530 single: path (in module sys)
531
Georg Brandl60203b42010-10-06 10:11:56 +0000532The basic initialization function is :c:func:`Py_Initialize`. This initializes
Georg Brandl116aa622007-08-15 14:28:22 +0000533the table of loaded modules, and creates the fundamental modules
Éric Araujo8b8f2ec2011-03-26 07:22:01 +0100534:mod:`builtins`, :mod:`__main__`, and :mod:`sys`. It also
Georg Brandl116aa622007-08-15 14:28:22 +0000535initializes the module search path (``sys.path``).
536
Benjamin Peterson2ebf8ce2010-06-27 21:48:35 +0000537.. index:: single: PySys_SetArgvEx()
Georg Brandl116aa622007-08-15 14:28:22 +0000538
Georg Brandl60203b42010-10-06 10:11:56 +0000539:c:func:`Py_Initialize` does not set the "script argument list" (``sys.argv``).
Benjamin Peterson2ebf8ce2010-06-27 21:48:35 +0000540If this variable is needed by Python code that will be executed later, it must
541be set explicitly with a call to ``PySys_SetArgvEx(argc, argv, updatepath)``
Georg Brandl60203b42010-10-06 10:11:56 +0000542after the call to :c:func:`Py_Initialize`.
Georg Brandl116aa622007-08-15 14:28:22 +0000543
544On most systems (in particular, on Unix and Windows, although the details are
Georg Brandl60203b42010-10-06 10:11:56 +0000545slightly different), :c:func:`Py_Initialize` calculates the module search path
Georg Brandl116aa622007-08-15 14:28:22 +0000546based upon its best guess for the location of the standard Python interpreter
547executable, assuming that the Python library is found in a fixed location
548relative to the Python interpreter executable. In particular, it looks for a
549directory named :file:`lib/python{X.Y}` relative to the parent directory
550where the executable named :file:`python` is found on the shell command search
551path (the environment variable :envvar:`PATH`).
552
553For instance, if the Python executable is found in
554:file:`/usr/local/bin/python`, it will assume that the libraries are in
555:file:`/usr/local/lib/python{X.Y}`. (In fact, this particular path is also
556the "fallback" location, used when no executable file named :file:`python` is
557found along :envvar:`PATH`.) The user can override this behavior by setting the
558environment variable :envvar:`PYTHONHOME`, or insert additional directories in
559front of the standard path by setting :envvar:`PYTHONPATH`.
560
561.. index::
562 single: Py_SetProgramName()
563 single: Py_GetPath()
564 single: Py_GetPrefix()
565 single: Py_GetExecPrefix()
566 single: Py_GetProgramFullPath()
567
568The embedding application can steer the search by calling
Georg Brandl60203b42010-10-06 10:11:56 +0000569``Py_SetProgramName(file)`` *before* calling :c:func:`Py_Initialize`. Note that
Georg Brandl116aa622007-08-15 14:28:22 +0000570:envvar:`PYTHONHOME` still overrides this and :envvar:`PYTHONPATH` is still
571inserted in front of the standard path. An application that requires total
Georg Brandl60203b42010-10-06 10:11:56 +0000572control has to provide its own implementation of :c:func:`Py_GetPath`,
573:c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, and
574:c:func:`Py_GetProgramFullPath` (all defined in :file:`Modules/getpath.c`).
Georg Brandl116aa622007-08-15 14:28:22 +0000575
576.. index:: single: Py_IsInitialized()
577
578Sometimes, it is desirable to "uninitialize" Python. For instance, the
579application may want to start over (make another call to
Georg Brandl60203b42010-10-06 10:11:56 +0000580:c:func:`Py_Initialize`) or the application is simply done with its use of
Georg Brandl116aa622007-08-15 14:28:22 +0000581Python and wants to free memory allocated by Python. This can be accomplished
Martin Panterb4ce1fc2015-11-30 03:18:29 +0000582by calling :c:func:`Py_FinalizeEx`. The function :c:func:`Py_IsInitialized` returns
Georg Brandl116aa622007-08-15 14:28:22 +0000583true if Python is currently in the initialized state. More information about
Martin Panterb4ce1fc2015-11-30 03:18:29 +0000584these functions is given in a later chapter. Notice that :c:func:`Py_FinalizeEx`
Georg Brandl116aa622007-08-15 14:28:22 +0000585does *not* free all memory allocated by the Python interpreter, e.g. memory
586allocated by extension modules currently cannot be released.
587
588
589.. _api-debugging:
590
591Debugging Builds
592================
593
594Python can be built with several macros to enable extra checks of the
595interpreter and extension modules. These checks tend to add a large amount of
596overhead to the runtime so they are not enabled by default.
597
598A full list of the various types of debugging builds is in the file
599:file:`Misc/SpecialBuilds.txt` in the Python source distribution. Builds are
600available that support tracing of reference counts, debugging the memory
601allocator, or low-level profiling of the main interpreter loop. Only the most
602frequently-used builds will be described in the remainder of this section.
603
Georg Brandl60203b42010-10-06 10:11:56 +0000604Compiling the interpreter with the :c:macro:`Py_DEBUG` macro defined produces
605what is generally meant by "a debug build" of Python. :c:macro:`Py_DEBUG` is
Éric Araujod2f8cec2011-06-08 05:29:39 +0200606enabled in the Unix build by adding ``--with-pydebug`` to the
607:file:`./configure` command. It is also implied by the presence of the
Georg Brandl60203b42010-10-06 10:11:56 +0000608not-Python-specific :c:macro:`_DEBUG` macro. When :c:macro:`Py_DEBUG` is enabled
Georg Brandl116aa622007-08-15 14:28:22 +0000609in the Unix build, compiler optimization is disabled.
610
611In addition to the reference count debugging described below, the following
612extra checks are performed:
613
614* Extra checks are added to the object allocator.
615
616* Extra checks are added to the parser and compiler.
617
618* Downcasts from wide types to narrow types are checked for loss of information.
619
620* A number of assertions are added to the dictionary and set implementations.
621 In addition, the set object acquires a :meth:`test_c_api` method.
622
623* Sanity checks of the input arguments are added to frame creation.
624
Mark Dickinsonbf5c6a92009-01-17 10:21:23 +0000625* The storage for ints is initialized with a known invalid pattern to catch
Georg Brandl116aa622007-08-15 14:28:22 +0000626 reference to uninitialized digits.
627
628* Low-level tracing and extra exception checking are added to the runtime
629 virtual machine.
630
631* Extra checks are added to the memory arena implementation.
632
633* Extra debugging is added to the thread module.
634
635There may be additional checks not mentioned here.
636
Georg Brandl60203b42010-10-06 10:11:56 +0000637Defining :c:macro:`Py_TRACE_REFS` enables reference tracing. When defined, a
Georg Brandl116aa622007-08-15 14:28:22 +0000638circular doubly linked list of active objects is maintained by adding two extra
Georg Brandl60203b42010-10-06 10:11:56 +0000639fields to every :c:type:`PyObject`. Total allocations are tracked as well. Upon
Georg Brandl116aa622007-08-15 14:28:22 +0000640exit, all existing references are printed. (In interactive mode this happens
Georg Brandl60203b42010-10-06 10:11:56 +0000641after every statement run by the interpreter.) Implied by :c:macro:`Py_DEBUG`.
Georg Brandl116aa622007-08-15 14:28:22 +0000642
643Please refer to :file:`Misc/SpecialBuilds.txt` in the Python source distribution
644for more detailed information.
645