blob: 3004174bcdca408b8221afb80254af2edf3ff170 [file] [log] [blame]
Georg Brandlc1660762010-12-28 11:38:12 +00001This file describes some special Python build types enabled via compile-time
2preprocessor defines.
Tim Peters6045d482002-07-09 18:35:34 +00003
Georg Brandlc1660762010-12-28 11:38:12 +00004IMPORTANT: if you want to build a debug-enabled Python, it is recommended that
5you use ``./configure --with-pydebug``, rather than the options listed here.
Antoine Pitrou3cbb24d2010-12-15 15:47:34 +00006
7However, if you wish to define some of these options individually, it is best
8to define them in the EXTRA_CFLAGS make variable;
Thomas Wouters49fd7fa2006-04-21 10:40:58 +00009``make EXTRA_CFLAGS="-DPy_REF_DEBUG"``.
Brett Cannona2675632005-04-19 20:28:09 +000010
Tim Peters6045d482002-07-09 18:35:34 +000011
Georg Brandlc1660762010-12-28 11:38:12 +000012Py_REF_DEBUG
13------------
14
15Turn on aggregate reference counting. This arranges that extern _Py_RefTotal
Benjamin Petersonff57aef2014-03-30 17:16:09 -040016hold a count of all references, the sum of ob_refcnt across all objects.
17Passing ``-X showrefcount`` on the command line causes the interactive
18interpreter to print the reference count total as well the number of memory
19blocks allocated after each statement:
Tim Peters6045d482002-07-09 18:35:34 +000020
21 >>> 23
22 23
Benjamin Petersonff57aef2014-03-30 17:16:09 -040023 [8288 refs, 14332 blocks]
Tim Peters6045d482002-07-09 18:35:34 +000024 >>>
25
26Note that if this count increases when you're not storing away new objects,
Georg Brandlc1660762010-12-28 11:38:12 +000027there's probably a leak. Remember, though, that in interactive mode the special
28name "_" holds a reference to the last result displayed!
Tim Peters6045d482002-07-09 18:35:34 +000029
Georg Brandlc1660762010-12-28 11:38:12 +000030Py_REF_DEBUG also checks after every decref to verify that the refcount hasn't
31gone negative, and causes an immediate fatal error if it has.
Tim Peters6045d482002-07-09 18:35:34 +000032
33Special gimmicks:
34
35sys.gettotalrefcount()
36 Return current total of all refcounts.
Tim Peters6045d482002-07-09 18:35:34 +000037
Georg Brandlc1660762010-12-28 11:38:12 +000038
39Py_TRACE_REFS
40-------------
41
42Turn on heavy reference debugging. This is major surgery. Every PyObject grows
43two more pointers, to maintain a doubly-linked list of all live heap-allocated
44objects. Most built-in type objects are not in this list, as they're statically
45allocated. Starting in Python 2.3, if COUNT_ALLOCS (see below) is also defined,
46a static type object T does appear in this list if at least one object of type T
47has been created.
Tim Peters78be7992003-03-23 02:51:01 +000048
49Note that because the fundamental PyObject layout changes, Python modules
Georg Brandlc1660762010-12-28 11:38:12 +000050compiled with Py_TRACE_REFS are incompatible with modules compiled without it.
Tim Peters6045d482002-07-09 18:35:34 +000051
52Py_TRACE_REFS implies Py_REF_DEBUG.
53
54Special gimmicks:
55
56sys.getobjects(max[, type])
Georg Brandlc1660762010-12-28 11:38:12 +000057 Return list of the (no more than) max most-recently allocated objects, most
58 recently allocated first in the list, least-recently allocated last in the
59 list. max=0 means no limit on list length. If an optional type object is
60 passed, the list is also restricted to objects of that type. The return
61 list itself, and some temp objects created just to call sys.getobjects(),
62 are excluded from the return list. Note that the list returned is just
63 another object, though, so may appear in the return list the next time you
64 call getobjects(); note that every object in the list is kept alive too,
65 simply by virtue of being in the list.
Tim Peters6045d482002-07-09 18:35:34 +000066
Georg Brandlc1660762010-12-28 11:38:12 +000067envvar PYTHONDUMPREFS
68 If this envvar exists, Py_Finalize() arranges to print a list of all
69 still-live heap objects. This is printed twice, in different formats,
70 before and after Py_Finalize has cleaned up everything it can clean up. The
71 first output block produces the repr() of each object so is more
72 informative; however, a lot of stuff destined to die is still alive then.
73 The second output block is much harder to work with (repr() can't be invoked
74 anymore -- the interpreter has been torn down too far), but doesn't list any
75 objects that will die. The tool script combinerefs.py can be run over this
76 to combine the info from both output blocks. The second output block, and
Tim Peters21d7d4d2003-04-18 00:45:59 +000077 combinerefs.py, were new in Python 2.3b1.
Georg Brandlc1660762010-12-28 11:38:12 +000078
79
80PYMALLOC_DEBUG
81--------------
Tim Peters6045d482002-07-09 18:35:34 +000082
Tim Peters889f61d2002-07-10 19:29:49 +000083When pymalloc is enabled (WITH_PYMALLOC is defined), calls to the PyObject_
Georg Brandlc1660762010-12-28 11:38:12 +000084memory routines are handled by Python's own small-object allocator, while calls
85to the PyMem_ memory routines are directed to the system malloc/ realloc/free.
86If PYMALLOC_DEBUG is also defined, calls to both PyObject_ and PyMem_ memory
87routines are directed to a special debugging mode of Python's small-object
88allocator.
Tim Peters889f61d2002-07-10 19:29:49 +000089
Georg Brandlc1660762010-12-28 11:38:12 +000090This mode fills dynamically allocated memory blocks with special, recognizable
91bit patterns, and adds debugging info on each end of dynamically allocated
92memory blocks. The special bit patterns are:
Tim Peters889f61d2002-07-10 19:29:49 +000093
94#define CLEANBYTE 0xCB /* clean (newly allocated) memory */
95#define DEADBYTE 0xDB /* dead (newly freed) memory */
Thomas Wouters89f507f2006-12-13 04:49:30 +000096#define FORBIDDENBYTE 0xFB /* forbidden -- untouchable bytes */
Tim Peters889f61d2002-07-10 19:29:49 +000097
98Strings of these bytes are unlikely to be valid addresses, floats, or 7-bit
99ASCII strings.
100
Georg Brandlc1660762010-12-28 11:38:12 +0000101Let S = sizeof(size_t). 2*S bytes are added at each end of each block of N bytes
102requested. The memory layout is like so, where p represents the address
103returned by a malloc-like or realloc-like function (p[i:j] means the slice of
104bytes from *(p+i) inclusive up to *(p+j) exclusive; note that the treatment of
105negative indices differs from a Python slice):
Tim Peters889f61d2002-07-10 19:29:49 +0000106
Thomas Wouters73e5a5b2006-06-08 15:35:45 +0000107p[-2*S:-S]
Georg Brandlc1660762010-12-28 11:38:12 +0000108 Number of bytes originally asked for. This is a size_t, big-endian (easier
109 to read in a memory dump).
Tim Petersdf099f52013-09-19 21:06:37 -0500110p[-S]
111 API ID. See PEP 445. This is a character, but seems undocumented.
112p[-S+1:0]
Tim Peters889f61d2002-07-10 19:29:49 +0000113 Copies of FORBIDDENBYTE. Used to catch under- writes and reads.
114p[0:N]
Tim Peters62fc52e2002-07-11 00:23:58 +0000115 The requested memory, filled with copies of CLEANBYTE, used to catch
Georg Brandlc1660762010-12-28 11:38:12 +0000116 reference to uninitialized memory. When a realloc-like function is called
117 requesting a larger memory block, the new excess bytes are also filled with
118 CLEANBYTE. When a free-like function is called, these are overwritten with
119 DEADBYTE, to catch reference to freed memory. When a realloc- like function
120 is called requesting a smaller memory block, the excess old bytes are also
121 filled with DEADBYTE.
Thomas Wouters73e5a5b2006-06-08 15:35:45 +0000122p[N:N+S]
Tim Peters889f61d2002-07-10 19:29:49 +0000123 Copies of FORBIDDENBYTE. Used to catch over- writes and reads.
Thomas Wouters73e5a5b2006-06-08 15:35:45 +0000124p[N+S:N+2*S]
Tim Peters889f61d2002-07-10 19:29:49 +0000125 A serial number, incremented by 1 on each call to a malloc-like or
Georg Brandlc1660762010-12-28 11:38:12 +0000126 realloc-like function. Big-endian size_t. If "bad memory" is detected
127 later, the serial number gives an excellent way to set a breakpoint on the
128 next run, to capture the instant at which this block was passed out. The
129 static function bumpserialno() in obmalloc.c is the only place the serial
130 number is incremented, and exists so you can set such a breakpoint easily.
Tim Peters889f61d2002-07-10 19:29:49 +0000131
Georg Brandlc1660762010-12-28 11:38:12 +0000132A realloc-like or free-like function first checks that the FORBIDDENBYTEs at
133each end are intact. If they've been altered, diagnostic output is written to
134stderr, and the program is aborted via Py_FatalError(). The other main failure
135mode is provoking a memory error when a program reads up one of the special bit
136patterns and tries to use it as an address. If you get in a debugger then and
137look at the object, you're likely to see that it's entirely filled with 0xDB
138(meaning freed memory is getting used) or 0xCB (meaning uninitialized memory is
139getting used).
Tim Peters889f61d2002-07-10 19:29:49 +0000140
141Note that PYMALLOC_DEBUG requires WITH_PYMALLOC.
142
Tim Peters6045d482002-07-09 18:35:34 +0000143Special gimmicks:
144
Georg Brandlc1660762010-12-28 11:38:12 +0000145envvar PYTHONMALLOCSTATS
146 If this envvar exists, a report of pymalloc summary statistics is printed to
147 stderr whenever a new arena is allocated, and also by Py_Finalize().
Thomas Wouters73e5a5b2006-06-08 15:35:45 +0000148
149Changed in 2.5: The number of extra bytes allocated is 4*sizeof(size_t).
150Before it was 16 on all boxes, reflecting that Python couldn't make use of
151allocations >= 2**32 bytes even on 64-bit boxes before 2.5.
Georg Brandlc1660762010-12-28 11:38:12 +0000152
153
154Py_DEBUG
155--------
Tim Peters6045d482002-07-09 18:35:34 +0000156
157This is what is generally meant by "a debug build" of Python.
158
Georg Brandlc1660762010-12-28 11:38:12 +0000159Py_DEBUG implies LLTRACE, Py_REF_DEBUG, Py_TRACE_REFS, and PYMALLOC_DEBUG (if
160WITH_PYMALLOC is enabled). In addition, C assert()s are enabled (via the C way:
161by not defining NDEBUG), and some routines do additional sanity checks inside
162"#ifdef Py_DEBUG" blocks.
163
164
165COUNT_ALLOCS
166------------
Tim Peters48ba6492002-07-09 19:24:54 +0000167
168Each type object grows three new members:
169
170 /* Number of times an object of this type was allocated. */
Guido van Rossum0c088642002-07-11 01:04:32 +0000171 int tp_allocs;
Tim Peters48ba6492002-07-09 19:24:54 +0000172
173 /* Number of times an object of this type was deallocated. */
Guido van Rossum0c088642002-07-11 01:04:32 +0000174 int tp_frees;
Tim Peters48ba6492002-07-09 19:24:54 +0000175
Guido van Rossum0c088642002-07-11 01:04:32 +0000176 /* Highwater mark: the maximum value of tp_allocs - tp_frees so
177 * far; or, IOW, the largest number of objects of this type alive at
178 * the same time.
179 */
180 int tp_maxalloc;
Tim Peters48ba6492002-07-09 19:24:54 +0000181
Georg Brandlc1660762010-12-28 11:38:12 +0000182Allocation and deallocation code keeps these counts up to date. Py_Finalize()
183displays a summary of the info returned by sys.getcounts() (see below), along
184with assorted other special allocation counts (like the number of tuple
185allocations satisfied by a tuple free-list, the number of 1-character strings
186allocated, etc).
Tim Peters48ba6492002-07-09 19:24:54 +0000187
188Before Python 2.2, type objects were immortal, and the COUNT_ALLOCS
Georg Brandlc1660762010-12-28 11:38:12 +0000189implementation relies on that. As of Python 2.2, heap-allocated type/ class
190objects can go away. COUNT_ALLOCS can blow up in 2.2 and 2.2.1 because of this;
191this was fixed in 2.2.2. Use of COUNT_ALLOCS makes all heap-allocated type
192objects immortal, except for those for which no object of that type is ever
193allocated.
Tim Peters48ba6492002-07-09 19:24:54 +0000194
Tim Peters78be7992003-03-23 02:51:01 +0000195Starting with Python 2.3, If Py_TRACE_REFS is also defined, COUNT_ALLOCS
Georg Brandlc1660762010-12-28 11:38:12 +0000196arranges to ensure that the type object for each allocated object appears in the
197doubly-linked list of all objects maintained by Py_TRACE_REFS.
Tim Peters78be7992003-03-23 02:51:01 +0000198
Tim Peters48ba6492002-07-09 19:24:54 +0000199Special gimmicks:
200
201sys.getcounts()
Georg Brandlc1660762010-12-28 11:38:12 +0000202 Return a list of 4-tuples, one entry for each type object for which at least
203 one object of that type was allocated. Each tuple is of the form:
Tim Peters48ba6492002-07-09 19:24:54 +0000204
205 (tp_name, tp_allocs, tp_frees, tp_maxalloc)
206
Georg Brandlc1660762010-12-28 11:38:12 +0000207 Each distinct type object gets a distinct entry in this list, even if two or
208 more type objects have the same tp_name (in which case there's no way to
209 distinguish them by looking at this list). The list is ordered by time of
210 first object allocation: the type object for which the first allocation of
211 an object of that type occurred most recently is at the front of the list.
212
213
214LLTRACE
215-------
Michael W. Hudsona6255232002-07-30 09:49:29 +0000216
Michael W. Hudson46e6d922005-01-18 15:53:59 +0000217Compile in support for Low Level TRACE-ing of the main interpreter loop.
Michael W. Hudsona6255232002-07-30 09:49:29 +0000218
Benjamin Peterson6e820c02015-03-01 20:59:22 -0500219When this preprocessor symbol is defined, before PyEval_EvalFrame executes a
220frame's code it checks the frame's global namespace for a variable
221"__lltrace__". If such a variable is found, mounds of information about what
222the interpreter is doing are sprayed to stdout, such as every opcode and opcode
223argument and values pushed onto and popped off the value stack.
Michael W. Hudsona6255232002-07-30 09:49:29 +0000224
225Not useful very often, but very useful when needed.
Jeremy Hylton985eba52003-02-05 23:13:00 +0000226
Georg Brandlc1660762010-12-28 11:38:12 +0000227
228CALL_PROFILE
229------------
Jeremy Hylton985eba52003-02-05 23:13:00 +0000230
231Count the number of function calls executed.
232
Georg Brandlc1660762010-12-28 11:38:12 +0000233When this symbol is defined, the ceval mainloop and helper functions count the
234number of function calls made. It keeps detailed statistics about what kind of
235object was called and whether the call hit any of the special fast paths in the
236code.
Michael W. Hudson800ba232004-08-12 18:19:17 +0000237
Michael W. Hudson800ba232004-08-12 18:19:17 +0000238
Georg Brandlc1660762010-12-28 11:38:12 +0000239WITH_TSC
240--------
241
242Super-lowlevel profiling of the interpreter. When enabled, the sys module grows
243a new function:
Michael W. Hudson800ba232004-08-12 18:19:17 +0000244
245settscdump(bool)
Georg Brandlc1660762010-12-28 11:38:12 +0000246 If true, tell the Python interpreter to dump VM measurements to stderr. If
247 false, turn off dump. The measurements are based on the processor's
248 time-stamp counter.
Michael W. Hudson800ba232004-08-12 18:19:17 +0000249
Georg Brandlc1660762010-12-28 11:38:12 +0000250This build option requires a small amount of platform specific code. Currently
251this code is present for linux/x86 and any PowerPC platform that uses GCC
252(i.e. OS X and linux/ppc).
Michael W. Hudson800ba232004-08-12 18:19:17 +0000253
Georg Brandlc1660762010-12-28 11:38:12 +0000254On the PowerPC the rate at which the time base register is incremented is not
255defined by the architecture specification, so you'll need to find the manual for
256your specific processor. For the 750CX, 750CXe and 750FX (all sold as the G3)
257we find:
Michael W. Hudson800ba232004-08-12 18:19:17 +0000258
Georg Brandlc1660762010-12-28 11:38:12 +0000259 The time base counter is clocked at a frequency that is one-fourth that of
260 the bus clock.
Michael W. Hudson800ba232004-08-12 18:19:17 +0000261
262This build is enabled by the --with-tsc flag to configure.