blob: 952ca42d99654d5c64a51d234f6c3df3f75d2306 [file] [log] [blame]
Tim Peters6045d482002-07-09 18:35:34 +00001This file describes some special Python build types enabled via
2compile-time preprocessor defines.
3
Martin v. Löwisd18d5a32006-04-16 18:55:07 +00004It is best to define these options in the EXTRA_CFLAGS make variable;
5``make EXTRA_CFLAGS="-DPy_REF_DEBUG"``.
Brett Cannona2675632005-04-19 20:28:09 +00006
Tim Peters6045d482002-07-09 18:35:34 +00007---------------------------------------------------------------------------
Tim Peters62fc52e2002-07-11 00:23:58 +00008Py_REF_DEBUG introduced in 1.4
9 named REF_DEBUG before 1.4
Tim Peters6045d482002-07-09 18:35:34 +000010
11Turn on aggregate reference counting. This arranges that extern
12_Py_RefTotal hold a count of all references, the sum of ob_refcnt across
13all objects. In a debug-mode build, this is where the "8288" comes from
14in
15
16 >>> 23
17 23
18 [8288 refs]
19 >>>
20
21Note that if this count increases when you're not storing away new objects,
22there's probably a leak. Remember, though, that in interactive mode the
23special name "_" holds a reference to the last result displayed!
24
25Py_REF_DEBUG also checks after every decref to verify that the refcount
26hasn't gone negative, and causes an immediate fatal error if it has.
27
28Special gimmicks:
29
30sys.gettotalrefcount()
31 Return current total of all refcounts.
32 Available under Py_REF_DEBUG in Python 2.3.
33 Before 2.3, Py_TRACE_REFS was required to enable this function.
34---------------------------------------------------------------------------
Tim Peters62fc52e2002-07-11 00:23:58 +000035Py_TRACE_REFS introduced in 1.4
36 named TRACE_REFS before 1.4
Tim Peters6045d482002-07-09 18:35:34 +000037
38Turn on heavy reference debugging. This is major surgery. Every PyObject
39grows two more pointers, to maintain a doubly-linked list of all live
Tim Peters78be7992003-03-23 02:51:01 +000040heap-allocated objects. Most builtin type objects are not in this list,
41as they're statically allocated. Starting in Python 2.3, if COUNT_ALLOCS
42(see below) is also defined, a static type object T does appear in this
43list if at least one object of type T has been created.
44
45Note that because the fundamental PyObject layout changes, Python modules
46compiled with Py_TRACE_REFS are incompatible with modules compiled without
47it.
Tim Peters6045d482002-07-09 18:35:34 +000048
49Py_TRACE_REFS implies Py_REF_DEBUG.
50
51Special gimmicks:
52
53sys.getobjects(max[, type])
Tim Petersa788f5e2002-07-10 18:47:03 +000054 Return list of the (no more than) max most-recently allocated objects,
55 most recently allocated first in the list, least-recently allocated
56 last in the list. max=0 means no limit on list length.
57 If an optional type object is passed, the list is also restricted to
58 objects of that type.
59 The return list itself, and some temp objects created just to call
60 sys.getobjects(), are excluded from the return list. Note that the
61 list returned is just another object, though, so may appear in the
62 return list the next time you call getobjects(); note that every
63 object in the list is kept alive too, simply by virtue of being in
64 the list.
Tim Peters6045d482002-07-09 18:35:34 +000065
66envar PYTHONDUMPREFS
67 If this envar exists, Py_Finalize() arranges to print a list of
Tim Peters21d7d4d2003-04-18 00:45:59 +000068 all still-live heap objects. This is printed twice, in different
69 formats, before and after Py_Finalize has cleaned up everything it
70 can clean up. The first output block produces the repr() of each
71 object so is more informative; however, a lot of stuff destined to
72 die is still alive then. The second output block is much harder
73 to work with (repr() can't be invoked anymore -- the interpreter
74 has been torn down too far), but doesn't list any objects that will
75 die. The tool script combinerefs.py can be run over this to combine
76 the info from both output blocks. The second output block, and
77 combinerefs.py, were new in Python 2.3b1.
Tim Peters6045d482002-07-09 18:35:34 +000078---------------------------------------------------------------------------
Tim Peters62fc52e2002-07-11 00:23:58 +000079PYMALLOC_DEBUG introduced in 2.3
Tim Peters6045d482002-07-09 18:35:34 +000080
Tim Peters889f61d2002-07-10 19:29:49 +000081When pymalloc is enabled (WITH_PYMALLOC is defined), calls to the PyObject_
82memory routines are handled by Python's own small-object allocator, while
83calls to the PyMem_ memory routines are directed to the system malloc/
84realloc/free. If PYMALLOC_DEBUG is also defined, calls to both PyObject_
85and PyMem_ memory routines are directed to a special debugging mode of
86Python's small-object allocator.
87
88This mode fills dynamically allocated memory blocks with special,
89recognizable bit patterns, and adds debugging info on each end of
90dynamically allocated memory blocks. The special bit patterns are:
91
92#define CLEANBYTE 0xCB /* clean (newly allocated) memory */
93#define DEADBYTE 0xDB /* dead (newly freed) memory */
94#define FORBIDDENBYTE 0xFB /* fordidden -- untouchable bytes */
95
96Strings of these bytes are unlikely to be valid addresses, floats, or 7-bit
97ASCII strings.
98
Tim Peters9ea89d22006-06-04 03:26:02 +000099Let S = sizeof(size_t). 2*S bytes are added at each end of each block of N
100bytes requested. The memory layout is like so, where p represents the
101address returned by a malloc-like or realloc-like function (p[i:j] means
102the slice of bytes from *(p+i) inclusive up to *(p+j) exclusive; note that
103the treatment of negative indices differs from a Python slice):
Tim Peters889f61d2002-07-10 19:29:49 +0000104
Tim Peters9ea89d22006-06-04 03:26:02 +0000105p[-2*S:-S]
106 Number of bytes originally asked for. This is a size_t, big-endian
107 (easier to read in a memory dump).
108p[-S:0]
Tim Peters889f61d2002-07-10 19:29:49 +0000109 Copies of FORBIDDENBYTE. Used to catch under- writes and reads.
110p[0:N]
Tim Peters62fc52e2002-07-11 00:23:58 +0000111 The requested memory, filled with copies of CLEANBYTE, used to catch
112 reference to uninitialized memory.
Tim Peters889f61d2002-07-10 19:29:49 +0000113 When a realloc-like function is called requesting a larger memory
114 block, the new excess bytes are also filled with CLEANBYTE.
115 When a free-like function is called, these are overwritten with
Tim Peters62fc52e2002-07-11 00:23:58 +0000116 DEADBYTE, to catch reference to freed memory. When a realloc-
Tim Peters889f61d2002-07-10 19:29:49 +0000117 like function is called requesting a smaller memory block, the excess
118 old bytes are also filled with DEADBYTE.
Tim Peters9ea89d22006-06-04 03:26:02 +0000119p[N:N+S]
Tim Peters889f61d2002-07-10 19:29:49 +0000120 Copies of FORBIDDENBYTE. Used to catch over- writes and reads.
Tim Peters9ea89d22006-06-04 03:26:02 +0000121p[N+S:N+2*S]
Tim Peters889f61d2002-07-10 19:29:49 +0000122 A serial number, incremented by 1 on each call to a malloc-like or
123 realloc-like function.
Tim Peters9ea89d22006-06-04 03:26:02 +0000124 Big-endian size_t.
Tim Peters889f61d2002-07-10 19:29:49 +0000125 If "bad memory" is detected later, the serial number gives an
126 excellent way to set a breakpoint on the next run, to capture the
Tim Peters20c8a042002-07-11 00:02:52 +0000127 instant at which this block was passed out. The static function
128 bumpserialno() in obmalloc.c is the only place the serial number
129 is incremented, and exists so you can set such a breakpoint easily.
Tim Peters889f61d2002-07-10 19:29:49 +0000130
Tim Peters62fc52e2002-07-11 00:23:58 +0000131A realloc-like or free-like function first checks that the FORBIDDENBYTEs
Tim Peters889f61d2002-07-10 19:29:49 +0000132at each end are intact. If they've been altered, diagnostic output is
Tim Peters62fc52e2002-07-11 00:23:58 +0000133written to stderr, and the program is aborted via Py_FatalError(). The
134other main failure mode is provoking a memory error when a program
135reads up one of the special bit patterns and tries to use it as an address.
136If you get in a debugger then and look at the object, you're likely
137to see that it's entirely filled with 0xDB (meaning freed memory is
138getting used) or 0xCB (meaning uninitialized memory is getting used).
Tim Peters889f61d2002-07-10 19:29:49 +0000139
140Note that PYMALLOC_DEBUG requires WITH_PYMALLOC.
141
Tim Peters6045d482002-07-09 18:35:34 +0000142Special gimmicks:
143
144envar PYTHONMALLOCSTATS
145 If this envar exists, a report of pymalloc summary statistics is
146 printed to stderr whenever a new arena is allocated, and also
147 by Py_Finalize().
Tim Peters9ea89d22006-06-04 03:26:02 +0000148
149Changed in 2.5: The number of extra bytes allocated is 4*sizeof(size_t).
150Before it was 16 on all boxes, reflecting that Python couldn't make use of
151allocations >= 2**32 bytes even on 64-bit boxes before 2.5.
Tim Peters6045d482002-07-09 18:35:34 +0000152---------------------------------------------------------------------------
Tim Peters62fc52e2002-07-11 00:23:58 +0000153Py_DEBUG introduced in 1.5
154 named DEBUG before 1.5
Tim Peters6045d482002-07-09 18:35:34 +0000155
156This is what is generally meant by "a debug build" of Python.
157
Michael W. Hudsona6255232002-07-30 09:49:29 +0000158Py_DEBUG implies LLTRACE, Py_REF_DEBUG, Py_TRACE_REFS, and
159PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled). In addition, C
160assert()s are enabled (via the C way: by not defining NDEBUG), and
161some routines do additional sanity checks inside "#ifdef Py_DEBUG"
162blocks.
Tim Peters6045d482002-07-09 18:35:34 +0000163---------------------------------------------------------------------------
Michael W. Hudson202a4b62002-07-30 15:25:57 +0000164COUNT_ALLOCS introduced in 0.9.9
165 partly broken in 2.2 and 2.2.1
Tim Peters48ba6492002-07-09 19:24:54 +0000166
167Each type object grows three new members:
168
169 /* Number of times an object of this type was allocated. */
Guido van Rossum0c088642002-07-11 01:04:32 +0000170 int tp_allocs;
Tim Peters48ba6492002-07-09 19:24:54 +0000171
172 /* Number of times an object of this type was deallocated. */
Guido van Rossum0c088642002-07-11 01:04:32 +0000173 int tp_frees;
Tim Peters48ba6492002-07-09 19:24:54 +0000174
Guido van Rossum0c088642002-07-11 01:04:32 +0000175 /* Highwater mark: the maximum value of tp_allocs - tp_frees so
176 * far; or, IOW, the largest number of objects of this type alive at
177 * the same time.
178 */
179 int tp_maxalloc;
Tim Peters48ba6492002-07-09 19:24:54 +0000180
181Allocation and deallocation code keeps these counts up to date.
182Py_Finalize() displays a summary of the info returned by sys.getcounts()
183(see below), along with assorted other special allocation counts (like
184the number of tuple allocations satisfied by a tuple free-list, the number
185of 1-character strings allocated, etc).
186
187Before Python 2.2, type objects were immortal, and the COUNT_ALLOCS
188implementation relies on that. As of Python 2.2, heap-allocated type/
189class objects can go away. COUNT_ALLOCS can blow up in 2.2 and 2.2.1
190because of this; this was fixed in 2.2.2. Use of COUNT_ALLOCS makes
191all heap-allocated type objects immortal, except for those for which no
192object of that type is ever allocated.
193
Tim Peters78be7992003-03-23 02:51:01 +0000194Starting with Python 2.3, If Py_TRACE_REFS is also defined, COUNT_ALLOCS
195arranges to ensure that the type object for each allocated object
196appears in the doubly-linked list of all objects maintained by
197Py_TRACE_REFS.
198
Tim Peters48ba6492002-07-09 19:24:54 +0000199Special gimmicks:
200
201sys.getcounts()
202 Return a list of 4-tuples, one entry for each type object for which
203 at least one object of that type was allocated. Each tuple is of
204 the form:
205
206 (tp_name, tp_allocs, tp_frees, tp_maxalloc)
207
Tim Peters44c1a7b2002-07-09 19:27:20 +0000208 Each distinct type object gets a distinct entry in this list, even
Tim Peters48ba6492002-07-09 19:24:54 +0000209 if two or more type objects have the same tp_name (in which case
210 there's no way to distinguish them by looking at this list). The
211 list is ordered by time of first object allocation: the type object
212 for which the first allocation of an object of that type occurred
213 most recently is at the front of the list.
214---------------------------------------------------------------------------
Michael W. Hudson202a4b62002-07-30 15:25:57 +0000215LLTRACE introduced well before 1.0
Michael W. Hudsona6255232002-07-30 09:49:29 +0000216
Michael W. Hudson46e6d922005-01-18 15:53:59 +0000217Compile in support for Low Level TRACE-ing of the main interpreter loop.
Michael W. Hudsona6255232002-07-30 09:49:29 +0000218
Michael W. Hudson46e6d922005-01-18 15:53:59 +0000219When this preprocessor symbol is defined, before PyEval_EvalFrame
220(eval_frame in 2.3 and 2.2, eval_code2 before that) executes a frame's code
221it checks the frame's global namespace for a variable "__lltrace__". If
222such a variable is found, mounds of information about what the interpreter
223is doing are sprayed to stdout, such as every opcode and opcode argument
224and values pushed onto and popped off the value stack.
Michael W. Hudsona6255232002-07-30 09:49:29 +0000225
226Not useful very often, but very useful when needed.
Jeremy Hylton985eba52003-02-05 23:13:00 +0000227
228---------------------------------------------------------------------------
229CALL_PROFILE introduced for Python 2.3
230
231Count the number of function calls executed.
232
233When this symbol is defined, the ceval mainloop and helper functions
234count the number of function calls made. It keeps detailed statistics
235about what kind of object was called and whether the call hit any of
236the special fast paths in the code.
Michael W. Hudson800ba232004-08-12 18:19:17 +0000237
238---------------------------------------------------------------------------
239WITH_TSC introduced for Python 2.4
240
241Super-lowlevel profiling of the interpreter. When enabled, the sys
242module grows a new function:
243
244settscdump(bool)
245 If true, tell the Python interpreter to dump VM measurements to
246 stderr. If false, turn off dump. The measurements are based on the
247 processor's time-stamp counter.
248
249This build option requires a small amount of platform specific code.
250Currently this code is present for linux/x86 and any PowerPC platform
251that uses GCC (i.e. OS X and linux/ppc).
252
253On the PowerPC the rate at which the time base register is incremented
254is not defined by the architecture specification, so you'll need to
Michael W. Hudson46e6d922005-01-18 15:53:59 +0000255find the manual for your specific processor. For the 750CX, 750CXe
256and 750FX (all sold as the G3) we find:
Michael W. Hudson800ba232004-08-12 18:19:17 +0000257
Tim Peters9ea89d22006-06-04 03:26:02 +0000258 The time base counter is clocked at a frequency that is
Michael W. Hudson800ba232004-08-12 18:19:17 +0000259 one-fourth that of the bus clock.
260
261This build is enabled by the --with-tsc flag to configure.