Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 1 | This file describes some special Python build types enabled via compile-time |
| 2 | preprocessor defines. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 3 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 4 | IMPORTANT: if you want to build a debug-enabled Python, it is recommended that |
| 5 | you use ``./configure --with-pydebug``, rather than the options listed here. |
| 6 | |
| 7 | However, if you wish to define some of these options individually, it is best |
| 8 | to define them in the EXTRA_CFLAGS make variable; |
Thomas Wouters | 49fd7fa | 2006-04-21 10:40:58 +0000 | [diff] [blame] | 9 | ``make EXTRA_CFLAGS="-DPy_REF_DEBUG"``. |
Brett Cannon | a267563 | 2005-04-19 20:28:09 +0000 | [diff] [blame] | 10 | |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 11 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 12 | Py_REF_DEBUG |
| 13 | ------------ |
| 14 | |
| 15 | Turn on aggregate reference counting. This arranges that extern _Py_RefTotal |
| 16 | hold a count of all references, the sum of ob_refcnt across all objects. In a |
| 17 | debug-mode build, this is where the "8288" comes from in |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 18 | |
| 19 | >>> 23 |
| 20 | 23 |
| 21 | [8288 refs] |
| 22 | >>> |
| 23 | |
| 24 | Note that if this count increases when you're not storing away new objects, |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 25 | there's probably a leak. Remember, though, that in interactive mode the special |
| 26 | name "_" holds a reference to the last result displayed! |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 27 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 28 | Py_REF_DEBUG also checks after every decref to verify that the refcount hasn't |
| 29 | gone negative, and causes an immediate fatal error if it has. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 30 | |
| 31 | Special gimmicks: |
| 32 | |
| 33 | sys.gettotalrefcount() |
| 34 | Return current total of all refcounts. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 35 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 36 | |
| 37 | Py_TRACE_REFS |
| 38 | ------------- |
| 39 | |
| 40 | Turn on heavy reference debugging. This is major surgery. Every PyObject grows |
| 41 | two more pointers, to maintain a doubly-linked list of all live heap-allocated |
| 42 | objects. Most built-in type objects are not in this list, as they're statically |
| 43 | allocated. Starting in Python 2.3, if COUNT_ALLOCS (see below) is also defined, |
| 44 | a static type object T does appear in this list if at least one object of type T |
| 45 | has been created. |
Tim Peters | 78be799 | 2003-03-23 02:51:01 +0000 | [diff] [blame] | 46 | |
| 47 | Note that because the fundamental PyObject layout changes, Python modules |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 48 | compiled with Py_TRACE_REFS are incompatible with modules compiled without it. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 49 | |
| 50 | Py_TRACE_REFS implies Py_REF_DEBUG. |
| 51 | |
| 52 | Special gimmicks: |
| 53 | |
| 54 | sys.getobjects(max[, type]) |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 55 | Return list of the (no more than) max most-recently allocated objects, most |
| 56 | recently allocated first in the list, least-recently allocated last in the |
| 57 | list. max=0 means no limit on list length. If an optional type object is |
| 58 | passed, the list is also restricted to objects of that type. The return |
| 59 | list itself, and some temp objects created just to call sys.getobjects(), |
| 60 | are excluded from the return list. Note that the list returned is just |
| 61 | another object, though, so may appear in the return list the next time you |
| 62 | call getobjects(); note that every object in the list is kept alive too, |
| 63 | simply by virtue of being in the list. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 64 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 65 | envvar PYTHONDUMPREFS |
| 66 | If this envvar exists, Py_Finalize() arranges to print a list of all |
| 67 | still-live heap objects. This is printed twice, in different formats, |
| 68 | before and after Py_Finalize has cleaned up everything it can clean up. The |
| 69 | first output block produces the repr() of each object so is more |
| 70 | informative; however, a lot of stuff destined to die is still alive then. |
| 71 | The second output block is much harder to work with (repr() can't be invoked |
| 72 | anymore -- the interpreter has been torn down too far), but doesn't list any |
| 73 | objects that will die. The tool script combinerefs.py can be run over this |
| 74 | to combine the info from both output blocks. The second output block, and |
Tim Peters | 21d7d4d | 2003-04-18 00:45:59 +0000 | [diff] [blame] | 75 | combinerefs.py, were new in Python 2.3b1. |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 76 | |
| 77 | |
| 78 | PYMALLOC_DEBUG |
| 79 | -------------- |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 80 | |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 81 | When pymalloc is enabled (WITH_PYMALLOC is defined), calls to the PyObject_ |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 82 | memory routines are handled by Python's own small-object allocator, while calls |
| 83 | to the PyMem_ memory routines are directed to the system malloc/ realloc/free. |
| 84 | If PYMALLOC_DEBUG is also defined, calls to both PyObject_ and PyMem_ memory |
| 85 | routines are directed to a special debugging mode of Python's small-object |
| 86 | allocator. |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 87 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 88 | This mode fills dynamically allocated memory blocks with special, recognizable |
| 89 | bit patterns, and adds debugging info on each end of dynamically allocated |
| 90 | memory blocks. The special bit patterns are: |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 91 | |
| 92 | #define CLEANBYTE 0xCB /* clean (newly allocated) memory */ |
| 93 | #define DEADBYTE 0xDB /* dead (newly freed) memory */ |
Thomas Wouters | 89f507f | 2006-12-13 04:49:30 +0000 | [diff] [blame] | 94 | #define FORBIDDENBYTE 0xFB /* forbidden -- untouchable bytes */ |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 95 | |
| 96 | Strings of these bytes are unlikely to be valid addresses, floats, or 7-bit |
| 97 | ASCII strings. |
| 98 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 99 | Let S = sizeof(size_t). 2*S bytes are added at each end of each block of N bytes |
| 100 | requested. The memory layout is like so, where p represents the address |
| 101 | returned by a malloc-like or realloc-like function (p[i:j] means the slice of |
| 102 | bytes from *(p+i) inclusive up to *(p+j) exclusive; note that the treatment of |
| 103 | negative indices differs from a Python slice): |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 104 | |
Thomas Wouters | 73e5a5b | 2006-06-08 15:35:45 +0000 | [diff] [blame] | 105 | p[-2*S:-S] |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 106 | Number of bytes originally asked for. This is a size_t, big-endian (easier |
| 107 | to read in a memory dump). |
Thomas Wouters | 73e5a5b | 2006-06-08 15:35:45 +0000 | [diff] [blame] | 108 | p[-S:0] |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 109 | Copies of FORBIDDENBYTE. Used to catch under- writes and reads. |
| 110 | p[0:N] |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 111 | The requested memory, filled with copies of CLEANBYTE, used to catch |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 112 | reference to uninitialized memory. When a realloc-like function is called |
| 113 | requesting a larger memory block, the new excess bytes are also filled with |
| 114 | CLEANBYTE. When a free-like function is called, these are overwritten with |
| 115 | DEADBYTE, to catch reference to freed memory. When a realloc- like function |
| 116 | is called requesting a smaller memory block, the excess old bytes are also |
| 117 | filled with DEADBYTE. |
Thomas Wouters | 73e5a5b | 2006-06-08 15:35:45 +0000 | [diff] [blame] | 118 | p[N:N+S] |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 119 | Copies of FORBIDDENBYTE. Used to catch over- writes and reads. |
Thomas Wouters | 73e5a5b | 2006-06-08 15:35:45 +0000 | [diff] [blame] | 120 | p[N+S:N+2*S] |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 121 | A serial number, incremented by 1 on each call to a malloc-like or |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 122 | realloc-like function. Big-endian size_t. If "bad memory" is detected |
| 123 | later, the serial number gives an excellent way to set a breakpoint on the |
| 124 | next run, to capture the instant at which this block was passed out. The |
| 125 | static function bumpserialno() in obmalloc.c is the only place the serial |
| 126 | number is incremented, and exists so you can set such a breakpoint easily. |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 127 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 128 | A realloc-like or free-like function first checks that the FORBIDDENBYTEs at |
| 129 | each end are intact. If they've been altered, diagnostic output is written to |
| 130 | stderr, and the program is aborted via Py_FatalError(). The other main failure |
| 131 | mode is provoking a memory error when a program reads up one of the special bit |
| 132 | patterns and tries to use it as an address. If you get in a debugger then and |
| 133 | look at the object, you're likely to see that it's entirely filled with 0xDB |
| 134 | (meaning freed memory is getting used) or 0xCB (meaning uninitialized memory is |
| 135 | getting used). |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 136 | |
| 137 | Note that PYMALLOC_DEBUG requires WITH_PYMALLOC. |
| 138 | |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 139 | Special gimmicks: |
| 140 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 141 | envvar PYTHONMALLOCSTATS |
| 142 | If this envvar exists, a report of pymalloc summary statistics is printed to |
| 143 | stderr whenever a new arena is allocated, and also by Py_Finalize(). |
Thomas Wouters | 73e5a5b | 2006-06-08 15:35:45 +0000 | [diff] [blame] | 144 | |
| 145 | Changed in 2.5: The number of extra bytes allocated is 4*sizeof(size_t). |
| 146 | Before it was 16 on all boxes, reflecting that Python couldn't make use of |
| 147 | allocations >= 2**32 bytes even on 64-bit boxes before 2.5. |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 148 | |
| 149 | |
| 150 | Py_DEBUG |
| 151 | -------- |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 152 | |
| 153 | This is what is generally meant by "a debug build" of Python. |
| 154 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 155 | Py_DEBUG implies LLTRACE, Py_REF_DEBUG, Py_TRACE_REFS, and PYMALLOC_DEBUG (if |
| 156 | WITH_PYMALLOC is enabled). In addition, C assert()s are enabled (via the C way: |
| 157 | by not defining NDEBUG), and some routines do additional sanity checks inside |
| 158 | "#ifdef Py_DEBUG" blocks. |
| 159 | |
| 160 | |
| 161 | COUNT_ALLOCS |
| 162 | ------------ |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 163 | |
| 164 | Each type object grows three new members: |
| 165 | |
| 166 | /* Number of times an object of this type was allocated. */ |
Guido van Rossum | 0c08864 | 2002-07-11 01:04:32 +0000 | [diff] [blame] | 167 | int tp_allocs; |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 168 | |
| 169 | /* Number of times an object of this type was deallocated. */ |
Guido van Rossum | 0c08864 | 2002-07-11 01:04:32 +0000 | [diff] [blame] | 170 | int tp_frees; |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 171 | |
Guido van Rossum | 0c08864 | 2002-07-11 01:04:32 +0000 | [diff] [blame] | 172 | /* Highwater mark: the maximum value of tp_allocs - tp_frees so |
| 173 | * far; or, IOW, the largest number of objects of this type alive at |
| 174 | * the same time. |
| 175 | */ |
| 176 | int tp_maxalloc; |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 177 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 178 | Allocation and deallocation code keeps these counts up to date. Py_Finalize() |
| 179 | displays a summary of the info returned by sys.getcounts() (see below), along |
| 180 | with assorted other special allocation counts (like the number of tuple |
| 181 | allocations satisfied by a tuple free-list, the number of 1-character strings |
| 182 | allocated, etc). |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 183 | |
| 184 | Before Python 2.2, type objects were immortal, and the COUNT_ALLOCS |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 185 | implementation relies on that. As of Python 2.2, heap-allocated type/ class |
| 186 | objects can go away. COUNT_ALLOCS can blow up in 2.2 and 2.2.1 because of this; |
| 187 | this was fixed in 2.2.2. Use of COUNT_ALLOCS makes all heap-allocated type |
| 188 | objects immortal, except for those for which no object of that type is ever |
| 189 | allocated. |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 190 | |
Tim Peters | 78be799 | 2003-03-23 02:51:01 +0000 | [diff] [blame] | 191 | Starting with Python 2.3, If Py_TRACE_REFS is also defined, COUNT_ALLOCS |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 192 | arranges to ensure that the type object for each allocated object appears in the |
| 193 | doubly-linked list of all objects maintained by Py_TRACE_REFS. |
Tim Peters | 78be799 | 2003-03-23 02:51:01 +0000 | [diff] [blame] | 194 | |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 195 | Special gimmicks: |
| 196 | |
| 197 | sys.getcounts() |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 198 | Return a list of 4-tuples, one entry for each type object for which at least |
| 199 | one object of that type was allocated. Each tuple is of the form: |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 200 | |
| 201 | (tp_name, tp_allocs, tp_frees, tp_maxalloc) |
| 202 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 203 | Each distinct type object gets a distinct entry in this list, even if two or |
| 204 | more type objects have the same tp_name (in which case there's no way to |
| 205 | distinguish them by looking at this list). The list is ordered by time of |
| 206 | first object allocation: the type object for which the first allocation of |
| 207 | an object of that type occurred most recently is at the front of the list. |
| 208 | |
| 209 | |
| 210 | LLTRACE |
| 211 | ------- |
Michael W. Hudson | a625523 | 2002-07-30 09:49:29 +0000 | [diff] [blame] | 212 | |
Michael W. Hudson | 46e6d92 | 2005-01-18 15:53:59 +0000 | [diff] [blame] | 213 | Compile in support for Low Level TRACE-ing of the main interpreter loop. |
Michael W. Hudson | a625523 | 2002-07-30 09:49:29 +0000 | [diff] [blame] | 214 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 215 | When this preprocessor symbol is defined, before PyEval_EvalFrame (eval_frame in |
| 216 | 2.3 and 2.2, eval_code2 before that) executes a frame's code it checks the |
| 217 | frame's global namespace for a variable "__lltrace__". If such a variable is |
| 218 | found, mounds of information about what the interpreter is doing are sprayed to |
| 219 | stdout, such as every opcode and opcode argument and values pushed onto and |
| 220 | popped off the value stack. |
Michael W. Hudson | a625523 | 2002-07-30 09:49:29 +0000 | [diff] [blame] | 221 | |
| 222 | Not useful very often, but very useful when needed. |
Jeremy Hylton | 985eba5 | 2003-02-05 23:13:00 +0000 | [diff] [blame] | 223 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 224 | |
| 225 | CALL_PROFILE |
| 226 | ------------ |
Jeremy Hylton | 985eba5 | 2003-02-05 23:13:00 +0000 | [diff] [blame] | 227 | |
| 228 | Count the number of function calls executed. |
| 229 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 230 | When this symbol is defined, the ceval mainloop and helper functions count the |
| 231 | number of function calls made. It keeps detailed statistics about what kind of |
| 232 | object was called and whether the call hit any of the special fast paths in the |
| 233 | code. |
Michael W. Hudson | 800ba23 | 2004-08-12 18:19:17 +0000 | [diff] [blame] | 234 | |
Michael W. Hudson | 800ba23 | 2004-08-12 18:19:17 +0000 | [diff] [blame] | 235 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 236 | WITH_TSC |
| 237 | -------- |
| 238 | |
| 239 | Super-lowlevel profiling of the interpreter. When enabled, the sys module grows |
| 240 | a new function: |
Michael W. Hudson | 800ba23 | 2004-08-12 18:19:17 +0000 | [diff] [blame] | 241 | |
| 242 | settscdump(bool) |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 243 | If true, tell the Python interpreter to dump VM measurements to stderr. If |
| 244 | false, turn off dump. The measurements are based on the processor's |
| 245 | time-stamp counter. |
Michael W. Hudson | 800ba23 | 2004-08-12 18:19:17 +0000 | [diff] [blame] | 246 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 247 | This build option requires a small amount of platform specific code. Currently |
| 248 | this code is present for linux/x86 and any PowerPC platform that uses GCC |
| 249 | (i.e. OS X and linux/ppc). |
Michael W. Hudson | 800ba23 | 2004-08-12 18:19:17 +0000 | [diff] [blame] | 250 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 251 | On the PowerPC the rate at which the time base register is incremented is not |
| 252 | defined by the architecture specification, so you'll need to find the manual for |
| 253 | your specific processor. For the 750CX, 750CXe and 750FX (all sold as the G3) |
| 254 | we find: |
Michael W. Hudson | 800ba23 | 2004-08-12 18:19:17 +0000 | [diff] [blame] | 255 | |
Georg Brandl | d98934c | 2011-02-25 10:03:34 +0000 | [diff] [blame] | 256 | The time base counter is clocked at a frequency that is one-fourth that of |
| 257 | the bus clock. |
Michael W. Hudson | 800ba23 | 2004-08-12 18:19:17 +0000 | [diff] [blame] | 258 | |
| 259 | This build is enabled by the --with-tsc flag to configure. |