Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 1 | This file describes some special Python build types enabled via |
| 2 | compile-time preprocessor defines. |
| 3 | |
| 4 | --------------------------------------------------------------------------- |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 5 | Py_REF_DEBUG introduced in 1.4 |
| 6 | named REF_DEBUG before 1.4 |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 7 | |
| 8 | Turn on aggregate reference counting. This arranges that extern |
| 9 | _Py_RefTotal hold a count of all references, the sum of ob_refcnt across |
| 10 | all objects. In a debug-mode build, this is where the "8288" comes from |
| 11 | in |
| 12 | |
| 13 | >>> 23 |
| 14 | 23 |
| 15 | [8288 refs] |
| 16 | >>> |
| 17 | |
| 18 | Note that if this count increases when you're not storing away new objects, |
| 19 | there's probably a leak. Remember, though, that in interactive mode the |
| 20 | special name "_" holds a reference to the last result displayed! |
| 21 | |
| 22 | Py_REF_DEBUG also checks after every decref to verify that the refcount |
| 23 | hasn't gone negative, and causes an immediate fatal error if it has. |
| 24 | |
| 25 | Special gimmicks: |
| 26 | |
| 27 | sys.gettotalrefcount() |
| 28 | Return current total of all refcounts. |
| 29 | Available under Py_REF_DEBUG in Python 2.3. |
| 30 | Before 2.3, Py_TRACE_REFS was required to enable this function. |
| 31 | --------------------------------------------------------------------------- |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 32 | Py_TRACE_REFS introduced in 1.4 |
| 33 | named TRACE_REFS before 1.4 |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 34 | |
| 35 | Turn on heavy reference debugging. This is major surgery. Every PyObject |
| 36 | grows two more pointers, to maintain a doubly-linked list of all live |
Tim Peters | 78be799 | 2003-03-23 02:51:01 +0000 | [diff] [blame] | 37 | heap-allocated objects. Most builtin type objects are not in this list, |
| 38 | as they're statically allocated. Starting in Python 2.3, if COUNT_ALLOCS |
| 39 | (see below) is also defined, a static type object T does appear in this |
| 40 | list if at least one object of type T has been created. |
| 41 | |
| 42 | Note that because the fundamental PyObject layout changes, Python modules |
| 43 | compiled with Py_TRACE_REFS are incompatible with modules compiled without |
| 44 | it. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 45 | |
| 46 | Py_TRACE_REFS implies Py_REF_DEBUG. |
| 47 | |
| 48 | Special gimmicks: |
| 49 | |
| 50 | sys.getobjects(max[, type]) |
Tim Peters | a788f5e | 2002-07-10 18:47:03 +0000 | [diff] [blame] | 51 | Return list of the (no more than) max most-recently allocated objects, |
| 52 | most recently allocated first in the list, least-recently allocated |
| 53 | last in the list. max=0 means no limit on list length. |
| 54 | If an optional type object is passed, the list is also restricted to |
| 55 | objects of that type. |
| 56 | The return list itself, and some temp objects created just to call |
| 57 | sys.getobjects(), are excluded from the return list. Note that the |
| 58 | list returned is just another object, though, so may appear in the |
| 59 | return list the next time you call getobjects(); note that every |
| 60 | object in the list is kept alive too, simply by virtue of being in |
| 61 | the list. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 62 | |
| 63 | envar PYTHONDUMPREFS |
| 64 | If this envar exists, Py_Finalize() arranges to print a list of |
Tim Peters | 21d7d4d | 2003-04-18 00:45:59 +0000 | [diff] [blame] | 65 | all still-live heap objects. This is printed twice, in different |
| 66 | formats, before and after Py_Finalize has cleaned up everything it |
| 67 | can clean up. The first output block produces the repr() of each |
| 68 | object so is more informative; however, a lot of stuff destined to |
| 69 | die is still alive then. The second output block is much harder |
| 70 | to work with (repr() can't be invoked anymore -- the interpreter |
| 71 | has been torn down too far), but doesn't list any objects that will |
| 72 | die. The tool script combinerefs.py can be run over this to combine |
| 73 | the info from both output blocks. The second output block, and |
| 74 | combinerefs.py, were new in Python 2.3b1. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 75 | --------------------------------------------------------------------------- |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 76 | PYMALLOC_DEBUG introduced in 2.3 |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 77 | |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 78 | When pymalloc is enabled (WITH_PYMALLOC is defined), calls to the PyObject_ |
| 79 | memory routines are handled by Python's own small-object allocator, while |
| 80 | calls to the PyMem_ memory routines are directed to the system malloc/ |
| 81 | realloc/free. If PYMALLOC_DEBUG is also defined, calls to both PyObject_ |
| 82 | and PyMem_ memory routines are directed to a special debugging mode of |
| 83 | Python's small-object allocator. |
| 84 | |
| 85 | This mode fills dynamically allocated memory blocks with special, |
| 86 | recognizable bit patterns, and adds debugging info on each end of |
| 87 | dynamically allocated memory blocks. The special bit patterns are: |
| 88 | |
| 89 | #define CLEANBYTE 0xCB /* clean (newly allocated) memory */ |
| 90 | #define DEADBYTE 0xDB /* dead (newly freed) memory */ |
| 91 | #define FORBIDDENBYTE 0xFB /* fordidden -- untouchable bytes */ |
| 92 | |
| 93 | Strings of these bytes are unlikely to be valid addresses, floats, or 7-bit |
| 94 | ASCII strings. |
| 95 | |
| 96 | 8 bytes are added at each end of each block of N bytes requested. The |
| 97 | memory layout is like so, where p represents the address returned by a |
Tim Peters | 20c8a04 | 2002-07-11 00:02:52 +0000 | [diff] [blame] | 98 | malloc-like or realloc-like function (p[i:j] means the slice of bytes |
| 99 | from *(p+i) inclusive up to *(p+j) exclusive; note that the treatment |
| 100 | of negative indices differs from a Python slice): |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 101 | |
| 102 | p[-8:-4] |
| 103 | Number of bytes originally asked for. 4-byte unsigned integer, |
| 104 | big-endian (easier to read in a memory dump). |
| 105 | p[-4:0] |
| 106 | Copies of FORBIDDENBYTE. Used to catch under- writes and reads. |
| 107 | p[0:N] |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 108 | The requested memory, filled with copies of CLEANBYTE, used to catch |
| 109 | reference to uninitialized memory. |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 110 | When a realloc-like function is called requesting a larger memory |
| 111 | block, the new excess bytes are also filled with CLEANBYTE. |
| 112 | When a free-like function is called, these are overwritten with |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 113 | DEADBYTE, to catch reference to freed memory. When a realloc- |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 114 | like function is called requesting a smaller memory block, the excess |
| 115 | old bytes are also filled with DEADBYTE. |
| 116 | p[N:N+4] |
| 117 | Copies of FORBIDDENBYTE. Used to catch over- writes and reads. |
| 118 | p[N+4:N+8] |
| 119 | A serial number, incremented by 1 on each call to a malloc-like or |
| 120 | realloc-like function. |
| 121 | 4-byte unsigned integer, big-endian. |
| 122 | If "bad memory" is detected later, the serial number gives an |
| 123 | excellent way to set a breakpoint on the next run, to capture the |
Tim Peters | 20c8a04 | 2002-07-11 00:02:52 +0000 | [diff] [blame] | 124 | instant at which this block was passed out. The static function |
| 125 | bumpserialno() in obmalloc.c is the only place the serial number |
| 126 | is incremented, and exists so you can set such a breakpoint easily. |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 127 | |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 128 | A realloc-like or free-like function first checks that the FORBIDDENBYTEs |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 129 | at each end are intact. If they've been altered, diagnostic output is |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 130 | written to stderr, and the program is aborted via Py_FatalError(). The |
| 131 | other main failure mode is provoking a memory error when a program |
| 132 | reads up one of the special bit patterns and tries to use it as an address. |
| 133 | If you get in a debugger then and look at the object, you're likely |
| 134 | to see that it's entirely filled with 0xDB (meaning freed memory is |
| 135 | getting used) or 0xCB (meaning uninitialized memory is getting used). |
Tim Peters | 889f61d | 2002-07-10 19:29:49 +0000 | [diff] [blame] | 136 | |
| 137 | Note that PYMALLOC_DEBUG requires WITH_PYMALLOC. |
| 138 | |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 139 | Special gimmicks: |
| 140 | |
| 141 | envar PYTHONMALLOCSTATS |
| 142 | If this envar exists, a report of pymalloc summary statistics is |
| 143 | printed to stderr whenever a new arena is allocated, and also |
| 144 | by Py_Finalize(). |
| 145 | --------------------------------------------------------------------------- |
Tim Peters | 62fc52e | 2002-07-11 00:23:58 +0000 | [diff] [blame] | 146 | Py_DEBUG introduced in 1.5 |
| 147 | named DEBUG before 1.5 |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 148 | |
| 149 | This is what is generally meant by "a debug build" of Python. |
| 150 | |
Michael W. Hudson | a625523 | 2002-07-30 09:49:29 +0000 | [diff] [blame] | 151 | Py_DEBUG implies LLTRACE, Py_REF_DEBUG, Py_TRACE_REFS, and |
| 152 | PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled). In addition, C |
| 153 | assert()s are enabled (via the C way: by not defining NDEBUG), and |
| 154 | some routines do additional sanity checks inside "#ifdef Py_DEBUG" |
| 155 | blocks. |
Tim Peters | 6045d48 | 2002-07-09 18:35:34 +0000 | [diff] [blame] | 156 | --------------------------------------------------------------------------- |
Michael W. Hudson | 202a4b6 | 2002-07-30 15:25:57 +0000 | [diff] [blame] | 157 | COUNT_ALLOCS introduced in 0.9.9 |
| 158 | partly broken in 2.2 and 2.2.1 |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 159 | |
| 160 | Each type object grows three new members: |
| 161 | |
| 162 | /* Number of times an object of this type was allocated. */ |
Guido van Rossum | 0c08864 | 2002-07-11 01:04:32 +0000 | [diff] [blame] | 163 | int tp_allocs; |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 164 | |
| 165 | /* Number of times an object of this type was deallocated. */ |
Guido van Rossum | 0c08864 | 2002-07-11 01:04:32 +0000 | [diff] [blame] | 166 | int tp_frees; |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 167 | |
Guido van Rossum | 0c08864 | 2002-07-11 01:04:32 +0000 | [diff] [blame] | 168 | /* Highwater mark: the maximum value of tp_allocs - tp_frees so |
| 169 | * far; or, IOW, the largest number of objects of this type alive at |
| 170 | * the same time. |
| 171 | */ |
| 172 | int tp_maxalloc; |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 173 | |
| 174 | Allocation and deallocation code keeps these counts up to date. |
| 175 | Py_Finalize() displays a summary of the info returned by sys.getcounts() |
| 176 | (see below), along with assorted other special allocation counts (like |
| 177 | the number of tuple allocations satisfied by a tuple free-list, the number |
| 178 | of 1-character strings allocated, etc). |
| 179 | |
| 180 | Before Python 2.2, type objects were immortal, and the COUNT_ALLOCS |
| 181 | implementation relies on that. As of Python 2.2, heap-allocated type/ |
| 182 | class objects can go away. COUNT_ALLOCS can blow up in 2.2 and 2.2.1 |
| 183 | because of this; this was fixed in 2.2.2. Use of COUNT_ALLOCS makes |
| 184 | all heap-allocated type objects immortal, except for those for which no |
| 185 | object of that type is ever allocated. |
| 186 | |
Tim Peters | 78be799 | 2003-03-23 02:51:01 +0000 | [diff] [blame] | 187 | Starting with Python 2.3, If Py_TRACE_REFS is also defined, COUNT_ALLOCS |
| 188 | arranges to ensure that the type object for each allocated object |
| 189 | appears in the doubly-linked list of all objects maintained by |
| 190 | Py_TRACE_REFS. |
| 191 | |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 192 | Special gimmicks: |
| 193 | |
| 194 | sys.getcounts() |
| 195 | Return a list of 4-tuples, one entry for each type object for which |
| 196 | at least one object of that type was allocated. Each tuple is of |
| 197 | the form: |
| 198 | |
| 199 | (tp_name, tp_allocs, tp_frees, tp_maxalloc) |
| 200 | |
Tim Peters | 44c1a7b | 2002-07-09 19:27:20 +0000 | [diff] [blame] | 201 | Each distinct type object gets a distinct entry in this list, even |
Tim Peters | 48ba649 | 2002-07-09 19:24:54 +0000 | [diff] [blame] | 202 | if two or more type objects have the same tp_name (in which case |
| 203 | there's no way to distinguish them by looking at this list). The |
| 204 | list is ordered by time of first object allocation: the type object |
| 205 | for which the first allocation of an object of that type occurred |
| 206 | most recently is at the front of the list. |
| 207 | --------------------------------------------------------------------------- |
Michael W. Hudson | 202a4b6 | 2002-07-30 15:25:57 +0000 | [diff] [blame] | 208 | LLTRACE introduced well before 1.0 |
Michael W. Hudson | a625523 | 2002-07-30 09:49:29 +0000 | [diff] [blame] | 209 | |
Michael W. Hudson | 202a4b6 | 2002-07-30 15:25:57 +0000 | [diff] [blame] | 210 | Compile in support of Low Level TRACE-ing of the main interpreter loop. |
Michael W. Hudson | a625523 | 2002-07-30 09:49:29 +0000 | [diff] [blame] | 211 | |
| 212 | When this preprocessor symbol is defined, before eval_frame |
Michael W. Hudson | 202a4b6 | 2002-07-30 15:25:57 +0000 | [diff] [blame] | 213 | (eval_code2 before 2.2) executes a frame's code it checks the frame's |
| 214 | global namespace for a variable "__lltrace__". If such a variable is |
| 215 | found, mounds of information about what the interpreter is doing are |
| 216 | sprayed to stdout, such as every opcode and opcode argument and values |
| 217 | pushed onto and popped off the value stack. |
Michael W. Hudson | a625523 | 2002-07-30 09:49:29 +0000 | [diff] [blame] | 218 | |
| 219 | Not useful very often, but very useful when needed. |
Jeremy Hylton | 985eba5 | 2003-02-05 23:13:00 +0000 | [diff] [blame] | 220 | |
| 221 | --------------------------------------------------------------------------- |
| 222 | CALL_PROFILE introduced for Python 2.3 |
| 223 | |
| 224 | Count the number of function calls executed. |
| 225 | |
| 226 | When this symbol is defined, the ceval mainloop and helper functions |
| 227 | count the number of function calls made. It keeps detailed statistics |
| 228 | about what kind of object was called and whether the call hit any of |
| 229 | the special fast paths in the code. |