1 This file describes some special Python build types enabled via compile-time 2 preprocessor defines. 3 4 IMPORTANT: if you want to build a debug-enabled Python, it is recommended that 5 you use ``./configure --with-pydebug``, rather than the options listed here. 6 7 However, if you wish to define some of these options individually, it is best 8 to define them in the EXTRA_CFLAGS make variable; 9 ``make EXTRA_CFLAGS="-DPy_REF_DEBUG"``. 10 11 12 Py_REF_DEBUG 13 ------------ 14 15 Turn on aggregate reference counting. This arranges that extern _Py_RefTotal 16 hold a count of all references, the sum of ob_refcnt across all objects. 17 Passing ``-X showrefcount`` on the command line causes the interactive 18 interpreter to print the reference count total as well the number of memory 19 blocks allocated after each statement: 20 21 >>> 23 22 23 23 [8288 refs, 14332 blocks] 24 >>> 25 26 Note that if this count increases when you're not storing away new objects, 27 there's probably a leak. Remember, though, that in interactive mode the special 28 name "_" holds a reference to the last result displayed! 29 30 Py_REF_DEBUG also checks after every decref to verify that the refcount hasn't 31 gone negative, and causes an immediate fatal error if it has. 32 33 Special gimmicks: 34 35 sys.gettotalrefcount() 36 Return current total of all refcounts. 37 38 39 Py_TRACE_REFS 40 ------------- 41 42 Turn on heavy reference debugging. This is major surgery. Every PyObject grows 43 two more pointers, to maintain a doubly-linked list of all live heap-allocated 44 objects. Most built-in type objects are not in this list, as they're statically 45 allocated. Starting in Python 2.3, if COUNT_ALLOCS (see below) is also defined, 46 a static type object T does appear in this list if at least one object of type T 47 has been created. 48 49 Note that because the fundamental PyObject layout changes, Python modules 50 compiled with Py_TRACE_REFS are incompatible with modules compiled without it. 51 52 Py_TRACE_REFS implies Py_REF_DEBUG. 53 54 Special gimmicks: 55 56 sys.getobjects(max[, type]) 57 Return list of the (no more than) max most-recently allocated objects, most 58 recently allocated first in the list, least-recently allocated last in the 59 list. max=0 means no limit on list length. If an optional type object is 60 passed, the list is also restricted to objects of that type. The return 61 list itself, and some temp objects created just to call sys.getobjects(), 62 are excluded from the return list. Note that the list returned is just 63 another object, though, so may appear in the return list the next time you 64 call getobjects(); note that every object in the list is kept alive too, 65 simply by virtue of being in the list. 66 67 envvar PYTHONDUMPREFS 68 If this envvar exists, Py_FinalizeEx() arranges to print a list of all 69 still-live heap objects. This is printed twice, in different formats, 70 before and after Py_FinalizeEx has cleaned up everything it can clean up. The 71 first output block produces the repr() of each object so is more 72 informative; however, a lot of stuff destined to die is still alive then. 73 The second output block is much harder to work with (repr() can't be invoked 74 anymore -- the interpreter has been torn down too far), but doesn't list any 75 objects that will die. The tool script combinerefs.py can be run over this 76 to combine the info from both output blocks. The second output block, and 77 combinerefs.py, were new in Python 2.3b1. 78 79 80 PYMALLOC_DEBUG 81 -------------- 82 83 When pymalloc is enabled (WITH_PYMALLOC is defined), calls to the PyObject_ 84 memory routines are handled by Python's own small-object allocator, while calls 85 to the PyMem_ memory routines are directed to the system malloc/ realloc/free. 86 If PYMALLOC_DEBUG is also defined, calls to both PyObject_ and PyMem_ memory 87 routines are directed to a special debugging mode of Python's small-object 88 allocator. 89 90 This mode fills dynamically allocated memory blocks with special, recognizable 91 bit patterns, and adds debugging info on each end of dynamically allocated 92 memory blocks. The special bit patterns are: 93 94 #define CLEANBYTE 0xCB /* clean (newly allocated) memory */ 95 #define DEADBYTE 0xDB /* dead (newly freed) memory */ 96 #define FORBIDDENBYTE 0xFB /* forbidden -- untouchable bytes */ 97 98 Strings of these bytes are unlikely to be valid addresses, floats, or 7-bit 99 ASCII strings. 100 101 Let S = sizeof(size_t). 2*S bytes are added at each end of each block of N bytes 102 requested. The memory layout is like so, where p represents the address 103 returned by a malloc-like or realloc-like function (p[i:j] means the slice of 104 bytes from *(p+i) inclusive up to *(p+j) exclusive; note that the treatment of 105 negative indices differs from a Python slice): 106 107 p[-2*S:-S] 108 Number of bytes originally asked for. This is a size_t, big-endian (easier 109 to read in a memory dump). 110 p[-S] 111 API ID. See PEP 445. This is a character, but seems undocumented. 112 p[-S+1:0] 113 Copies of FORBIDDENBYTE. Used to catch under- writes and reads. 114 p[0:N] 115 The requested memory, filled with copies of CLEANBYTE, used to catch 116 reference to uninitialized memory. When a realloc-like function is called 117 requesting a larger memory block, the new excess bytes are also filled with 118 CLEANBYTE. When a free-like function is called, these are overwritten with 119 DEADBYTE, to catch reference to freed memory. When a realloc- like function 120 is called requesting a smaller memory block, the excess old bytes are also 121 filled with DEADBYTE. 122 p[N:N+S] 123 Copies of FORBIDDENBYTE. Used to catch over- writes and reads. 124 p[N+S:N+2*S] 125 A serial number, incremented by 1 on each call to a malloc-like or 126 realloc-like function. Big-endian size_t. If "bad memory" is detected 127 later, the serial number gives an excellent way to set a breakpoint on the 128 next run, to capture the instant at which this block was passed out. The 129 static function bumpserialno() in obmalloc.c is the only place the serial 130 number is incremented, and exists so you can set such a breakpoint easily. 131 132 A realloc-like or free-like function first checks that the FORBIDDENBYTEs at 133 each end are intact. If they've been altered, diagnostic output is written to 134 stderr, and the program is aborted via Py_FatalError(). The other main failure 135 mode is provoking a memory error when a program reads up one of the special bit 136 patterns and tries to use it as an address. If you get in a debugger then and 137 look at the object, you're likely to see that it's entirely filled with 0xDB 138 (meaning freed memory is getting used) or 0xCB (meaning uninitialized memory is 139 getting used). 140 141 Note that PYMALLOC_DEBUG requires WITH_PYMALLOC. 142 143 Special gimmicks: 144 145 envvar PYTHONMALLOCSTATS 146 If this envvar exists, a report of pymalloc summary statistics is printed to 147 stderr whenever a new arena is allocated, and also by Py_FinalizeEx(). 148 149 Changed in 2.5: The number of extra bytes allocated is 4*sizeof(size_t). 150 Before it was 16 on all boxes, reflecting that Python couldn't make use of 151 allocations >= 2**32 bytes even on 64-bit boxes before 2.5. 152 153 154 Py_DEBUG 155 -------- 156 157 This is what is generally meant by "a debug build" of Python. 158 159 Py_DEBUG implies LLTRACE, Py_REF_DEBUG, Py_TRACE_REFS, and PYMALLOC_DEBUG (if 160 WITH_PYMALLOC is enabled). In addition, C assert()s are enabled (via the C way: 161 by not defining NDEBUG), and some routines do additional sanity checks inside 162 "#ifdef Py_DEBUG" blocks. 163 164 165 COUNT_ALLOCS 166 ------------ 167 168 Each type object grows three new members: 169 170 /* Number of times an object of this type was allocated. */ 171 int tp_allocs; 172 173 /* Number of times an object of this type was deallocated. */ 174 int tp_frees; 175 176 /* Highwater mark: the maximum value of tp_allocs - tp_frees so 177 * far; or, IOW, the largest number of objects of this type alive at 178 * the same time. 179 */ 180 int tp_maxalloc; 181 182 Allocation and deallocation code keeps these counts up to date. Py_FinalizeEx() 183 displays a summary of the info returned by sys.getcounts() (see below), along 184 with assorted other special allocation counts (like the number of tuple 185 allocations satisfied by a tuple free-list, the number of 1-character strings 186 allocated, etc). 187 188 Before Python 2.2, type objects were immortal, and the COUNT_ALLOCS 189 implementation relies on that. As of Python 2.2, heap-allocated type/ class 190 objects can go away. COUNT_ALLOCS can blow up in 2.2 and 2.2.1 because of this; 191 this was fixed in 2.2.2. Use of COUNT_ALLOCS makes all heap-allocated type 192 objects immortal, except for those for which no object of that type is ever 193 allocated. 194 195 Starting with Python 2.3, If Py_TRACE_REFS is also defined, COUNT_ALLOCS 196 arranges to ensure that the type object for each allocated object appears in the 197 doubly-linked list of all objects maintained by Py_TRACE_REFS. 198 199 Special gimmicks: 200 201 sys.getcounts() 202 Return a list of 4-tuples, one entry for each type object for which at least 203 one object of that type was allocated. Each tuple is of the form: 204 205 (tp_name, tp_allocs, tp_frees, tp_maxalloc) 206 207 Each distinct type object gets a distinct entry in this list, even if two or 208 more type objects have the same tp_name (in which case there's no way to 209 distinguish them by looking at this list). The list is ordered by time of 210 first object allocation: the type object for which the first allocation of 211 an object of that type occurred most recently is at the front of the list. 212 213 214 LLTRACE 215 ------- 216 217 Compile in support for Low Level TRACE-ing of the main interpreter loop. 218 219 When this preprocessor symbol is defined, before PyEval_EvalFrame executes a 220 frame's code it checks the frame's global namespace for a variable 221 "__ltrace__". If such a variable is found, mounds of information about what 222 the interpreter is doing are sprayed to stdout, such as every opcode and opcode 223 argument and values pushed onto and popped off the value stack. 224 225 Not useful very often, but very useful when needed. 226
