1 This file describes some special Python build types enabled via

     2 compile-time preprocessor defines.

     3 

     4 It is best to define these options in the EXTRA_CFLAGS make variable;

     5 ``make EXTRA_CFLAGS="-DPy_REF_DEBUG"``.

     6 

     7 ---------------------------------------------------------------------------

     8 Py_REF_DEBUG                                              introduced in 1.4

     9                                                  named REF_DEBUG before 1.4

    10 

    11 Turn on aggregate reference counting.  This arranges that extern

    12 _Py_RefTotal hold a count of all references, the sum of ob_refcnt across

    13 all objects.  In a debug-mode build, this is where the "8288" comes from

    14 in

    15 

    16     >>> 23

    17     23

    18     [8288 refs]

    19     >>>

    20 

    21 Note that if this count increases when you're not storing away new objects,

    22 there's probably a leak.  Remember, though, that in interactive mode the

    23 special name "_" holds a reference to the last result displayed!

    24 

    25 Py_REF_DEBUG also checks after every decref to verify that the refcount

    26 hasn't gone negative, and causes an immediate fatal error if it has.

    27 

    28 Special gimmicks:

    29 

    30 sys.gettotalrefcount()

    31     Return current total of all refcounts.

    32     Available under Py_REF_DEBUG in Python 2.3.

    33     Before 2.3, Py_TRACE_REFS was required to enable this function.

    34 ---------------------------------------------------------------------------

    35 Py_TRACE_REFS                                             introduced in 1.4

    36                                                 named TRACE_REFS before 1.4

    37 

    38 Turn on heavy reference debugging.  This is major surgery.  Every PyObject

    39 grows two more pointers, to maintain a doubly-linked list of all live

    40 heap-allocated objects.  Most builtin type objects are not in this list,

    41 as they're statically allocated.  Starting in Python 2.3, if COUNT_ALLOCS

    42 (see below) is also defined, a static type object T does appear in this

    43 list if at least one object of type T has been created.

    44 

    45 Note that because the fundamental PyObject layout changes, Python modules

    46 compiled with Py_TRACE_REFS are incompatible with modules compiled without

    47 it.

    48 

    49 Py_TRACE_REFS implies Py_REF_DEBUG.

    50 

    51 Special gimmicks:

    52 

    53 sys.getobjects(max[, type])

    54     Return list of the (no more than) max most-recently allocated objects,

    55     most recently allocated first in the list, least-recently allocated

    56     last in the list.  max=0 means no limit on list length.

    57     If an optional type object is passed, the list is also restricted to

    58     objects of that type.

    59     The return list itself, and some temp objects created just to call

    60     sys.getobjects(), are excluded from the return list.  Note that the

    61     list returned is just another object, though, so may appear in the

    62     return list the next time you call getobjects(); note that every

    63     object in the list is kept alive too, simply by virtue of being in

    64     the list.

    65 

    66 envar PYTHONDUMPREFS

    67     If this envar exists, Py_Finalize() arranges to print a list of

    68     all still-live heap objects.  This is printed twice, in different

    69     formats, before and after Py_Finalize has cleaned up everything it

    70     can clean up.  The first output block produces the repr() of each

    71     object so is more informative; however, a lot of stuff destined to

    72     die is still alive then.  The second output block is much harder

    73     to work with (repr() can't be invoked anymore -- the interpreter

    74     has been torn down too far), but doesn't list any objects that will

    75     die.  The tool script combinerefs.py can be run over this to combine

    76     the info from both output blocks.  The second output block, and

    77     combinerefs.py, were new in Python 2.3b1.

    78 ---------------------------------------------------------------------------

    79 PYMALLOC_DEBUG                                            introduced in 2.3

    80 

    81 When pymalloc is enabled (WITH_PYMALLOC is defined), calls to the PyObject_

    82 memory routines are handled by Python's own small-object allocator, while

    83 calls to the PyMem_ memory routines are directed to the system malloc/

    84 realloc/free.  If PYMALLOC_DEBUG is also defined, calls to both PyObject_

    85 and PyMem_ memory routines are directed to a special debugging mode of

    86 Python's small-object allocator.

    87 

    88 This mode fills dynamically allocated memory blocks with special,

    89 recognizable bit patterns, and adds debugging info on each end of

    90 dynamically allocated memory blocks.  The special bit patterns are:

    91 

    92 #define CLEANBYTE     0xCB   /* clean (newly allocated) memory */

    93 #define DEADBYTE      0xDB   /* dead (newly freed) memory */

    94 #define FORBIDDENBYTE 0xFB   /* forbidden -- untouchable bytes */

    95 

    96 Strings of these bytes are unlikely to be valid addresses, floats, or 7-bit

    97 ASCII strings.

    98 

    99 Let S = sizeof(size_t). 2*S bytes are added at each end of each block of N

   100 bytes requested.  The memory layout is like so, where p represents the

   101 address returned by a malloc-like or realloc-like function (p[i:j] means

   102 the slice of bytes from *(p+i) inclusive up to *(p+j) exclusive; note that

   103 the treatment of negative indices differs from a Python slice):

   104 

   105 p[-2*S:-S]

   106     Number of bytes originally asked for.  This is a size_t, big-endian

   107     (easier to read in a memory dump).

   108 p[-S:0]

   109     Copies of FORBIDDENBYTE.  Used to catch under- writes and reads.

   110 p[0:N]

   111     The requested memory, filled with copies of CLEANBYTE, used to catch

   112     reference to uninitialized memory.

   113     When a realloc-like function is called requesting a larger memory

   114     block, the new excess bytes are also filled with CLEANBYTE.

   115     When a free-like function is called, these are overwritten with

   116     DEADBYTE, to catch reference to freed memory.  When a realloc-

   117     like function is called requesting a smaller memory block, the excess

   118     old bytes are also filled with DEADBYTE.

   119 p[N:N+S]

   120     Copies of FORBIDDENBYTE.  Used to catch over- writes and reads.

   121 p[N+S:N+2*S]

   122     A serial number, incremented by 1 on each call to a malloc-like or

   123     realloc-like function.

   124     Big-endian size_t.

   125     If "bad memory" is detected later, the serial number gives an

   126     excellent way to set a breakpoint on the next run, to capture the

   127     instant at which this block was passed out.  The static function

   128     bumpserialno() in obmalloc.c is the only place the serial number

   129     is incremented, and exists so you can set such a breakpoint easily.

   130 

   131 A realloc-like or free-like function first checks that the FORBIDDENBYTEs

   132 at each end are intact.  If they've been altered, diagnostic output is

   133 written to stderr, and the program is aborted via Py_FatalError().  The

   134 other main failure mode is provoking a memory error when a program

   135 reads up one of the special bit patterns and tries to use it as an address.

   136 If you get in a debugger then and look at the object, you're likely

   137 to see that it's entirely filled with 0xDB (meaning freed memory is

   138 getting used) or 0xCB (meaning uninitialized memory is getting used).

   139 

   140 Note that PYMALLOC_DEBUG requires WITH_PYMALLOC.

   141 

   142 Special gimmicks:

   143 

   144 envar PYTHONMALLOCSTATS

   145     If this envar exists, a report of pymalloc summary statistics is

   146     printed to stderr whenever a new arena is allocated, and also

   147     by Py_Finalize().

   148 

   149 Changed in 2.5:  The number of extra bytes allocated is 4*sizeof(size_t).

   150 Before it was 16 on all boxes, reflecting that Python couldn't make use of

   151 allocations >= 2**32 bytes even on 64-bit boxes before 2.5.

   152 ---------------------------------------------------------------------------

   153 Py_DEBUG                                                  introduced in 1.5

   154                                                      named DEBUG before 1.5

   155 

   156 This is what is generally meant by "a debug build" of Python.

   157 

   158 Py_DEBUG implies LLTRACE, Py_REF_DEBUG, Py_TRACE_REFS, and

   159 PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled).  In addition, C

   160 assert()s are enabled (via the C way: by not defining NDEBUG), and

   161 some routines do additional sanity checks inside "#ifdef Py_DEBUG"

   162 blocks.

   163 ---------------------------------------------------------------------------

   164 COUNT_ALLOCS                                            introduced in 0.9.9

   165                                              partly broken in 2.2 and 2.2.1

   166 

   167 Each type object grows three new members:

   168 

   169     /* Number of times an object of this type was allocated. */

   170     int tp_allocs;

   171 

   172     /* Number of times an object of this type was deallocated. */

   173     int tp_frees;

   174 

   175     /* Highwater mark:  the maximum value of tp_allocs - tp_frees so

   176      * far; or, IOW, the largest number of objects of this type alive at

   177      * the same time.

   178      */

   179     int tp_maxalloc;

   180 

   181 Allocation and deallocation code keeps these counts up to date.

   182 Py_Finalize() displays a summary of the info returned by sys.getcounts()

   183 (see below), along with assorted other special allocation counts (like

   184 the number of tuple allocations satisfied by a tuple free-list, the number

   185 of 1-character strings allocated, etc).

   186 

   187 Before Python 2.2, type objects were immortal, and the COUNT_ALLOCS

   188 implementation relies on that.  As of Python 2.2, heap-allocated type/

   189 class objects can go away.  COUNT_ALLOCS can blow up in 2.2 and 2.2.1

   190 because of this; this was fixed in 2.2.2.  Use of COUNT_ALLOCS makes

   191 all heap-allocated type objects immortal, except for those for which no

   192 object of that type is ever allocated.

   193 

   194 Starting with Python 2.3, If Py_TRACE_REFS is also defined, COUNT_ALLOCS

   195 arranges to ensure that the type object for each allocated object

   196 appears in the doubly-linked list of all objects maintained by

   197 Py_TRACE_REFS.

   198 

   199 Special gimmicks:

   200 

   201 sys.getcounts()

   202     Return a list of 4-tuples, one entry for each type object for which

   203     at least one object of that type was allocated.  Each tuple is of

   204     the form:

   205 

   206         (tp_name, tp_allocs, tp_frees, tp_maxalloc)

   207 

   208     Each distinct type object gets a distinct entry in this list, even

   209     if two or more type objects have the same tp_name (in which case

   210     there's no way to distinguish them by looking at this list).  The

   211     list is ordered by time of first object allocation:  the type object

   212     for which the first allocation of an object of that type occurred

   213     most recently is at the front of the list.

   214 ---------------------------------------------------------------------------

   215 LLTRACE                                          introduced well before 1.0

   216 

   217 Compile in support for Low Level TRACE-ing of the main interpreter loop.

   218 

   219 When this preprocessor symbol is defined, before PyEval_EvalFrame

   220 (eval_frame in 2.3 and 2.2, eval_code2 before that) executes a frame's code

   221 it checks the frame's global namespace for a variable "__lltrace__".  If

   222 such a variable is found, mounds of information about what the interpreter

   223 is doing are sprayed to stdout, such as every opcode and opcode argument

   224 and values pushed onto and popped off the value stack.

   225 

   226 Not useful very often, but very useful when needed.

   227 

   228 ---------------------------------------------------------------------------

   229 CALL_PROFILE                                      introduced for Python 2.3

   230 

   231 Count the number of function calls executed.

   232 

   233 When this symbol is defined, the ceval mainloop and helper functions

   234 count the number of function calls made.  It keeps detailed statistics

   235 about what kind of object was called and whether the call hit any of

   236 the special fast paths in the code.

   237 

   238 ---------------------------------------------------------------------------

   239 WITH_TSC                                          introduced for Python 2.4

   240 

   241 Super-lowlevel profiling of the interpreter.  When enabled, the sys

   242 module grows a new function:

   243 

   244 settscdump(bool)

   245     If true, tell the Python interpreter to dump VM measurements to

   246     stderr.  If false, turn off dump.  The measurements are based on the

   247     processor's time-stamp counter.

   248 

   249 This build option requires a small amount of platform specific code.

   250 Currently this code is present for linux/x86 and any PowerPC platform

   251 that uses GCC (i.e. OS X and linux/ppc).

   252 

   253 On the PowerPC the rate at which the time base register is incremented

   254 is not defined by the architecture specification, so you'll need to

   255 find the manual for your specific processor.  For the 750CX, 750CXe

   256 and 750FX (all sold as the G3) we find:

   257 

   258     The time base counter is clocked at a frequency that is

   259     one-fourth that of the bus clock.

   260 

   261 This build is enabled by the --with-tsc flag to configure.