1 .. highlightlang:: c

     2 

     3 

     4 .. _memory:

     5 

     6 *****************

     7 Memory Management

     8 *****************

     9 

    10 .. sectionauthor:: Vladimir Marangozov <Vladimir.Marangozov@inrialpes.fr>

    11 

    12 

    13 

    14 .. _memoryoverview:

    15 

    16 Overview

    17 ========

    18 

    19 Memory management in Python involves a private heap containing all Python

    20 objects and data structures. The management of this private heap is ensured

    21 internally by the *Python memory manager*.  The Python memory manager has

    22 different components which deal with various dynamic storage management aspects,

    23 like sharing, segmentation, preallocation or caching.

    24 

    25 At the lowest level, a raw memory allocator ensures that there is enough room in

    26 the private heap for storing all Python-related data by interacting with the

    27 memory manager of the operating system. On top of the raw memory allocator,

    28 several object-specific allocators operate on the same heap and implement

    29 distinct memory management policies adapted to the peculiarities of every object

    30 type. For example, integer objects are managed differently within the heap than

    31 strings, tuples or dictionaries because integers imply different storage

    32 requirements and speed/space tradeoffs. The Python memory manager thus delegates

    33 some of the work to the object-specific allocators, but ensures that the latter

    34 operate within the bounds of the private heap.

    35 

    36 It is important to understand that the management of the Python heap is

    37 performed by the interpreter itself and that the user has no control over it,

    38 even if she regularly manipulates object pointers to memory blocks inside that

    39 heap.  The allocation of heap space for Python objects and other internal

    40 buffers is performed on demand by the Python memory manager through the Python/C

    41 API functions listed in this document.

    42 

    43 .. index::

    44    single: malloc()

    45    single: calloc()

    46    single: realloc()

    47    single: free()

    48 

    49 To avoid memory corruption, extension writers should never try to operate on

    50 Python objects with the functions exported by the C library: :cfunc:`malloc`,

    51 :cfunc:`calloc`, :cfunc:`realloc` and :cfunc:`free`.  This will result in  mixed

    52 calls between the C allocator and the Python memory manager with fatal

    53 consequences, because they implement different algorithms and operate on

    54 different heaps.  However, one may safely allocate and release memory blocks

    55 with the C library allocator for individual purposes, as shown in the following

    56 example::

    57 

    58    PyObject *res;

    59    char *buf = (char *) malloc(BUFSIZ); /* for I/O */

    60 

    61    if (buf == NULL)

    62        return PyErr_NoMemory();

    63    ...Do some I/O operation involving buf...

    64    res = PyString_FromString(buf);

    65    free(buf); /* malloc'ed */

    66    return res;

    67 

    68 In this example, the memory request for the I/O buffer is handled by the C

    69 library allocator. The Python memory manager is involved only in the allocation

    70 of the string object returned as a result.

    71 

    72 In most situations, however, it is recommended to allocate memory from the

    73 Python heap specifically because the latter is under control of the Python

    74 memory manager. For example, this is required when the interpreter is extended

    75 with new object types written in C. Another reason for using the Python heap is

    76 the desire to *inform* the Python memory manager about the memory needs of the

    77 extension module. Even when the requested memory is used exclusively for

    78 internal, highly-specific purposes, delegating all memory requests to the Python

    79 memory manager causes the interpreter to have a more accurate image of its

    80 memory footprint as a whole. Consequently, under certain circumstances, the

    81 Python memory manager may or may not trigger appropriate actions, like garbage

    82 collection, memory compaction or other preventive procedures. Note that by using

    83 the C library allocator as shown in the previous example, the allocated memory

    84 for the I/O buffer escapes completely the Python memory manager.

    85 

    86 

    87 .. _memoryinterface:

    88 

    89 Memory Interface

    90 ================

    91 

    92 The following function sets, modeled after the ANSI C standard, but specifying

    93 behavior when requesting zero bytes, are available for allocating and releasing

    94 memory from the Python heap:

    95 

    96 

    97 .. cfunction:: void* PyMem_Malloc(size_t n)

    98 

    99    Allocates *n* bytes and returns a pointer of type :ctype:`void\*` to the

   100    allocated memory, or *NULL* if the request fails. Requesting zero bytes returns

   101    a distinct non-*NULL* pointer if possible, as if :cfunc:`PyMem_Malloc(1)` had

   102    been called instead. The memory will not have been initialized in any way.

   103 

   104 

   105 .. cfunction:: void* PyMem_Realloc(void *p, size_t n)

   106 

   107    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be

   108    unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, the

   109    call is equivalent to :cfunc:`PyMem_Malloc(n)`; else if *n* is equal to zero,

   110    the memory block is resized but is not freed, and the returned pointer is

   111    non-*NULL*.  Unless *p* is *NULL*, it must have been returned by a previous call

   112    to :cfunc:`PyMem_Malloc` or :cfunc:`PyMem_Realloc`. If the request fails,

   113    :cfunc:`PyMem_Realloc` returns *NULL* and *p* remains a valid pointer to the

   114    previous memory area.

   115 

   116 

   117 .. cfunction:: void PyMem_Free(void *p)

   118 

   119    Frees the memory block pointed to by *p*, which must have been returned by a

   120    previous call to :cfunc:`PyMem_Malloc` or :cfunc:`PyMem_Realloc`.  Otherwise, or

   121    if :cfunc:`PyMem_Free(p)` has been called before, undefined behavior occurs. If

   122    *p* is *NULL*, no operation is performed.

   123 

   124 The following type-oriented macros are provided for convenience.  Note  that

   125 *TYPE* refers to any C type.

   126 

   127 

   128 .. cfunction:: TYPE* PyMem_New(TYPE, size_t n)

   129 

   130    Same as :cfunc:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of

   131    memory.  Returns a pointer cast to :ctype:`TYPE\*`.  The memory will not have

   132    been initialized in any way.

   133 

   134 

   135 .. cfunction:: TYPE* PyMem_Resize(void *p, TYPE, size_t n)

   136 

   137    Same as :cfunc:`PyMem_Realloc`, but the memory block is resized to ``(n *

   138    sizeof(TYPE))`` bytes.  Returns a pointer cast to :ctype:`TYPE\*`. On return,

   139    *p* will be a pointer to the new memory area, or *NULL* in the event of

   140    failure.  This is a C preprocessor macro; p is always reassigned.  Save

   141    the original value of p to avoid losing memory when handling errors.

   142 

   143 

   144 .. cfunction:: void PyMem_Del(void *p)

   145 

   146    Same as :cfunc:`PyMem_Free`.

   147 

   148 In addition, the following macro sets are provided for calling the Python memory

   149 allocator directly, without involving the C API functions listed above. However,

   150 note that their use does not preserve binary compatibility across Python

   151 versions and is therefore deprecated in extension modules.

   152 

   153 :cfunc:`PyMem_MALLOC`, :cfunc:`PyMem_REALLOC`, :cfunc:`PyMem_FREE`.

   154 

   155 :cfunc:`PyMem_NEW`, :cfunc:`PyMem_RESIZE`, :cfunc:`PyMem_DEL`.

   156 

   157 

   158 .. _memoryexamples:

   159 

   160 Examples

   161 ========

   162 

   163 Here is the example from section :ref:`memoryoverview`, rewritten so that the

   164 I/O buffer is allocated from the Python heap by using the first function set::

   165 

   166    PyObject *res;

   167    char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */

   168 

   169    if (buf == NULL)

   170        return PyErr_NoMemory();

   171    /* ...Do some I/O operation involving buf... */

   172    res = PyString_FromString(buf);

   173    PyMem_Free(buf); /* allocated with PyMem_Malloc */

   174    return res;

   175 

   176 The same code using the type-oriented function set::

   177 

   178    PyObject *res;

   179    char *buf = PyMem_New(char, BUFSIZ); /* for I/O */

   180 

   181    if (buf == NULL)

   182        return PyErr_NoMemory();

   183    /* ...Do some I/O operation involving buf... */

   184    res = PyString_FromString(buf);

   185    PyMem_Del(buf); /* allocated with PyMem_New */

   186    return res;

   187 

   188 Note that in the two examples above, the buffer is always manipulated via

   189 functions belonging to the same set. Indeed, it is required to use the same

   190 memory API family for a given memory block, so that the risk of mixing different

   191 allocators is reduced to a minimum. The following code sequence contains two

   192 errors, one of which is labeled as *fatal* because it mixes two different

   193 allocators operating on different heaps. ::

   194 

   195    char *buf1 = PyMem_New(char, BUFSIZ);

   196    char *buf2 = (char *) malloc(BUFSIZ);

   197    char *buf3 = (char *) PyMem_Malloc(BUFSIZ);

   198    ...

   199    PyMem_Del(buf3);  /* Wrong -- should be PyMem_Free() */

   200    free(buf2);       /* Right -- allocated via malloc() */

   201    free(buf1);       /* Fatal -- should be PyMem_Del()  */

   202 

   203 In addition to the functions aimed at handling raw memory blocks from the Python

   204 heap, objects in Python are allocated and released with :cfunc:`PyObject_New`,

   205 :cfunc:`PyObject_NewVar` and :cfunc:`PyObject_Del`.

   206 

   207 These will be explained in the next chapter on defining and implementing new

   208 object types in C.

   209