1 .. highlightlang:: c 2 3 4 .. _memory: 5 6 ***************** 7 Memory Management 8 ***************** 9 10 .. sectionauthor:: Vladimir Marangozov <Vladimir.Marangozov (a] inrialpes.fr> 11 12 13 14 .. _memoryoverview: 15 16 Overview 17 ======== 18 19 Memory management in Python involves a private heap containing all Python 20 objects and data structures. The management of this private heap is ensured 21 internally by the *Python memory manager*. The Python memory manager has 22 different components which deal with various dynamic storage management aspects, 23 like sharing, segmentation, preallocation or caching. 24 25 At the lowest level, a raw memory allocator ensures that there is enough room in 26 the private heap for storing all Python-related data by interacting with the 27 memory manager of the operating system. On top of the raw memory allocator, 28 several object-specific allocators operate on the same heap and implement 29 distinct memory management policies adapted to the peculiarities of every object 30 type. For example, integer objects are managed differently within the heap than 31 strings, tuples or dictionaries because integers imply different storage 32 requirements and speed/space tradeoffs. The Python memory manager thus delegates 33 some of the work to the object-specific allocators, but ensures that the latter 34 operate within the bounds of the private heap. 35 36 It is important to understand that the management of the Python heap is 37 performed by the interpreter itself and that the user has no control over it, 38 even if she regularly manipulates object pointers to memory blocks inside that 39 heap. The allocation of heap space for Python objects and other internal 40 buffers is performed on demand by the Python memory manager through the Python/C 41 API functions listed in this document. 42 43 .. index:: 44 single: malloc() 45 single: calloc() 46 single: realloc() 47 single: free() 48 49 To avoid memory corruption, extension writers should never try to operate on 50 Python objects with the functions exported by the C library: :c:func:`malloc`, 51 :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`. This will result in mixed 52 calls between the C allocator and the Python memory manager with fatal 53 consequences, because they implement different algorithms and operate on 54 different heaps. However, one may safely allocate and release memory blocks 55 with the C library allocator for individual purposes, as shown in the following 56 example:: 57 58 PyObject *res; 59 char *buf = (char *) malloc(BUFSIZ); /* for I/O */ 60 61 if (buf == NULL) 62 return PyErr_NoMemory(); 63 ...Do some I/O operation involving buf... 64 res = PyString_FromString(buf); 65 free(buf); /* malloc'ed */ 66 return res; 67 68 In this example, the memory request for the I/O buffer is handled by the C 69 library allocator. The Python memory manager is involved only in the allocation 70 of the string object returned as a result. 71 72 In most situations, however, it is recommended to allocate memory from the 73 Python heap specifically because the latter is under control of the Python 74 memory manager. For example, this is required when the interpreter is extended 75 with new object types written in C. Another reason for using the Python heap is 76 the desire to *inform* the Python memory manager about the memory needs of the 77 extension module. Even when the requested memory is used exclusively for 78 internal, highly-specific purposes, delegating all memory requests to the Python 79 memory manager causes the interpreter to have a more accurate image of its 80 memory footprint as a whole. Consequently, under certain circumstances, the 81 Python memory manager may or may not trigger appropriate actions, like garbage 82 collection, memory compaction or other preventive procedures. Note that by using 83 the C library allocator as shown in the previous example, the allocated memory 84 for the I/O buffer escapes completely the Python memory manager. 85 86 87 .. _memoryinterface: 88 89 Memory Interface 90 ================ 91 92 The following function sets, modeled after the ANSI C standard, but specifying 93 behavior when requesting zero bytes, are available for allocating and releasing 94 memory from the Python heap: 95 96 97 .. c:function:: void* PyMem_Malloc(size_t n) 98 99 Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the 100 allocated memory, or *NULL* if the request fails. Requesting zero bytes returns 101 a distinct non-*NULL* pointer if possible, as if ``PyMem_Malloc(1)`` had 102 been called instead. The memory will not have been initialized in any way. 103 104 105 .. c:function:: void* PyMem_Realloc(void *p, size_t n) 106 107 Resizes the memory block pointed to by *p* to *n* bytes. The contents will be 108 unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, the 109 call is equivalent to ``PyMem_Malloc(n)``; else if *n* is equal to zero, 110 the memory block is resized but is not freed, and the returned pointer is 111 non-*NULL*. Unless *p* is *NULL*, it must have been returned by a previous call 112 to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. If the request fails, 113 :c:func:`PyMem_Realloc` returns *NULL* and *p* remains a valid pointer to the 114 previous memory area. 115 116 117 .. c:function:: void PyMem_Free(void *p) 118 119 Frees the memory block pointed to by *p*, which must have been returned by a 120 previous call to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. Otherwise, or 121 if ``PyMem_Free(p)`` has been called before, undefined behavior occurs. If 122 *p* is *NULL*, no operation is performed. 123 124 The following type-oriented macros are provided for convenience. Note that 125 *TYPE* refers to any C type. 126 127 128 .. c:function:: TYPE* PyMem_New(TYPE, size_t n) 129 130 Same as :c:func:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of 131 memory. Returns a pointer cast to :c:type:`TYPE\*`. The memory will not have 132 been initialized in any way. 133 134 135 .. c:function:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) 136 137 Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * 138 sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, 139 *p* will be a pointer to the new memory area, or *NULL* in the event of 140 failure. This is a C preprocessor macro; p is always reassigned. Save 141 the original value of p to avoid losing memory when handling errors. 142 143 144 .. c:function:: void PyMem_Del(void *p) 145 146 Same as :c:func:`PyMem_Free`. 147 148 In addition, the following macro sets are provided for calling the Python memory 149 allocator directly, without involving the C API functions listed above. However, 150 note that their use does not preserve binary compatibility across Python 151 versions and is therefore deprecated in extension modules. 152 153 :c:func:`PyMem_MALLOC`, :c:func:`PyMem_REALLOC`, :c:func:`PyMem_FREE`. 154 155 :c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`. 156 157 158 .. _memoryexamples: 159 160 Examples 161 ======== 162 163 Here is the example from section :ref:`memoryoverview`, rewritten so that the 164 I/O buffer is allocated from the Python heap by using the first function set:: 165 166 PyObject *res; 167 char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */ 168 169 if (buf == NULL) 170 return PyErr_NoMemory(); 171 /* ...Do some I/O operation involving buf... */ 172 res = PyString_FromString(buf); 173 PyMem_Free(buf); /* allocated with PyMem_Malloc */ 174 return res; 175 176 The same code using the type-oriented function set:: 177 178 PyObject *res; 179 char *buf = PyMem_New(char, BUFSIZ); /* for I/O */ 180 181 if (buf == NULL) 182 return PyErr_NoMemory(); 183 /* ...Do some I/O operation involving buf... */ 184 res = PyString_FromString(buf); 185 PyMem_Del(buf); /* allocated with PyMem_New */ 186 return res; 187 188 Note that in the two examples above, the buffer is always manipulated via 189 functions belonging to the same set. Indeed, it is required to use the same 190 memory API family for a given memory block, so that the risk of mixing different 191 allocators is reduced to a minimum. The following code sequence contains two 192 errors, one of which is labeled as *fatal* because it mixes two different 193 allocators operating on different heaps. :: 194 195 char *buf1 = PyMem_New(char, BUFSIZ); 196 char *buf2 = (char *) malloc(BUFSIZ); 197 char *buf3 = (char *) PyMem_Malloc(BUFSIZ); 198 ... 199 PyMem_Del(buf3); /* Wrong -- should be PyMem_Free() */ 200 free(buf2); /* Right -- allocated via malloc() */ 201 free(buf1); /* Fatal -- should be PyMem_Del() */ 202 203 In addition to the functions aimed at handling raw memory blocks from the Python 204 heap, objects in Python are allocated and released with :c:func:`PyObject_New`, 205 :c:func:`PyObject_NewVar` and :c:func:`PyObject_Del`. 206 207 These will be explained in the next chapter on defining and implementing new 208 object types in C. 209 210
