Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | .. highlightlang:: c |
| 2 | |
| 3 | |
| 4 | .. _memory: |
| 5 | |
| 6 | ***************** |
| 7 | Memory Management |
| 8 | ***************** |
| 9 | |
| 10 | .. sectionauthor:: Vladimir Marangozov <Vladimir.Marangozov@inrialpes.fr> |
| 11 | |
| 12 | |
| 13 | |
| 14 | .. _memoryoverview: |
| 15 | |
| 16 | Overview |
| 17 | ======== |
| 18 | |
| 19 | Memory management in Python involves a private heap containing all Python |
| 20 | objects and data structures. The management of this private heap is ensured |
| 21 | internally by the *Python memory manager*. The Python memory manager has |
| 22 | different components which deal with various dynamic storage management aspects, |
| 23 | like sharing, segmentation, preallocation or caching. |
| 24 | |
| 25 | At the lowest level, a raw memory allocator ensures that there is enough room in |
| 26 | the private heap for storing all Python-related data by interacting with the |
| 27 | memory manager of the operating system. On top of the raw memory allocator, |
| 28 | several object-specific allocators operate on the same heap and implement |
| 29 | distinct memory management policies adapted to the peculiarities of every object |
| 30 | type. For example, integer objects are managed differently within the heap than |
| 31 | strings, tuples or dictionaries because integers imply different storage |
| 32 | requirements and speed/space tradeoffs. The Python memory manager thus delegates |
| 33 | some of the work to the object-specific allocators, but ensures that the latter |
| 34 | operate within the bounds of the private heap. |
| 35 | |
| 36 | It is important to understand that the management of the Python heap is |
| 37 | performed by the interpreter itself and that the user has no control over it, |
| 38 | even if she regularly manipulates object pointers to memory blocks inside that |
| 39 | heap. The allocation of heap space for Python objects and other internal |
| 40 | buffers is performed on demand by the Python memory manager through the Python/C |
| 41 | API functions listed in this document. |
| 42 | |
| 43 | .. index:: |
| 44 | single: malloc() |
| 45 | single: calloc() |
| 46 | single: realloc() |
| 47 | single: free() |
| 48 | |
| 49 | To avoid memory corruption, extension writers should never try to operate on |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 50 | Python objects with the functions exported by the C library: :c:func:`malloc`, |
| 51 | :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`. This will result in mixed |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 52 | calls between the C allocator and the Python memory manager with fatal |
| 53 | consequences, because they implement different algorithms and operate on |
| 54 | different heaps. However, one may safely allocate and release memory blocks |
| 55 | with the C library allocator for individual purposes, as shown in the following |
| 56 | example:: |
| 57 | |
| 58 | PyObject *res; |
| 59 | char *buf = (char *) malloc(BUFSIZ); /* for I/O */ |
| 60 | |
| 61 | if (buf == NULL) |
| 62 | return PyErr_NoMemory(); |
| 63 | ...Do some I/O operation involving buf... |
| 64 | res = PyString_FromString(buf); |
| 65 | free(buf); /* malloc'ed */ |
| 66 | return res; |
| 67 | |
| 68 | In this example, the memory request for the I/O buffer is handled by the C |
| 69 | library allocator. The Python memory manager is involved only in the allocation |
| 70 | of the string object returned as a result. |
| 71 | |
| 72 | In most situations, however, it is recommended to allocate memory from the |
| 73 | Python heap specifically because the latter is under control of the Python |
| 74 | memory manager. For example, this is required when the interpreter is extended |
| 75 | with new object types written in C. Another reason for using the Python heap is |
| 76 | the desire to *inform* the Python memory manager about the memory needs of the |
| 77 | extension module. Even when the requested memory is used exclusively for |
| 78 | internal, highly-specific purposes, delegating all memory requests to the Python |
| 79 | memory manager causes the interpreter to have a more accurate image of its |
| 80 | memory footprint as a whole. Consequently, under certain circumstances, the |
| 81 | Python memory manager may or may not trigger appropriate actions, like garbage |
| 82 | collection, memory compaction or other preventive procedures. Note that by using |
| 83 | the C library allocator as shown in the previous example, the allocated memory |
| 84 | for the I/O buffer escapes completely the Python memory manager. |
| 85 | |
| 86 | |
| 87 | .. _memoryinterface: |
| 88 | |
| 89 | Memory Interface |
| 90 | ================ |
| 91 | |
| 92 | The following function sets, modeled after the ANSI C standard, but specifying |
| 93 | behavior when requesting zero bytes, are available for allocating and releasing |
| 94 | memory from the Python heap: |
| 95 | |
| 96 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 97 | .. c:function:: void* PyMem_Malloc(size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 98 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 99 | Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 100 | allocated memory, or *NULL* if the request fails. Requesting zero bytes returns |
Andrew Svetlov | 7dbee38 | 2012-08-09 21:26:34 +0300 | [diff] [blame] | 101 | a distinct non-*NULL* pointer if possible, as if ``PyMem_Malloc(1)`` had |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 102 | been called instead. The memory will not have been initialized in any way. |
| 103 | |
| 104 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 105 | .. c:function:: void* PyMem_Realloc(void *p, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 106 | |
| 107 | Resizes the memory block pointed to by *p* to *n* bytes. The contents will be |
| 108 | unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, the |
Andrew Svetlov | 7dbee38 | 2012-08-09 21:26:34 +0300 | [diff] [blame] | 109 | call is equivalent to ``PyMem_Malloc(n)``; else if *n* is equal to zero, |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 110 | the memory block is resized but is not freed, and the returned pointer is |
| 111 | non-*NULL*. Unless *p* is *NULL*, it must have been returned by a previous call |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 112 | to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. If the request fails, |
| 113 | :c:func:`PyMem_Realloc` returns *NULL* and *p* remains a valid pointer to the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 114 | previous memory area. |
| 115 | |
| 116 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 117 | .. c:function:: void PyMem_Free(void *p) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 118 | |
| 119 | Frees the memory block pointed to by *p*, which must have been returned by a |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 120 | previous call to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. Otherwise, or |
Andrew Svetlov | 7dbee38 | 2012-08-09 21:26:34 +0300 | [diff] [blame] | 121 | if ``PyMem_Free(p)`` has been called before, undefined behavior occurs. If |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 122 | *p* is *NULL*, no operation is performed. |
| 123 | |
| 124 | The following type-oriented macros are provided for convenience. Note that |
| 125 | *TYPE* refers to any C type. |
| 126 | |
| 127 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 128 | .. c:function:: TYPE* PyMem_New(TYPE, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 129 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 130 | Same as :c:func:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of |
| 131 | memory. Returns a pointer cast to :c:type:`TYPE\*`. The memory will not have |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 132 | been initialized in any way. |
| 133 | |
| 134 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 135 | .. c:function:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 136 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 137 | Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * |
| 138 | sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, |
Georg Brandl | d492ad8 | 2008-07-23 16:13:07 +0000 | [diff] [blame] | 139 | *p* will be a pointer to the new memory area, or *NULL* in the event of |
| 140 | failure. This is a C preprocessor macro; p is always reassigned. Save |
| 141 | the original value of p to avoid losing memory when handling errors. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 142 | |
| 143 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 144 | .. c:function:: void PyMem_Del(void *p) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 145 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 146 | Same as :c:func:`PyMem_Free`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 147 | |
| 148 | In addition, the following macro sets are provided for calling the Python memory |
| 149 | allocator directly, without involving the C API functions listed above. However, |
| 150 | note that their use does not preserve binary compatibility across Python |
| 151 | versions and is therefore deprecated in extension modules. |
| 152 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 153 | :c:func:`PyMem_MALLOC`, :c:func:`PyMem_REALLOC`, :c:func:`PyMem_FREE`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 154 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 155 | :c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 156 | |
| 157 | |
| 158 | .. _memoryexamples: |
| 159 | |
| 160 | Examples |
| 161 | ======== |
| 162 | |
| 163 | Here is the example from section :ref:`memoryoverview`, rewritten so that the |
| 164 | I/O buffer is allocated from the Python heap by using the first function set:: |
| 165 | |
| 166 | PyObject *res; |
| 167 | char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */ |
| 168 | |
| 169 | if (buf == NULL) |
| 170 | return PyErr_NoMemory(); |
| 171 | /* ...Do some I/O operation involving buf... */ |
| 172 | res = PyString_FromString(buf); |
| 173 | PyMem_Free(buf); /* allocated with PyMem_Malloc */ |
| 174 | return res; |
| 175 | |
| 176 | The same code using the type-oriented function set:: |
| 177 | |
| 178 | PyObject *res; |
| 179 | char *buf = PyMem_New(char, BUFSIZ); /* for I/O */ |
| 180 | |
| 181 | if (buf == NULL) |
| 182 | return PyErr_NoMemory(); |
| 183 | /* ...Do some I/O operation involving buf... */ |
| 184 | res = PyString_FromString(buf); |
| 185 | PyMem_Del(buf); /* allocated with PyMem_New */ |
| 186 | return res; |
| 187 | |
| 188 | Note that in the two examples above, the buffer is always manipulated via |
| 189 | functions belonging to the same set. Indeed, it is required to use the same |
| 190 | memory API family for a given memory block, so that the risk of mixing different |
| 191 | allocators is reduced to a minimum. The following code sequence contains two |
| 192 | errors, one of which is labeled as *fatal* because it mixes two different |
| 193 | allocators operating on different heaps. :: |
| 194 | |
| 195 | char *buf1 = PyMem_New(char, BUFSIZ); |
| 196 | char *buf2 = (char *) malloc(BUFSIZ); |
| 197 | char *buf3 = (char *) PyMem_Malloc(BUFSIZ); |
| 198 | ... |
| 199 | PyMem_Del(buf3); /* Wrong -- should be PyMem_Free() */ |
| 200 | free(buf2); /* Right -- allocated via malloc() */ |
| 201 | free(buf1); /* Fatal -- should be PyMem_Del() */ |
| 202 | |
| 203 | In addition to the functions aimed at handling raw memory blocks from the Python |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 204 | heap, objects in Python are allocated and released with :c:func:`PyObject_New`, |
| 205 | :c:func:`PyObject_NewVar` and :c:func:`PyObject_Del`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 206 | |
| 207 | These will be explained in the next chapter on defining and implementing new |
| 208 | object types in C. |
| 209 | |