Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | .. highlightlang:: c |
| 2 | |
| 3 | |
| 4 | .. _memory: |
| 5 | |
| 6 | ***************** |
| 7 | Memory Management |
| 8 | ***************** |
| 9 | |
| 10 | .. sectionauthor:: Vladimir Marangozov <Vladimir.Marangozov@inrialpes.fr> |
| 11 | |
| 12 | |
| 13 | |
| 14 | .. _memoryoverview: |
| 15 | |
| 16 | Overview |
| 17 | ======== |
| 18 | |
| 19 | Memory management in Python involves a private heap containing all Python |
| 20 | objects and data structures. The management of this private heap is ensured |
| 21 | internally by the *Python memory manager*. The Python memory manager has |
| 22 | different components which deal with various dynamic storage management aspects, |
| 23 | like sharing, segmentation, preallocation or caching. |
| 24 | |
| 25 | At the lowest level, a raw memory allocator ensures that there is enough room in |
| 26 | the private heap for storing all Python-related data by interacting with the |
| 27 | memory manager of the operating system. On top of the raw memory allocator, |
| 28 | several object-specific allocators operate on the same heap and implement |
| 29 | distinct memory management policies adapted to the peculiarities of every object |
| 30 | type. For example, integer objects are managed differently within the heap than |
| 31 | strings, tuples or dictionaries because integers imply different storage |
| 32 | requirements and speed/space tradeoffs. The Python memory manager thus delegates |
| 33 | some of the work to the object-specific allocators, but ensures that the latter |
| 34 | operate within the bounds of the private heap. |
| 35 | |
| 36 | It is important to understand that the management of the Python heap is |
| 37 | performed by the interpreter itself and that the user has no control over it, |
Andrés Delfino | 5092439 | 2018-06-18 01:34:30 -0300 | [diff] [blame^] | 38 | even if they regularly manipulate object pointers to memory blocks inside that |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 39 | heap. The allocation of heap space for Python objects and other internal |
| 40 | buffers is performed on demand by the Python memory manager through the Python/C |
| 41 | API functions listed in this document. |
| 42 | |
| 43 | .. index:: |
| 44 | single: malloc() |
| 45 | single: calloc() |
| 46 | single: realloc() |
| 47 | single: free() |
| 48 | |
| 49 | To avoid memory corruption, extension writers should never try to operate on |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 50 | Python objects with the functions exported by the C library: :c:func:`malloc`, |
| 51 | :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`. This will result in mixed |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 52 | calls between the C allocator and the Python memory manager with fatal |
| 53 | consequences, because they implement different algorithms and operate on |
| 54 | different heaps. However, one may safely allocate and release memory blocks |
| 55 | with the C library allocator for individual purposes, as shown in the following |
| 56 | example:: |
| 57 | |
| 58 | PyObject *res; |
| 59 | char *buf = (char *) malloc(BUFSIZ); /* for I/O */ |
| 60 | |
| 61 | if (buf == NULL) |
| 62 | return PyErr_NoMemory(); |
| 63 | ...Do some I/O operation involving buf... |
Gregory P. Smith | 4b52ae8 | 2013-03-22 13:43:30 -0700 | [diff] [blame] | 64 | res = PyBytes_FromString(buf); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 65 | free(buf); /* malloc'ed */ |
| 66 | return res; |
| 67 | |
| 68 | In this example, the memory request for the I/O buffer is handled by the C |
| 69 | library allocator. The Python memory manager is involved only in the allocation |
| 70 | of the string object returned as a result. |
| 71 | |
| 72 | In most situations, however, it is recommended to allocate memory from the |
| 73 | Python heap specifically because the latter is under control of the Python |
| 74 | memory manager. For example, this is required when the interpreter is extended |
| 75 | with new object types written in C. Another reason for using the Python heap is |
| 76 | the desire to *inform* the Python memory manager about the memory needs of the |
| 77 | extension module. Even when the requested memory is used exclusively for |
| 78 | internal, highly-specific purposes, delegating all memory requests to the Python |
| 79 | memory manager causes the interpreter to have a more accurate image of its |
| 80 | memory footprint as a whole. Consequently, under certain circumstances, the |
| 81 | Python memory manager may or may not trigger appropriate actions, like garbage |
| 82 | collection, memory compaction or other preventive procedures. Note that by using |
| 83 | the C library allocator as shown in the previous example, the allocated memory |
| 84 | for the I/O buffer escapes completely the Python memory manager. |
| 85 | |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 86 | .. seealso:: |
| 87 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 88 | The :envvar:`PYTHONMALLOC` environment variable can be used to configure |
| 89 | the memory allocators used by Python. |
| 90 | |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 91 | The :envvar:`PYTHONMALLOCSTATS` environment variable can be used to print |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 92 | statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a |
| 93 | new pymalloc object arena is created, and on shutdown. |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 94 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 95 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 96 | Raw Memory Interface |
| 97 | ==================== |
| 98 | |
| 99 | The following function sets are wrappers to the system allocator. These |
| 100 | functions are thread-safe, the :term:`GIL <global interpreter lock>` does not |
| 101 | need to be held. |
| 102 | |
Victor Stinner | 5d39e04 | 2017-11-29 17:20:38 +0100 | [diff] [blame] | 103 | The :ref:`default raw memory allocator <default-memory-allocators>` uses |
| 104 | the following functions: :c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` |
| 105 | and :c:func:`free`; call ``malloc(1)`` (or ``calloc(1, 1)``) when requesting |
| 106 | zero bytes. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 107 | |
| 108 | .. versionadded:: 3.4 |
| 109 | |
| 110 | .. c:function:: void* PyMem_RawMalloc(size_t n) |
| 111 | |
| 112 | Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 113 | allocated memory, or *NULL* if the request fails. |
| 114 | |
| 115 | Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as |
| 116 | if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 117 | been initialized in any way. |
| 118 | |
| 119 | |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 120 | .. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize) |
| 121 | |
| 122 | Allocates *nelem* elements each whose size in bytes is *elsize* and returns |
| 123 | a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 124 | request fails. The memory is initialized to zeros. |
| 125 | |
| 126 | Requesting zero elements or elements of size zero bytes returns a distinct |
| 127 | non-*NULL* pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been |
| 128 | called instead. |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 129 | |
| 130 | .. versionadded:: 3.5 |
| 131 | |
| 132 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 133 | .. c:function:: void* PyMem_RawRealloc(void *p, size_t n) |
| 134 | |
| 135 | Resizes the memory block pointed to by *p* to *n* bytes. The contents will |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 136 | be unchanged to the minimum of the old and the new sizes. |
| 137 | |
| 138 | If *p* is *NULL*, the call is equivalent to ``PyMem_RawMalloc(n)``; else if |
| 139 | *n* is equal to zero, the memory block is resized but is not freed, and the |
| 140 | returned pointer is non-*NULL*. |
| 141 | |
| 142 | Unless *p* is *NULL*, it must have been returned by a previous call to |
| 143 | :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or |
| 144 | :c:func:`PyMem_RawCalloc`. |
| 145 | |
| 146 | If the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* |
| 147 | remains a valid pointer to the previous memory area. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 148 | |
| 149 | |
| 150 | .. c:function:: void PyMem_RawFree(void *p) |
| 151 | |
| 152 | Frees the memory block pointed to by *p*, which must have been returned by a |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 153 | previous call to :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or |
Victor Stinner | ec2cbdd | 2017-10-31 09:37:25 -0700 | [diff] [blame] | 154 | :c:func:`PyMem_RawCalloc`. Otherwise, or if ``PyMem_RawFree(p)`` has been |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 155 | called before, undefined behavior occurs. |
| 156 | |
| 157 | If *p* is *NULL*, no operation is performed. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 158 | |
| 159 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 160 | .. _memoryinterface: |
| 161 | |
| 162 | Memory Interface |
| 163 | ================ |
| 164 | |
| 165 | The following function sets, modeled after the ANSI C standard, but specifying |
| 166 | behavior when requesting zero bytes, are available for allocating and releasing |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 167 | memory from the Python heap. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 168 | |
Victor Stinner | 5d39e04 | 2017-11-29 17:20:38 +0100 | [diff] [blame] | 169 | The :ref:`default memory allocator <default-memory-allocators>` uses the |
| 170 | :ref:`pymalloc memory allocator <pymalloc>`. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 171 | |
| 172 | .. warning:: |
| 173 | |
| 174 | The :term:`GIL <global interpreter lock>` must be held when using these |
| 175 | functions. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 176 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 177 | .. versionchanged:: 3.6 |
| 178 | |
| 179 | The default allocator is now pymalloc instead of system :c:func:`malloc`. |
| 180 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 181 | .. c:function:: void* PyMem_Malloc(size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 182 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 183 | Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 184 | allocated memory, or *NULL* if the request fails. |
| 185 | |
| 186 | Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as |
| 187 | if ``PyMem_Malloc(1)`` had been called instead. The memory will not have |
| 188 | been initialized in any way. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 189 | |
| 190 | |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 191 | .. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize) |
| 192 | |
| 193 | Allocates *nelem* elements each whose size in bytes is *elsize* and returns |
| 194 | a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 195 | request fails. The memory is initialized to zeros. |
| 196 | |
| 197 | Requesting zero elements or elements of size zero bytes returns a distinct |
| 198 | non-*NULL* pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called |
| 199 | instead. |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 200 | |
| 201 | .. versionadded:: 3.5 |
| 202 | |
| 203 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 204 | .. c:function:: void* PyMem_Realloc(void *p, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 205 | |
| 206 | Resizes the memory block pointed to by *p* to *n* bytes. The contents will be |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 207 | unchanged to the minimum of the old and the new sizes. |
| 208 | |
| 209 | If *p* is *NULL*, the call is equivalent to ``PyMem_Malloc(n)``; else if *n* |
| 210 | is equal to zero, the memory block is resized but is not freed, and the |
| 211 | returned pointer is non-*NULL*. |
| 212 | |
| 213 | Unless *p* is *NULL*, it must have been returned by a previous call to |
| 214 | :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or :c:func:`PyMem_Calloc`. |
| 215 | |
| 216 | If the request fails, :c:func:`PyMem_Realloc` returns *NULL* and *p* remains |
| 217 | a valid pointer to the previous memory area. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 218 | |
| 219 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 220 | .. c:function:: void PyMem_Free(void *p) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 221 | |
| 222 | Frees the memory block pointed to by *p*, which must have been returned by a |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 223 | previous call to :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or |
| 224 | :c:func:`PyMem_Calloc`. Otherwise, or if ``PyMem_Free(p)`` has been called |
| 225 | before, undefined behavior occurs. |
| 226 | |
| 227 | If *p* is *NULL*, no operation is performed. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 228 | |
| 229 | The following type-oriented macros are provided for convenience. Note that |
| 230 | *TYPE* refers to any C type. |
| 231 | |
| 232 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 233 | .. c:function:: TYPE* PyMem_New(TYPE, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 234 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 235 | Same as :c:func:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of |
| 236 | memory. Returns a pointer cast to :c:type:`TYPE\*`. The memory will not have |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 237 | been initialized in any way. |
| 238 | |
| 239 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 240 | .. c:function:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 241 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 242 | Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * |
| 243 | sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, |
Georg Brandl | d492ad8 | 2008-07-23 16:13:07 +0000 | [diff] [blame] | 244 | *p* will be a pointer to the new memory area, or *NULL* in the event of |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 245 | failure. |
| 246 | |
| 247 | This is a C preprocessor macro; *p* is always reassigned. Save the original |
| 248 | value of *p* to avoid losing memory when handling errors. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 249 | |
| 250 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 251 | .. c:function:: void PyMem_Del(void *p) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 252 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 253 | Same as :c:func:`PyMem_Free`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 254 | |
| 255 | In addition, the following macro sets are provided for calling the Python memory |
| 256 | allocator directly, without involving the C API functions listed above. However, |
| 257 | note that their use does not preserve binary compatibility across Python |
| 258 | versions and is therefore deprecated in extension modules. |
| 259 | |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 260 | * ``PyMem_MALLOC(size)`` |
| 261 | * ``PyMem_NEW(type, size)`` |
| 262 | * ``PyMem_REALLOC(ptr, size)`` |
| 263 | * ``PyMem_RESIZE(ptr, type, size)`` |
| 264 | * ``PyMem_FREE(ptr)`` |
| 265 | * ``PyMem_DEL(ptr)`` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 266 | |
| 267 | |
Victor Stinner | ec2cbdd | 2017-10-31 09:37:25 -0700 | [diff] [blame] | 268 | Object allocators |
| 269 | ================= |
| 270 | |
| 271 | The following function sets, modeled after the ANSI C standard, but specifying |
| 272 | behavior when requesting zero bytes, are available for allocating and releasing |
| 273 | memory from the Python heap. |
| 274 | |
Victor Stinner | 5d39e04 | 2017-11-29 17:20:38 +0100 | [diff] [blame] | 275 | The :ref:`default object allocator <default-memory-allocators>` uses the |
| 276 | :ref:`pymalloc memory allocator <pymalloc>`. |
Victor Stinner | ec2cbdd | 2017-10-31 09:37:25 -0700 | [diff] [blame] | 277 | |
| 278 | .. warning:: |
| 279 | |
| 280 | The :term:`GIL <global interpreter lock>` must be held when using these |
| 281 | functions. |
| 282 | |
| 283 | .. c:function:: void* PyObject_Malloc(size_t n) |
| 284 | |
| 285 | Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the |
| 286 | allocated memory, or *NULL* if the request fails. |
| 287 | |
| 288 | Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as |
| 289 | if ``PyObject_Malloc(1)`` had been called instead. The memory will not have |
| 290 | been initialized in any way. |
| 291 | |
| 292 | |
| 293 | .. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize) |
| 294 | |
| 295 | Allocates *nelem* elements each whose size in bytes is *elsize* and returns |
| 296 | a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the |
| 297 | request fails. The memory is initialized to zeros. |
| 298 | |
| 299 | Requesting zero elements or elements of size zero bytes returns a distinct |
| 300 | non-*NULL* pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called |
| 301 | instead. |
| 302 | |
| 303 | .. versionadded:: 3.5 |
| 304 | |
| 305 | |
| 306 | .. c:function:: void* PyObject_Realloc(void *p, size_t n) |
| 307 | |
| 308 | Resizes the memory block pointed to by *p* to *n* bytes. The contents will be |
| 309 | unchanged to the minimum of the old and the new sizes. |
| 310 | |
| 311 | If *p* is *NULL*, the call is equivalent to ``PyObject_Malloc(n)``; else if *n* |
| 312 | is equal to zero, the memory block is resized but is not freed, and the |
| 313 | returned pointer is non-*NULL*. |
| 314 | |
| 315 | Unless *p* is *NULL*, it must have been returned by a previous call to |
| 316 | :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or :c:func:`PyObject_Calloc`. |
| 317 | |
| 318 | If the request fails, :c:func:`PyObject_Realloc` returns *NULL* and *p* remains |
| 319 | a valid pointer to the previous memory area. |
| 320 | |
| 321 | |
| 322 | .. c:function:: void PyObject_Free(void *p) |
| 323 | |
| 324 | Frees the memory block pointed to by *p*, which must have been returned by a |
| 325 | previous call to :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or |
| 326 | :c:func:`PyObject_Calloc`. Otherwise, or if ``PyObject_Free(p)`` has been called |
| 327 | before, undefined behavior occurs. |
| 328 | |
| 329 | If *p* is *NULL*, no operation is performed. |
| 330 | |
| 331 | |
Victor Stinner | 5d39e04 | 2017-11-29 17:20:38 +0100 | [diff] [blame] | 332 | .. _default-memory-allocators: |
| 333 | |
| 334 | Default Memory Allocators |
| 335 | ========================= |
| 336 | |
| 337 | Default memory allocators: |
| 338 | |
| 339 | =============================== ==================== ================== ===================== ==================== |
| 340 | Configuration Name PyMem_RawMalloc PyMem_Malloc PyObject_Malloc |
| 341 | =============================== ==================== ================== ===================== ==================== |
| 342 | Release build ``"pymalloc"`` ``malloc`` ``pymalloc`` ``pymalloc`` |
| 343 | Debug build ``"pymalloc_debug"`` ``malloc`` + debug ``pymalloc`` + debug ``pymalloc`` + debug |
| 344 | Release build, without pymalloc ``"malloc"`` ``malloc`` ``malloc`` ``malloc`` |
| 345 | Release build, without pymalloc ``"malloc_debug"`` ``malloc`` + debug ``malloc`` + debug ``malloc`` + debug |
| 346 | =============================== ==================== ================== ===================== ==================== |
| 347 | |
| 348 | Legend: |
| 349 | |
| 350 | * Name: value for :envvar:`PYTHONMALLOC` environment variable |
| 351 | * ``malloc``: system allocators from the standard C library, C functions: |
| 352 | :c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free` |
| 353 | * ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>` |
| 354 | * "+ debug": with debug hooks installed by :c:func:`PyMem_SetupDebugHooks` |
| 355 | |
| 356 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 357 | Customize Memory Allocators |
| 358 | =========================== |
| 359 | |
| 360 | .. versionadded:: 3.4 |
| 361 | |
Victor Stinner | d8f0d92 | 2014-06-02 21:57:10 +0200 | [diff] [blame] | 362 | .. c:type:: PyMemAllocatorEx |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 363 | |
| 364 | Structure used to describe a memory block allocator. The structure has |
| 365 | four fields: |
| 366 | |
| 367 | +----------------------------------------------------------+---------------------------------------+ |
| 368 | | Field | Meaning | |
| 369 | +==========================================================+=======================================+ |
| 370 | | ``void *ctx`` | user context passed as first argument | |
| 371 | +----------------------------------------------------------+---------------------------------------+ |
| 372 | | ``void* malloc(void *ctx, size_t size)`` | allocate a memory block | |
| 373 | +----------------------------------------------------------+---------------------------------------+ |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 374 | | ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized | |
| 375 | | | with zeros | |
| 376 | +----------------------------------------------------------+---------------------------------------+ |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 377 | | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block | |
| 378 | +----------------------------------------------------------+---------------------------------------+ |
| 379 | | ``void free(void *ctx, void *ptr)`` | free a memory block | |
| 380 | +----------------------------------------------------------+---------------------------------------+ |
| 381 | |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 382 | .. versionchanged:: 3.5 |
Victor Stinner | d8f0d92 | 2014-06-02 21:57:10 +0200 | [diff] [blame] | 383 | The :c:type:`PyMemAllocator` structure was renamed to |
| 384 | :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. |
| 385 | |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 386 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 387 | .. c:type:: PyMemAllocatorDomain |
| 388 | |
| 389 | Enum used to identify an allocator domain. Domains: |
| 390 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 391 | .. c:var:: PYMEM_DOMAIN_RAW |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 392 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 393 | Functions: |
| 394 | |
| 395 | * :c:func:`PyMem_RawMalloc` |
| 396 | * :c:func:`PyMem_RawRealloc` |
| 397 | * :c:func:`PyMem_RawCalloc` |
| 398 | * :c:func:`PyMem_RawFree` |
| 399 | |
| 400 | .. c:var:: PYMEM_DOMAIN_MEM |
| 401 | |
| 402 | Functions: |
| 403 | |
| 404 | * :c:func:`PyMem_Malloc`, |
| 405 | * :c:func:`PyMem_Realloc` |
| 406 | * :c:func:`PyMem_Calloc` |
| 407 | * :c:func:`PyMem_Free` |
| 408 | |
| 409 | .. c:var:: PYMEM_DOMAIN_OBJ |
| 410 | |
| 411 | Functions: |
| 412 | |
| 413 | * :c:func:`PyObject_Malloc` |
| 414 | * :c:func:`PyObject_Realloc` |
| 415 | * :c:func:`PyObject_Calloc` |
| 416 | * :c:func:`PyObject_Free` |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 417 | |
Victor Stinner | d8f0d92 | 2014-06-02 21:57:10 +0200 | [diff] [blame] | 418 | .. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 419 | |
| 420 | Get the memory block allocator of the specified domain. |
| 421 | |
| 422 | |
Victor Stinner | d8f0d92 | 2014-06-02 21:57:10 +0200 | [diff] [blame] | 423 | .. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 424 | |
| 425 | Set the memory block allocator of the specified domain. |
| 426 | |
| 427 | The new allocator must return a distinct non-NULL pointer when requesting |
| 428 | zero bytes. |
| 429 | |
| 430 | For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be |
| 431 | thread-safe: the :term:`GIL <global interpreter lock>` is not held when the |
| 432 | allocator is called. |
| 433 | |
| 434 | If the new allocator is not a hook (does not call the previous allocator), |
| 435 | the :c:func:`PyMem_SetupDebugHooks` function must be called to reinstall the |
| 436 | debug hooks on top on the new allocator. |
| 437 | |
| 438 | |
| 439 | .. c:function:: void PyMem_SetupDebugHooks(void) |
| 440 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 441 | Setup hooks to detect bugs in the Python memory allocator functions. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 442 | |
| 443 | Newly allocated memory is filled with the byte ``0xCB``, freed memory is |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 444 | filled with the byte ``0xDB``. |
| 445 | |
| 446 | Runtime checks: |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 447 | |
Victor Stinner | c4aec36 | 2016-03-14 22:26:53 +0100 | [diff] [blame] | 448 | - Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 449 | allocated by :c:func:`PyMem_Malloc` |
Victor Stinner | c4aec36 | 2016-03-14 22:26:53 +0100 | [diff] [blame] | 450 | - Detect write before the start of the buffer (buffer underflow) |
| 451 | - Detect write after the end of the buffer (buffer overflow) |
| 452 | - Check that the :term:`GIL <global interpreter lock>` is held when |
Victor Stinner | c2fc568 | 2016-03-18 11:04:31 +0100 | [diff] [blame] | 453 | allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: |
| 454 | :c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex: |
| 455 | :c:func:`PyMem_Malloc`) domains are called |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 456 | |
Victor Stinner | 0611c26 | 2016-03-15 22:22:13 +0100 | [diff] [blame] | 457 | On error, the debug hooks use the :mod:`tracemalloc` module to get the |
| 458 | traceback where a memory block was allocated. The traceback is only |
| 459 | displayed if :mod:`tracemalloc` is tracing Python memory allocations and the |
| 460 | memory block was traced. |
| 461 | |
Victor Stinner | 5d39e04 | 2017-11-29 17:20:38 +0100 | [diff] [blame] | 462 | These hooks are :ref:`installed by default <default-memory-allocators>` if |
| 463 | Python is compiled in debug |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 464 | mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install |
| 465 | debug hooks on a Python compiled in release mode. |
| 466 | |
| 467 | .. versionchanged:: 3.6 |
| 468 | This function now also works on Python compiled in release mode. |
Victor Stinner | 0611c26 | 2016-03-15 22:22:13 +0100 | [diff] [blame] | 469 | On error, the debug hooks now use :mod:`tracemalloc` to get the traceback |
Victor Stinner | c2fc568 | 2016-03-18 11:04:31 +0100 | [diff] [blame] | 470 | where a memory block was allocated. The debug hooks now also check |
Victor Stinner | 9b46a57 | 2016-03-18 15:10:43 +0100 | [diff] [blame] | 471 | if the GIL is held when functions of :c:data:`PYMEM_DOMAIN_OBJ` and |
Victor Stinner | c2fc568 | 2016-03-18 11:04:31 +0100 | [diff] [blame] | 472 | :c:data:`PYMEM_DOMAIN_MEM` domains are called. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 473 | |
| 474 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 475 | .. _pymalloc: |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 476 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 477 | The pymalloc allocator |
| 478 | ====================== |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 479 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 480 | Python has a *pymalloc* allocator optimized for small objects (smaller or equal |
| 481 | to 512 bytes) with a short lifetime. It uses memory mappings called "arenas" |
Victor Stinner | 8c663fd | 2017-11-08 14:44:44 -0800 | [diff] [blame] | 482 | with a fixed size of 256 KiB. It falls back to :c:func:`PyMem_RawMalloc` and |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 483 | :c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes. |
| 484 | |
Victor Stinner | 5d39e04 | 2017-11-29 17:20:38 +0100 | [diff] [blame] | 485 | *pymalloc* is the :ref:`default allocator <default-memory-allocators>` of the |
| 486 | :c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) and |
| 487 | :c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) domains. |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 488 | |
| 489 | The arena allocator uses the following functions: |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 490 | |
| 491 | * :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows, |
| 492 | * :c:func:`mmap` and :c:func:`munmap` if available, |
| 493 | * :c:func:`malloc` and :c:func:`free` otherwise. |
| 494 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 495 | Customize pymalloc Arena Allocator |
| 496 | ---------------------------------- |
| 497 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 498 | .. versionadded:: 3.4 |
| 499 | |
| 500 | .. c:type:: PyObjectArenaAllocator |
| 501 | |
| 502 | Structure used to describe an arena allocator. The structure has |
| 503 | three fields: |
| 504 | |
| 505 | +--------------------------------------------------+---------------------------------------+ |
| 506 | | Field | Meaning | |
| 507 | +==================================================+=======================================+ |
| 508 | | ``void *ctx`` | user context passed as first argument | |
| 509 | +--------------------------------------------------+---------------------------------------+ |
| 510 | | ``void* alloc(void *ctx, size_t size)`` | allocate an arena of size bytes | |
| 511 | +--------------------------------------------------+---------------------------------------+ |
| 512 | | ``void free(void *ctx, size_t size, void *ptr)`` | free an arena | |
| 513 | +--------------------------------------------------+---------------------------------------+ |
| 514 | |
| 515 | .. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator) |
| 516 | |
| 517 | Get the arena allocator. |
| 518 | |
| 519 | .. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) |
| 520 | |
| 521 | Set the arena allocator. |
| 522 | |
| 523 | |
Victor Stinner | 5ea4c06 | 2017-06-20 17:46:36 +0200 | [diff] [blame] | 524 | tracemalloc C API |
| 525 | ================= |
| 526 | |
| 527 | .. versionadded:: 3.7 |
| 528 | |
| 529 | .. c:function: int PyTraceMalloc_Track(unsigned int domain, uintptr_t ptr, size_t size) |
| 530 | |
| 531 | Track an allocated memory block in the :mod:`tracemalloc` module. |
| 532 | |
Serhiy Storchaka | 5bb0005 | 2018-02-09 13:31:19 +0200 | [diff] [blame] | 533 | Return ``0`` on success, return ``-1`` on error (failed to allocate memory to |
Victor Stinner | 5ea4c06 | 2017-06-20 17:46:36 +0200 | [diff] [blame] | 534 | store the trace). Return ``-2`` if tracemalloc is disabled. |
| 535 | |
| 536 | If memory block is already tracked, update the existing trace. |
| 537 | |
| 538 | .. c:function: int PyTraceMalloc_Untrack(unsigned int domain, uintptr_t ptr) |
| 539 | |
| 540 | Untrack an allocated memory block in the :mod:`tracemalloc` module. |
| 541 | Do nothing if the block was not tracked. |
| 542 | |
| 543 | Return ``-2`` if tracemalloc is disabled, otherwise return ``0``. |
| 544 | |
| 545 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 546 | .. _memoryexamples: |
| 547 | |
| 548 | Examples |
| 549 | ======== |
| 550 | |
| 551 | Here is the example from section :ref:`memoryoverview`, rewritten so that the |
| 552 | I/O buffer is allocated from the Python heap by using the first function set:: |
| 553 | |
| 554 | PyObject *res; |
| 555 | char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */ |
| 556 | |
| 557 | if (buf == NULL) |
| 558 | return PyErr_NoMemory(); |
| 559 | /* ...Do some I/O operation involving buf... */ |
Gregory P. Smith | 4b52ae8 | 2013-03-22 13:43:30 -0700 | [diff] [blame] | 560 | res = PyBytes_FromString(buf); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 561 | PyMem_Free(buf); /* allocated with PyMem_Malloc */ |
| 562 | return res; |
| 563 | |
| 564 | The same code using the type-oriented function set:: |
| 565 | |
| 566 | PyObject *res; |
| 567 | char *buf = PyMem_New(char, BUFSIZ); /* for I/O */ |
| 568 | |
| 569 | if (buf == NULL) |
| 570 | return PyErr_NoMemory(); |
| 571 | /* ...Do some I/O operation involving buf... */ |
Gregory P. Smith | 4b52ae8 | 2013-03-22 13:43:30 -0700 | [diff] [blame] | 572 | res = PyBytes_FromString(buf); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 573 | PyMem_Del(buf); /* allocated with PyMem_New */ |
| 574 | return res; |
| 575 | |
| 576 | Note that in the two examples above, the buffer is always manipulated via |
| 577 | functions belonging to the same set. Indeed, it is required to use the same |
| 578 | memory API family for a given memory block, so that the risk of mixing different |
| 579 | allocators is reduced to a minimum. The following code sequence contains two |
| 580 | errors, one of which is labeled as *fatal* because it mixes two different |
| 581 | allocators operating on different heaps. :: |
| 582 | |
| 583 | char *buf1 = PyMem_New(char, BUFSIZ); |
| 584 | char *buf2 = (char *) malloc(BUFSIZ); |
| 585 | char *buf3 = (char *) PyMem_Malloc(BUFSIZ); |
| 586 | ... |
| 587 | PyMem_Del(buf3); /* Wrong -- should be PyMem_Free() */ |
| 588 | free(buf2); /* Right -- allocated via malloc() */ |
| 589 | free(buf1); /* Fatal -- should be PyMem_Del() */ |
| 590 | |
| 591 | In addition to the functions aimed at handling raw memory blocks from the Python |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 592 | heap, objects in Python are allocated and released with :c:func:`PyObject_New`, |
| 593 | :c:func:`PyObject_NewVar` and :c:func:`PyObject_Del`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 594 | |
| 595 | These will be explained in the next chapter on defining and implementing new |
| 596 | object types in C. |
| 597 | |