Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | .. highlightlang:: c |
| 2 | |
| 3 | |
| 4 | .. _memory: |
| 5 | |
| 6 | ***************** |
| 7 | Memory Management |
| 8 | ***************** |
| 9 | |
| 10 | .. sectionauthor:: Vladimir Marangozov <Vladimir.Marangozov@inrialpes.fr> |
| 11 | |
| 12 | |
| 13 | |
| 14 | .. _memoryoverview: |
| 15 | |
| 16 | Overview |
| 17 | ======== |
| 18 | |
| 19 | Memory management in Python involves a private heap containing all Python |
| 20 | objects and data structures. The management of this private heap is ensured |
| 21 | internally by the *Python memory manager*. The Python memory manager has |
| 22 | different components which deal with various dynamic storage management aspects, |
| 23 | like sharing, segmentation, preallocation or caching. |
| 24 | |
| 25 | At the lowest level, a raw memory allocator ensures that there is enough room in |
| 26 | the private heap for storing all Python-related data by interacting with the |
| 27 | memory manager of the operating system. On top of the raw memory allocator, |
| 28 | several object-specific allocators operate on the same heap and implement |
| 29 | distinct memory management policies adapted to the peculiarities of every object |
| 30 | type. For example, integer objects are managed differently within the heap than |
| 31 | strings, tuples or dictionaries because integers imply different storage |
| 32 | requirements and speed/space tradeoffs. The Python memory manager thus delegates |
| 33 | some of the work to the object-specific allocators, but ensures that the latter |
| 34 | operate within the bounds of the private heap. |
| 35 | |
| 36 | It is important to understand that the management of the Python heap is |
| 37 | performed by the interpreter itself and that the user has no control over it, |
| 38 | even if she regularly manipulates object pointers to memory blocks inside that |
| 39 | heap. The allocation of heap space for Python objects and other internal |
| 40 | buffers is performed on demand by the Python memory manager through the Python/C |
| 41 | API functions listed in this document. |
| 42 | |
| 43 | .. index:: |
| 44 | single: malloc() |
| 45 | single: calloc() |
| 46 | single: realloc() |
| 47 | single: free() |
| 48 | |
| 49 | To avoid memory corruption, extension writers should never try to operate on |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 50 | Python objects with the functions exported by the C library: :c:func:`malloc`, |
| 51 | :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`. This will result in mixed |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 52 | calls between the C allocator and the Python memory manager with fatal |
| 53 | consequences, because they implement different algorithms and operate on |
| 54 | different heaps. However, one may safely allocate and release memory blocks |
| 55 | with the C library allocator for individual purposes, as shown in the following |
| 56 | example:: |
| 57 | |
| 58 | PyObject *res; |
| 59 | char *buf = (char *) malloc(BUFSIZ); /* for I/O */ |
| 60 | |
| 61 | if (buf == NULL) |
| 62 | return PyErr_NoMemory(); |
| 63 | ...Do some I/O operation involving buf... |
Gregory P. Smith | 4b52ae8 | 2013-03-22 13:43:30 -0700 | [diff] [blame] | 64 | res = PyBytes_FromString(buf); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 65 | free(buf); /* malloc'ed */ |
| 66 | return res; |
| 67 | |
| 68 | In this example, the memory request for the I/O buffer is handled by the C |
| 69 | library allocator. The Python memory manager is involved only in the allocation |
| 70 | of the string object returned as a result. |
| 71 | |
| 72 | In most situations, however, it is recommended to allocate memory from the |
| 73 | Python heap specifically because the latter is under control of the Python |
| 74 | memory manager. For example, this is required when the interpreter is extended |
| 75 | with new object types written in C. Another reason for using the Python heap is |
| 76 | the desire to *inform* the Python memory manager about the memory needs of the |
| 77 | extension module. Even when the requested memory is used exclusively for |
| 78 | internal, highly-specific purposes, delegating all memory requests to the Python |
| 79 | memory manager causes the interpreter to have a more accurate image of its |
| 80 | memory footprint as a whole. Consequently, under certain circumstances, the |
| 81 | Python memory manager may or may not trigger appropriate actions, like garbage |
| 82 | collection, memory compaction or other preventive procedures. Note that by using |
| 83 | the C library allocator as shown in the previous example, the allocated memory |
| 84 | for the I/O buffer escapes completely the Python memory manager. |
| 85 | |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 86 | .. seealso:: |
| 87 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 88 | The :envvar:`PYTHONMALLOC` environment variable can be used to configure |
| 89 | the memory allocators used by Python. |
| 90 | |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 91 | The :envvar:`PYTHONMALLOCSTATS` environment variable can be used to print |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 92 | statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a |
| 93 | new pymalloc object arena is created, and on shutdown. |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 94 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 95 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 96 | Raw Memory Interface |
| 97 | ==================== |
| 98 | |
| 99 | The following function sets are wrappers to the system allocator. These |
| 100 | functions are thread-safe, the :term:`GIL <global interpreter lock>` does not |
| 101 | need to be held. |
| 102 | |
| 103 | The default raw memory block allocator uses the following functions: |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 104 | :c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call |
| 105 | ``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 106 | |
| 107 | .. versionadded:: 3.4 |
| 108 | |
| 109 | .. c:function:: void* PyMem_RawMalloc(size_t n) |
| 110 | |
| 111 | Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 112 | allocated memory, or *NULL* if the request fails. |
| 113 | |
| 114 | Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as |
| 115 | if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 116 | been initialized in any way. |
| 117 | |
| 118 | |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 119 | .. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize) |
| 120 | |
| 121 | Allocates *nelem* elements each whose size in bytes is *elsize* and returns |
| 122 | a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 123 | request fails. The memory is initialized to zeros. |
| 124 | |
| 125 | Requesting zero elements or elements of size zero bytes returns a distinct |
| 126 | non-*NULL* pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been |
| 127 | called instead. |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 128 | |
| 129 | .. versionadded:: 3.5 |
| 130 | |
| 131 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 132 | .. c:function:: void* PyMem_RawRealloc(void *p, size_t n) |
| 133 | |
| 134 | Resizes the memory block pointed to by *p* to *n* bytes. The contents will |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 135 | be unchanged to the minimum of the old and the new sizes. |
| 136 | |
| 137 | If *p* is *NULL*, the call is equivalent to ``PyMem_RawMalloc(n)``; else if |
| 138 | *n* is equal to zero, the memory block is resized but is not freed, and the |
| 139 | returned pointer is non-*NULL*. |
| 140 | |
| 141 | Unless *p* is *NULL*, it must have been returned by a previous call to |
| 142 | :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or |
| 143 | :c:func:`PyMem_RawCalloc`. |
| 144 | |
| 145 | If the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* |
| 146 | remains a valid pointer to the previous memory area. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 147 | |
| 148 | |
| 149 | .. c:function:: void PyMem_RawFree(void *p) |
| 150 | |
| 151 | Frees the memory block pointed to by *p*, which must have been returned by a |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 152 | previous call to :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or |
Victor Stinner | ec2cbdd | 2017-10-31 09:37:25 -0700 | [diff] [blame^] | 153 | :c:func:`PyMem_RawCalloc`. Otherwise, or if ``PyMem_RawFree(p)`` has been |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 154 | called before, undefined behavior occurs. |
| 155 | |
| 156 | If *p* is *NULL*, no operation is performed. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 157 | |
| 158 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 159 | .. _memoryinterface: |
| 160 | |
| 161 | Memory Interface |
| 162 | ================ |
| 163 | |
| 164 | The following function sets, modeled after the ANSI C standard, but specifying |
| 165 | behavior when requesting zero bytes, are available for allocating and releasing |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 166 | memory from the Python heap. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 167 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 168 | By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 169 | |
| 170 | .. warning:: |
| 171 | |
| 172 | The :term:`GIL <global interpreter lock>` must be held when using these |
| 173 | functions. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 174 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 175 | .. versionchanged:: 3.6 |
| 176 | |
| 177 | The default allocator is now pymalloc instead of system :c:func:`malloc`. |
| 178 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 179 | .. c:function:: void* PyMem_Malloc(size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 180 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 181 | Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 182 | allocated memory, or *NULL* if the request fails. |
| 183 | |
| 184 | Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as |
| 185 | if ``PyMem_Malloc(1)`` had been called instead. The memory will not have |
| 186 | been initialized in any way. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 187 | |
| 188 | |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 189 | .. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize) |
| 190 | |
| 191 | Allocates *nelem* elements each whose size in bytes is *elsize* and returns |
| 192 | a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 193 | request fails. The memory is initialized to zeros. |
| 194 | |
| 195 | Requesting zero elements or elements of size zero bytes returns a distinct |
| 196 | non-*NULL* pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called |
| 197 | instead. |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 198 | |
| 199 | .. versionadded:: 3.5 |
| 200 | |
| 201 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 202 | .. c:function:: void* PyMem_Realloc(void *p, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 203 | |
| 204 | Resizes the memory block pointed to by *p* to *n* bytes. The contents will be |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 205 | unchanged to the minimum of the old and the new sizes. |
| 206 | |
| 207 | If *p* is *NULL*, the call is equivalent to ``PyMem_Malloc(n)``; else if *n* |
| 208 | is equal to zero, the memory block is resized but is not freed, and the |
| 209 | returned pointer is non-*NULL*. |
| 210 | |
| 211 | Unless *p* is *NULL*, it must have been returned by a previous call to |
| 212 | :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or :c:func:`PyMem_Calloc`. |
| 213 | |
| 214 | If the request fails, :c:func:`PyMem_Realloc` returns *NULL* and *p* remains |
| 215 | a valid pointer to the previous memory area. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 216 | |
| 217 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 218 | .. c:function:: void PyMem_Free(void *p) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 219 | |
| 220 | Frees the memory block pointed to by *p*, which must have been returned by a |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 221 | previous call to :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or |
| 222 | :c:func:`PyMem_Calloc`. Otherwise, or if ``PyMem_Free(p)`` has been called |
| 223 | before, undefined behavior occurs. |
| 224 | |
| 225 | If *p* is *NULL*, no operation is performed. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 226 | |
| 227 | The following type-oriented macros are provided for convenience. Note that |
| 228 | *TYPE* refers to any C type. |
| 229 | |
| 230 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 231 | .. c:function:: TYPE* PyMem_New(TYPE, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 232 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 233 | Same as :c:func:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of |
| 234 | memory. Returns a pointer cast to :c:type:`TYPE\*`. The memory will not have |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 235 | been initialized in any way. |
| 236 | |
| 237 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 238 | .. c:function:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 239 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 240 | Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * |
| 241 | sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, |
Georg Brandl | d492ad8 | 2008-07-23 16:13:07 +0000 | [diff] [blame] | 242 | *p* will be a pointer to the new memory area, or *NULL* in the event of |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 243 | failure. |
| 244 | |
| 245 | This is a C preprocessor macro; *p* is always reassigned. Save the original |
| 246 | value of *p* to avoid losing memory when handling errors. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 247 | |
| 248 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 249 | .. c:function:: void PyMem_Del(void *p) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 250 | |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 251 | Same as :c:func:`PyMem_Free`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 252 | |
| 253 | In addition, the following macro sets are provided for calling the Python memory |
| 254 | allocator directly, without involving the C API functions listed above. However, |
| 255 | note that their use does not preserve binary compatibility across Python |
| 256 | versions and is therefore deprecated in extension modules. |
| 257 | |
Victor Stinner | 29bf27f | 2016-03-09 14:49:52 +0100 | [diff] [blame] | 258 | * ``PyMem_MALLOC(size)`` |
| 259 | * ``PyMem_NEW(type, size)`` |
| 260 | * ``PyMem_REALLOC(ptr, size)`` |
| 261 | * ``PyMem_RESIZE(ptr, type, size)`` |
| 262 | * ``PyMem_FREE(ptr)`` |
| 263 | * ``PyMem_DEL(ptr)`` |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 264 | |
| 265 | |
Victor Stinner | ec2cbdd | 2017-10-31 09:37:25 -0700 | [diff] [blame^] | 266 | Object allocators |
| 267 | ================= |
| 268 | |
| 269 | The following function sets, modeled after the ANSI C standard, but specifying |
| 270 | behavior when requesting zero bytes, are available for allocating and releasing |
| 271 | memory from the Python heap. |
| 272 | |
| 273 | By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`. |
| 274 | |
| 275 | .. warning:: |
| 276 | |
| 277 | The :term:`GIL <global interpreter lock>` must be held when using these |
| 278 | functions. |
| 279 | |
| 280 | .. c:function:: void* PyObject_Malloc(size_t n) |
| 281 | |
| 282 | Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the |
| 283 | allocated memory, or *NULL* if the request fails. |
| 284 | |
| 285 | Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as |
| 286 | if ``PyObject_Malloc(1)`` had been called instead. The memory will not have |
| 287 | been initialized in any way. |
| 288 | |
| 289 | |
| 290 | .. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize) |
| 291 | |
| 292 | Allocates *nelem* elements each whose size in bytes is *elsize* and returns |
| 293 | a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the |
| 294 | request fails. The memory is initialized to zeros. |
| 295 | |
| 296 | Requesting zero elements or elements of size zero bytes returns a distinct |
| 297 | non-*NULL* pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called |
| 298 | instead. |
| 299 | |
| 300 | .. versionadded:: 3.5 |
| 301 | |
| 302 | |
| 303 | .. c:function:: void* PyObject_Realloc(void *p, size_t n) |
| 304 | |
| 305 | Resizes the memory block pointed to by *p* to *n* bytes. The contents will be |
| 306 | unchanged to the minimum of the old and the new sizes. |
| 307 | |
| 308 | If *p* is *NULL*, the call is equivalent to ``PyObject_Malloc(n)``; else if *n* |
| 309 | is equal to zero, the memory block is resized but is not freed, and the |
| 310 | returned pointer is non-*NULL*. |
| 311 | |
| 312 | Unless *p* is *NULL*, it must have been returned by a previous call to |
| 313 | :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or :c:func:`PyObject_Calloc`. |
| 314 | |
| 315 | If the request fails, :c:func:`PyObject_Realloc` returns *NULL* and *p* remains |
| 316 | a valid pointer to the previous memory area. |
| 317 | |
| 318 | |
| 319 | .. c:function:: void PyObject_Free(void *p) |
| 320 | |
| 321 | Frees the memory block pointed to by *p*, which must have been returned by a |
| 322 | previous call to :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or |
| 323 | :c:func:`PyObject_Calloc`. Otherwise, or if ``PyObject_Free(p)`` has been called |
| 324 | before, undefined behavior occurs. |
| 325 | |
| 326 | If *p* is *NULL*, no operation is performed. |
| 327 | |
| 328 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 329 | Customize Memory Allocators |
| 330 | =========================== |
| 331 | |
| 332 | .. versionadded:: 3.4 |
| 333 | |
Victor Stinner | d8f0d92 | 2014-06-02 21:57:10 +0200 | [diff] [blame] | 334 | .. c:type:: PyMemAllocatorEx |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 335 | |
| 336 | Structure used to describe a memory block allocator. The structure has |
| 337 | four fields: |
| 338 | |
| 339 | +----------------------------------------------------------+---------------------------------------+ |
| 340 | | Field | Meaning | |
| 341 | +==========================================================+=======================================+ |
| 342 | | ``void *ctx`` | user context passed as first argument | |
| 343 | +----------------------------------------------------------+---------------------------------------+ |
| 344 | | ``void* malloc(void *ctx, size_t size)`` | allocate a memory block | |
| 345 | +----------------------------------------------------------+---------------------------------------+ |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 346 | | ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized | |
| 347 | | | with zeros | |
| 348 | +----------------------------------------------------------+---------------------------------------+ |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 349 | | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block | |
| 350 | +----------------------------------------------------------+---------------------------------------+ |
| 351 | | ``void free(void *ctx, void *ptr)`` | free a memory block | |
| 352 | +----------------------------------------------------------+---------------------------------------+ |
| 353 | |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 354 | .. versionchanged:: 3.5 |
Victor Stinner | d8f0d92 | 2014-06-02 21:57:10 +0200 | [diff] [blame] | 355 | The :c:type:`PyMemAllocator` structure was renamed to |
| 356 | :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. |
| 357 | |
Victor Stinner | db067af | 2014-05-02 22:31:14 +0200 | [diff] [blame] | 358 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 359 | .. c:type:: PyMemAllocatorDomain |
| 360 | |
| 361 | Enum used to identify an allocator domain. Domains: |
| 362 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 363 | .. c:var:: PYMEM_DOMAIN_RAW |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 364 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 365 | Functions: |
| 366 | |
| 367 | * :c:func:`PyMem_RawMalloc` |
| 368 | * :c:func:`PyMem_RawRealloc` |
| 369 | * :c:func:`PyMem_RawCalloc` |
| 370 | * :c:func:`PyMem_RawFree` |
| 371 | |
| 372 | .. c:var:: PYMEM_DOMAIN_MEM |
| 373 | |
| 374 | Functions: |
| 375 | |
| 376 | * :c:func:`PyMem_Malloc`, |
| 377 | * :c:func:`PyMem_Realloc` |
| 378 | * :c:func:`PyMem_Calloc` |
| 379 | * :c:func:`PyMem_Free` |
| 380 | |
| 381 | .. c:var:: PYMEM_DOMAIN_OBJ |
| 382 | |
| 383 | Functions: |
| 384 | |
| 385 | * :c:func:`PyObject_Malloc` |
| 386 | * :c:func:`PyObject_Realloc` |
| 387 | * :c:func:`PyObject_Calloc` |
| 388 | * :c:func:`PyObject_Free` |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 389 | |
Victor Stinner | d8f0d92 | 2014-06-02 21:57:10 +0200 | [diff] [blame] | 390 | .. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 391 | |
| 392 | Get the memory block allocator of the specified domain. |
| 393 | |
| 394 | |
Victor Stinner | d8f0d92 | 2014-06-02 21:57:10 +0200 | [diff] [blame] | 395 | .. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 396 | |
| 397 | Set the memory block allocator of the specified domain. |
| 398 | |
| 399 | The new allocator must return a distinct non-NULL pointer when requesting |
| 400 | zero bytes. |
| 401 | |
| 402 | For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be |
| 403 | thread-safe: the :term:`GIL <global interpreter lock>` is not held when the |
| 404 | allocator is called. |
| 405 | |
| 406 | If the new allocator is not a hook (does not call the previous allocator), |
| 407 | the :c:func:`PyMem_SetupDebugHooks` function must be called to reinstall the |
| 408 | debug hooks on top on the new allocator. |
| 409 | |
| 410 | |
| 411 | .. c:function:: void PyMem_SetupDebugHooks(void) |
| 412 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 413 | Setup hooks to detect bugs in the Python memory allocator functions. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 414 | |
| 415 | Newly allocated memory is filled with the byte ``0xCB``, freed memory is |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 416 | filled with the byte ``0xDB``. |
| 417 | |
| 418 | Runtime checks: |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 419 | |
Victor Stinner | c4aec36 | 2016-03-14 22:26:53 +0100 | [diff] [blame] | 420 | - Detect API violations, ex: :c:func:`PyObject_Free` called on a buffer |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 421 | allocated by :c:func:`PyMem_Malloc` |
Victor Stinner | c4aec36 | 2016-03-14 22:26:53 +0100 | [diff] [blame] | 422 | - Detect write before the start of the buffer (buffer underflow) |
| 423 | - Detect write after the end of the buffer (buffer overflow) |
| 424 | - Check that the :term:`GIL <global interpreter lock>` is held when |
Victor Stinner | c2fc568 | 2016-03-18 11:04:31 +0100 | [diff] [blame] | 425 | allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: |
| 426 | :c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex: |
| 427 | :c:func:`PyMem_Malloc`) domains are called |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 428 | |
Victor Stinner | 0611c26 | 2016-03-15 22:22:13 +0100 | [diff] [blame] | 429 | On error, the debug hooks use the :mod:`tracemalloc` module to get the |
| 430 | traceback where a memory block was allocated. The traceback is only |
| 431 | displayed if :mod:`tracemalloc` is tracing Python memory allocations and the |
| 432 | memory block was traced. |
| 433 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 434 | These hooks are installed by default if Python is compiled in debug |
| 435 | mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install |
| 436 | debug hooks on a Python compiled in release mode. |
| 437 | |
| 438 | .. versionchanged:: 3.6 |
| 439 | This function now also works on Python compiled in release mode. |
Victor Stinner | 0611c26 | 2016-03-15 22:22:13 +0100 | [diff] [blame] | 440 | On error, the debug hooks now use :mod:`tracemalloc` to get the traceback |
Victor Stinner | c2fc568 | 2016-03-18 11:04:31 +0100 | [diff] [blame] | 441 | where a memory block was allocated. The debug hooks now also check |
Victor Stinner | 9b46a57 | 2016-03-18 15:10:43 +0100 | [diff] [blame] | 442 | if the GIL is held when functions of :c:data:`PYMEM_DOMAIN_OBJ` and |
Victor Stinner | c2fc568 | 2016-03-18 11:04:31 +0100 | [diff] [blame] | 443 | :c:data:`PYMEM_DOMAIN_MEM` domains are called. |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 444 | |
| 445 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 446 | .. _pymalloc: |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 447 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 448 | The pymalloc allocator |
| 449 | ====================== |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 450 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 451 | Python has a *pymalloc* allocator optimized for small objects (smaller or equal |
| 452 | to 512 bytes) with a short lifetime. It uses memory mappings called "arenas" |
| 453 | with a fixed size of 256 KB. It falls back to :c:func:`PyMem_RawMalloc` and |
| 454 | :c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes. |
| 455 | |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 456 | *pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_MEM` (ex: |
INADA Naoki | f669fff | 2017-02-27 22:42:37 +0900 | [diff] [blame] | 457 | :c:func:`PyMem_Malloc`) and :c:data:`PYMEM_DOMAIN_OBJ` (ex: |
Victor Stinner | f5c4b99 | 2016-04-22 16:26:23 +0200 | [diff] [blame] | 458 | :c:func:`PyObject_Malloc`) domains. |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 459 | |
| 460 | The arena allocator uses the following functions: |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 461 | |
| 462 | * :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows, |
| 463 | * :c:func:`mmap` and :c:func:`munmap` if available, |
| 464 | * :c:func:`malloc` and :c:func:`free` otherwise. |
| 465 | |
Victor Stinner | 34be807 | 2016-03-14 12:04:26 +0100 | [diff] [blame] | 466 | Customize pymalloc Arena Allocator |
| 467 | ---------------------------------- |
| 468 | |
Victor Stinner | 0507bf5 | 2013-07-07 02:05:46 +0200 | [diff] [blame] | 469 | .. versionadded:: 3.4 |
| 470 | |
| 471 | .. c:type:: PyObjectArenaAllocator |
| 472 | |
| 473 | Structure used to describe an arena allocator. The structure has |
| 474 | three fields: |
| 475 | |
| 476 | +--------------------------------------------------+---------------------------------------+ |
| 477 | | Field | Meaning | |
| 478 | +==================================================+=======================================+ |
| 479 | | ``void *ctx`` | user context passed as first argument | |
| 480 | +--------------------------------------------------+---------------------------------------+ |
| 481 | | ``void* alloc(void *ctx, size_t size)`` | allocate an arena of size bytes | |
| 482 | +--------------------------------------------------+---------------------------------------+ |
| 483 | | ``void free(void *ctx, size_t size, void *ptr)`` | free an arena | |
| 484 | +--------------------------------------------------+---------------------------------------+ |
| 485 | |
| 486 | .. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator) |
| 487 | |
| 488 | Get the arena allocator. |
| 489 | |
| 490 | .. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) |
| 491 | |
| 492 | Set the arena allocator. |
| 493 | |
| 494 | |
Victor Stinner | 5ea4c06 | 2017-06-20 17:46:36 +0200 | [diff] [blame] | 495 | tracemalloc C API |
| 496 | ================= |
| 497 | |
| 498 | .. versionadded:: 3.7 |
| 499 | |
| 500 | .. c:function: int PyTraceMalloc_Track(unsigned int domain, uintptr_t ptr, size_t size) |
| 501 | |
| 502 | Track an allocated memory block in the :mod:`tracemalloc` module. |
| 503 | |
| 504 | Return 0 on success, return ``-1`` on error (failed to allocate memory to |
| 505 | store the trace). Return ``-2`` if tracemalloc is disabled. |
| 506 | |
| 507 | If memory block is already tracked, update the existing trace. |
| 508 | |
| 509 | .. c:function: int PyTraceMalloc_Untrack(unsigned int domain, uintptr_t ptr) |
| 510 | |
| 511 | Untrack an allocated memory block in the :mod:`tracemalloc` module. |
| 512 | Do nothing if the block was not tracked. |
| 513 | |
| 514 | Return ``-2`` if tracemalloc is disabled, otherwise return ``0``. |
| 515 | |
| 516 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 517 | .. _memoryexamples: |
| 518 | |
| 519 | Examples |
| 520 | ======== |
| 521 | |
| 522 | Here is the example from section :ref:`memoryoverview`, rewritten so that the |
| 523 | I/O buffer is allocated from the Python heap by using the first function set:: |
| 524 | |
| 525 | PyObject *res; |
| 526 | char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */ |
| 527 | |
| 528 | if (buf == NULL) |
| 529 | return PyErr_NoMemory(); |
| 530 | /* ...Do some I/O operation involving buf... */ |
Gregory P. Smith | 4b52ae8 | 2013-03-22 13:43:30 -0700 | [diff] [blame] | 531 | res = PyBytes_FromString(buf); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 532 | PyMem_Free(buf); /* allocated with PyMem_Malloc */ |
| 533 | return res; |
| 534 | |
| 535 | The same code using the type-oriented function set:: |
| 536 | |
| 537 | PyObject *res; |
| 538 | char *buf = PyMem_New(char, BUFSIZ); /* for I/O */ |
| 539 | |
| 540 | if (buf == NULL) |
| 541 | return PyErr_NoMemory(); |
| 542 | /* ...Do some I/O operation involving buf... */ |
Gregory P. Smith | 4b52ae8 | 2013-03-22 13:43:30 -0700 | [diff] [blame] | 543 | res = PyBytes_FromString(buf); |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 544 | PyMem_Del(buf); /* allocated with PyMem_New */ |
| 545 | return res; |
| 546 | |
| 547 | Note that in the two examples above, the buffer is always manipulated via |
| 548 | functions belonging to the same set. Indeed, it is required to use the same |
| 549 | memory API family for a given memory block, so that the risk of mixing different |
| 550 | allocators is reduced to a minimum. The following code sequence contains two |
| 551 | errors, one of which is labeled as *fatal* because it mixes two different |
| 552 | allocators operating on different heaps. :: |
| 553 | |
| 554 | char *buf1 = PyMem_New(char, BUFSIZ); |
| 555 | char *buf2 = (char *) malloc(BUFSIZ); |
| 556 | char *buf3 = (char *) PyMem_Malloc(BUFSIZ); |
| 557 | ... |
| 558 | PyMem_Del(buf3); /* Wrong -- should be PyMem_Free() */ |
| 559 | free(buf2); /* Right -- allocated via malloc() */ |
| 560 | free(buf1); /* Fatal -- should be PyMem_Del() */ |
| 561 | |
| 562 | In addition to the functions aimed at handling raw memory blocks from the Python |
Georg Brandl | 60203b4 | 2010-10-06 10:11:56 +0000 | [diff] [blame] | 563 | heap, objects in Python are allocated and released with :c:func:`PyObject_New`, |
| 564 | :c:func:`PyObject_NewVar` and :c:func:`PyObject_Del`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 565 | |
| 566 | These will be explained in the next chapter on defining and implementing new |
| 567 | object types in C. |
| 568 | |