Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | :mod:`mmap` --- Memory-mapped file support |
| 2 | ========================================== |
| 3 | |
| 4 | .. module:: mmap |
| 5 | :synopsis: Interface to memory-mapped files for Unix and Windows. |
| 6 | |
| 7 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 8 | Memory-mapped file objects behave like both :class:`bytes` and like file |
| 9 | objects. Unlike normal :class:`bytes` objects, however, these are mutable. |
| 10 | You can use mmap objects in most places where :class:`bytes` are expected; for |
| 11 | example, you can use the :mod:`re` module to search through a memory-mapped file. |
| 12 | Since they're mutable, you can change a single byte by doing ``obj[index] = 97``, |
| 13 | or change a subsequence by assigning to a slice: ``obj[i1:i2] = b'...'``. |
| 14 | You can also read and write data starting at the current file position, and |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 15 | :meth:`seek` through the file to different positions. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 16 | |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 17 | A memory-mapped file is created by the :class:`mmap` constructor, which is |
| 18 | different on Unix and on Windows. In either case you must provide a file |
| 19 | descriptor for a file opened for update. If you wish to map an existing Python |
| 20 | file object, use its :meth:`fileno` method to obtain the correct value for the |
| 21 | *fileno* parameter. Otherwise, you can open the file using the |
| 22 | :func:`os.open` function, which returns a file descriptor directly (the file |
| 23 | still needs to be closed when done). |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 24 | |
Georg Brandl | 86def6c | 2008-01-21 20:36:10 +0000 | [diff] [blame] | 25 | For both the Unix and Windows versions of the constructor, *access* may be |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 26 | specified as an optional keyword parameter. *access* accepts one of three |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 27 | values: :const:`ACCESS_READ`, :const:`ACCESS_WRITE`, or :const:`ACCESS_COPY` |
| 28 | to specify read-only, write-through or copy-on-write memory respectively. |
| 29 | *access* can be used on both Unix and Windows. If *access* is not specified, |
| 30 | Windows mmap returns a write-through mapping. The initial memory values for |
| 31 | all three access types are taken from the specified file. Assignment to an |
| 32 | :const:`ACCESS_READ` memory map raises a :exc:`TypeError` exception. |
| 33 | Assignment to an :const:`ACCESS_WRITE` memory map affects both memory and the |
| 34 | underlying file. Assignment to an :const:`ACCESS_COPY` memory map affects |
| 35 | memory but does not update the underlying file. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 36 | |
Georg Brandl | 55ac8f0 | 2007-09-01 13:51:09 +0000 | [diff] [blame] | 37 | To map anonymous memory, -1 should be passed as the fileno along with the length. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 38 | |
Georg Brandl | cd7f32b | 2009-06-08 09:13:45 +0000 | [diff] [blame] | 39 | .. class:: mmap(fileno, length, tagname=None, access=ACCESS_DEFAULT[, offset]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 40 | |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 41 | **(Windows version)** Maps *length* bytes from the file specified by the |
| 42 | file handle *fileno*, and creates a mmap object. If *length* is larger |
| 43 | than the current size of the file, the file is extended to contain *length* |
| 44 | bytes. If *length* is ``0``, the maximum length of the map is the current |
| 45 | size of the file, except that if the file is empty Windows raises an |
| 46 | exception (you cannot create an empty mapping on Windows). |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 47 | |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 48 | *tagname*, if specified and not ``None``, is a string giving a tag name for |
| 49 | the mapping. Windows allows you to have many different mappings against |
| 50 | the same file. If you specify the name of an existing tag, that tag is |
| 51 | opened, otherwise a new tag of this name is created. If this parameter is |
| 52 | omitted or ``None``, the mapping is created without a name. Avoiding the |
| 53 | use of the tag parameter will assist in keeping your code portable between |
| 54 | Unix and Windows. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 55 | |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 56 | *offset* may be specified as a non-negative integer offset. mmap references |
| 57 | will be relative to the offset from the beginning of the file. *offset* |
| 58 | defaults to 0. *offset* must be a multiple of the ALLOCATIONGRANULARITY. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 59 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 60 | |
Georg Brandl | cd7f32b | 2009-06-08 09:13:45 +0000 | [diff] [blame] | 61 | .. class:: mmap(fileno, length, flags=MAP_SHARED, prot=PROT_WRITE|PROT_READ, access=ACCESS_DEFAULT[, offset]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 62 | :noindex: |
| 63 | |
| 64 | **(Unix version)** Maps *length* bytes from the file specified by the file |
| 65 | descriptor *fileno*, and returns a mmap object. If *length* is ``0``, the |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 66 | maximum length of the map will be the current size of the file when |
| 67 | :class:`mmap` is called. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 68 | |
| 69 | *flags* specifies the nature of the mapping. :const:`MAP_PRIVATE` creates a |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 70 | private copy-on-write mapping, so changes to the contents of the mmap |
| 71 | object will be private to this process, and :const:`MAP_SHARED` creates a |
| 72 | mapping that's shared with all other processes mapping the same areas of |
| 73 | the file. The default value is :const:`MAP_SHARED`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 74 | |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 75 | *prot*, if specified, gives the desired memory protection; the two most |
| 76 | useful values are :const:`PROT_READ` and :const:`PROT_WRITE`, to specify |
| 77 | that the pages may be read or written. *prot* defaults to |
| 78 | :const:`PROT_READ \| PROT_WRITE`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 79 | |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 80 | *access* may be specified in lieu of *flags* and *prot* as an optional |
| 81 | keyword parameter. It is an error to specify both *flags*, *prot* and |
| 82 | *access*. See the description of *access* above for information on how to |
| 83 | use this parameter. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 84 | |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 85 | *offset* may be specified as a non-negative integer offset. mmap references |
| 86 | will be relative to the offset from the beginning of the file. *offset* |
| 87 | defaults to 0. *offset* must be a multiple of the PAGESIZE or |
| 88 | ALLOCATIONGRANULARITY. |
Georg Brandl | 48310cd | 2009-01-03 21:18:54 +0000 | [diff] [blame] | 89 | |
Georg Brandl | 86def6c | 2008-01-21 20:36:10 +0000 | [diff] [blame] | 90 | This example shows a simple way of using :class:`mmap`:: |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 91 | |
| 92 | import mmap |
| 93 | |
| 94 | # write a simple example file |
Benjamin Peterson | e0124bd | 2009-03-09 21:04:33 +0000 | [diff] [blame] | 95 | with open("hello.txt", "wb") as f: |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 96 | f.write(b"Hello Python!\n") |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 97 | |
Benjamin Peterson | e0124bd | 2009-03-09 21:04:33 +0000 | [diff] [blame] | 98 | with open("hello.txt", "r+b") as f: |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 99 | # memory-map the file, size 0 means whole file |
| 100 | map = mmap.mmap(f.fileno(), 0) |
| 101 | # read content via standard file methods |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 102 | print(map.readline()) # prints b"Hello Python!\n" |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 103 | # read content via slice notation |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 104 | print(map[:5]) # prints b"Hello" |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 105 | # update content using slice notation; |
| 106 | # note that new content must have same size |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 107 | map[6:] = b" world!\n" |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 108 | # ... and read again using standard file methods |
| 109 | map.seek(0) |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 110 | print(map.readline()) # prints b"Hello world!\n" |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 111 | # close the map |
| 112 | map.close() |
| 113 | |
| 114 | |
Georg Brandl | 0bccc18 | 2010-08-01 14:50:00 +0000 | [diff] [blame] | 115 | :class:`mmap` can also be used as a context manager in a :keyword:`with` |
| 116 | statement.:: |
| 117 | |
| 118 | import mmap |
| 119 | |
| 120 | with mmap.mmap(-1, 13) as map: |
| 121 | map.write("Hello world!") |
| 122 | |
| 123 | .. versionadded:: 3.2 |
| 124 | Context manager support. |
| 125 | |
| 126 | |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 127 | The next example demonstrates how to create an anonymous map and exchange |
| 128 | data between the parent and child processes:: |
| 129 | |
| 130 | import mmap |
| 131 | import os |
| 132 | |
| 133 | map = mmap.mmap(-1, 13) |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 134 | map.write(b"Hello world!") |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 135 | |
| 136 | pid = os.fork() |
| 137 | |
| 138 | if pid == 0: # In a child process |
| 139 | map.seek(0) |
Georg Brandl | a09ca38 | 2007-12-02 18:20:12 +0000 | [diff] [blame] | 140 | print(map.readline()) |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 141 | |
| 142 | map.close() |
| 143 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 144 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 145 | Memory-mapped file objects support the following methods: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 146 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 147 | .. method:: close() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 148 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 149 | Close the file. Subsequent calls to other methods of the object will |
| 150 | result in an exception being raised. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 151 | |
| 152 | |
Georg Brandl | 0bccc18 | 2010-08-01 14:50:00 +0000 | [diff] [blame] | 153 | .. attribute:: closed |
| 154 | |
| 155 | True if the file is closed. |
| 156 | |
| 157 | .. versionadded:: 3.2 |
| 158 | |
| 159 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 160 | .. method:: find(sub[, start[, end]]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 161 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 162 | Returns the lowest index in the object where the subsequence *sub* is |
| 163 | found, such that *sub* is contained in the range [*start*, *end*]. |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 164 | Optional arguments *start* and *end* are interpreted as in slice notation. |
| 165 | Returns ``-1`` on failure. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 166 | |
| 167 | |
Georg Brandl | cd7f32b | 2009-06-08 09:13:45 +0000 | [diff] [blame] | 168 | .. method:: flush([offset[, size]]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 169 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 170 | Flushes changes made to the in-memory copy of a file back to disk. Without |
| 171 | use of this call there is no guarantee that changes are written back before |
| 172 | the object is destroyed. If *offset* and *size* are specified, only |
| 173 | changes to the given range of bytes will be flushed to disk; otherwise, the |
| 174 | whole extent of the mapping is flushed. |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 175 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 176 | **(Windows version)** A nonzero value returned indicates success; zero |
| 177 | indicates failure. |
Christian Heimes | dae2a89 | 2008-04-19 00:55:37 +0000 | [diff] [blame] | 178 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 179 | **(Unix version)** A zero value is returned to indicate success. An |
| 180 | exception is raised when the call failed. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 181 | |
| 182 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 183 | .. method:: move(dest, src, count) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 184 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 185 | Copy the *count* bytes starting at offset *src* to the destination index |
| 186 | *dest*. If the mmap was created with :const:`ACCESS_READ`, then calls to |
| 187 | move will throw a :exc:`TypeError` exception. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 188 | |
| 189 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 190 | .. method:: read(num) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 191 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 192 | Return a :class:`bytes` containing up to *num* bytes starting from the |
| 193 | current file position; the file position is updated to point after the |
| 194 | bytes that were returned. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 195 | |
| 196 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 197 | .. method:: read_byte() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 198 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 199 | Returns a byte at the current file position as an integer, and advances |
| 200 | the file position by 1. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 201 | |
| 202 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 203 | .. method:: readline() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 204 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 205 | Returns a single line, starting at the current file position and up to the |
| 206 | next newline. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 207 | |
| 208 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 209 | .. method:: resize(newsize) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 210 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 211 | Resizes the map and the underlying file, if any. If the mmap was created |
| 212 | with :const:`ACCESS_READ` or :const:`ACCESS_COPY`, resizing the map will |
| 213 | throw a :exc:`TypeError` exception. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 214 | |
| 215 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 216 | .. method:: rfind(sub[, start[, end]]) |
Georg Brandl | fceab5a | 2008-01-19 20:08:23 +0000 | [diff] [blame] | 217 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 218 | Returns the highest index in the object where the subsequence *sub* is |
| 219 | found, such that *sub* is contained in the range [*start*, *end*]. |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 220 | Optional arguments *start* and *end* are interpreted as in slice notation. |
| 221 | Returns ``-1`` on failure. |
Georg Brandl | fceab5a | 2008-01-19 20:08:23 +0000 | [diff] [blame] | 222 | |
| 223 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 224 | .. method:: seek(pos[, whence]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 225 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 226 | Set the file's current position. *whence* argument is optional and |
| 227 | defaults to ``os.SEEK_SET`` or ``0`` (absolute file positioning); other |
| 228 | values are ``os.SEEK_CUR`` or ``1`` (seek relative to the current |
| 229 | position) and ``os.SEEK_END`` or ``2`` (seek relative to the file's end). |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 230 | |
| 231 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 232 | .. method:: size() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 233 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 234 | Return the length of the file, which can be larger than the size of the |
| 235 | memory-mapped area. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 236 | |
| 237 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 238 | .. method:: tell() |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 239 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 240 | Returns the current position of the file pointer. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 241 | |
| 242 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 243 | .. method:: write(bytes) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 244 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 245 | Write the bytes in *bytes* into memory at the current position of the |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 246 | file pointer; the file position is updated to point after the bytes that |
| 247 | were written. If the mmap was created with :const:`ACCESS_READ`, then |
| 248 | writing to it will throw a :exc:`TypeError` exception. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 249 | |
| 250 | |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 251 | .. method:: write_byte(byte) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 252 | |
Benjamin Peterson | e099b37 | 2009-04-04 17:09:35 +0000 | [diff] [blame] | 253 | Write the the integer *byte* into memory at the current |
Benjamin Peterson | e41251e | 2008-04-25 01:59:09 +0000 | [diff] [blame] | 254 | position of the file pointer; the file position is advanced by ``1``. If |
| 255 | the mmap was created with :const:`ACCESS_READ`, then writing to it will |
| 256 | throw a :exc:`TypeError` exception. |