Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | |
| 2 | :mod:`mmap` --- Memory-mapped file support |
| 3 | ========================================== |
| 4 | |
| 5 | .. module:: mmap |
| 6 | :synopsis: Interface to memory-mapped files for Unix and Windows. |
| 7 | |
| 8 | |
| 9 | Memory-mapped file objects behave like both strings and like file objects. |
| 10 | Unlike normal string objects, however, these are mutable. You can use mmap |
| 11 | objects in most places where strings are expected; for example, you can use the |
| 12 | :mod:`re` module to search through a memory-mapped file. Since they're mutable, |
| 13 | you can change a single character by doing ``obj[index] = 'a'``, or change a |
| 14 | substring by assigning to a slice: ``obj[i1:i2] = '...'``. You can also read |
| 15 | and write data starting at the current file position, and :meth:`seek` through |
| 16 | the file to different positions. |
| 17 | |
| 18 | A memory-mapped file is created by the :func:`mmap` function, which is different |
| 19 | on Unix and on Windows. In either case you must provide a file descriptor for a |
| 20 | file opened for update. If you wish to map an existing Python file object, use |
| 21 | its :meth:`fileno` method to obtain the correct value for the *fileno* |
| 22 | parameter. Otherwise, you can open the file using the :func:`os.open` function, |
| 23 | which returns a file descriptor directly (the file still needs to be closed when |
| 24 | done). |
| 25 | |
| 26 | For both the Unix and Windows versions of the function, *access* may be |
| 27 | specified as an optional keyword parameter. *access* accepts one of three |
| 28 | values: :const:`ACCESS_READ`, :const:`ACCESS_WRITE`, or :const:`ACCESS_COPY` to |
| 29 | specify readonly, write-through or copy-on-write memory respectively. *access* |
| 30 | can be used on both Unix and Windows. If *access* is not specified, Windows |
| 31 | mmap returns a write-through mapping. The initial memory values for all three |
| 32 | access types are taken from the specified file. Assignment to an |
| 33 | :const:`ACCESS_READ` memory map raises a :exc:`TypeError` exception. Assignment |
| 34 | to an :const:`ACCESS_WRITE` memory map affects both memory and the underlying |
| 35 | file. Assignment to an :const:`ACCESS_COPY` memory map affects memory but does |
| 36 | not update the underlying file. |
| 37 | |
Georg Brandl | 55ac8f0 | 2007-09-01 13:51:09 +0000 | [diff] [blame] | 38 | To map anonymous memory, -1 should be passed as the fileno along with the length. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 39 | |
| 40 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 41 | .. function:: mmap(fileno, length[, tagname[, access[, offset]]]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 42 | |
| 43 | **(Windows version)** Maps *length* bytes from the file specified by the file |
| 44 | handle *fileno*, and returns a mmap object. If *length* is larger than the |
| 45 | current size of the file, the file is extended to contain *length* bytes. If |
| 46 | *length* is ``0``, the maximum length of the map is the current size of the |
| 47 | file, except that if the file is empty Windows raises an exception (you cannot |
| 48 | create an empty mapping on Windows). |
| 49 | |
| 50 | *tagname*, if specified and not ``None``, is a string giving a tag name for the |
| 51 | mapping. Windows allows you to have many different mappings against the same |
| 52 | file. If you specify the name of an existing tag, that tag is opened, otherwise |
| 53 | a new tag of this name is created. If this parameter is omitted or ``None``, |
| 54 | the mapping is created without a name. Avoiding the use of the tag parameter |
| 55 | will assist in keeping your code portable between Unix and Windows. |
| 56 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 57 | *offset* may be specified as a non-negative integer offset. mmap references will |
| 58 | be relative to the offset from the beginning of the file. *offset* defaults to 0. |
| 59 | *offset* must be a multiple of the ALLOCATIONGRANULARITY. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 60 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 61 | |
| 62 | .. function:: mmap(fileno, length[, flags[, prot[, access[, offset]]]]) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 63 | :noindex: |
| 64 | |
| 65 | **(Unix version)** Maps *length* bytes from the file specified by the file |
| 66 | descriptor *fileno*, and returns a mmap object. If *length* is ``0``, the |
| 67 | maximum length of the map will be the current size of the file when :func:`mmap` |
| 68 | is called. |
| 69 | |
| 70 | *flags* specifies the nature of the mapping. :const:`MAP_PRIVATE` creates a |
| 71 | private copy-on-write mapping, so changes to the contents of the mmap object |
| 72 | will be private to this process, and :const:`MAP_SHARED` creates a mapping |
| 73 | that's shared with all other processes mapping the same areas of the file. The |
| 74 | default value is :const:`MAP_SHARED`. |
| 75 | |
| 76 | *prot*, if specified, gives the desired memory protection; the two most useful |
| 77 | values are :const:`PROT_READ` and :const:`PROT_WRITE`, to specify that the pages |
| 78 | may be read or written. *prot* defaults to :const:`PROT_READ \| PROT_WRITE`. |
| 79 | |
| 80 | *access* may be specified in lieu of *flags* and *prot* as an optional keyword |
| 81 | parameter. It is an error to specify both *flags*, *prot* and *access*. See |
| 82 | the description of *access* above for information on how to use this parameter. |
| 83 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 84 | *offset* may be specified as a non-negative integer offset. mmap references will |
| 85 | be relative to the offset from the beginning of the file. *offset* defaults to 0. |
| 86 | *offset* must be a multiple of the PAGESIZE or ALLOCATIONGRANULARITY. |
Christian Heimes | d8654cf | 2007-12-02 15:22:16 +0000 | [diff] [blame] | 87 | |
| 88 | This example shows a simple way of using :func:`mmap`:: |
| 89 | |
| 90 | import mmap |
| 91 | |
| 92 | # write a simple example file |
| 93 | with open("hello.txt", "w") as f: |
| 94 | f.write("Hello Python!\n") |
| 95 | |
| 96 | with open("hello.txt", "r+") as f: |
| 97 | # memory-map the file, size 0 means whole file |
| 98 | map = mmap.mmap(f.fileno(), 0) |
| 99 | # read content via standard file methods |
| 100 | print map.readline() # prints "Hello Python!" |
| 101 | # read content via slice notation |
| 102 | print map[:5] # prints "Hello" |
| 103 | # update content using slice notation; |
| 104 | # note that new content must have same size |
| 105 | map[6:] = " world!\n" |
| 106 | # ... and read again using standard file methods |
| 107 | map.seek(0) |
| 108 | print map.readline() # prints "Hello world!" |
| 109 | # close the map |
| 110 | map.close() |
| 111 | |
| 112 | |
| 113 | The next example demonstrates how to create an anonymous map and exchange |
| 114 | data between the parent and child processes:: |
| 115 | |
| 116 | import mmap |
| 117 | import os |
| 118 | |
| 119 | map = mmap.mmap(-1, 13) |
| 120 | map.write("Hello world!") |
| 121 | |
| 122 | pid = os.fork() |
| 123 | |
| 124 | if pid == 0: # In a child process |
| 125 | map.seek(0) |
| 126 | print map.readline() |
| 127 | |
| 128 | map.close() |
| 129 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 130 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 131 | Memory-mapped file objects support the following methods: |
| 132 | |
| 133 | |
| 134 | .. method:: mmap.close() |
| 135 | |
| 136 | Close the file. Subsequent calls to other methods of the object will result in |
| 137 | an exception being raised. |
| 138 | |
| 139 | |
| 140 | .. method:: mmap.find(string[, start]) |
| 141 | |
| 142 | Returns the lowest index in the object where the substring *string* is found. |
| 143 | Returns ``-1`` on failure. *start* is the index at which the search begins, and |
| 144 | defaults to zero. |
| 145 | |
| 146 | |
| 147 | .. method:: mmap.flush([offset, size]) |
| 148 | |
| 149 | Flushes changes made to the in-memory copy of a file back to disk. Without use |
| 150 | of this call there is no guarantee that changes are written back before the |
| 151 | object is destroyed. If *offset* and *size* are specified, only changes to the |
| 152 | given range of bytes will be flushed to disk; otherwise, the whole extent of the |
| 153 | mapping is flushed. |
| 154 | |
| 155 | |
| 156 | .. method:: mmap.move(dest, src, count) |
| 157 | |
| 158 | Copy the *count* bytes starting at offset *src* to the destination index *dest*. |
| 159 | If the mmap was created with :const:`ACCESS_READ`, then calls to move will throw |
| 160 | a :exc:`TypeError` exception. |
| 161 | |
| 162 | |
| 163 | .. method:: mmap.read(num) |
| 164 | |
| 165 | Return a string containing up to *num* bytes starting from the current file |
| 166 | position; the file position is updated to point after the bytes that were |
| 167 | returned. |
| 168 | |
| 169 | |
| 170 | .. method:: mmap.read_byte() |
| 171 | |
| 172 | Returns a string of length 1 containing the character at the current file |
| 173 | position, and advances the file position by 1. |
| 174 | |
| 175 | |
| 176 | .. method:: mmap.readline() |
| 177 | |
| 178 | Returns a single line, starting at the current file position and up to the next |
| 179 | newline. |
| 180 | |
| 181 | |
| 182 | .. method:: mmap.resize(newsize) |
| 183 | |
| 184 | Resizes the map and the underlying file, if any. If the mmap was created with |
| 185 | :const:`ACCESS_READ` or :const:`ACCESS_COPY`, resizing the map will throw a |
| 186 | :exc:`TypeError` exception. |
| 187 | |
| 188 | |
| 189 | .. method:: mmap.seek(pos[, whence]) |
| 190 | |
| 191 | Set the file's current position. *whence* argument is optional and defaults to |
| 192 | ``os.SEEK_SET`` or ``0`` (absolute file positioning); other values are |
| 193 | ``os.SEEK_CUR`` or ``1`` (seek relative to the current position) and |
| 194 | ``os.SEEK_END`` or ``2`` (seek relative to the file's end). |
| 195 | |
| 196 | |
| 197 | .. method:: mmap.size() |
| 198 | |
| 199 | Return the length of the file, which can be larger than the size of the |
| 200 | memory-mapped area. |
| 201 | |
| 202 | |
| 203 | .. method:: mmap.tell() |
| 204 | |
| 205 | Returns the current position of the file pointer. |
| 206 | |
| 207 | |
| 208 | .. method:: mmap.write(string) |
| 209 | |
| 210 | Write the bytes in *string* into memory at the current position of the file |
| 211 | pointer; the file position is updated to point after the bytes that were |
| 212 | written. If the mmap was created with :const:`ACCESS_READ`, then writing to it |
| 213 | will throw a :exc:`TypeError` exception. |
| 214 | |
| 215 | |
| 216 | .. method:: mmap.write_byte(byte) |
| 217 | |
| 218 | Write the single-character string *byte* into memory at the current position of |
| 219 | the file pointer; the file position is advanced by ``1``. If the mmap was |
| 220 | created with :const:`ACCESS_READ`, then writing to it will throw a |
| 221 | :exc:`TypeError` exception. |
| 222 | |
Georg Brandl | 9afde1c | 2007-11-01 20:32:30 +0000 | [diff] [blame] | 223 | |