| .. highlightlang:: c |
| |
| .. _bufferobjects: |
| |
| Buffer Objects |
| -------------- |
| |
| .. sectionauthor:: Greg Stein <gstein@lyra.org> |
| .. sectionauthor:: Benjamin Peterson |
| |
| |
| .. index:: |
| object: buffer |
| single: buffer interface |
| |
| Python objects implemented in C can export a "buffer interface." These |
| functions can be used by an object to expose its data in a raw, byte-oriented |
| format. Clients of the object can use the buffer interface to access the object |
| data directly, without needing to copy it first. |
| |
| Two examples of objects that support the buffer interface are bytes and |
| arrays. The bytes object exposes the character contents in the buffer |
| interface's byte-oriented form. An array can also expose its contents, but it |
| should be noted that array elements may be multi-byte values. |
| |
| An example user of the buffer interface is the file object's :meth:`write` |
| method. Any object that can export a series of bytes through the buffer |
| interface can be written to a file. There are a number of format codes to |
| :cfunc:`PyArg_ParseTuple` that operate against an object's buffer interface, |
| returning data from the target object. |
| |
| .. index:: single: PyBufferProcs |
| |
| More information on the buffer interface is provided in the section |
| :ref:`buffer-structs`, under the description for :ctype:`PyBufferProcs`. |
| |
| Buffer objects are useful as a way to expose the data from another object's |
| buffer interface to the Python programmer. They can also be used as a zero-copy |
| slicing mechanism. Using their ability to reference a block of memory, it is |
| possible to expose any data to the Python programmer quite easily. The memory |
| could be a large, constant array in a C extension, it could be a raw block of |
| memory for manipulation before passing to an operating system library, or it |
| could be used to pass around structured data in its native, in-memory format. |
| |
| |
| .. ctype:: Py_buffer |
| |
| .. cmember:: void *buf |
| |
| A pointer to the start of the memory for the object. |
| |
| .. cmember:: Py_ssize_t len |
| :noindex: |
| |
| The total length of the memory in bytes. |
| |
| .. cmember:: int readonly |
| |
| An indicator of whether the buffer is read only. |
| |
| .. cmember:: const char *format |
| :noindex: |
| |
| A *NULL* terminated string in :mod:`struct` module style syntax giving the |
| contents of the elements available through the buffer. If this is *NULL*, |
| ``"B"`` (unsigned bytes) is assumed. |
| |
| .. cmember:: int ndim |
| |
| The number of dimensions the memory represents as a multi-dimensional |
| array. If it is 0, :cdata:`strides` and :cdata:`suboffsets` must be |
| *NULL*. |
| |
| .. cmember:: Py_ssize_t *shape |
| |
| An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the |
| shape of the memory as a multi-dimensional array. Note that |
| ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to |
| :cdata:`len`. |
| |
| .. cmember:: Py_ssize_t *strides |
| |
| An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the |
| number of bytes to skip to get to a new element in each dimension. |
| |
| .. cmember:: Py_ssize_t *suboffsets |
| |
| An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim`. If these |
| suboffset numbers are greater than or equal to 0, then the value stored |
| along the indicated dimension is a pointer and the suboffset value |
| dictates how many bytes to add to the pointer after de-referencing. A |
| suboffset value that it negative indicates that no de-referencing should |
| occur (striding in a contiguous memory block). |
| |
| Here is a function that returns a pointer to the element in an N-D array |
| pointed to by an N-dimesional index when there are both non-NULL strides |
| and suboffsets:: |
| |
| void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, |
| Py_ssize_t *suboffsets, Py_ssize_t *indices) { |
| char *pointer = (char*)buf; |
| int i; |
| for (i = 0; i < ndim; i++) { |
| pointer += strides[i] * indices[i]; |
| if (suboffsets[i] >=0 ) { |
| pointer = *((char**)pointer) + suboffsets[i]; |
| } |
| } |
| return (void*)pointer; |
| } |
| |
| |
| .. cmember:: Py_ssize_t itemsize |
| |
| This is a storage for the itemsize (in bytes) of each element of the |
| shared memory. It is technically un-necessary as it can be obtained using |
| :cfunc:`PyBuffer_SizeFromFormat`, however an exporter may know this |
| information without parsing the format string and it is necessary to know |
| the itemsize for proper interpretation of striding. Therefore, storing it |
| is more convenient and faster. |
| |
| .. cmember:: void *internal |
| |
| This is for use internally by the exporting object. For example, this |
| might be re-cast as an integer by the exporter and used to store flags |
| about whether or not the shape, strides, and suboffsets arrays must be |
| freed when the buffer is released. The consumer should never alter this |
| value. |
| |
| |
| Buffer related functions |
| ======================== |
| |
| |
| .. cfunction:: int PyObject_CheckBuffer(PyObject *obj) |
| |
| Return 1 if *obj* supports the buffer interface otherwise 0. |
| |
| |
| .. cfunction:: int PyObject_GetBuffer(PyObject *obj, PyObject *view, int flags) |
| |
| Export *obj* into a :ctype:`Py_buffer`, *view*. These arguments must |
| never be *NULL*. The *flags* argument is a bit field indicating what kind |
| of buffer the caller is prepared to deal with and therefore what kind of |
| buffer the exporter is allowed to return. The buffer interface allows for |
| complicated memory sharing possibilities, but some caller may not be able |
| to handle all the complexibity but may want to see if the exporter will |
| let them take a simpler view to its memory. |
| |
| Some exporters may not be able to share memory in every possible way and |
| may need to raise errors to signal to some consumers that something is |
| just not possible. These errors should be a :exc:`BufferError` unless |
| there is another error that is actually causing the problem. The exporter |
| can use flags information to simplify how much of the :cdata:`Py_buffer` |
| structure is filled in with non-default values and/or raise an error if |
| the object can't support a simpler view of its memory. |
| |
| 0 is returned on success and -1 on error. |
| |
| The following table gives possible values to the *flags* arguments. |
| |
| +------------------------------+-----------------------------------------------+ |
| | Flag | Description | |
| +==============================+===============================================+ |
| | :cmacro:`PyBUF_SIMPLE` |This is the default flag state. The returned | |
| | |buffer may or may not have writable memory. | |
| | |The format will be assumed to be unsigned bytes| |
| | |. This is a "stand-alone" flag constant. It | |
| | |never needs to be |'d to the others. The | |
| | |exporter will raise an error if it cannot | |
| | |provide such a contiguous buffer of bytes. | |
| | | | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_WRITABLE` |The returned buffer must be writable. If it is | |
| | |not writable, then raise an error. | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_STRIDES` |This implies :cmacro:`PyBUF_ND`. The returned | |
| | |buffer must provide strides information | |
| | |(i.e. the strides cannot be NULL). This would | |
| | |be used when the consumer can handle strided, | |
| | |discontiguous arrays. Handling strides | |
| | |automatically assumes you can handle shape. The| |
| | |exporter may raise an error if cannot provide a| |
| | |strided-only representation of the data | |
| | |(i.e. without the suboffsets). | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_ND` |The returned buffer must provide shape | |
| | |information. The memory will be assumed C-style| |
| | |contiguous (last dimension varies the | |
| | |fastest). The exporter may raise an error if it| |
| | |cannot provide this kind of contiguous | |
| | |buffer. If this is not given then shape will be| |
| | |*NULL*. | |
| | | | |
| | | | |
| +------------------------------+-----------------------------------------------+ |
| |:cmacro:`PyBUF_C_CONTIGUOUS` |These flags indicate that the contiguoity | |
| |:cmacro:`PyBUF_F_CONTIGUOUS` |returned buffer must be respectively, | |
| |:cmacro:`PyBUF_ANY_CONTIGUOUS`|C-contiguous (last dimension varies the | |
| | |fastest), Fortran contiguous (first dimension | |
| | |varies the fastest) or either one. All of | |
| | |these flags imply :cmacro:`PyBUF_STRIDES` and | |
| | |guarantee that the strides buffer info | |
| | |structure will be filled in correctly. | |
| | | | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_INDIRECT` |This implies :cmacro:`PyBUF_STRIDES`. The | |
| | |returned buffer must have suboffsets | |
| | |information (which can be NULL if no suboffsets| |
| | |are needed). This would be used when the | |
| | |consumer can handle indirect array referencing | |
| | |implied by these suboffsets. | |
| | | | |
| | | | |
| | | | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_FORMAT` |The returned buffer must have true format | |
| | |information if this flag is provided. This | |
| | |would be used when the consumer is going to be | |
| | |checking for what 'kind' of data is actually | |
| | |stored. An exporter should always be able to | |
| | |provide this information if requested. If | |
| | |format is not explicitly requested then the | |
| | |format must be returned as *NULL* (which means | |
| | |``'B'``, or unsigned bytes) | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_STRIDED` |This is equivalent to ``(PyBUF_STRIDES | | |
| | |PyBUF_WRITABLE)``. | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_STRIDED_RO` |This is equivalent to ``(PyBUF_STRIDES)``. | |
| | | | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_RECORDS` |This is equivalent to ``(PyBUF_STRIDES | | |
| | |PyBUF_FORMAT | PyBUF_WRITABLE)``. | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_RECORDS_RO` |This is equivalent to ``(PyBUF_STRIDES | | |
| | |PyBUF_FORMAT)``. | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_FULL` |This is equivalent to ``(PyBUF_INDIRECT | | |
| | |PyBUF_FORMAT | PyBUF_WRITABLE)``. | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_FULL_RO`` |This is equivalent to ``(PyBUF_INDIRECT | | |
| | |PyBUF_FORMAT)``. | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_CONTIG` |This is equivalent to ``(PyBUF_ND | | |
| | |PyBUF_WRITABLE)``. | |
| +------------------------------+-----------------------------------------------+ |
| | :cmacro:`PyBUF_CONTIG_RO` |This is equivalent to ``(PyBUF_ND)``. | |
| | | | |
| +------------------------------+-----------------------------------------------+ |
| |
| |
| .. cfunction:: void PyBuffer_Release(PyObject *obj, Py_buffer *view) |
| |
| Release the buffer *view* over *obj*. This shouldd be called when the buffer |
| is no longer being used as it may free memory from it. |
| |
| |
| .. cfunction:: Py_ssize_t PyBuffer_SizeFromFormat(const char *) |
| |
| Return the implied :cdata:`~Py_buffer.itemsize` from the struct-stype |
| :cdata:`~Py_buffer.format`. |
| |
| |
| .. cfunction:: int PyObject_CopyToObject(PyObject *obj, void *buf, Py_ssize_t len, char fortran) |
| |
| Copy *len* bytes of data pointed to by the contiguous chunk of memory pointed |
| to by *buf* into the buffer exported by obj. The buffer must of course be |
| writable. Return 0 on success and return -1 and raise an error on failure. |
| If the object does not have a writable buffer, then an error is raised. If |
| *fortran* is ``'F'``, then if the object is multi-dimensional, then the data |
| will be copied into the array in Fortran-style (first dimension varies the |
| fastest). If *fortran* is ``'C'``, then the data will be copied into the |
| array in C-style (last dimension varies the fastest). If *fortran* is |
| ``'A'``, then it does not matter and the copy will be made in whatever way is |
| more efficient. |
| |
| |
| .. cfunction:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran) |
| |
| Return 1 if the memory defined by the *view* is C-style (*fortran* is |
| ``'C'``) or Fortran-style (*fortran* is ``'F'``) contiguous or either one |
| (*fortran* is ``'A'``). Return 0 otherwise. |
| |
| |
| .. cfunction:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char fortran) |
| |
| Fill the *strides* array with byte-strides of a contiguous (C-style if |
| *fortran* is ``'C'`` or Fortran-style if *fortran* is ``'F'`` array of the |
| given shape with the given number of bytes per element. |
| |
| |
| .. cfunction:: int PyBuffer_FillInfo(Py_buffer *view, void *buf, Py_ssize_t len, int readonly, int infoflags) |
| |
| Fill in a buffer-info structure, *view*, correctly for an exporter that can |
| only share a contiguous chunk of memory of "unsigned bytes" of the given |
| length. Return 0 on success and -1 (with raising an error) on error. |
| |
| |
| MemoryView objects |
| ================== |
| |
| A memoryview object is an extended buffer object that could replace the buffer |
| object (but doesn't have to as that could be kept as a simple 1-d memoryview |
| object). It, unlike :ctype:`Py_buffer`, is a Python object (exposed as |
| :class:`memoryview` in :mod:`builtins`), so it can be used with Python code. |
| |
| .. cfunction:: PyObject* PyMemoryView_FromObject(PyObject *obj) |
| |
| Return a memoryview object from an object that defines the buffer interface. |