Georg Brandl | fe7b00f | 2012-10-06 13:49:34 +0200 | [diff] [blame] | 1 | .. _xml: |
| 2 | |
| 3 | XML Processing Modules |
| 4 | ====================== |
| 5 | |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 6 | .. module:: xml |
| 7 | :synopsis: Package containing XML processing modules |
| 8 | .. sectionauthor:: Christian Heimes <christian@python.org> |
| 9 | .. sectionauthor:: Georg Brandl <georg@python.org> |
| 10 | |
| 11 | |
Georg Brandl | fe7b00f | 2012-10-06 13:49:34 +0200 | [diff] [blame] | 12 | Python's interfaces for processing XML are grouped in the ``xml`` package. |
| 13 | |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 14 | .. warning:: |
| 15 | |
| 16 | The XML modules are not secure against erroneous or maliciously |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 17 | constructed data. If you need to parse untrusted or |
| 18 | unauthenticated data see the :ref:`xml-vulnerabilities` and |
| 19 | :ref:`defused-packages` sections. |
Christian Heimes | 768f6a5 | 2013-03-26 17:47:23 +0100 | [diff] [blame] | 20 | |
Georg Brandl | fe7b00f | 2012-10-06 13:49:34 +0200 | [diff] [blame] | 21 | It is important to note that modules in the :mod:`xml` package require that |
| 22 | there be at least one SAX-compliant XML parser available. The Expat parser is |
| 23 | included with Python, so the :mod:`xml.parsers.expat` module will always be |
| 24 | available. |
| 25 | |
| 26 | The documentation for the :mod:`xml.dom` and :mod:`xml.sax` packages are the |
| 27 | definition of the Python bindings for the DOM and SAX interfaces. |
| 28 | |
| 29 | The XML handling submodules are: |
| 30 | |
| 31 | * :mod:`xml.etree.ElementTree`: the ElementTree API, a simple and lightweight |
Zachary Ware | 19c1f3d | 2014-01-31 11:30:36 -0600 | [diff] [blame] | 32 | XML processor |
Georg Brandl | fe7b00f | 2012-10-06 13:49:34 +0200 | [diff] [blame] | 33 | |
| 34 | .. |
| 35 | |
| 36 | * :mod:`xml.dom`: the DOM API definition |
Antoine Pitrou | f20ea13 | 2013-12-22 01:57:01 +0100 | [diff] [blame] | 37 | * :mod:`xml.dom.minidom`: a minimal DOM implementation |
Georg Brandl | fe7b00f | 2012-10-06 13:49:34 +0200 | [diff] [blame] | 38 | * :mod:`xml.dom.pulldom`: support for building partial DOM trees |
| 39 | |
| 40 | .. |
| 41 | |
| 42 | * :mod:`xml.sax`: SAX2 base classes and convenience functions |
| 43 | * :mod:`xml.parsers.expat`: the Expat parser binding |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 44 | |
| 45 | |
| 46 | .. _xml-vulnerabilities: |
| 47 | |
| 48 | XML vulnerabilities |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 49 | ------------------- |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 50 | |
| 51 | The XML processing modules are not secure against maliciously constructed data. |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 52 | An attacker can abuse XML features to carry out denial of service attacks, |
| 53 | access local files, generate network connections to other machines, or |
| 54 | circumvent firewalls. |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 55 | |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 56 | The following table gives an overview of the known attacks and whether |
| 57 | the various modules are vulnerable to them. |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 58 | |
| 59 | ========================= ======== ========= ========= ======== ========= |
| 60 | kind sax etree minidom pulldom xmlrpc |
| 61 | ========================= ======== ========= ========= ======== ========= |
Georg Brandl | 57f936e | 2013-10-12 18:19:33 +0200 | [diff] [blame] | 62 | billion laughs **Yes** **Yes** **Yes** **Yes** **Yes** |
| 63 | quadratic blowup **Yes** **Yes** **Yes** **Yes** **Yes** |
| 64 | external entity expansion **Yes** No (1) No (2) **Yes** No (3) |
| 65 | DTD retrieval **Yes** No No **Yes** No |
| 66 | decompression bomb No No No No **Yes** |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 67 | ========================= ======== ========= ========= ======== ========= |
| 68 | |
| 69 | 1. :mod:`xml.etree.ElementTree` doesn't expand external entities and raises a |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 70 | :exc:`ParserError` when an entity occurs. |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 71 | 2. :mod:`xml.dom.minidom` doesn't expand external entities and simply returns |
| 72 | the unexpanded entity verbatim. |
| 73 | 3. :mod:`xmlrpclib` doesn't expand external entities and omits them. |
| 74 | |
| 75 | |
| 76 | billion laughs / exponential entity expansion |
| 77 | The `Billion Laughs`_ attack -- also known as exponential entity expansion -- |
| 78 | uses multiple levels of nested entities. Each entity refers to another entity |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 79 | several times, and the final entity definition contains a small string. |
| 80 | The exponential expansion results in several gigabytes of text and |
| 81 | consumes lots of memory and CPU time. |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 82 | |
| 83 | quadratic blowup entity expansion |
| 84 | A quadratic blowup attack is similar to a `Billion Laughs`_ attack; it abuses |
| 85 | entity expansion, too. Instead of nested entities it repeats one large entity |
| 86 | with a couple of thousand chars over and over again. The attack isn't as |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 87 | efficient as the exponential case but it avoids triggering parser countermeasures |
| 88 | that forbid deeply-nested entities. |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 89 | |
| 90 | external entity expansion |
| 91 | Entity declarations can contain more than just text for replacement. They can |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 92 | also point to external resources or local files. The XML |
| 93 | parser accesses the resource and embeds the content into the XML document. |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 94 | |
| 95 | DTD retrieval |
R David Murray | 66c9350 | 2014-01-13 13:51:17 -0500 | [diff] [blame] | 96 | Some XML libraries like Python's :mod:`xml.dom.pulldom` retrieve document type |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 97 | definitions from remote or local locations. The feature has similar |
| 98 | implications as the external entity expansion issue. |
| 99 | |
| 100 | decompression bomb |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 101 | Decompression bombs (aka `ZIP bomb`_) apply to all XML libraries |
| 102 | that can parse compressed XML streams such as gzipped HTTP streams or |
| 103 | LZMA-compressed |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 104 | files. For an attacker it can reduce the amount of transmitted data by three |
| 105 | magnitudes or more. |
| 106 | |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 107 | The documentation for `defusedxml`_ on PyPI has further information about |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 108 | all known attack vectors with examples and references. |
| 109 | |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 110 | .. _defused-packages: |
| 111 | |
| 112 | The :mod:`defusedxml` and :mod:`defusedexpat` Packages |
| 113 | ------------------------------------------------------ |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 114 | |
| 115 | `defusedxml`_ is a pure Python package with modified subclasses of all stdlib |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 116 | XML parsers that prevent any potentially malicious operation. Use of this |
| 117 | package is recommended for any server code that parses untrusted XML data. The |
| 118 | package also ships with example exploits and extended documentation on more |
| 119 | XML exploits such as XPath injection. |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 120 | |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 121 | `defusedexpat`_ provides a modified libexpat and a patched |
| 122 | :mod:`pyexpat` module that have countermeasures against entity expansion |
| 123 | DoS attacks. The :mod:`defusedexpat` module still allows a sane and configurable amount of entity |
| 124 | expansions. The modifications may be included in some future release of Python, |
| 125 | but will not be included in any bugfix releases of |
| 126 | Python because they break backward compatibility. |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 127 | |
| 128 | |
Georg Brandl | 6ba6b13 | 2013-03-28 09:11:44 +0100 | [diff] [blame] | 129 | .. _defusedxml: https://pypi.python.org/pypi/defusedxml/ |
| 130 | .. _defusedexpat: https://pypi.python.org/pypi/defusedexpat/ |
Christian Heimes | 7380a67 | 2013-03-26 17:35:55 +0100 | [diff] [blame] | 131 | .. _Billion Laughs: http://en.wikipedia.org/wiki/Billion_laughs |
| 132 | .. _ZIP bomb: http://en.wikipedia.org/wiki/Zip_bomb |
| 133 | .. _DTD: http://en.wikipedia.org/wiki/Document_Type_Definition |