Fred Drake | 3c50ea4 | 2008-05-17 22:02:32 +0000 | [diff] [blame] | 1 | :mod:`html.parser` --- Simple HTML and XHTML parser |
| 2 | =================================================== |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 3 | |
Fred Drake | 3c50ea4 | 2008-05-17 22:02:32 +0000 | [diff] [blame] | 4 | .. module:: html.parser |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 5 | :synopsis: A simple parser that can handle HTML and XHTML. |
| 6 | |
| 7 | |
Georg Brandl | 9087b7f | 2008-05-18 07:53:01 +0000 | [diff] [blame] | 8 | .. index:: |
| 9 | single: HTML |
| 10 | single: XHTML |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 11 | |
| 12 | This module defines a class :class:`HTMLParser` which serves as the basis for |
| 13 | parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 14 | |
| 15 | .. class:: HTMLParser() |
| 16 | |
| 17 | The :class:`HTMLParser` class is instantiated without arguments. |
| 18 | |
Fred Drake | 3c50ea4 | 2008-05-17 22:02:32 +0000 | [diff] [blame] | 19 | An :class:`HTMLParser` instance is fed HTML data and calls handler functions when tags |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 20 | begin and end. The :class:`HTMLParser` class is meant to be overridden by the |
| 21 | user to provide a desired behavior. |
| 22 | |
Georg Brandl | 877b10a | 2008-06-01 21:25:55 +0000 | [diff] [blame] | 23 | This parser does not check that end tags match start tags or call the end-tag |
| 24 | handler for elements which are closed implicitly by closing an outer element. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 25 | |
| 26 | An exception is defined as well: |
| 27 | |
| 28 | |
| 29 | .. exception:: HTMLParseError |
| 30 | |
| 31 | Exception raised by the :class:`HTMLParser` class when it encounters an error |
| 32 | while parsing. This exception provides three attributes: :attr:`msg` is a brief |
| 33 | message explaining the error, :attr:`lineno` is the number of the line on which |
| 34 | the broken construct was detected, and :attr:`offset` is the number of |
| 35 | characters into the line at which the construct starts. |
| 36 | |
| 37 | :class:`HTMLParser` instances have the following methods: |
| 38 | |
| 39 | |
| 40 | .. method:: HTMLParser.reset() |
| 41 | |
| 42 | Reset the instance. Loses all unprocessed data. This is called implicitly at |
| 43 | instantiation time. |
| 44 | |
| 45 | |
| 46 | .. method:: HTMLParser.feed(data) |
| 47 | |
| 48 | Feed some text to the parser. It is processed insofar as it consists of |
| 49 | complete elements; incomplete data is buffered until more data is fed or |
| 50 | :meth:`close` is called. |
| 51 | |
| 52 | |
| 53 | .. method:: HTMLParser.close() |
| 54 | |
| 55 | Force processing of all buffered data as if it were followed by an end-of-file |
| 56 | mark. This method may be redefined by a derived class to define additional |
| 57 | processing at the end of the input, but the redefined version should always call |
| 58 | the :class:`HTMLParser` base class method :meth:`close`. |
| 59 | |
| 60 | |
| 61 | .. method:: HTMLParser.getpos() |
| 62 | |
| 63 | Return current line number and offset. |
| 64 | |
| 65 | |
| 66 | .. method:: HTMLParser.get_starttag_text() |
| 67 | |
| 68 | Return the text of the most recently opened start tag. This should not normally |
| 69 | be needed for structured processing, but may be useful in dealing with HTML "as |
| 70 | deployed" or for re-generating input with minimal changes (whitespace between |
| 71 | attributes can be preserved, etc.). |
| 72 | |
| 73 | |
| 74 | .. method:: HTMLParser.handle_starttag(tag, attrs) |
| 75 | |
| 76 | This method is called to handle the start of a tag. It is intended to be |
| 77 | overridden by a derived class; the base class implementation does nothing. |
| 78 | |
| 79 | The *tag* argument is the name of the tag converted to lower case. The *attrs* |
| 80 | argument is a list of ``(name, value)`` pairs containing the attributes found |
| 81 | inside the tag's ``<>`` brackets. The *name* will be translated to lower case, |
| 82 | and quotes in the *value* have been removed, and character and entity references |
| 83 | have been replaced. For instance, for the tag ``<A |
| 84 | HREF="http://www.cwi.nl/">``, this method would be called as |
| 85 | ``handle_starttag('a', [('href', 'http://www.cwi.nl/')])``. |
| 86 | |
Georg Brandl | 9087b7f | 2008-05-18 07:53:01 +0000 | [diff] [blame] | 87 | All entity references from :mod:`html.entities` are replaced in the attribute |
| 88 | values. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 89 | |
| 90 | |
| 91 | .. method:: HTMLParser.handle_startendtag(tag, attrs) |
| 92 | |
| 93 | Similar to :meth:`handle_starttag`, but called when the parser encounters an |
| 94 | XHTML-style empty tag (``<a .../>``). This method may be overridden by |
| 95 | subclasses which require this particular lexical information; the default |
| 96 | implementation simple calls :meth:`handle_starttag` and :meth:`handle_endtag`. |
| 97 | |
| 98 | |
| 99 | .. method:: HTMLParser.handle_endtag(tag) |
| 100 | |
| 101 | This method is called to handle the end tag of an element. It is intended to be |
| 102 | overridden by a derived class; the base class implementation does nothing. The |
| 103 | *tag* argument is the name of the tag converted to lower case. |
| 104 | |
| 105 | |
| 106 | .. method:: HTMLParser.handle_data(data) |
| 107 | |
| 108 | This method is called to process arbitrary data. It is intended to be |
| 109 | overridden by a derived class; the base class implementation does nothing. |
| 110 | |
| 111 | |
| 112 | .. method:: HTMLParser.handle_charref(name) |
| 113 | |
| 114 | This method is called to process a character reference of the form ``&#ref;``. |
| 115 | It is intended to be overridden by a derived class; the base class |
| 116 | implementation does nothing. |
| 117 | |
| 118 | |
| 119 | .. method:: HTMLParser.handle_entityref(name) |
| 120 | |
| 121 | This method is called to process a general entity reference of the form |
| 122 | ``&name;`` where *name* is an general entity reference. It is intended to be |
| 123 | overridden by a derived class; the base class implementation does nothing. |
| 124 | |
| 125 | |
| 126 | .. method:: HTMLParser.handle_comment(data) |
| 127 | |
| 128 | This method is called when a comment is encountered. The *comment* argument is |
| 129 | a string containing the text between the ``--`` and ``--`` delimiters, but not |
| 130 | the delimiters themselves. For example, the comment ``<!--text-->`` will cause |
| 131 | this method to be called with the argument ``'text'``. It is intended to be |
| 132 | overridden by a derived class; the base class implementation does nothing. |
| 133 | |
| 134 | |
| 135 | .. method:: HTMLParser.handle_decl(decl) |
| 136 | |
Georg Brandl | feb4c88 | 2010-08-01 21:09:54 +0000 | [diff] [blame] | 137 | Method called when an SGML ``doctype`` declaration is read by the parser. |
| 138 | The *decl* parameter will be the entire contents of the declaration inside |
| 139 | the ``<!...>`` markup. It is intended to be overridden by a derived class; |
| 140 | the base class implementation does nothing. |
| 141 | |
| 142 | |
| 143 | .. method:: HTMLParser.unknown_decl(data) |
| 144 | |
| 145 | Method called when an unrecognized SGML declaration is read by the parser. |
| 146 | The *data* parameter will be the entire contents of the declaration inside |
| 147 | the ``<!...>`` markup. It is sometimes useful to be be overridden by a |
| 148 | derived class; the base class implementation throws an :exc:`HTMLParseError`. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 149 | |
| 150 | |
| 151 | .. method:: HTMLParser.handle_pi(data) |
| 152 | |
| 153 | Method called when a processing instruction is encountered. The *data* |
| 154 | parameter will contain the entire processing instruction. For example, for the |
| 155 | processing instruction ``<?proc color='red'>``, this method would be called as |
| 156 | ``handle_pi("proc color='red'")``. It is intended to be overridden by a derived |
| 157 | class; the base class implementation does nothing. |
| 158 | |
| 159 | .. note:: |
| 160 | |
| 161 | The :class:`HTMLParser` class uses the SGML syntactic rules for processing |
| 162 | instructions. An XHTML processing instruction using the trailing ``'?'`` will |
| 163 | cause the ``'?'`` to be included in *data*. |
| 164 | |
| 165 | |
| 166 | .. _htmlparser-example: |
| 167 | |
| 168 | Example HTML Parser Application |
| 169 | ------------------------------- |
| 170 | |
| 171 | As a basic example, below is a very basic HTML parser that uses the |
| 172 | :class:`HTMLParser` class to print out tags as they are encountered:: |
| 173 | |
Georg Brandl | f78dd34 | 2009-08-13 08:58:24 +0000 | [diff] [blame] | 174 | >>> from html.parser import HTMLParser |
| 175 | >>> |
| 176 | >>> class MyHTMLParser(HTMLParser): |
| 177 | ... def handle_starttag(self, tag, attrs): |
| 178 | ... print("Encountered a {} start tag".format(tag)) |
| 179 | ... def handle_endtag(self, tag): |
| 180 | ... print("Encountered a {} end tag".format(tag)) |
| 181 | ... |
| 182 | >>> page = """<html><h1>Title</h1><p>I'm a paragraph!</p></html>""" |
| 183 | >>> |
| 184 | >>> myparser = MyHTMLParser() |
| 185 | >>> myparser.feed(page) |
| 186 | Encountered a html start tag |
| 187 | Encountered a h1 start tag |
| 188 | Encountered a h1 end tag |
| 189 | Encountered a p start tag |
| 190 | Encountered a p end tag |
| 191 | Encountered a html end tag |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 192 | |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 193 | |