Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / python / cpython2 / 8f1dd224e002f93b67821a7a55db35152928d6da / . / Doc / library / htmlparser.rst
blob: 6740b43367fa531a41fa315b64802c4816a314d1 [file] [log] [blame]
Fred Draked995e112008-05-20 06:08:38 +0000[diff] [blame]1
2:mod:`HTMLParser` --- Simple HTML and XHTML parser
3==================================================
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]4
Fred Drake20b56602008-05-17 21:23:02 +0000[diff] [blame]5.. module:: HTMLParser
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]6 :synopsis: A simple parser that can handle HTML and XHTML.
7
Fred Drake20b56602008-05-17 21:23:02 +0000[diff] [blame]8.. note::
Georg Brandl3682dfe2008-05-20 07:21:58 +0000[diff] [blame]9
10 The :mod:`HTMLParser` module has been renamed to :mod:`html.parser` in Python
Ezio Melotti87033522011-10-28 14:20:08 +0300[diff] [blame]11 3. The :term:`2to3` tool will automatically adapt imports when converting
12 your sources to Python 3.
Fred Drake20b56602008-05-17 21:23:02 +0000[diff] [blame]13
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]14
15.. versionadded:: 2.2
16
17.. index::
18 single: HTML
19 single: XHTML
20
Éric Araujo29a0b572011-08-19 02:14:03 +0200[diff] [blame]21**Source code:** :source:`Lib/HTMLParser.py`
22
23--------------
24
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]25This module defines a class :class:`.HTMLParser` which serves as the basis for
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]26parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML.
27Unlike the parser in :mod:`htmllib`, this parser is not based on the SGML parser
28in :mod:`sgmllib`.
29
30
31.. class:: HTMLParser()
32
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]33 An :class:`.HTMLParser` instance is fed HTML data and calls handler methods
34 when start tags, end tags, text, comments, and other markup elements are
35 encountered. The user should subclass :class:`.HTMLParser` and override its
36 methods to implement the desired behavior.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]37
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]38 The :class:`.HTMLParser` class is instantiated without arguments.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]39
40 Unlike the parser in :mod:`htmllib`, this parser does not check that end tags
41 match start tags or call the end-tag handler for elements which are closed
42 implicitly by closing an outer element.
43
44An exception is defined as well:
45
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]46.. exception:: HTMLParseError
47
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]48 :class:`.HTMLParser` is able to handle broken markup, but in some cases it
49 might raise this exception when it encounters an error while parsing.
50 This exception provides three attributes: :attr:`msg` is a brief
51 message explaining the error, :attr:`lineno` is the number of the line on
52 which the broken construct was detected, and :attr:`offset` is the number of
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]53 characters into the line at which the construct starts.
54
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]55
56Example HTML Parser Application
57-------------------------------
58
59As a basic example, below is a simple HTML parser that uses the
60:class:`.HTMLParser` class to print out start tags, end tags and data
61as they are encountered::
62
63 from HTMLParser import HTMLParser
64
65 # create a subclass and override the handler methods
66 class MyHTMLParser(HTMLParser):
67 def handle_starttag(self, tag, attrs):
68 print "Encountered a start tag:", tag
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]69
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]70 def handle_endtag(self, tag):
71 print "Encountered an end tag :", tag
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]72
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]73 def handle_data(self, data):
74 print "Encountered some data :", data
75
76 # instantiate the parser and fed it some HTML
77 parser = MyHTMLParser()
78 parser.feed('<html><head><title>Test</title></head>'
79 '<body><h1>Parse me!</h1></body></html>')
80
Martin Panter8f1dd222016-07-26 11:18:21 +0200[diff] [blame^]81The output will then be:
82
83.. code-block:: none
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]84
85 Encountered a start tag: html
86 Encountered a start tag: head
87 Encountered a start tag: title
88 Encountered some data : Test
89 Encountered an end tag : title
90 Encountered an end tag : head
91 Encountered a start tag: body
92 Encountered a start tag: h1
93 Encountered some data : Parse me!
94 Encountered an end tag : h1
95 Encountered an end tag : body
96 Encountered an end tag : html
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]97
98
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]99:class:`.HTMLParser` Methods
100----------------------------
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]101
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]102:class:`.HTMLParser` instances have the following methods:
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]103
104
105.. method:: HTMLParser.feed(data)
106
107 Feed some text to the parser. It is processed insofar as it consists of
108 complete elements; incomplete data is buffered until more data is fed or
Ezio Melottid0ffcd62011-12-19 07:15:26 +0200[diff] [blame]109 :meth:`close` is called. *data* can be either :class:`unicode` or
110 :class:`str`, but passing :class:`unicode` is advised.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]111
112
113.. method:: HTMLParser.close()
114
115 Force processing of all buffered data as if it were followed by an end-of-file
116 mark. This method may be redefined by a derived class to define additional
117 processing at the end of the input, but the redefined version should always call
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]118 the :class:`.HTMLParser` base class method :meth:`close`.
119
120
121.. method:: HTMLParser.reset()
122
123 Reset the instance. Loses all unprocessed data. This is called implicitly at
124 instantiation time.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]125
126
127.. method:: HTMLParser.getpos()
128
129 Return current line number and offset.
130
131
132.. method:: HTMLParser.get_starttag_text()
133
134 Return the text of the most recently opened start tag. This should not normally
135 be needed for structured processing, but may be useful in dealing with HTML "as
136 deployed" or for re-generating input with minimal changes (whitespace between
137 attributes can be preserved, etc.).
138
139
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]140The following methods are called when data or markup elements are encountered
141and they are meant to be overridden in a subclass. The base class
142implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`):
143
144
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]145.. method:: HTMLParser.handle_starttag(tag, attrs)
146
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]147 This method is called to handle the start of a tag (e.g. ``<div id="main">``).
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]148
149 The *tag* argument is the name of the tag converted to lower case. The *attrs*
150 argument is a list of ``(name, value)`` pairs containing the attributes found
151 inside the tag's ``<>`` brackets. The *name* will be translated to lower case,
152 and quotes in the *value* have been removed, and character and entity references
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]153 have been replaced.
154
Serhiy Storchakab4905ef2016-05-07 10:50:12 +0300[diff] [blame]155 For instance, for the tag ``<A HREF="https://www.cwi.nl/">``, this method
156 would be called as ``handle_starttag('a', [('href', 'https://www.cwi.nl/')])``.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]157
158 .. versionchanged:: 2.6
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]159 All entity references from :mod:`htmlentitydefs` are now replaced in the
160 attribute values.
161
162
163.. method:: HTMLParser.handle_endtag(tag)
164
165 This method is called to handle the end tag of an element (e.g. ``</div>``).
166
167 The *tag* argument is the name of the tag converted to lower case.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]168
169
170.. method:: HTMLParser.handle_startendtag(tag, attrs)
171
172 Similar to :meth:`handle_starttag`, but called when the parser encounters an
Ezio Melotti87033522011-10-28 14:20:08 +0300[diff] [blame]173 XHTML-style empty tag (``<img ... />``). This method may be overridden by
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]174 subclasses which require this particular lexical information; the default
Ezio Melotti87033522011-10-28 14:20:08 +0300[diff] [blame]175 implementation simply calls :meth:`handle_starttag` and :meth:`handle_endtag`.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]176
177
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]178.. method:: HTMLParser.handle_data(data)
179
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]180 This method is called to process arbitrary data (e.g. text nodes and the
181 content of ``<script>...</script>`` and ``<style>...</style>``).
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]182
183
184.. method:: HTMLParser.handle_entityref(name)
185
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]186 This method is called to process a named character reference of the form
187 ``&name;`` (e.g. ``&gt;``), where *name* is a general entity reference
188 (e.g. ``'gt'``).
189
190
191.. method:: HTMLParser.handle_charref(name)
192
193 This method is called to process decimal and hexadecimal numeric character
194 references of the form ``&#NNN;`` and ``&#xNNN;``. For example, the decimal
195 equivalent for ``&gt;`` is ``&#62;``, whereas the hexadecimal is ``&#x3E;``;
196 in this case the method will receive ``'62'`` or ``'x3E'``.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]197
198
199.. method:: HTMLParser.handle_comment(data)
200
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]201 This method is called when a comment is encountered (e.g. ``<!--comment-->``).
202
203 For example, the comment ``<!-- comment -->`` will cause this method to be
204 called with the argument ``' comment '``.
205
206 The content of Internet Explorer conditional comments (condcoms) will also be
207 sent to this method, so, for ``<!--[if IE 9]>IE9-specific content<![endif]-->``,
R David Murrayf79aa582015-08-24 12:50:50 -0400[diff] [blame]208 this method will receive ``'[if IE 9]>IE9-specific content<![endif]'``.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]209
210
211.. method:: HTMLParser.handle_decl(decl)
212
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]213 This method is called to handle an HTML doctype declaration (e.g.
214 ``<!DOCTYPE html>``).
215
Georg Brandlc79d4322010-08-01 21:10:57 +0000[diff] [blame]216 The *decl* parameter will be the entire contents of the declaration inside
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]217 the ``<!...>`` markup (e.g. ``'DOCTYPE html'``).
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]218
219
220.. method:: HTMLParser.handle_pi(data)
221
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]222 This method is called when a processing instruction is encountered. The *data*
223 parameter will contain the entire processing instruction. For example, for the
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]224 processing instruction ``<?proc color='red'>``, this method would be called as
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]225 ``handle_pi("proc color='red'")``.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]226
227 .. note::
228
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]229 The :class:`.HTMLParser` class uses the SGML syntactic rules for processing
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]230 instructions. An XHTML processing instruction using the trailing ``'?'`` will
231 cause the ``'?'`` to be included in *data*.
232
233
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]234.. method:: HTMLParser.unknown_decl(data)
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]235
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]236 This method is called when an unrecognized declaration is read by the parser.
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]237
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]238 The *data* parameter will be the entire contents of the declaration inside
239 the ``<![...]>`` markup. It is sometimes useful to be overridden by a
240 derived class.
241
242
243.. _htmlparser-examples:
244
245Examples
246--------
247
248The following class implements a parser that will be used to illustrate more
249examples::
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]250
Fred Draked995e112008-05-20 06:08:38 +0000[diff] [blame]251 from HTMLParser import HTMLParser
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]252 from htmlentitydefs import name2codepoint
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]253
254 class MyHTMLParser(HTMLParser):
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]255 def handle_starttag(self, tag, attrs):
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]256 print "Start tag:", tag
257 for attr in attrs:
258 print " attr:", attr
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]259
Georg Brandl8ec7f652007-08-15 14:28:01 +0000[diff] [blame]260 def handle_endtag(self, tag):
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]261 print "End tag :", tag
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]262
Ezio Melottif9cc80d2011-10-28 14:14:34 +0300[diff] [blame]263 def handle_data(self, data):
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]264 print "Data :", data
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]265
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]266 def handle_comment(self, data):
267 print "Comment :", data
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]268
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]269 def handle_entityref(self, name):
270 c = unichr(name2codepoint[name])
271 print "Named ent:", c
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]272
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]273 def handle_charref(self, name):
274 if name.startswith('x'):
275 c = unichr(int(name[1:], 16))
276 else:
277 c = unichr(int(name))
278 print "Num ent :", c
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]279
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]280 def handle_decl(self, data):
281 print "Decl :", data
Ezio Melottif9cc80d2011-10-28 14:14:34 +0300[diff] [blame]282
283 parser = MyHTMLParser()
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]284
285Parsing a doctype::
286
287 >>> parser.feed('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" '
288 ... '"http://www.w3.org/TR/html4/strict.dtd">')
289 Decl : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"
290
291Parsing an element with a few attributes and a title::
292
293 >>> parser.feed('<img src="python-logo.png" alt="The Python logo">')
294 Start tag: img
295 attr: ('src', 'python-logo.png')
296 attr: ('alt', 'The Python logo')
297 >>>
298 >>> parser.feed('<h1>Python</h1>')
299 Start tag: h1
300 Data : Python
301 End tag : h1
302
303The content of ``script`` and ``style`` elements is returned as is, without
304further parsing::
305
306 >>> parser.feed('<style type="text/css">#python { color: green }</style>')
307 Start tag: style
308 attr: ('type', 'text/css')
309 Data : #python { color: green }
310 End tag : style
Serhiy Storchaka12d547a2016-05-10 13:45:32 +0300[diff] [blame]311
Ezio Melottic39b5522012-02-18 01:46:04 +0200[diff] [blame]312 >>> parser.feed('<script type="text/javascript">'
313 ... 'alert("<strong>hello!</strong>");</script>')
314 Start tag: script
315 attr: ('type', 'text/javascript')
316 Data : alert("<strong>hello!</strong>");
317 End tag : script
318
319Parsing comments::
320
321 >>> parser.feed('<!-- a comment -->'
322 ... '<!--[if IE 9]>IE-specific content<![endif]-->')
323 Comment : a comment
324 Comment : [if IE 9]>IE-specific content<![endif]
325
326Parsing named and numeric character references and converting them to the
327correct char (note: these 3 references are all equivalent to ``'>'``)::
328
329 >>> parser.feed('&gt;&#62;&#x3E;')
330 Named ent: >
331 Num ent : >
332 Num ent : >
333
334Feeding incomplete chunks to :meth:`~HTMLParser.feed` works, but
335:meth:`~HTMLParser.handle_data` might be called more than once::
336
337 >>> for chunk in ['<sp', 'an>buff', 'ered ', 'text</s', 'pan>']:
338 ... parser.feed(chunk)
339 ...
340 Start tag: span
341 Data : buff
342 Data : ered
343 Data : text
344 End tag : span
345
346Parsing invalid HTML (e.g. unquoted attributes) also works::
347
348 >>> parser.feed('<p><a class=link href=#main>tag soup</p ></a>')
349 Start tag: p
350 Start tag: a
351 attr: ('class', 'link')
352 attr: ('href', '#main')
353 Data : tag soup
354 End tag : p
355 End tag : a