Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / libxml2 / 2644ab270ea5db157fff0e92a6c095a47577ce4f / . / doc / encoding.html
blob: 04d54e795789c96b23e9740f43e4cb3954e5b9c9 [file] [log] [blame]
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]1<?xml version="1.0" encoding="ISO-8859-1"?>
2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3<html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" /><link rel="SHORTCUT ICON" href="/favicon.ico" /><style type="text/css">
Daniel Veillard373a4752002-02-21 14:46:29 +0000[diff] [blame]4TD {font-family: Verdana,Arial,Helvetica}
5BODY {font-family: Verdana,Arial,Helvetica; margin-top: 2em; margin-left: 0em; margin-right: 0em}
6H1 {font-family: Verdana,Arial,Helvetica}
7H2 {font-family: Verdana,Arial,Helvetica}
8H3 {font-family: Verdana,Arial,Helvetica}
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]9A:link, A:visited, A:active { text-decoration: underline }
Daniel Veillardd49370e2005-04-11 23:28:16 +0000[diff] [blame]10</style><title>Encodings support</title></head><body bgcolor="#8b7765" text="#000000" link="#a06060" vlink="#000000"><table border="0" width="100%" cellpadding="5" cellspacing="0" align="center"><tr><td width="120"><a href="http://swpat.ffii.org/"><img src="epatents.png" alt="Action against software patents" /></a></td><td width="180"><a href="http://www.gnome.org/"><img src="gnome2.png" alt="Gnome2 Logo" /></a><a href="http://www.w3.org/Status"><img src="w3c.png" alt="W3C Logo" /></a><a href="http://www.redhat.com/"><img src="redhat.gif" alt="Red Hat Logo" /></a><div align="left"><a href="http://xmlsoft.org/"><img src="Libxml2-Logo-180x168.gif" alt="Made with Libxml2 Logo" /></a></div></td><td><table border="0" width="90%" cellpadding="2" cellspacing="0" align="center" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3" bgcolor="#fffacd"><tr><td align="center"><h1>The XML C parser and toolkit of Gnome</h1><h2>Encodings support</h2></td></tr></table></td></tr></table></td></tr></table><table border="0" cellpadding="4" cellspacing="0" width="100%" align="center"><tr><td bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="2" width="100%"><tr><td valign="top" width="200" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3"><tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Main Menu</b></center></td></tr><tr><td bgcolor="#fffacd"><form action="search.php" enctype="application/x-www-form-urlencoded" method="get"><input name="query" type="text" size="20" value="" /><input name="submit" type="submit" value="Search ..." /></form><ul><li><a href="index.html">Home</a></li><li><a href="http://xmlsoft.org/wiki">Wiki</a></li><li><a href="html/index.html">Reference Manual</a></li><li><a href="intro.html">Introduction</a></li><li><a href="FAQ.html">FAQ</a></li><li><a href="docs.html" style="font-weight:bold">Developer Menu</a></li><li><a href="bugs.html">Reporting bugs and getting help</a></li><li><a href="help.html">How to help</a></li><li><a href="downloads.html">Downloads</a></li><li><a href="news.html">Releases</a></li><li><a href="XMLinfo.html">XML</a></li><li><a href="XSLT.html">XSLT</a></li><li><a href="xmldtd.html">Validation &amp; DTDs</a></li><li><a href="encoding.html">Encodings support</a></li><li><a href="catalog.html">Catalog support</a></li><li><a href="namespaces.html">Namespaces</a></li><li><a href="contribs.html">Contributions</a></li><li><a href="examples/index.html" style="font-weight:bold">Code Examples</a></li><li><a href="html/index.html" style="font-weight:bold">API Menu</a></li><li><a href="guidelines.html">XML Guidelines</a></li><li><a href="ChangeLog.html">Recent Changes</a></li></ul></td></tr></table><table width="100%" border="0" cellspacing="1" cellpadding="3"><tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Related links</b></center></td></tr><tr><td bgcolor="#fffacd"><ul><li><a href="http://mail.gnome.org/archives/xml/">Mail archive</a></li><li><a href="http://xmlsoft.org/XSLT/">XSLT libxslt</a></li><li><a href="http://phd.cs.unibo.it/gdome2/">DOM gdome2</a></li><li><a href="http://www.aleksey.com/xmlsec/">XML-DSig xmlsec</a></li><li><a href="ftp://xmlsoft.org/">FTP</a></li><li><a href="http://www.zlatkovic.com/projects/libxml/">Windows binaries</a></li><li><a href="http://www.blastwave.org/packages.php/libxml2">Solaris binaries</a></li><li><a href="http://www.explain.com.au/oss/libxml2xslt.html">MacOsX binaries</a></li><li><a href="http://libxmlplusplus.sourceforge.net/">C++ bindings</a></li><li><a href="http://www.zend.com/php5/articles/php5-xmlphp.php#Heading4">PHP bindings</a></li><li><a href="http://sourceforge.net/projects/libxml2-pas/">Pascal bindings</a></li><li><a href="http://rubyforge.org/projects/xml-tools/">Ruby bindings</a></li><li><a href="http://tclxml.sourceforge.net/">Tcl bindings</a></li><li><a href="http://bugzilla.gnome.org/buglist.cgi?product=libxml2">Bug Tracker</a></li></ul></td></tr></table></td></tr></table></td><td valign="top" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%"><tr><td><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table border="0" cellpadding="3" cellspacing="1" width="100%"><tr><td bgcolor="#fffacd"><p>If you are not really familiar with Internationalization (usual shortcut
Daniel Veillarde5d68de2005-03-10 15:03:40 +0000[diff] [blame]11is I18N) , Unicode, characters and glyphs, I suggest you read a <a href="http://www.tbray.org/ongoing/When/200x/2003/04/06/Unicode">presentation</a>
12by Tim Bray on Unicode and why you should care about it.</p><p>If you don't understand why <b>it does not make sense to have a string
13without knowing what encoding it uses</b>, then as Joel Spolsky said
14<a href="http://www.joelonsoftware.com/articles/Unicode.html">please do
15not write another line of code until you finish reading that article.</a>.
16It is a prerequisite to understand this page, and avoid a lot of problems
17with libxml2, XML or text processing in general.</p><p>Table of Content:</p><ol><li><a href="encoding.html#What">What does internationalization support
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]18 mean ?</a></li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]19 <li><a href="encoding.html#internal">The internal encoding, how and
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]20 why</a></li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]21 <li><a href="encoding.html#implemente">How is it implemented ?</a></li>
22 <li><a href="encoding.html#Default">Default supported encodings</a></li>
23 <li><a href="encoding.html#extend">How to extend the existing
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]24 support</a></li>
Daniel Veillarde5d68de2005-03-10 15:03:40 +0000[diff] [blame]25</ol><h3><a name="What" id="What">What does internationalization support mean ?</a></h3><p>XML was designed from the start to allow the support of any character set
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]26by using Unicode. Any conformant XML parser has to support the UTF-8 and
27UTF-16 default encodings which can both express the full unicode ranges. UTF8
Daniel Veillard63d83142002-05-20 06:51:05 +0000[diff] [blame]28is a variable length encoding whose greatest points are to reuse the same
29encoding for ASCII and to save space for Western encodings, but it is a bit
Daniel Veillardabfca612004-01-07 23:38:02 +0000[diff] [blame]30more complex to handle in practice. UTF-16 use 2 bytes per character (and
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]31sometimes combines two pairs), it makes implementation easier, but looks a
32bit overkill for Western languages encoding. Moreover the XML specification
Daniel Veillard189f46b2004-01-25 21:03:04 +0000[diff] [blame]33allows the document to be encoded in other encodings at the condition that
34they are clearly labeled as such. For example the following is a wellformed
35XML document encoded in ISO-8859-1 and using accentuated letters that we
36French like for both markup and content:</p><pre>&lt;?xml version="1.0" encoding="ISO-8859-1"?&gt;
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]37&lt;très&gt;là&lt;/très&gt;</pre><p>Having internationalization support in libxml2 means the following:</p><ul><li>the document is properly parsed</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]38 <li>informations about it's encoding are saved</li>
39 <li>it can be modified</li>
40 <li>it can be saved in its original encoding</li>
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]41 <li>it can also be saved in another encoding supported by libxml2 (for
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]42 example straight UTF8 or even an ASCII form)</li>
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]43</ul><p>Another very important point is that the whole libxml2 API, with the
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]44exception of a few routines to read with a specific encoding or save to a
45specific encoding, is completely agnostic about the original encoding of the
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]46document.</p><p>It should be noted too that the HTML parser embedded in libxml2 now obey
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]47the same rules too, the following document will be (as of 2.2.2) handled in
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]48an internationalized fashion by libxml2 too:</p><pre>&lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
49 "http://www.w3.org/TR/REC-html40/loose.dtd"&gt;
50&lt;html lang="fr"&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]51&lt;head&gt;
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]52 &lt;META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1"&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]53&lt;/head&gt;
54&lt;body&gt;
55&lt;p&gt;W3C crée des standards pour le Web.&lt;/body&gt;
Daniel Veillardabfca612004-01-07 23:38:02 +0000[diff] [blame]56&lt;/html&gt;</pre><h3><a name="internal" id="internal">The internal encoding, how and why</a></h3><p>One of the core decisions was to force all documents to be converted to a
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]57default internal encoding, and that encoding to be UTF-8, here are the
Daniel Veillardabfca612004-01-07 23:38:02 +0000[diff] [blame]58rationales for those choices:</p><ul><li>keeping the native encoding in the internal form would force the libxml
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]59 users (or the code associated) to be fully aware of the encoding of the
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]60 original document, for examples when adding a text node to a document,
61 the content would have to be provided in the document encoding, i.e. the
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]62 client code would have to check it before hand, make sure it's conformant
63 to the encoding, etc ... Very hard in practice, though in some specific
64 cases this may make sense.</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]65 <li>the second decision was which encoding. From the XML spec only UTF8 and
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]66 UTF16 really makes sense as being the two only encodings for which there
Daniel Veillard63d83142002-05-20 06:51:05 +0000[diff] [blame]67 is mandatory support. UCS-4 (32 bits fixed size encoding) could be
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]68 considered an intelligent choice too since it's a direct Unicode mapping
69 support. I selected UTF-8 on the basis of efficiency and compatibility
70 with surrounding software:
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]71 <ul><li>UTF-8 while a bit more complex to convert from/to (i.e. slightly
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]72 more costly to import and export CPU wise) is also far more compact
73 than UTF-16 (and UCS-4) for a majority of the documents I see it used
74 for right now (RPM RDF catalogs, advogato data, various configuration
75 file formats, etc.) and the key point for today's computer
76 architecture is efficient uses of caches. If one nearly double the
77 memory requirement to store the same amount of data, this will trash
78 caches (main memory/external caches/internal caches) and my take is
79 that this harms the system far more than the CPU requirements needed
80 for the conversion to UTF-8</li>
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]81 <li>Most of libxml2 version 1 users were using it with straight ASCII
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]82 most of the time, doing the conversion with an internal encoding
83 requiring all their code to be rewritten was a serious show-stopper
84 for using UTF-16 or UCS-4.</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]85 <li>UTF-8 is being used as the de-facto internal encoding standard for
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]86 related code like the <a href="http://www.pango.org/">pango</a>
Daniel Veillardabfca612004-01-07 23:38:02 +0000[diff] [blame]87 upcoming Gnome text widget, and a lot of Unix code (yet another place
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]88 where Unix programmer base takes a different approach from Microsoft
89 - they are using UTF-16)</li>
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]90 </ul></li>
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]91</ul><p>What does this mean in practice for the libxml2 user:</p><ul><li>xmlChar, the libxml2 data type is a byte, those bytes must be assembled
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]92 as UTF-8 valid strings. The proper way to terminate an xmlChar * string
93 is simply to append 0 byte, as usual.</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]94 <li>One just need to make sure that when using chars outside the ASCII set,
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]95 the values has been properly converted to UTF-8</li>
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]96</ul><h3><a name="implemente" id="implemente">How is it implemented ?</a></h3><p>Let's describe how all this works within libxml, basically the I18N
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]97(internationalization) support get triggered only during I/O operation, i.e.
98when reading a document or saving one. Let's look first at the reading
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]99sequence:</p><ol><li>when a document is processed, we usually don't know the encoding, a
Daniel Veillard189f46b2004-01-25 21:03:04 +0000[diff] [blame]100 simple heuristic allows to detect UTF-16 and UCS-4 from encodings where
101 the ASCII range (0-0x7F) maps with ASCII</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]102 <li>the xml declaration if available is parsed, including the encoding
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]103 declaration. At that point, if the autodetected encoding is different
104 from the one declared a call to xmlSwitchEncoding() is issued.</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]105 <li>If there is no encoding declaration, then the input has to be in either
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]106 UTF-8 or UTF-16, if it is not then at some point when processing the
107 input, the converter/checker of UTF-8 form will raise an encoding error.
108 You may end-up with a garbled document, or no document at all ! Example:
109 <pre>~/XML -&gt; ./xmllint err.xml
110err.xml:1: error: Input is not proper UTF-8, indicate encoding !
111&lt;très&gt;là&lt;/très&gt;
112 ^
113err.xml:1: error: Bytes: 0xE8 0x73 0x3E 0x6C
114&lt;très&gt;là&lt;/très&gt;
115 ^</pre>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]116 </li>
117 <li>xmlSwitchEncoding() does an encoding name lookup, canonicalize it, and
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]118 then search the default registered encoding converters for that encoding.
119 If it's not within the default set and iconv() support has been compiled
120 it, it will ask iconv for such an encoder. If this fails then the parser
121 will report an error and stops processing:
122 <pre>~/XML -&gt; ./xmllint err2.xml
123err2.xml:1: error: Unsupported encoding UnsupportedEnc
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]124&lt;?xml version="1.0" encoding="UnsupportedEnc"?&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]125 ^</pre>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]126 </li>
127 <li>From that point the encoder processes progressively the input (it is
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]128 plugged as a front-end to the I/O module) for that entity. It captures
Daniel Veillardabfca612004-01-07 23:38:02 +0000[diff] [blame]129 and converts on-the-fly the document to be parsed to UTF-8. The parser
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]130 itself just does UTF-8 checking of this input and process it
131 transparently. The only difference is that the encoding information has
132 been added to the parsing context (more precisely to the input
133 corresponding to this entity).</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]134 <li>The result (when using DOM) is an internal form completely in UTF-8
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]135 with just an encoding information on the document node.</li>
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]136</ol><p>Ok then what happens when saving the document (assuming you
Daniel Veillard63d83142002-05-20 06:51:05 +0000[diff] [blame]137collected/built an xmlDoc DOM like structure) ? It depends on the function
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]138called, xmlSaveFile() will just try to save in the original encoding, while
139xmlSaveFileTo() and xmlSaveFileEnc() can optionally save to a given
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]140encoding:</p><ol><li>if no encoding is given, libxml2 will look for an encoding value
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]141 associated to the document and if it exists will try to save to that
142 encoding,
143 <p>otherwise everything is written in the internal form, i.e. UTF-8</p>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]144 </li>
145 <li>so if an encoding was specified, either at the API level or on the
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]146 document, libxml2 will again canonicalize the encoding name, lookup for a
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]147 converter in the registered set or through iconv. If not found the
148 function will return an error code</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]149 <li>the converter is placed before the I/O buffer layer, as another kind of
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]150 buffer, then libxml2 will simply push the UTF-8 serialization to through
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]151 that buffer, which will then progressively be converted and pushed onto
152 the I/O layer.</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]153 <li>It is possible that the converter code fails on some input, for example
Daniel Veillard63d83142002-05-20 06:51:05 +0000[diff] [blame]154 trying to push an UTF-8 encoded Chinese character through the UTF-8 to
Daniel Veillard0d6b1702000-08-22 23:52:16 +0000[diff] [blame]155 ISO-8859-1 converter won't work. Since the encoders are progressive they
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]156 will just report the error and the number of bytes converted, at that
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]157 point libxml2 will decode the offending character, remove it from the
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]158 buffer and replace it with the associated charRef encoding &amp;#123; and
Daniel Veillard63d83142002-05-20 06:51:05 +0000[diff] [blame]159 resume the conversion. This guarantees that any document will be saved
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]160 without losses (except for markup names where this is not legal, this is
Daniel Veillard63d83142002-05-20 06:51:05 +0000[diff] [blame]161 a problem in the current version, in practice avoid using non-ascii
Daniel Veillard189f46b2004-01-25 21:03:04 +0000[diff] [blame]162 characters for tag or attribute names). A special "ascii" encoding name
163 is used to save documents to a pure ascii form can be used when
Daniel Veillard0d6b1702000-08-22 23:52:16 +0000[diff] [blame]164 portability is really crucial</li>
Daniel Veillardabfca612004-01-07 23:38:02 +0000[diff] [blame]165</ol><p>Here are a few examples based on the same test document:</p><pre>~/XML -&gt; ./xmllint isolat1
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]166&lt;?xml version="1.0" encoding="ISO-8859-1"?&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]167&lt;très&gt;là&lt;/très&gt;
168~/XML -&gt; ./xmllint --encode UTF-8 isolat1
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]169&lt;?xml version="1.0" encoding="UTF-8"?&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]170&lt;très&gt;là  &lt;/très&gt;
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]171~/XML -&gt; </pre><p>The same processing is applied (and reuse most of the code) for HTML I18N
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]172processing. Looking up and modifying the content encoding is a bit more
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]173difficult since it is located in a &lt;meta&gt; tag under the &lt;head&gt;,
174so a couple of functions htmlGetMetaEncoding() and htmlSetMetaEncoding() have
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]175been provided. The parser also attempts to switch encoding on the fly when
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]176detecting such a tag on input. Except for that the processing is the same
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]177(and again reuses the same code).</p><h3><a name="Default" id="Default">Default supported encodings</a></h3><p>libxml2 has a set of default converters for the following encodings
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]178(located in encoding.c):</p><ol><li>UTF-8 is supported by default (null handlers)</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]179 <li>UTF-16, both little and big endian</li>
180 <li>ISO-Latin-1 (ISO-8859-1) covering most western languages</li>
181 <li>ASCII, useful mostly for saving</li>
182 <li>HTML, a specific handler for the conversion of UTF-8 to ASCII with HTML
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]183 predefined entities like &amp;copy; for the Copyright sign.</li>
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]184</ol><p>More over when compiled on an Unix platform with iconv support the full
Daniel Veillardc0801af2002-05-28 16:28:42 +0000[diff] [blame]185set of encodings supported by iconv can be instantly be used by libxml. On a
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]186linux machine with glibc-2.1 the list of supported encodings and aliases fill
1873 full pages, and include UCS-4, the full set of ISO-Latin encodings, and the
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]188various Japanese ones.</p><h4>Encoding aliases</h4><p>From 2.2.3, libxml2 has support to register encoding names aliases. The
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]189goal is to be able to parse document whose encoding is supported but where
190the name differs (for example from the default set of names accepted by
191iconv). The following functions allow to register and handle new aliases for
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]192existing encodings. Once registered libxml2 will automatically lookup the
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]193aliases when handling a document:</p><ul><li>int xmlAddEncodingAlias(const char *name, const char *alias);</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]194 <li>int xmlDelEncodingAlias(const char *alias);</li>
195 <li>const char * xmlGetEncodingAlias(const char *alias);</li>
196 <li>void xmlCleanupEncodingAliases(void);</li>
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]197</ul><h3><a name="extend" id="extend">How to extend the existing support</a></h3><p>Well adding support for new encoding, or overriding one of the encoders
Daniel Veillardabfca612004-01-07 23:38:02 +0000[diff] [blame]198(assuming it is buggy) should not be hard, just write input and output
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]199conversion routines to/from UTF-8, and register them using
200xmlNewCharEncodingHandler(name, xxxToUTF8, UTF8Toxxx), and they will be
201called automatically if the parser(s) encounter such an encoding name
202(register it uppercase, this will help). The description of the encoders,
203their arguments and expected return values are described in the encoding.h
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]204header.</p><p>A quick note on the topic of subverting the parser to use a different
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]205internal encoding than UTF-8, in some case people will absolutely want to
206keep the internal encoding different, I think it's still possible (but the
207encoding must be compliant with ASCII on the same subrange) though I didn't
208tried it. The key is to override the default conversion routines (by
209registering null encoders/decoders for your charsets), and bypass the UTF-8
210checking of the parser by setting the parser context charset
211(ctxt-&gt;charset) to something different than XML_CHAR_ENCODING_UTF8, but
Daniel Veillard63d83142002-05-20 06:51:05 +0000[diff] [blame]212there is no guarantee that this will work. You may also have some troubles
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]213saving back.</p><p>Basically proper I18N support is important, this requires at least
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]214libxml-2.0.0, but a lot of features and corrections are really available only
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]215starting 2.2.</p><p><a href="bugs.html">Daniel Veillard</a></p></td></tr></table></td></tr></table></td></tr></table></td></tr></table></td></tr></table></body></html>