Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / libxml2 / 7cb3fa9d51a0a57d1e538854e4be84db82b75098 / . / doc / encoding.html
blob: 4b6d0d47888b107891e5a33295b3aad19cd53977 [file] [log] [blame]
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]1<?xml version="1.0" encoding="ISO-8859-1"?>
2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3<html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" /><link rel="SHORTCUT ICON" href="/favicon.ico" /><style type="text/css">
Daniel Veillard373a4752002-02-21 14:46:29 +0000[diff] [blame]4TD {font-family: Verdana,Arial,Helvetica}
5BODY {font-family: Verdana,Arial,Helvetica; margin-top: 2em; margin-left: 0em; margin-right: 0em}
6H1 {font-family: Verdana,Arial,Helvetica}
7H2 {font-family: Verdana,Arial,Helvetica}
8H3 {font-family: Verdana,Arial,Helvetica}
Daniel Veillardb8cfbd12001-10-25 10:53:28 +0000[diff] [blame]9A:link, A:visited, A:active { text-decoration: underline }
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]10</style><title>Encodings support</title></head><body bgcolor="#8b7765" text="#000000" link="#a06060" vlink="#000000"><table border="0" width="100%" cellpadding="5" cellspacing="0" align="center"><tr><td width="120"><a href="http://swpat.ffii.org/"><img src="epatents.png" alt="Action against software patents" /></a></td><td width="180"><a href="http://www.gnome.org/"><img src="gnome2.png" alt="Gnome2 Logo" /></a><a href="http://www.w3.org/Status"><img src="w3c.png" alt="W3C Logo" /></a><a href="http://www.redhat.com/"><img src="redhat.gif" alt="Red Hat Logo" /></a><div align="left"><a href="http://xmlsoft.org/"><img src="Libxml2-Logo-180x168.gif" alt="Made with Libxml2 Logo" /></a></div></td><td><table border="0" width="90%" cellpadding="2" cellspacing="0" align="center" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3" bgcolor="#fffacd"><tr><td align="center"><h1>The XML C parser and toolkit of Gnome</h1><h2>Encodings support</h2></td></tr></table></td></tr></table></td></tr></table><table border="0" cellpadding="4" cellspacing="0" width="100%" align="center"><tr><td bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="2" width="100%"><tr><td valign="top" width="200" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3"><tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Main Menu</b></center></td></tr><tr><td bgcolor="#fffacd"><form action="search.php" enctype="application/x-www-form-urlencoded" method="get"><input name="query" type="text" size="20" value="" /><input name="submit" type="submit" value="Search ..." /></form><ul><li><a href="index.html">Home</a></li><li><a href="html/index.html">Reference Manual</a></li><li><a href="intro.html">Introduction</a></li><li><a href="FAQ.html">FAQ</a></li><li><a href="docs.html" style="font-weight:bold">Developer Menu</a></li><li><a href="bugs.html">Reporting bugs and getting help</a></li><li><a href="help.html">How to help</a></li><li><a href="downloads.html">Downloads</a></li><li><a href="news.html">Releases</a></li><li><a href="XMLinfo.html">XML</a></li><li><a href="XSLT.html">XSLT</a></li><li><a href="xmldtd.html">Validation &amp; DTDs</a></li><li><a href="encoding.html">Encodings support</a></li><li><a href="catalog.html">Catalog support</a></li><li><a href="namespaces.html">Namespaces</a></li><li><a href="contribs.html">Contributions</a></li><li><a href="examples/index.html" style="font-weight:bold">Code Examples</a></li><li><a href="html/index.html" style="font-weight:bold">API Menu</a></li><li><a href="guidelines.html">XML Guidelines</a></li><li><a href="ChangeLog.html">Recent Changes</a></li></ul></td></tr></table><table width="100%" border="0" cellspacing="1" cellpadding="3"><tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Related links</b></center></td></tr><tr><td bgcolor="#fffacd"><ul><li><a href="http://mail.gnome.org/archives/xml/">Mail archive</a></li><li><a href="http://xmlsoft.org/XSLT/">XSLT libxslt</a></li><li><a href="http://phd.cs.unibo.it/gdome2/">DOM gdome2</a></li><li><a href="http://www.aleksey.com/xmlsec/">XML-DSig xmlsec</a></li><li><a href="ftp://xmlsoft.org/">FTP</a></li><li><a href="http://www.zlatkovic.com/projects/libxml/">Windows binaries</a></li><li><a href="http://www.blastwave.org/packages.php/libxml2">Solaris binaries</a></li><li><a href="http://www.explain.com.au/oss/libxml2xslt.html">MacOsX binaries</a></li><li><a href="http://libxmlplusplus.sourceforge.net/">C++ bindings</a></li><li><a href="http://www.zend.com/php5/articles/php5-xmlphp.php#Heading4">PHP bindings</a></li><li><a href="http://sourceforge.net/projects/libxml2-pas/">Pascal bindings</a></li><li><a href="http://libxml.rubyforge.org/">Ruby bindings</a></li><li><a href="http://tclxml.sourceforge.net/">Tcl bindings</a></li><li><a href="http://bugzilla.gnome.org/buglist.cgi?product=libxml2">Bug Tracker</a></li></ul></td></tr></table></td></tr></table></td><td valign="top" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%"><tr><td><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table border="0" cellpadding="3" cellspacing="1" width="100%"><tr><td bgcolor="#fffacd"><p>If you are not really familiar with Internationalization (usual shortcutis
11I18N) , Unicode, characters and glyphs, I suggest you read a <a href="http://www.tbray.org/ongoing/When/200x/2003/04/06/Unicode">presentation</a>by
12Tim Bray on Unicode and why you should care about it.</p><p>If you don't understand why <b>it does not make sense to have a
13stringwithout knowing what encoding it uses</b>, then as Joel Spolsky said <a href="http://www.joelonsoftware.com/articles/Unicode.html">please do notwrite
14another line of code until you finish reading that article.</a>. It isa
15prerequisite to understand this page, and avoid a lot of problems
16withlibxml2, XML or text processing in general.</p><p>Table of Content:</p><ol><li><a href="encoding.html#What">What does internationalization supportmean
17 ?</a></li>
18 <li><a href="encoding.html#internal">The internal encoding, how
19 andwhy</a></li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]20 <li><a href="encoding.html#implemente">How is it implemented ?</a></li>
21 <li><a href="encoding.html#Default">Default supported encodings</a></li>
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]22 <li><a href="encoding.html#extend">How to extend the
23 existingsupport</a></li>
24</ol><h3><a name="What" id="What">What does internationalization support mean ?</a></h3><p>XML was designed from the start to allow the support of any character
25setby using Unicode. Any conformant XML parser has to support the UTF-8
26andUTF-16 default encodings which can both express the full unicode ranges.
27UTF8is a variable length encoding whose greatest points are to reuse the
28sameencoding for ASCII and to save space for Western encodings, but it is a
29bitmore complex to handle in practice. UTF-16 use 2 bytes per character
30(andsometimes combines two pairs), it makes implementation easier, but looks
31abit overkill for Western languages encoding. Moreover the XML
32specificationallows the document to be encoded in other encodings at the
33condition thatthey are clearly labeled as such. For example the following is
34a wellformedXML document encoded in ISO-8859-1 and using accentuated letters
35that weFrench like for both markup and content:</p><pre>&lt;?xml version="1.0" encoding="ISO-8859-1"?&gt;
Daniel Veillard8a469172003-06-12 16:05:07 +0000[diff] [blame]36&lt;très&gt;là&lt;/très&gt;</pre><p>Having internationalization support in libxml2 means the following:</p><ul><li>the document is properly parsed</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]37 <li>informations about it's encoding are saved</li>
38 <li>it can be modified</li>
39 <li>it can be saved in its original encoding</li>
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]40 <li>it can also be saved in another encoding supported by libxml2
41 (forexample straight UTF8 or even an ASCII form)</li>
42</ul><p>Another very important point is that the whole libxml2 API, with
43theexception of a few routines to read with a specific encoding or save to
44aspecific encoding, is completely agnostic about the original encoding of
45thedocument.</p><p>It should be noted too that the HTML parser embedded in libxml2 now
46obeythe same rules too, the following document will be (as of 2.2.2) handled
47inan internationalized fashion by libxml2 too:</p><pre>&lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]48 "http://www.w3.org/TR/REC-html40/loose.dtd"&gt;
49&lt;html lang="fr"&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]50&lt;head&gt;
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]51 &lt;META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1"&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]52&lt;/head&gt;
53&lt;body&gt;
54&lt;p&gt;W3C crée des standards pour le Web.&lt;/body&gt;
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]55&lt;/html&gt;</pre><h3><a name="internal" id="internal">The internal encoding, how and why</a></h3><p>One of the core decisions was to force all documents to be converted to
56adefault internal encoding, and that encoding to be UTF-8, here are
57therationales for those choices:</p><ul><li>keeping the native encoding in the internal form would force the
58 libxmlusers (or the code associated) to be fully aware of the encoding of
59 theoriginal document, for examples when adding a text node to a
60 document,the content would have to be provided in the document encoding,
61 i.e. theclient code would have to check it before hand, make sure it's
62 conformantto the encoding, etc ... Very hard in practice, though in some
63 specificcases this may make sense.</li>
64 <li>the second decision was which encoding. From the XML spec only UTF8
65 andUTF16 really makes sense as being the two only encodings for which
66 thereis mandatory support. UCS-4 (32 bits fixed size encoding) could
67 beconsidered an intelligent choice too since it's a direct Unicode
68 mappingsupport. I selected UTF-8 on the basis of efficiency and
69 compatibilitywith surrounding software:
70 <ul><li>UTF-8 while a bit more complex to convert from/to (i.e.
71 slightlymore costly to import and export CPU wise) is also far more
72 compactthan UTF-16 (and UCS-4) for a majority of the documents I see
73 it usedfor right now (RPM RDF catalogs, advogato data, various
74 configurationfile formats, etc.) and the key point for today's
75 computerarchitecture is efficient uses of caches. If one nearly
76 double thememory requirement to store the same amount of data, this
77 will trashcaches (main memory/external caches/internal caches) and my
78 take isthat this harms the system far more than the CPU requirements
79 neededfor the conversion to UTF-8</li>
80 <li>Most of libxml2 version 1 users were using it with straight
81 ASCIImost of the time, doing the conversion with an internal
82 encodingrequiring all their code to be rewritten was a serious
83 show-stopperfor using UTF-16 or UCS-4.</li>
84 <li>UTF-8 is being used as the de-facto internal encoding standard
85 forrelated code like the <a href="http://www.pango.org/">pango</a>upcoming Gnome text widget, and
86 a lot of Unix code (yet another placewhere Unix programmer base takes
87 a different approach from Microsoft- they are using UTF-16)</li>
Daniel Veillard1177ca42003-04-26 22:29:54 +0000[diff] [blame]88 </ul></li>
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]89</ul><p>What does this mean in practice for the libxml2 user:</p><ul><li>xmlChar, the libxml2 data type is a byte, those bytes must be
90 assembledas UTF-8 valid strings. The proper way to terminate an xmlChar *
91 stringis simply to append 0 byte, as usual.</li>
92 <li>One just need to make sure that when using chars outside the ASCII
93 set,the values has been properly converted to UTF-8</li>
94</ul><h3><a name="implemente" id="implemente">How is it implemented ?</a></h3><p>Let's describe how all this works within libxml, basically the
95I18N(internationalization) support get triggered only during I/O operation,
96i.e.when reading a document or saving one. Let's look first at the
97readingsequence:</p><ol><li>when a document is processed, we usually don't know the encoding,
98 asimple heuristic allows to detect UTF-16 and UCS-4 from encodings
99 wherethe ASCII range (0-0x7F) maps with ASCII</li>
100 <li>the xml declaration if available is parsed, including the
101 encodingdeclaration. At that point, if the autodetected encoding is
102 differentfrom the one declared a call to xmlSwitchEncoding() is
103 issued.</li>
104 <li>If there is no encoding declaration, then the input has to be in
105 eitherUTF-8 or UTF-16, if it is not then at some point when processing
106 theinput, the converter/checker of UTF-8 form will raise an encoding
107 error.You may end-up with a garbled document, or no document at all !
108 Example:
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]109 <pre>~/XML -&gt; ./xmllint err.xml
110err.xml:1: error: Input is not proper UTF-8, indicate encoding !
111&lt;très&gt;là&lt;/très&gt;
112 ^
113err.xml:1: error: Bytes: 0xE8 0x73 0x3E 0x6C
114&lt;très&gt;là&lt;/très&gt;
115 ^</pre>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]116 </li>
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]117 <li>xmlSwitchEncoding() does an encoding name lookup, canonicalize it,
118 andthen search the default registered encoding converters for that
119 encoding.If it's not within the default set and iconv() support has been
120 compiledit, it will ask iconv for such an encoder. If this fails then the
121 parserwill report an error and stops processing:
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]122 <pre>~/XML -&gt; ./xmllint err2.xml
123err2.xml:1: error: Unsupported encoding UnsupportedEnc
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]124&lt;?xml version="1.0" encoding="UnsupportedEnc"?&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]125 ^</pre>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]126 </li>
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]127 <li>From that point the encoder processes progressively the input (it
128 isplugged as a front-end to the I/O module) for that entity. It
129 capturesand converts on-the-fly the document to be parsed to UTF-8. The
130 parseritself just does UTF-8 checking of this input and process
131 ittransparently. The only difference is that the encoding information
132 hasbeen added to the parsing context (more precisely to the
133 inputcorresponding to this entity).</li>
134 <li>The result (when using DOM) is an internal form completely in UTF-8with
135 just an encoding information on the document node.</li>
136</ol><p>Ok then what happens when saving the document (assuming youcollected/built
137an xmlDoc DOM like structure) ? It depends on the functioncalled,
138xmlSaveFile() will just try to save in the original encoding,
139whilexmlSaveFileTo() and xmlSaveFileEnc() can optionally save to a
140givenencoding:</p><ol><li>if no encoding is given, libxml2 will look for an encoding
141 valueassociated to the document and if it exists will try to save to
142 thatencoding,
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]143 <p>otherwise everything is written in the internal form, i.e. UTF-8</p>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]144 </li>
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]145 <li>so if an encoding was specified, either at the API level or on
146 thedocument, libxml2 will again canonicalize the encoding name, lookup
147 for aconverter in the registered set or through iconv. If not found
148 thefunction will return an error code</li>
149 <li>the converter is placed before the I/O buffer layer, as another kind
150 ofbuffer, then libxml2 will simply push the UTF-8 serialization to
151 throughthat buffer, which will then progressively be converted and pushed
152 ontothe I/O layer.</li>
153 <li>It is possible that the converter code fails on some input, for
154 exampletrying to push an UTF-8 encoded Chinese character through the
155 UTF-8 toISO-8859-1 converter won't work. Since the encoders are
156 progressive theywill just report the error and the number of bytes
157 converted, at thatpoint libxml2 will decode the offending character,
158 remove it from thebuffer and replace it with the associated charRef
159 encoding &amp;#123; andresume the conversion. This guarantees that any
160 document will be savedwithout losses (except for markup names where this
161 is not legal, this isa problem in the current version, in practice avoid
162 using non-asciicharacters for tag or attribute names). A special "ascii"
163 encoding nameis used to save documents to a pure ascii form can be used
164 whenportability is really crucial</li>
Daniel Veillardabfca612004-01-07 23:38:02 +0000[diff] [blame]165</ol><p>Here are a few examples based on the same test document:</p><pre>~/XML -&gt; ./xmllint isolat1
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]166&lt;?xml version="1.0" encoding="ISO-8859-1"?&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]167&lt;très&gt;là&lt;/très&gt;
168~/XML -&gt; ./xmllint --encode UTF-8 isolat1
Daniel Veillard024f1992003-12-10 16:43:49 +0000[diff] [blame]169&lt;?xml version="1.0" encoding="UTF-8"?&gt;
Daniel Veillardbe40c8b2000-07-14 12:10:59 +0000[diff] [blame]170&lt;très&gt;là  &lt;/très&gt;
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]171~/XML -&gt; </pre><p>The same processing is applied (and reuse most of the code) for HTML
172I18Nprocessing. Looking up and modifying the content encoding is a bit
173moredifficult since it is located in a &lt;meta&gt; tag under the
174&lt;head&gt;,so a couple of functions htmlGetMetaEncoding() and
175htmlSetMetaEncoding() havebeen provided. The parser also attempts to switch
176encoding on the fly whendetecting such a tag on input. Except for that the
177processing is the same(and again reuses the same code).</p><h3><a name="Default" id="Default">Default supported encodings</a></h3><p>libxml2 has a set of default converters for the following
178encodings(located in encoding.c):</p><ol><li>UTF-8 is supported by default (null handlers)</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]179 <li>UTF-16, both little and big endian</li>
180 <li>ISO-Latin-1 (ISO-8859-1) covering most western languages</li>
181 <li>ASCII, useful mostly for saving</li>
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]182 <li>HTML, a specific handler for the conversion of UTF-8 to ASCII with
183 HTMLpredefined entities like &amp;copy; for the Copyright sign.</li>
184</ol><p>More over when compiled on an Unix platform with iconv support the fullset
185of encodings supported by iconv can be instantly be used by libxml. On alinux
186machine with glibc-2.1 the list of supported encodings and aliases fill3 full
187pages, and include UCS-4, the full set of ISO-Latin encodings, and thevarious
188Japanese ones.</p><p>To convert from the UTF-8 values returned from the API to another
189encodingthen it is possible to use the function provided from <a href="html/libxml-encoding.html">the encoding module</a>like <a href="html/libxml-encoding.html#UTF8Toisolat1">UTF8Toisolat1</a>, or use
190thePOSIX <a href="http://www.opengroup.org/onlinepubs/009695399/functions/iconv.html">iconv()</a>API
191directly.</p><h4>Encoding aliases</h4><p>From 2.2.3, libxml2 has support to register encoding names aliases.
192Thegoal is to be able to parse document whose encoding is supported but
193wherethe name differs (for example from the default set of names accepted
194byiconv). The following functions allow to register and handle new aliases
195forexisting encodings. Once registered libxml2 will automatically lookup
196thealiases when handling a document:</p><ul><li>int xmlAddEncodingAlias(const char *name, const char *alias);</li>
Daniel Veillard0b28e882002-07-24 23:47:05 +0000[diff] [blame]197 <li>int xmlDelEncodingAlias(const char *alias);</li>
198 <li>const char * xmlGetEncodingAlias(const char *alias);</li>
199 <li>void xmlCleanupEncodingAliases(void);</li>
Daniel Veillard69839ba2006-06-06 13:27:03 +0000[diff] [blame]200</ul><h3><a name="extend" id="extend">How to extend the existing support</a></h3><p>Well adding support for new encoding, or overriding one of the
201encoders(assuming it is buggy) should not be hard, just write input and
202outputconversion routines to/from UTF-8, and register them
203usingxmlNewCharEncodingHandler(name, xxxToUTF8, UTF8Toxxx), and they will
204becalled automatically if the parser(s) encounter such an encoding
205name(register it uppercase, this will help). The description of the
206encoders,their arguments and expected return values are described in the
207encoding.hheader.</p><p><a href="bugs.html">Daniel Veillard</a></p></td></tr></table></td></tr></table></td></tr></table></td></tr></table></td></tr></table></body></html>