Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 2 | <html> |
| 3 | <head> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 4 | <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> |
| 5 | <style type="text/css"><!-- |
| 6 | TD {font-size: 10pt; font-family: Verdana,Arial,Helvetica} |
| 7 | BODY {font-size: 10pt; font-family: Verdana,Arial,Helvetica; margin-top: 5pt; margin-left: 0pt; margin-right: 0pt} |
| 8 | H1 {font-size: 16pt; font-family: Verdana,Arial,Helvetica} |
| 9 | H2 {font-size: 14pt; font-family: Verdana,Arial,Helvetica} |
| 10 | H3 {font-size: 12pt; font-family: Verdana,Arial,Helvetica} |
| 11 | A:link, A:visited, A:active { text-decoration: underline } |
| 12 | --></style> |
| 13 | <title>Memory Management</title> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 14 | </head> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 15 | <body bgcolor="#8b7765" text="#000000" link="#000000" vlink="#000000"> |
| 16 | <table border="0" width="100%" cellpadding="5" cellspacing="0" align="center"><tr> |
| 17 | <td width="180"> |
| 18 | <a href="http://www.gnome.org/"><img src="smallfootonly.gif" alt="Gnome Logo"></a><a href="http://www.w3.org/Status"><img src="w3c.png" alt="W3C Logo"></a><a href="http://www.redhat.com/"><img src="redhat.gif" alt="Red Hat Logo"></a> |
| 19 | </td> |
| 20 | <td><table border="0" width="90%" cellpadding="2" cellspacing="0" align="center" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3" bgcolor="#fffacd"><tr><td align="center"> |
| 21 | <h1>The XML C library for Gnome</h1> |
| 22 | <h2>Memory Management</h2> |
| 23 | </td></tr></table></td></tr></table></td> |
| 24 | </tr></table> |
| 25 | <table border="0" cellpadding="4" cellspacing="0" width="100%" align="center"><tr><td bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="2" width="100%"><tr> |
| 26 | <td valign="top" width="200" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td> |
| 27 | <table width="100%" border="0" cellspacing="1" cellpadding="3"> |
| 28 | <tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Main Menu</b></center></td></tr> |
| 29 | <tr><td bgcolor="#fffacd"><ul style="margin-left: -2pt"> |
| 30 | <li><a href="index.html">Home</a></li> |
| 31 | <li><a href="intro.html">Introduction</a></li> |
| 32 | <li><a href="FAQ.html">FAQ</a></li> |
| 33 | <li><a href="docs.html">Documentation</a></li> |
| 34 | <li><a href="bugs.html">Reporting bugs and getting help</a></li> |
| 35 | <li><a href="help.html">How to help</a></li> |
| 36 | <li><a href="downloads.html">Downloads</a></li> |
| 37 | <li><a href="news.html">News</a></li> |
| 38 | <li><a href="XML.html">XML</a></li> |
| 39 | <li><a href="XSLT.html">XSLT</a></li> |
| 40 | <li><a href="architecture.html">libxml architecture</a></li> |
| 41 | <li><a href="tree.html">The tree output</a></li> |
| 42 | <li><a href="interface.html">The SAX interface</a></li> |
| 43 | <li><a href="xmldtd.html">Validation & DTDs</a></li> |
| 44 | <li><a href="xmlmem.html">Memory Management</a></li> |
| 45 | <li><a href="encoding.html">Encodings support</a></li> |
| 46 | <li><a href="xmlio.html">I/O Interfaces</a></li> |
| 47 | <li><a href="catalog.html">Catalog support</a></li> |
| 48 | <li><a href="library.html">The parser interfaces</a></li> |
| 49 | <li><a href="entities.html">Entities or no entities</a></li> |
| 50 | <li><a href="namespaces.html">Namespaces</a></li> |
| 51 | <li><a href="upgrade.html">Upgrading 1.x code</a></li> |
| 52 | <li><a href="DOM.html">DOM Principles</a></li> |
| 53 | <li><a href="example.html">A real example</a></li> |
| 54 | <li><a href="contribs.html">Contributions</a></li> |
| 55 | <li> |
| 56 | <a href="xml.html">flat page</a>, <a href="site.xsl">stylesheet</a> |
| 57 | </li> |
| 58 | </ul></td></tr> |
| 59 | </table> |
| 60 | <table width="100%" border="0" cellspacing="1" cellpadding="3"> |
| 61 | <tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Related links</b></center></td></tr> |
| 62 | <tr><td bgcolor="#fffacd"><ul style="margin-left: -2pt"> |
| 63 | <li><a href="http://mail.gnome.org/archives/xml/">Mail archive</a></li> |
| 64 | <li><a href="http://xmlsoft.org/XSLT/">XSLT libxslt</a></li> |
| 65 | <li><a href="http://www.cs.unibo.it/~casarini/gdome2/">DOM gdome2</a></li> |
| 66 | <li><a href="ftp://xmlsoft.org/">FTP</a></li> |
| 67 | <li><a href="http://www.fh-frankfurt.de/~igor/projects/libxml/">Windows binaries</a></li> |
| 68 | <li><a href="http://pages.eidosnet.co.uk/~garypen/libxml/">Solaris binaries</a></li> |
Daniel Veillard | c6271d2 | 2001-10-27 07:50:58 +0000 | [diff] [blame] | 69 | <li><a href="http://bugzilla.gnome.org/buglist.cgi?product=libxml">Bug Tracker</a></li> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 70 | </ul></td></tr> |
| 71 | </table> |
| 72 | </td></tr></table></td> |
| 73 | <td valign="top" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%"><tr><td><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table border="0" cellpadding="3" cellspacing="1" width="100%"><tr><td bgcolor="#fffacd"> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 74 | <p>Table of Content:</p> |
| 75 | <ol> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 76 | <li><a href="#General3">General overview</a></li> |
Daniel Veillard | 9c46682 | 2001-10-25 12:03:39 +0000 | [diff] [blame] | 77 | <li><a href="#setting">Setting libxml set of memory routines</a></li> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 78 | <li><a href="#cleanup">Cleaning up after parsing</a></li> |
| 79 | <li><a href="#Debugging">Debugging routines</a></li> |
| 80 | <li><a href="#General4">General memory requirements</a></li> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 81 | </ol> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 82 | <h3><a name="General3">General overview</a></h3> |
| 83 | <p>The module <code><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlmemory.h</a></code> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 84 | provides the interfaces to the libxml memory system:</p> |
| 85 | <ul> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 86 | <li>libxml does not use the libc memory allocator directly but xmlFree(), |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 87 | xmlMalloc() and xmlRealloc()</li> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 88 | <li>those routines can be reallocated to a specific set of routine, by |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 89 | default the libc ones i.e. free(), malloc() and realloc()</li> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 90 | <li>the xmlmemory.c module includes a set of debugging routine</li> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 91 | </ul> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 92 | <h3><a name="setting">Setting libxml set of memory routines</a></h3> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 93 | <p>It is sometimes useful to not use the default memory allocator, either for |
| 94 | debugging, analysis or to implement a specific behaviour on memory management |
| 95 | (like on embedded systems). Two function calls are available to do so:</p> |
| 96 | <ul> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 97 | <li> |
| 98 | <a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemGet ()</a> |
| 99 | which return the current set of functions in use by the parser</li> |
| 100 | <li> |
| 101 | <a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemSetup()</a> |
| 102 | which allow to set up a new set of memory allocation functions</li> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 103 | </ul> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 104 | <p>Of course a call to xmlMemSetup() should probably be done before calling |
| 105 | any other libxml routines (unless you are sure your allocations routines are |
| 106 | compatibles).</p> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 107 | <h3><a name="cleanup">Cleaning up after parsing</a></h3> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 108 | <p>Libxml is not stateless, there is a few set of memory structures needing |
| 109 | allocation before the parser is fully functionnal (some encoding structures |
| 110 | for example). This also mean that once parsing is finished there is a tiny |
| 111 | amount of memory (a few hundred bytes) which can be recollected if you don't |
| 112 | reuse the parser immediately:</p> |
| 113 | <ul> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 114 | <li> |
| 115 | <a href="http://xmlsoft.org/html/libxml-parser.html">xmlCleanupParser |
| 116 | ()</a> |
| 117 | is a centralized routine to free the parsing states. Note that it won't |
| 118 | deallocate any produced tree if any (use the xmlFreeDoc() and related |
| 119 | routines for this).</li> |
| 120 | <li> |
| 121 | <a href="http://xmlsoft.org/html/libxml-parser.html">xmlInitParser |
| 122 | ()</a> |
| 123 | is the dual routine allowing to preallocate the parsing state which can |
| 124 | be useful for example to avoid initialization reentrancy problems when |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 125 | using libxml in multithreaded applications</li> |
| 126 | </ul> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 127 | <p>Generally xmlCleanupParser() is safe, if needed the state will be rebuild |
| 128 | at the next invocation of parser routines, but be careful of the consequences |
| 129 | in multithreaded applications.</p> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 130 | <h3><a name="Debugging">Debugging routines</a></h3> |
| 131 | <p>When configured using --with-mem-debug flag (off by default), libxml uses |
| 132 | a set of memory allocation debugging routineskeeping track of all allocated |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 133 | blocks and the location in the code where the routine was called. A couple of |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 134 | other debugging routines allow to dump the memory allocated infos to a file |
| 135 | or call a specific routine when a given block number is allocated:</p> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 136 | <ul> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 137 | <li> |
| 138 | <a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMallocLoc()</a><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlReallocLoc()</a> |
| 139 | and <a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemStrdupLoc()</a> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 140 | are the memory debugging replacement allocation routines</li> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 141 | <li> |
| 142 | <a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemoryDump |
| 143 | ()</a> |
| 144 | dumps all the informations about the allocated memory block lefts in the |
| 145 | <code>.memdump</code> file</li> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 146 | </ul> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 147 | <p>When developping libxml memory debug is enabled, the tests programs call |
| 148 | xmlMemoryDump () and the "make test" regression tests will check for any |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 149 | memory leak during the full regression test sequence, this helps a lot |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 150 | ensuring that libxml does not leak memory and bullet proof memory |
| 151 | allocations use (some libc implementations are known to be far too permissive |
| 152 | resulting in major portability problems!).</p> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 153 | <p>If the .memdump reports a leak, it displays the allocation function and |
| 154 | also tries to give some informations about the content and structure of the |
| 155 | allocated blocks left. This is sufficient in most cases to find the culprit, |
| 156 | but not always. Assuming the allocation problem is reproductible, it is |
| 157 | possible to find more easilly:</p> |
| 158 | <ol> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 159 | <li>write down the block number xxxx not allocated</li> |
| 160 | <li>export the environement variable XML_MEM_BREAKPOINT=xxxx</li> |
| 161 | <li>run the program under a debugger and set a breakpoint on |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 162 | xmlMallocBreakpoint() a specific function called when this precise block |
| 163 | is allocated</li> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 164 | <li>when the breakpoint is reached you can then do a fine analysis of the |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 165 | allocation an step to see the condition resulting in the missing |
| 166 | deallocation.</li> |
| 167 | </ol> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 168 | <p>I used to use a commercial tool to debug libxml memory problems but after |
| 169 | noticing that it was not detecting memory leaks that simple mechanism was |
| 170 | used and proved extremely efficient until now.</p> |
| 171 | <h3><a name="General4">General memory requirements</a></h3> |
| 172 | <p>How much libxml memory require ? It's hard to tell in average it depends |
| 173 | of a number of things:</p> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 174 | <ul> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 175 | <li>the parser itself should work in a fixed amout of memory, except for |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 176 | information maintained about the stacks of names and entities locations. |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 177 | The I/O and encoding handlers will probably account for a few KBytes. |
| 178 | This is true for both the XML and HTML parser (though the HTML parser |
| 179 | need more state).</li> |
| 180 | <li>If you are generating the DOM tree then memory requirements will grow |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 181 | nearly lineary with the size of the data. In general for a balanced |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 182 | textual document the internal memory requirement is about 4 times the |
| 183 | size of the UTF8 serialization of this document (exmple the XML-1.0 |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 184 | recommendation is a bit more of 150KBytes and takes 650KBytes of main |
| 185 | memory when parsed). Validation will add a amount of memory required for |
| 186 | maintaining the external Dtd state which should be linear with the |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 187 | complexity of the content model defined by the Dtd</li> |
| 188 | <li>If you don't care about the advanced features of libxml like |
| 189 | validation, DOM, XPath or XPointer, but really need to work fixed memory |
| 190 | requirements, then the SAX interface should be used.</li> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 191 | </ul> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 192 | <p> |
Daniel Veillard | c5d6434 | 2001-06-24 12:13:24 +0000 | [diff] [blame] | 193 | <p><a href="mailto:daniel@veillard.com">Daniel Veillard</a></p> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 194 | </td></tr></table></td></tr></table></td></tr></table></td> |
| 195 | </tr></table></td></tr></table> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 196 | </body> |
| 197 | </html> |