Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 1 | <?xml version="1.0" encoding="ISO-8859-1"?> |
| 2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| 3 | <html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" /><link rel="SHORTCUT ICON" href="/favicon.ico" /><style type="text/css"> |
Daniel Veillard | 373a475 | 2002-02-21 14:46:29 +0000 | [diff] [blame] | 4 | TD {font-family: Verdana,Arial,Helvetica} |
| 5 | BODY {font-family: Verdana,Arial,Helvetica; margin-top: 2em; margin-left: 0em; margin-right: 0em} |
| 6 | H1 {font-family: Verdana,Arial,Helvetica} |
| 7 | H2 {font-family: Verdana,Arial,Helvetica} |
| 8 | H3 {font-family: Verdana,Arial,Helvetica} |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 9 | A:link, A:visited, A:active { text-decoration: underline } |
William M. Brack | fd9e70d | 2004-02-26 11:01:57 +0000 | [diff] [blame] | 10 | </style><title>Memory Management</title></head><body bgcolor="#8b7765" text="#000000" link="#000000" vlink="#000000"><table border="0" width="100%" cellpadding="5" cellspacing="0" align="center"><tr><td width="120"><a href="http://swpat.ffii.org/"><img src="epatents.png" alt="Action against software patents" /></a></td><td width="180"><a href="http://www.gnome.org/"><img src="gnome2.png" alt="Gnome2 Logo" /></a><a href="http://www.w3.org/Status"><img src="w3c.png" alt="W3C Logo" /></a><a href="http://www.redhat.com/"><img src="redhat.gif" alt="Red Hat Logo" /></a><div align="left"><a href="http://xmlsoft.org/"><img src="Libxml2-Logo-180x168.gif" alt="Made with Libxml2 Logo" /></a></div></td><td><table border="0" width="90%" cellpadding="2" cellspacing="0" align="center" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3" bgcolor="#fffacd"><tr><td align="center"><h1>The XML C parser and toolkit of Gnome</h1><h2>Memory Management</h2></td></tr></table></td></tr></table></td></tr></table><table border="0" cellpadding="4" cellspacing="0" width="100%" align="center"><tr><td bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="2" width="100%"><tr><td valign="top" width="200" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3"><tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Developer Menu</b></center></td></tr><tr><td bgcolor="#fffacd"><form action="search.php" enctype="application/x-www-form-urlencoded" method="get"><input name="query" type="text" size="20" value="" /><input name="submit" type="submit" value="Search ..." /></form><ul><li><a href="index.html" style="font-weight:bold">Main Menu</a></li><li><a href="examples/index.html" style="font-weight:bold">Code Examples</a></li><li><a href="guidelines.html">XML Guidelines</a></li><li><a href="tutorial/index.html">Tutorial</a></li><li><a href="xmlreader.html">The Reader Interface</a></li><li><a href="XSLT.html">XSLT</a></li><li><a href="python.html">Python and bindings</a></li><li><a href="architecture.html">libxml2 architecture</a></li><li><a href="tree.html">The tree output</a></li><li><a href="interface.html">The SAX interface</a></li><li><a href="xmlmem.html">Memory Management</a></li><li><a href="xmlio.html">I/O Interfaces</a></li><li><a href="library.html">The parser interfaces</a></li><li><a href="entities.html">Entities or no entities</a></li><li><a href="namespaces.html">Namespaces</a></li><li><a href="upgrade.html">Upgrading 1.x code</a></li><li><a href="threads.html">Thread safety</a></li><li><a href="DOM.html">DOM Principles</a></li><li><a href="example.html">A real example</a></li><li><a href="xml.html">flat page</a>, <a href="site.xsl">stylesheet</a></li></ul></td></tr></table><table width="100%" border="0" cellspacing="1" cellpadding="3"><tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>API Indexes</b></center></td></tr><tr><td bgcolor="#fffacd"><ul><li><a href="APIchunk0.html">Alphabetic</a></li><li><a href="APIconstructors.html">Constructors</a></li><li><a href="APIfunctions.html">Functions/Types</a></li><li><a href="APIfiles.html">Modules</a></li><li><a href="APIsymbols.html">Symbols</a></li></ul></td></tr></table><table width="100%" border="0" cellspacing="1" cellpadding="3"><tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Related links</b></center></td></tr><tr><td bgcolor="#fffacd"><ul><li><a href="http://mail.gnome.org/archives/xml/">Mail archive</a></li><li><a href="http://xmlsoft.org/XSLT/">XSLT libxslt</a></li><li><a href="http://phd.cs.unibo.it/gdome2/">DOM gdome2</a></li><li><a href="http://www.aleksey.com/xmlsec/">XML-DSig xmlsec</a></li><li><a href="ftp://xmlsoft.org/">FTP</a></li><li><a href="http://www.zlatkovic.com/projects/libxml/">Windows binaries</a></li><li><a href="http://garypennington.net/libxml2/">Solaris binaries</a></li><li><a href="http://www.zveno.com/open_source/libxml2xslt.html">MacOsX binaries</a></li><li><a href="http://sourceforge.net/projects/libxml2-pas/">Pascal bindings</a></li><li><a href="http://bugzilla.gnome.org/buglist.cgi?product=libxml2">Bug Tracker</a></li></ul></td></tr></table></td></tr></table></td><td valign="top" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%"><tr><td><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table border="0" cellpadding="3" cellspacing="1" width="100%"><tr><td bgcolor="#fffacd"><p>Table of Content:</p><ol><li><a href="#General3">General overview</a></li> |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 11 | <li><a href="#setting">Setting libxml2 set of memory routines</a></li> |
Daniel Veillard | 0b28e88 | 2002-07-24 23:47:05 +0000 | [diff] [blame] | 12 | <li><a href="#cleanup">Cleaning up after parsing</a></li> |
| 13 | <li><a href="#Debugging">Debugging routines</a></li> |
| 14 | <li><a href="#General4">General memory requirements</a></li> |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 15 | </ol><h3><a name="General3" id="General3">General overview</a></h3><p>The module <code><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlmemory.h</a></code> |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 16 | provides the interfaces to the libxml2 memory system:</p><ul><li>libxml2 does not use the libc memory allocator directly but xmlFree(), |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 17 | xmlMalloc() and xmlRealloc()</li> |
Daniel Veillard | 0b28e88 | 2002-07-24 23:47:05 +0000 | [diff] [blame] | 18 | <li>those routines can be reallocated to a specific set of routine, by |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 19 | default the libc ones i.e. free(), malloc() and realloc()</li> |
Daniel Veillard | 0b28e88 | 2002-07-24 23:47:05 +0000 | [diff] [blame] | 20 | <li>the xmlmemory.c module includes a set of debugging routine</li> |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 21 | </ul><h3><a name="setting" id="setting">Setting libxml2 set of memory routines</a></h3><p>It is sometimes useful to not use the default memory allocator, either for |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 22 | debugging, analysis or to implement a specific behaviour on memory management |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 23 | (like on embedded systems). Two function calls are available to do so:</p><ul><li><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemGet |
Daniel Veillard | af43f63 | 2002-03-08 15:05:20 +0000 | [diff] [blame] | 24 | ()</a> which return the current set of functions in use by the parser</li> |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 25 | <li><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemSetup()</a> |
Daniel Veillard | af43f63 | 2002-03-08 15:05:20 +0000 | [diff] [blame] | 26 | which allow to set up a new set of memory allocation functions</li> |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 27 | </ul><p>Of course a call to xmlMemSetup() should probably be done before calling |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 28 | any other libxml2 routines (unless you are sure your allocations routines are |
| 29 | compatibles).</p><h3><a name="cleanup" id="cleanup">Cleaning up after parsing</a></h3><p>Libxml2 is not stateless, there is a few set of memory structures needing |
Daniel Veillard | 63d8314 | 2002-05-20 06:51:05 +0000 | [diff] [blame] | 30 | allocation before the parser is fully functional (some encoding structures |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 31 | for example). This also mean that once parsing is finished there is a tiny |
| 32 | amount of memory (a few hundred bytes) which can be recollected if you don't |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 33 | reuse the parser immediately:</p><ul><li><a href="http://xmlsoft.org/html/libxml-parser.html">xmlCleanupParser |
Daniel Veillard | af43f63 | 2002-03-08 15:05:20 +0000 | [diff] [blame] | 34 | ()</a> is a centralized routine to free the parsing states. Note that it |
| 35 | won't deallocate any produced tree if any (use the xmlFreeDoc() and |
| 36 | related routines for this).</li> |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 37 | <li><a href="http://xmlsoft.org/html/libxml-parser.html">xmlInitParser |
Daniel Veillard | af43f63 | 2002-03-08 15:05:20 +0000 | [diff] [blame] | 38 | ()</a> is the dual routine allowing to preallocate the parsing state |
| 39 | which can be useful for example to avoid initialization reentrancy |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 40 | problems when using libxml2 in multithreaded applications</li> |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 41 | </ul><p>Generally xmlCleanupParser() is safe, if needed the state will be rebuild |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 42 | at the next invocation of parser routines, but be careful of the consequences |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 43 | in multithreaded applications.</p><h3><a name="Debugging" id="Debugging">Debugging routines</a></h3><p>When configured using --with-mem-debug flag (off by default), libxml2 uses |
Daniel Veillard | 63d8314 | 2002-05-20 06:51:05 +0000 | [diff] [blame] | 44 | a set of memory allocation debugging routines keeping track of all allocated |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 45 | blocks and the location in the code where the routine was called. A couple of |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 46 | other debugging routines allow to dump the memory allocated infos to a file |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 47 | or call a specific routine when a given block number is allocated:</p><ul><li><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMallocLoc()</a> |
Daniel Veillard | 0b28e88 | 2002-07-24 23:47:05 +0000 | [diff] [blame] | 48 | <a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlReallocLoc()</a> |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 49 | and <a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemStrdupLoc()</a> |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 50 | are the memory debugging replacement allocation routines</li> |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 51 | <li><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemoryDump |
Daniel Veillard | af43f63 | 2002-03-08 15:05:20 +0000 | [diff] [blame] | 52 | ()</a> dumps all the informations about the allocated memory block lefts |
| 53 | in the <code>.memdump</code> file</li> |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 54 | </ul><p>When developing libxml2 memory debug is enabled, the tests programs call |
Daniel Veillard | 024f199 | 2003-12-10 16:43:49 +0000 | [diff] [blame] | 55 | xmlMemoryDump () and the "make test" regression tests will check for any |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 56 | memory leak during the full regression test sequence, this helps a lot |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 57 | ensuring that libxml2 does not leak memory and bullet proof memory |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 58 | allocations use (some libc implementations are known to be far too permissive |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 59 | resulting in major portability problems!).</p><p>If the .memdump reports a leak, it displays the allocation function and |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 60 | also tries to give some informations about the content and structure of the |
| 61 | allocated blocks left. This is sufficient in most cases to find the culprit, |
Daniel Veillard | 63d8314 | 2002-05-20 06:51:05 +0000 | [diff] [blame] | 62 | but not always. Assuming the allocation problem is reproducible, it is |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 63 | possible to find more easily:</p><ol><li>write down the block number xxxx not allocated</li> |
Daniel Veillard | 0b28e88 | 2002-07-24 23:47:05 +0000 | [diff] [blame] | 64 | <li>export the environment variable XML_MEM_BREAKPOINT=xxxx , the easiest |
Daniel Veillard | a8a89fe | 2002-04-12 21:03:34 +0000 | [diff] [blame] | 65 | when using GDB is to simply give the command |
| 66 | <p><code>set environment XML_MEM_BREAKPOINT xxxx</code></p> |
Daniel Veillard | 0b28e88 | 2002-07-24 23:47:05 +0000 | [diff] [blame] | 67 | <p>before running the program.</p> |
| 68 | </li> |
| 69 | <li>run the program under a debugger and set a breakpoint on |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 70 | xmlMallocBreakpoint() a specific function called when this precise block |
| 71 | is allocated</li> |
Daniel Veillard | 0b28e88 | 2002-07-24 23:47:05 +0000 | [diff] [blame] | 72 | <li>when the breakpoint is reached you can then do a fine analysis of the |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 73 | allocation an step to see the condition resulting in the missing |
| 74 | deallocation.</li> |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 75 | </ol><p>I used to use a commercial tool to debug libxml2 memory problems but after |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 76 | noticing that it was not detecting memory leaks that simple mechanism was |
Daniel Veillard | a8a89fe | 2002-04-12 21:03:34 +0000 | [diff] [blame] | 77 | used and proved extremely efficient until now. Lately I have also used <a href="http://developer.kde.org/~sewardj/">valgrind</a> with quite some |
| 78 | success, it is tied to the i386 architecture since it works by emulating the |
| 79 | processor and instruction set, it is slow but extremely efficient, i.e. it |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 80 | spot memory usage errors in a very precise way.</p><h3><a name="General4" id="General4">General memory requirements</a></h3><p>How much libxml2 memory require ? It's hard to tell in average it depends |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 81 | of a number of things:</p><ul><li>the parser itself should work in a fixed amount of memory, except for |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 82 | information maintained about the stacks of names and entities locations. |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 83 | The I/O and encoding handlers will probably account for a few KBytes. |
| 84 | This is true for both the XML and HTML parser (though the HTML parser |
| 85 | need more state).</li> |
Daniel Veillard | 0b28e88 | 2002-07-24 23:47:05 +0000 | [diff] [blame] | 86 | <li>If you are generating the DOM tree then memory requirements will grow |
Daniel Veillard | 63d8314 | 2002-05-20 06:51:05 +0000 | [diff] [blame] | 87 | nearly linear with the size of the data. In general for a balanced |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 88 | textual document the internal memory requirement is about 4 times the |
Daniel Veillard | 63d8314 | 2002-05-20 06:51:05 +0000 | [diff] [blame] | 89 | size of the UTF8 serialization of this document (example the XML-1.0 |
Daniel Veillard | 189446d | 2000-10-13 10:23:06 +0000 | [diff] [blame] | 90 | recommendation is a bit more of 150KBytes and takes 650KBytes of main |
| 91 | memory when parsed). Validation will add a amount of memory required for |
| 92 | maintaining the external Dtd state which should be linear with the |
Daniel Veillard | b8cfbd1 | 2001-10-25 10:53:28 +0000 | [diff] [blame] | 93 | complexity of the content model defined by the Dtd</li> |
Daniel Veillard | ce192eb | 2003-04-16 15:58:05 +0000 | [diff] [blame] | 94 | <li>If you need to work with fixed memory requirements or don't need the |
| 95 | full DOM tree then using the <a href="xmlreader.html">xmlReader |
| 96 | interface</a> is probably the best way to proceed, it still allows to |
| 97 | validate or operate on subset of the tree if needed.</li> |
Daniel Veillard | 8a46917 | 2003-06-12 16:05:07 +0000 | [diff] [blame] | 98 | <li>If you don't care about the advanced features of libxml2 like |
Daniel Veillard | ce192eb | 2003-04-16 15:58:05 +0000 | [diff] [blame] | 99 | validation, DOM, XPath or XPointer, don't use entities, need to work with |
| 100 | fixed memory requirements, and try to get the fastest parsing possible |
| 101 | then the SAX interface should be used, but it has known restrictions.</li> |
Daniel Veillard | 1177ca4 | 2003-04-26 22:29:54 +0000 | [diff] [blame] | 102 | </ul><p></p><p><a href="bugs.html">Daniel Veillard</a></p></td></tr></table></td></tr></table></td></tr></table></td></tr></table></td></tr></table></body></html> |