doc/xmlmem.html - platform/external/libxml2 - Gitiles

 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
                       "http://www.w3.org/TR/REC-html40/loose.dtd">
 <html>
 <head>
   <title>Libxml memory management</title>
   <meta name="GENERATOR" content="amaya V3.2">
   <meta http-equiv="Content-Type" content="text/html">
 </head>

 <body bgcolor="#ffffff">
 <h1 align="center">Libxml memory management</h1>

 <p>Location: <a
 href="http://xmlsoft.org/xmlmem.html">http://xmlsoft.org/xmlmem.html</a></p>

 <p>Libxml home page: <a href="http://xmlsoft.org/">http://xmlsoft.org/</a></p>

 <p>Mailing-list archive:  <a
 href="http://xmlsoft.org/messages/">http://xmlsoft.org/messages/</a></p>

 <p>Version: $Revision$</p>

 <p>Table of Content:</p>
 <ol>
   <li><a href="#General">General overview</a></li>
   <li><a href="#setting">Setting libxml set of memory routines</a></li>
   <li><a href="#cleanup">Cleaning up after parsing</a></li>
   <li><a href="#Debugging">Debugging routines</a></li>
   <li><a href="#General">General memory requirements</a></li>
 </ol>

 <h2><a name="General">General overview</a></h2>

 <p>The module <code><a
 href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlmemory.h</a></code>
 provides the interfaces to the libxml memory system:</p>
 <ul>
   <li>libxml does not use the libc memory allocator directly but xmlFree(),
     xmlMalloc() and xmlRealloc()</li>
   <li>those routines can be reallocated to a specific set of routine, by
     default the libc ones i.e. free(), malloc() and realloc()</li>
   <li>the xmlmemory.c module includes a set of debugging routine</li>
 </ul>

 <h2><a name="setting">Setting libxml set of memory routines</a></h2>

 <p>It is sometimes useful to not use the default memory allocator, either for
 debugging, analysis or to implement a specific behaviour on memory management
 (like on embedded systems). Two function calls are available to do so:</p>
 <ul>
   <li><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemGet
     ()</a> which return the current set of functions in use by the parser</li>
   <li><a
     href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemSetup()</a>
     which allow to set up a new set of memory allocation functions</li>
 </ul>

 <p>Of course a call to xmlMemSetup() should probably be done before calling
 any other libxml routines (unless you are sure your allocations routines are
 compatibles).</p>

 <h2><a name="cleanup">Cleaning up after parsing</a></h2>

 <p>Libxml is not stateless, there is a few set of memory structures needing
 allocation before the parser is fully functionnal (some encoding structures
 for example). This also mean that once parsing is finished there is a tiny
 amount of memory (a few hundred bytes) which can be recollected if you don't
 reuse the parser immediately:</p>
 <ul>
   <li><a href="http://xmlsoft.org/html/libxml-parser.html">xmlCleanupParser
     ()</a> is a centralized routine to free the parsing states. Note that it
     won't deallocate any produced tree if any (use the xmlFreeDoc() and
     related routines for this).</li>
   <li><a href="http://xmlsoft.org/html/libxml-parser.html">xmlInitParser
     ()</a> is the dual routine allowing to preallocate the parsing state which
     can be useful for example to avoid initialization reentrancy problems when
     using libxml in multithreaded applications</li>
 </ul>

 <p>Generally xmlCleanupParser() is safe, if needed the state will be rebuild
 at the next invocation of parser routines, but be careful of the consequences
 in multithreaded applications.</p>

 <h2><a name="Debugging">Debugging routines</a></h2>

 <p>When configured using --with-mem-debug flag (off by default), libxml uses a
 set of memory allocation debugging routineskeeping track of all allocated
 blocks and the location in the code where the routine was called. A couple of
 other debugging routines allow to dump the memory allocated infos to a file or
 call a specific routine when a given block number is allocated:</p>
 <ul>
   <li><a
     href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMallocLoc()</a>
     <a
     href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlReallocLoc()</a>
     and <a
     href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemStrdupLoc()</a>
     are the memory debugging replacement allocation routines</li>
   <li><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemoryDump
     ()</a> dumps all the informations about the allocated memory block lefts
     in the <code>.memdump</code> file</li>
 </ul>

 <p> When developping libxml memory debug is enabled, the tests programs call
 xmlMemoryDump () and the "make test" regression tests will check for any
 memory leak during the full regression test sequence, this helps a lot
 ensuring that libxml  does not leak memory and bullet proof memory allocations
 use (some libc implementations are known to be far too permissive resulting in
 major portability problems!). </p>

 <p>If the .memdump reports a leak, it displays the allocation function and
 also tries to give some informations about the content and structure of the
 allocated blocks left. This is sufficient in most cases to find the culprit,
 but not always. Assuming the allocation problem is reproductible, it is
 possible to find more easilly:</p>
 <ol>
   <li>write down the block number xxxx not allocated</li>
   <li>export the environement variable XML_MEM_BREAKPOINT=xxxx</li>
   <li>run the program under a debugger and set a breakpoint on
     xmlMallocBreakpoint() a specific function called when this precise block
     is allocated</li>
   <li>when the breakpoint is reached you can then do a fine analysis of the
     allocation an step  to see the condition resulting in the missing
     deallocation.</li>
 </ol>

 <p> I used to use a commercial tool to debug libxml memory problems but after
 noticing that it was not detecting memory leaks that simple mechanism was used
 and proved extremely efficient until now.</p>

 <h2><a name="General">General memory requirements</a></h2>

 <p>How much libxml memory require ? It's hard to tell in average it depends of
 a number of things:</p>
 <ul>
   <li>the parser itself should work  in a fixed amout of memory, except for
     information maintained about the stacks of names and  entities locations.
     The I/O and encoding handlers will probably account for a few KBytes. This
     is true for both the XML and HTML parser (though the HTML parser need more
     state).</li>
   <li>If you are generating the DOM tree then memory requirements will grow
     nearly lineary with the size of the data. In general for a balanced
     textual document the internal memory requirement is about 4 times the size
     of the UTF8 serialization of this document (exmple the XML-1.0
     recommendation is a bit more of 150KBytes and takes 650KBytes of main
     memory when parsed). Validation will add a amount of memory required for
     maintaining the external Dtd state which should be linear with the
     complexity of the content model defined by the Dtd </li>
   <li>If you don't care about the advanced features of libxml like validation,
     DOM, XPath or XPointer, but really need to work fixed memory requirements,
     then the SAX interface should be used.</li>
 </ul>

 <p></p>

 <p><a href="mailto:daniel@veillard.com">Daniel Veillard</a></p>

 <p>$Id$</p>
 </body>
 </html>
	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
	"http://www.w3.org/TR/REC-html40/loose.dtd">
	<html>
	<head>
	<title>Libxml memory management</title>
	<meta name="GENERATOR" content="amaya V3.2">
	<meta http-equiv="Content-Type" content="text/html">
	</head>

	<body bgcolor="#ffffff">
	<h1 align="center">Libxml memory management</h1>

	<p>Location: <a
	href="http://xmlsoft.org/xmlmem.html">http://xmlsoft.org/xmlmem.html</a></p>

	<p>Libxml home page: <a href="http://xmlsoft.org/">http://xmlsoft.org/</a></p>

	<p>Mailing-list archive: <a
	href="http://xmlsoft.org/messages/">http://xmlsoft.org/messages/</a></p>

	<p>Version: $Revision$</p>

	<p>Table of Content:</p>
	<ol>
	<li><a href="#General">General overview</a></li>
	<li><a href="#setting">Setting libxml set of memory routines</a></li>
	<li><a href="#cleanup">Cleaning up after parsing</a></li>
	<li><a href="#Debugging">Debugging routines</a></li>
	<li><a href="#General">General memory requirements</a></li>
	</ol>

	<h2><a name="General">General overview</a></h2>

	<p>The module <code><a
	href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlmemory.h</a></code>
	provides the interfaces to the libxml memory system:</p>
	<ul>
	<li>libxml does not use the libc memory allocator directly but xmlFree(),
	xmlMalloc() and xmlRealloc()</li>
	<li>those routines can be reallocated to a specific set of routine, by
	default the libc ones i.e. free(), malloc() and realloc()</li>
	<li>the xmlmemory.c module includes a set of debugging routine</li>
	</ul>

	<h2><a name="setting">Setting libxml set of memory routines</a></h2>

	<p>It is sometimes useful to not use the default memory allocator, either for
	debugging, analysis or to implement a specific behaviour on memory management
	(like on embedded systems). Two function calls are available to do so:</p>
	<ul>
	<li><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemGet
	()</a> which return the current set of functions in use by the parser</li>
	<li><a
	href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemSetup()</a>
	which allow to set up a new set of memory allocation functions</li>
	</ul>

	<p>Of course a call to xmlMemSetup() should probably be done before calling
	any other libxml routines (unless you are sure your allocations routines are
	compatibles).</p>

	<h2><a name="cleanup">Cleaning up after parsing</a></h2>

	<p>Libxml is not stateless, there is a few set of memory structures needing
	allocation before the parser is fully functionnal (some encoding structures
	for example). This also mean that once parsing is finished there is a tiny
	amount of memory (a few hundred bytes) which can be recollected if you don't
	reuse the parser immediately:</p>
	<ul>
	<li><a href="http://xmlsoft.org/html/libxml-parser.html">xmlCleanupParser
	()</a> is a centralized routine to free the parsing states. Note that it
	won't deallocate any produced tree if any (use the xmlFreeDoc() and
	related routines for this).</li>
	<li><a href="http://xmlsoft.org/html/libxml-parser.html">xmlInitParser
	()</a> is the dual routine allowing to preallocate the parsing state which
	can be useful for example to avoid initialization reentrancy problems when
	using libxml in multithreaded applications</li>
	</ul>

	<p>Generally xmlCleanupParser() is safe, if needed the state will be rebuild
	at the next invocation of parser routines, but be careful of the consequences
	in multithreaded applications.</p>

	<h2><a name="Debugging">Debugging routines</a></h2>

	<p>When configured using --with-mem-debug flag (off by default), libxml uses a
	set of memory allocation debugging routineskeeping track of all allocated
	blocks and the location in the code where the routine was called. A couple of
	other debugging routines allow to dump the memory allocated infos to a file or
	call a specific routine when a given block number is allocated:</p>
	<ul>
	<li><a
	href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMallocLoc()</a>
	<a
	href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlReallocLoc()</a>
	and <a
	href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemStrdupLoc()</a>
	are the memory debugging replacement allocation routines</li>
	<li><a href="http://xmlsoft.org/html/libxml-xmlmemory.html">xmlMemoryDump
	()</a> dumps all the informations about the allocated memory block lefts
	in the <code>.memdump</code> file</li>
	</ul>

	<p> When developping libxml memory debug is enabled, the tests programs call
	xmlMemoryDump () and the "make test" regression tests will check for any
	memory leak during the full regression test sequence, this helps a lot
	ensuring that libxml does not leak memory and bullet proof memory allocations
	use (some libc implementations are known to be far too permissive resulting in
	major portability problems!). </p>

	<p>If the .memdump reports a leak, it displays the allocation function and
	also tries to give some informations about the content and structure of the
	allocated blocks left. This is sufficient in most cases to find the culprit,
	but not always. Assuming the allocation problem is reproductible, it is
	possible to find more easilly:</p>
	<ol>
	<li>write down the block number xxxx not allocated</li>
	<li>export the environement variable XML_MEM_BREAKPOINT=xxxx</li>
	<li>run the program under a debugger and set a breakpoint on
	xmlMallocBreakpoint() a specific function called when this precise block
	is allocated</li>
	<li>when the breakpoint is reached you can then do a fine analysis of the
	allocation an step to see the condition resulting in the missing
	deallocation.</li>
	</ol>

	<p> I used to use a commercial tool to debug libxml memory problems but after
	noticing that it was not detecting memory leaks that simple mechanism was used
	and proved extremely efficient until now.</p>

	<h2><a name="General">General memory requirements</a></h2>

	<p>How much libxml memory require ? It's hard to tell in average it depends of
	a number of things:</p>
	<ul>
	<li>the parser itself should work in a fixed amout of memory, except for
	information maintained about the stacks of names and entities locations.
	The I/O and encoding handlers will probably account for a few KBytes. This
	is true for both the XML and HTML parser (though the HTML parser need more
	state).</li>
	<li>If you are generating the DOM tree then memory requirements will grow
	nearly lineary with the size of the data. In general for a balanced
	textual document the internal memory requirement is about 4 times the size
	of the UTF8 serialization of this document (exmple the XML-1.0
	recommendation is a bit more of 150KBytes and takes 650KBytes of main
	memory when parsed). Validation will add a amount of memory required for
	maintaining the external Dtd state which should be linear with the
	complexity of the content model defined by the Dtd </li>
	<li>If you don't care about the advanced features of libxml like validation,
	DOM, XPath or XPointer, but really need to work fixed memory requirements,
	then the SAX interface should be used.</li>
	</ul>

	<p></p>

	<p><a href="mailto:daniel@veillard.com">Daniel Veillard</a></p>

	<p>$Id$</p>
	</body>
	</html>