Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / valgrind / a0664b9ca67b594bd6f570a61d3301167a24750c / . / docs / html / hg-manual.html
blob: 4156c05259efe5f34c294e220d08ce25e24223a3 [file] [log] [blame]
Elliott Hughesa0664b92017-04-18 17:46:52 -0700[diff] [blame^]1<html>
2<head>
3<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
4<title>7. Helgrind: a thread error detector</title>
5<link rel="stylesheet" type="text/css" href="vg_basic.css">
6<meta name="generator" content="DocBook XSL Stylesheets V1.78.1">
7<link rel="home" href="index.html" title="Valgrind Documentation">
8<link rel="up" href="manual.html" title="Valgrind User Manual">
9<link rel="prev" href="cl-manual.html" title="6. Callgrind: a call-graph generating cache and branch prediction profiler">
10<link rel="next" href="drd-manual.html" title="8. DRD: a thread error detector">
11</head>
12<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
13<div><table class="nav" width="100%" cellspacing="3" cellpadding="3" border="0" summary="Navigation header"><tr>
14<td width="22px" align="center" valign="middle"><a accesskey="p" href="cl-manual.html"><img src="images/prev.png" width="18" height="21" border="0" alt="Prev"></a></td>
15<td width="25px" align="center" valign="middle"><a accesskey="u" href="manual.html"><img src="images/up.png" width="21" height="18" border="0" alt="Up"></a></td>
16<td width="31px" align="center" valign="middle"><a accesskey="h" href="index.html"><img src="images/home.png" width="27" height="20" border="0" alt="Up"></a></td>
17<th align="center" valign="middle">Valgrind User Manual</th>
18<td width="22px" align="center" valign="middle"><a accesskey="n" href="drd-manual.html"><img src="images/next.png" width="18" height="21" border="0" alt="Next"></a></td>
19</tr></table></div>
20<div class="chapter">
21<div class="titlepage"><div><div><h1 class="title">
22<a name="hg-manual"></a>7. Helgrind: a thread error detector</h1></div></div></div>
23<div class="toc">
24<p><b>Table of Contents</b></p>
25<dl class="toc">
26<dt><span class="sect1"><a href="hg-manual.html#hg-manual.overview">7.1. Overview</a></span></dt>
27<dt><span class="sect1"><a href="hg-manual.html#hg-manual.api-checks">7.2. Detected errors: Misuses of the POSIX pthreads API</a></span></dt>
28<dt><span class="sect1"><a href="hg-manual.html#hg-manual.lock-orders">7.3. Detected errors: Inconsistent Lock Orderings</a></span></dt>
29<dt><span class="sect1"><a href="hg-manual.html#hg-manual.data-races">7.4. Detected errors: Data Races</a></span></dt>
30<dd><dl>
31<dt><span class="sect2"><a href="hg-manual.html#hg-manual.data-races.example">7.4.1. A Simple Data Race</a></span></dt>
32<dt><span class="sect2"><a href="hg-manual.html#hg-manual.data-races.algorithm">7.4.2. Helgrind's Race Detection Algorithm</a></span></dt>
33<dt><span class="sect2"><a href="hg-manual.html#hg-manual.data-races.errmsgs">7.4.3. Interpreting Race Error Messages</a></span></dt>
34</dl></dd>
35<dt><span class="sect1"><a href="hg-manual.html#hg-manual.effective-use">7.5. Hints and Tips for Effective Use of Helgrind</a></span></dt>
36<dt><span class="sect1"><a href="hg-manual.html#hg-manual.options">7.6. Helgrind Command-line Options</a></span></dt>
37<dt><span class="sect1"><a href="hg-manual.html#hg-manual.monitor-commands">7.7. Helgrind Monitor Commands</a></span></dt>
38<dt><span class="sect1"><a href="hg-manual.html#hg-manual.client-requests">7.8. Helgrind Client Requests</a></span></dt>
39<dt><span class="sect1"><a href="hg-manual.html#hg-manual.todolist">7.9. A To-Do List for Helgrind</a></span></dt>
40</dl>
41</div>
42<p>To use this tool, you must specify
43<code class="option">--tool=helgrind</code> on the Valgrind
44command line.</p>
45<div class="sect1">
46<div class="titlepage"><div><div><h2 class="title" style="clear: both">
47<a name="hg-manual.overview"></a>7.1. Overview</h2></div></div></div>
48<p>Helgrind is a Valgrind tool for detecting synchronisation errors
49in C, C++ and Fortran programs that use the POSIX pthreads
50threading primitives.</p>
51<p>The main abstractions in POSIX pthreads are: a set of threads
52sharing a common address space, thread creation, thread joining,
53thread exit, mutexes (locks), condition variables (inter-thread event
54notifications), reader-writer locks, spinlocks, semaphores and
55barriers.</p>
56<p>Helgrind can detect three classes of errors, which are discussed
57in detail in the next three sections:</p>
58<div class="orderedlist"><ol class="orderedlist" type="1">
59<li class="listitem"><p><a class="link" href="hg-manual.html#hg-manual.api-checks" title="7.2. Detected errors: Misuses of the POSIX pthreads API">
60 Misuses of the POSIX pthreads API.</a></p></li>
61<li class="listitem"><p><a class="link" href="hg-manual.html#hg-manual.lock-orders" title="7.3. Detected errors: Inconsistent Lock Orderings">
62 Potential deadlocks arising from lock
63 ordering problems.</a></p></li>
64<li class="listitem"><p><a class="link" href="hg-manual.html#hg-manual.data-races" title="7.4. Detected errors: Data Races">
65 Data races -- accessing memory without adequate locking
66 or synchronisation</a>.
67 </p></li>
68</ol></div>
69<p>Problems like these often result in unreproducible,
70timing-dependent crashes, deadlocks and other misbehaviour, and
71can be difficult to find by other means.</p>
72<p>Helgrind is aware of all the pthread abstractions and tracks
73their effects as accurately as it can. On x86 and amd64 platforms, it
74understands and partially handles implicit locking arising from the
75use of the LOCK instruction prefix. On PowerPC/POWER and ARM
76platforms, it partially handles implicit locking arising from
77load-linked and store-conditional instruction pairs.
78</p>
79<p>Helgrind works best when your application uses only the POSIX
80pthreads API. However, if you want to use custom threading
81primitives, you can describe their behaviour to Helgrind using the
82<code class="varname">ANNOTATE_*</code> macros defined
83in <code class="varname">helgrind.h</code>.</p>
84<p>Following those is a section containing
85<a class="link" href="hg-manual.html#hg-manual.effective-use" title="7.5. Hints and Tips for Effective Use of Helgrind">
86hints and tips on how to get the best out of Helgrind.</a>
87</p>
88<p>Then there is a
89<a class="link" href="hg-manual.html#hg-manual.options" title="7.6. Helgrind Command-line Options">summary of command-line
90options.</a>
91</p>
92<p>Finally, there is
93<a class="link" href="hg-manual.html#hg-manual.todolist" title="7.9. A To-Do List for Helgrind">a brief summary of areas in which Helgrind
94could be improved.</a>
95</p>
96</div>
97<div class="sect1">
98<div class="titlepage"><div><div><h2 class="title" style="clear: both">
99<a name="hg-manual.api-checks"></a>7.2. Detected errors: Misuses of the POSIX pthreads API</h2></div></div></div>
100<p>Helgrind intercepts calls to many POSIX pthreads functions, and
101is therefore able to report on various common problems. Although
102these are unglamourous errors, their presence can lead to undefined
103program behaviour and hard-to-find bugs later on. The detected errors
104are:</p>
105<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
106<li class="listitem"><p>unlocking an invalid mutex</p></li>
107<li class="listitem"><p>unlocking a not-locked mutex</p></li>
108<li class="listitem"><p>unlocking a mutex held by a different
109 thread</p></li>
110<li class="listitem"><p>destroying an invalid or a locked mutex</p></li>
111<li class="listitem"><p>recursively locking a non-recursive mutex</p></li>
112<li class="listitem"><p>deallocation of memory that contains a
113 locked mutex</p></li>
114<li class="listitem"><p>passing mutex arguments to functions expecting
115 reader-writer lock arguments, and vice
116 versa</p></li>
117<li class="listitem"><p>when a POSIX pthread function fails with an
118 error code that must be handled</p></li>
119<li class="listitem"><p>when a thread exits whilst still holding locked
120 locks</p></li>
121<li class="listitem"><p>calling <code class="function">pthread_cond_wait</code>
122 with a not-locked mutex, an invalid mutex,
123 or one locked by a different
124 thread</p></li>
125<li class="listitem"><p>inconsistent bindings between condition
126 variables and their associated mutexes</p></li>
127<li class="listitem"><p>invalid or duplicate initialisation of a pthread
128 barrier</p></li>
129<li class="listitem"><p>initialisation of a pthread barrier on which threads
130 are still waiting</p></li>
131<li class="listitem"><p>destruction of a pthread barrier object which was
132 never initialised, or on which threads are still
133 waiting</p></li>
134<li class="listitem"><p>waiting on an uninitialised pthread
135 barrier</p></li>
136<li class="listitem"><p>for all of the pthreads functions that Helgrind
137 intercepts, an error is reported, along with a stack
138 trace, if the system threading library routine returns
139 an error code, even if Helgrind itself detected no
140 error</p></li>
141</ul></div>
142<p>Checks pertaining to the validity of mutexes are generally also
143performed for reader-writer locks.</p>
144<p>Various kinds of this-can't-possibly-happen events are also
145reported. These usually indicate bugs in the system threading
146library.</p>
147<p>Reported errors always contain a primary stack trace indicating
148where the error was detected. They may also contain auxiliary stack
149traces giving additional information. In particular, most errors
150relating to mutexes will also tell you where that mutex first came to
151Helgrind's attention (the "<code class="computeroutput">was first observed
152at</code>" part), so you have a chance of figuring out which
153mutex it is referring to. For example:</p>
154<pre class="programlisting">
155Thread #1 unlocked a not-locked lock at 0x7FEFFFA90
156 at 0x4C2408D: pthread_mutex_unlock (hg_intercepts.c:492)
157 by 0x40073A: nearly_main (tc09_bad_unlock.c:27)
158 by 0x40079B: main (tc09_bad_unlock.c:50)
159 Lock at 0x7FEFFFA90 was first observed
160 at 0x4C25D01: pthread_mutex_init (hg_intercepts.c:326)
161 by 0x40071F: nearly_main (tc09_bad_unlock.c:23)
162 by 0x40079B: main (tc09_bad_unlock.c:50)
163</pre>
164<p>Helgrind has a way of summarising thread identities, as
165you see here with the text "<code class="computeroutput">Thread
166#1</code>". This is so that it can speak about threads and
167sets of threads without overwhelming you with details. See
168<a class="link" href="hg-manual.html#hg-manual.data-races.errmsgs" title="7.4.3. Interpreting Race Error Messages">below</a>
169for more information on interpreting error messages.</p>
170</div>
171<div class="sect1">
172<div class="titlepage"><div><div><h2 class="title" style="clear: both">
173<a name="hg-manual.lock-orders"></a>7.3. Detected errors: Inconsistent Lock Orderings</h2></div></div></div>
174<p>In this section, and in general, to "acquire" a lock simply
175means to lock that lock, and to "release" a lock means to unlock
176it.</p>
177<p>Helgrind monitors the order in which threads acquire locks.
178This allows it to detect potential deadlocks which could arise from
179the formation of cycles of locks. Detecting such inconsistencies is
180useful because, whilst actual deadlocks are fairly obvious, potential
181deadlocks may never be discovered during testing and could later lead
182to hard-to-diagnose in-service failures.</p>
183<p>The simplest example of such a problem is as
184follows.</p>
185<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
186<li class="listitem"><p>Imagine some shared resource R, which, for whatever
187 reason, is guarded by two locks, L1 and L2, which must both be held
188 when R is accessed.</p></li>
189<li class="listitem"><p>Suppose a thread acquires L1, then L2, and proceeds
190 to access R. The implication of this is that all threads in the
191 program must acquire the two locks in the order first L1 then L2.
192 Not doing so risks deadlock.</p></li>
193<li class="listitem"><p>The deadlock could happen if two threads -- call them
194 T1 and T2 -- both want to access R. Suppose T1 acquires L1 first,
195 and T2 acquires L2 first. Then T1 tries to acquire L2, and T2 tries
196 to acquire L1, but those locks are both already held. So T1 and T2
197 become deadlocked.</p></li>
198</ul></div>
199<p>Helgrind builds a directed graph indicating the order in which
200locks have been acquired in the past. When a thread acquires a new
201lock, the graph is updated, and then checked to see if it now contains
202a cycle. The presence of a cycle indicates a potential deadlock involving
203the locks in the cycle.</p>
204<p>In general, Helgrind will choose two locks involved in the cycle
205and show you how their acquisition ordering has become inconsistent.
206It does this by showing the program points that first defined the
207ordering, and the program points which later violated it. Here is a
208simple example involving just two locks:</p>
209<pre class="programlisting">
210Thread #1: lock order "0x7FF0006D0 before 0x7FF0006A0" violated
211
212Observed (incorrect) order is: acquisition of lock at 0x7FF0006A0
213 at 0x4C2BC62: pthread_mutex_lock (hg_intercepts.c:494)
214 by 0x400825: main (tc13_laog1.c:23)
215
216 followed by a later acquisition of lock at 0x7FF0006D0
217 at 0x4C2BC62: pthread_mutex_lock (hg_intercepts.c:494)
218 by 0x400853: main (tc13_laog1.c:24)
219
220Required order was established by acquisition of lock at 0x7FF0006D0
221 at 0x4C2BC62: pthread_mutex_lock (hg_intercepts.c:494)
222 by 0x40076D: main (tc13_laog1.c:17)
223
224 followed by a later acquisition of lock at 0x7FF0006A0
225 at 0x4C2BC62: pthread_mutex_lock (hg_intercepts.c:494)
226 by 0x40079B: main (tc13_laog1.c:18)
227</pre>
228<p>When there are more than two locks in the cycle, the error is
229equally serious. However, at present Helgrind does not show the locks
230involved, sometimes because that information is not available, but
231also so as to avoid flooding you with information. For example, a
232naive implementation of the famous Dining Philosophers problem
233involves a cycle of five locks
234(see <code class="computeroutput">helgrind/tests/tc14_laog_dinphils.c</code>).
235In this case Helgrind has detected that all 5 philosophers could
236simultaneously pick up their left fork and then deadlock whilst
237waiting to pick up their right forks.</p>
238<pre class="programlisting">
239Thread #6: lock order "0x80499A0 before 0x8049A00" violated
240
241Observed (incorrect) order is: acquisition of lock at 0x8049A00
242 at 0x40085BC: pthread_mutex_lock (hg_intercepts.c:495)
243 by 0x80485B4: dine (tc14_laog_dinphils.c:18)
244 by 0x400BDA4: mythread_wrapper (hg_intercepts.c:219)
245 by 0x39B924: start_thread (pthread_create.c:297)
246 by 0x2F107D: clone (clone.S:130)
247
248 followed by a later acquisition of lock at 0x80499A0
249 at 0x40085BC: pthread_mutex_lock (hg_intercepts.c:495)
250 by 0x80485CD: dine (tc14_laog_dinphils.c:19)
251 by 0x400BDA4: mythread_wrapper (hg_intercepts.c:219)
252 by 0x39B924: start_thread (pthread_create.c:297)
253 by 0x2F107D: clone (clone.S:130)
254</pre>
255</div>
256<div class="sect1">
257<div class="titlepage"><div><div><h2 class="title" style="clear: both">
258<a name="hg-manual.data-races"></a>7.4. Detected errors: Data Races</h2></div></div></div>
259<p>A data race happens, or could happen, when two threads access a
260shared memory location without using suitable locks or other
261synchronisation to ensure single-threaded access. Such missing
262locking can cause obscure timing dependent bugs. Ensuring programs
263are race-free is one of the central difficulties of threaded
264programming.</p>
265<p>Reliably detecting races is a difficult problem, and most
266of Helgrind's internals are devoted to dealing with it.
267We begin with a simple example.</p>
268<div class="sect2">
269<div class="titlepage"><div><div><h3 class="title">
270<a name="hg-manual.data-races.example"></a>7.4.1. A Simple Data Race</h3></div></div></div>
271<p>About the simplest possible example of a race is as follows. In
272this program, it is impossible to know what the value
273of <code class="computeroutput">var</code> is at the end of the program.
274Is it 2 ? Or 1 ?</p>
275<pre class="programlisting">
276#include &lt;pthread.h&gt;
277
278int var = 0;
279
280void* child_fn ( void* arg ) {
281 var++; /* Unprotected relative to parent */ /* this is line 6 */
282 return NULL;
283}
284
285int main ( void ) {
286 pthread_t child;
287 pthread_create(&amp;child, NULL, child_fn, NULL);
288 var++; /* Unprotected relative to child */ /* this is line 13 */
289 pthread_join(child, NULL);
290 return 0;
291}
292</pre>
293<p>The problem is there is nothing to
294stop <code class="varname">var</code> being updated simultaneously
295by both threads. A correct program would
296protect <code class="varname">var</code> with a lock of type
297<code class="function">pthread_mutex_t</code>, which is acquired
298before each access and released afterwards. Helgrind's output for
299this program is:</p>
300<pre class="programlisting">
301Thread #1 is the program's root thread
302
303Thread #2 was created
304 at 0x511C08E: clone (in /lib64/libc-2.8.so)
305 by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
306 by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
307 by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
308 by 0x400605: main (simple_race.c:12)
309
310Possible data race during read of size 4 at 0x601038 by thread #1
311Locks held: none
312 at 0x400606: main (simple_race.c:13)
313
314This conflicts with a previous write of size 4 by thread #2
315Locks held: none
316 at 0x4005DC: child_fn (simple_race.c:6)
317 by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
318 by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
319 by 0x511C0CC: clone (in /lib64/libc-2.8.so)
320
321Location 0x601038 is 0 bytes inside global var "var"
322declared at simple_race.c:3
323</pre>
324<p>This is quite a lot of detail for an apparently simple error.
325The last clause is the main error message. It says there is a race as
326a result of a read of size 4 (bytes), at 0x601038, which is the
327address of <code class="computeroutput">var</code>, happening in
328function <code class="computeroutput">main</code> at line 13 in the
329program.</p>
330<p>Two important parts of the message are:</p>
331<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
332<li class="listitem">
333<p>Helgrind shows two stack traces for the error, not one. By
334 definition, a race involves two different threads accessing the
335 same location in such a way that the result depends on the relative
336 speeds of the two threads.</p>
337<p>
338 The first stack trace follows the text "<code class="computeroutput">Possible
339 data race during read of size 4 ...</code>" and the
340 second trace follows the text "<code class="computeroutput">This conflicts with
341 a previous write of size 4 ...</code>". Helgrind is
342 usually able to show both accesses involved in a race. At least
343 one of these will be a write (since two concurrent, unsynchronised
344 reads are harmless), and they will of course be from different
345 threads.</p>
346<p>By examining your program at the two locations, you should be
347 able to get at least some idea of what the root cause of the
348 problem is. For each location, Helgrind shows the set of locks
349 held at the time of the access. This often makes it clear which
350 thread, if any, failed to take a required lock. In this example
351 neither thread holds a lock during the access.</p>
352</li>
353<li class="listitem">
354<p>For races which occur on global or stack variables, Helgrind
355 tries to identify the name and defining point of the variable.
356 Hence the text "<code class="computeroutput">Location 0x601038 is 0 bytes inside
357 global var "var" declared at simple_race.c:3</code>".</p>
358<p>Showing names of stack and global variables carries no
359 run-time overhead once Helgrind has your program up and running.
360 However, it does require Helgrind to spend considerable extra time
361 and memory at program startup to read the relevant debug info.
362 Hence this facility is disabled by default. To enable it, you need
363 to give the <code class="varname">--read-var-info=yes</code> option to
364 Helgrind.</p>
365</li>
366</ul></div>
367<p>The following section explains Helgrind's race detection
368algorithm in more detail.</p>
369</div>
370<div class="sect2">
371<div class="titlepage"><div><div><h3 class="title">
372<a name="hg-manual.data-races.algorithm"></a>7.4.2. Helgrind's Race Detection Algorithm</h3></div></div></div>
373<p>Most programmers think about threaded programming in terms of
374the basic functionality provided by the threading library (POSIX
375Pthreads): thread creation, thread joining, locks, condition
376variables, semaphores and barriers.</p>
377<p>The effect of using these functions is to impose
378constraints upon the order in which memory accesses can
379happen. This implied ordering is generally known as the
380"happens-before relation". Once you understand the happens-before
381relation, it is easy to see how Helgrind finds races in your code.
382Fortunately, the happens-before relation is itself easy to understand,
383and is by itself a useful tool for reasoning about the behaviour of
384parallel programs. We now introduce it using a simple example.</p>
385<p>Consider first the following buggy program:</p>
386<pre class="programlisting">
387Parent thread: Child thread:
388
389int var;
390
391// create child thread
392pthread_create(...)
393var = 20; var = 10;
394 exit
395
396// wait for child
397pthread_join(...)
398printf("%d\n", var);
399</pre>
400<p>The parent thread creates a child. Both then write different
401values to some variable <code class="computeroutput">var</code>, and the
402parent then waits for the child to exit.</p>
403<p>What is the value of <code class="computeroutput">var</code> at the
404end of the program, 10 or 20? We don't know. The program is
405considered buggy (it has a race) because the final value
406of <code class="computeroutput">var</code> depends on the relative rates
407of progress of the parent and child threads. If the parent is fast
408and the child is slow, then the child's assignment may happen later,
409so the final value will be 10; and vice versa if the child is faster
410than the parent.</p>
411<p>The relative rates of progress of parent vs child is not something
412the programmer can control, and will often change from run to run.
413It depends on factors such as the load on the machine, what else is
414running, the kernel's scheduling strategy, and many other factors.</p>
415<p>The obvious fix is to use a lock to
416protect <code class="computeroutput">var</code>. It is however
417instructive to consider a somewhat more abstract solution, which is to
418send a message from one thread to the other:</p>
419<pre class="programlisting">
420Parent thread: Child thread:
421
422int var;
423
424// create child thread
425pthread_create(...)
426var = 20;
427// send message to child
428 // wait for message to arrive
429 var = 10;
430 exit
431
432// wait for child
433pthread_join(...)
434printf("%d\n", var);
435</pre>
436<p>Now the program reliably prints "10", regardless of the speed of
437the threads. Why? Because the child's assignment cannot happen until
438after it receives the message. And the message is not sent until
439after the parent's assignment is done.</p>
440<p>The message transmission creates a "happens-before" dependency
441between the two assignments: <code class="computeroutput">var = 20;</code>
442must now happen-before <code class="computeroutput">var = 10;</code>.
443And so there is no longer a race
444on <code class="computeroutput">var</code>.
445</p>
446<p>Note that it's not significant that the parent sends a message
447to the child. Sending a message from the child (after its assignment)
448to the parent (before its assignment) would also fix the problem, causing
449the program to reliably print "20".</p>
450<p>Helgrind's algorithm is (conceptually) very simple. It monitors all
451accesses to memory locations. If a location -- in this example,
452<code class="computeroutput">var</code>,
453is accessed by two different threads, Helgrind checks to see if the
454two accesses are ordered by the happens-before relation. If so,
455that's fine; if not, it reports a race.</p>
456<p>It is important to understand that the happens-before relation
457creates only a partial ordering, not a total ordering. An example of
458a total ordering is comparison of numbers: for any two numbers
459<code class="computeroutput">x</code> and
460<code class="computeroutput">y</code>, either
461<code class="computeroutput">x</code> is less than, equal to, or greater
462than
463<code class="computeroutput">y</code>. A partial ordering is like a
464total ordering, but it can also express the concept that two elements
465are neither equal, less or greater, but merely unordered with respect
466to each other.</p>
467<p>In the fixed example above, we say that
468<code class="computeroutput">var = 20;</code> "happens-before"
469<code class="computeroutput">var = 10;</code>. But in the original
470version, they are unordered: we cannot say that either happens-before
471the other.</p>
472<p>What does it mean to say that two accesses from different
473threads are ordered by the happens-before relation? It means that
474there is some chain of inter-thread synchronisation operations which
475cause those accesses to happen in a particular order, irrespective of
476the actual rates of progress of the individual threads. This is a
477required property for a reliable threaded program, which is why
478Helgrind checks for it.</p>
479<p>The happens-before relations created by standard threading
480primitives are as follows:</p>
481<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
482<li class="listitem"><p>When a mutex is unlocked by thread T1 and later (or
483 immediately) locked by thread T2, then the memory accesses in T1
484 prior to the unlock must happen-before those in T2 after it acquires
485 the lock.</p></li>
486<li class="listitem"><p>The same idea applies to reader-writer locks,
487 although with some complication so as to allow correct handling of
488 reads vs writes.</p></li>
489<li class="listitem"><p>When a condition variable (CV) is signalled on by
490 thread T1 and some other thread T2 is thereby released from a wait
491 on the same CV, then the memory accesses in T1 prior to the
492 signalling must happen-before those in T2 after it returns from the
493 wait. If no thread was waiting on the CV then there is no
494 effect.</p></li>
495<li class="listitem"><p>If instead T1 broadcasts on a CV, then all of the
496 waiting threads, rather than just one of them, acquire a
497 happens-before dependency on the broadcasting thread at the point it
498 did the broadcast.</p></li>
499<li class="listitem"><p>A thread T2 that continues after completing sem_wait
500 on a semaphore that thread T1 posts on, acquires a happens-before
501 dependence on the posting thread, a bit like dependencies caused
502 mutex unlock-lock pairs. However, since a semaphore can be posted
503 on many times, it is unspecified from which of the post calls the
504 wait call gets its happens-before dependency.</p></li>
505<li class="listitem"><p>For a group of threads T1 .. Tn which arrive at a
506 barrier and then move on, each thread after the call has a
507 happens-after dependency from all threads before the
508 barrier.</p></li>
509<li class="listitem"><p>A newly-created child thread acquires an initial
510 happens-after dependency on the point where its parent created it.
511 That is, all memory accesses performed by the parent prior to
512 creating the child are regarded as happening-before all the accesses
513 of the child.</p></li>
514<li class="listitem"><p>Similarly, when an exiting thread is reaped via a
515 call to <code class="function">pthread_join</code>, once the call returns, the
516 reaping thread acquires a happens-after dependency relative to all memory
517 accesses made by the exiting thread.</p></li>
518</ul></div>
519<p>In summary: Helgrind intercepts the above listed events, and builds a
520directed acyclic graph represented the collective happens-before
521dependencies. It also monitors all memory accesses.</p>
522<p>If a location is accessed by two different threads, but Helgrind
523cannot find any path through the happens-before graph from one access
524to the other, then it reports a race.</p>
525<p>There are a couple of caveats:</p>
526<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
527<li class="listitem"><p>Helgrind doesn't check for a race in the case where
528 both accesses are reads. That would be silly, since concurrent
529 reads are harmless.</p></li>
530<li class="listitem"><p>Two accesses are considered to be ordered by the
531 happens-before dependency even through arbitrarily long chains of
532 synchronisation events. For example, if T1 accesses some location
533 L, and then <code class="function">pthread_cond_signals</code> T2, which later
534 <code class="function">pthread_cond_signals</code> T3, which then accesses L, then
535 a suitable happens-before dependency exists between the first and second
536 accesses, even though it involves two different inter-thread
537 synchronisation events.</p></li>
538</ul></div>
539</div>
540<div class="sect2">
541<div class="titlepage"><div><div><h3 class="title">
542<a name="hg-manual.data-races.errmsgs"></a>7.4.3. Interpreting Race Error Messages</h3></div></div></div>
543<p>Helgrind's race detection algorithm collects a lot of
544information, and tries to present it in a helpful way when a race is
545detected. Here's an example:</p>
546<pre class="programlisting">
547Thread #2 was created
548 at 0x511C08E: clone (in /lib64/libc-2.8.so)
549 by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
550 by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
551 by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
552 by 0x4008F2: main (tc21_pthonce.c:86)
553
554Thread #3 was created
555 at 0x511C08E: clone (in /lib64/libc-2.8.so)
556 by 0x4E333A4: do_clone (in /lib64/libpthread-2.8.so)
557 by 0x4E33A30: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.8.so)
558 by 0x4C299D4: pthread_create@* (hg_intercepts.c:214)
559 by 0x4008F2: main (tc21_pthonce.c:86)
560
561Possible data race during read of size 4 at 0x601070 by thread #3
562Locks held: none
563 at 0x40087A: child (tc21_pthonce.c:74)
564 by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
565 by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
566 by 0x511C0CC: clone (in /lib64/libc-2.8.so)
567
568This conflicts with a previous write of size 4 by thread #2
569Locks held: none
570 at 0x400883: child (tc21_pthonce.c:74)
571 by 0x4C29AFF: mythread_wrapper (hg_intercepts.c:194)
572 by 0x4E3403F: start_thread (in /lib64/libpthread-2.8.so)
573 by 0x511C0CC: clone (in /lib64/libc-2.8.so)
574
575Location 0x601070 is 0 bytes inside local var "unprotected2"
576declared at tc21_pthonce.c:51, in frame #0 of thread 3
577</pre>
578<p>Helgrind first announces the creation points of any threads
579referenced in the error message. This is so it can speak concisely
580about threads without repeatedly printing their creation point call
581stacks. Each thread is only ever announced once, the first time it
582appears in any Helgrind error message.</p>
583<p>The main error message begins at the text
584"<code class="computeroutput">Possible data race during read</code>". At
585the start is information you would expect to see -- address and size
586of the racing access, whether a read or a write, and the call stack at
587the point it was detected.</p>
588<p>A second call stack is presented starting at the text
589"<code class="computeroutput">This conflicts with a previous
590write</code>". This shows a previous access which also
591accessed the stated address, and which is believed to be racing
592against the access in the first call stack. Note that this second
593call stack is limited to a maximum of 8 entries to limit the
594memory usage.</p>
595<p>Finally, Helgrind may attempt to give a description of the
596raced-on address in source level terms. In this example, it
597identifies it as a local variable, shows its name, declaration point,
598and in which frame (of the first call stack) it lives. Note that this
599information is only shown when <code class="varname">--read-var-info=yes</code>
600is specified on the command line. That's because reading the DWARF3
601debug information in enough detail to capture variable type and
602location information makes Helgrind much slower at startup, and also
603requires considerable amounts of memory, for large programs.
604</p>
605<p>Once you have your two call stacks, how do you find the root
606cause of the race?</p>
607<p>The first thing to do is examine the source locations referred
608to by each call stack. They should both show an access to the same
609location, or variable.</p>
610<p>Now figure out how how that location should have been made
611thread-safe:</p>
612<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
613<li class="listitem"><p>Perhaps the location was intended to be protected by
614 a mutex? If so, you need to lock and unlock the mutex at both
615 access points, even if one of the accesses is reported to be a read.
616 Did you perhaps forget the locking at one or other of the accesses?
617 To help you do this, Helgrind shows the set of locks held by each
618 threads at the time they accessed the raced-on location.</p></li>
619<li class="listitem">
620<p>Alternatively, perhaps you intended to use a some
621 other scheme to make it safe, such as signalling on a condition
622 variable. In all such cases, try to find a synchronisation event
623 (or a chain thereof) which separates the earlier-observed access (as
624 shown in the second call stack) from the later-observed access (as
625 shown in the first call stack). In other words, try to find
626 evidence that the earlier access "happens-before" the later access.
627 See the previous subsection for an explanation of the happens-before
628 relation.</p>
629<p>
630 The fact that Helgrind is reporting a race means it did not observe
631 any happens-before relation between the two accesses. If
632 Helgrind is working correctly, it should also be the case that you
633 also cannot find any such relation, even on detailed inspection
634 of the source code. Hopefully, though, your inspection of the code
635 will show where the missing synchronisation operation(s) should have
636 been.</p>
637</li>
638</ul></div>
639</div>
640</div>
641<div class="sect1">
642<div class="titlepage"><div><div><h2 class="title" style="clear: both">
643<a name="hg-manual.effective-use"></a>7.5. Hints and Tips for Effective Use of Helgrind</h2></div></div></div>
644<p>Helgrind can be very helpful in finding and resolving
645threading-related problems. Like all sophisticated tools, it is most
646effective when you understand how to play to its strengths.</p>
647<p>Helgrind will be less effective when you merely throw an
648existing threaded program at it and try to make sense of any reported
649errors. It will be more effective if you design threaded programs
650from the start in a way that helps Helgrind verify correctness. The
651same is true for finding memory errors with Memcheck, but applies more
652here, because thread checking is a harder problem. Consequently it is
653much easier to write a correct program for which Helgrind falsely
654reports (threading) errors than it is to write a correct program for
655which Memcheck falsely reports (memory) errors.</p>
656<p>With that in mind, here are some tips, listed most important first,
657for getting reliable results and avoiding false errors. The first two
658are critical. Any violations of them will swamp you with huge numbers
659of false data-race errors.</p>
660<div class="orderedlist"><ol class="orderedlist" type="1">
661<li class="listitem">
662<p>Make sure your application, and all the libraries it uses,
663 use the POSIX threading primitives. Helgrind needs to be able to
664 see all events pertaining to thread creation, exit, locking and
665 other synchronisation events. To do so it intercepts many POSIX
666 pthreads functions.</p>
667<p>Do not roll your own threading primitives (mutexes, etc)
668 from combinations of the Linux futex syscall, atomic counters, etc.
669 These throw Helgrind's internal what's-going-on models
670 way off course and will give bogus results.</p>
671<p>Also, do not reimplement existing POSIX abstractions using
672 other POSIX abstractions. For example, don't build your own
673 semaphore routines or reader-writer locks from POSIX mutexes and
674 condition variables. Instead use POSIX reader-writer locks and
675 semaphores directly, since Helgrind supports them directly.</p>
676<p>Helgrind directly supports the following POSIX threading
677 abstractions: mutexes, reader-writer locks, condition variables
678 (but see below), semaphores and barriers. Currently spinlocks
679 are not supported, although they could be in future.</p>
680<p>At the time of writing, the following popular Linux packages
681 are known to implement their own threading primitives:</p>
682<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
683<li class="listitem"><p>Qt version 4.X. Qt 3.X is harmless in that it
684 only uses POSIX pthreads primitives. Unfortunately Qt 4.X
685 has its own implementation of mutexes (QMutex) and thread reaping.
686 Helgrind 3.4.x contains direct support
687 for Qt 4.X threading, which is experimental but is believed to
688 work fairly well. A side effect of supporting Qt 4 directly is
689 that Helgrind can be used to debug KDE4 applications. As this
690 is an experimental feature, we would particularly appreciate
691 feedback from folks who have used Helgrind to successfully debug
692 Qt 4 and/or KDE4 applications.</p></li>
693<li class="listitem">
694<p>Runtime support library for GNU OpenMP (part of
695 GCC), at least for GCC versions 4.2 and 4.3. The GNU OpenMP runtime
696 library (<code class="filename">libgomp.so</code>) constructs its own
697 synchronisation primitives using combinations of atomic memory
698 instructions and the futex syscall, which causes total chaos since in
699 Helgrind since it cannot "see" those.</p>
700<p>Fortunately, this can be solved using a configuration-time
701 option (for GCC). Rebuild GCC from source, and configure using
702 <code class="varname">--disable-linux-futex</code>.
703 This makes libgomp.so use the standard
704 POSIX threading primitives instead. Note that this was tested
705 using GCC 4.2.3 and has not been re-tested using more recent GCC
706 versions. We would appreciate hearing about any successes or
707 failures with more recent versions.</p>
708</li>
709</ul></div>
710<p>If you must implement your own threading primitives, there
711 are a set of client request macros
712 in <code class="computeroutput">helgrind.h</code> to help you
713 describe your primitives to Helgrind. You should be able to
714 mark up mutexes, condition variables, etc, without difficulty.
715 </p>
716<p>
717 It is also possible to mark up the effects of thread-safe
718 reference counting using the
719 <code class="computeroutput">ANNOTATE_HAPPENS_BEFORE</code>,
720 <code class="computeroutput">ANNOTATE_HAPPENS_AFTER</code> and
721 <code class="computeroutput">ANNOTATE_HAPPENS_BEFORE_FORGET_ALL</code>,
722 macros. Thread-safe reference counting using an atomically
723 incremented/decremented refcount variable causes Helgrind
724 problems because a one-to-zero transition of the reference count
725 means the accessing thread has exclusive ownership of the
726 associated resource (normally, a C++ object) and can therefore
727 access it (normally, to run its destructor) without locking.
728 Helgrind doesn't understand this, and markup is essential to
729 avoid false positives.
730 </p>
731<p>
732 Here are recommended guidelines for marking up thread safe
733 reference counting in C++. You only need to mark up your
734 release methods -- the ones which decrement the reference count.
735 Given a class like this:
736 </p>
737<pre class="programlisting">
738class MyClass {
739 unsigned int mRefCount;
740
741 void Release ( void ) {
742 unsigned int newCount = atomic_decrement(&amp;mRefCount);
743 if (newCount == 0) {
744 delete this;
745 }
746 }
747}
748</pre>
749<p>
750 the release method should be marked up as follows:
751 </p>
752<pre class="programlisting">
753 void Release ( void ) {
754 unsigned int newCount = atomic_decrement(&amp;mRefCount);
755 if (newCount == 0) {
756 ANNOTATE_HAPPENS_AFTER(&amp;mRefCount);
757 ANNOTATE_HAPPENS_BEFORE_FORGET_ALL(&amp;mRefCount);
758 delete this;
759 } else {
760 ANNOTATE_HAPPENS_BEFORE(&amp;mRefCount);
761 }
762 }
763</pre>
764<p>
765 There are a number of complex, mostly-theoretical objections to
766 this scheme. From a theoretical standpoint it appears to be
767 impossible to devise a markup scheme which is completely correct
768 in the sense of guaranteeing to remove all false races. The
769 proposed scheme however works well in practice.
770 </p>
771</li>
772<li class="listitem">
773<p>Avoid memory recycling. If you can't avoid it, you must use
774 tell Helgrind what is going on via the
775 <code class="function">VALGRIND_HG_CLEAN_MEMORY</code> client request (in
776 <code class="computeroutput">helgrind.h</code>).</p>
777<p>Helgrind is aware of standard heap memory allocation and
778 deallocation that occurs via
779 <code class="function">malloc</code>/<code class="function">free</code>/<code class="function">new</code>/<code class="function">delete</code>
780 and from entry and exit of stack frames. In particular, when memory is
781 deallocated via <code class="function">free</code>, <code class="function">delete</code>,
782 or function exit, Helgrind considers that memory clean, so when it is
783 eventually reallocated, its history is irrelevant.</p>
784<p>However, it is common practice to implement memory recycling
785 schemes. In these, memory to be freed is not handed to
786 <code class="function">free</code>/<code class="function">delete</code>, but instead put
787 into a pool of free buffers to be handed out again as required. The
788 problem is that Helgrind has no
789 way to know that such memory is logically no longer in use, and
790 its history is irrelevant. Hence you must make that explicit,
791 using the <code class="function">VALGRIND_HG_CLEAN_MEMORY</code> client request
792 to specify the relevant address ranges. It's easiest to put these
793 requests into the pool manager code, and use them either when memory is
794 returned to the pool, or is allocated from it.</p>
795</li>
796<li class="listitem">
797<p>Avoid POSIX condition variables. If you can, use POSIX
798 semaphores (<code class="function">sem_t</code>, <code class="function">sem_post</code>,
799 <code class="function">sem_wait</code>) to do inter-thread event signalling.
800 Semaphores with an initial value of zero are particularly useful for
801 this.</p>
802<p>Helgrind only partially correctly handles POSIX condition
803 variables. This is because Helgrind can see inter-thread
804 dependencies between a <code class="function">pthread_cond_wait</code> call and a
805 <code class="function">pthread_cond_signal</code>/<code class="function">pthread_cond_broadcast</code>
806 call only if the waiting thread actually gets to the rendezvous first
807 (so that it actually calls
808 <code class="function">pthread_cond_wait</code>). It can't see dependencies
809 between the threads if the signaller arrives first. In the latter case,
810 POSIX guidelines imply that the associated boolean condition still
811 provides an inter-thread synchronisation event, but one which is
812 invisible to Helgrind.</p>
813<p>The result of Helgrind missing some inter-thread
814 synchronisation events is to cause it to report false positives.
815 </p>
816<p>The root cause of this synchronisation lossage is
817 particularly hard to understand, so an example is helpful. It was
818 discussed at length by Arndt Muehlenfeld ("Runtime Race Detection
819 in Multi-Threaded Programs", Dissertation, TU Graz, Austria). The
820 canonical POSIX-recommended usage scheme for condition variables
821 is as follows:</p>
822<pre class="programlisting">
823b is a Boolean condition, which is False most of the time
824cv is a condition variable
825mx is its associated mutex
826
827Signaller: Waiter:
828
829lock(mx) lock(mx)
830b = True while (b == False)
831signal(cv) wait(cv,mx)
832unlock(mx) unlock(mx)
833</pre>
834<p>Assume <code class="computeroutput">b</code> is False most of
835 the time. If the waiter arrives at the rendezvous first, it
836 enters its while-loop, waits for the signaller to signal, and
837 eventually proceeds. Helgrind sees the signal, notes the
838 dependency, and all is well.</p>
839<p>If the signaller arrives
840 first, <code class="computeroutput">b</code> is set to true, and the
841 signal disappears into nowhere. When the waiter later arrives, it
842 does not enter its while-loop and simply carries on. But even in
843 this case, the waiter code following the while-loop cannot execute
844 until the signaller sets <code class="computeroutput">b</code> to
845 True. Hence there is still the same inter-thread dependency, but
846 this time it is through an arbitrary in-memory condition, and
847 Helgrind cannot see it.</p>
848<p>By comparison, Helgrind's detection of inter-thread
849 dependencies caused by semaphore operations is believed to be
850 exactly correct.</p>
851<p>As far as I know, a solution to this problem that does not
852 require source-level annotation of condition-variable wait loops
853 is beyond the current state of the art.</p>
854</li>
855<li class="listitem"><p>Make sure you are using a supported Linux distribution. At
856 present, Helgrind only properly supports glibc-2.3 or later. This
857 in turn means we only support glibc's NPTL threading
858 implementation. The old LinuxThreads implementation is not
859 supported.</p></li>
860<li class="listitem"><p>If your application is using thread local variables,
861 helgrind might report false positive race conditions on these
862 variables, despite being very probably race free. On Linux, you can
863 use <code class="option">--sim-hints=deactivate-pthread-stack-cache-via-hack</code>
864 to avoid such false positive error messages
865 (see <a class="xref" href="manual-core.html#opt.sim-hints">--sim-hints</a>).
866 </p></li>
867<li class="listitem">
868<p>Round up all finished threads using
869 <code class="function">pthread_join</code>. Avoid
870 detaching threads: don't create threads in the detached state, and
871 don't call <code class="function">pthread_detach</code> on existing threads.</p>
872<p>Using <code class="function">pthread_join</code> to round up finished
873 threads provides a clear synchronisation point that both Helgrind and
874 programmers can see. If you don't call
875 <code class="function">pthread_join</code> on a thread, Helgrind has no way to
876 know when it finishes, relative to any
877 significant synchronisation points for other threads in the program. So
878 it assumes that the thread lingers indefinitely and can potentially
879 interfere indefinitely with the memory state of the program. It
880 has every right to assume that -- after all, it might really be
881 the case that, for scheduling reasons, the exiting thread did run
882 very slowly in the last stages of its life.</p>
883</li>
884<li class="listitem">
885<p>Perform thread debugging (with Helgrind) and memory
886 debugging (with Memcheck) together.</p>
887<p>Helgrind tracks the state of memory in detail, and memory
888 management bugs in the application are liable to cause confusion.
889 In extreme cases, applications which do many invalid reads and
890 writes (particularly to freed memory) have been known to crash
891 Helgrind. So, ideally, you should make your application
892 Memcheck-clean before using Helgrind.</p>
893<p>It may be impossible to make your application Memcheck-clean
894 unless you first remove threading bugs. In particular, it may be
895 difficult to remove all reads and writes to freed memory in
896 multithreaded C++ destructor sequences at program termination.
897 So, ideally, you should make your application Helgrind-clean
898 before using Memcheck.</p>
899<p>Since this circularity is obviously unresolvable, at least
900 bear in mind that Memcheck and Helgrind are to some extent
901 complementary, and you may need to use them together.</p>
902</li>
903<li class="listitem">
904<p>POSIX requires that implementations of standard I/O
905 (<code class="function">printf</code>, <code class="function">fprintf</code>,
906 <code class="function">fwrite</code>, <code class="function">fread</code>, etc) are thread
907 safe. Unfortunately GNU libc implements this by using internal locking
908 primitives that Helgrind is unable to intercept. Consequently Helgrind
909 generates many false race reports when you use these functions.</p>
910<p>Helgrind attempts to hide these errors using the standard
911 Valgrind error-suppression mechanism. So, at least for simple
912 test cases, you don't see any. Nevertheless, some may slip
913 through. Just something to be aware of.</p>
914</li>
915<li class="listitem">
916<p>Helgrind's error checks do not work properly inside the
917 system threading library itself
918 (<code class="computeroutput">libpthread.so</code>), and it usually
919 observes large numbers of (false) errors in there. Valgrind's
920 suppression system then filters these out, so you should not see
921 them.</p>
922<p>If you see any race errors reported
923 where <code class="computeroutput">libpthread.so</code> or
924 <code class="computeroutput">ld.so</code> is the object associated
925 with the innermost stack frame, please file a bug report at
926 <a class="ulink" href="http://www.valgrind.org/" target="_top">http://www.valgrind.org/</a>.
927 </p>
928</li>
929</ol></div>
930</div>
931<div class="sect1">
932<div class="titlepage"><div><div><h2 class="title" style="clear: both">
933<a name="hg-manual.options"></a>7.6. Helgrind Command-line Options</h2></div></div></div>
934<p>The following end-user options are available:</p>
935<div class="variablelist">
936<a name="hg.opts.list"></a><dl class="variablelist">
937<dt>
938<a name="opt.free-is-write"></a><span class="term">
939 <code class="option">--free-is-write=no|yes
940 [default: no] </code>
941 </span>
942</dt>
943<dd>
944<p>When enabled (not the default), Helgrind treats freeing of
945 heap memory as if the memory was written immediately before
946 the free. This exposes races where memory is referenced by
947 one thread, and freed by another, but there is no observable
948 synchronisation event to ensure that the reference happens
949 before the free.
950 </p>
951<p>This functionality is new in Valgrind 3.7.0, and is
952 regarded as experimental. It is not enabled by default
953 because its interaction with custom memory allocators is not
954 well understood at present. User feedback is welcomed.
955 </p>
956</dd>
957<dt>
958<a name="opt.track-lockorders"></a><span class="term">
959 <code class="option">--track-lockorders=no|yes
960 [default: yes] </code>
961 </span>
962</dt>
963<dd><p>When enabled (the default), Helgrind performs lock order
964 consistency checking. For some buggy programs, the large number
965 of lock order errors reported can become annoying, particularly
966 if you're only interested in race errors. You may therefore find
967 it helpful to disable lock order checking.</p></dd>
968<dt>
969<a name="opt.history-level"></a><span class="term">
970 <code class="option">--history-level=none|approx|full
971 [default: full] </code>
972 </span>
973</dt>
974<dd>
975<p><code class="option">--history-level=full</code> (the default) causes
976 Helgrind collects enough information about "old" accesses that
977 it can produce two stack traces in a race report -- both the
978 stack trace for the current access, and the trace for the
979 older, conflicting access. To limit memory usage, "old" accesses
980 stack traces are limited to a maximum of 8 entries, even if
981 <code class="option">--num-callers</code> value is bigger.</p>
982<p>Collecting such information is expensive in both speed and
983 memory, particularly for programs that do many inter-thread
984 synchronisation events (locks, unlocks, etc). Without such
985 information, it is more difficult to track down the root
986 causes of races. Nonetheless, you may not need it in
987 situations where you just want to check for the presence or
988 absence of races, for example, when doing regression testing
989 of a previously race-free program.</p>
990<p><code class="option">--history-level=none</code> is the opposite
991 extreme. It causes Helgrind not to collect any information
992 about previous accesses. This can be dramatically faster
993 than <code class="option">--history-level=full</code>.</p>
994<p><code class="option">--history-level=approx</code> provides a
995 compromise between these two extremes. It causes Helgrind to
996 show a full trace for the later access, and approximate
997 information regarding the earlier access. This approximate
998 information consists of two stacks, and the earlier access is
999 guaranteed to have occurred somewhere between program points
1000 denoted by the two stacks. This is not as useful as showing
1001 the exact stack for the previous access
1002 (as <code class="option">--history-level=full</code> does), but it is
1003 better than nothing, and it is almost as fast as
1004 <code class="option">--history-level=none</code>.</p>
1005</dd>
1006<dt>
1007<a name="opt.conflict-cache-size"></a><span class="term">
1008 <code class="option">--conflict-cache-size=N
1009 [default: 1000000] </code>
1010 </span>
1011</dt>
1012<dd>
1013<p>This flag only has any effect
1014 at <code class="option">--history-level=full</code>.</p>
1015<p>Information about "old" conflicting accesses is stored in
1016 a cache of limited size, with LRU-style management. This is
1017 necessary because it isn't practical to store a stack trace
1018 for every single memory access made by the program.
1019 Historical information on not recently accessed locations is
1020 periodically discarded, to free up space in the cache.</p>
1021<p>This option controls the size of the cache, in terms of the
1022 number of different memory addresses for which
1023 conflicting access information is stored. If you find that
1024 Helgrind is showing race errors with only one stack instead of
1025 the expected two stacks, try increasing this value.</p>
1026<p>The minimum value is 10,000 and the maximum is 30,000,000
1027 (thirty times the default value). Increasing the value by 1
1028 increases Helgrind's memory requirement by very roughly 100
1029 bytes, so the maximum value will easily eat up three extra
1030 gigabytes or so of memory.</p>
1031</dd>
1032<dt>
1033<a name="opt.check-stack-refs"></a><span class="term">
1034 <code class="option">--check-stack-refs=no|yes
1035 [default: yes] </code>
1036 </span>
1037</dt>
1038<dd><p>
1039 By default Helgrind checks all data memory accesses made by your
1040 program. This flag enables you to skip checking for accesses
1041 to thread stacks (local variables). This can improve
1042 performance, but comes at the cost of missing races on
1043 stack-allocated data.
1044 </p></dd>
1045<dt>
1046<a name="opt.ignore-thread-creation"></a><span class="term">
1047 <code class="option">--ignore-thread-creation=&lt;yes|no&gt;
1048 [default: no]</code>
1049 </span>
1050</dt>
1051<dd>
1052<p>
1053 Controls whether all activities during thread creation should be
1054 ignored. By default enabled only on Solaris.
1055 Solaris provides higher throughput, parallelism and scalability than
1056 other operating systems, at the cost of more fine-grained locking
1057 activity. This means for example that when a thread is created under
1058 glibc, just one big lock is used for all thread setup. Solaris libc
1059 uses several fine-grained locks and the creator thread resumes its
1060 activities as soon as possible, leaving for example stack and TLS setup
1061 sequence to the created thread.
1062 This situation confuses Helgrind as it assumes there is some false
1063 ordering in place between creator and created thread; and therefore many
1064 types of race conditions in the application would not be reported.
1065 To prevent such false ordering, this command line option is set to
1066 <code class="computeroutput">yes</code> by default on Solaris.
1067 All activity (loads, stores, client requests) is therefore ignored
1068 during:</p>
1069<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1070<li class="listitem"><p>
1071 pthread_create() call in the creator thread
1072 </p></li>
1073<li class="listitem"><p>
1074 thread creation phase (stack and TLS setup) in the created thread
1075 </p></li>
1076</ul></div>
1077<p>
1078 Also new memory allocated during thread creation is untracked,
1079 that is race reporting is suppressed there. DRD does the same thing
1080 implicitly. This is necessary because Solaris libc caches many objects
1081 and reuses them for different threads and that confuses
1082 Helgrind.</p>
1083</dd>
1084</dl>
1085</div>
1086</div>
1087<div class="sect1">
1088<div class="titlepage"><div><div><h2 class="title" style="clear: both">
1089<a name="hg-manual.monitor-commands"></a>7.7. Helgrind Monitor Commands</h2></div></div></div>
1090<p>The Helgrind tool provides monitor commands handled by Valgrind's
1091built-in gdbserver (see <a class="xref" href="manual-core-adv.html#manual-core-adv.gdbserver-commandhandling" title="3.2.5. Monitor command handling by the Valgrind gdbserver">Monitor command handling by the Valgrind gdbserver</a>).
1092</p>
1093<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1094<li class="listitem">
1095<p><code class="varname">info locks [lock_addr]</code> shows the list of locks
1096 and their status. If <code class="varname">lock_addr</code> is given, only shows
1097 the lock located at this address. </p>
1098<p>
1099 In the following example, helgrind knows about one lock. This
1100 lock is located at the guest address <code class="varname">ga
1101 0x8049a20</code>. The lock kind is <code class="varname">rdwr</code>
1102 indicating a reader-writer lock. Other possible lock kinds
1103 are <code class="varname">nonRec</code> (simple mutex, non recursive)
1104 and <code class="varname">mbRec</code> (simple mutex, possibly recursive).
1105 The lock kind is then followed by the list of threads helding the
1106 lock. In the below example, <code class="varname">R1:thread #6 tid 3</code>
1107 indicates that the helgrind thread #6 has acquired (once, as the
1108 counter following the letter R is one) the lock in read mode. The
1109 helgrind thread nr is incremented for each started thread. The
1110 presence of 'tid 3' indicates that the thread #6 is has not exited
1111 yet and is the valgrind tid 3. If a thread has terminated, then
1112 this is indicated with 'tid (exited)'.
1113 </p>
1114<pre class="programlisting">
1115(gdb) monitor info locks
1116Lock ga 0x8049a20 {
1117 kind rdwr
1118 { R1:thread #6 tid 3 }
1119}
1120(gdb)
1121</pre>
1122<p> If you give the option <code class="varname">--read-var-info=yes</code>,
1123 then more information will be provided about the lock location, such as
1124 the global variable or the heap block that contains the lock:
1125 </p>
1126<pre class="programlisting">
1127Lock ga 0x8049a20 {
1128 Location 0x8049a20 is 0 bytes inside global var "s_rwlock"
1129 declared at rwlock_race.c:17
1130 kind rdwr
1131 { R1:thread #3 tid 3 }
1132}
1133</pre>
1134</li>
1135<li class="listitem">
1136<p><code class="varname">accesshistory &lt;addr&gt; [&lt;len&gt;]</code>
1137 shows the access history recorded for &lt;len&gt; (default 1) bytes
1138 starting at &lt;addr&gt;. For each recorded access that overlaps
1139 with the given range, <code class="varname">accesshistory</code> shows the operation
1140 type (read or write), the address and size read or written, the helgrind
1141 thread nr/valgrind tid number that did the operation and the locks held
1142 by the thread at the time of the operation.
1143 The oldest access is shown first, the most recent access is shown last.
1144 </p>
1145<p>
1146 In the following example, we see first a recorded write of 4 bytes by
1147 thread #7 that has modified the given 2 bytes range.
1148 The second recorded write is the most recent recorded write : thread #9
1149 modified the same 2 bytes as part of a 4 bytes write operation.
1150 The list of locks held by each thread at the time of the write operation
1151 are also shown.
1152 </p>
1153<pre class="programlisting">
1154(gdb) monitor accesshistory 0x8049D8A 2
1155write of size 4 at 0x8049D88 by thread #7 tid 3
1156==6319== Locks held: 2, at address 0x8049D8C (and 1 that can't be shown)
1157==6319== at 0x804865F: child_fn1 (locked_vs_unlocked2.c:29)
1158==6319== by 0x400AE61: mythread_wrapper (hg_intercepts.c:234)
1159==6319== by 0x39B924: start_thread (pthread_create.c:297)
1160==6319== by 0x2F107D: clone (clone.S:130)
1161
1162write of size 4 at 0x8049D88 by thread #9 tid 2
1163==6319== Locks held: 2, at addresses 0x8049DA4 0x8049DD4
1164==6319== at 0x804877B: child_fn2 (locked_vs_unlocked2.c:45)
1165==6319== by 0x400AE61: mythread_wrapper (hg_intercepts.c:234)
1166==6319== by 0x39B924: start_thread (pthread_create.c:297)
1167==6319== by 0x2F107D: clone (clone.S:130)
1168
1169</pre>
1170</li>
1171</ul></div>
1172</div>
1173<div class="sect1">
1174<div class="titlepage"><div><div><h2 class="title" style="clear: both">
1175<a name="hg-manual.client-requests"></a>7.8. Helgrind Client Requests</h2></div></div></div>
1176<p>The following client requests are defined in
1177<code class="filename">helgrind.h</code>. See that file for exact details of their
1178arguments.</p>
1179<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1180<li class="listitem">
1181<p><code class="function">VALGRIND_HG_CLEAN_MEMORY</code></p>
1182<p>This makes Helgrind forget everything it knows about a
1183 specified memory range. This is particularly useful for memory
1184 allocators that wish to recycle memory.</p>
1185</li>
1186<li class="listitem"><p><code class="function">ANNOTATE_HAPPENS_BEFORE</code></p></li>
1187<li class="listitem"><p><code class="function">ANNOTATE_HAPPENS_AFTER</code></p></li>
1188<li class="listitem"><p><code class="function">ANNOTATE_NEW_MEMORY</code></p></li>
1189<li class="listitem"><p><code class="function">ANNOTATE_RWLOCK_CREATE</code></p></li>
1190<li class="listitem"><p><code class="function">ANNOTATE_RWLOCK_DESTROY</code></p></li>
1191<li class="listitem"><p><code class="function">ANNOTATE_RWLOCK_ACQUIRED</code></p></li>
1192<li class="listitem">
1193<p><code class="function">ANNOTATE_RWLOCK_RELEASED</code></p>
1194<p>These are used to describe to Helgrind, the behaviour of
1195 custom (non-POSIX) synchronisation primitives, which it otherwise
1196 has no way to understand. See comments
1197 in <code class="filename">helgrind.h</code> for further
1198 documentation.</p>
1199</li>
1200</ul></div>
1201</div>
1202<div class="sect1">
1203<div class="titlepage"><div><div><h2 class="title" style="clear: both">
1204<a name="hg-manual.todolist"></a>7.9. A To-Do List for Helgrind</h2></div></div></div>
1205<p>The following is a list of loose ends which should be tidied up
1206some time.</p>
1207<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1208<li class="listitem"><p>For lock order errors, print the complete lock
1209 cycle, rather than only doing for size-2 cycles as at
1210 present.</p></li>
1211<li class="listitem"><p>The conflicting access mechanism sometimes
1212 mysteriously fails to show the conflicting access' stack, even
1213 when provided with unbounded storage for conflicting access info.
1214 This should be investigated.</p></li>
1215<li class="listitem"><p>Document races caused by GCC's thread-unsafe code
1216 generation for speculative stores. In the interim see
1217 <code class="computeroutput">http://gcc.gnu.org/ml/gcc/2007-10/msg00266.html
1218 </code>
1219 and <code class="computeroutput">http://lkml.org/lkml/2007/10/24/673</code>.
1220 </p></li>
1221<li class="listitem"><p>Don't update the lock-order graph, and don't check
1222 for errors, when a "try"-style lock operation happens (e.g.
1223 <code class="function">pthread_mutex_trylock</code>). Such calls do not add any real
1224 restrictions to the locking order, since they can always fail to
1225 acquire the lock, resulting in the caller going off and doing Plan
1226 B (presumably it will have a Plan B). Doing such checks could
1227 generate false lock-order errors and confuse users.</p></li>
1228<li class="listitem"><p> Performance can be very poor. Slowdowns on the
1229 order of 100:1 are not unusual. There is limited scope for
1230 performance improvements.
1231 </p></li>
1232</ul></div>
1233</div>
1234</div>
1235<div>
1236<br><table class="nav" width="100%" cellspacing="3" cellpadding="2" border="0" summary="Navigation footer">
1237<tr>
1238<td rowspan="2" width="40%" align="left">
1239<a accesskey="p" href="cl-manual.html">&lt;&lt; 6. Callgrind: a call-graph generating cache and branch prediction profiler</a> </td>
1240<td width="20%" align="center"><a accesskey="u" href="manual.html">Up</a></td>
1241<td rowspan="2" width="40%" align="right"> <a accesskey="n" href="drd-manual.html">8. DRD: a thread error detector &gt;&gt;</a>
1242</td>
1243</tr>
1244<tr><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td></tr>
1245</table>
1246</div>
1247</body>
1248</html>