
A mini-FAQ for valgrind, version 1.9.6
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Last revised 5 May 2003
~~~~~~~~~~~~~~~~~~~~~~~

-----------------------------------------------------------------

Q1. Programs run OK on valgrind, but at exit produce a bunch
    of errors a bit like this

    ==20755== Invalid read of size 4
    ==20755==    at 0x40281C8A: _nl_unload_locale (loadlocale.c:238)
    ==20755==    by 0x4028179D: free_mem (findlocale.c:257)
    ==20755==    by 0x402E0962: __libc_freeres (set-freeres.c:34)
    ==20755==    by 0x40048DCC: vgPlain___libc_freeres_wrapper 
                                              (vg_clientfuncs.c:585)
    ==20755==    Address 0x40CC304C is 8 bytes inside a block of size 380 free'd
    ==20755==    at 0x400484C9: free (vg_clientfuncs.c:180)
    ==20755==    by 0x40281CBA: _nl_unload_locale (loadlocale.c:246)
    ==20755==    by 0x40281218: free_mem (setlocale.c:461)
    ==20755==    by 0x402E0962: __libc_freeres (set-freeres.c:34)

    and then die with a segmentation fault.

A1. When the program exits, valgrind runs the procedure
    __libc_freeres() in glibc.  This is a hook for memory debuggers,
    so they can ask glibc to free up any memory it has used.  Doing
    that is needed to ensure that valgrind doesn't incorrectly
    report space leaks in glibc.

    Problem is that running __libc_freeres() in older glibc versions
    causes this crash.  

    WORKAROUND FOR 1.1.X and later versions of valgrind: use the
    --run-libc-freeres=no flag.  You may then get space leak
    reports for glibc-allocations (please _don't_ report these
    to the glibc people, since they are not real leaks), but at
    least the program runs.

-----------------------------------------------------------------

Q2. My program dies complaining that syscall 197 is unimplemented.

A2. 197, which is fstat64, is supported by valgrind.  The problem is
    that the /usr/include/asm/unistd.h on the machine on which your
    valgrind was built, doesn't match your kernel -- or, to be more 
    specific, glibc is asking your kernel to do a syscall which is
    not listed in /usr/include/asm/unistd.h.

    The fix is simple.  Somewhere near the top of
    coregrind/vg_syscalls.c, add the following line:

       #define __NR_fstat64            197

    Rebuild and try again.  The above line should appear before any
    uses of the __NR_fstat64 symbol in that file.  If you look at the
    place where __NR_fstat64 is used in vg_syscalls.c, it will be
    obvious why this fix works.

-----------------------------------------------------------------

Q3. My (buggy) program dies like this:
      valgrind: vg_malloc2.c:442 (bszW_to_pszW): 
                Assertion `pszW >= 0' failed.
    And/or my (buggy) program runs OK on valgrind, but dies like 
    this on cachegrind.

A3. If valgrind shows any invalid reads, invalid writes and invalid
    frees in your program, the above may happen.  Reason is that your
    program may trash valgrind's low-level memory manager, which then
    dies with the above assertion, or something like this.  The cure
    is to fix your program so that it doesn't do any illegal memory
    accesses.  The above failure will hopefully go away after that.

-----------------------------------------------------------------

Q4. I'm running Red Hat Advanced Server.  Valgrind always segfaults at
    startup.  

A4. Known issue with RHAS 2.1, due to funny stack permissions at
    startup.  However, valgrind-1.9.4 and later automatically handle
    this correctly, and should not segfault.

-----------------------------------------------------------------

Q5. I try running "valgrind my_program", but my_program runs normally,
    and Valgrind doesn't emit any output at all.

A5. Is my_program statically linked?  Valgrind doesn't work with
    statically linked binaries.  my_program must rely on at least one 
    shared object.  To determine if a my_program is statically linked, 
    run:

      ldd my_program

    It will show what shared objects my_program relies on, or say:

      not a dynamic executable

    if my_program is statically linked.

-----------------------------------------------------------------

Q6. I try running "valgrind my_program" and get Valgrind's startup message,
    but I don't get any errors and I know my program has errors.
   
A6. By default, Valgrind only traces the top-level process.  So if your
    program spawns children, they won't be traced by Valgrind by default.
    Also, if your program is started by a shell script, Perl script, or
    something similar, Valgrind will trace the shell, or the Perl 
    interpreter, or equivalent.

    To trace child processes, use the --trace-children=yes option.

    If you are tracing large trees of processes, it can be less
    disruptive to have the output sent over the network.  Give
    valgrind the flag --logsocket=127.0.0.1:12345 (if you want 
    logging output sent to port 12345 on localhost).  You can
    use the valgrind-listener program to listen on that port:
       valgrind-listener 12345
    Obviously you have to start the listener process first.
    See the documentation for more details.

-----------------------------------------------------------------

Q7. My threaded server process runs unbelievably slowly on
    valgrind.  So slowly, in fact, that at first I thought it
    had completely locked up.

A7. We are not completely sure about this, but one possibility
    is that laptops with power management fool valgrind's 
    timekeeping mechanism, which is (somewhat in error) based
    on the x86 RDTSC instruction.  A "fix" which is claimed to
    work is to run some other cpu-intensive process at the same
    time, so that the laptop's power-management clock-slowing
    does not kick in.  We would be interested in hearing more
    feedback on this.

    Another possible cause is that versions prior to 1.9.6
    did not support threading on glibc 2.3.X systems well.
    Hopefully the situation is much improved with 1.9.6.

-----------------------------------------------------------------

Q8. My program dies (exactly) like this:

      REPE then 0xF
      valgrind: the `impossible' happened:
         Unhandled REPE case

A8. Yeah ... that I believe is a SSE or SSE2 instruction.  Are you
    building your app with -march=pentium4 or -march=athlon or
    something like that?  If you can somehow dissuade gcc from 
    producing SSE/SSE2 instructions, you may be able to avoid this.
    Some folks have reported that removing the flag -march=...
    works around this.

    I'd be interested to hear if you can get rid of it by changing
    your application build flags.

-----------------------------------------------------------------

Q9. My program dies complaining that __libc_current_sigrtmin
    is unimplemented.

A9. Should be fixed in 1.9.6.  I would appreciate confirmation 
    of that.

-----------------------------------------------------------------

Q10. I upgraded to Red Hat 9 and threaded programs now act
     strange / deadlock when they didn't before.

A10. Thread support on glibc 2.3.2+ with NPTL is not as 
     good as on older LinuxThreads-based systems.  We have
     this under consideration.  Avoid Red Hat >= 8.1 for
     the time being, if you can.

     5 May 03: 1.9.6 should be significantly improved on 
     Red Hat 9, SuSE 8.2 and other glibc-2.3.2 systems.

-----------------------------------------------------------------

Q11. I really need to use the NVidia libGL.so in my app.
     Help!

A11. NVidia also noticed this it seems, and the "latest" drivers
     (version 4349, apparently) come with this text

        DISABLING CPU SPECIFIC FEATURES

        Setting the environment variable __GL_FORCE_GENERIC_CPU to a
        non-zero value will inhibit the use of CPU specific features
        such as MMX, SSE, or 3DNOW!.  Use of this option may result in
        performance loss.  This option may be useful in conjunction with
        software such as the Valgrind memory debugger.

     Set __GL_FORCE_GENERIC_CPU=1 and Valgrind should work.  This has
     been confirmed by various people.  Thanks NVidia!

-----------------------------------------------------------------

Q12. My program dies like this (often at exit):

     VG_(mash_LD_PRELOAD_and_LD_LIBRARY_PATH): internal error:
     (loads of text)

A12. One possible cause is that your program modifies its
     environment variables, possibly including zeroing them
     all.  Valgrind relies on the LD_PRELOAD, LD_LIBRARY_PATH and
     VG_ARGS variables.  Zeroing them will break things.

     As of 1.9.6, Valgrind only uses these variables with
     --trace-children=no, when executing execve() or using the
     --stop-after=yes flag.  This should reduce the potential for
     problems.

-----------------------------------------------------------------

Q13.  My program dies like this:

      error: /lib/librt.so.1: symbol __pthread_clock_settime, version
      GLIBC_PRIVATE not defined in file libpthread.so.0 with link time
      reference

A13.  This is a total swamp.  Nevertheless there is a way out.
      It's a problem which is not easy to fix.  Really the problem is
      that /lib/librt.so.1 refers to some symbols
      __pthread_clock_settime and __pthread_clock_gettime in
      /lib/libpthread.so which are not intended to be exported, ie
      they are private.

      Best solution is to ensure your program does not use
      /lib/librt.so.1.

      However .. since you're probably not using it directly, or even
      knowingly, that's hard to do.  You might instead be able to fix
      it by playing around with coregrind/vg_libpthread.vs.  Things to
      try:

      Remove this

         GLIBC_PRIVATE {
            __pthread_clock_gettime;
            __pthread_clock_settime;
         };

      or maybe remove this

         GLIBC_2.2.3 {
            __pthread_clock_gettime;
            __pthread_clock_settime;
         } GLIBC_2.2;

      or maybe add this

         GLIBC_2.2.4 {
            __pthread_clock_gettime;
            __pthread_clock_settime;
         } GLIBC_2.2;

         GLIBC_2.2.5 {
            __pthread_clock_gettime;
            __pthread_clock_settime;
         } GLIBC_2.2;

      or some combination of the above.  After each change you need to
      delete coregrind/libpthread.so and do make && make install.

      I just don't know if any of the above will work.  If you can
      find a solution which works, I would be interested to hear it.

      To which someone replied:

      I deleted this:

          GLIBC_2.2.3 { 
             __pthread_clock_gettime; 
             __pthread_clock_settime; 
          } GLIBC_2.2; 

      and it worked.

-----------------------------------------------------------------

Q14.  My program uses the C++ STL and string classes.  Valgrind
      reports 'still reachable' memory leaks involving these classes
      at the exit of the program, but there should be none.

A14.  First of all: relax, it's probably not a bug, but a feature.
      Many implementations of the C++ standard libraries use their own
      memory pool allocators.  Memory for quite a number of destructed
      objects is not immediately freed and given back to the OS, but
      kept in the pool(s) for later re-use.  The fact that the pools
      are not freed at the exit() of the program cause valgrind to
      report this memory as still reachable.  The behaviour not to
      free pools at the exit() could be called a bug of the library
      though.

      Using gcc, you can force the STL to use malloc and to free
      memory as soon as possible by globally disabling memory caching.
      Beware!  Doing so will probably slow down your program,
      sometimes drastically.

      - With gcc 2.91, 2.95, 3.0 and 3.1, compile all source using the
        STL with -D__USE_MALLOC. Beware! This is removed from gcc
        starting with version 3.3.

      - With 3.2.2 and later, you should export the environment
        variable GLIBCPP_FORCE_NEW before running your program.

      There are other ways to disable memory pooling: using the
      malloc_alloc template with your objects (not portable, but
      should work for gcc) or even writing your own memory
      allocators. But all this goes beyond the scope of this
      FAQ.  Start by reading
         http://gcc.gnu.org/onlinedocs/libstdc++/ext/howto.html#3
      if you absolutely want to do that. But beware:

         1) there are currently changes underway for gcc which are not
            totally reflected in the docs right now 
            ("now" == 26 Apr 03)

         2) allocators belong to the more messy parts of the STL and
            people went at great lengths to make it portable across
            platforms. Chances are good that your solution will work
            on your platform, but not on others.

-----------------------------------------------------------------

Q15.  My program dies with a segmentation fault, but Valgrind doesn't give
      any error messages before it, or none that look related.

A15.  The one kind of segmentation fault that Valgrind won't give any
      warnings about is writes to read-only memory.  Maybe your program is
      writing to a static string like this:

        char* s = "hello";
        s[0] = 'j';

      or something similar.  Writing to read-only memory can also apparently
      make LinuxThreads behave strangely.

-----------------------------------------------------------------

Q16.  When I trying building Valgrind, 'make' dies partway with an
      assertion failure, something like this: make: expand.c:489:
      
        allocated_variable_append: Assertion 
        `current_variable_set_list->next != 0' failed.
      
A16.  It's probably a bug in 'make'.  Some, but not all, instances of
      version 3.79.1 have this bug, see 
      www.mail-archive.com/bug-make@gnu.org/msg01658.html.  Try upgrading to a
      more recent version of 'make'.

-----------------------------------------------------------------

Q17.  I tried writing a suppression but it didn't work.  Can you 
      write my suppression for me?

A17.  Yes!  Use the --gen-suppressions=yes feature to spit out
      suppressions automatically for you.  You can then edit them
      if you like, eg. combining similar automatically generated
      suppressions using wildcards like '*'.

      If you really want to write suppressions by hand, read the 
      manual carefully.  Note particularly that C++ function names
      must be _mangled_.

-----------------------------------------------------------------

(this is the end of the FAQ.)