Implement destructors for thread-local heap data.
Description of problem:
Use of __thread variables is great for creating a thread-safe variable, but
only insofar as the contents of that variable can safely be abandoned on
pthread_exit(). The moment you store malloc()d data into a __thread void*
variable, you have leaked memory when the thread exits, since there is no way
to associate a destructor with __thread variables.
The _only_ safe way to use thread-local caching of malloc()d data is to use
pthread_key_create, and associate a destructor that will call free() on the
resulting data when the thread exits.
libselinux is guilty of abusing __thread variables to store malloc()d data as a
form of a cache, to minimize computation by reusing earlier results from the
same thread. As a result of this memory leak, repeated starting and stopping
of domains via libvirt can result in the OOM killer triggering, since libvirt
fires up a thread per domain, and each thread uses selinux calls such as
fgetfilecon.
Version-Release number of selected component (if applicable):
libselinux-2.0.94-2.el6.x86_64
libvirt-0.8.1-27.el6.x86_64
How reproducible:
100%
Steps to Reproduce:
0. These steps are run as root, assuming hardware kvm support and existence of
a VM named fedora (adjust the steps below as appropriate); if desired, I can
reduce this to a simpler test case that does not rely on libvirt, by using a
single .c file that links against libselinux and repeatedly spawns threads.
1. service libvirtd stop
2. valgrind --quiet --leak-check=full /usr/sbin/libvirtd& pid=$!
3. virsh start fedora
4. kill $pid
Actual results:
The biggest leak reported is due to libselinux' abuse of __thread:
==26696== 829,730 (40 direct, 829,690 indirect) bytes in 1 blocks are
definitely lost in loss record 500 of 500
==26696== at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==26696== by 0x3022E0D48C: selabel_open (label.c:165)
==26696== by 0x3022E11646: matchpathcon_init_prefix (matchpathcon.c:296)
==26696== by 0x3022E1190D: matchpathcon (matchpathcon.c:317)
==26696== by 0x3033ED7FB5: SELinuxRestoreSecurityFileLabel (security_selinux.c:381)
==26696== by 0x3033ED8539: SELinuxRestoreSecurityAllLabel (security_selinux.c:749)
==26696== by 0x459153: qemuSecurityStackedRestoreSecurityAllLabel (qemu_security_stacked.c:257)
==26696== by 0x43F0C5: qemudShutdownVMDaemon (qemu_driver.c:4311)
==26696== by 0x4555C9: qemudStartVMDaemon (qemu_driver.c:4234)
==26696== by 0x458416: qemudDomainObjStart (qemu_driver.c:7268)
==26696== by 0x45896F: qemudDomainStart (qemu_driver.c:7308)
==26696== by 0x3033E75412: virDomainCreate (libvirt.c:4881)
==26696==
Basically, libvirt created a thread that used matchpathcon during 'virsh start
fedora', and matchpathcon stuffed over 800k of malloc'd data into:
static __thread char **con_array;
which are then inaccessible when libvirt exits the thread as part of shutting
down on SIGTERM.
Expected results:
valgrind should not report any memory leaks related to libselinux.
Signed-off-by: Eamon Walsh <ewalsh@tycho.nsa.gov>
Reported-by: Eric Blake <eblake@redhat.com>
Tested-by: Eric Blake <eblake@redhat.com>
3 files changed