outer/inner setup: new perf/vg_perf options to run perf tests + support translation chaining in inner.
* perf/vg_perf:
Similarly to tests/vg_regtest, perf/vg_perf now accepts the 3
optional arguments:
--outer-valgrind
--outer-tool
--outer-args
This allows easy analysis or comparison of performance between
different Valgrind versions (e.g. using callgrind, or cachegrind/cg_diff).
* See README_DEVELOPERS for more details.
* vg_regtest modified so as to use the 'in-place' build of inner, rather
than the installed version.
* added option --smc-check=all-non-file to vg_perf and vg_regtest
outer default arguments (needed when evaluating a Valgrind which does
translation chaining).
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@12496 a5019735-40e9-0310-863c-91ae7b9d1cf9
diff --git a/README_DEVELOPERS b/README_DEVELOPERS
index d09917a..e6a005e 100644
--- a/README_DEVELOPERS
+++ b/README_DEVELOPERS
@@ -136,7 +136,15 @@
Self-hosting
~~~~~~~~~~~~
-To run Valgrind under Valgrind:
+This section explains :
+ (A) How to configure Valgrind to run under Valgrind.
+ Such a setup is called self hosting, or outer/inner setup.
+ (B) How to run Valgrind regression tests in a 'self-hosting' mode,
+ e.g. to verify Valgrind has no bugs such as memory leaks.
+ (C) How to run Valgrind performance tests in a 'self-hosting' mode,
+ to analyse and optimise the performance of Valgrind and its tools.
+
+(A) How to configure Valgrind to run under Valgrind:
(1) Check out 2 trees, "Inner" and "Outer". Inner runs the app
directly. Outer runs Inner.
@@ -148,6 +156,7 @@
(4) Choose a very simple program (date) and try
outer/.../bin/valgrind --sim-hints=enable-outer --trace-children=yes \
+ --smc-check=all-non-file \
--run-libc-freeres=no --tool=cachegrind -v \
inner/.../bin/valgrind --vgdb-prefix=./inner --tool=none -v prog
@@ -156,6 +165,10 @@
it will try to find and run __libc_freeres in the inner, while libc is not
used by the inner. Inner needs --vgdb-prefix=./inner to avoid inner
gdbserver colliding with outer gdbserver.
+Currently, inner does *not* use the client request
+VALGRIND_DISCARD_TRANSLATIONS for the JITted code or the code patched for
+translation chaining. So the outer needs --smc-check=all-non-file to
+detect the modified code.
Debugging the whole thing might imply to use up to 3 GDB:
* a GDB attached to the Outer valgrind, allowing
@@ -186,7 +199,8 @@
When using self-hosting with an outer Callgrind tool, use '--pop-on-jump'
(on the outer). Otherwise, Callgrind has much higher memory requirements.
-Regression tests in an outer/inner setup:
+(B) Regression tests in an outer/inner setup:
+
To run all the regression tests with an outer memcheck, do :
perl test/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \
--all
@@ -197,7 +211,7 @@
To run regression tests with another outer tool:
perl tests/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \
- --outer-tool=helgrind " --all
+ --outer-tool=helgrind --all
--outer-args allows to give specific arguments to the outer tool,
replacing the default one provided by vg_regtest.
@@ -211,6 +225,48 @@
The file tests/outer_inner.supp contains suppressions for
the irrelevant or benign errors found in the inner.
+(C) Performance tests in an outer/inner setup:
+
+ To run all the performance tests with an outer cachegrind, do :
+ perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind perf
+
+ To run a specific perf test (e.g. bz2) in this setup, do :
+ perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind perf/bz2
+
+ To run all the performance tests with an outer callgrind, do :
+ perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind \
+ --outer-tool=callgrind perf
+
+ To compare the performance of multiple Valgrind versions, do :
+ perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind \
+ --vg=../inner_xxxx --vg=../inner_yyyy perf
+ (where inner_xxxx and inner_yyyy are the versions to compare).
+ Cachegrind and cg_diff are particularly handy to obtain a delta
+ between the two versions.
+
+When the outer tool is callgrind or cachegrind, the following
+output files will be created for each test:
+ <outertoolname>.out.<inner_valgrind_dir>.<tt>.<perftestname>.<pid>
+ <outertoolname>.outer.log.<inner_valgrind_dir>.<tt>.<perftestname>.<pid>
+ (where tt is the two letters abbreviation for the inner tool(s) run).
+
+For example, the command
+ perl perf/vg_perf \
+ --outer-valgrind=../outer_trunk/install/bin/valgrind \
+ --outer-tool=callgrind \
+ --vg=../inner_tchain --vg=../inner_trunk perf/many-loss-records
+
+produces the files
+ callgrind.out.inner_tchain.no.many-loss-records.18465
+ callgrind.outer.log.inner_tchain.no.many-loss-records.18465
+ callgrind.out.inner_tchain.me.many-loss-records.21899
+ callgrind.outer.log.inner_tchain.me.many-loss-records.21899
+ callgrind.out.inner_trunk.no.many-loss-records.21224
+ callgrind.outer.log.inner_trunk.no.many-loss-records.21224
+ callgrind.out.inner_trunk.me.many-loss-records.22916
+ callgrind.outer.log.inner_trunk.me.many-loss-records.22916
+
+
Printing out problematic blocks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to print out a disassembly of a particular block that
diff --git a/perf/vg_perf.in b/perf/vg_perf.in
index 2188a31..9be023e 100644
--- a/perf/vg_perf.in
+++ b/perf/vg_perf.in
@@ -58,9 +58,14 @@
-h --help show this message
--reps=<n> number of repeats for each program [1]
--tools=<t1,t2,t3> tools to run [Nulgrind and Memcheck]
- --vg Valgrind(s) to measure (can be specified multiple
- times). The "in-place" build is used.
- [Valgrind in the current directory]
+ --vg Valgrind(s) to measure [Valgrind in the current directory]
+ (can be specified multiple times).
+ The "in-place" build is used.
+
+ --outer-valgrind: run these Valgrind(s) under the given outer valgrind.
+ These Valgrind(s) must be configured with --enable-inner.
+ --outer-tool: tool to use by the outer valgrind (default cachegrind).
+ --outer-args: use this as outer tool args.
Any tools named in --tools must be present in all directories specified
with --vg. (This is not checked.)
@@ -79,6 +84,14 @@
my @vgdirs; # Dirs of the various Valgrinds being measured.
my @tools = ("none", "memcheck"); # tools being measured
+# Outer valgrind to use, and args to use for it.
+# If this is set, --valgrind should be set to the installed inner valgrind,
+# and --valgrind-lib will be ignore
+my $outer_valgrind;
+my $outer_tool = "cachegrind";
+my $outer_args;
+
+
my $num_tests_done = 0;
my $num_timings_done = 0;
@@ -119,7 +132,6 @@
{
my ($vgdir) = @_;
if ($vgdir !~ /^\//) { $vgdir = "$tests_dir/$vgdir"; }
- validate_program($vgdir, "./coregrind/valgrind", 1, 1);
push(@vgdirs, $vgdir);
}
@@ -137,6 +149,12 @@
add_vgdir($1);
} elsif ($arg =~ /^--tools=(.+)$/) {
@tools = split(/,/, $1);
+ } elsif ($arg =~ /^--outer-valgrind=(.*)$/) {
+ $outer_valgrind = $1;
+ } elsif ($arg =~ /^--outer-tool=(.*)$/) {
+ $outer_tool = $1;
+ } elsif ($arg =~ /^--outer-args=(.*)$/) {
+ $outer_args = $1;
} else {
die $usage;
}
@@ -256,6 +274,17 @@
my $cmd = "$timecmd $prog $args";
my $tNative = time_prog($cmd, $n_reps);
+ if (defined $outer_valgrind) {
+ $outer_valgrind = validate_program($tests_dir, $outer_valgrind, 1, 1);
+ foreach my $vgdir (@vgdirs) {
+ validate_program($vgdir, "./coregrind/valgrind", 1, 1);
+ }
+ } else {
+ foreach my $vgdir (@vgdirs) {
+ validate_program($vgdir, "./coregrind/valgrind", 1, 1);
+ }
+ }
+
foreach my $vgdir (@vgdirs) {
# Benchmark name
printf("%-8s ", $name);
@@ -272,20 +301,50 @@
# First two chars of toolname for abbreviation
my $tool_abbrev = $tool;
$tool_abbrev =~ s/(..).*/$1/;
-
- # Do the tool run(s). Set both VALGRIND_LIB and VALGRIND_LIB_INNER
- # in case this Valgrind was configured with --enable-inner. And
- # also VALGRINDLIB, which was the old name for the variable, to
- # allow comparison against old Valgrind versions (eg. 2.4.X).
printf(" %s:", $tool_abbrev);
- my $vgsetup = "VALGRINDLIB=$vgdir/.in_place "
- . "VALGRIND_LIB=$vgdir/.in_place "
- . "VALGRIND_LIB_INNER=$vgdir/.in_place ";
+ my $run_outer_args = "";
+ if (not defined $outer_args) {
+ $run_outer_args =
+ " -v --command-line-only=yes"
+ . " --run-libc-freeres=no --sim-hints=enable-outer"
+ . " --smc-check=all-non-file"
+ . " --vgdb=no --trace-children=yes --read-var-info=no"
+ . " --suppressions=../tests/outer_inner.supp"
+ . " --memcheck:leak-check=full --memcheck:show-reachable=no"
+ . " --cachegrind:cache-sim=yes --cachegrind:branch-sim=yes"
+ . " --cachegrind:cachegrind-out-file=cachegrind.out.$vgdirname.$tool_abbrev.$name.%p"
+ . " --callgrind:cache-sim=yes --callgrind:branch-sim=yes"
+ . " --callgrind:dump-instr=yes --callgrind:collect-jumps=yes"
+ . " --callgrind:callgrind-out-file=callgrind.out.$vgdirname.$tool_abbrev.$name.%p"
+ . " ";
+ } else {
+ $run_outer_args = $outer_args;
+ }
+
+ my $vgsetup = "";
my $vgcmd = "$vgdir/coregrind/valgrind "
. "--command-line-only=yes --tool=$tool -q "
. "--memcheck:leak-check=no "
. "--trace-children=yes "
. "$vgopts ";
+ # Do the tool run(s).
+ if (defined $outer_valgrind ) {
+ # in an outer-inner setup, only set VALGRIND_LIB_INNER
+ $vgsetup = "VALGRIND_LIB_INNER=$vgdir/.in_place ";
+ $vgcmd = "$outer_valgrind "
+ . "--tool=" . $outer_tool . " "
+ . "$run_outer_args "
+ . "--log-file=" . "$outer_tool.outer.log.$vgdirname.$tool_abbrev.$name.%p "
+ . $vgcmd;
+ } else {
+ # Set both VALGRIND_LIB and VALGRIND_LIB_INNER
+ # in case this Valgrind was configured with --enable-inner. And
+ # also VALGRINDLIB, which was the old name for the variable, to
+ # allow comparison against old Valgrind versions (eg. 2.4.X).
+ $vgsetup = "VALGRINDLIB=$vgdir/.in_place "
+ . "VALGRIND_LIB=$vgdir/.in_place "
+ . "VALGRIND_LIB_INNER=$vgdir/.in_place ";
+ }
my $cmd = "$vgsetup $timecmd $vgcmd $prog $args";
my $tTool = time_prog($cmd, $n_reps);
printf("%4.1fs (%4.1fx,", $tTool, $tTool/$tNative);
diff --git a/tests/vg_regtest.in b/tests/vg_regtest.in
index 9ea1d05..224385f 100755
--- a/tests/vg_regtest.in
+++ b/tests/vg_regtest.in
@@ -164,8 +164,6 @@
chomp(my $tests_dir = `pwd`);
# Outer valgrind to use, and args to use for it.
-# If this is set, --valgrind should be set to the installed inner valgrind,
-# and --valgrind-lib will be ignore
my $outer_valgrind;
my $outer_tool = "memcheck";
my $outer_args;
@@ -238,11 +236,12 @@
$valgrind = validate_program($tests_dir, $valgrind, 1, 0);
if (defined $outer_valgrind) {
- $outer_valgrind = validate_program($tests_dir, $outer_valgrind, 1, 0);
+ $outer_valgrind = validate_program($tests_dir, $outer_valgrind, 1, 1);
if (not defined $outer_args) {
$outer_args =
- " --command-line-only=yes"
+ " --command-line-only=yes"
. " --run-libc-freeres=no --sim-hints=enable-outer"
+ . " --smc-check=all-non-file"
. " --vgdb=no --trace-children=yes --read-var-info=no"
. " --suppressions="
. validate_program($tests_dir,"./tests/outer_inner.supp",1,0)
@@ -457,12 +456,12 @@
}
# Pass the appropriate --tool option for the directory (can be overridden
- # by an "args:" line, though). Set both VALGRIND_LIB and
- # VALGRIND_LIB_INNER in case this Valgrind was configured with
- # --enable-inner.
+ # by an "args:" line, though).
my $tool=determine_tool();
if (defined $outer_valgrind ) {
- mysystem("$outer_valgrind "
+ # in an outer-inner setup, only set VALGRIND_LIB_INNER
+ mysystem( "VALGRIND_LIB_INNER=$valgrind_lib "
+ . "$outer_valgrind "
. "--tool=" . $outer_tool . " "
. "$outer_args "
. "--log-file=" . "$name.outer.log "
@@ -471,7 +470,9 @@
. "--tool=$tool $extraopts $vgopts "
. "$prog $args > $name.stdout.out 2> $name.stderr.out");
} else {
- mysystem("VALGRIND_LIB=$valgrind_lib VALGRIND_LIB_INNER=$valgrind_lib "
+ # Set both VALGRIND_LIB and VALGRIND_LIB_INNER in case this Valgrind
+ # was configured with --enable-inner.
+ mysystem( "VALGRIND_LIB=$valgrind_lib VALGRIND_LIB_INNER=$valgrind_lib "
. "$valgrind --command-line-only=yes --memcheck:leak-check=no "
. "--tool=$tool $extraopts $vgopts "
. "$prog $args > $name.stdout.out 2> $name.stderr.out");