perf/README

=============================================================================
Notes about performance benchmarks
=============================================================================
For each benchmark, here is a brief description and notes about its
strengths and weaknesses.

-----------------------------------------------------------------------------
Artificial stress tests
-----------------------------------------------------------------------------
bigcode1, bigcode2:
- Description: Executes a lot of (nonsensical) code.
- Strengths:   Demonstrates the cost of translation which is a large part
               of runtime, particularly on larger programs.
- Weaknesses:  Highly artificial.

heap:
- Description: Does a lot of heap allocation and deallocation, and has a lot
               of heap blocks live while doing so.
- Strengths:   Stress test for an important sub-system; bug #105039 showed
               that inefficiencies in heap allocation can make a big
               difference to programs that allocate a lot.
- Weaknesses:  Highly artificial -- allocation pattern is not real, and only
               a few different size allocations are used.

sarp:
- Description: Does a lot of stack allocation and deallocation.
- Strengths:   Tests for a specific performance bug that existed in 3.1.0 and
               all earlier versions.
- Weaknesses:  Highly artificial.

-----------------------------------------------------------------------------
Real programs
-----------------------------------------------------------------------------
bz2:
- Description: Burrows-Wheeler compression and decompression.
- Strengths:   A real, widely used program, very similar to the 256.bzip2
               SPEC2000 benchmark.  Not dominated by any code, the hottest
               55 blocks account for only 90% of execution.  Has lots of
               short blocks and stresses the memory system hard.
- Weaknesses:  None, really, it's a good benchmark.

fbench:
- Description: Does some ray-tracing.
- Strengths:   Moderately realistic program.
- Weaknesses:  Dominated by sin and cos, which are not widely used, and are
               hardware-supported on x86 but not on other platforms such as
               PPC.

ffbench: 
- Description: Does a Fast Fourier Transform (FFT).
- Strengths:   Tests common FP ops (mostly adding and multiplying array
               elements), FFT is a very important operation.
- Weaknesses:  Dominated by the inner loop, which is quite long and flatters
               Valgrind due to the small dispatcher overhead.

tinycc:
- Description: A very small and fast C compiler.  A munged version of
               Fabrice Bellard's TinyCC compiling itself multiple times.
- Strengths:   A real program, lots of code (top 100 blocks only account for
               47% of execution), involves large irregular data structures
               (presumably, since it's a compiler).  Does lots of
               malloc/free calls and so changes that make a big improvement
               to perf/heap typically cause a small improvement.
- Weaknesses   None, really, it's a good benchmark.