Tools/pybench/README

________________________________________________________________________

PYBENCH - A Python Benchmark Suite
________________________________________________________________________

     Extendable suite of of low-level benchmarks for measuring
          the performance of the Python implementation 
                 (interpreter, compiler or VM).

pybench is a collection of tests that provides a standardized way to
measure the performance of Python implementations. It takes a very
close look at different aspects of Python programs and let's you
decide which factors are more important to you than others, rather
than wrapping everything up in one number, like the other performance
tests do (e.g. pystone which is included in the Python Standard
Library).

pybench has been used in the past by several Python developers to
track down performance bottlenecks or to demonstrate the impact of
optimizations and new features in Python.

The command line interface for pybench is the file pybench.py. Run
this script with option '--help' to get a listing of the possible
options. Without options, pybench will simply execute the benchmark
and then print out a report to stdout.


Micro-Manual
------------

Run 'pybench.py -h' to see the help screen.
Run 'pybench.py' to just let the benchmark suite do it's thing and
'pybench.py -f <file>' to have it store the results in a file too.

This is the current output of pybench.py --help:

Synopsis:
 pybench.py [option] files...

Options and default settings:
  -n arg           number of rounds (10)
  -f arg           save benchmark to file arg ()
  -c arg           compare benchmark with the one in file arg ()
  -s arg           show benchmark in file arg, then exit ()
  -S               show statistics of benchmarks (0)
  -w arg           set warp factor to arg (20)
  -d               hide noise in compares (0)
  --no-gc          disable garbage collection (0)
  --no-syscheck    "disable" sys check interval (set to sys.maxint) (0)
  -t arg           tests containing substring ()
  -C arg           number of calibration runs (20)
  -v               generate verbose output
  -h               show this help text
  --help           show this help text
  --debug          enable debugging
  --copyright      show copyright
  --examples       show examples of usage

Version:
 1.3

The normal operation is to run the suite and display the
results. Use -f to save them for later reuse or comparisms.

Examples:

python1.5 pybench.py -w 100 -f p15
python1.4 pybench.py -w 100 -f p14
python pybench.py -s p15 -c p14


License
-------

See LICENSE file.


Sample output
-------------

PYBENCH 1.3

Machine Details:
   Platform ID:  Linux-2.6.8-24.19-default-x86_64-with-SuSE-9.2-x86-64
   Executable:   /home/lemburg/projects/Python/Installation/bin/python
   Python:       2.5a1.0
   Compiler:     GCC 3.3.4 (pre 3.3.5 20040809)
   Build:        Apr  9 2006 01:50:57 (#trunk)

Searching for tests...
   BuiltinFunctionCalls
   BuiltinMethodLookup
   CompareFloats
   CompareFloatsIntegers
   CompareIntegers
   CompareInternedStrings
   CompareLongs
   CompareStrings
   CompareUnicode
   ConcatStrings
   ConcatUnicode
   CreateInstances
   CreateStringsWithConcat
   CreateUnicodeWithConcat
   DictCreation
   DictWithFloatKeys
   DictWithIntegerKeys
   DictWithStringKeys
   ForLoops
   IfThenElse
   ListSlicing
   NestedForLoops
   NormalClassAttribute
   NormalInstanceAttribute
   PythonFunctionCalls
   PythonMethodCalls
   Recursion
   SecondImport
   SecondPackageImport
   SecondSubmoduleImport
   SimpleComplexArithmetic
   SimpleDictManipulation
   SimpleFloatArithmetic
   SimpleIntFloatArithmetic
   SimpleIntegerArithmetic
   SimpleListManipulation
   SimpleLongArithmetic
   SmallLists
   SmallTuples
   SpecialClassAttribute
   SpecialInstanceAttribute
   StringMappings
   StringPredicates
   StringSlicing
   TryExcept
   TryRaiseExcept
   TupleSlicing
   UnicodeMappings
   UnicodePredicates
   UnicodeProperties
   UnicodeSlicing

Running 10 round(s) of the suite:

...

 Round 10                         real   abs    overhead
          BuiltinFunctionCalls:   0.030r 0.030a 0.000o
           BuiltinMethodLookup:   0.059r 0.060a 0.001o
                 CompareFloats:   0.050r 0.050a 0.000o
         CompareFloatsIntegers:   0.050r 0.050a 0.000o
               CompareIntegers:   0.070r 0.070a 0.000o
        CompareInternedStrings:   0.039r 0.040a 0.001o
                  CompareLongs:   0.050r 0.050a 0.000o
                CompareStrings:   0.060r 0.060a 0.000o
                CompareUnicode:   0.060r 0.060a 0.000o
                 ConcatStrings:   0.040r 0.040a 0.000o
                 ConcatUnicode:   0.050r 0.050a 0.000o
               CreateInstances:   0.050r 0.050a 0.000o
       CreateStringsWithConcat:   0.029r 0.030a 0.001o
       CreateUnicodeWithConcat:   0.060r 0.060a 0.000o
                  DictCreation:   0.040r 0.040a 0.000o
             DictWithFloatKeys:   0.089r 0.090a 0.000o
           DictWithIntegerKeys:   0.059r 0.060a 0.001o
            DictWithStringKeys:   0.070r 0.070a 0.001o
                      ForLoops:   0.050r 0.050a 0.000o
                    IfThenElse:   0.070r 0.070a 0.000o
                   ListSlicing:   0.030r 0.030a 0.000o
                NestedForLoops:   0.030r 0.030a 0.000o
          NormalClassAttribute:   0.060r 0.060a 0.000o
       NormalInstanceAttribute:   0.060r 0.060a 0.000o
           PythonFunctionCalls:   0.060r 0.060a 0.000o
             PythonMethodCalls:   0.050r 0.050a 0.000o
                     Recursion:   0.050r 0.050a 0.000o
                  SecondImport:   0.030r 0.030a 0.000o
           SecondPackageImport:   0.030r 0.030a 0.000o
         SecondSubmoduleImport:   0.040r 0.040a 0.000o
       SimpleComplexArithmetic:   0.030r 0.030a 0.000o
        SimpleDictManipulation:   0.040r 0.040a 0.000o
         SimpleFloatArithmetic:   0.050r 0.050a 0.001o
      SimpleIntFloatArithmetic:   0.060r 0.060a 0.000o
       SimpleIntegerArithmetic:   0.060r 0.060a 0.000o
        SimpleListManipulation:   0.030r 0.030a 0.000o
          SimpleLongArithmetic:   0.030r 0.030a 0.000o
                    SmallLists:   0.050r 0.050a 0.000o
                   SmallTuples:   0.050r 0.050a 0.000o
         SpecialClassAttribute:   0.060r 0.060a 0.000o
      SpecialInstanceAttribute:   0.079r 0.080a 0.001o
                StringMappings:   0.060r 0.060a 0.000o
              StringPredicates:   0.049r 0.050a 0.001o
                 StringSlicing:   0.039r 0.040a 0.000o
                     TryExcept:   0.079r 0.080a 0.001o
                TryRaiseExcept:   0.059r 0.060a 0.001o
                  TupleSlicing:   0.050r 0.050a 0.000o
               UnicodeMappings:   0.070r 0.070a 0.001o
             UnicodePredicates:   0.059r 0.060a 0.001o
             UnicodeProperties:   0.059r 0.060a 0.001o
                UnicodeSlicing:   0.050r 0.050a 0.000o
                                 ----------------------
            Average round time:      2.937 seconds


Tests:                              per run    per oper.   overhead
------------------------------------------------------------------------
          BuiltinFunctionCalls:      29.85 ms    0.23 us    0.00 ms
           BuiltinMethodLookup:      66.85 ms    0.13 us    0.50 ms
                 CompareFloats:      43.00 ms    0.10 us    0.00 ms
         CompareFloatsIntegers:      51.80 ms    0.12 us    0.00 ms
               CompareIntegers:      70.70 ms    0.08 us    0.50 ms
        CompareInternedStrings:      41.40 ms    0.08 us    0.50 ms
                  CompareLongs:      47.90 ms    0.11 us    0.00 ms
                CompareStrings:      58.50 ms    0.12 us    0.50 ms
                CompareUnicode:      56.55 ms    0.15 us    0.50 ms
                 ConcatStrings:      44.75 ms    0.30 us    0.00 ms
                 ConcatUnicode:      54.55 ms    0.36 us    0.50 ms
               CreateInstances:      50.95 ms    1.21 us    0.00 ms
       CreateStringsWithConcat:      28.85 ms    0.14 us    0.50 ms
       CreateUnicodeWithConcat:      53.75 ms    0.27 us    0.00 ms
                  DictCreation:      41.90 ms    0.28 us    0.00 ms
             DictWithFloatKeys:      88.50 ms    0.15 us    0.50 ms
           DictWithIntegerKeys:      62.55 ms    0.10 us    0.50 ms
            DictWithStringKeys:      60.50 ms    0.10 us    0.50 ms
                      ForLoops:      46.90 ms    4.69 us    0.00 ms
                    IfThenElse:      60.55 ms    0.09 us    0.00 ms
                   ListSlicing:      29.90 ms    8.54 us    0.00 ms
                NestedForLoops:      33.95 ms    0.10 us    0.00 ms
          NormalClassAttribute:      62.75 ms    0.10 us    0.50 ms
       NormalInstanceAttribute:      61.80 ms    0.10 us    0.50 ms
           PythonFunctionCalls:      60.00 ms    0.36 us    0.00 ms
             PythonMethodCalls:      50.00 ms    0.67 us    0.00 ms
                     Recursion:      46.85 ms    3.75 us    0.00 ms
                  SecondImport:      35.00 ms    1.40 us    0.00 ms
           SecondPackageImport:      32.00 ms    1.28 us    0.00 ms
         SecondSubmoduleImport:      38.00 ms    1.52 us    0.00 ms
       SimpleComplexArithmetic:      26.85 ms    0.12 us    0.00 ms
        SimpleDictManipulation:      40.85 ms    0.14 us    0.00 ms
         SimpleFloatArithmetic:      48.70 ms    0.09 us    0.50 ms
      SimpleIntFloatArithmetic:      57.70 ms    0.09 us    0.00 ms
       SimpleIntegerArithmetic:      58.75 ms    0.09 us    0.50 ms
        SimpleListManipulation:      34.80 ms    0.13 us    0.00 ms
          SimpleLongArithmetic:      30.95 ms    0.19 us    0.50 ms
                    SmallLists:      47.60 ms    0.19 us    0.00 ms
                   SmallTuples:      48.80 ms    0.20 us    0.50 ms
         SpecialClassAttribute:      61.70 ms    0.10 us    0.00 ms
      SpecialInstanceAttribute:      76.70 ms    0.13 us    0.50 ms
                StringMappings:      58.70 ms    0.47 us    0.00 ms
              StringPredicates:      50.00 ms    0.18 us    1.00 ms
                 StringSlicing:      39.65 ms    0.23 us    0.50 ms
                     TryExcept:      84.45 ms    0.06 us    0.50 ms
                TryRaiseExcept:      61.75 ms    4.12 us    0.50 ms
                  TupleSlicing:      48.95 ms    0.47 us    0.00 ms
               UnicodeMappings:      71.50 ms    3.97 us    0.50 ms
             UnicodePredicates:      52.75 ms    0.23 us    1.00 ms
             UnicodeProperties:      61.90 ms    0.31 us    1.00 ms
                UnicodeSlicing:      53.75 ms    0.31 us    0.50 ms
------------------------------------------------------------------------
            Average round time:    2937.00 ms

________________________________________________________________________

Writing New Tests
________________________________________________________________________

pybench tests are simple modules defining one or more pybench.Test
subclasses.

Writing a test essentially boils down to providing two methods:
.test() which runs .rounds number of .operations test operations each
and .calibrate() which does the same except that it doesn't actually
execute the operations.


Here's an example:
------------------

from pybench import Test

class IntegerCounting(Test):

    # Version number of the test as float (x.yy); this is important
    # for comparisons of benchmark runs - tests with unequal version
    # number will not get compared.
    version = 1.0
    
    # The number of abstract operations done in each round of the
    # test. An operation is the basic unit of what you want to
    # measure. The benchmark will output the amount of run-time per
    # operation. Note that in order to raise the measured timings
    # significantly above noise level, it is often required to repeat
    # sets of operations more than once per test round. The measured
    # overhead per test round should be less than 1 second.
    operations = 20

    # Number of rounds to execute per test run. This should be
    # adjusted to a figure that results in a test run-time of between
    # 20-50 seconds.
    rounds = 100000

    def test(self):

	""" Run the test.

	    The test needs to run self.rounds executing
	    self.operations number of operations each.

        """
        # Init the test
        a = 1

        # Run test rounds
	#
        # NOTE: Use xrange() for all test loops unless you want to face
	# a 20MB process !
	#
        for i in xrange(self.rounds):

            # Repeat the operations per round to raise the run-time
            # per operation significantly above the noise level of the
            # for-loop overhead. 

	    # Execute 20 operations (a += 1):
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1
            a += 1

    def calibrate(self):

	""" Calibrate the test.

	    This method should execute everything that is needed to
	    setup and run the test - except for the actual operations
	    that you intend to measure. pybench uses this method to
            measure the test implementation overhead.

        """
        # Init the test
        a = 1

        # Run test rounds (without actually doing any operation)
        for i in xrange(self.rounds):

	    # Skip the actual execution of the operations, since we
	    # only want to measure the test's administration overhead.
            pass

Registering a new test module
-----------------------------

To register a test module with pybench, the classes need to be
imported into the pybench.Setup module. pybench will then scan all the
symbols defined in that module for subclasses of pybench.Test and
automatically add them to the benchmark suite.


Breaking Comparability
----------------------

If a change is made to any individual test that means it is no
longer strictly comparable with previous runs, the '.version' class
variable should be updated. Therefafter, comparisons with previous
versions of the test will list as "n/a" to reflect the change.

Have fun,
--
Marc-Andre Lemburg
mal@lemburg.com