| ________________________________________________________________________ |
| |
| PYBENCH - A Python Benchmark Suite |
| ________________________________________________________________________ |
| |
| Extendable suite of of low-level benchmarks for measuring |
| the performance of the Python implementation |
| (interpreter, compiler or VM). |
| |
| pybench is a collection of tests that provides a standardized way to |
| measure the performance of Python implementations. It takes a very |
| close look at different aspects of Python programs and let's you |
| decide which factors are more important to you than others, rather |
| than wrapping everything up in one number, like the other performance |
| tests do (e.g. pystone which is included in the Python Standard |
| Library). |
| |
| pybench has been used in the past by several Python developers to |
| track down performance bottlenecks or to demonstrate the impact of |
| optimizations and new features in Python. |
| |
| The command line interface for pybench is the file pybench.py. Run |
| this script with option '--help' to get a listing of the possible |
| options. Without options, pybench will simply execute the benchmark |
| and then print out a report to stdout. |
| |
| |
| Micro-Manual |
| ------------ |
| |
| Run 'pybench.py -h' to see the help screen. Run 'pybench.py' to run |
| the benchmark suite using default settings and 'pybench.py -f <file>' |
| to have it store the results in a file too. |
| |
| It is usually a good idea to run pybench.py multiple times to see |
| whether the environment, timers and benchmark run-times are suitable |
| for doing benchmark tests. |
| |
| You can use the comparison feature of pybench.py ('pybench.py -c |
| <file>') to check how well the system behaves in comparison to a |
| reference run. |
| |
| If the differences are well below 10% for each test, then you have a |
| system that is good for doing benchmark testings. Of you get random |
| differences of more than 10% or significant differences between the |
| values for minimum and average time, then you likely have some |
| background processes running which cause the readings to become |
| inconsistent. Examples include: web-browsers, email clients, RSS |
| readers, music players, backup programs, etc. |
| |
| If you are only interested in a few tests of the whole suite, you can |
| use the filtering option, e.g. 'pybench.py -t string' will only |
| run/show the tests that have 'string' in their name. |
| |
| This is the current output of pybench.py --help: |
| |
| """ |
| ------------------------------------------------------------------------ |
| PYBENCH - a benchmark test suite for Python interpreters/compilers. |
| ------------------------------------------------------------------------ |
| |
| Synopsis: |
| pybench.py [option] files... |
| |
| Options and default settings: |
| -n arg number of rounds (10) |
| -f arg save benchmark to file arg () |
| -c arg compare benchmark with the one in file arg () |
| -s arg show benchmark in file arg, then exit () |
| -w arg set warp factor to arg (10) |
| -t arg run only tests with names matching arg () |
| -C arg set the number of calibration runs to arg (20) |
| -d hide noise in comparisons (0) |
| -v verbose output (not recommended) (0) |
| --with-gc enable garbage collection (0) |
| --with-syscheck use default sys check interval (0) |
| --timer arg use given timer (time.time) |
| -h show this help text |
| --help show this help text |
| --debug enable debugging |
| --copyright show copyright |
| --examples show examples of usage |
| |
| Version: |
| 2.0 |
| |
| The normal operation is to run the suite and display the |
| results. Use -f to save them for later reuse or comparisons. |
| |
| Available timers: |
| |
| time.time |
| time.clock |
| systimes.processtime |
| |
| Examples: |
| |
| python2.1 pybench.py -f p21.pybench |
| python2.5 pybench.py -f p25.pybench |
| python pybench.py -s p25.pybench -c p21.pybench |
| """ |
| |
| License |
| ------- |
| |
| See LICENSE file. |
| |
| |
| Sample output |
| ------------- |
| |
| """ |
| ------------------------------------------------------------------------------- |
| PYBENCH 2.0 |
| ------------------------------------------------------------------------------- |
| * using Python 2.4.2 |
| * disabled garbage collection |
| * system check interval set to maximum: 2147483647 |
| * using timer: time.time |
| |
| Calibrating tests. Please wait... |
| |
| Running 10 round(s) of the suite at warp factor 10: |
| |
| * Round 1 done in 6.388 seconds. |
| * Round 2 done in 6.485 seconds. |
| * Round 3 done in 6.786 seconds. |
| ... |
| * Round 10 done in 6.546 seconds. |
| |
| ------------------------------------------------------------------------------- |
| Benchmark: 2006-06-12 12:09:25 |
| ------------------------------------------------------------------------------- |
| |
| Rounds: 10 |
| Warp: 10 |
| Timer: time.time |
| |
| Machine Details: |
| Platform ID: Linux-2.6.8-24.19-default-x86_64-with-SuSE-9.2-x86-64 |
| Processor: x86_64 |
| |
| Python: |
| Executable: /usr/local/bin/python |
| Version: 2.4.2 |
| Compiler: GCC 3.3.4 (pre 3.3.5 20040809) |
| Bits: 64bit |
| Build: Oct 1 2005 15:24:35 (#1) |
| Unicode: UCS2 |
| |
| |
| Test minimum average operation overhead |
| ------------------------------------------------------------------------------- |
| BuiltinFunctionCalls: 126ms 145ms 0.28us 0.274ms |
| BuiltinMethodLookup: 124ms 130ms 0.12us 0.316ms |
| CompareFloats: 109ms 110ms 0.09us 0.361ms |
| CompareFloatsIntegers: 100ms 104ms 0.12us 0.271ms |
| CompareIntegers: 137ms 138ms 0.08us 0.542ms |
| CompareInternedStrings: 124ms 127ms 0.08us 1.367ms |
| CompareLongs: 100ms 104ms 0.10us 0.316ms |
| CompareStrings: 111ms 115ms 0.12us 0.929ms |
| CompareUnicode: 108ms 128ms 0.17us 0.693ms |
| ConcatStrings: 142ms 155ms 0.31us 0.562ms |
| ConcatUnicode: 119ms 127ms 0.42us 0.384ms |
| CreateInstances: 123ms 128ms 1.14us 0.367ms |
| CreateNewInstances: 121ms 126ms 1.49us 0.335ms |
| CreateStringsWithConcat: 130ms 135ms 0.14us 0.916ms |
| CreateUnicodeWithConcat: 130ms 135ms 0.34us 0.361ms |
| DictCreation: 108ms 109ms 0.27us 0.361ms |
| DictWithFloatKeys: 149ms 153ms 0.17us 0.678ms |
| DictWithIntegerKeys: 124ms 126ms 0.11us 0.915ms |
| DictWithStringKeys: 114ms 117ms 0.10us 0.905ms |
| ForLoops: 110ms 111ms 4.46us 0.063ms |
| IfThenElse: 118ms 119ms 0.09us 0.685ms |
| ListSlicing: 116ms 120ms 8.59us 0.103ms |
| NestedForLoops: 125ms 137ms 0.09us 0.019ms |
| NormalClassAttribute: 124ms 136ms 0.11us 0.457ms |
| NormalInstanceAttribute: 110ms 117ms 0.10us 0.454ms |
| PythonFunctionCalls: 107ms 113ms 0.34us 0.271ms |
| PythonMethodCalls: 140ms 149ms 0.66us 0.141ms |
| Recursion: 156ms 166ms 3.32us 0.452ms |
| SecondImport: 112ms 118ms 1.18us 0.180ms |
| SecondPackageImport: 118ms 127ms 1.27us 0.180ms |
| SecondSubmoduleImport: 140ms 151ms 1.51us 0.180ms |
| SimpleComplexArithmetic: 128ms 139ms 0.16us 0.361ms |
| SimpleDictManipulation: 134ms 136ms 0.11us 0.452ms |
| SimpleFloatArithmetic: 110ms 113ms 0.09us 0.571ms |
| SimpleIntFloatArithmetic: 106ms 111ms 0.08us 0.548ms |
| SimpleIntegerArithmetic: 106ms 109ms 0.08us 0.544ms |
| SimpleListManipulation: 103ms 113ms 0.10us 0.587ms |
| SimpleLongArithmetic: 112ms 118ms 0.18us 0.271ms |
| SmallLists: 105ms 116ms 0.17us 0.366ms |
| SmallTuples: 108ms 128ms 0.24us 0.406ms |
| SpecialClassAttribute: 119ms 136ms 0.11us 0.453ms |
| SpecialInstanceAttribute: 143ms 155ms 0.13us 0.454ms |
| StringMappings: 115ms 121ms 0.48us 0.405ms |
| StringPredicates: 120ms 129ms 0.18us 2.064ms |
| StringSlicing: 111ms 127ms 0.23us 0.781ms |
| TryExcept: 125ms 126ms 0.06us 0.681ms |
| TryRaiseExcept: 133ms 137ms 2.14us 0.361ms |
| TupleSlicing: 117ms 120ms 0.46us 0.066ms |
| UnicodeMappings: 156ms 160ms 4.44us 0.429ms |
| UnicodePredicates: 117ms 121ms 0.22us 2.487ms |
| UnicodeProperties: 115ms 153ms 0.38us 2.070ms |
| UnicodeSlicing: 126ms 129ms 0.26us 0.689ms |
| ------------------------------------------------------------------------------- |
| Totals: 6283ms 6673ms |
| """ |
| ________________________________________________________________________ |
| |
| Writing New Tests |
| ________________________________________________________________________ |
| |
| pybench tests are simple modules defining one or more pybench.Test |
| subclasses. |
| |
| Writing a test essentially boils down to providing two methods: |
| .test() which runs .rounds number of .operations test operations each |
| and .calibrate() which does the same except that it doesn't actually |
| execute the operations. |
| |
| |
| Here's an example: |
| ------------------ |
| |
| from pybench import Test |
| |
| class IntegerCounting(Test): |
| |
| # Version number of the test as float (x.yy); this is important |
| # for comparisons of benchmark runs - tests with unequal version |
| # number will not get compared. |
| version = 1.0 |
| |
| # The number of abstract operations done in each round of the |
| # test. An operation is the basic unit of what you want to |
| # measure. The benchmark will output the amount of run-time per |
| # operation. Note that in order to raise the measured timings |
| # significantly above noise level, it is often required to repeat |
| # sets of operations more than once per test round. The measured |
| # overhead per test round should be less than 1 second. |
| operations = 20 |
| |
| # Number of rounds to execute per test run. This should be |
| # adjusted to a figure that results in a test run-time of between |
| # 1-2 seconds (at warp 1). |
| rounds = 100000 |
| |
| def test(self): |
| |
| """ Run the test. |
| |
| The test needs to run self.rounds executing |
| self.operations number of operations each. |
| |
| """ |
| # Init the test |
| a = 1 |
| |
| # Run test rounds |
| # |
| # NOTE: Use xrange() for all test loops unless you want to face |
| # a 20MB process ! |
| # |
| for i in xrange(self.rounds): |
| |
| # Repeat the operations per round to raise the run-time |
| # per operation significantly above the noise level of the |
| # for-loop overhead. |
| |
| # Execute 20 operations (a += 1): |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| a += 1 |
| |
| def calibrate(self): |
| |
| """ Calibrate the test. |
| |
| This method should execute everything that is needed to |
| setup and run the test - except for the actual operations |
| that you intend to measure. pybench uses this method to |
| measure the test implementation overhead. |
| |
| """ |
| # Init the test |
| a = 1 |
| |
| # Run test rounds (without actually doing any operation) |
| for i in xrange(self.rounds): |
| |
| # Skip the actual execution of the operations, since we |
| # only want to measure the test's administration overhead. |
| pass |
| |
| Registering a new test module |
| ----------------------------- |
| |
| To register a test module with pybench, the classes need to be |
| imported into the pybench.Setup module. pybench will then scan all the |
| symbols defined in that module for subclasses of pybench.Test and |
| automatically add them to the benchmark suite. |
| |
| |
| Breaking Comparability |
| ---------------------- |
| |
| If a change is made to any individual test that means it is no |
| longer strictly comparable with previous runs, the '.version' class |
| variable should be updated. Therefafter, comparisons with previous |
| versions of the test will list as "n/a" to reflect the change. |
| |
| |
| Version History |
| --------------- |
| |
| 2.0: rewrote parts of pybench which resulted in more repeatable |
| timings: |
| - made timer a parameter |
| - changed the platform default timer to use high-resolution |
| timers rather than process timers (which have a much lower |
| resolution) |
| - added option to select timer |
| - added process time timer (using systimes.py) |
| - changed to use min() as timing estimator (average |
| is still taken as well to provide an idea of the difference) |
| - garbage collection is turned off per default |
| - sys check interval is set to the highest possible value |
| - calibration is now a separate step and done using |
| a different strategy that allows measuring the test |
| overhead more accurately |
| - modified the tests to each give a run-time of between |
| 100-200ms using warp 10 |
| - changed default warp factor to 10 (from 20) |
| - compared results with timeit.py and confirmed measurements |
| - bumped all test versions to 2.0 |
| - updated platform.py to the latest version |
| - changed the output format a bit to make it look |
| nicer |
| - refactored the APIs somewhat |
| 1.3+: Steve Holden added the NewInstances test and the filtering |
| option during the NeedForSpeed sprint; this also triggered a long |
| discussion on how to improve benchmark timing and finally |
| resulted in the release of 2.0 |
| 1.3: initial checkin into the Python SVN repository |
| |
| |
| Have fun, |
| -- |
| Marc-Andre Lemburg |
| mal@lemburg.com |