1 # Benchmark Tools 2 3 ## compare_bench.py 4 5 The `compare_bench.py` utility which can be used to compare the result of benchmarks. 6 The program is invoked like: 7 8 ``` bash 9 $ compare_bench.py <old-benchmark> <new-benchmark> [benchmark options]... 10 ``` 11 12 Where `<old-benchmark>` and `<new-benchmark>` either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file. 13 14 The sample output using the JSON test files under `Inputs/` gives: 15 16 ``` bash 17 $ ./compare_bench.py ./gbench/Inputs/test1_run1.json ./gbench/Inputs/test1_run2.json 18 Comparing ./gbench/Inputs/test1_run1.json to ./gbench/Inputs/test1_run2.json 19 Benchmark Time CPU 20 ---------------------------------------------- 21 BM_SameTimes +0.00 +0.00 22 BM_2xFaster -0.50 -0.50 23 BM_2xSlower +1.00 +1.00 24 BM_10PercentFaster -0.10 -0.10 25 BM_10PercentSlower +0.10 +0.10 26 ``` 27 28 When a benchmark executable is run, the raw output from the benchmark is printed in real time to stdout. The sample output using `benchmark/basic_test` for both arguments looks like: 29 30 ``` 31 ./compare_bench.py test/basic_test test/basic_test --benchmark_filter=BM_empty.* 32 RUNNING: test/basic_test --benchmark_filter=BM_empty.* 33 Run on (4 X 4228.32 MHz CPU s) 34 2016-08-02 19:21:33 35 Benchmark Time CPU Iterations 36 -------------------------------------------------------------------- 37 BM_empty 9 ns 9 ns 79545455 38 BM_empty/threads:4 4 ns 9 ns 75268816 39 BM_empty_stop_start 8 ns 8 ns 83333333 40 BM_empty_stop_start/threads:4 3 ns 8 ns 83333332 41 RUNNING: test/basic_test --benchmark_filter=BM_empty.* 42 Run on (4 X 4228.32 MHz CPU s) 43 2016-08-02 19:21:35 44 Benchmark Time CPU Iterations 45 -------------------------------------------------------------------- 46 BM_empty 9 ns 9 ns 76086957 47 BM_empty/threads:4 4 ns 9 ns 76086956 48 BM_empty_stop_start 8 ns 8 ns 87500000 49 BM_empty_stop_start/threads:4 3 ns 8 ns 88607596 50 Comparing test/basic_test to test/basic_test 51 Benchmark Time CPU 52 --------------------------------------------------------- 53 BM_empty +0.00 +0.00 54 BM_empty/threads:4 +0.00 +0.00 55 BM_empty_stop_start +0.00 +0.00 56 BM_empty_stop_start/threads:4 +0.00 +0.00 57 ``` 58 59 Obviously this example doesn't give any useful output, but it's intended to show the output format when 'compare_bench.py' needs to run benchmarks. 60
