1 2 ## Fruit benchmarks 3 4 Fruit contains some code to benchmark various metrics (e.g. performance, compile time, executable size) in an automated 5 way. 6 7 ### Benchmark suites 8 9 The `suites` folder contains suites of Fruit (or Fruit-related) benchmarks that can be run using `run_benchmarks.py`. 10 For example: 11 12 ```bash 13 $ ~/projects/fruit/extras/benchmark/run_benchmarks.py \ 14 --continue-benchmark=true \ 15 --benchmark-definition ~/projects/fruit/extras/benchmark/suites/fruit_full.yml 16 --output-file ~/fruit_bench_results.txt \ 17 --fruit-sources-dir ~/projects/fruit \ 18 --fruit-benchmark-sources-dir ~/projects/fruit \ 19 --boost-di-sources-dir ~/projects/boost-di 20 ``` 21 22 Once the benchmark run completes, you can format the results using some pre-defined tables, see the section below. 23 24 The following benchmark suites are defined: 25 26 * `fruit_full.yml`: full set of Fruit benchmarks (using the Fruit 3.x API). 27 * `fruit_mostly_full.yml`: a subset of the tests in `fruit_full.yml`. 28 * `fruit_quick.yml`: this is an even smaller subset, and the number of runs is capped at 10 so 29 the confidence intervals might be wider. It's useful as a quicker (around 10-15min) way to get a rough idea of the 30 performance (e.g. to evaluate the performance impact of a commit, during development). 31 * `fruit_single.yml`: runs the Fruit runtime benchs under a single compiler and with just 1 combination of flags. This 32 also caps the number of runs at 8, so the resulting confidence intervals might be wider than they would be with 33 `fruit_full.yml`. This is a quick benchmark that can used during development of performance optimizations. 34 * `fruit_debug.yml`: a suite used to debug Fruit's benchmarking code. This is very quick, but the actual results are 35 not meaningful. Run this after changing any benchmarking code, to check that it still works. 36 * `boost_di`: unlike the others, this benchmark suite exercises the Boost.DI library (still in boost-experimental at the 37 time of writing) instead of Fruit. 38 39 ### Benchmark tables 40 41 The `tables` folder contains some table definitions that can be used to get a human-readable representations of 42 benchmark results generated using `run_benchmarks.py`. 43 44 Note that there *isn't* a 1:1 mapping between benchmark suites and benchmark tables; the same table definition can be 45 used with multiple benchmark suites (for example, a full suite and a quick variant that only has a subset of the 46 dimensions) and multiple table definitions might be appropriate to display the results of a single suite (for example, 47 there could be a table definition that displays only metrics meaningful to Fruit users and one that also displays 48 more fine-grained metrics that are only meaningful to Fruit developers). 49 50 Example usage of `format_bench_results.py` with one of these table definitions: 51 52 ```bash 53 $ ~/projects/fruit/extras/benchmark/format_bench_results.py \ 54 --benchmark-results ~/fruit_bench_results.txt \ 55 --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml 56 ``` 57 58 It's also possible to compare two benchmark results (for example, when running the same benchmarks before and after 59 a Fruit commit): 60 61 ```bash 62 $ ~/projects/fruit/extras/benchmark/format_bench_results.py \ 63 --benchmark-results ~/fruit_bench_results_after.txt \ 64 --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml \ 65 --baseline-benchmark-results ~/fruit_bench_results_before.txt 66 ``` 67 68 The following tables are defined: 69 70 * `fruit_wiki.yml`: the "main" table definition, with the tables that are in Fruit's wiki. 71 * `fruit_internal.yml`: a more detailed version of `fruit_wiki.yml`, also displaying metrics that are only meaningful 72 to Fruit developers (e.g. splitting the setup time into component creation time and normalization time). 73 74 ### Manual benchmarks 75 76 In some cases, you might want to run the benchmarks manually (e.g. if you want to use `callgrind` to profile the 77 benchmark run). This is how you can do that: 78 79 ```bash 80 $ cd ~/projects/fruit 81 $ mkdir build 82 $ cd build 83 $ CXX=g++-6 cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo 84 $ make -j 10 85 $ cd .. 86 $ mkdir generated-benchs 87 $ extras/benchmark/generate_benchmark.py \ 88 --compiler g++-6 \ 89 --fruit-sources-dir ~/projects/fruit \ 90 --fruit-build-dir ~/projects/fruit/build \ 91 --num-components-with-no-deps 10 \ 92 --num-components-with-deps 90 \ 93 --num-deps 10 \ 94 --output-dir generated-benchs \ 95 --generate-debuginfo=true 96 $ cd generated-benchs 97 $ make -j 10 98 $ valgrind \ 99 --tool=callgrind \ 100 --simulate-cache=yes \ 101 --dump-instr=yes \ 102 ./main 10000 103 ``` 104
