1 { 2 "cells": [ 3 { 4 "cell_type": "markdown", 5 "metadata": {}, 6 "source": [ 7 "# Executor API - Executor\n", 8 "\n", 9 " A tests executor is a module which supports the execution of a configured set of experiments.<br><br>\n", 10 " Each experiment is composed by:\n", 11 " - a target configuration\n", 12 " - a workload to execute\n", 13 "\n", 14 "The executor module can be configured to run a set of workloads (wloads) in each different target configuration of a specified set (confs). These wloads and confs can be specified by the \"experiments_conf\" input dictionary which is described below at **Experiments Configuration**.<br><br>\n", 15 "All the results generated by each experiment will be collected in a results folder. The format and content fo the results forlder is detailed in the last cell of **Tests execution** below.\n" 16 ] 17 }, 18 { 19 "cell_type": "code", 20 "execution_count": 1, 21 "metadata": { 22 "collapsed": false 23 }, 24 "outputs": [ 25 { 26 "name": "stderr", 27 "output_type": "stream", 28 "text": [ 29 "2016-12-08 11:58:22,693 INFO : root : Using LISA logging configuration:\n", 30 "2016-12-08 11:58:22,694 INFO : root : /home/vagrant/lisa/logging.conf\n" 31 ] 32 } 33 ], 34 "source": [ 35 "import logging\n", 36 "\n", 37 "from conf import LisaLogging\n", 38 "LisaLogging.setup()" 39 ] 40 }, 41 { 42 "cell_type": "code", 43 "execution_count": 2, 44 "metadata": { 45 "collapsed": false 46 }, 47 "outputs": [], 48 "source": [ 49 "import os\n", 50 "import json\n", 51 "\n", 52 "from env import TestEnv\n", 53 "from executor import Executor" 54 ] 55 }, 56 { 57 "cell_type": "markdown", 58 "metadata": {}, 59 "source": [ 60 "## Target Configuration\n", 61 "The target configuration it's used to describe and configure your test environment.\n", 62 "You can find more details in **examples/utils/testenv_example.ipynb**." 63 ] 64 }, 65 { 66 "cell_type": "code", 67 "execution_count": 3, 68 "metadata": { 69 "collapsed": false 70 }, 71 "outputs": [], 72 "source": [ 73 "# Setup a target configuration\n", 74 "my_target_conf = {\n", 75 " \n", 76 " # Target platform and board\n", 77 " \"platform\" : 'linux',\n", 78 " \"board\" : 'juno',\n", 79 " \n", 80 " # Target board IP/MAC address\n", 81 " \"host\" : '192.168.0.1',\n", 82 " \n", 83 " # Login credentials\n", 84 " \"username\" : 'root',\n", 85 " \"password\" : 'juno',\n", 86 "\n", 87 "}" 88 ] 89 }, 90 { 91 "cell_type": "markdown", 92 "metadata": {}, 93 "source": [ 94 "## Experiments Configuration\n", 95 "\n", 96 "The experiments configuration defines the software setups that we need on our hardware target.<br>\n", 97 "This can be given as an argument to an Executor instance or to a TestEnv one.<br> <br>\n", 98 "Elements of the experiments configuration:\n", 99 " - **confs**: **mandatory** platform configurations to be tested.\n", 100 " - tag: relevant string to identify your configuration.\n", 101 " - flags: ftrace (to enable ftrace events) is the only one supported at the moment.\n", 102 " - sched_features: features to be added to /sys/kernel/debug/sched_features.\n", 103 " - cpufreq: CpuFreq governor and tunables.\n", 104 " - cgroups: CGroups configuration (controller). The default CGroup will be used otherwise.\n", 105 " - **wloads**: **mandatory** workloads to run on each platform configuration.\n", 106 " - **iterations**: number of iterations for each workload.\n", 107 " - **tools**: binary tools (available under ./tools/$ARCH/) to install by default; these will be merged with the ones in the target configuration.\n", 108 " - **ftrace**: FTrace events to collect for all the experiments configurations which have the \"ftrace\" flag enabled.\n", 109 " - **modules**: modules required by the experiments resulted from the experiments configurations.\n", 110 " - **exclude_modules** - modules to be disabled.\n", 111 " - **results_dir**: results directory - experiments configuration results directory overrides target one." 112 ] 113 }, 114 { 115 "cell_type": "code", 116 "execution_count": 4, 117 "metadata": { 118 "collapsed": false, 119 "scrolled": false 120 }, 121 "outputs": [], 122 "source": [ 123 "my_experiments_conf = {\n", 124 "\n", 125 " # Folder where all the results will be collected\n", 126 " \"results_dir\" : \"ExecutorExample\",\n", 127 "\n", 128 " # Platform configurations to test: you can specify any number of configurations\n", 129 " \"confs\" : [\n", 130 " { \n", 131 " \"tag\" : \"base\", # Relevant string to identify configuration\n", 132 " \"flags\" : [\"ftrace\", \"freeze_userspace\"], # Enable FTrace events, freeze userspace while running \n", 133 " \"sched_features\" : \"NO_ENERGY_AWARE\", # Disable EAS\n", 134 " \"cpufreq\" : { # Use PERFORMANCE CpuFreq\n", 135 " \"governor\" : \"performance\",\n", 136 " },\n", 137 " },\n", 138 " {\n", 139 " \"tag\" : \"eas\", # Relevant string to identify configuration\n", 140 " \"flags\" : [\"ftrace\", \"freeze_userspace\"], # Enable FTrace events, freeze userspace while running\n", 141 " \"sched_features\" : \"ENERGY_AWARE\", # Enable EAS\n", 142 " \"cpufreq\" : { # Use PERFORMANCE CpuFreq\n", 143 " \"governor\" : \"performance\",\n", 144 " },\n", 145 " },\n", 146 " ],\n", 147 " \n", 148 " # Workloads to run (on each platform configuration)\n", 149 " \"wloads\" : {\n", 150 " # Run hackbench with 1 group using pipes\n", 151 " \"perf\" : {\n", 152 " \"type\" : \"perf_bench\",\n", 153 " \"conf\" : {\n", 154 " \"class\" : \"messaging\",\n", 155 " \"params\" : {\n", 156 " \"group\" : 1,\n", 157 " \"loop\" : 10,\n", 158 " \"pipe\" : True,\n", 159 " \"thread\": True,\n", 160 " }\n", 161 " }\n", 162 " },\n", 163 " # Run a 20% duty-cycle periodic task\n", 164 " \"rta\" : {\n", 165 " \"type\" : \"rt-app\",\n", 166 " \"loadref\" : \"big\",\n", 167 " \"conf\" : {\n", 168 " \"class\" : \"profile\",\n", 169 " \"params\" : {\n", 170 " \"p20\" : {\n", 171 " \"kind\" : \"Periodic\",\n", 172 " \"params\" : {\n", 173 " \"duty_cycle_pct\" : 20,\n", 174 " },\n", 175 " },\n", 176 " },\n", 177 " },\n", 178 " },\n", 179 " },\n", 180 " \n", 181 " # Number of iterations for each workloaditerations\n", 182 " \"iterations\" : 1,\n", 183 "}\n", 184 " \n", 185 "my_test_conf = {\n", 186 " # FTrace events to collect for all the tests configuration which have\n", 187 " # the \"ftrace\" flag enabled\n", 188 " \"ftrace\" : {\n", 189 " \"events\" : [\n", 190 " \"sched_switch\",\n", 191 " \"sched_wakeup\",\n", 192 " \"sched_wakeup_new\",\n", 193 " \"cpu_frequency\",\n", 194 " ],\n", 195 " \"buffsize\" : 80 * 1024,\n", 196 " },\n", 197 " \n", 198 " # Tools required by the experiments\n", 199 " \"tools\" : [ 'trace-cmd', 'perf' ],\n", 200 " \n", 201 " # Modules required by these experiments\n", 202 " \"modules\" : [ 'bl', 'cpufreq', 'cgroups' ],\n", 203 "\n", 204 "}" 205 ] 206 }, 207 { 208 "cell_type": "markdown", 209 "metadata": {}, 210 "source": [ 211 "## Tests execution" 212 ] 213 }, 214 { 215 "cell_type": "code", 216 "execution_count": 5, 217 "metadata": { 218 "collapsed": false 219 }, 220 "outputs": [ 221 { 222 "name": "stderr", 223 "output_type": "stream", 224 "text": [ 225 "2016-12-07 10:17:28,037 INFO : TestEnv : Using base path: /home/vagrant/lisa\n", 226 "2016-12-07 10:17:28,039 INFO : TestEnv : Loading custom (inline) target configuration\n", 227 "2016-12-07 10:17:28,039 INFO : TestEnv : Devlib modules to load: ['bl', 'hwmon', 'cpufreq']\n", 228 "2016-12-07 10:17:28,040 INFO : TestEnv : Connecting linux target:\n", 229 "2016-12-07 10:17:28,040 INFO : TestEnv : username : root\n", 230 "2016-12-07 10:17:28,041 INFO : TestEnv : host : 192.168.0.1\n", 231 "2016-12-07 10:17:28,041 INFO : TestEnv : password : juno\n", 232 "2016-12-07 10:17:28,041 INFO : TestEnv : Connection settings:\n", 233 "2016-12-07 10:17:28,042 INFO : TestEnv : {'username': 'root', 'host': '192.168.0.1', 'password': 'juno'}\n", 234 "2016-12-07 10:17:45,282 INFO : TestEnv : Initializing target workdir:\n", 235 "2016-12-07 10:17:45,283 INFO : TestEnv : /root/devlib-target\n", 236 "2016-12-07 10:17:51,006 INFO : TestEnv : Topology:\n", 237 "2016-12-07 10:17:51,007 INFO : TestEnv : [[0, 3, 4, 5], [1, 2]]\n", 238 "2016-12-07 10:17:52,248 INFO : EnergyMeter : Scanning for HWMON channels, may take some time...\n", 239 "2016-12-07 10:17:52,250 INFO : EnergyMeter : Channels selected for energy sampling:\n", 240 "2016-12-07 10:17:52,250 INFO : EnergyMeter : BOARDBIG_energy\n", 241 "2016-12-07 10:17:52,251 INFO : EnergyMeter : BOARDLITTLE_energy\n", 242 "2016-12-07 10:17:52,251 INFO : TestEnv : Set results folder to:\n", 243 "2016-12-07 10:17:52,251 INFO : TestEnv : /home/vagrant/lisa/results/20161207_101752\n", 244 "2016-12-07 10:17:52,252 INFO : TestEnv : Experiment results available also in:\n", 245 "2016-12-07 10:17:52,252 INFO : TestEnv : /home/vagrant/lisa/results_latest\n", 246 "2016-12-07 10:17:52,253 INFO : Executor : Loading custom (inline) test configuration\n", 247 "2016-12-07 10:17:52,253 INFO : Executor : \n", 248 "2016-12-07 10:17:52,254 INFO : Executor : ################################################################################\n", 249 "2016-12-07 10:17:52,254 INFO : Executor : Experiments configuration\n", 250 "2016-12-07 10:17:52,254 INFO : Executor : ################################################################################\n", 251 "2016-12-07 10:17:52,255 INFO : Executor : Configured to run:\n", 252 "2016-12-07 10:17:52,255 INFO : Executor : 2 target configurations:\n", 253 "2016-12-07 10:17:52,256 INFO : Executor : base, eas\n", 254 "2016-12-07 10:17:52,256 INFO : Executor : 2 workloads (1 iterations each)\n", 255 "2016-12-07 10:17:52,257 INFO : Executor : rta, perf\n", 256 "2016-12-07 10:17:52,257 INFO : Executor : Total: 4 experiments\n", 257 "2016-12-07 10:17:52,257 INFO : Executor : Results will be collected under:\n", 258 "2016-12-07 10:17:52,258 INFO : Executor : /home/vagrant/lisa/results/20161207_101752\n" 259 ] 260 } 261 ], 262 "source": [ 263 "executor = Executor(TestEnv(target_conf=my_target_conf, test_conf=my_test_conf), my_experiments_conf)" 264 ] 265 }, 266 { 267 "cell_type": "code", 268 "execution_count": 6, 269 "metadata": { 270 "collapsed": false 271 }, 272 "outputs": [ 273 { 274 "name": "stderr", 275 "output_type": "stream", 276 "text": [ 277 "2016-12-07 10:17:59,239 INFO : Executor : \n", 278 "2016-12-07 10:17:59,239 INFO : Executor : ################################################################################\n", 279 "2016-12-07 10:17:59,240 INFO : Executor : Experiments execution\n", 280 "2016-12-07 10:17:59,240 INFO : Executor : ################################################################################\n", 281 "2016-12-07 10:17:59,241 INFO : Executor : \n", 282 "2016-12-07 10:17:59,241 INFO : Executor : ================================================================================\n", 283 "2016-12-07 10:17:59,241 INFO : Executor : configuring target for [base] experiments\n", 284 "2016-12-07 10:18:00,663 INFO : Executor : Set scheduler feature: NO_ENERGY_AWARE\n", 285 "2016-12-07 10:18:01,469 INFO : Executor : Configuring all CPUs to use [performance] cpufreq governor\n", 286 "2016-12-07 10:18:02,274 INFO : Workload : Setup new workload rta\n", 287 "2016-12-07 10:18:02,274 INFO : Workload : Workload duration defined by longest task\n", 288 "2016-12-07 10:18:02,275 INFO : Workload : Default policy: SCHED_OTHER\n", 289 "2016-12-07 10:18:02,275 INFO : Workload : ------------------------\n", 290 "2016-12-07 10:18:02,276 INFO : Workload : task [task_p200], sched: using default policy\n", 291 "2016-12-07 10:18:02,276 INFO : Workload : | calibration CPU: 1\n", 292 "2016-12-07 10:18:02,277 INFO : Workload : | loops count: 1\n", 293 "2016-12-07 10:18:02,277 INFO : Workload : + phase_000001: duration 1.000000 [s] (10 loops)\n", 294 "2016-12-07 10:18:02,278 INFO : Workload : | period 100000 [us], duty_cycle 20 %\n", 295 "2016-12-07 10:18:02,278 INFO : Workload : | run_time 20000 [us], sleep_time 80000 [us]\n", 296 "2016-12-07 10:18:05,732 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", 297 "2016-12-07 10:18:05,734 INFO : Executor : Experiment 0/4, [base:rta] 1/1\n", 298 "2016-12-07 10:18:06,361 INFO : Workload : Workload execution START:\n", 299 "2016-12-07 10:18:06,363 INFO : Workload : /root/devlib-target/bin/rt-app /root/devlib-target/run_dir/rta_00.json 2>&1\n", 300 "2016-12-07 10:18:13,384 INFO : Executor : --------------------------------------------------------------------------------\n", 301 "2016-12-07 10:18:13,386 INFO : Workload : Setup new workload perf\n", 302 "2016-12-07 10:18:13,712 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", 303 "2016-12-07 10:18:13,712 INFO : Executor : Experiment 1/4, [base:perf] 1/1\n", 304 "2016-12-07 10:18:14,337 INFO : Workload : Workload execution START:\n", 305 "2016-12-07 10:18:14,338 INFO : Workload : /root/devlib-target/bin/perf bench sched messaging --pipe --thread --group 1 --loop 10\n", 306 "2016-12-07 10:18:14,664 INFO : perf_bench : PerfBench - Completion time: 0.010000, Performance 100.000000\n", 307 "2016-12-07 10:18:15,286 INFO : Executor : --------------------------------------------------------------------------------\n", 308 "2016-12-07 10:18:15,288 INFO : Executor : \n", 309 "2016-12-07 10:18:15,290 INFO : Executor : ================================================================================\n", 310 "2016-12-07 10:18:15,292 INFO : Executor : configuring target for [eas] experiments\n", 311 "2016-12-07 10:18:16,713 INFO : Executor : Set scheduler feature: ENERGY_AWARE\n", 312 "2016-12-07 10:18:17,519 INFO : Executor : Configuring all CPUs to use [performance] cpufreq governor\n", 313 "2016-12-07 10:18:18,325 INFO : Workload : Setup new workload rta\n", 314 "2016-12-07 10:18:18,326 INFO : Workload : Workload duration defined by longest task\n", 315 "2016-12-07 10:18:18,327 INFO : Workload : Default policy: SCHED_OTHER\n", 316 "2016-12-07 10:18:18,329 INFO : Workload : ------------------------\n", 317 "2016-12-07 10:18:18,330 INFO : Workload : task [task_p200], sched: using default policy\n", 318 "2016-12-07 10:18:18,331 INFO : Workload : | calibration CPU: 1\n", 319 "2016-12-07 10:18:18,332 INFO : Workload : | loops count: 1\n", 320 "2016-12-07 10:18:18,332 INFO : Workload : + phase_000001: duration 1.000000 [s] (10 loops)\n", 321 "2016-12-07 10:18:18,333 INFO : Workload : | period 100000 [us], duty_cycle 20 %\n", 322 "2016-12-07 10:18:18,333 INFO : Workload : | run_time 20000 [us], sleep_time 80000 [us]\n", 323 "2016-12-07 10:18:21,414 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", 324 "2016-12-07 10:18:21,415 INFO : Executor : Experiment 2/4, [eas:rta] 1/1\n", 325 "2016-12-07 10:18:22,043 INFO : Workload : Workload execution START:\n", 326 "2016-12-07 10:18:22,045 INFO : Workload : /root/devlib-target/bin/rt-app /root/devlib-target/run_dir/rta_00.json 2>&1\n", 327 "2016-12-07 10:18:28,802 INFO : Executor : --------------------------------------------------------------------------------\n", 328 "2016-12-07 10:18:28,806 INFO : Workload : Setup new workload perf\n", 329 "2016-12-07 10:18:29,134 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", 330 "2016-12-07 10:18:29,135 INFO : Executor : Experiment 3/4, [eas:perf] 1/1\n", 331 "2016-12-07 10:18:29,760 INFO : Workload : Workload execution START:\n", 332 "2016-12-07 10:18:29,761 INFO : Workload : /root/devlib-target/bin/perf bench sched messaging --pipe --thread --group 1 --loop 10\n", 333 "2016-12-07 10:18:30,089 INFO : perf_bench : PerfBench - Completion time: 0.009000, Performance 111.111111\n", 334 "2016-12-07 10:18:30,710 INFO : Executor : --------------------------------------------------------------------------------\n", 335 "2016-12-07 10:18:30,712 INFO : Executor : \n", 336 "2016-12-07 10:18:30,714 INFO : Executor : ################################################################################\n", 337 "2016-12-07 10:18:30,715 INFO : Executor : Experiments execution completed\n", 338 "2016-12-07 10:18:30,716 INFO : Executor : ################################################################################\n", 339 "2016-12-07 10:18:30,716 INFO : Executor : Results available in:\n", 340 "2016-12-07 10:18:30,717 INFO : Executor : /home/vagrant/lisa/results/20161207_101752\n" 341 ] 342 } 343 ], 344 "source": [ 345 "executor.run()" 346 ] 347 }, 348 { 349 "cell_type": "code", 350 "execution_count": 7, 351 "metadata": { 352 "collapsed": false 353 }, 354 "outputs": [ 355 { 356 "name": "stdout", 357 "output_type": "stream", 358 "text": [ 359 "/home/vagrant/lisa/results/20161207_101752\r\n", 360 " perf_bench_messaging:base:perf\r\n", 361 " 1\r\n", 362 " energy.json\r\n", 363 " output.log\r\n", 364 " performance.json\r\n", 365 " kernel.config\r\n", 366 " kernel.version\r\n", 367 " platform.json\r\n", 368 " perf_bench_messaging:eas:perf\r\n", 369 " 1\r\n", 370 " energy.json\r\n", 371 " output.log\r\n", 372 " performance.json\r\n", 373 " kernel.config\r\n", 374 " kernel.version\r\n", 375 " platform.json\r\n", 376 " rtapp:base:rta\r\n", 377 " 1\r\n", 378 " energy.json\r\n", 379 " output.log\r\n", 380 " rta_00.json\r\n", 381 " rt-app-task_p200-0.log\r\n", 382 " kernel.config\r\n", 383 " kernel.version\r\n", 384 " platform.json\r\n", 385 " rtapp:eas:rta\r\n", 386 " 1\r\n", 387 " energy.json\r\n", 388 " output.log\r\n", 389 " rta_00.json\r\n", 390 " rt-app-task_p200-0.log\r\n", 391 " kernel.config\r\n", 392 " kernel.version\r\n", 393 " platform.json\r\n", 394 "\r\n", 395 "8 directories, 26 files\r\n" 396 ] 397 } 398 ], 399 "source": [ 400 "!tree {executor.te.res_dir}" 401 ] 402 } 403 ], 404 "metadata": { 405 "kernelspec": { 406 "display_name": "Python 2", 407 "language": "python", 408 "name": "python2" 409 }, 410 "language_info": { 411 "codemirror_mode": { 412 "name": "ipython", 413 "version": 2 414 }, 415 "file_extension": ".py", 416 "mimetype": "text/x-python", 417 "name": "python", 418 "nbconvert_exporter": "python", 419 "pygments_lexer": "ipython2", 420 "version": "2.7.6" 421 } 422 }, 423 "nbformat": 4, 424 "nbformat_minor": 0 425 } 426
