| { |
| "cells": [ |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "# Executor API - Executor\n", |
| "\n", |
| " A tests executor is a module which supports the execution of a configured set of experiments.<br><br>\n", |
| " Each experiment is composed by:\n", |
| " - a target configuration\n", |
| " - a workload to execute\n", |
| "\n", |
| "The executor module can be configured to run a set of workloads (wloads) in each different target configuration of a specified set (confs). These wloads and confs can be specified by the \"experiments_conf\" input dictionary which is described below at **Experiments Configuration**.<br><br>\n", |
| "All the results generated by each experiment will be collected in a results folder. The format and content fo the results forlder is detailed in the last cell of **Tests execution** below.\n" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 1, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [ |
| { |
| "name": "stderr", |
| "output_type": "stream", |
| "text": [ |
| "2016-12-08 11:58:22,693 INFO : root : Using LISA logging configuration:\n", |
| "2016-12-08 11:58:22,694 INFO : root : /home/vagrant/lisa/logging.conf\n" |
| ] |
| } |
| ], |
| "source": [ |
| "import logging\n", |
| "\n", |
| "from conf import LisaLogging\n", |
| "LisaLogging.setup()" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 2, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [], |
| "source": [ |
| "import os\n", |
| "import json\n", |
| "\n", |
| "from env import TestEnv\n", |
| "from executor import Executor" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "## Target Configuration\n", |
| "The target configuration it's used to describe and configure your test environment.\n", |
| "You can find more details in **examples/utils/testenv_example.ipynb**." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 3, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [], |
| "source": [ |
| "# Setup a target configuration\n", |
| "my_target_conf = {\n", |
| " \n", |
| " # Target platform and board\n", |
| " \"platform\" : 'linux',\n", |
| " \"board\" : 'juno',\n", |
| " \n", |
| " # Target board IP/MAC address\n", |
| " \"host\" : '192.168.0.1',\n", |
| " \n", |
| " # Login credentials\n", |
| " \"username\" : 'root',\n", |
| " \"password\" : 'juno',\n", |
| "\n", |
| "}" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "## Experiments Configuration\n", |
| "\n", |
| "The experiments configuration defines the software setups that we need on our hardware target.<br>\n", |
| "This can be given as an argument to an Executor instance or to a TestEnv one.<br> <br>\n", |
| "Elements of the experiments configuration:\n", |
| " - **confs**: **mandatory** platform configurations to be tested.\n", |
| " - tag: relevant string to identify your configuration.\n", |
| " - flags: ftrace (to enable ftrace events) is the only one supported at the moment.\n", |
| " - sched_features: features to be added to /sys/kernel/debug/sched_features.\n", |
| " - cpufreq: CpuFreq governor and tunables.\n", |
| " - cgroups: CGroups configuration (controller). The default CGroup will be used otherwise.\n", |
| " - **wloads**: **mandatory** workloads to run on each platform configuration.\n", |
| " - **iterations**: number of iterations for each workload.\n", |
| " - **tools**: binary tools (available under ./tools/$ARCH/) to install by default; these will be merged with the ones in the target configuration.\n", |
| " - **ftrace**: FTrace events to collect for all the experiments configurations which have the \"ftrace\" flag enabled.\n", |
| " - **modules**: modules required by the experiments resulted from the experiments configurations.\n", |
| " - **exclude_modules** - modules to be disabled.\n", |
| " - **results_dir**: results directory - experiments configuration results directory overrides target one." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 4, |
| "metadata": { |
| "collapsed": false, |
| "scrolled": false |
| }, |
| "outputs": [], |
| "source": [ |
| "my_experiments_conf = {\n", |
| "\n", |
| " # Folder where all the results will be collected\n", |
| " \"results_dir\" : \"ExecutorExample\",\n", |
| "\n", |
| " # Platform configurations to test: you can specify any number of configurations\n", |
| " \"confs\" : [\n", |
| " {\n", |
| " \"tag\" : \"base\", # Relevant string to identify configuration\n", |
| " \"flags\" : \"ftrace\", # Enable FTrace events\n", |
| " \"sched_features\" : \"NO_ENERGY_AWARE\", # Disable EAS\n", |
| " \"cpufreq\" : { # Use PERFORMANCE CpuFreq\n", |
| " \"governor\" : \"performance\",\n", |
| " },\n", |
| " },\n", |
| " {\n", |
| " \"tag\" : \"eas\", # Relevant string to identify configuration\n", |
| " \"flags\" : \"ftrace\", # Enable FTrace events\n", |
| " \"sched_features\" : \"ENERGY_AWARE\", # Enable EAS\n", |
| " \"cpufreq\" : { # Use PERFORMANCE CpuFreq\n", |
| " \"governor\" : \"performance\",\n", |
| " },\n", |
| " },\n", |
| " ],\n", |
| " \n", |
| " # Workloads to run (on each platform configuration)\n", |
| " \"wloads\" : {\n", |
| " # Run hackbench with 1 group using pipes\n", |
| " \"perf\" : {\n", |
| " \"type\" : \"perf_bench\",\n", |
| " \"conf\" : {\n", |
| " \"class\" : \"messaging\",\n", |
| " \"params\" : {\n", |
| " \"group\" : 1,\n", |
| " \"loop\" : 10,\n", |
| " \"pipe\" : True,\n", |
| " \"thread\": True,\n", |
| " }\n", |
| " }\n", |
| " },\n", |
| " # Run a 20% duty-cycle periodic task\n", |
| " \"rta\" : {\n", |
| " \"type\" : \"rt-app\",\n", |
| " \"loadref\" : \"big\",\n", |
| " \"conf\" : {\n", |
| " \"class\" : \"profile\",\n", |
| " \"params\" : {\n", |
| " \"p20\" : {\n", |
| " \"kind\" : \"Periodic\",\n", |
| " \"params\" : {\n", |
| " \"duty_cycle_pct\" : 20,\n", |
| " },\n", |
| " },\n", |
| " },\n", |
| " },\n", |
| " },\n", |
| " },\n", |
| " \n", |
| " # Number of iterations for each workloaditerations\n", |
| " \"iterations\" : 1,\n", |
| " \n", |
| " # FTrace events to collect for all the tests configuration which have\n", |
| " # the \"ftrace\" flag enabled\n", |
| " \"ftrace\" : {\n", |
| " \"events\" : [\n", |
| " \"sched_switch\",\n", |
| " \"sched_wakeup\",\n", |
| " \"sched_wakeup_new\",\n", |
| " \"cpu_frequency\",\n", |
| " ],\n", |
| " \"buffsize\" : 80 * 1024,\n", |
| " },\n", |
| " \n", |
| " # Tools required by the experiments\n", |
| " \"tools\" : [ 'trace-cmd', 'perf' ],\n", |
| " \n", |
| " # Modules required by these experiments\n", |
| " \"modules\" : [ 'bl', 'cpufreq' ],\n", |
| "\n", |
| "}" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "## Tests execution" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 5, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [ |
| { |
| "name": "stderr", |
| "output_type": "stream", |
| "text": [ |
| "2016-12-07 10:17:28,037 INFO : TestEnv : Using base path: /home/vagrant/lisa\n", |
| "2016-12-07 10:17:28,039 INFO : TestEnv : Loading custom (inline) target configuration\n", |
| "2016-12-07 10:17:28,039 INFO : TestEnv : Devlib modules to load: ['bl', 'hwmon', 'cpufreq']\n", |
| "2016-12-07 10:17:28,040 INFO : TestEnv : Connecting linux target:\n", |
| "2016-12-07 10:17:28,040 INFO : TestEnv : username : root\n", |
| "2016-12-07 10:17:28,041 INFO : TestEnv : host : 192.168.0.1\n", |
| "2016-12-07 10:17:28,041 INFO : TestEnv : password : juno\n", |
| "2016-12-07 10:17:28,041 INFO : TestEnv : Connection settings:\n", |
| "2016-12-07 10:17:28,042 INFO : TestEnv : {'username': 'root', 'host': '192.168.0.1', 'password': 'juno'}\n", |
| "2016-12-07 10:17:45,282 INFO : TestEnv : Initializing target workdir:\n", |
| "2016-12-07 10:17:45,283 INFO : TestEnv : /root/devlib-target\n", |
| "2016-12-07 10:17:51,006 INFO : TestEnv : Topology:\n", |
| "2016-12-07 10:17:51,007 INFO : TestEnv : [[0, 3, 4, 5], [1, 2]]\n", |
| "2016-12-07 10:17:52,248 INFO : EnergyMeter : Scanning for HWMON channels, may take some time...\n", |
| "2016-12-07 10:17:52,250 INFO : EnergyMeter : Channels selected for energy sampling:\n", |
| "2016-12-07 10:17:52,250 INFO : EnergyMeter : BOARDBIG_energy\n", |
| "2016-12-07 10:17:52,251 INFO : EnergyMeter : BOARDLITTLE_energy\n", |
| "2016-12-07 10:17:52,251 INFO : TestEnv : Set results folder to:\n", |
| "2016-12-07 10:17:52,251 INFO : TestEnv : /home/vagrant/lisa/results/20161207_101752\n", |
| "2016-12-07 10:17:52,252 INFO : TestEnv : Experiment results available also in:\n", |
| "2016-12-07 10:17:52,252 INFO : TestEnv : /home/vagrant/lisa/results_latest\n", |
| "2016-12-07 10:17:52,253 INFO : Executor : Loading custom (inline) test configuration\n", |
| "2016-12-07 10:17:52,253 INFO : Executor : \n", |
| "2016-12-07 10:17:52,254 INFO : Executor : ################################################################################\n", |
| "2016-12-07 10:17:52,254 INFO : Executor : Experiments configuration\n", |
| "2016-12-07 10:17:52,254 INFO : Executor : ################################################################################\n", |
| "2016-12-07 10:17:52,255 INFO : Executor : Configured to run:\n", |
| "2016-12-07 10:17:52,255 INFO : Executor : 2 target configurations:\n", |
| "2016-12-07 10:17:52,256 INFO : Executor : base, eas\n", |
| "2016-12-07 10:17:52,256 INFO : Executor : 2 workloads (1 iterations each)\n", |
| "2016-12-07 10:17:52,257 INFO : Executor : rta, perf\n", |
| "2016-12-07 10:17:52,257 INFO : Executor : Total: 4 experiments\n", |
| "2016-12-07 10:17:52,257 INFO : Executor : Results will be collected under:\n", |
| "2016-12-07 10:17:52,258 INFO : Executor : /home/vagrant/lisa/results/20161207_101752\n" |
| ] |
| } |
| ], |
| "source": [ |
| "executor = Executor(TestEnv(my_target_conf), my_experiments_conf)" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 6, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [ |
| { |
| "name": "stderr", |
| "output_type": "stream", |
| "text": [ |
| "2016-12-07 10:17:59,239 INFO : Executor : \n", |
| "2016-12-07 10:17:59,239 INFO : Executor : ################################################################################\n", |
| "2016-12-07 10:17:59,240 INFO : Executor : Experiments execution\n", |
| "2016-12-07 10:17:59,240 INFO : Executor : ################################################################################\n", |
| "2016-12-07 10:17:59,241 INFO : Executor : \n", |
| "2016-12-07 10:17:59,241 INFO : Executor : ================================================================================\n", |
| "2016-12-07 10:17:59,241 INFO : Executor : configuring target for [base] experiments\n", |
| "2016-12-07 10:18:00,663 INFO : Executor : Set scheduler feature: NO_ENERGY_AWARE\n", |
| "2016-12-07 10:18:01,469 INFO : Executor : Configuring all CPUs to use [performance] cpufreq governor\n", |
| "2016-12-07 10:18:02,274 INFO : Workload : Setup new workload rta\n", |
| "2016-12-07 10:18:02,274 INFO : Workload : Workload duration defined by longest task\n", |
| "2016-12-07 10:18:02,275 INFO : Workload : Default policy: SCHED_OTHER\n", |
| "2016-12-07 10:18:02,275 INFO : Workload : ------------------------\n", |
| "2016-12-07 10:18:02,276 INFO : Workload : task [task_p200], sched: using default policy\n", |
| "2016-12-07 10:18:02,276 INFO : Workload : | calibration CPU: 1\n", |
| "2016-12-07 10:18:02,277 INFO : Workload : | loops count: 1\n", |
| "2016-12-07 10:18:02,277 INFO : Workload : + phase_000001: duration 1.000000 [s] (10 loops)\n", |
| "2016-12-07 10:18:02,278 INFO : Workload : | period 100000 [us], duty_cycle 20 %\n", |
| "2016-12-07 10:18:02,278 INFO : Workload : | run_time 20000 [us], sleep_time 80000 [us]\n", |
| "2016-12-07 10:18:05,732 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", |
| "2016-12-07 10:18:05,734 INFO : Executor : Experiment 0/4, [base:rta] 1/1\n", |
| "2016-12-07 10:18:06,361 INFO : Workload : Workload execution START:\n", |
| "2016-12-07 10:18:06,363 INFO : Workload : /root/devlib-target/bin/rt-app /root/devlib-target/run_dir/rta_00.json 2>&1\n", |
| "2016-12-07 10:18:13,384 INFO : Executor : --------------------------------------------------------------------------------\n", |
| "2016-12-07 10:18:13,386 INFO : Workload : Setup new workload perf\n", |
| "2016-12-07 10:18:13,712 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", |
| "2016-12-07 10:18:13,712 INFO : Executor : Experiment 1/4, [base:perf] 1/1\n", |
| "2016-12-07 10:18:14,337 INFO : Workload : Workload execution START:\n", |
| "2016-12-07 10:18:14,338 INFO : Workload : /root/devlib-target/bin/perf bench sched messaging --pipe --thread --group 1 --loop 10\n", |
| "2016-12-07 10:18:14,664 INFO : perf_bench : PerfBench - Completion time: 0.010000, Performance 100.000000\n", |
| "2016-12-07 10:18:15,286 INFO : Executor : --------------------------------------------------------------------------------\n", |
| "2016-12-07 10:18:15,288 INFO : Executor : \n", |
| "2016-12-07 10:18:15,290 INFO : Executor : ================================================================================\n", |
| "2016-12-07 10:18:15,292 INFO : Executor : configuring target for [eas] experiments\n", |
| "2016-12-07 10:18:16,713 INFO : Executor : Set scheduler feature: ENERGY_AWARE\n", |
| "2016-12-07 10:18:17,519 INFO : Executor : Configuring all CPUs to use [performance] cpufreq governor\n", |
| "2016-12-07 10:18:18,325 INFO : Workload : Setup new workload rta\n", |
| "2016-12-07 10:18:18,326 INFO : Workload : Workload duration defined by longest task\n", |
| "2016-12-07 10:18:18,327 INFO : Workload : Default policy: SCHED_OTHER\n", |
| "2016-12-07 10:18:18,329 INFO : Workload : ------------------------\n", |
| "2016-12-07 10:18:18,330 INFO : Workload : task [task_p200], sched: using default policy\n", |
| "2016-12-07 10:18:18,331 INFO : Workload : | calibration CPU: 1\n", |
| "2016-12-07 10:18:18,332 INFO : Workload : | loops count: 1\n", |
| "2016-12-07 10:18:18,332 INFO : Workload : + phase_000001: duration 1.000000 [s] (10 loops)\n", |
| "2016-12-07 10:18:18,333 INFO : Workload : | period 100000 [us], duty_cycle 20 %\n", |
| "2016-12-07 10:18:18,333 INFO : Workload : | run_time 20000 [us], sleep_time 80000 [us]\n", |
| "2016-12-07 10:18:21,414 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", |
| "2016-12-07 10:18:21,415 INFO : Executor : Experiment 2/4, [eas:rta] 1/1\n", |
| "2016-12-07 10:18:22,043 INFO : Workload : Workload execution START:\n", |
| "2016-12-07 10:18:22,045 INFO : Workload : /root/devlib-target/bin/rt-app /root/devlib-target/run_dir/rta_00.json 2>&1\n", |
| "2016-12-07 10:18:28,802 INFO : Executor : --------------------------------------------------------------------------------\n", |
| "2016-12-07 10:18:28,806 INFO : Workload : Setup new workload perf\n", |
| "2016-12-07 10:18:29,134 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", |
| "2016-12-07 10:18:29,135 INFO : Executor : Experiment 3/4, [eas:perf] 1/1\n", |
| "2016-12-07 10:18:29,760 INFO : Workload : Workload execution START:\n", |
| "2016-12-07 10:18:29,761 INFO : Workload : /root/devlib-target/bin/perf bench sched messaging --pipe --thread --group 1 --loop 10\n", |
| "2016-12-07 10:18:30,089 INFO : perf_bench : PerfBench - Completion time: 0.009000, Performance 111.111111\n", |
| "2016-12-07 10:18:30,710 INFO : Executor : --------------------------------------------------------------------------------\n", |
| "2016-12-07 10:18:30,712 INFO : Executor : \n", |
| "2016-12-07 10:18:30,714 INFO : Executor : ################################################################################\n", |
| "2016-12-07 10:18:30,715 INFO : Executor : Experiments execution completed\n", |
| "2016-12-07 10:18:30,716 INFO : Executor : ################################################################################\n", |
| "2016-12-07 10:18:30,716 INFO : Executor : Results available in:\n", |
| "2016-12-07 10:18:30,717 INFO : Executor : /home/vagrant/lisa/results/20161207_101752\n" |
| ] |
| } |
| ], |
| "source": [ |
| "executor.run()" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 7, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [ |
| { |
| "name": "stdout", |
| "output_type": "stream", |
| "text": [ |
| "/home/vagrant/lisa/results/20161207_101752\r\n", |
| "├── perf_bench_messaging:base:perf\r\n", |
| "│ ├── 1\r\n", |
| "│ │ ├── energy.json\r\n", |
| "│ │ ├── output.log\r\n", |
| "│ │ └── performance.json\r\n", |
| "│ ├── kernel.config\r\n", |
| "│ ├── kernel.version\r\n", |
| "│ └── platform.json\r\n", |
| "├── perf_bench_messaging:eas:perf\r\n", |
| "│ ├── 1\r\n", |
| "│ │ ├── energy.json\r\n", |
| "│ │ ├── output.log\r\n", |
| "│ │ └── performance.json\r\n", |
| "│ ├── kernel.config\r\n", |
| "│ ├── kernel.version\r\n", |
| "│ └── platform.json\r\n", |
| "├── rtapp:base:rta\r\n", |
| "│ ├── 1\r\n", |
| "│ │ ├── energy.json\r\n", |
| "│ │ ├── output.log\r\n", |
| "│ │ ├── rta_00.json\r\n", |
| "│ │ └── rt-app-task_p200-0.log\r\n", |
| "│ ├── kernel.config\r\n", |
| "│ ├── kernel.version\r\n", |
| "│ └── platform.json\r\n", |
| "└── rtapp:eas:rta\r\n", |
| " ├── 1\r\n", |
| " │ ├── energy.json\r\n", |
| " │ ├── output.log\r\n", |
| " │ ├── rta_00.json\r\n", |
| " │ └── rt-app-task_p200-0.log\r\n", |
| " ├── kernel.config\r\n", |
| " ├── kernel.version\r\n", |
| " └── platform.json\r\n", |
| "\r\n", |
| "8 directories, 26 files\r\n" |
| ] |
| } |
| ], |
| "source": [ |
| "!tree {executor.te.res_dir}" |
| ] |
| } |
| ], |
| "metadata": { |
| "kernelspec": { |
| "display_name": "Python 2", |
| "language": "python", |
| "name": "python2" |
| }, |
| "language_info": { |
| "codemirror_mode": { |
| "name": "ipython", |
| "version": 2 |
| }, |
| "file_extension": ".py", |
| "mimetype": "text/x-python", |
| "name": "python", |
| "nbconvert_exporter": "python", |
| "pygments_lexer": "ipython2", |
| "version": "2.7.6" |
| } |
| }, |
| "nbformat": 4, |
| "nbformat_minor": 0 |
| } |