blob: f32a5f07f8ab8883c7c3508482af17449062b2e5 [file] [log] [blame]
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Test environment API - TestEnv\n",
"\n",
"The test environment is primarily defined by the target configuration (see **conf** below).\n",
"\n",
"One can also pass the test configuration - definining which software setups are needed on the hardware target, and a location for the results of the experiments.\n",
"\n",
"Parameters:\n",
" - target configuration\n",
" - test configuration - more information on this can be found in **examples/utils/executor_example.ipynb**\n",
" - wipe - if to clean up all previous content from the results folder\n",
" - force new - if to create a new TestEnv object even if there is one available"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# One initial cell for imports\n",
"import json\n",
"import time\n",
"import os\n",
"import logging"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2016-12-12 11:51:14,376 INFO : root : Using LISA logging configuration:\n",
"2016-12-12 11:51:14,376 INFO : root : /home/vagrant/lisa/logging.conf\n"
]
}
],
"source": [
"from conf import LisaLogging\n",
"LisaLogging.setup()\n",
"# For debug information use:\n",
"# LisaLogging.setup(level=logging.DEBUG)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Test environment setup"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Target configuration**:\n",
"\n",
" - **platform** - the currently supported boards are:\n",
" - linux - accessed via SSH connection\n",
" - android - accessed via ADB connection\n",
" - host - ran on local host\n",
" \n",
" - **board** - the currently supported boards are:\n",
" - juno - target is a JUNO board\n",
" - tc2 - target is a TC2 board\n",
" - oak - target is MT8173 platform model\n",
" - pixel - target is a Pixel device\n",
" - hikey - target is a Hikey development platform\n",
" - nexus5x - target is a Nexus 5X device\n",
" \n",
" - **host** - target IP or MAC address\n",
" \n",
" - **device** - target Android device ID\n",
" \n",
" - **port** - port for Android connection - default port is 5555\n",
" \n",
" - **ANDROID_HOME** - path to android-sdk-linux\n",
" \n",
" - **username**\n",
" \n",
" - **password**\n",
" \n",
" - **keyfile** - you can either specify a password or a keyfile\n",
" \n",
" - **rtapp-calib** - these values are not supposed to be specified at the first run on a target.\n",
" After the first run, it's best to fill this array with values reported in the log messges for\n",
" your specific target, for these not to be obtained again.\n",
" \n",
" - **tftp** - tftp server from where the target gets kernel/dtb images at each boot\n",
" \n",
" - **modules** - devlib modules to be enabled\n",
" \n",
" - **exclude_modules** - devlib modules to be disabled\n",
" \n",
" - **tools** - binary tools (available under ./tools/$ARCH/) to install by default\n",
" \n",
" - **ping_time** - wait time before trying to access the target after reboot\n",
" \n",
" - **reboot_time** - maximum time to wait after rebooting the target\n",
" \n",
" - **__features__** - list of test environment features to enable\n",
" - no-kernel - do not deploy kernel/dtb images\n",
" - no-reboot - do not force reboot the target at each configuration change\n",
" - debug - enable debugging messages\n",
" \n",
" - **ftrace** - ftrace configuration\n",
" - events\n",
" - functions\n",
" - buffsize\n",
" \n",
" - **results_dir** - location of results of the experiments\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Setup a target configuration\n",
"conf = {\n",
"\n",
" # Platform\n",
" \"platform\" : \"linux\",\n",
" # Board\n",
" \"board\" : \"juno\",\n",
"\n",
" # Login credentials\n",
" \"host\" : \"192.168.0.1\",\n",
" \"username\" : \"root\",\n",
" # You can specify either a password or keyfile\n",
" \"password\" : \"juno\",\n",
" # \"keyfile\" : \"/complete/path/of/your/keyfile\",\n",
"\n",
" # Tools to deploy\n",
" \"tools\" : [ \"rt-app\", \"taskset\" ],\n",
"\n",
" \"tftp\" : {\n",
" \"folder\" : \"/var/lib/tftpboot/\",\n",
" \"kernel\" : \"Image\",\n",
" \"dtb\" : \"juno.dtb\"\n",
" },\n",
"\n",
" #\"ping_time\" : \"15\",\n",
" #\"reboot_time\" : \"180\",\n",
"\n",
" # RTApp calibration values (comment to let LISA do a calibration run)\n",
" \"rtapp-calib\" : {\n",
" \"0\": 358, \"1\": 138, \"2\": 138, \"3\": 357, \"4\": 359, \"5\": 355\n",
" },\n",
"\n",
" # FTrace configuration\n",
" \"ftrace\" : {\n",
" \"events\" : [\n",
" \"cpu_idle\",\n",
" \"sched_switch\",\n",
" ],\n",
" \"buffsize\" : 10240,\n",
" },\n",
" \n",
" # Where results are collected\n",
" \"results_dir\" : \"TestEnvExample\",\n",
"\n",
" #\"__features__\" : \"no-kernel no-reboot\"\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Test environment initialisation"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [],
"source": [
"from env import TestEnv\n",
"\n",
"# Initialize a test environment using the provided configuration\n",
"te = TestEnv(conf)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Attributes\n",
"\n",
"The initialisation of the test environment pre-initialises some useful environment variables.\n",
"\n",
"This is some of the information available via the TestEnv object:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"username\": \"root\", \n",
" \"ftrace\": {\n",
" \"buffsize\": 10240, \n",
" \"events\": [\n",
" \"cpu_idle\", \n",
" \"sched_switch\"\n",
" ]\n",
" }, \n",
" \"rtapp-calib\": {\n",
" \"1\": 138, \n",
" \"0\": 358, \n",
" \"3\": 357, \n",
" \"2\": 138, \n",
" \"5\": 355, \n",
" \"4\": 359\n",
" }, \n",
" \"host\": \"192.168.0.1\", \n",
" \"password\": \"juno\", \n",
" \"tools\": [\n",
" \"rt-app\", \n",
" \"taskset\", \n",
" \"trace-cmd\", \n",
" \"taskset\", \n",
" \"trace-cmd\", \n",
" \"perf\", \n",
" \"cgroup_run_into.sh\"\n",
" ], \n",
" \"results_dir\": \"TestEnvExample\", \n",
" \"platform\": \"linux\", \n",
" \"board\": \"juno\", \n",
" \"__features__\": [], \n",
" \"tftp\": {\n",
" \"kernel\": \"Image\", \n",
" \"folder\": \"/var/lib/tftpboot/\", \n",
" \"dtb\": \"juno.dtb\"\n",
" }\n",
"}\n"
]
}
],
"source": [
"# The complete configuration of the target we have configured\n",
"print json.dumps(te.conf, indent=4)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None\n",
"None\n"
]
}
],
"source": [
"# Last configured kernel and DTB image\n",
"print te.kernel\n",
"print te.dtb"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"192.168.0.1\n",
"None\n"
]
}
],
"source": [
"# The IP and MAC address of the target\n",
"print te.ip\n",
"print te.mac"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"nrg_model\": {\n",
" \"big\": {\n",
" \"cluster\": {\n",
" \"nrg_max\": 64\n",
" }, \n",
" \"cpu\": {\n",
" \"cap_max\": 1024, \n",
" \"nrg_max\": 616\n",
" }\n",
" }, \n",
" \"little\": {\n",
" \"cluster\": {\n",
" \"nrg_max\": 57\n",
" }, \n",
" \"cpu\": {\n",
" \"cap_max\": 447, \n",
" \"nrg_max\": 93\n",
" }\n",
" }\n",
" }, \n",
" \"clusters\": {\n",
" \"big\": [\n",
" 1, \n",
" 2\n",
" ], \n",
" \"little\": [\n",
" 0, \n",
" 3, \n",
" 4, \n",
" 5\n",
" ]\n",
" }, \n",
" \"cpus_count\": 6, \n",
" \"freqs\": {\n",
" \"big\": [\n",
" 450000, \n",
" 625000, \n",
" 800000, \n",
" 950000, \n",
" 1100000\n",
" ], \n",
" \"little\": [\n",
" 450000, \n",
" 575000, \n",
" 700000, \n",
" 775000, \n",
" 850000\n",
" ]\n",
" }, \n",
" \"topology\": [\n",
" [\n",
" 0, \n",
" 3, \n",
" 4, \n",
" 5\n",
" ], \n",
" [\n",
" 1, \n",
" 2\n",
" ]\n",
" ]\n",
"}\n"
]
}
],
"source": [
"# A full platform descriptor\n",
"print json.dumps(te.platform, indent=4)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"'/home/vagrant/lisa/results/TestEnvExample'"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# This is a pre-created folder to host the tests results generated using this\n",
"# test environment. Notice that the suite could add additional information\n",
"# in this folder, like for example a copy of the target configuration\n",
"# and other target specific collected information.\n",
"te.res_dir"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"'/data/local/schedtest'"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# The working directory on the target\n",
"te.workdir"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"cluster [[0, 3, 4, 5], [1, 2]]\n",
"cpu [[0], [1], [2], [3], [4], [5]]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# The target topology, which can be used to build BART assertions\n",
"te.topology"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Functions\n",
"\n",
"Some methods are also exposed to test developers which could be used to ease the creation of tests.\n",
"\n",
"These are some of the methods available:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"{0: 358, 1: 138, 2: 138, 3: 357, 4: 359, 5: 355}"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Calibrate RT-App (if required) and get the most updated calibration value\n",
"te.calibration()"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"({'clusters': {'big': [1, 2], 'little': [0, 3, 4, 5]},\n",
" 'cpus_count': 6,\n",
" 'freqs': {'big': [450000, 625000, 800000, 950000, 1100000],\n",
" 'little': [450000, 575000, 700000, 775000, 850000]},\n",
" 'nrg_model': {u'big': {u'cluster': {u'nrg_max': 64},\n",
" u'cpu': {u'cap_max': 1024, u'nrg_max': 616}},\n",
" u'little': {u'cluster': {u'nrg_max': 57},\n",
" u'cpu': {u'cap_max': 447, u'nrg_max': 93}}},\n",
" 'topology': [[0, 3, 4, 5], [1, 2]]},\n",
" '/tmp/platform.json')"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Generate a JSON file with the complete platform description\n",
"te.platform_dump(dest_dir='/tmp')"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2016-12-12 11:56:15,281 INFO : TestEnv : Target (00:02:f7:00:5d:d7) at IP address: 192.168.0.1\n",
"2016-12-12 11:56:16,087 INFO : TestEnv : Waiting up to 360[s] for target [192.168.0.1] to reboot...\n",
"2016-12-12 11:57:21,143 INFO : TestEnv : Devlib modules to load: ['bl', 'hwmon', 'cpufreq']\n",
"2016-12-12 11:57:21,144 INFO : TestEnv : Connecting linux target:\n",
"2016-12-12 11:57:21,145 INFO : TestEnv : username : root\n",
"2016-12-12 11:57:21,146 INFO : TestEnv : host : 192.168.0.1\n",
"2016-12-12 11:57:21,146 INFO : TestEnv : password : juno\n",
"2016-12-12 11:57:21,147 INFO : TestEnv : Connection settings:\n",
"2016-12-12 11:57:21,147 INFO : TestEnv : {'username': 'root', 'host': '192.168.0.1', 'password': 'juno'}\n",
"2016-12-12 11:57:37,176 INFO : TestEnv : Initializing target workdir:\n",
"2016-12-12 11:57:37,177 INFO : TestEnv : /root/devlib-target\n",
"2016-12-12 11:57:43,908 INFO : TestEnv : Topology:\n",
"2016-12-12 11:57:43,908 INFO : TestEnv : [[0, 3, 4, 5], [1, 2]]\n",
"2016-12-12 11:57:45,155 INFO : TestEnv : Loading default EM:\n",
"2016-12-12 11:57:45,156 INFO : TestEnv : /home/vagrant/lisa/libs/utils/platforms/juno.json\n",
"2016-12-12 11:57:48,681 INFO : TestEnv : Enabled tracepoints:\n",
"2016-12-12 11:57:48,684 INFO : TestEnv : cpu_idle\n",
"2016-12-12 11:57:48,685 INFO : TestEnv : sched_switch\n",
"2016-12-12 11:57:48,688 INFO : EnergyMeter : Scanning for HWMON channels, may take some time...\n",
"2016-12-12 11:57:48,691 INFO : EnergyMeter : Channels selected for energy sampling:\n",
"2016-12-12 11:57:48,691 INFO : EnergyMeter : BOARDBIG_energy\n",
"2016-12-12 11:57:48,692 INFO : EnergyMeter : BOARDLITTLE_energy\n"
]
}
],
"source": [
"# Force a reboot of the target (and wait specified [s] before reconnect)\n",
"# Keep in mind that a reboot can be disabled from __features__ in the target configuration\n",
"te.reboot(reboot_time=360, ping_time=15)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"06:03:00 INFO : HostResolver - Target (00:02:F7:00:5A:5B) at IP address: 192.168.0.1\n"
]
},
{
"data": {
"text/plain": [
"('00:02:F7:00:5A:5B', '192.168.0.1')"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Resolve a MAC address into an IP address\n",
"te.resolv_host(host='00:02:F7:00:5A:5B')"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"06:03:00 INFO : TFTP - Deploy /etc/group into /var/lib/tftpboot/group\n"
]
}
],
"source": [
"# Copy the specified file into the TFTP server folder defined by configuration\n",
"te.tftp_deploy('/etc/group')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Attributes: target\n",
"### Access to the devlib API\n",
"\n",
"A special attribute of TestEnv is **target**, which represents a devlib instance. Using the target attribute we have access to the full set of devlib provided functionalities. A small set of these are exemplified below. For a more extensive set check the **examples/devlib** notebooks."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"'Hello Test Environment'"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Run a command on the target\n",
"te.target.execute(\"echo -n 'Hello Test Environment'\", as_root=False)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"''"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Spawn a command in background on the target\n",
"te.target.kick_off(\"sleep 10\", as_root=True)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ABI : arm64\n",
"big Core Family : A57\n",
"LITTLE Core Family : A53\n",
"CPU's Clusters IDs : [0, 1, 1, 0, 0, 0]\n",
"CPUs type : ['A53', 'A57', 'A57', 'A53', 'A53', 'A53']\n"
]
}
],
"source": [
"# Acces to many target specific information\n",
"print \"ABI : \", te.target.abi\n",
"print \"big Core Family : \", te.target.big_core\n",
"print \"LITTLE Core Family : \", te.target.little_core\n",
"print \"CPU's Clusters IDs : \", te.target.core_clusters\n",
"print \"CPUs type : \", te.target.core_names"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"big CPUs IDs : [1, 2]\n",
"LITTLE CPUs IDs : [0, 3, 4, 5]\n",
"big CPUs freqs : 450000\n",
"big CPUs governor : interactive\n"
]
}
],
"source": [
"# Access to big.LITTLE specific information\n",
"print \"big CPUs IDs : \", te.target.bl.bigs\n",
"print \"LITTLE CPUs IDs : \", te.target.bl.littles\n",
"print \"big CPUs freqs : {}\".format(te.target.bl.get_bigs_frequency())\n",
"print \"big CPUs governor : {}\".format(te.target.bl.get_bigs_governor())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Attributes: emeter (energy meter)\n",
"\n",
"In order to sample energy from the target:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"First read: {\n",
" \"BOARDBIG\": {\n",
" \"total\": 0.03712299999999935, \n",
" \"last\": 5.61159, \n",
" \"delta\": 0.019406000000000034\n",
" }, \n",
" \"BOARDLITTLE\": {\n",
" \"total\": 0.017602000000000118, \n",
" \"last\": 4.954883, \n",
" \"delta\": 0.008766999999999747\n",
" }\n",
"}\n",
"Second read: {\n",
" \"BOARDBIG\": {\n",
" \"total\": 0.11209199999999964, \n",
" \"last\": 5.686559, \n",
" \"delta\": 0.018203999999999887\n",
" }, \n",
" \"BOARDLITTLE\": {\n",
" \"total\": 0.06513600000000075, \n",
" \"last\": 5.002417, \n",
" \"delta\": 0.009789000000000492\n",
" }\n",
"}\n"
]
}
],
"source": [
"# Reset and sample energy counters\n",
"te.emeter.reset()\n",
"nrg = te.emeter.sample()\n",
"nrg = json.dumps(te.emeter.sample(), indent=4)\n",
"print \"First read: \", nrg\n",
"time.sleep(2)\n",
"nrg = te.emeter.sample()\n",
"nrg = json.dumps(te.emeter.sample(), indent=4)\n",
"print \"Second read: \", nrg"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Attribute: ftrace\n",
"\n",
"You can configure FTrace for a specific experiment using the following:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2016-12-12 11:58:44,190 INFO : TestEnv : Enabled tracepoints:\n",
"2016-12-12 11:58:44,192 INFO : TestEnv : cpu_idle\n",
"2016-12-12 11:58:44,193 INFO : TestEnv : cpu_capacity\n",
"2016-12-12 11:58:44,194 INFO : TestEnv : cpu_frequency\n",
"2016-12-12 11:58:44,194 INFO : TestEnv : sched_switch\n"
]
}
],
"source": [
"# Configure a specific set of events to trace\n",
"te.ftrace_conf(\n",
" { \n",
" \"events\" : [ \n",
" \"cpu_idle\", \n",
" \"cpu_capacity\",\n",
" \"cpu_frequency\",\n",
" \"sched_switch\",\n",
" ], \n",
" \"buffsize\" : 10240 \n",
" }\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Start/Stop a FTrace session\n",
"te.ftrace.start()\n",
"te.target.execute(\"uname -a\")\n",
"te.ftrace.stop()"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Collect and visualize the trace\n",
"trace_file = os.path.join(te.res_dir, 'trace.dat')\n",
"te.ftrace.get_trace(trace_file)\n",
"\n",
"# There might be a different display value on your machine\n",
"# Check by issuing \"echo $DISPLAY\" in the LISA VM\n",
"output = os.popen(\"DISPLAY=:10.0 kernelshark {}\".format(trace_file))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}