| { |
| "cells": [ |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "# EAS Testing - Recents Fling on Android" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "The goal of this experiment is to collect frame statistics while swiping up and down tabs of recently opened applications on a Nexus N5X running Android with an EAS kernel. This process is name **Recents Fling**. The Analysis phase will consist in comparing EAS with other schedulers, that is comparing *sched* governor with:\n", |
| "\n", |
| " - interactive\n", |
| " - performance\n", |
| " - powersave\n", |
| " - ondemand\n", |
| " \n", |
| "For this experiment it is recommended to open many applications so that we can swipe over more recently opened applications." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 1, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [], |
| "source": [ |
| "import logging\n", |
| "from conf import LisaLogging\n", |
| "LisaLogging.setup()" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 2, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [ |
| { |
| "name": "stdout", |
| "output_type": "stream", |
| "text": [ |
| "Populating the interactive namespace from numpy and matplotlib\n" |
| ] |
| } |
| ], |
| "source": [ |
| "%pylab inline\n", |
| "\n", |
| "import os\n", |
| "from time import sleep\n", |
| "\n", |
| "# Support to access the remote target\n", |
| "import devlib\n", |
| "from env import TestEnv\n", |
| "\n", |
| "from devlib.utils.android import adb_command\n", |
| "\n", |
| "# Support for trace events analysis\n", |
| "from trace import Trace\n", |
| "\n", |
| "# Suport for FTrace events parsing and visualization\n", |
| "import trappy" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "# Test Environment set up" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "Devlib requires the ANDROID_HOME environment variable configured to point to your local installation of the Android SDK. If you have not this variable configured in the shell used to start the notebook server, you need to run the next cell to define where your Android SDK is installed." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 3, |
| "metadata": { |
| "collapsed": true |
| }, |
| "outputs": [], |
| "source": [ |
| "import os\n", |
| "os.environ['ANDROID_HOME'] = '/ext/android-sdk-linux/'" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "In case more than one Android device are conencted to the host, you must specify the ID of the device you want to target in `my_target_conf`. Run `adb devices` on your host to get the ID." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 4, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [], |
| "source": [ |
| "# Setup a target configuration\n", |
| "my_conf = {\n", |
| " \n", |
| " # Target platform and board\n", |
| " \"platform\" : 'android',\n", |
| "\n", |
| " # Device ID\n", |
| " # \"device\" : \"0123456789abcdef\",\n", |
| "\n", |
| " # Folder where all the results will be collected\n", |
| " \"results_dir\" : \"Android_RecentsFling\",\n", |
| " \n", |
| " # Define devlib modules to load\n", |
| " \"modules\" : [\n", |
| " 'cpufreq' # enable CPUFreq support\n", |
| " ],\n", |
| "\n", |
| " # FTrace events to collect for all the tests configuration which have\n", |
| " # the \"ftrace\" flag enabled\n", |
| " \"ftrace\" : {\n", |
| " \"events\" : [\n", |
| " \"sched_switch\",\n", |
| " \"sched_load_avg_cpu\",\n", |
| " \"cpu_frequency\",\n", |
| " \"cpu_capacity\"\n", |
| " ],\n", |
| " \"buffsize\" : 10 * 1024,\n", |
| " },\n", |
| "\n", |
| " # Tools required by the experiments\n", |
| " \"tools\" : [ 'trace-cmd' ],\n", |
| "}" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 5, |
| "metadata": { |
| "collapsed": false, |
| "scrolled": true |
| }, |
| "outputs": [ |
| { |
| "name": "stderr", |
| "output_type": "stream", |
| "text": [ |
| "2016-04-27 10:03:27,438 INFO : Target - Using base path: /home/derkling/Code/lisa\n", |
| "2016-04-27 10:03:27,439 INFO : Target - Loading custom (inline) target configuration\n", |
| "2016-04-27 10:03:27,440 INFO : Target - Devlib modules to load: ['cpufreq']\n", |
| "2016-04-27 10:03:27,441 INFO : Target - Connecting Android target [DEFAULT]\n", |
| "2016-04-27 10:03:29,160 INFO : Target - Initializing target workdir:\n", |
| "2016-04-27 10:03:29,161 INFO : Target - /data/local/tmp/devlib-target\n", |
| "2016-04-27 10:03:32,592 INFO : Target - Topology:\n", |
| "2016-04-27 10:03:32,593 INFO : Target - [[0, 1], [2, 3]]\n", |
| "2016-04-27 10:03:33,947 WARNING : Event [sched_load_avg_cpu] not available for tracing\n", |
| "2016-04-27 10:03:33,949 WARNING : Event [cpu_capacity] not available for tracing\n", |
| "2016-04-27 10:03:33,950 INFO : FTrace - Enabled tracepoints:\n", |
| "2016-04-27 10:03:33,950 INFO : FTrace - sched_switch\n", |
| "2016-04-27 10:03:33,951 INFO : FTrace - sched_load_avg_cpu\n", |
| "2016-04-27 10:03:33,951 INFO : FTrace - cpu_frequency\n", |
| "2016-04-27 10:03:33,952 INFO : FTrace - cpu_capacity\n", |
| "2016-04-27 10:03:33,953 WARNING : TestEnv - Wipe previous contents of the results folder:\n", |
| "2016-04-27 10:03:33,953 WARNING : TestEnv - /home/derkling/Code/lisa/results/Android_RecentsFling\n", |
| "2016-04-27 10:03:33,963 INFO : TestEnv - Set results folder to:\n", |
| "2016-04-27 10:03:33,963 INFO : TestEnv - /home/derkling/Code/lisa/results/Android_RecentsFling\n", |
| "2016-04-27 10:03:33,964 INFO : TestEnv - Experiment results available also in:\n", |
| "2016-04-27 10:03:33,964 INFO : TestEnv - /home/derkling/Code/lisa/results_latest\n" |
| ] |
| } |
| ], |
| "source": [ |
| "# Initialize a test environment using:\n", |
| "te = TestEnv(my_conf)\n", |
| "target = te.target" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "# Support Functions" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "This set of support functions will help us running the benchmark using different CPUFreq governors." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 6, |
| "metadata": { |
| "collapsed": true |
| }, |
| "outputs": [], |
| "source": [ |
| "def set_performance():\n", |
| " target.cpufreq.set_all_governors('performance')\n", |
| "\n", |
| "def set_powersave():\n", |
| " target.cpufreq.set_all_governors('powersave')\n", |
| "\n", |
| "def set_interactive():\n", |
| " target.cpufreq.set_all_governors('interactive')\n", |
| "\n", |
| "def set_sched():\n", |
| " target.cpufreq.set_all_governors('sched')\n", |
| "\n", |
| "def set_ondemand():\n", |
| " target.cpufreq.set_all_governors('ondemand')\n", |
| " \n", |
| " for cpu in target.list_online_cpus():\n", |
| " tunables = target.cpufreq.get_governor_tunables(cpu)\n", |
| " target.cpufreq.set_governor_tunables(\n", |
| " cpu,\n", |
| " 'ondemand',\n", |
| " **{'sampling_rate' : tunables['sampling_rate_min']}\n", |
| " )" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 7, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [], |
| "source": [ |
| "# CPUFreq configurations to test\n", |
| "confs = {\n", |
| " 'performance' : {\n", |
| " 'label' : 'prf',\n", |
| " 'set' : set_performance,\n", |
| " },\n", |
| " #'powersave' : {\n", |
| " # 'label' : 'pws',\n", |
| " # 'set' : set_powersave,\n", |
| " #},\n", |
| " 'interactive' : {\n", |
| " 'label' : 'int',\n", |
| " 'set' : set_interactive,\n", |
| " },\n", |
| " #'sched' : {\n", |
| " # 'label' : 'sch',\n", |
| " # 'set' : set_sched,\n", |
| " #},\n", |
| " #'ondemand' : {\n", |
| " # 'label' : 'odm',\n", |
| " # 'set' : set_ondemand,\n", |
| " #}\n", |
| "}\n", |
| "\n", |
| "# The set of results for each comparison test\n", |
| "results = {}" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 8, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [], |
| "source": [ |
| "def open_apps(n):\n", |
| " \"\"\"\n", |
| " Open `n` apps on the device\n", |
| " \n", |
| " :param n: number of apps to open\n", |
| " :type n: int\n", |
| " \"\"\"\n", |
| " # Get a list of third-party packages\n", |
| " android_version = target.getprop('ro.build.version.release')\n", |
| " if android_version >= 'N':\n", |
| " packages = target.execute('cmd package list packages | cut -d: -f 2')\n", |
| " packages = packages.splitlines()\n", |
| " else:\n", |
| " packages = target.execute('pm list packages -3 | cut -d: -f 2')\n", |
| " packages = packages.splitlines()\n", |
| "\n", |
| " # As a safe fallback let's use a list of standard Android AOSP apps which are always available\n", |
| " if len(packages) < 8:\n", |
| " packages = [\n", |
| " 'com.android.messaging',\n", |
| " 'com.android.calendar',\n", |
| " 'com.android.settings',\n", |
| " 'com.android.calculator2',\n", |
| " 'com.android.email',\n", |
| " 'com.android.music',\n", |
| " 'com.android.deskclock',\n", |
| " 'com.android.contacts',\n", |
| " ]\n", |
| " \n", |
| " LAUNCH_CMD = 'monkey -p {} -c android.intent.category.LAUNCHER 1 '\n", |
| " \n", |
| " if n > len(packages):\n", |
| " n = len(packages)\n", |
| " \n", |
| " logging.info('Trying to open %d apps...', n)\n", |
| " started = 0\n", |
| " for app in packages:\n", |
| " logging.debug(' Launching %s', app)\n", |
| " try:\n", |
| " target.execute(LAUNCH_CMD.format(app))\n", |
| " started = started + 1\n", |
| " logging.info(' %2d starting %s...', started, app)\n", |
| " except Exception:\n", |
| " pass\n", |
| " if started >= n:\n", |
| " break\n", |
| " \n", |
| " # Close Recents\n", |
| " target.execute('input keyevent KEYCODE_HOME')\n" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 9, |
| "metadata": { |
| "collapsed": true |
| }, |
| "outputs": [], |
| "source": [ |
| "def recentsfling_run(exp_dir):\n", |
| " # Open Recents on the target device\n", |
| " target.execute('input keyevent KEYCODE_APP_SWITCH')\n", |
| " # Allow the activity to start\n", |
| " sleep(5)\n", |
| " # Reset framestats collection\n", |
| " target.execute('dumpsys gfxinfo --reset')\n", |
| " \n", |
| " w, h = target.screen_resolution\n", |
| " x = w/2\n", |
| " yl = int(0.2*h)\n", |
| " yh = int(0.9*h)\n", |
| " \n", |
| " logging.info('Start Swiping Recents')\n", |
| " for i in range(5):\n", |
| " # Simulate two fast UP and DOWN swipes\n", |
| " target.execute('input swipe {} {} {} {} 50'.format(x, yl, x, yh))\n", |
| " sleep(0.3)\n", |
| " target.execute('input swipe {} {} {} {} 50'.format(x, yh, x, yl))\n", |
| " sleep(0.7)\n", |
| " logging.info('Swiping Recents Completed')\n", |
| " \n", |
| " # Get frame stats\n", |
| " framestats_file = os.path.join(exp_dir, \"framestats.txt\")\n", |
| " adb_command(target.adb_name, 'shell dumpsys gfxinfo com.android.systemui > {}'.format(framestats_file))\n", |
| " \n", |
| " # Close Recents\n", |
| " target.execute('input keyevent KEYCODE_HOME')\n", |
| "\n", |
| " return framestats_file" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 10, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [], |
| "source": [ |
| "def experiment(governor, exp_dir):\n", |
| " os.system('mkdir -p {}'.format(exp_dir));\n", |
| "\n", |
| " logging.info('------------------------')\n", |
| " logging.info('Run workload using %s governor', governor)\n", |
| " confs[governor]['set']()\n", |
| " \n", |
| " # Start FTrace\n", |
| " te.ftrace.start()\n", |
| " \n", |
| " ### Run the benchmark ###\n", |
| " framestats_file = recentsfling_run(exp_dir)\n", |
| " \n", |
| " # Stop FTrace\n", |
| " te.ftrace.stop() \n", |
| "\n", |
| " # Collect and keep track of the trace\n", |
| " trace_file = os.path.join(exp_dir, 'trace.dat')\n", |
| " te.ftrace.get_trace(trace_file)\n", |
| " \n", |
| " # Parse trace\n", |
| " tr = Trace(te.platform, exp_dir,\n", |
| " events=my_conf['ftrace']['events'])\n", |
| " \n", |
| " # return all the experiment data\n", |
| " return {\n", |
| " 'dir' : exp_dir,\n", |
| " 'framestats_file' : framestats_file,\n", |
| " 'trace_file' : trace_file,\n", |
| " 'ftrace' : tr.ftrace,\n", |
| " 'trace' : tr\n", |
| " }" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "# Run Flinger" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "## Prepare Environment" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 11, |
| "metadata": { |
| "collapsed": true |
| }, |
| "outputs": [], |
| "source": [ |
| "N_APPS = 20" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 12, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [ |
| { |
| "name": "stderr", |
| "output_type": "stream", |
| "text": [ |
| "2016-04-27 10:04:15,735 INFO : Trying to open 20 apps...\n", |
| "2016-04-27 10:04:19,062 INFO : 1 starting com.android.documentsui...\n", |
| "2016-04-27 10:04:20,636 INFO : 2 starting com.android.quicksearchbox...\n", |
| "2016-04-27 10:04:22,291 INFO : 3 starting com.android.messaging...\n", |
| "2016-04-27 10:04:26,125 INFO : 4 starting com.android.contacts...\n", |
| "2016-04-27 10:04:30,385 INFO : 5 starting com.android.calendar...\n", |
| "2016-04-27 10:04:33,122 INFO : 6 starting com.htc.android.ssdtest...\n", |
| "2016-04-27 10:04:37,992 INFO : 7 starting com.android.dialer...\n", |
| "2016-04-27 10:04:38,592 INFO : 8 starting com.android.gallery3d...\n", |
| "2016-04-27 10:04:45,168 INFO : 9 starting com.android.settings...\n", |
| "2016-04-27 10:04:45,718 INFO : 10 starting com.android.calculator2...\n", |
| "2016-04-27 10:04:46,798 INFO : 11 starting com.android.email...\n", |
| "2016-04-27 10:04:47,373 INFO : 12 starting com.android.music...\n", |
| "2016-04-27 10:04:49,977 INFO : 13 starting com.android.deskclock...\n", |
| "2016-04-27 10:04:52,633 INFO : 14 starting com.android.development...\n" |
| ] |
| } |
| ], |
| "source": [ |
| "open_apps(N_APPS)\n", |
| "\n", |
| "# Give apps enough time to open\n", |
| "sleep(5)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": {}, |
| "source": [ |
| "## Run workload and collect traces" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 13, |
| "metadata": { |
| "collapsed": false, |
| "scrolled": false |
| }, |
| "outputs": [ |
| { |
| "name": "stderr", |
| "output_type": "stream", |
| "text": [ |
| "2016-04-27 10:05:16,261 INFO : ------------------------\n", |
| "2016-04-27 10:05:16,261 INFO : Run workload using performance governor\n", |
| "2016-04-27 10:05:23,910 INFO : Start Swiping Recents\n", |
| "2016-04-27 10:05:34,043 INFO : Swiping Recents Completed\n", |
| "2016-04-27 10:05:37,908 INFO : Parsing FTrace format...\n", |
| "2016-04-27 10:05:41,305 INFO : Collected events spans a 16.850 [s] time interval\n", |
| "2016-04-27 10:05:41,306 INFO : Set plots time range to (0.000000, 16.849682)[s]\n", |
| "2016-04-27 10:05:41,659 INFO : ------------------------\n", |
| "2016-04-27 10:05:41,660 INFO : Run workload using interactive governor\n", |
| "2016-04-27 10:05:49,387 INFO : Start Swiping Recents\n", |
| "2016-04-27 10:06:00,339 INFO : Swiping Recents Completed\n", |
| "2016-04-27 10:06:05,640 INFO : Parsing FTrace format...\n", |
| "2016-04-27 10:06:08,702 INFO : Collected events spans a 17.899 [s] time interval\n", |
| "2016-04-27 10:06:08,703 INFO : Set plots time range to (0.000000, 17.898734)[s]\n" |
| ] |
| } |
| ], |
| "source": [ |
| "# Unlock device screen (assume no password required)\n", |
| "target.execute('input keyevent 82')\n", |
| "\n", |
| "# Run the benchmark in all the configured governors\n", |
| "for governor in confs:\n", |
| " test_dir = os.path.join(te.res_dir, governor)\n", |
| " results[governor] = experiment(governor, test_dir)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "metadata": { |
| "collapsed": false |
| }, |
| "source": [ |
| "# UI Performance Analysis" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 14, |
| "metadata": { |
| "collapsed": false |
| }, |
| "outputs": [ |
| { |
| "name": "stdout", |
| "output_type": "stream", |
| "text": [ |
| "Frame Statistics for PERFORMANCE governor\n", |
| "Stats since: 31757612331ns\n", |
| "Total frames rendered: 1052\n", |
| "Janky frames: 124 (11.79%)\n", |
| "50th percentile: 9ms\n", |
| "90th percentile: 17ms\n", |
| "95th percentile: 23ms\n", |
| "99th percentile: 34ms\n", |
| "\n", |
| "Frame Statistics for INTERACTIVE governor\n", |
| "Stats since: 31757612331ns\n", |
| "Total frames rendered: 1527\n", |
| "Janky frames: 230 (15.06%)\n", |
| "50th percentile: 10ms\n", |
| "90th percentile: 19ms\n", |
| "95th percentile: 23ms\n", |
| "99th percentile: 34ms\n", |
| "\n" |
| ] |
| } |
| ], |
| "source": [ |
| "for governor in confs:\n", |
| " framestats_file = results[governor]['framestats_file']\n", |
| " print \"Frame Statistics for {} governor\".format(governor.upper())\n", |
| " !sed '/Stats since/,/99th/!d;/99th/q' $framestats_file\n", |
| " print \"\"" |
| ] |
| } |
| ], |
| "metadata": { |
| "kernelspec": { |
| "display_name": "Python 2", |
| "language": "python", |
| "name": "python2" |
| }, |
| "language_info": { |
| "codemirror_mode": { |
| "name": "ipython", |
| "version": 2 |
| }, |
| "file_extension": ".py", |
| "mimetype": "text/x-python", |
| "name": "python", |
| "nbconvert_exporter": "python", |
| "pygments_lexer": "ipython2", |
| "version": "2.7.9" |
| }, |
| "toc": { |
| "toc_cell": false, |
| "toc_number_sections": true, |
| "toc_threshold": 6, |
| "toc_window_display": false |
| } |
| }, |
| "nbformat": 4, |
| "nbformat_minor": 0 |
| } |