Atest

Atest is a command line tool that allows users to build, install and run Android tests locally. This markdown will explain how to use atest on the commandline to run android tests.

For instructions on writing tests go here. Importantly, when writing your test's build script file (Android.mk), make sure to include the variable LOCAL_COMPATIBILITY_SUITE. A good default to use for it is device-test.

Curious about how atest works? Want to add a feature but not sure where to begin? Go here. Just want to learn about the overall structure? Go here.

Table of Contents

1. Environment Setup
2. Basic Usage
3. Identifying Tests
4. Specifying Steps: Build, Install or Run
5. Running Specific Methods
6. Running Multiple Classes
7. Detecting Metrics Regression

Environment Setup

Before you can run atest, you first have to setup your environment.

1. Run envsetup.sh

From the root of the android source checkout, run the following command:

$ source build/envsetup.sh

2. Run lunch

Run the $ lunch command to bring up a "menu" of supported devices. Find the device that matches the device you have connected locally and run that command.

For instance, if you have an ARM device connected, you would run the following command:

$ lunch aosp_arm64-eng

This will set various environment variables necessary for running atest. It will also add the atest command to your $PATH.

Basic Usage

Atest commands take the following form:

      atest <optional arguments> <tests to run>

<optional arguments>

More information about -b, -i and -t can be found below under Specifying Steps: Build, Install or Run.

<tests to run>

The positional argument <tests to run> should be a reference to one or more of the tests you'd like to run. Multiple tests can be run by separating test references with spaces.

Usage template:   atest <reference_to_test_1> <reference_to_test_2>

Here are some simple examples:

      atest FrameworksServicesTests
      atest example/reboot
      atest FrameworksServicesTests CtsJankDeviceTestCases
      atest FrameworksServicesTests:ScreenDecorWindowTests

More information on how to reference a test can be found under Identifying Tests.

Identifying Tests

A <reference_to_test> can be satisfied by the test's Module Name, Module:Class, Class Name, TF Integration Test, File Path or Package Name.

Module Name

Identifying a test by its module name will run the entire module. Input the name as it appears in the LOCAL_MODULE or LOCAL_PACKAGE_NAME variables in that test's Android.mk or Android.bp file.

Note: Use TF Integration Test to run non-module tests integrated directly into TradeFed.

Examples:
      atest FrameworksServicesTests
      atest CtsJankDeviceTestCases

Module:Class

Identifying a test by its class name will run just the tests in that class and not the whole module. Module:Class is the preferred way to run a single class. Module is the same as described above. Class is the name of the test class in the .java file. It can either be the fully qualified class name or just the basic name.

Examples:
       atest FrameworksServicesTests:ScreenDecorWindowTests
       atest FrameworksServicesTests:com.android.server.wm.ScreenDecorWindowTests
       atest CtsJankDeviceTestCases:CtsDeviceJankUi

Class Name

A single class can also be run by referencing the class name without the module name.

Examples:
      atest ScreenDecorWindowTests
      atest CtsDeviceJankUi

However, this will take more time than the equivalent Module:Class reference, so we suggest using a Module:Class reference whenever possible. Examples below are ordered by performance from the fastest to the slowest:

Examples:
       atest FrameworksServicesTests:com.android.server.wm.ScreenDecorWindowTests
       atest FrameworksServicesTests:ScreenDecorWindowTests
       atest ScreenDecorWindowTests

TF Integration Test

To run tests that are integrated directly into TradeFed (non-modules), input the name as it appears in the output of the "tradefed.sh list configs" cmd.

Example:
      atest example/reboot  (runs this test)
      atest native-benchmark  (runs this test)

File Path

Both module-based tests and integration-based tests can be run by inputting the path to their test file or dir as appropriate. A single class can also be run by inputting the path to the class's java file. Both relative and absolute paths are supported.

Example - 2 ways to run the CtsJankDeviceTestCases module via path:

1. run module from android <repo root>: atest cts/tests/jank
2. from android <repo root>/cts/tests/jank: atest .

Example - run a specific class within CtsJankDeviceTestCases module via path:
      From android <repo root>: atest cts/tests/jank/src/android/jank/cts/ui/CtsDeviceJankUi.java

Example - run an integration test via path:
      From android <repo root>: atest tools/tradefederation/contrib/res/config/example/reboot.xml

Package Name

Atest supports searching tests from package name as well.

Examples:
      atest com.android.server.wm
      atest android.jank.cts

Specifying Steps: Build, Install or Run

The -b, -i and -t options allow you to specify which steps you want to run. If none of those options are given, then all steps are run. If any of these options are provided then only the listed steps are run.

Note: -i alone is not currently support and can only be included with -t. Both -b and -t can be run alone.

• Build targets only: atest -b <test>
• Run tests only: atest -t <test>
• Install apk and run tests: atest -it <test>
• Build and run, but don't install: atest -bt <test>

Atest now has the ability to force a test to skip its cleanup/teardown step. Many tests, e.g. CTS, cleanup the device after the test is run, so trying to rerun your test with -t will fail without having the --disable-teardown parameter. Use -d before -t to skip the test clean up step and test iteratively.

      atest -d <test>
      atest -t <test>

Note that -t disables both setup/install and teardown/cleanup of the device. So you can continue to rerun your test with just atest -t <test> as many times as you want.

Running Specific Methods

It is possible to run only specific methods within a test class. While the whole module will still need to be built, this will greatly speed up the time needed to run the tests. To run only specific methods, identify the class in any of the ways supported for identifying a class (MODULE:CLASS, FILE PATH, etc) and then append the name of the method or method using the following template:

      <reference_to_class>#<method1>

Multiple methods can be specified with commas:

      <reference_to_class>#<method1>,<method2>,<method3>

Examples:
       atest com.android.server.wm.ScreenDecorWindowTests#testMultipleDecors
       atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval

Here are the two preferred ways to run a single method, we're specifying the testFlagChange method. These are preferred over just the class name, because specifying the module or the java file location allows atest to find the test much faster:

1. atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange
2. From android <repo root>:
   atest frameworks/base/services/tests/servicestests/src/com/android/server/wm/ScreenDecorWindowTests.java#testFlagChange

Multiple methods can be ran from different classes and modules:
       atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval ScreenDecorWindowTests#testMultipleDecors

Running Multiple Classes

To run multiple classes, deliminate them with spaces just like you would when running multiple tests. Atest will handle building and running classes in the most efficient way possible, so specifying a subset of classes in a module will improve performance over running the whole module.

Examples:

• two classes in same module:
         atest FrameworksServicesTests:ScreenDecorWindowTests FrameworksServicesTests:DimmerTests

• two classes, different modules:
         atest FrameworksServicesTests:ScreenDecorWindowTests CtsJankDeviceTestCases:CtsDeviceJankUi

Running Native Tests

Atest can be used to run native tests as well.

Examples:

• input tests:
         atest -a libinput_tests inputflinger_tests

Use the option '-a' to run the tests for all available device architectures, which in this example would be armeabi-v7a (ARM 32-bit) and arm64-v8a (ARM 64-bit).

Detecting Metrics Regression

Generate pre-patch or post-patch metrics without running regression detection:

Examples:
      atest <test> --generate-baseline <optional iter>
      atest <test> --generate-new-metrics <optional iter>

Local regression detection can be run in three options:

1. Provide a folder containing baseline (pre-patch) metrics (generated previously). Atest will run the tests n (default 5) iterations, generate a new set of post-patch metrics, and compare those against existing metrics.

   Example:
          atest <test> --detect-regression </path/to/baseline> --generate-new-metrics <optional iter>

2. Provide a folder containing post-patch metrics (generated previously). Atest will run the tests n (default 5) iterations, generate a new set of pre-patch metrics, and compare those against those provided. Note: the developer needs to revert the device/tests to pre-patch state to generate baseline metrics.

   Example:
          atest <test> --detect-regression </path/to/new> --generate-baseline <optional iter>

3. Provide 2 folders containing both pre-patch and post-patch metrics. Atest will run no tests but the regression detection algorithm.

   Example:
          atest --detect-regression </path/to/baseline> </path/to/new>