Atest is a command line tool that allows users to build, install and run Android tests locally. This markdown will explain how to use atest on the commandline to run android tests.
For instructions on writing tests go here. Importantly, when writing your test's build script file (Android.mk), make sure to include the variable LOCAL_COMPATIBILITY_SUITE. A good default to use for it is device-test.
Before you can run atest, you first have to setup your environment.
From the root of the android source checkout, run the following command:
$ source build/envsetup.sh
Run the $ lunch command to bring up a "menu" of supported devices. Find the device that matches the device you have connected locally and run that command.
For instance, if you have a marlin device connected, you would run the following command:
$ lunch marlin-userdebug
This will set various environment variables necessary for running atest. It will also add the atest command to your $PATH.
Atest commands take the following form:
atest <optional arguments> <tests to run>
-b --build Build test targets.
-i --install Install test artifacts (APKs) on device.
-t --test Run the tests.
-w --wait-for-debugger Only for instrumentation tests. Waits for debugger prior to execution.
-v --verbose Display DEBUG level logging.
-h --help show this help message and exit
More information about -b, -i and -t can be found below under Specifying Steps: Build, Install or Run.
The positional argument <tests to run> should be a reference to one or more of the tests you'd like to run. Multiple tests can be run by separating test references with spaces.
Usage form: atest <reference_to_test_1> <reference_to_test_2>
Here are some simple examples:
atest FrameworksServicesTests
atest example/reboot
atest FrameworksServicesTests CtsJankDeviceTestCases
atest FrameworksServicesTests:ScreenDecorWindowTests
More information on how to reference a test can be found under Identifying Tests.
You can identify a test by inputting it's Module Name, Module:Class, Class Name, TF Integration Test or File Path.
Identifying a test by its module name will run the entire module. Input the name as it appears in the LOCAL_MODULE or LOCAL_PACKAGE_NAME variables in that test's Android.mk or Android.bp file.
Note: Use TF Integration Test below to run non-module tests integrated directly into TradeFed.
Examples:
atest FrameworksServicesTests
atest CtsJankDeviceTestCases
Identifying a test by its class name will run just the tests in that class and not the whole module. Module:Class is the preferred way to run a single class. Module is the same as described above. Class is the name of the test class in the .java file. It can either be the fully qualified class name or just the basic name.
Examples:
atest FrameworksServicesTests:ScreenDecorWindowTests
atest PtsBatteryTestCases:com.google.android.battery.pts.BatteryTest
atest CtsJankDeviceTestCases:CtsDeviceJankUi
A single class can also be run by referencing the class name without the module name. However, this will take more time than the equivalent Module:Class reference, so we suggest using a Module:Class reference whenever possible.
Examples:
atest ScreenDecorWindowTests
atest com.google.android.battery.pts.BatteryTest
atest CtsDeviceJankUi
To run tests that are integrated directly into TradeFed (non-modules), input the name as it appears in the output of the "tradefed.sh list configs" cmd.
Example:
atest example/reboot (runs this test)
atest native-benchmark (runs this test)
Both module-based tests and integration-based tests can be run by inputting the path to their test file or dir as appropriate. A single class can also be run by inputting the path to the class's java file. Both relative and absolute paths are supported.
Example - 4 ways to run the CtsJankDeviceTestCases module via path:
atest cts/tests/jankatest cts/tests/jank/Android.mkatest .atest .Example - run a specific class within CtsJankDeviceTestCases module via path:
From android repo root: atest cts/tests/jank/src/android/jank/cts/ui/CtsDeviceJankUi.java
Example - run an integration test via path:
From android repo root: atest tools/tradefederation/contrib/res/config/example/reboot.xml
The -b, -i and -t options allow you to specify which steps you want to run. If none of those options are given, then all steps are run. If any of these options are provided then only the listed steps are run.
Note: -i alone is not currently support and can only be included with -t. Both -b and -t can be run alone.
atest -b <test>atest -t <test> atest -it <test> atest -bt <test> It is possible to run only specific methods within a test class. While the whole module will still need to be built, this will greatly speed up the time needed to run the tests. To run only specific methods, identify the class in any of the ways supported for identifying a class (MODULE:CLASS, FILE PATH, etc) and then append the name of the method or method using the following template:
<reference_to_class>#<method1>
Multiple methods can be specified with commas:
<reference_to_class>#<method1>,<method2>,<method3>
Examples:
atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval
atest com.google.android.battery.pts.BatteryTest#testDischarge
To run multiple classes, deliminate them with spaces just like you would when running multiple tests. Atest will handle building and running classes in the most efficient way possible, so specifying a subset of classes in a module will improve performance over running the whole module.
Examples:
atest FrameworksServicesTests:ScreenDecorWindowTests FrameworksServicesTest:DimmerTests
atest FrameworksServicesTests:ScreenDecorWindowTests CtsJankDeviceTestCases:CtsDeviceJankUi
Generate pre-patch or post-patch metrics without running regression detection:
Example:
atest <test> --generate-baseline <optional iter>
atest <test> --generate-new-metrics <optional iter>
Local regression detection can be run in three options:
Provide a folder containing baseline (pre-patch) metrics (generated previously). Atest will run the tests n (default 5) iterations, generate a new set of post-patch metrics, and compare those against existing metrics.
Example:
atest <test> --detect-regression </path/to/baseline> --generate-new-metrics <optional iter>
Provide a folder containing post-patch metrics (generated previously). Atest will run the tests n (default 5) iterations, generate a new set of pre-patch metrics, and compare those against those provided. Note: the developer needs to revert the device/tests to pre-patch state to generate baseline metrics.
Example:
atest <test> --detect-regression </path/to/new> --generate-baseline <optional iter>
Provide 2 folders containing both pre-patch and post-patch metrics. Atest will run no tests but the regression detection algorithm.
Example:
atest --detect-regression </path/to/baseline> </path/to/new>
Here are the two preferred ways to run a single method, we're specifying the testFlagChange method. These are preferred over just the class name, because specifying the module or the java file location allows atest to find the test much faster:
atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChangeatest frameworks/base/services/tests/servicestests/src/com/android/server/wm/ScreenDecorWindowTests.java#testFlagChangeHere we run multiple methods from different classes and modules.
atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval PtsBatteryTestCases:BatteryTest#testDischarge