Przemyslaw Szczepaniak | d222d00 | 2018-10-15 11:44:03 +0100 | [diff] [blame] | 1 | Copyright 2017 The Android Open Source Project |
| 2 | |
| 3 | Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | you may not use this file except in compliance with the License. |
| 5 | You may obtain a copy of the License at |
| 6 | |
| 7 | http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | |
| 9 | Unless required by applicable law or agreed to in writing, software |
| 10 | distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | See the License for the specific language governing permissions and |
| 13 | limitations under the License. |
| 14 | ------------------------------------------------------------------ |
| 15 | |
| 16 | This directory contains files for the Android MLTS (Machine Learning |
| 17 | Test Suite). MLTS allows to evaluate NNAPI acceleration latency and accuracy |
| 18 | on an Android device, using few selected ML models and datesets. |
| 19 | |
| 20 | Models and datasets used description and licensing can be found in |
| 21 | platform/test/mlts/models/README.txt file. |
| 22 | |
| 23 | Usage: |
| 24 | * Connect a target device to your workstation, make sure it's |
| 25 | reachable through adb. Export target device ANDROID_SERIAL |
| 26 | environment variable if more than one device is connected. |
| 27 | * cd into android top-level source directory |
| 28 | > source build/envsetup.sh |
| 29 | > lunch aosp_arm-userdebug # Or aosp_arm64-userdebug if available. |
| 30 | > ./test/mlts/benchmark/build_and_run_benchmark.sh |
| 31 | * At the end of a benchmark run, its results will be |
| 32 | presented as html page, passed to xdg-open. |
| 33 | |
Stefano Galarraga | 4edc2ea | 2020-03-02 15:38:02 +0000 | [diff] [blame] | 34 | # Crash test |
Przemyslaw Szczepaniak | d222d00 | 2018-10-15 11:44:03 +0100 | [diff] [blame] | 35 | |
Stefano Galarraga | 4edc2ea | 2020-03-02 15:38:02 +0000 | [diff] [blame] | 36 | The MLTS suite contains a series of tests to validate the behaviour of the drivers under stress or |
| 37 | in corner case conditions. |
| 38 | |
| 39 | To run the tests use the specific targets available in the build_and_run_benchmark.sh script. |
Stefano Galarraga | 06ec72b | 2020-05-29 10:47:31 +0100 | [diff] [blame] | 40 | By default, every test gets run on each available accelerator in isolation. It is possible to filter the |
| 41 | accelerators to test against by invoking the build_and_run_benchmark.sh script with the option |
| 42 | -f (--filter-driver) and specifying a regular expression to filter the acccelerator names with. |
| 43 | It is also possible to run additional tests without specified target accelerator to let NNAPI |
| 44 | partition the model and assign the best available one(s) by using the |
| 45 | -r (--include-nnapi-reference) option. |
| 46 | |
Stefano Galarraga | f03d573 | 2020-06-08 19:18:24 +0100 | [diff] [blame] | 47 | Currently available tests are: |
Stefano Galarraga | 4edc2ea | 2020-03-02 15:38:02 +0000 | [diff] [blame] | 48 | |
Stefano Galarraga | f03d573 | 2020-06-08 19:18:24 +0100 | [diff] [blame] | 49 | * parallel-inference-stress: to test the behaviour of drivers with different amount of inference |
Stefano Galarraga | 4edc2ea | 2020-03-02 15:38:02 +0000 | [diff] [blame] | 50 | executed in parallel. Tests are running in a separate process so crashes can be detected and |
Stefano Galarraga | f03d573 | 2020-06-08 19:18:24 +0100 | [diff] [blame] | 51 | notified as test failures. |
Stefano Galarraga | 4edc2ea | 2020-03-02 15:38:02 +0000 | [diff] [blame] | 52 | |
Stefano Galarraga | f03d573 | 2020-06-08 19:18:24 +0100 | [diff] [blame] | 53 | * parallel-inference-stress-in-process: same as parallel-inference-stress but the tests are running |
Stefano Galarraga | 4edc2ea | 2020-03-02 15:38:02 +0000 | [diff] [blame] | 54 | in the same process of the test so in case of crash the testing app will crash too |
Stefano Galarraga | f03d573 | 2020-06-08 19:18:24 +0100 | [diff] [blame] | 55 | |
| 56 | * client-early-termination-stress: to test the resilience of device drivers to failing clients. |
| 57 | It spawns a separate process each running a set of parallel threads compiling different models. |
| 58 | The process is then forcibly terminated. The test validates that the targeted driver is not |
| 59 | crashing or hanging |
| 60 | |
| 61 | * multi-process-inference-stress: this extends the `parallel-inference-stress` running inference |
| 62 | on a single model in multiple processes and threads with different probabilities in client process |
| 63 | early termination |
| 64 | |
| 65 | * multi-process-model-load-stress: this extends the `parallel-inference-stress` running inference |
| 66 | on a single model in multiple processes and threads with different probabilities in client process |
Stefano Galarraga | 965e78e | 2020-05-12 17:11:54 +0100 | [diff] [blame] | 67 | early termination |
| 68 | |
| 69 | * memory-mapped-model-load-stress: runs a series of parallel model compilation with memory mapped |
Stefano Galarraga | 04f119a | 2020-05-13 19:19:32 +0100 | [diff] [blame] | 70 | TFLite models |
| 71 | |
Stefano Galarraga | f19cd01 | 2020-06-09 18:52:17 +0100 | [diff] [blame] | 72 | * model-load-random-stress: test compiling a large set of randomly generated models |
| 73 | |
Stefano Galarraga | 3edb25c | 2020-06-16 18:27:35 +0100 | [diff] [blame] | 74 | * inference-random-stress: test running a large set of randomly generated models |
| 75 | |
| 76 | * performance-degradation-stress: verifies that accelerator inference speed is not degrading over |
Stefano Galarraga | 57b6e4c | 2021-09-10 15:18:12 +0100 | [diff] [blame] | 77 | a certain threshold when running concurrent workload |
| 78 | |
| 79 | # Testing a NNAPI Support Library implementation |
| 80 | |
| 81 | All tests documented above can be run using a NNAPI Support Library implementation. |
Stefano Galarraga | fdbb44e | 2021-10-04 17:20:17 +0100 | [diff] [blame] | 82 | To do so you need to: |
| 83 | |
| 84 | - copy all the shared objects part of the libraries under the `sl_prebuilt` |
| 85 | folder |
| 86 | - Use the `-s` or `--use-nnapi-sl` option when running `build_and_run_benchmark.sh`. |
| 87 | |
| 88 | By default the system will use the sl_prebuilt/Android.bp.template to map |
| 89 | every library under sl_prebuilt to a native library to include in the APK. |
| 90 | The file is already configured for the Qualcomm NNAPI SL binaries. |
| 91 | If you have different libraries that the ones defined under sl_prebuilt/Android.bp.template |
| 92 | you should |
| 93 | |
| 94 | - configure a sl_prebuilt/Android.bp with the list of binaries you added. |
| 95 | You can use the sl_prebuilt/Android.bp.template file as an example template. |
| 96 | |
| 97 | - Set in in Android.mk the SL_LIBS variable with the list of drivers |