Ruben Brunk | 370e243 | 2014-10-14 18:33:23 -0700 | [diff] [blame] | 1 | # Copyright 2013 The Android Open Source Project |
| 2 | # |
| 3 | # Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | # you may not use this file except in compliance with the License. |
| 5 | # You may obtain a copy of the License at |
| 6 | # |
| 7 | # http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | # |
| 9 | # Unless required by applicable law or agreed to in writing, software |
| 10 | # distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | # See the License for the specific language governing permissions and |
| 13 | # limitations under the License. |
| 14 | |
| 15 | |
| 16 | Android Camera Imaging Test Suite (ITS) |
| 17 | ======================================= |
| 18 | |
| 19 | 1. Introduction |
| 20 | --------------- |
| 21 | |
| 22 | The ITS is a framework for running tests on the images produced by an Android |
| 23 | camera. The general goal of each test is to configure the camera in a desired |
| 24 | manner and capture one or more shots, and then examine the shots to see if |
| 25 | they contain the expected image data. Many of the tests will require that the |
| 26 | camera is pointed at a specific target chart or be illuminated at a specific |
| 27 | intensity. |
| 28 | |
| 29 | 2. Setup |
| 30 | -------- |
| 31 | |
| 32 | There are two components to the ITS: |
| 33 | 1. The Android device running CtsVerifier.apk. |
| 34 | 2. A host machine connected to the Android device that runs Python tests. |
| 35 | |
| 36 | 2.1. Device setup |
| 37 | ----------------- |
| 38 | |
| 39 | Build and install CtsVerifier.apk for your device. Along with the regular |
| 40 | CTS verifier tests, this includes the ITS service used by the ITS python |
| 41 | scripts to capture image data. After setting up your shell for Android builds, |
| 42 | from the cts/apps/CameraITS directory run the following commands: |
| 43 | |
| 44 | cd <BUILD_PATH>/cts/apps/CtsVerifier |
| 45 | mma -j32 |
| 46 | adb install -r <YOUR_OUTPUT_PATH>/CtsVerifier.apk |
| 47 | |
| 48 | using whatever path is appropriate to your output CtsVerifier.apk file. |
| 49 | |
| 50 | Note: If you are installing this from CTS Verifier bundle |
| 51 | (android-cts-verifier.zip) rather than from a full Android source tree, both |
| 52 | the CtsVerifier.apk and CameraITS directory can be found at the top level of |
| 53 | the zipped android-cts-verifier folder. |
| 54 | |
| 55 | 2.2. Host PC setup |
| 56 | ------------------ |
| 57 | |
| 58 | The first pre-requisite is the Android SDK, as adb is used to communicate with |
| 59 | the device. |
| 60 | |
| 61 | The test framework is based on Python on the host machine. It requires |
| 62 | Python 2.7 and the scipy/numpy stack, including the Python Imaging Library. |
| 63 | |
| 64 | (For Ubuntu users) |
| 65 | |
| 66 | sudo apt-get install python-numpy python-scipy python-matplotlib |
| 67 | |
| 68 | (For other users) |
| 69 | |
| 70 | All of these pieces can be installed on your host machine separately, |
| 71 | however it is highly recommended to install a bundled distribution of |
| 72 | Python that comes with these modules. Some different bundles are listed |
| 73 | here: |
| 74 | |
| 75 | http://www.scipy.org/install.html |
| 76 | |
| 77 | Of these, Anaconda has been verified to work with these scripts, and it is |
| 78 | available on Mac, Linux, and Windows from here: |
| 79 | |
| 80 | http://continuum.io/downloads |
| 81 | |
| 82 | Note that the Anaconda python executable's directory must be at the front of |
| 83 | your PATH environment variable, assuming that you are using this Python |
| 84 | distribution. The Anaconda installer may set this up for you automatically. |
| 85 | |
| 86 | Once your Python installation is ready, set up the test environment. |
| 87 | |
| 88 | 2.2.1. Linux + Mac OS X |
| 89 | ----------------------- |
| 90 | |
| 91 | On Linux or Mac OS X, run the following command (in a terminal) from the |
| 92 | cts/apps/CameraITS directory, from a bash shell: |
| 93 | |
| 94 | source build/envsetup.sh |
| 95 | |
| 96 | This will do some basic sanity checks on your Python installation, and set up |
| 97 | the PYTHONPATH environment variable. |
| 98 | |
| 99 | 2.2.2. Windows |
| 100 | -------------- |
| 101 | |
| 102 | On Windows, the bash script won't run (unless you have cygwin (which has not |
| 103 | been tested)), but all you need to do is set your PYTHONPATH environment |
| 104 | variable in your shell to point to the cts/apps/CameraITS/pymodules directory, |
| 105 | giving an absolute path. Without this, you'll get "import" errors when running |
| 106 | the test scripts. |
| 107 | |
| 108 | 3. Python framework overview |
| 109 | ---------------------------- |
| 110 | |
| 111 | The Python modules are under the pymodules directory, in the "its" package. |
| 112 | |
| 113 | * its.device: encapsulates communication with ITS service included in the |
| 114 | CtsVerifier.apk running on the device |
| 115 | * its.objects: contains a collection of functions for creating Python objects |
| 116 | corresponding to the Java objects which the ITS service uses |
| 117 | * its.image: contains a collection of functions (built on numpy arrays) for |
| 118 | processing captured images |
| 119 | * its.error: the exception/error class used in this framework |
| 120 | * its.target: functions to set and measure the exposure level to use for |
| 121 | manual shots in tests, to ensure that the images are exposed well for the |
| 122 | target scene |
| 123 | * its.dng: functions to work with DNG metadata |
| 124 | |
| 125 | All of these module have associated unit tests; to run the unit tests, execute |
| 126 | the modules (rather than importing them). |
| 127 | |
| 128 | 3.1. Device control |
| 129 | ------------------- |
| 130 | |
| 131 | The its.device.ItsSession class encapsulates a session with a connected device |
| 132 | under test (which is running CtsVerifier.apk). The session is over TCP, which is |
| 133 | forwarded over adb. |
| 134 | |
| 135 | As an overview, the ItsSession.do_capture() function takes a Python dictionary |
| 136 | object as an argument, converts that object to JSON, and sends it to the |
| 137 | device over tcp which then deserializes from the JSON object representation to |
| 138 | Camera2 Java objects (CaptureRequests) which are used to specify one or more |
| 139 | captures. Once the captures are complete, the resultant images are copied back |
| 140 | to the host machine (over tcp again), along with JSON representations of the |
| 141 | CaptureResult and other objects that describe the shot that was actually taken. |
| 142 | |
| 143 | The Python capture request object(s) can contain key/value entries corresponding |
| 144 | to any of the Java CaptureRequest object fields. |
| 145 | |
| 146 | The output surface's width, height, and format can also be specified. Currently |
| 147 | supported formats are "jpg", "raw", "raw10", "dng", and "yuv", where "yuv" is |
| 148 | YUV420 fully planar. The default output surface is a full sensor YUV420 frame. |
| 149 | |
| 150 | The metadata that is returned along with the captured images is also in JSON |
| 151 | format, serialized from the CaptureRequest and CaptureResult objects that were |
| 152 | passed to the capture listener, as well as the CameraProperties object. |
| 153 | |
| 154 | 3.2. Image processing and analysis |
| 155 | ---------------------------------- |
| 156 | |
| 157 | The its.image module is a collection of Python functions, built on top of numpy |
| 158 | arrays, for manipulating captured images. Some functions of note include: |
| 159 | |
| 160 | load_yuv420_to_rgb_image |
| 161 | apply_lut_to_image |
| 162 | apply_matrix_to_image |
| 163 | write_image |
| 164 | |
| 165 | The scripts in the tests directory make use of these modules. |
| 166 | |
| 167 | Note that it's important to do heavy image processing using the efficient numpy |
| 168 | ndarray operations, rather than writing complex loops in standard Python to |
| 169 | process pixels. Refer to online docs and examples of numpy for information on |
| 170 | this. |
| 171 | |
| 172 | 3.3. Tests |
| 173 | ---------- |
| 174 | |
| 175 | The tests directory contains a number of self-contained test scripts. All |
| 176 | tests should pass if the tree is in a good state. |
| 177 | |
| 178 | Most of the tests save various files in the current directory. To have all the |
| 179 | output files put into a separate directory, run the script from that directory, |
| 180 | for example: |
| 181 | |
| 182 | mkdir out |
| 183 | cd out |
| 184 | python ../tests/scene1/test_linearity.py |
| 185 | |
| 186 | Any test can be specified to reboot the camera prior to capturing any shots, by |
| 187 | adding a "reboot" or "reboot=N" command line argument, where N is the number of |
| 188 | seconds to wait after rebooting the device before sending any commands; the |
| 189 | default is 30 seconds. |
| 190 | |
| 191 | python tests/scene1/test_linearity.py reboot |
| 192 | python tests/scene1/test_linearity.py reboot=20 |
| 193 | |
| 194 | It's possible that a test could leave the camera in a bad state, in particular |
| 195 | if there are any bugs in the HAL or the camera framework. Rebooting the device |
| 196 | can be used to get it into a known clean state again. |
| 197 | |
| 198 | Each test assumes some sort of target or scene. There are multiple scene<N> |
| 199 | folders under the tests directory, and each contains a README file which |
| 200 | describes the scene for the scripts in that folder. |
| 201 | |
| 202 | By default, camera device id=0 is opened when the script connects to the unit, |
| 203 | however this can be specified by adding a "camera=1" or similar argument to |
| 204 | the script command line. On a typical device, camera=0 is the main (rear) |
| 205 | camera, and camera=1 is the front-facing camera. |
| 206 | |
| 207 | python tests/scene1/test_linearity.py camera=1 |
| 208 | |
| 209 | The tools/run_all_tests.py script should be executed from the top-level |
| 210 | CameraITS directory, and it will run all of the tests in an automated fashion, |
| 211 | saving the generated output files along with the stdout and stderr dumps to |
| 212 | a temporary directory. |
| 213 | |
| 214 | python tools/run_all_tests.py |
| 215 | |
| 216 | This can be run with the "noinit" argument, and in general any args provided |
| 217 | to this command line will be passed to each script as it is executed. |
| 218 | |
| 219 | The tests/inprog directory contains a mix of unfinished, in-progress, and |
| 220 | incomplete tests. These may or may not be useful in testing a HAL impl., |
| 221 | and as these tests are copmleted they will be moved into the scene<N> folders. |
| 222 | |
| 223 | When running individual tests from the command line (as in the examples here), |
| 224 | each test run will ensure that the ITS service is running on the device and is |
| 225 | ready to accept TCP connections. When using a separate test harness to control |
| 226 | this infrastructure, the "noinit" command line argument can be provided to |
| 227 | skip this step; in this case, the test will just try to open a socket to the |
| 228 | service on the device, and will fail if it's not running and ready. |
| 229 | |
| 230 | python tests/scene1/test_linearity.py noinit |
| 231 | |
| 232 | 3.4. Target exposure |
| 233 | -------------------- |
| 234 | |
| 235 | The tools/config.py script is a wrapper for the its.target module, which is |
| 236 | used to set an exposure level based on the scene that the camera is imaging. |
| 237 | The purpose of this is to be able to have tests which use hard-coded manual |
| 238 | exposure controls, while at the same time ensuring that the captured images |
| 239 | are properly exposed for the test (and aren't clamped to white or black). |
| 240 | |
| 241 | If no argument is provided, the script will use the camera to measure the |
| 242 | scene to determine the exposure level. An argument can be provided to hard- |
| 243 | code the exposure level. |
| 244 | |
| 245 | python tools/config.py |
| 246 | python tools/config.py 16531519962 |
| 247 | |
| 248 | This creates a file named its.target.cfg in the current directory, storing the |
| 249 | target exposure level. Tests that use the its.target module will be reusing |
| 250 | this value, if they are run from the same directory and if they contain the |
| 251 | "target" command line argument: |
| 252 | |
| 253 | python tests/scene1/test_linearity.py target |
| 254 | |
| 255 | If the "target" argument isn't present, then the script won't use any cached |
| 256 | its.target.cfg values that may be present in the current directory. |
| 257 | |
| 258 | 3.5. Docs |
| 259 | --------- |
| 260 | |
| 261 | The pydoc tool can generate HTML docs for the ITS Python modules, using the |
| 262 | following command (run after PYTHONPATH has been set up as described above): |
| 263 | |
| 264 | pydoc -w its its.device its.image its.error its.objects its.dng its.target |
| 265 | |
| 266 | There is a tutorial script in the tests folder (named tutorial.py). It |
| 267 | illustrates a number of the its.image and its.device primitives, and shows |
| 268 | how to work with image data in general using this infrastructure. (Its code |
| 269 | is commented with explanatory remarks.) |
| 270 | |
| 271 | python tests/tutorial.py |
| 272 | |
| 273 | 3.6. List of command line args |
| 274 | --------------------------------- |
| 275 | |
| 276 | The above doc sections describe the following command line arguments that may |
| 277 | be provided when running a test: |
| 278 | |
| 279 | reboot |
| 280 | reboot=N |
| 281 | target |
| 282 | noinit |
| 283 | camera=N |
| 284 | |
| 285 | 4. Known issues |
| 286 | --------------- |
| 287 | |
| 288 | The Python test scripts don't work if multiple devices are connected to the |
| 289 | host machine; currently, the its.device module uses a simplistic "adb -d" |
| 290 | approach to communicating with the device, assuming that there is only one |
| 291 | device connected. Fixing this is a TODO. |
| 292 | |