Merge "Create Tutorial page"
diff --git a/src/tech/test_infra/tradefed/sidebar3.md b/src/tech/test_infra/tradefed/sidebar3.md
index 574fe47..b525258 100644
--- a/src/tech/test_infra/tradefed/sidebar3.md
+++ b/src/tech/test_infra/tradefed/sidebar3.md
@@ -1,4 +1,4 @@
# Trade Federation Docs #
- [Getting Started](getting_started.html)
-
+- [Tutorial](tutorial.html)
diff --git a/src/tech/test_infra/tradefed/tutorial.md b/src/tech/test_infra/tradefed/tutorial.md
new file mode 100644
index 0000000..e9b5e5f
--- /dev/null
+++ b/src/tech/test_infra/tradefed/tutorial.md
@@ -0,0 +1,424 @@
+<!--
+ Copyright 2012 The Android Open Source Project
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Tutorial
+
+This tutorial guides you through the construction of a "hello world" Trade Federation test
+configuration, and gives you a hands-on introduction to the Trade Federation framework. Starting
+from the Tf development environment, it guides you through the process of creating a simple Trade
+Federation config and gradually adding more features to it.
+
+The tutorial presents the TF test development process as a set of exercises, each consisting of
+several steps. The exercises demonstrate how to gradually build and refine your configuration, and
+provide all the sample code you need to complete the test configuration.
+
+When you are finished with the tutorial, you will have created a functioning TF configuration and
+will have learned many of the most important concepts in the TF framework.
+
+
+## Set up TradeFederation development environment
+
+See (FIXME: link) for how to setup the development environment. The rest of this tutorial assumes you have a shell open that has been initialized to the TradeFederation environment.
+
+For simplicity, this tutorial will illustrate adding a configuration and its classes to the TradeFederation framework core library. Later tutorials/documentation will show how to create your own library that extends TradeFederation.
+
+
+## Creating a test class
+
+Lets create a hello world test that just dumps a message to stdout. A TradeFederation test must
+implement the (FIXME: link) IRemoteTest interface.
+
+Here's an implementation for the HelloWorldTest:
+
+ package com.android.tradefed.example;
+
+ import com.android.tradefed.device.DeviceNotAvailableException;
+ import com.android.tradefed.result.ITestInvocationListener;
+ import com.android.tradefed.testtype.IRemoteTest;
+
+
+ public class HelloWorldTest implements IRemoteTest {
+ @Override
+ public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
+ System.out.println("Hello, TF World!");
+ }
+ }
+
+FIXME: prod-tests
+Save this sample code to
+`<git home>/tools/tradefederation/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java`
+and rebuild tradefed from your shell:
+
+ m -j6
+
+If the build does not succeed, please consult the (FIXME: link)Development Environment page to
+ensure you did not miss any steps.
+
+
+## Creating a configuration
+
+Trade Federation tests are defined in a "Configuration". A Configuration is an XML file that
+instructs tradefed which test (or set of tests) to run.
+
+Lets create a new Configuration for our HelloWorldTest.
+
+ <configuration description="Runs the hello world test">
+ <test class="com.android.tradefed.example.HelloWorldTest" />
+ </configuration>
+
+TF will parse the Configuration XML file, load the specified class using reflection, instantiate it,
+cast it to a IRemoteTest, and call its 'run' method.
+
+Note that we've specified the full class name of the HelloWorldTest. Save this data to a
+`helloworld.xml` file anywhere on your local filesystem (eg `/tmp/helloworld.xml`).
+
+
+## Running the configuration
+
+From your shell, launch the tradefed console
+
+ $ ./tradefed.sh
+
+Ensure a device is connected to the host machine that is visible to tradefed
+
+ tf> list devices
+
+Configurations can be run using the `run <config>` console command. Try this now
+
+FIXME: redo this
+
+ tf> run /tmp/helloworld.xml
+ 05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC
+ Hello, TF World!
+
+You should see "Hello, TF World!" outputted on the terminal.
+
+
+## Adding the configuration to the classpath
+FIXME: prod-tests
+For convenience of deployment, you can also bundle configuration files into the TradeFederation jars
+themselves. Tradefed will automatically recognize all configurations placed in 'config' folders on
+the classpath.
+
+Lets illustrate this now by moving the helloworld.xml into the tradefed core library.
+
+Move the `helloworld.xml` file into
+`<git root>/tools/tradefederation/prod-tests/res/config/example/helloworld.xml`.
+
+Rebuild tradefed, and restart the tradefed console.
+
+Ask tradefed to display the list of configurations on the classpath:
+
+ tf> list configs
+ […]
+ example/helloworld: Runs the hello world test
+
+You can now run the helloworld config via the following command
+
+ tf >run example/helloworld
+ 05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC
+ Hello, TF World!
+
+
+## Interacting with a device
+
+So far our hello world test isn't doing anything interesting. Tradefed is intended to run tests using Android devices, so lets add an Android device to the test.
+
+Tests can get a reference to an Android device by implementing the IDeviceTest interface.
+
+Here's a sample implementation of what this looks like:
+
+ public class HelloWorldTest implements IRemoteTest, IDeviceTest {
+ private ITestDevice mDevice;
+ @Override
+ public void setDevice(ITestDevice device) {
+ mDevice = device;
+ }
+
+ @Override
+ public ITestDevice getDevice() {
+ return mDevice;
+ }
+ …
+ }
+
+The TradeFederation framework will inject the ITestDevice reference into your test via the
+IDeviceTest#setDevice method, before the IRemoteTest#run method is called.
+
+Lets add an additional print message to the HelloWorldTest displaying the serial number of the
+device.
+
+ @Override
+ public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
+ System.out.println("Hello, TF World! I have a device " + getDevice().getSerialNumber());
+ }
+
+Now rebuild tradefed, and do (FIXME: update)
+
+ $ tradefed.sh
+ tf> list devices
+ Available devices: [30315E38655500EC]
+ …
+
+Take note of the serial number listed in Available devices above. That is the device that should be allocated to HelloWorld.
+
+ tf >run example/helloworld
+ 05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC
+ Hello world, TF! I have a device 30315E38655500EC
+
+You should see the new print message displaying the serial number of the device.
+
+
+## Sending test results
+
+IRemoteTests report results by calling methods on the ITestInvocationListener instance provided to
+their `#run` method.
+
+The TradeFederation framework is responsible for reporting the start and end of an Invocation (via
+the ITestInvocationListener#invocationStarted and ITestInvocationListener#invocationEnded methods
+respectively).
+
+A `test run` is a logical collection of tests. To report test results, IRemoteTests are responsible
+for reporting the start of a test run, the start and end of each test, and the end of the test run.
+
+Here's what the HelloWorldTest implementation looks like with a single failed test result.
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
+ System.out.println("Hello, TF World! I have a device " + getDevice().getSerialNumber());
+
+ TestIdentifier testId = new TestIdentifier("com.example.MyTestClassName", "sampleTest");
+ listener.testRunStarted("helloworldrun", 1);
+ listener.testStarted(testId);
+ listener.testFailed(TestFailure.FAILURE, testId, "oh noes, test failed");
+ listener.testEnded(testId, Collections.EMPTY_MAP);
+ listener.testRunEnded(0, Collections.EMPTY_MAP);
+ }
+
+Note that TradeFederation also includes several IRemoteTest implementations that you can reuse
+instead of writing your own from scratch. (such as InstrumentationTest, which can run an Android
+application's tests remotely on an Android device, parse the results, and forward them to the
+ITestInvocationListener). See the Test Types documentation for more details.
+
+
+## Storing test results
+
+By default, a TradeFederation configuration will use the TextResultReporter as the test listener
+implementation for the configuration. TextResultReporter will dump the results of an invocation to
+stdout. To illustrate, try running the hello-world config from previous section now:
+
+ $ ./tradefed.sh
+ tf >run example/helloworld
+ 05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC
+ Hello world, TF! I have a device 30315E38655500EC
+ 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests
+ Test FAILURE: com.example.MyTestClassName#sampleTest
+ stack: oh noes, test failed
+ 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms
+
+If you want to store the results of an invocation elsewhere, say to a file, you would need to
+specify a custom "result_reporter" in your configuration, that specifies the custom
+ITestInvocationListener class you want to use.
+
+The TradeFederation framework includes a result_reporter (XmlResultReporter) that will write test
+results to an XML file, in a format similar to the ant JUnit XML writer.
+
+Lets specify the result_reporter in the configuration now. Edit the
+`tools/tradefederation/res/config/example/helloworld.xml` like this:
+
+ <configuration description="Runs the hello world test">
+ <test class="com.android.tradefed.example.HelloWorldTest" />
+ <result_reporter class="com.android.tradefed.result.XmlResultReporter" />
+ </configuration>
+
+Now rebuild tradefed and re-run the hello world sample:
+FIXME: paths
+
+ tf >run example/helloworld
+ 05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC
+ Hello world, TF! I have a device 30315E38655500EC
+ 05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /var/folders/++/++2Pz+++6+0++4RjPqRgNE+-4zk/-Tmp-/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt
+ 05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /var/folders/++/++2Pz+++6+0++4RjPqRgNE+-4zk/-Tmp-/0/inv_2991649128735283633/host_log_6307746032218561704.txt
+ 05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /var/folders/++/++2Pz+++6+0++4RjPqRgNE+-4zk/-Tmp-/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0
+
+Notice the log message stating an XML file has been generated. The generated file should look like this:
+
+ <?xml version='1.0' encoding='UTF-8' ?>
+ <testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost">
+ <properties />
+ <testcase name="sampleTest" classname="com.example.MyTestClassName" time="0">
+ <failure>oh noes, test failed
+ </failure>
+ </testcase>
+ </testsuite>
+
+Note that you can write your own custom result_reporter. It just needs to implement the
+ITestInvocationListener interface.
+
+Also note that Tradefed supports multiple result_reporters, meaning that you can send test results
+to multiple independent destinations. Just specify multiple <result_reporter> tags in your config to
+do this.
+
+
+## Logging
+
+TradeFederation includes two logging facilities:
+
+1. ability to capture logs from the device (aka device logcat)
+2. ability to record logs from the TradeFederation framework running on the host machine (aka the
+ host log)
+
+Lets focus on 2 for now. Trade Federation's host logs are reported using the CLog wrapper for the
+ddmlib Log class.
+
+Lets convert the previous System.out.println call in HelloWorldTest to a CLog call:
+
+ @Override
+ public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
+ CLog.i("Hello world, TF! I have a device " + getDevice().getSerialNumber());
+
+Now rebuild and rerun. You should see the log message on stdout.
+
+ tf> run example/helloworld
+ …
+ 05-16 21:30:46 I/HelloWorldTest: Hello world, TF! I have a device 30315E38655500EC
+ …
+
+By default, TradeFederation will output host log messages to stdout. TradeFederation also includes a
+log implementation that will write messages to a file: FileLogger. To add file logging, add a
+'logger' tag to the configuration xml, specifying the full class name of FileLogger.
+
+ <configuration description="Runs the hello world test">
+ <test class="com.android.tradefed.example.HelloWorldTest" />
+ <result_reporter class="com.android.tradefed.result.XmlResultReporter" />
+ <logger class="com.android.tradefed.log.FileLogger" />
+ </configuration>
+
+Now rebuild and run the helloworld example again.
+
+ tf >run example/helloworld
+ …
+ 05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /var/folders/++/++2Pz+++6+0++4RjPqRgNE+-4zk/-Tmp-/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt
+ 05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
+ …
+
+Note the log message indicating the path of the host log. View the contents of that file, and you
+should see your HelloWorldTest log message
+
+ $ more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
+ …
+ 05-16 21:38:21 I/HelloWorldTest: Hello world, TF! I have a device 30315E38655500EC
+
+The TradeFederation framework will also automatically capture the logcat from the allocated device,
+and send it the the result_reporter for processing. XmlResultReporter will save the captured device
+logcat as a file.
+
+
+## Command line options
+Objects loaded from a TradeFederation Configuration (aka "Configuration objects") also have the
+ability to receive data from command line arguments.
+
+This is accomplished via the `@Option` annotation. To participate, a Configuration object class
+would apply the `@Option` annotation to a member field, and provide it a unique name. This would
+allow that member field's value to be populated via a command line option, and would also
+automatically add that option to the configuration help system (Note: not all field types are
+supported: see the OptionSetter javadoc for a description of supported types).
+
+Lets add an Option to the HelloWorldTest.
+
+ @Option(name="my_option",
+ shortName='m',
+ description="this is the option's help text",
+ // always display this option in the default help text
+ importance=Importance.ALWAYS)
+ private String mMyOption = "thisisthedefault";
+
+And lets add a log message to display the value of the option in HelloWorldTest, so we can
+demonstrate that it was received correctly.
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
+ …
+ Log.logAndDisplay(LogLevel.INFO, "HelloWorldTest", "I received this option " + mMyOption);
+
+Rebuild TF and run helloworld: you should see a log message with the my_option's default value.
+
+ tf> run example/helloworld
+ …
+ 05-24 18:30:05 I/HelloWorldTest: I received this option thisisthedefault
+
+Now pass in a value for my_option: you should see my_option getting populated with that value
+
+ tf> run example/helloworld --my_option foo
+ …
+ 05-24 18:33:44 I/HelloWorldTest: I received this option foo
+
+TF configurations also include a help system, which automatically displays help text for @Option
+fields. Try it now, and you should see the help text for 'my_option':
+
+ tf> run --help example/helloworld
+ Printing help for only the important options. To see help for all options, use the --help-all flag
+
+ cmd_options options:
+ --[no-]help display the help text for the most important/critical options. Default: false.
+ --[no-]help-all display the full help text for all options. Default: false.
+ --[no-]loop keep running continuously. Default: false.
+
+ test options:
+ -m, --my_option this is the option's help text Default: thisisthedefault.
+
+ 'file' logger options:
+ --log-level-display the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error.
+FIXME: redo with enum help
+
+Note the message at the top about 'printing only the important options'. To reduce option help
+clutter, TF uses the Option#importance attribute to determine whether to show an Option's help text
+when '--help' is specified. '--help-all' will always show all options' help regardless of
+importance. See Option.Importance javadoc for details.
+
+You can also specify an Option's value within the configuration xml by adding a
+`<option name="" value="">` element. Lets see how this looks in the helloworld.xml:
+
+ <test class="com.android.tradefed.example.HelloWorldTest" >
+ <option name="my_option" value="fromxml" />
+ </test>
+
+Re-building and running helloworld should now produce this output:
+
+ 05-24 20:38:25 I/HelloWorldTest: I received this option fromxml
+
+The configuration help should also be updated to indicate my_option's new default value:
+
+ tf> run --help example/helloworld
+ test options:
+ -m, --my_option this is the option's help text Default: fromxml.
+
+Also note that other configuration objects included in the helloworld config, like FileLogger, also have options. '--log-level-display' is of interest because it filters the logs that show up on stdout. You may have noticed from earlier in the tutorial the 'Hello world, TF! I have a device ..' log message stopped getting displayed on stdout once we switched to using FileLogger. You can increase the verbosity of logging to stdout by passing in log-level-display arg.
+
+Try this now, and you should see the 'I have a device' log message reappear on stdout, in addition to getting logged to a file.
+
+ tf >run --log-level-display info example/helloworld
+ …
+ 05-24 18:53:50 I/HelloWorldTest: Hello world, TF! I have a device XXXXXX
+
+<!-- To make future debugging in this tutorial easier, edit the helloworld.xml to default log-level-display to debug:
+
+ <logger class="com.android.tradefed.log.FileLogger" >
+ <option name="log-level-display" value="debug" />
+ </logger>
+-->