This tutorial guides you through the construction of a "hello world" Trade Federation test configuration, and gives you a hands-on introduction to the Trade Federation framework. Starting from the Tf development environment, it guides you through the process of creating a simple Trade Federation config and gradually adding more features to it.
The tutorial presents the TF test development process as a set of exercises, each consisting of several steps. The exercises demonstrate how to gradually build and refine your configuration, and provide all the sample code you need to complete the test configuration.
When you are finished with the tutorial, you will have created a functioning TF configuration and will have learned many of the most important concepts in the TF framework.
See (FIXME: link) for how to setup the development environment. The rest of this tutorial assumes you have a shell open that has been initialized to the TradeFederation environment.
For simplicity, this tutorial will illustrate adding a configuration and its classes to the TradeFederation framework core library. Later tutorials/documentation will show how to create your own library that extends TradeFederation.
Lets create a hello world test that just dumps a message to stdout. A TradeFederation test must implement the (FIXME: link) IRemoteTest interface.
Here's an implementation for the HelloWorldTest:
package com.android.tradefed.example;
import com.android.tradefed.device.DeviceNotAvailableException;
import com.android.tradefed.result.ITestInvocationListener;
import com.android.tradefed.testtype.IRemoteTest;
public class HelloWorldTest implements IRemoteTest {
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World!");
}
}
FIXME: prod-tests Save this sample code to <git home>/tools/tradefederation/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java and rebuild tradefed from your shell:
m -j6
If the build does not succeed, please consult the (FIXME: link)Development Environment page to ensure you did not miss any steps.
Trade Federation tests are defined in a "Configuration". A Configuration is an XML file that instructs tradefed which test (or set of tests) to run.
Lets create a new Configuration for our HelloWorldTest.
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
</configuration>
TF will parse the Configuration XML file, load the specified class using reflection, instantiate it, cast it to a IRemoteTest, and call its 'run' method.
Note that we've specified the full class name of the HelloWorldTest. Save this data to a helloworld.xml file anywhere on your local filesystem (eg /tmp/helloworld.xml).
From your shell, launch the tradefed console
$ ./tradefed.sh
Ensure a device is connected to the host machine that is visible to tradefed
tf> list devices
Configurations can be run using the run <config> console command. Try this now
FIXME: redo this
tf> run /tmp/helloworld.xml 05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC Hello, TF World!
You should see "Hello, TF World!" outputted on the terminal.
FIXME: prod-tests For convenience of deployment, you can also bundle configuration files into the TradeFederation jars themselves. Tradefed will automatically recognize all configurations placed in 'config' folders on the classpath.
Lets illustrate this now by moving the helloworld.xml into the tradefed core library.
Move the helloworld.xml file into <git root>/tools/tradefederation/prod-tests/res/config/example/helloworld.xml.
Rebuild tradefed, and restart the tradefed console.
Ask tradefed to display the list of configurations on the classpath:
tf> list configs […] example/helloworld: Runs the hello world test
You can now run the helloworld config via the following command
tf >run example/helloworld 05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC Hello, TF World!
So far our hello world test isn't doing anything interesting. Tradefed is intended to run tests using Android devices, so lets add an Android device to the test.
Tests can get a reference to an Android device by implementing the IDeviceTest interface.
Here's a sample implementation of what this looks like:
public class HelloWorldTest implements IRemoteTest, IDeviceTest {
private ITestDevice mDevice;
@Override
public void setDevice(ITestDevice device) {
mDevice = device;
}
@Override
public ITestDevice getDevice() {
return mDevice;
}
…
}
The TradeFederation framework will inject the ITestDevice reference into your test via the IDeviceTest#setDevice method, before the IRemoteTest#run method is called.
Lets add an additional print message to the HelloWorldTest displaying the serial number of the device.
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World! I have a device " + getDevice().getSerialNumber());
}
Now rebuild tradefed, and do (FIXME: update)
$ tradefed.sh tf> list devices Available devices: [30315E38655500EC] …
Take note of the serial number listed in Available devices above. That is the device that should be allocated to HelloWorld.
tf >run example/helloworld 05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC Hello world, TF! I have a device 30315E38655500EC
You should see the new print message displaying the serial number of the device.
IRemoteTests report results by calling methods on the ITestInvocationListener instance provided to their #run method.
The TradeFederation framework is responsible for reporting the start and end of an Invocation (via the ITestInvocationListener#invocationStarted and ITestInvocationListener#invocationEnded methods respectively).
A test run is a logical collection of tests. To report test results, IRemoteTests are responsible for reporting the start of a test run, the start and end of each test, and the end of the test run.
Here's what the HelloWorldTest implementation looks like with a single failed test result.
@SuppressWarnings("unchecked")
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World! I have a device " + getDevice().getSerialNumber());
TestIdentifier testId = new TestIdentifier("com.example.MyTestClassName", "sampleTest");
listener.testRunStarted("helloworldrun", 1);
listener.testStarted(testId);
listener.testFailed(TestFailure.FAILURE, testId, "oh noes, test failed");
listener.testEnded(testId, Collections.EMPTY_MAP);
listener.testRunEnded(0, Collections.EMPTY_MAP);
}
Note that TradeFederation also includes several IRemoteTest implementations that you can reuse instead of writing your own from scratch. (such as InstrumentationTest, which can run an Android application's tests remotely on an Android device, parse the results, and forward them to the ITestInvocationListener). See the Test Types documentation for more details.
By default, a TradeFederation configuration will use the TextResultReporter as the test listener implementation for the configuration. TextResultReporter will dump the results of an invocation to stdout. To illustrate, try running the hello-world config from previous section now:
$ ./tradefed.sh tf >run example/helloworld 05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC Hello world, TF! I have a device 30315E38655500EC 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests Test FAILURE: com.example.MyTestClassName#sampleTest stack: oh noes, test failed 05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms
If you want to store the results of an invocation elsewhere, say to a file, you would need to specify a custom "result_reporter" in your configuration, that specifies the custom ITestInvocationListener class you want to use.
The TradeFederation framework includes a result_reporter (XmlResultReporter) that will write test results to an XML file, in a format similar to the ant JUnit XML writer.
Lets specify the result_reporter in the configuration now. Edit the tools/tradefederation/res/config/example/helloworld.xml like this:
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
<result_reporter class="com.android.tradefed.result.XmlResultReporter" />
</configuration>
Now rebuild tradefed and re-run the hello world sample: FIXME: paths
tf >run example/helloworld 05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 30315E38655500EC Hello world, TF! I have a device 30315E38655500EC 05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /var/folders/++/++2Pz+++6+0++4RjPqRgNE+-4zk/-Tmp-/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt 05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /var/folders/++/++2Pz+++6+0++4RjPqRgNE+-4zk/-Tmp-/0/inv_2991649128735283633/host_log_6307746032218561704.txt 05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /var/folders/++/++2Pz+++6+0++4RjPqRgNE+-4zk/-Tmp-/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0
Notice the log message stating an XML file has been generated. The generated file should look like this:
<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost">
<properties />
<testcase name="sampleTest" classname="com.example.MyTestClassName" time="0">
<failure>oh noes, test failed
</failure>
</testcase>
</testsuite>
Note that you can write your own custom result_reporter. It just needs to implement the ITestInvocationListener interface.
Also note that Tradefed supports multiple result_reporters, meaning that you can send test results to multiple independent destinations. Just specify multiple <result_reporter> tags in your config to do this.
TradeFederation includes two logging facilities:
Lets focus on 2 for now. Trade Federation's host logs are reported using the CLog wrapper for the ddmlib Log class.
Lets convert the previous System.out.println call in HelloWorldTest to a CLog call:
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
CLog.i("Hello world, TF! I have a device " + getDevice().getSerialNumber());
Now rebuild and rerun. You should see the log message on stdout.
tf> run example/helloworld … 05-16 21:30:46 I/HelloWorldTest: Hello world, TF! I have a device 30315E38655500EC …
By default, TradeFederation will output host log messages to stdout. TradeFederation also includes a log implementation that will write messages to a file: FileLogger. To add file logging, add a 'logger' tag to the configuration xml, specifying the full class name of FileLogger.
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
<result_reporter class="com.android.tradefed.result.XmlResultReporter" />
<logger class="com.android.tradefed.log.FileLogger" />
</configuration>
Now rebuild and run the helloworld example again.
tf >run example/helloworld … 05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /var/folders/++/++2Pz+++6+0++4RjPqRgNE+-4zk/-Tmp-/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt 05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt …
Note the log message indicating the path of the host log. View the contents of that file, and you should see your HelloWorldTest log message
$ more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt … 05-16 21:38:21 I/HelloWorldTest: Hello world, TF! I have a device 30315E38655500EC
The TradeFederation framework will also automatically capture the logcat from the allocated device, and send it the the result_reporter for processing. XmlResultReporter will save the captured device logcat as a file.
Objects loaded from a TradeFederation Configuration (aka "Configuration objects") also have the ability to receive data from command line arguments.
This is accomplished via the @Option annotation. To participate, a Configuration object class would apply the @Option annotation to a member field, and provide it a unique name. This would allow that member field's value to be populated via a command line option, and would also automatically add that option to the configuration help system (Note: not all field types are supported: see the OptionSetter javadoc for a description of supported types).
Lets add an Option to the HelloWorldTest.
@Option(name="my_option",
shortName='m',
description="this is the option's help text",
// always display this option in the default help text
importance=Importance.ALWAYS)
private String mMyOption = "thisisthedefault";
And lets add a log message to display the value of the option in HelloWorldTest, so we can demonstrate that it was received correctly.
@SuppressWarnings("unchecked")
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
…
Log.logAndDisplay(LogLevel.INFO, "HelloWorldTest", "I received this option " + mMyOption);
Rebuild TF and run helloworld: you should see a log message with the my_option's default value.
tf> run example/helloworld … 05-24 18:30:05 I/HelloWorldTest: I received this option thisisthedefault
Now pass in a value for my_option: you should see my_option getting populated with that value
tf> run example/helloworld --my_option foo … 05-24 18:33:44 I/HelloWorldTest: I received this option foo
TF configurations also include a help system, which automatically displays help text for @Option fields. Try it now, and you should see the help text for 'my_option':
tf> run --help example/helloworld
Printing help for only the important options. To see help for all options, use the --help-all flag
cmd_options options:
--[no-]help display the help text for the most important/critical options. Default: false.
--[no-]help-all display the full help text for all options. Default: false.
--[no-]loop keep running continuously. Default: false.
test options:
-m, --my_option this is the option's help text Default: thisisthedefault.
'file' logger options:
--log-level-display the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error.
FIXME: redo with enum help
Note the message at the top about 'printing only the important options'. To reduce option help clutter, TF uses the Option#importance attribute to determine whether to show an Option's help text when '--help' is specified. '--help-all' will always show all options' help regardless of importance. See Option.Importance javadoc for details.
You can also specify an Option's value within the configuration xml by adding a <option name="" value=""> element. Lets see how this looks in the helloworld.xml:
<test class="com.android.tradefed.example.HelloWorldTest" >
<option name="my_option" value="fromxml" />
</test>
Re-building and running helloworld should now produce this output:
05-24 20:38:25 I/HelloWorldTest: I received this option fromxml
The configuration help should also be updated to indicate my_option's new default value:
tf> run --help example/helloworld
test options:
-m, --my_option this is the option's help text Default: fromxml.
Also note that other configuration objects included in the helloworld config, like FileLogger, also have options. '--log-level-display' is of interest because it filters the logs that show up on stdout. You may have noticed from earlier in the tutorial the 'Hello world, TF! I have a device ..' log message stopped getting displayed on stdout once we switched to using FileLogger. You can increase the verbosity of logging to stdout by passing in log-level-display arg.
Try this now, and you should see the 'I have a device' log message reappear on stdout, in addition to getting logged to a file.
tf >run --log-level-display info example/helloworld … 05-24 18:53:50 I/HelloWorldTest: Hello world, TF! I have a device XXXXXX