Support Native Tests in CTS
Bug 4970300
CTS uses XML files to determine what tests to run. The buildCts.py
script runs JavaDoc over Java files to create these listings. However,
this doesn't work for native tests which are executables written in
C++. This adds a Java tool called cts-native-xml-generator that
scans the C++ source files to find tests and then generates a
XML file that buildCts.py can pull in to form the CTS test plans.
The generate XMLs don't use a lot of the existing attributes
were intended for Java. I have also added a new "testType" attribute,
because each new test has so far added another boolean like
"vmHostTest=true" or "isHosttest=true"...
New build rules call cts-native-xml-generator in test_executable.mk.
Replace calls to include BUILD_EXECUTABLE with BUILD_CTS_EXECUTABLE
to not only create the executable like usual but to define a rule
that will call upon cts-native-xml-generator to create the test
package XML. An additional variable called LOCAL_CTS_TEST_PACKAGE
is required to give the native tests a parent test package which
is required by CTS.
CtsNativeTestCase.mk is a Makefile that defines a temporary location
to store the generated XMLs that are then copied to the final
location by build/core/cts.mk. This Makefile is included by the main
CTS Android.mk so that BUILD_CTS_EXECUTABLE can be used throughout
the project.
CtsTestCaseList also includes CtsNativeTestCase.mk to convert the
simple test executable names like "CtsNativeTestcases" to its
full path and the corresponding generated XML's path. These paths
are then passed to build/core/cts.mk. Both end up in the testcases
directory. We define two new variables, CTS_TEST_CASES and
CTS_TEST_XMLS. This is for future planning as discussed later.
Finally, GeeTest and GeeTestResultParser are added to handle
running the native tests. While TradeFed already has similar
classes, these two get around the limitations to work on user
builds and append the package name defined back as
LOCAL_CTS_TEST_PACKAGE. Integrating these changes back to the
TradeFed project will need to be done in a separate change.
A lot of this could have been done by adding more to buildCts.py,
but I didn't want to add to what is a single serial monolithic
step in the CTS build process. By having a separate rule to
handle each XML, make can take advantage of the cores on the
command line rather than sitting there on one core waiting
for the buildCts.py script to finish.
The CTS_TEST_CASES and CTS_TEST_XMLS are made more general to
handle copying any file type whether executable or APK to
the testcases directory. The XMLs could also be for native
packages or they could be for Java tests too. They are made
general on purpose so we do not need to modify
build/core/tasks/cts.mk everytime we add a new test type.
We just need to update CtsTestCaseList.mk without ever
having to do a multiproject change. Furthermore, this
generality allows us to easily convert over the existing
Java test cases.
Change-Id: I68f784bfedda5182b9795de2e6b2a5b7ed49626e
diff --git a/Android.mk b/Android.mk
index e3f9ba8..159bda0 100644
--- a/Android.mk
+++ b/Android.mk
@@ -14,6 +14,6 @@
# limitations under the License.
#
+include cts/CtsNativeTestCase.mk
include cts/CtsTestCoverage.mk
-
include $(call all-subdir-makefiles)
diff --git a/CtsNativeTestCase.mk b/CtsNativeTestCase.mk
new file mode 100644
index 0000000..a648fa0
--- /dev/null
+++ b/CtsNativeTestCase.mk
@@ -0,0 +1,32 @@
+# Copyright (C) 2011 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Include this file to gain access to functions to build native CTS
+# test packages. Replace "include $(BUILD_EXECUTABLE)" with
+# "include $(BUILD_CTS_EXECUTABLE)".
+
+LOCAL_PATH := $(call my-dir)
+BUILD_CTS_EXECUTABLE := $(LOCAL_PATH)/tools/build/test_executable.mk
+
+CTS_NATIVE_XML_OUT := $(HOST_OUT)/cts-native-xml
+
+CTS_NATIVE_XML_GENERATOR := $(HOST_OUT_EXECUTABLES)/cts-native-xml-generator
+
+define cts-get-native-paths
+ $(foreach exe,$(1),$(call intermediates-dir-for,EXECUTABLES,$(1))/$(1))
+endef
+
+define cts-get-native-xmls
+ $(foreach exe,$(1),$(CTS_NATIVE_XML_OUT)/$(exe).xml)
+endef
diff --git a/CtsTestCaseList.mk b/CtsTestCaseList.mk
index 3923aac..060e824 100644
--- a/CtsTestCaseList.mk
+++ b/CtsTestCaseList.mk
@@ -75,5 +75,12 @@
$(CTS_COVERAGE_TEST_CASE_LIST) \
$(CTS_SECURITY_APPS_LIST)
+CTS_NATIVE_EXES := \
+ CtsNativeMediaTestCases
+
+CTS_TEST_CASES := $(call cts-get-native-paths,$(CTS_NATIVE_EXES))
+
+CTS_TEST_XMLS := $(call cts-get-native-xmls,$(CTS_NATIVE_EXES))
+
# The following files will be placed in the tools directory of the CTS distribution
CTS_TOOLS_LIST :=
diff --git a/development/ide/eclipse/.classpath b/development/ide/eclipse/.classpath
index a38f53a..30c63d8 100644
--- a/development/ide/eclipse/.classpath
+++ b/development/ide/eclipse/.classpath
@@ -58,6 +58,7 @@
<classpathentry kind="src" path="cts/tests/tests/webkit/src"/>
<classpathentry kind="src" path="cts/tests/tests/widget/src"/>
<classpathentry kind="src" path="cts/tools/cts-api-coverage/src"/>
+ <classpathentry kind="src" path="cts/tools/cts-native-xml-generator/src"/>
<classpathentry kind="src" path="cts/tools/cts-reference-app-lib/src"/>
<classpathentry kind="src" path="cts/tools/dasm/src"/>
<classpathentry kind="src" path="cts/tools/device-setup/TestDeviceSetup/src"/>
diff --git a/tests/tests/nativemedia/Android.mk b/tests/tests/nativemedia/Android.mk
index 928ebcd..94bc83e 100644
--- a/tests/tests/nativemedia/Android.mk
+++ b/tests/tests/nativemedia/Android.mk
@@ -3,7 +3,7 @@
LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)
-LOCAL_MODULE_TAGS := tests
+LOCAL_MODULE_TAGS := optional
LOCAL_C_INCLUDES:= \
bionic \
@@ -27,8 +27,9 @@
LOCAL_CFLAGS += -DXP_UNIX
-LOCAL_MODULE:= NativeMediaTest
+LOCAL_MODULE := CtsNativeMediaTestCases
LOCAL_MODULE_PATH := $(TARGET_OUT_DATA)/nativetest
-include $(BUILD_EXECUTABLE)
\ No newline at end of file
+LOCAL_CTS_TEST_PACKAGE := android.nativemedia
+include $(BUILD_CTS_EXECUTABLE)
diff --git a/tools/build/test_executable.mk b/tools/build/test_executable.mk
new file mode 100644
index 0000000..bfd5bee
--- /dev/null
+++ b/tools/build/test_executable.mk
@@ -0,0 +1,38 @@
+# Copyright (C) 2011 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Builds an executable and defines a rule to generate the associated test
+# package XML needed by CTS.
+#
+# 1. Replace "include $(BUILD_EXECUTABLE)"
+# with "include $(BUILD_CTS_EXECUTABLE)"
+#
+# 2. Define LOCAL_CTS_TEST_PACKAGE to group the tests under a package
+# as needed by CTS.
+#
+
+include $(BUILD_EXECUTABLE)
+
+cts_executable_xml := $(CTS_NATIVE_XML_OUT)/$(LOCAL_MODULE).xml
+
+$(cts_executable_xml): PRIVATE_PATH := $(LOCAL_PATH)
+$(cts_executable_xml): PRIVATE_TEST_PACKAGE := $(LOCAL_CTS_TEST_PACKAGE)
+$(cts_executable_xml): PRIVATE_EXECUTABLE := $(LOCAL_MODULE)
+$(cts_executable_xml): $(addprefix $(LOCAL_PATH)/,$(LOCAL_SRC_FILES)) | $(CTS_NATIVE_XML_GENERATOR)
+ $(hide) echo Generating test description for native package $(PRIVATE_TEST_PACKAGE)
+ $(hide) $(CTS_NATIVE_XML_GENERATOR) -p $(PRIVATE_TEST_PACKAGE) \
+ -n $(PRIVATE_EXECUTABLE) \
+ -s $(PRIVATE_PATH) \
+ -o $@
diff --git a/tools/cts-native-xml-generator/Android.mk b/tools/cts-native-xml-generator/Android.mk
new file mode 100644
index 0000000..c02d745
--- /dev/null
+++ b/tools/cts-native-xml-generator/Android.mk
@@ -0,0 +1,43 @@
+# Copyright (C) 2011 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+LOCAL_PATH := $(call my-dir)
+
+# We use copy-file-to-new-target so that the installed
+# script file's timestamp is at least as new as the
+# .jar file it wraps.
+
+# the hat script
+# ============================================================
+include $(CLEAR_VARS)
+LOCAL_IS_HOST_MODULE := true
+LOCAL_MODULE_CLASS := EXECUTABLES
+LOCAL_MODULE := cts-native-xml-generator
+LOCAL_MODULE_TAGS := optional
+
+include $(BUILD_SYSTEM)/base_rules.mk
+
+$(LOCAL_BUILT_MODULE): $(HOST_OUT_JAVA_LIBRARIES)/$(LOCAL_MODULE)$(COMMON_JAVA_PACKAGE_SUFFIX)
+$(LOCAL_BUILT_MODULE): $(LOCAL_PATH)/etc/$(LOCAL_MODULE) | $(ACP)
+ @echo "Copy: $(PRIVATE_MODULE) ($@)"
+ $(copy-file-to-new-target)
+ $(hide) chmod 755 $@
+
+# the other stuff
+# ============================================================
+subdirs := $(addprefix $(LOCAL_PATH)/,$(addsuffix /Android.mk, \
+ src \
+ ))
+
+include $(subdirs)
diff --git a/tools/cts-native-xml-generator/etc/cts-native-xml-generator b/tools/cts-native-xml-generator/etc/cts-native-xml-generator
new file mode 100644
index 0000000..cae5f4a
--- /dev/null
+++ b/tools/cts-native-xml-generator/etc/cts-native-xml-generator
@@ -0,0 +1,46 @@
+#!/bin/bash
+#
+# Copyright (C) 2011, The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Set up prog to be the path of this script, including following symlinks,
+# and set up progdir to be the fully-qualified pathname of its directory.
+prog="$0"
+while [ -h "${prog}" ]; do
+ newProg=`/bin/ls -ld "${prog}"`
+ newProg=`expr "${newProg}" : ".* -> \(.*\)$"`
+ if expr "x${newProg}" : 'x/' >/dev/null; then
+ prog="${newProg}"
+ else
+ progdir=`dirname "${prog}"`
+ prog="${progdir}/${newProg}"
+ fi
+done
+oldwd=`pwd`
+progdir=`dirname "${prog}"`
+cd "${progdir}"
+progdir=`pwd`
+prog="${progdir}"/`basename "${prog}"`
+cd "${oldwd}"
+
+libdir=`dirname $progdir`/framework
+
+javaOpts=""
+while expr "x$1" : 'x-J' >/dev/null; do
+ opt=`expr "$1" : '-J\(.*\)'`
+ javaOpts="${javaOpts} -${opt}"
+ shift
+done
+
+exec java $javaOpts -jar $libdir/cts-native-xml-generator.jar "$@"
diff --git a/tools/cts-native-xml-generator/src/Android.mk b/tools/cts-native-xml-generator/src/Android.mk
new file mode 100644
index 0000000..70b9f49
--- /dev/null
+++ b/tools/cts-native-xml-generator/src/Android.mk
@@ -0,0 +1,29 @@
+# Copyright (C) 2011 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+LOCAL_PATH := $(call my-dir)
+
+
+# cts-native-xml-generator java library
+# ============================================================
+include $(CLEAR_VARS)
+
+LOCAL_SRC_FILES := $(call all-subdir-java-files)
+LOCAL_JAVA_RESOURCE_DIRS := res
+LOCAL_JAR_MANIFEST := MANIFEST.mf
+
+LOCAL_MODULE := cts-native-xml-generator
+LOCAL_MODULE_TAGS := optional
+
+include $(BUILD_HOST_JAVA_LIBRARY)
diff --git a/tools/cts-native-xml-generator/src/MANIFEST.mf b/tools/cts-native-xml-generator/src/MANIFEST.mf
new file mode 100644
index 0000000..b8bde72
--- /dev/null
+++ b/tools/cts-native-xml-generator/src/MANIFEST.mf
@@ -0,0 +1,2 @@
+Manifest-Version: 1.0
+Main-Class: com.android.cts.nativexml.CtsNativeXmlGenerator
diff --git a/tools/cts-native-xml-generator/src/com/android/cts/nativexml/CtsNativeXmlGenerator.java b/tools/cts-native-xml-generator/src/com/android/cts/nativexml/CtsNativeXmlGenerator.java
new file mode 100644
index 0000000..1a436dd
--- /dev/null
+++ b/tools/cts-native-xml-generator/src/com/android/cts/nativexml/CtsNativeXmlGenerator.java
@@ -0,0 +1,89 @@
+/*
+ * Copyright (C) 2011 The Android Open Source Project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.android.cts.nativexml;
+
+import java.io.File;
+import java.util.Arrays;
+
+/**
+ * Class that searches a source directory for native gTests and outputs a
+ * test package xml.
+ */
+public class CtsNativeXmlGenerator {
+
+ private static void usage(String[] args) {
+ System.err.println("Arguments: " + Arrays.asList(args));
+ System.err.println("Usage: cts-native-xml-generator -p PACKAGE_NAME -n EXECUTABLE_NAME "
+ + "-s SOURCE_DIR [-o OUTPUT_FILE]");
+ System.exit(1);
+ }
+
+ public static void main(String[] args) throws Exception {
+ String appPackageName = null;
+ String name = null;
+ File sourceDir = null;
+ String outputPath = null;
+
+ for (int i = 0; i < args.length; i++) {
+ if ("-p".equals(args[i])) {
+ if (i + 1 < args.length) {
+ appPackageName = args[++i];
+ } else {
+ System.err.println("Missing value for test package");
+ usage(args);
+ }
+ } else if ("-n".equals(args[i])) {
+ if (i + 1 < args.length) {
+ name = args[++i];
+ } else {
+ System.err.println("Missing value for executable name");
+ usage(args);
+ }
+ } else if ("-o".equals(args[i])) {
+ if (i + 1 < args.length) {
+ outputPath = args[++i];
+ } else {
+ System.err.println("Missing value for output file");
+ usage(args);
+ }
+ } else if ("-s".equals(args[i])) {
+ if (i + 1 < args.length) {
+ sourceDir = new File(args[++i]);
+ } else {
+ System.err.println("Missing value for source directory");
+ usage(args);
+ }
+ } else {
+ System.err.println("Unsupported flag: " + args[i]);
+ usage(args);
+ }
+ }
+
+ if (appPackageName == null) {
+ System.out.println("Package name is required");
+ usage(args);
+ } else if (name == null) {
+ System.out.println("Executable name is required");
+ usage(args);
+ } else if (sourceDir == null) {
+ System.out.println("Source directory is required");
+ usage(args);
+ }
+
+ Generator generator = new Generator(appPackageName, name, sourceDir, outputPath);
+ generator.writePackageXml();
+ }
+}
diff --git a/tools/cts-native-xml-generator/src/com/android/cts/nativexml/Generator.java b/tools/cts-native-xml-generator/src/com/android/cts/nativexml/Generator.java
new file mode 100644
index 0000000..3a75e49
--- /dev/null
+++ b/tools/cts-native-xml-generator/src/com/android/cts/nativexml/Generator.java
@@ -0,0 +1,174 @@
+/*
+ * Copyright (C) 2011 The Android Open Source Project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.android.cts.nativexml;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.FilenameFilter;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.Scanner;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Generator of TestPackage XML files for native gTests.
+ *
+ * It scours all the C++ source files in a given source directory looking
+ * for test declarations and outputs a XML test listing.
+ */
+class Generator {
+
+ /** Test package name like "android.nativemedia" to group the tests. */
+ private final String mAppPackageName;
+
+ /** Name of the native executable. */
+ private final String mName;
+
+ /** Directory to recursively scan for gTest test declarations. */
+ private final File mSourceDir;
+
+ /** Path to output file or null to just dump to standard out. */
+ private final String mOutputPath;
+
+ Generator(String appPackageName, String name, File sourceDir, String outputPath) {
+ mAppPackageName = appPackageName;
+ mName = name;
+ mSourceDir = sourceDir;
+ mOutputPath = outputPath;
+ }
+
+ public void writePackageXml() throws IOException {
+ OutputStream output = System.out;
+ if (mOutputPath != null) {
+ File outputFile = new File(mOutputPath);
+ File outputDir = outputFile.getParentFile();
+ if (!outputDir.exists() && !outputDir.mkdirs()) {
+ System.err.println("Couldn't make output directory: " + mOutputPath);
+ System.exit(1);
+ }
+ output = new FileOutputStream(outputFile);
+ }
+
+ PrintWriter writer = null;
+ try {
+ writer = new PrintWriter(output);
+ writer.println("<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
+ writeTestPackage(writer);
+ } finally {
+ if (writer != null) {
+ writer.close();
+ }
+ }
+ }
+
+ private void writeTestPackage(PrintWriter writer) throws FileNotFoundException {
+ writer.append("<TestPackage appPackageName=\"")
+ .append(mAppPackageName)
+ .append("\" name=\"")
+ .append(mName)
+ .println("\" testType=\"native\" version=\"1.0\">");
+ writeTestSuite(writer);
+ writer.println("</TestPackage>");
+ }
+
+ private void writeTestSuite(PrintWriter writer) throws FileNotFoundException {
+ /*
+ * Given "android.foo.bar.baz"...
+ *
+ * <TestSuite name="android">
+ * <TestSuite name="foo">
+ * <TestSuite name="bar">
+ * <TestSuite name="baz">
+ */
+ Scanner scanner = null;
+ try {
+ scanner = new Scanner(mAppPackageName);
+ scanner.useDelimiter("\\.");
+
+ int numLevels = 0;
+ for (; scanner.hasNext(); numLevels++) {
+ String packagePart = scanner.next();
+ writer.append("<TestSuite name=\"").append(packagePart).println("\">");
+ }
+
+ writeTestCases(writer, mSourceDir);
+
+ for (; numLevels > 0; numLevels--) {
+ writer.println("</TestSuite>");
+ }
+ } finally {
+ if (scanner != null) {
+ scanner.close();
+ }
+ }
+ }
+
+ private void writeTestCases(PrintWriter writer, File dir) throws FileNotFoundException {
+ // Find both C++ files to find tests and directories to look for more tests!
+ File[] files = dir.listFiles(new FilenameFilter() {
+ @Override
+ public boolean accept(File dir, String filename) {
+ return filename.endsWith(".cpp") || new File(dir, filename).isDirectory();
+ }
+ });
+
+ for (int i = 0; i < files.length; i++) {
+ File file = files[i];
+ if (file.isDirectory()) {
+ writeTestCases(writer, file);
+ } else {
+ // Take the test name from the name of the file. It's probably
+ // more accurate to take the name from inside the file...
+ String fileName = file.getName();
+ int extension = fileName.lastIndexOf('.');
+ if (extension != -1) {
+ fileName = fileName.substring(0, extension);
+ }
+
+ writer.append("<TestCase name=\"").append(fileName).println("\">");
+ writeTests(writer, file);
+ writer.println("</TestCase>");
+ }
+ }
+ }
+
+ // We want to find lines like TEST_F(SLObjectCreationTest, testAudioPlayerFromFdCreation) { ...
+ // and extract the "testAudioPlayerFromFdCreation" as group #1
+ private static final Pattern TEST_REGEX = Pattern.compile("\\s*TEST_F\\(\\w+,\\s*(\\w+)\\).*");
+
+ private void writeTests(PrintWriter writer, File file) throws FileNotFoundException {
+ Scanner scanner = null;
+ try {
+ scanner = new Scanner(file);
+ while (scanner.hasNextLine()) {
+ String line = scanner.nextLine();
+ Matcher matcher = TEST_REGEX.matcher(line);
+ if (matcher.matches()) {
+ String name = matcher.group(1);
+ writer.append("<Test name=\"").append(name).println("\" />");
+ }
+ }
+ } finally {
+ if (scanner != null) {
+ scanner.close();
+ }
+ }
+ }
+}
diff --git a/tools/tradefed-host/src/com/android/cts/tradefed/testtype/GeeTest.java b/tools/tradefed-host/src/com/android/cts/tradefed/testtype/GeeTest.java
new file mode 100644
index 0000000..5cfafc6
--- /dev/null
+++ b/tools/tradefed-host/src/com/android/cts/tradefed/testtype/GeeTest.java
@@ -0,0 +1,139 @@
+/*
+ * Copyright (C) 2011 The Android Open Source Project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.android.cts.tradefed.testtype;
+
+import com.android.cts.tradefed.build.CtsBuildHelper;
+import com.android.ddmlib.testrunner.ITestRunListener;
+import com.android.tradefed.build.IBuildInfo;
+import com.android.tradefed.device.DeviceNotAvailableException;
+import com.android.tradefed.device.ITestDevice;
+import com.android.tradefed.log.LogUtil.CLog;
+import com.android.tradefed.result.ITestInvocationListener;
+import com.android.tradefed.testtype.IBuildReceiver;
+import com.android.tradefed.testtype.IDeviceTest;
+import com.android.tradefed.testtype.IRemoteTest;
+
+import java.io.File;
+
+/**
+ * Test runner for native gTests.
+ *
+ * TODO: This is similar to Tradefed's existing GTest, but it doesn't confirm
+ * each directory segment exists using ddmlib's file service. This was
+ * a problem since /data is not visible on a user build, but it is
+ * executable. It's also a lot more verbose when it comes to errors.
+ */
+public class GeeTest implements IBuildReceiver, IDeviceTest, IRemoteTest {
+
+ private static final String NATIVE_TESTS_DIRECTORY = "/data/local/tmp/cts-native-tests";
+
+ private int mMaxTestTimeMs = 1 * 60 * 1000;
+
+ private CtsBuildHelper mCtsBuild;
+ private ITestDevice mDevice;
+
+ private final String mPackageName;
+ private final String mExeName;
+
+ public GeeTest(String packageName, String exeName) {
+ mPackageName = packageName;
+ mExeName = exeName;
+ }
+
+ @Override
+ public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
+ if (installTest()) {
+ runTest(listener);
+ } else {
+ CLog.e("Failed to install native tests");
+ }
+ }
+
+ private boolean installTest() throws DeviceNotAvailableException {
+ if (!createRemoteDir(NATIVE_TESTS_DIRECTORY)) {
+ CLog.e("Could not create directory for native tests: " + NATIVE_TESTS_DIRECTORY);
+ return false;
+ }
+
+ File nativeExe = new File(mCtsBuild.getTestCasesDir(), mExeName);
+ if (!nativeExe.exists()) {
+ CLog.e("Native test not found: " + nativeExe);
+ return false;
+ }
+
+ File devicePath = new File(NATIVE_TESTS_DIRECTORY, mExeName);
+ if (!mDevice.pushFile(nativeExe, devicePath.toString())) {
+ CLog.e("Failed to push native test to device");
+ return false;
+ }
+ return true;
+ }
+
+ private boolean createRemoteDir(String remoteFilePath) throws DeviceNotAvailableException {
+ if (mDevice.doesFileExist(remoteFilePath)) {
+ return true;
+ }
+ File remoteFile = new File(remoteFilePath);
+ String parentPath = remoteFile.getParent();
+ if (parentPath != null) {
+ if (!createRemoteDir(parentPath)) {
+ return false;
+ }
+ }
+ mDevice.executeShellCommand(String.format("mkdir %s", remoteFilePath));
+ return mDevice.doesFileExist(remoteFilePath);
+ }
+
+ void runTest(ITestRunListener listener) throws DeviceNotAvailableException {
+ GeeTestResultParser resultParser = new GeeTestResultParser(mPackageName, listener);
+ resultParser.setFakePackagePrefix(mPackageName + ".");
+
+ String fullPath = NATIVE_TESTS_DIRECTORY + File.separator + mExeName;
+ String flags = "";
+ CLog.v("Running gtest %s %s on %s", fullPath, flags, mDevice.getSerialNumber());
+ // force file to be executable
+ CLog.v("%s", mDevice.executeShellCommand(String.format("chmod 755 %s", fullPath)));
+
+ try {
+ mDevice.executeShellCommand(String.format("%s %s", fullPath, flags), resultParser,
+ mMaxTestTimeMs /* maxTimeToShellOutputResponse */,
+ 0 /* retryAttempts */);
+ } catch (DeviceNotAvailableException e) {
+ resultParser.flush();
+ throw e;
+ } catch (RuntimeException e) {
+ resultParser.flush();
+ throw e;
+ }
+ }
+
+
+ @Override
+ public void setBuild(IBuildInfo buildInfo) {
+ mCtsBuild = CtsBuildHelper.createBuildHelper(buildInfo);
+ }
+
+ @Override
+ public void setDevice(ITestDevice device) {
+ mDevice = device;
+ }
+
+ @Override
+ public ITestDevice getDevice() {
+ return mDevice;
+ }
+}
diff --git a/tools/tradefed-host/src/com/android/cts/tradefed/testtype/GeeTestResultParser.java b/tools/tradefed-host/src/com/android/cts/tradefed/testtype/GeeTestResultParser.java
new file mode 100644
index 0000000..c01da20
--- /dev/null
+++ b/tools/tradefed-host/src/com/android/cts/tradefed/testtype/GeeTestResultParser.java
@@ -0,0 +1,670 @@
+/*
+ * Copyright (C) 2011 The Android Open Source Project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.android.cts.tradefed.testtype;
+
+import com.android.ddmlib.IShellOutputReceiver;
+import com.android.ddmlib.Log;
+import com.android.ddmlib.MultiLineReceiver;
+import com.android.ddmlib.testrunner.ITestRunListener;
+import com.android.ddmlib.testrunner.TestIdentifier;
+import com.android.tradefed.testtype.testdefs.XmlDefsTest;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * TODO: Merge change to add package prefix to tradefed's GTestResultParser.
+ * Otherwise this file should be the same as TradeFed's version.
+ *
+ * Parses the 'raw output mode' results of native tests using GTest that run from shell, and informs
+ * a ITestRunListener of the results.
+ * <p>Sample format of output expected:
+ *
+ * <pre>
+ * [==========] Running 15 tests from 1 test case.
+ * [----------] Global test environment set-up.
+ * [----------] 15 tests from MessageTest
+ * [ RUN ] MessageTest.DefaultConstructor
+ * [ OK ] MessageTest.DefaultConstructor (1 ms)
+ * [ RUN ] MessageTest.CopyConstructor
+ * external/gtest/test/gtest-message_test.cc:67: Failure
+ * Value of: 5
+ * Expected: 2
+ * external/gtest/test/gtest-message_test.cc:68: Failure
+ * Value of: 1 == 1
+ * Actual: true
+ * Expected: false
+ * [ FAILED ] MessageTest.CopyConstructor (2 ms)
+ * ...
+ * [ RUN ] MessageTest.DoesNotTakeUpMuchStackSpace
+ * [ OK ] MessageTest.DoesNotTakeUpMuchStackSpace (0 ms)
+ * [----------] 15 tests from MessageTest (26 ms total)
+ *
+ * [----------] Global test environment tear-down
+ * [==========] 15 tests from 1 test case ran. (26 ms total)
+ * [ PASSED ] 6 tests.
+ * [ FAILED ] 9 tests, listed below:
+ * [ FAILED ] MessageTest.CopyConstructor
+ * [ FAILED ] MessageTest.ConstructsFromCString
+ * [ FAILED ] MessageTest.StreamsCString
+ * [ FAILED ] MessageTest.StreamsNullCString
+ * [ FAILED ] MessageTest.StreamsString
+ * [ FAILED ] MessageTest.StreamsStringWithEmbeddedNUL
+ * [ FAILED ] MessageTest.StreamsNULChar
+ * [ FAILED ] MessageTest.StreamsInt
+ * [ FAILED ] MessageTest.StreamsBasicIoManip
+ * 9 FAILED TESTS
+ * </pre>
+ *
+ * <p>where the following tags are used to signal certain events:
+ * <pre>
+ * [==========]: the first occurrence indicates a new run started, including the number of tests
+ * to be expected in this run
+ * [ RUN ]: indicates a new test has started to run; a series of zero or more lines may
+ * follow a test start, and will be captured in case of a test failure or error
+ * [ OK ]: the preceding test has completed successfully, optionally including the time it
+ * took to run (in ms)
+ * [ FAILED ]: the preceding test has failed, optionally including the time it took to run (in ms)
+ * [==========]: the preceding test run has completed, optionally including the time it took to run
+ * (in ms)
+ * </pre>
+ *
+ * All other lines are ignored.
+ */
+public class GeeTestResultParser extends MultiLineReceiver {
+ private static final String LOG_TAG = "GTestResultParser";
+
+ // Variables to keep track of state
+ private TestResult mCurrentTestResult = null;
+ private int mNumTestsRun = 0;
+ private int mNumTestsExpected = 0;
+ private long mTotalRunTime = 0;
+ private boolean mTestInProgress = false;
+ private boolean mTestRunInProgress = false;
+ private final String mTestRunName;
+ private final Collection<ITestRunListener> mTestListeners;
+
+ /** Fake adding a package prefix if the test listener needs it. */
+ private String mFakePackagePrefix = "";
+
+ /** True if start of test has already been reported to listener. */
+ private boolean mTestRunStartReported = false;
+
+ /** True if current test run has been canceled by user. */
+ private boolean mIsCancelled = false;
+
+ private String mCoverageTarget = null;
+
+ /**
+ * Test result data
+ */
+ private static class TestResult {
+ private String mTestName = null;
+ private String mTestClass = null;
+ private StringBuilder mStackTrace = null;
+ @SuppressWarnings("unused")
+ private Long mRunTime = null;
+
+ /** Returns whether expected values have been parsed
+ *
+ * @return true if all expected values have been parsed
+ */
+ boolean isComplete() {
+ return mTestName != null && mTestClass != null;
+ }
+
+ /** Returns whether there is currently a stack trace
+ *
+ * @return true if there is currently a stack trace, false otherwise
+ */
+ boolean hasStackTrace() {
+ return mStackTrace != null;
+ }
+
+ /**
+ * Returns the stack trace of the current test.
+ *
+ * @return a String representation of the current test's stack trace; if there is not
+ * a current stack trace, it returns an error string. Use {@link TestResult#hasStackTrace}
+ * if you need to know whether there is a stack trace.
+ */
+ String getTrace() {
+ if (hasStackTrace()) {
+ return mStackTrace.toString();
+ } else {
+ Log.e(LOG_TAG, "Could not find stack trace for failed test");
+ return new Throwable("Unknown failure").toString();
+ }
+ }
+
+ /** Provides a more user readable string for TestResult, if possible */
+ @Override
+ public String toString() {
+ StringBuilder output = new StringBuilder();
+ if (mTestClass != null ) {
+ output.append(mTestClass);
+ output.append('#');
+ }
+ if (mTestName != null) {
+ output.append(mTestName);
+ }
+ if (output.length() > 0) {
+ return output.toString();
+ }
+ return "unknown result";
+ }
+ }
+
+ /** Internal helper struct to store parsed test info. */
+ private static class ParsedTestInfo {
+ String mTestName = null;
+ String mTestClassName = null;
+ String mTestRunTime = null;
+
+ public ParsedTestInfo(String testName, String testClassName, String testRunTime) {
+ mTestName = testName;
+ mTestClassName = testClassName;
+ mTestRunTime = testRunTime;
+ }
+ }
+
+ /** Prefixes used to demarcate and identify output. */
+ private static class Prefixes {
+ @SuppressWarnings("unused")
+ private static final String INFORMATIONAL_MARKER = "[----------]";
+ private static final String START_TEST_RUN_MARKER = "[==========] Running";
+ private static final String TEST_RUN_MARKER = "[==========]";
+ private static final String START_TEST_MARKER = "[ RUN ]";
+ private static final String OK_TEST_MARKER = "[ OK ]";
+ private static final String FAILED_TEST_MARKER = "[ FAILED ]";
+ }
+
+ /**
+ * Creates the GTestResultParser.
+ *
+ * @param testRunName the test run name to provide to
+ * {@link ITestRunListener#testRunStarted(String, int)}
+ * @param listeners informed of test results as the tests are executing
+ */
+ public GeeTestResultParser(String testRunName, Collection<ITestRunListener> listeners) {
+ mTestRunName = testRunName;
+ mTestListeners = new ArrayList<ITestRunListener>(listeners);
+ }
+
+ /**
+ * Creates the GTestResultParser for a single listener.
+ *
+ * @param testRunName the test run name to provide to
+ * {@link ITestRunListener#testRunStarted(String, int)}
+ * @param listener informed of test results as the tests are executing
+ */
+ public GeeTestResultParser(String testRunName, ITestRunListener listener) {
+ mTestRunName = testRunName;
+ mTestListeners = new ArrayList<ITestRunListener>(1);
+ mTestListeners.add(listener);
+ }
+
+ /**
+ * Package prefix to be added to test names when they are reported like
+ * "android.nativemedia." You may need to add the dot if you need it.
+ */
+ public void setFakePackagePrefix(String prefix) {
+ mFakePackagePrefix = prefix;
+ }
+
+ /**
+ * Returns the current TestResult for test in progress, or a new default one.
+ *
+ * @return The TestResult for the current test run
+ */
+ private TestResult getCurrentTestResult() {
+ if (mCurrentTestResult == null) {
+ mCurrentTestResult = new TestResult();
+ }
+ return mCurrentTestResult;
+ }
+
+
+ /**
+ * Clears out the current TestResult.
+ */
+ private void clearCurrentTestResult() {
+ mCurrentTestResult = null;
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void processNewLines(String[] lines) {
+ for (String line : lines) {
+ parse(line);
+ }
+ }
+
+ /**
+ * Parse an individual output line.
+ *
+ * @param line Text output line
+ */
+ private void parse(String line) {
+ String message = null;
+
+ if (mTestRunInProgress || line.startsWith(Prefixes.TEST_RUN_MARKER)) {
+ if (line.startsWith(Prefixes.START_TEST_MARKER)) {
+ // Individual test started
+ message = line.substring(Prefixes.START_TEST_MARKER.length()).trim();
+ processTestStartedTag(message);
+ }
+ else if (line.startsWith(Prefixes.OK_TEST_MARKER)) {
+ // Individual test completed successfully
+ message = line.substring(Prefixes.OK_TEST_MARKER.length()).trim();
+ processOKTag(message);
+ clearCurrentTestResult();
+ }
+ else if (line.startsWith(Prefixes.FAILED_TEST_MARKER)) {
+ // Individual test completed with failure
+ message = line.substring(Prefixes.FAILED_TEST_MARKER.length()).trim();
+ processFailedTag(message);
+ clearCurrentTestResult();
+ }
+ else if (line.startsWith(Prefixes.START_TEST_RUN_MARKER)) {
+ // Test run started
+ // Make sure to leave the "Running" in the string
+ message = line.substring(Prefixes.TEST_RUN_MARKER.length()).trim();
+ processRunStartedTag(message);
+ }
+ else if (line.startsWith(Prefixes.TEST_RUN_MARKER)) {
+ // Test run ended
+ // This is for the end of the test suite run, so make sure this else-if is after the
+ // check for START_TEST_SUITE_MARKER
+ message = line.substring(Prefixes.TEST_RUN_MARKER.length()).trim();
+ processRunCompletedTag(message);
+ }
+ else if (testInProgress()) {
+ // Note this does not handle the case of an error outside an actual test run
+ appendTestOutputLine(line);
+ }
+ }
+ }
+
+ /**
+ * Returns true if test run canceled.
+ *
+ * @see IShellOutputReceiver#isCancelled()
+ */
+ public boolean isCancelled() {
+ return mIsCancelled;
+ }
+
+ /**
+ * Requests cancellation of test run.
+ */
+ public void cancel() {
+ mIsCancelled = true;
+ }
+
+ /**
+ * Returns whether we're in the middle of running a test.
+ *
+ * @return True if a test was started, false otherwise
+ */
+ private boolean testInProgress() {
+ return mTestInProgress;
+ }
+
+ /**
+ * Set state to indicate we've started running a test.
+ *
+ */
+ private void setTestStarted() {
+ mTestInProgress = true;
+ }
+
+ /**
+ * Set state to indicate we've started running a test.
+ *
+ */
+ private void setTestEnded() {
+ mTestInProgress = false;
+ }
+
+ /**
+ * Reports the start of a test run, and the total test count, if it has not been previously
+ * reported.
+ */
+ private void reportTestRunStarted() {
+ // if start test run not reported yet
+ if (!mTestRunStartReported) {
+ for (ITestRunListener listener : mTestListeners) {
+ listener.testRunStarted(mTestRunName, mNumTestsExpected);
+ }
+ mTestRunStartReported = true;
+ }
+ }
+
+ /**
+ * Reports the end of a test run, and resets that test
+ */
+ private void reportTestRunEnded() {
+ for (ITestRunListener listener : mTestListeners) {
+ listener.testRunEnded(mTotalRunTime, getRunMetrics());
+ }
+ mTestRunStartReported = false;
+ }
+
+ /**
+ * Create the run metrics {@link Map} to report.
+ *
+ * @return a {@link Map} of run metrics data
+ */
+ private Map<String, String> getRunMetrics() {
+ Map<String, String> metricsMap = new HashMap<String, String>();
+ if (mCoverageTarget != null) {
+ metricsMap.put(XmlDefsTest.COVERAGE_TARGET_KEY, mCoverageTarget);
+ }
+ return metricsMap;
+ }
+
+ /**
+ * Parse the test identifier (class and test name), and optional time info.
+ *
+ * @param identifier Raw identifier of the form classname.testname, with an optional time
+ * element in the format of (XX ms) at the end
+ * @return A ParsedTestInfo representing the parsed info from the identifier string.
+ *
+ * If no time tag was detected, then the third element in the array (time_in_ms) will
+ * be null. If the line failed to parse properly (eg: could not determine name of
+ * test/class) then an "UNKNOWN" string value will be returned for the classname and
+ * testname. This method guarantees a string will always be returned for the class and
+ * test names (but not for the time value).
+ */
+ private ParsedTestInfo parseTestIdentifier(String identifier) {
+ ParsedTestInfo returnInfo = new ParsedTestInfo("UNKNOWN_CLASS", "UNKNOWN_TEST", null);
+
+ Pattern timePattern = Pattern.compile(".*(\\((\\d+) ms\\))"); // eg: (XX ms)
+ Matcher time = timePattern.matcher(identifier);
+
+ // Try to find a time
+ if (time.find()) {
+ String timeString = time.group(2); // the "XX" in "(XX ms)"
+ String discardPortion = time.group(1); // everything after the test class/name
+ identifier = identifier.substring(0, identifier.lastIndexOf(discardPortion)).trim();
+ returnInfo.mTestRunTime = timeString;
+ }
+
+ String[] testId = identifier.split("\\.");
+ if (testId.length < 2) {
+ Log.e(LOG_TAG, "Could not detect the test class and test name, received: " +
+ identifier);
+ }
+ else {
+ returnInfo.mTestClassName = testId[0];
+ returnInfo.mTestName = testId[1];
+ }
+ return returnInfo;
+ }
+
+ /**
+ * Parses and stores the test identifier (class and test name).
+ *
+ * @param identifier Raw identifier
+ */
+ private void processRunStartedTag(String identifier) {
+ // eg: (Running XX tests from 1 test case.)
+ Pattern numTestsPattern = Pattern.compile("Running (\\d+) test[s]? from .*");
+ Matcher numTests = numTestsPattern.matcher(identifier);
+
+ // Try to find number of tests
+ if (numTests.find()) {
+ try {
+ mNumTestsExpected = Integer.parseInt(numTests.group(1));
+ }
+ catch (NumberFormatException e) {
+ Log.e(LOG_TAG, "Unable to determine number of tests expected, received: " +
+ numTests.group(1));
+ }
+ }
+ if (mNumTestsExpected > 0) {
+ reportTestRunStarted();
+ mNumTestsRun = 0;
+ mTestRunInProgress = true;
+ }
+ }
+
+ /**
+ * Processes and informs listener when we encounter a tag indicating that a test suite is done.
+ *
+ * @param identifier Raw log output from the suite ended tag
+ */
+ private void processRunCompletedTag(String identifier) {
+ Pattern timePattern = Pattern.compile(".*\\((\\d+) ms total\\)"); // eg: (XX ms total)
+ Matcher time = timePattern.matcher(identifier);
+
+ // Try to find the total run time
+ if (time.find()) {
+ try {
+ mTotalRunTime = Long.parseLong(time.group(1));
+ }
+ catch (NumberFormatException e) {
+ Log.e(LOG_TAG, "Unable to determine the total running time, received: " +
+ time.group(1));
+ }
+ }
+ reportTestRunEnded();
+ mTestRunInProgress = false;
+ }
+
+ /**
+ * Processes and informs listener when we encounter a tag indicating that a test has started.
+ *
+ * @param identifier Raw log output of the form classname.testname, with an optional time (x ms)
+ */
+ private void processTestStartedTag(String identifier) {
+ ParsedTestInfo parsedResults = parseTestIdentifier(identifier);
+ TestResult testResult = getCurrentTestResult();
+ testResult.mTestClass = parsedResults.mTestClassName;
+ testResult.mTestName = parsedResults.mTestName;
+
+ TestIdentifier testId = new TestIdentifier(mFakePackagePrefix + testResult.mTestClass,
+ testResult.mTestName);
+
+ for (ITestRunListener listener : mTestListeners) {
+ listener.testStarted(testId);
+ }
+ setTestStarted();
+ }
+
+ /**
+ * Helper method to do the work necessary when a test has ended.
+ *
+ * @param identifier Raw log output of the form "classname.testname" with an optional (XX ms)
+ * at the end indicating the running time.
+ * @param testPassed Indicates whether the test passed or failed (set to true if passed, false
+ * if failed)
+ */
+ private void doTestEnded(String identifier, boolean testPassed) {
+ ParsedTestInfo parsedResults = parseTestIdentifier(identifier);
+ TestResult testResult = getCurrentTestResult();
+ TestIdentifier testId = new TestIdentifier(mFakePackagePrefix + testResult.mTestClass,
+ testResult.mTestName);
+
+ // Error - trying to end a test when one isn't in progress
+ if (!testInProgress()) {
+ Log.e(LOG_TAG, "Test currently not in progress when trying to end test: " + identifier);
+ return;
+ }
+
+ // Save the run time for this test if one exists
+ if (parsedResults.mTestRunTime != null) {
+ try {
+ testResult.mRunTime = new Long(parsedResults.mTestRunTime);
+ }
+ catch (NumberFormatException e) {
+ Log.e(LOG_TAG, "Test run time value is invalid, received: " +
+ parsedResults.mTestRunTime);
+ }
+ }
+
+ // Check that the test result is for the same test/class we're expecting it to be for
+ boolean encounteredUnexpectedTest = false;
+ if (!testResult.isComplete()) {
+ Log.e(LOG_TAG, "No test/class name is currently recorded as running!");
+ }
+ else {
+ if (testResult.mTestClass.compareTo(parsedResults.mTestClassName) != 0) {
+ Log.e(LOG_TAG, "Name for current test class does not match class we started " +
+ "with, expected: " + testResult.mTestClass + " but got: " +
+ parsedResults.mTestClassName);
+ encounteredUnexpectedTest = true;
+ }
+ if (testResult.mTestName.compareTo(parsedResults.mTestName) != 0) {
+ Log.e(LOG_TAG, "Name for current test does not match test we started with," +
+ "expected: " + testResult.mTestName + " bug got: " +
+ parsedResults.mTestName);
+ encounteredUnexpectedTest = true;
+ }
+ }
+
+ if (encounteredUnexpectedTest) {
+ // If the test name of the result changed from what we started with, report that
+ // the last known test failed, regardless of whether we received a pass or fail tag.
+ for (ITestRunListener listener : mTestListeners) {
+ listener.testFailed(ITestRunListener.TestFailure.ERROR, testId,
+ mCurrentTestResult.getTrace());
+ }
+ }
+ else if (!testPassed) { // test failed
+ for (ITestRunListener listener : mTestListeners) {
+ listener.testFailed(ITestRunListener.TestFailure.FAILURE, testId,
+ mCurrentTestResult.getTrace());
+ }
+ }
+ // For all cases (pass or fail), we ultimately need to report test has ended
+ Map <String, String> emptyMap = Collections.emptyMap();
+ for (ITestRunListener listener : mTestListeners) {
+ // @TODO: Add reporting of test run time to ITestRunListener
+ listener.testEnded(testId, emptyMap);
+ }
+
+ setTestEnded();
+ ++mNumTestsRun;
+ }
+
+ /**
+ * Processes and informs listener when we encounter the OK tag.
+ *
+ * @param identifier Raw log output of the form "classname.testname" with an optional (XX ms)
+ * at the end indicating the running time.
+ */
+ private void processOKTag(String identifier) {
+ doTestEnded(identifier, true);
+ }
+
+ /**
+ * Processes and informs listener when we encounter the FAILED tag.
+ *
+ * @param identifier Raw log output of the form "classname.testname" with an optional (XX ms)
+ * at the end indicating the running time.
+ */
+ private void processFailedTag(String identifier) {
+ doTestEnded(identifier, false);
+ }
+
+ /**
+ * Appends the test output to the current TestResult.
+ *
+ * @param line Raw test result line of output.
+ */
+ private void appendTestOutputLine(String line) {
+ TestResult testResult = getCurrentTestResult();
+ if (testResult.mStackTrace == null) {
+ testResult.mStackTrace = new StringBuilder();
+ }
+ else {
+ testResult.mStackTrace.append("\r\n");
+ }
+ testResult.mStackTrace.append(line);
+ }
+
+ /**
+ * Process an instrumentation run failure
+ *
+ * @param errorMsg The message to output about the nature of the error
+ */
+ private void handleTestRunFailed(String errorMsg) {
+ errorMsg = (errorMsg == null ? "Unknown error" : errorMsg);
+ Log.i(LOG_TAG, String.format("Test run failed: %s", errorMsg));
+ String testRunStackTrace = "";
+
+ // Report that the last known test failed
+ if ((mCurrentTestResult != null) && (mCurrentTestResult.isComplete())) {
+ // current test results are cleared out after every complete test run,
+ // if it's not null, assume the last test caused this and report as a test failure
+ TestIdentifier testId = new TestIdentifier(mCurrentTestResult.mTestClass,
+ mCurrentTestResult.mTestName);
+
+ // If there was any stack trace during the test run, append it to the "test failed"
+ // error message so we have an idea of what caused the crash/failure.
+ Map<String, String> emptyMap = Collections.emptyMap();
+ if (mCurrentTestResult.hasStackTrace()) {
+ testRunStackTrace = mCurrentTestResult.getTrace();
+ }
+ for (ITestRunListener listener : mTestListeners) {
+ listener.testFailed(ITestRunListener.TestFailure.ERROR, testId,
+ "No test results.\r\n" + testRunStackTrace);
+ listener.testEnded(testId, emptyMap);
+ }
+ clearCurrentTestResult();
+ }
+ // Report the test run failed
+ for (ITestRunListener listener : mTestListeners) {
+ listener.testRunFailed(errorMsg);
+ listener.testRunEnded(mTotalRunTime, getRunMetrics());
+ }
+ }
+
+ /**
+ * Called by parent when adb session is complete.
+ */
+ @Override
+ public void done() {
+ super.done();
+ if (mNumTestsExpected > mNumTestsRun) {
+ handleTestRunFailed(String.format("Test run incomplete. Expected %d tests, received %d",
+ mNumTestsExpected, mNumTestsRun));
+ }
+ else if (mTestRunInProgress) {
+ handleTestRunFailed("No test results");
+ }
+ }
+
+ /**
+ * Sets the coverage target for this test.
+ * <p/>
+ * Will be sent as a metric to test listeners.
+ *
+ * @param coverageTarget the coverage target
+ */
+ public void setCoverageTarget(String coverageTarget) {
+ mCoverageTarget = coverageTarget;
+ }
+}
diff --git a/tools/tradefed-host/src/com/android/cts/tradefed/testtype/TestPackageDef.java b/tools/tradefed-host/src/com/android/cts/tradefed/testtype/TestPackageDef.java
index 8844667..8cccab0 100644
--- a/tools/tradefed-host/src/com/android/cts/tradefed/testtype/TestPackageDef.java
+++ b/tools/tradefed-host/src/com/android/cts/tradefed/testtype/TestPackageDef.java
@@ -50,6 +50,7 @@
private String mRunner = null;
private boolean mIsHostSideTest = false;
private boolean mIsVMHostTest = false;
+ private String mTestType = null;
private String mJarPath = null;
private boolean mIsSignatureTest = false;
private boolean mIsReferenceAppTest = false;
@@ -78,6 +79,7 @@
/**
* {@inheritDoc}
*/
+ @Override
public String getUri() {
return mUri;
}
@@ -128,6 +130,10 @@
return mIsVMHostTest;
}
+ void setTestType(String testType) {
+ mTestType = testType;
+ }
+
void setJarPath(String jarPath) {
mJarPath = jarPath;
}
@@ -229,6 +235,8 @@
vmHostTest.setTests(mTests);
mDigest = generateDigest(testCaseDir, mJarPath);
return vmHostTest;
+ } else if ("native".equals(mTestType)) {
+ return new GeeTest(mUri, mName);
} else if (mIsSignatureTest) {
// TODO: hardcode the runner/class/method for now, since current package xml points to
// specialized instrumentation. Eventually this special case for signatureTest can be
@@ -302,6 +310,7 @@
/**
* {@inheritDoc}
*/
+ @Override
public boolean isKnownTest(TestIdentifier testDef) {
return mTests.contains(testDef);
}
@@ -309,6 +318,7 @@
/**
* {@inheritDoc}
*/
+ @Override
public boolean isKnownTestClass(String className) {
return mTestClasses.contains(className);
}
diff --git a/tools/tradefed-host/src/com/android/cts/tradefed/testtype/TestPackageXmlParser.java b/tools/tradefed-host/src/com/android/cts/tradefed/testtype/TestPackageXmlParser.java
index 64706a2..f1b6ed0 100644
--- a/tools/tradefed-host/src/com/android/cts/tradefed/testtype/TestPackageXmlParser.java
+++ b/tools/tradefed-host/src/com/android/cts/tradefed/testtype/TestPackageXmlParser.java
@@ -68,6 +68,7 @@
final String runnerName = attributes.getValue("runner");
final String hostSideTest = attributes.getValue("hostSideOnly");
final String vmHostTest = attributes.getValue("vmHostTest");
+ final String testType = attributes.getValue("testType");
final String jarPath = attributes.getValue("jarPath");
final String signatureCheck = attributes.getValue("signatureCheck");
final String referenceApp = attributes.getValue("referenceAppTest");
@@ -84,6 +85,7 @@
mPackageDef.setRunner(runnerName);
mPackageDef.setIsHostSideTest(parseBoolean(hostSideTest));
mPackageDef.setIsVMHostTest(parseBoolean(vmHostTest));
+ mPackageDef.setTestType(testType);
mPackageDef.setJarPath(jarPath);
mPackageDef.setIsSignatureCheck(parseBoolean(signatureCheck));
mPackageDef.setIsReferenceApp(parseBoolean(referenceApp));