[autotest] run_suite refactoring
This cl refactor the code that pulls the result into a class called
ResultCollector. ResultCollector goes through the following step
to collect test results (See ResultCollector.run()).
1) Pull test views of the suite job and only keep relevant views, i.e.
SERVER_JOB, and test views that have no subdir (meaning that the test
was not executed, see crosreview.com/191002).
If a test has been executed, we would have test views of the child
job. In such case, we will refer to the child job' test views.
2) Pull test views of the child jobs
3) Generate 'display names' for each test. It removes
the 'build/suite' prefeix if any, and append a 'exprimental'
prefix if it is an experimental test.
4) Generate web and buildbot links for each test
5) Calculate suite timings
6) Compute the return code based on all the test results.
Other than code refactoring, the major change here is
1) We pull child job's test views and use them whenever it is possible.
2) Because of 1), we now able to read child jobs' job keyvals.
We deprecated the old way of figuring out whether a test
is experimental using name hash, which is known to cause problems
when multiple tests have the same test name.
This cl also includes a small change to suite.py and
dynamic_suite/tools.py. When recording the bug id and bug count
as job keyvals, it now includes afe_job_id as part of the key, so
that if bugs are filed for two tests with the same name, they
won't end up with having the same keys.
BUG=chromium:322561,chromium:326294,chromium:353164
TEST=Call run_suite with an exisiting cautotest job id in mock mode:
site_utils/run_suite.py -b BOARD -s suite -i BUILD_NAME -m JOB_ID
Pick up jobs that meet these critera:
1) All passes with TEST_NA and experimental test FAIL
2) With FAIL
3) With WARN
4) With ABORT (aborted before/after running)
5) With child server job SERVER_JOB failure.
6) Suite job was aborted
For each of the above cases, check the following:
A. return code
B. test names, experimental prefix.
C. Suite timings
D. Weblinks and buildbot links(with bugs).
Run dummy and bvt suite locally.
Change-Id: Ie224d69bc9a753426fe1b2e3c2e71ef5cd84c9ad
Reviewed-on: https://chromium-review.googlesource.com/193726
Tested-by: Fang Deng <fdeng@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
diff --git a/server/cros/dynamic_suite/tools.py b/server/cros/dynamic_suite/tools.py
index 1ffb59d..608d069 100644
--- a/server/cros/dynamic_suite/tools.py
+++ b/server/cros/dynamic_suite/tools.py
@@ -189,22 +189,25 @@
_BUG_COUNT_KEYVAL = '-Bug_Count'
-def create_bug_keyvals(testname, bug_info):
+def create_bug_keyvals(job_id, testname, bug_info):
"""Create keyvals to record a bug filed against a test failure.
@param testname Name of the test for which to record a bug.
@param bug_info Pair with the id of the bug and the count of
the number of times the bug has been seen.
+ @param job_id The afe job id of job which the test is associated to.
+ job_id will be a part of the key.
@return Keyvals to be recorded for the given test.
"""
- keyval_base = _testname_to_keyval_key(testname)
+ testname = _testname_to_keyval_key(testname)
+ keyval_base = '%s_%s' % (job_id, testname) if job_id else testname
return {
keyval_base + _BUG_ID_KEYVAL: bug_info[0],
keyval_base + _BUG_COUNT_KEYVAL: bug_info[1]
}
-def get_test_failure_bug_info(keyvals, testname):
+def get_test_failure_bug_info(keyvals, job_id, testname):
"""Extract information about a bug filed against a test failure.
This method tries to extract bug_id and bug_count from the keyvals
@@ -221,12 +224,14 @@
the bug has occured in the buildbot stages.
@param keyvals Keyvals associated with a suite job.
+ @param job_id The afe job id of the job that runs the test.
@param testname Name of a test from the suite.
@return None if there is no bug info, or a pair with the
id of the bug, and the count of the number of
times the bug has been seen.
"""
- keyval_base = _testname_to_keyval_key(testname)
+ testname = _testname_to_keyval_key(testname)
+ keyval_base = '%s_%s' % (job_id, testname) if job_id else testname
bug_id = keyvals.get(keyval_base + _BUG_ID_KEYVAL)
if not bug_id:
return None, None
@@ -260,4 +265,4 @@
lumpy-release/R31-1234.0.0/bvt/dummy_Pass_SERVER_JOB.
@return: the test name, e.g., dummy_Pass_SERVER_JOB.
"""
- return test_job_name.replace('%s/%s/' % (build, suite), '')
\ No newline at end of file
+ return test_job_name.replace('%s/%s/' % (build, suite), '')