[autotest] run_suite refactoring
This cl refactor the code that pulls the result into a class called
ResultCollector. ResultCollector goes through the following step
to collect test results (See ResultCollector.run()).
1) Pull test views of the suite job and only keep relevant views, i.e.
SERVER_JOB, and test views that have no subdir (meaning that the test
was not executed, see crosreview.com/191002).
If a test has been executed, we would have test views of the child
job. In such case, we will refer to the child job' test views.
2) Pull test views of the child jobs
3) Generate 'display names' for each test. It removes
the 'build/suite' prefeix if any, and append a 'exprimental'
prefix if it is an experimental test.
4) Generate web and buildbot links for each test
5) Calculate suite timings
6) Compute the return code based on all the test results.
Other than code refactoring, the major change here is
1) We pull child job's test views and use them whenever it is possible.
2) Because of 1), we now able to read child jobs' job keyvals.
We deprecated the old way of figuring out whether a test
is experimental using name hash, which is known to cause problems
when multiple tests have the same test name.
This cl also includes a small change to suite.py and
dynamic_suite/tools.py. When recording the bug id and bug count
as job keyvals, it now includes afe_job_id as part of the key, so
that if bugs are filed for two tests with the same name, they
won't end up with having the same keys.
BUG=chromium:322561,chromium:326294,chromium:353164
TEST=Call run_suite with an exisiting cautotest job id in mock mode:
site_utils/run_suite.py -b BOARD -s suite -i BUILD_NAME -m JOB_ID
Pick up jobs that meet these critera:
1) All passes with TEST_NA and experimental test FAIL
2) With FAIL
3) With WARN
4) With ABORT (aborted before/after running)
5) With child server job SERVER_JOB failure.
6) Suite job was aborted
For each of the above cases, check the following:
A. return code
B. test names, experimental prefix.
C. Suite timings
D. Weblinks and buildbot links(with bugs).
Run dummy and bvt suite locally.
Change-Id: Ie224d69bc9a753426fe1b2e3c2e71ef5cd84c9ad
Reviewed-on: https://chromium-review.googlesource.com/193726
Tested-by: Fang Deng <fdeng@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
3 files changed