[autotest] Dyanamic suite schedule retries
Make dynamic schedule a new job for a test that
requires retry if the original job failed.
Maximum times of retry can be speicified in control file.
No retry will be scheduled if
a) Job is aborted before it starts.
b) Job is not scheduled because dependencies are not
satisfied.
Major changes:
1) Add a new field 'job_retries' in ControlData
2) Add RetryHandler in suite.py which handles retry state.
3) Suite.wait schedules retry job if necessary.
4) job_status.wait_for_children and wait_for_results
are modified so that they can handle newly added jobs properly.
6) Retry feature can be enabled/disabled at suite level by passing
job_retry=True/False to dynamic_suite.reimage_and_run.
7) Added unittests for job retry.
This is a part of the implemenation of test retry.
CQ-DEPEND=CL:191002
TEST=unittests; dummy_suite runs ok with both run_suite and test_that.
And integration test as following
1) cherry-pick cl 188365, 191002
2) Add "JOB_RETRIES=2" to dummy_Fail/control and
dummy_Pass/control.
Create a trybot image that has the change.
3) run_suite.py -b lumpy -s dummy -i
trybot-lumpy-release/R35-5653.0.0-b1948
4) Observe that dummy_Fail is retried twice, and
'invalid' and 'invalidates_job_id' are properly set
in tko_tests.
Observe that dummy_Pass is not retried.
5) Test the same flow but dependencies are not satisfied
Observed no retry was scheduled
6) Test the same flow but abandon dummy_Fail from frontend
before the job started. Observed no retry was scheduled.
And test for bug filing:
1) Test bug is filed as normal when no retry is needed
2) Test bug is filed for the last retry job if retry is needed
BUG=chromium:348677
Change-Id: I80878ce959948ae51b759c2c58632f9de4d8b035
Reviewed-on: https://chromium-review.googlesource.com/190585
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
diff --git a/server/cros/dynamic_suite/constants.py b/server/cros/dynamic_suite/constants.py
index abd574b..4b0776e 100644
--- a/server/cros/dynamic_suite/constants.py
+++ b/server/cros/dynamic_suite/constants.py
@@ -9,6 +9,7 @@
# Job keyval indicating whether a job is for an experimental test.
JOB_EXPERIMENTAL_KEY = 'experimental'
+RETRY_ORIGINAL_JOB_ID = 'retry_original_job_id'
# Job attribute and label names
EXPERIMENTAL_PREFIX = 'experimental_'