[autotest] Add retries to provisioning.
There is now a setting in global_config.ini which controls provision
retries, and this value can be reloaded on-the-fly in the scheduler.
Be cautioned that provision failures are basically silently hid.
There's currently no sort of reporting to indicate that a retry
happened.
Implementing this also pointed out the way to clean up the ProvisionTask
code, so there's also some free cleanup packaged into this CL.
BUG=chromium:279667
DEPLOY=scheduler
TEST=forced provision failures, and watched the HQE get requeued a
finite number of times.
Change-Id: I66d967fb8f3ab9f199571764821e1a39d0e81f39
Reviewed-on: https://chromium-review.googlesource.com/167990
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
Reviewed-by: Prashanth Balasubramanian <beeps@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
diff --git a/global_config.ini b/global_config.ini
index 89d320d..bdc634d 100644
--- a/global_config.ini
+++ b/global_config.ini
@@ -103,8 +103,8 @@
copy_parse_log_back: False
tick_debug: True
extra_debugging: False
-try_job_timeout_mins: 120
-hqe_maximum_abort_rate_float: .5
+max_repair_limit: 2
+max_provision_retries: 1
[HOSTS]