[autotest] Add retries to provisioning.
There is now a setting in global_config.ini which controls provision
retries, and this value can be reloaded on-the-fly in the scheduler.
Be cautioned that provision failures are basically silently hid.
There's currently no sort of reporting to indicate that a retry
happened.
Implementing this also pointed out the way to clean up the ProvisionTask
code, so there's also some free cleanup packaged into this CL.
BUG=chromium:279667
DEPLOY=scheduler
TEST=forced provision failures, and watched the HQE get requeued a
finite number of times.
Change-Id: I66d967fb8f3ab9f199571764821e1a39d0e81f39
Reviewed-on: https://chromium-review.googlesource.com/167990
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
Reviewed-by: Prashanth Balasubramanian <beeps@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
diff --git a/scheduler/scheduler_config.py b/scheduler/scheduler_config.py
index 96d4371..91c8fb1 100644
--- a/scheduler/scheduler_config.py
+++ b/scheduler/scheduler_config.py
@@ -18,6 +18,8 @@
'secs_to_wait_for_atomic_group_hosts',
'reverify_period_minutes': 'reverify_period_minutes',
'reverify_max_hosts_at_once': 'reverify_max_hosts_at_once',
+ 'max_repair_limit': 'max_repair_limit',
+ 'max_provision_retries': 'max_provision_retries',
}