[autotest] Add retries to provisioning.
There is now a setting in global_config.ini which controls provision
retries, and this value can be reloaded on-the-fly in the scheduler.
Be cautioned that provision failures are basically silently hid.
There's currently no sort of reporting to indicate that a retry
happened.
Implementing this also pointed out the way to clean up the ProvisionTask
code, so there's also some free cleanup packaged into this CL.
BUG=chromium:279667
DEPLOY=scheduler
TEST=forced provision failures, and watched the HQE get requeued a
finite number of times.
Change-Id: I66d967fb8f3ab9f199571764821e1a39d0e81f39
Reviewed-on: https://chromium-review.googlesource.com/167990
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
Reviewed-by: Prashanth Balasubramanian <beeps@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
3 files changed