Reduce timeout conditions for Chrome OS for efficiency.
Many of the Autotest timeouts are far too long for Chrome OS. Where a
normal machine might take 10+ minutes to recover, Chromebooks take only
seconds. As such, I've tweaked many of the boot and crash timeouts to
reflect this.
When a host goes down the scheduler idles the job for four hours while
waiting for crash collection, causing two main problems:
1. Metahost jobs take four hours to requeue upon verify failure.
2. Results are delayed four hours when a host dies in jobs.
Additional issues fixed:
1. Timeout after reboot was essentially doubled in the default Host
implementation of wait_for_reboot. Works for us, but will need to query
upstream to ensure it works for everyone.
2. Unified timeouts across autotest.py and base_classes.py so boot
timeouts exist in only one place. Should be okay since all hosts extend
from Host in base_classes.py. Timings are now configurable in
global_config.
3. Global SSH options to ensure better timings. In order to do this, it
was necessary to add site extensibility to abstract_ssh.
Changes to crashcollect.py, autotest.py, abstract_ssh.py, and
base_classes.py will be upstreamed.
BUG=none
TEST=run_remote_tests. Will test in production as well.
Change-Id: I92964b140337e4ab919288b5190bb2e6d5a4fa2d
Reviewed-on: http://gerrit.chromium.org/gerrit/2269
Reviewed-by: Paul Pendlebury <pauldean@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Dale Curtis <dalecurtis@chromium.org>
diff --git a/server/crashcollect.py b/server/crashcollect.py
index 6e1de7f..dc582ac 100644
--- a/server/crashcollect.py
+++ b/server/crashcollect.py
@@ -1,5 +1,6 @@
import os, time, pickle, logging, shutil
+from autotest_lib.client.common_lib import global_config
from autotest_lib.server import utils
@@ -33,7 +34,12 @@
collect_uncollected_logs(host)
-def wait_for_machine_to_recover(host, hours_to_wait=4.0):
+# Load default for number of hours to wait before giving up on crash collection.
+HOURS_TO_WAIT = global_config.global_config.get_config_value(
+ 'SERVER', 'crash_collection_hours_to_wait', type=float, default=4.0)
+
+
+def wait_for_machine_to_recover(host, hours_to_wait=HOURS_TO_WAIT):
"""Wait for a machine (possibly down) to become accessible again.
@param host: A RemoteHost instance to wait on