[autotest] Add retrying logic to devserver calls, autoupdate 'test'
Have the autoupdate 'test' control file that we use to reimage devices
use a RetryingAFE, so that it doesn't die in the event of frontend
restarts.
Also, re-use the same retrying logic to have the DevServer class retry
calls in the event of URLErrors. We don't do this for
DevServerExceptions, because that would cause us to retry for things
like attempts to trigger a download with a malformed image name, which
we don't want to do.
BUG=chromium-os:31418
TEST=unit
TEST=run_suite using a local dev server that's not running initially.
TEST=It should keep trying until you start the dev server, at which point things should progress normally.
Change-Id: Id8c7b07ce19d528ae35e7df6a7359234329869df
Reviewed-on: https://gerrit.chromium.org/gerrit/28101
Tested-by: Chris Masone <cmasone@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: Chris Masone <cmasone@chromium.org>
diff --git a/client/common_lib/cros/retry.py b/client/common_lib/cros/retry.py
new file mode 100644
index 0000000..6401990
--- /dev/null
+++ b/client/common_lib/cros/retry.py
@@ -0,0 +1,49 @@
+# Copyright (c) 2012 The Chromium OS Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import logging, random, time
+from autotest_lib.client.common_lib import error
+from autotest_lib.frontend.afe.json_rpc import proxy
+
+
+def retry(ExceptionToCheck, timeout_min=1, delay_sec=3):
+ """Retry calling the decorated function using a delay with jitter.
+
+ Will raise RPC ValidationError exceptions from the decorated
+ function without retrying; a malformed RPC isn't going to
+ magically become good.
+
+ original from:
+ http://www.saltycrane.com/blog/2009/11/trying-out-retry-decorator-python/
+
+ @param ExceptionToCheck: the exception to check. May be a tuple of
+ exceptions to check.
+ @param timeout_min: timeout in minutes until giving up.
+ @param delay_sec: pre-jittered delay between retries in seconds. Actual
+ delays will be centered around this value, ranging up to
+ 50% off this midpoint.
+ """
+ def deco_retry(func):
+ random.seed()
+ def func_retry(*args, **kwargs):
+ deadline = time.time() + timeout_min * 60 # convert to seconds.
+ while time.time() < deadline:
+ try:
+ return func(*args, **kwargs)
+ except error.CrosDynamicSuiteException, e:
+ raise e
+ except proxy.ValidationError, e:
+ raise e
+ except ExceptionToCheck, e:
+ # 'Jitter' the delay, up to 50% in either direction.
+ delay = random.uniform(.5 * delay_sec, 1.5 * delay_sec)
+ logging.warning("%s(%s), Retrying in %f seconds...",
+ e.__class__, e, delay)
+ time.sleep(delay)
+ else:
+ # On the last try, run func() and allow exceptions to escape.
+ return func(*args, **kwargs)
+ return
+ return func_retry # true decorator
+ return deco_retry