[autotest] Allow suite scheduler to process boards in parallel
suite scheduler processes each board in a single threaded fashion. Each board
takes about 2-10 mins to process for a single event (new_build, nightly or
weekly). We have over 100 boards in the lab, that leads to nightly run takes
almost 8 hours to finish, and weekly run takes more than 36 hours.
This CL changes suite scheduler driver to process boards in parallel, to
reduce the run time needed to handle each event.
BUG=chromium:560951
TEST=local run, confirm suite can be created.
/usr/local/autotest/site_utils/suite_scheduler/suite_scheduler.py -b \
-d /usr/local/autotest/logs -f /usr/local/autotest/t_suite_scheduler.ini \
-r /tmp/_autotmp_P5J9Zj_suite_scheduler
unittest
Change-Id: I3d7eea2b6449d3c4ea4a8c191ff980064e0301a6
Reviewed-on: https://chromium-review.googlesource.com/318503
Commit-Ready: Dan Shi <dshi@google.com>
Tested-by: Dan Shi <dshi@google.com>
Reviewed-by: Simran Basi <sbasi@chromium.org>
diff --git a/client/common_lib/cros/retry.py b/client/common_lib/cros/retry.py
index d8512de..def586b 100644
--- a/client/common_lib/cros/retry.py
+++ b/client/common_lib/cros/retry.py
@@ -2,7 +2,12 @@
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
-import logging, random, signal, sys, time
+import logging
+import random
+import signal
+import sys
+import threading
+import time
from autotest_lib.client.common_lib import error
@@ -113,6 +118,10 @@
function without retrying; a malformed RPC isn't going to
magically become good. Will raise exceptions in blacklist as well.
+ If the retry is done in a child thread, timeout may not be enforced as
+ signal only works in main thread. Therefore, the retry inside a child
+ thread may run longer than timeout or even hang.
+
original from:
http://www.saltycrane.com/blog/2009/11/trying-out-retry-decorator-python/
@@ -144,7 +153,8 @@
exception_tuple = () if blacklist is None else tuple(blacklist)
start_time = time.time()
remaining_time = timeout_min * 60
-
+ is_main_thread = isinstance(threading.current_thread(),
+ threading._MainThread)
while remaining_time > 0:
if delayed_enabled:
delay()
@@ -153,10 +163,13 @@
try:
# Clear the cache
exc_info = None
- is_timeout, result = timeout(func, args, kwargs,
- remaining_time)
- if not is_timeout:
- return result
+ if is_main_thread:
+ is_timeout, result = timeout(func, args, kwargs,
+ remaining_time)
+ if not is_timeout:
+ return result
+ else:
+ return func(*args, **kwargs)
except exception_tuple:
raise
except error.CrosDynamicSuiteException: