faft: Retry killing and starting FAFT client on any failure
We saw some RPC errors when starting FAFT client. Retrying the whole FAFT
client (killing the previous and starting a new one) is a better way to
prevent such kind of RPC errors.
BUG=chrome-os-partner:16037
TEST=run a FAFT test and pass.
Change-Id: I4492f1cc727ab234aafc1a32dacd6aa5507c45a8
Reviewed-on: https://gerrit.chromium.org/gerrit/38606
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Ready: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>
diff --git a/server/cros/servo_test.py b/server/cros/servo_test.py
index 1c1fa09..1793b40 100644
--- a/server/cros/servo_test.py
+++ b/server/cros/servo_test.py
@@ -179,6 +179,24 @@
def launch_client(self, info):
+ """Launch a remote XML RPC connection on client with retrials.
+
+ Args:
+ info: A dict of remote info, see the definition of self._remote_infos.
+ """
+ retry = 3
+ try:
+ self._launch_client_once(info)
+ except AssertionError:
+ if retry:
+ retry -= 1
+ logging.info('Retry again...')
+ time.sleep(5)
+ else:
+ raise
+
+
+ def _launch_client_once(self, info):
"""Launch a remote process on client and set up an xmlrpc connection.
Args:
@@ -211,12 +229,6 @@
xmlrpclib.ServerProxy(remote_url, allow_none=True))
logging.info('Server proxy: %s', remote_url)
- # We found that the following RPC call retrial doesn't work all the
- # time and causes timeout error happened. So add this delay to wait
- # the client RPC server start-up as a work-around.
- # TODO(waihong@chromium.org): Find the root cause why retrial not work.
- time.sleep(5)
-
# Poll for client RPC server to come online.
timeout = 20
succeed = False