Autotest: Test servo device before using.
We have seen incidents where we can connect to the beaglebone's
xmlrpc server but when we try to interact with the servo device,
the call hangs for long amounts of time (~20 mins) and the
beaglebone is no longer reachable.
In order to prevent us from stalling our reimaging jobs, we now
ensure the beaglebone can interact with the servo and wrap this
call with a timeout. If it fails, the _connect_servod call errors
out and we won't get stuck waiting on servo calls.
BUG=chromium:255645,chromium:244981,chromium:221883,chromium:241228
TEST=local setup. Interacted with a host with a bad beagleone and
ensured the new test call to it indeed times out.
Change-Id: I3e9915d35eeb26cdd09774ff86ac3b78258c4439
Reviewed-on: https://gerrit.chromium.org/gerrit/62404
Reviewed-by: Fang Deng <fdeng@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Queue: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
diff --git a/server/cros/servo/servo.py b/server/cros/servo/servo.py
index 91da659..7f394cd 100644
--- a/server/cros/servo/servo.py
+++ b/server/cros/servo/servo.py
@@ -10,6 +10,7 @@
import logging, re, time, xmlrpclib
from autotest_lib.client.common_lib import error
+from autotest_lib.client.common_lib.cros import retry
from autotest_lib.server import utils
from autotest_lib.server.cros.servo import power_state_controller
from autotest_lib.server.cros.servo import programmer
@@ -61,6 +62,9 @@
# Time to keep USB power off before and after USB mux direction is changed
USB_POWEROFF_DELAY = 2
+ # Time to wait before timing out on a dut-control call.
+ CALL_TIMEOUT_SECS = 60
+
KEY_MATRIX_ALT_0 = {
'ctrl_refresh': ['0', '0', '0', '1'],
'ctrl_d': ['0', '1', '0', '0'],
@@ -576,10 +580,20 @@
remote = 'http://%s:%s' % (servo_host, servo_port)
self._server = xmlrpclib.ServerProxy(remote)
try:
- self._server.echo("ping-test")
- except:
- logging.error('Connection to servod failed')
- raise
+ # We have noticed incidents where we are able to connect but the
+ # actual servo calls go on for 20-30 mins. Thus _connect_servod now
+ # grabs the pwr_button state and if it times out raises an error.
+ timeout, result = retry.timeout(
+ self._server.get, args=('pwr_button', ),
+ timeout_sec=Servo.CALL_TIMEOUT_SECS)
+ if timeout:
+ raise error.AutotestError('Timed out getting value for '
+ 'pwr_button.')
+ except Exception as e:
+ # Make sure to catch any errors that can occur connecting to servod.
+ error_msg = 'Connection to servod failed. Failure: %s' % e
+ logging.error(error_msg)
+ raise error.AutotestError(error_msg)
def _scp_image(self, image_path):