faft: Wait for system to respond to ping before attempting ssh
In the course of faft testing there are a lot of random test failures
due to ssh timeouts which declare the DUT to be unresponsive, but when
looking at the host everything is fine. These tests will typically
pass with subsequent re-runs.
When debugging this I found that the wait_for_kernel_up() function
is not really waiting for the kernel to be up and is attempting to
initiate a ssh connection immediately after resetting or powering up
the device.
While this seems to work most of the time where ssh will eventually
connect it occasionally times out and does not connect and the test
will be declared to fail.
Under worst-case conditions the system will be in developer mode and
run EC+PD software sync and have a slow to connect network and can
take a minute before it is responsive.
It seems that adding a ping check with a generous timeout in the
wait_for_kernel_up() function increases the reliability of the tests
and does not seem to have a downside except if a device is actually
failed it can take longer to be considered a failure.
The function does not abort if the ping check fails, it will print a
warning and then attempt to ssh -- all in order to continue the
existing behavior in case ping fails but ssh is functional.
BUG=chrome-os-partner:30832
TEST=run suite:faft_bios on samus and get 100% pass rate
Change-Id: Ibfdf2d5ac1f3e3311b98ba36abdad12f892f3df5
Signed-off-by: Duncan Laurie <dlaurie@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/222011
Reviewed-by: Yusuf Mohsinally <mohsinally@chromium.org>
diff --git a/server/cros/faft/firmware_test.py b/server/cros/faft/firmware_test.py
index 7419826..6b541f8 100644
--- a/server/cros/faft/firmware_test.py
+++ b/server/cros/faft/firmware_test.py
@@ -1176,6 +1176,9 @@
@param install_deps: bool, install deps after boot.
"""
logging.info("-[FAFT]-[ start wait_for_kernel_up ]---")
+ # Wait for the system to respond to ping before attempting ssh
+ if not self._client.ping_wait_up(90):
+ logging.warning("-[FAFT]-[ system did not respond to ping ]")
try:
logging.info("Installing deps after boot : %s", install_deps)
self.wait_for_client(install_deps=install_deps)