faft: Retrieve the recovery reason when a test is trapped in recovery screen
In most of the firmware test failures, it is trapped in the recovery screen.
The test will report a reboot timeout error. It always needs manual rerun and
check the recovery reason again. This CL is to modify the reset logic and try
to retrieve the recovery reason then, such that we can easily check the reason
without rerun.
BUG=chromium-os:23309
TEST=manual
Run a test on a mis-behaved firmware which is trapped in the recovery screen.
$ run_remote_tests.sh --board link --remote dut CorruptBothFwSigAB.*dev
Check the result:
...
14:28:56 INFO | Server: Client machine is offline.
14:32:13 INFO | wait_for_client() timed out.
14:32:14 INFO | Try to retrieve recovery reason...
14:32:24 INFO | Setting usb_mux_sel1 to servo_sees_usbkey
14:32:34 INFO | Setting usb_mux_sel1 to dut_sees_usbkey
14:33:00 INFO | Server: Client machine is up.
...
14:33:15 INFO | Server: Relaunched remote faft.
14:33:15 INFO | Got the recovery reason 19.
14:33:15 INFO | Try cold reboot...
...
[ FAILED ]
firmware_CorruptBothFwSigAB.dev FAIL: Trapped in the recovery reason: 19
Change-Id: I59cfdeccf810f0a9768695517d70ca8a40eed508
Reviewed-on: https://gerrit.chromium.org/gerrit/36537
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Ready: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>
diff --git a/server/cros/faftsequence.py b/server/cros/faftsequence.py
index 1fe8e34..fb08b38 100644
--- a/server/cros/faftsequence.py
+++ b/server/cros/faftsequence.py
@@ -75,6 +75,8 @@
test image to be installed.
_firmware_update: Boolean. True if firmware update needed after
installing the image.
+ _trapped_in_recovery_reason: Keep the recovery reason when the test is
+ trapped in the recovery screen.
"""
version = 1
@@ -132,6 +134,7 @@
}
_install_image_path = None
_firmware_update = False
+ _trapped_in_recovery_reason = 0
_backup_firmware_sha = ()
@@ -263,6 +266,24 @@
# TODO(waihong@chromium.org): Implement replugging the Ethernet in the
# first reset item.
+ # DUT may be trapped in the recovery screen. Try to boot into USB to
+ # retrieve the recovery reason.
+ logging.info('Try to retrieve recovery reason...')
+ if self.servo.get('usb_mux_sel1') == 'dut_sees_usbkey':
+ self.wait_fw_screen_and_plug_usb()
+ else:
+ self.servo.set('usb_mux_sel1', 'dut_sees_usbkey')
+
+ try:
+ self.wait_for_client(install_deps=True)
+ lines = self.faft_client.run_shell_command_get_output(
+ 'crossystem recovery_reason')
+ self._trapped_in_recovery_reason = int(lines[0])
+ logging.info('Got the recovery reason %d.' %
+ self._trapped_in_recovery_reason)
+ except AssertionError:
+ logging.info('Failed to get the recovery reason.')
+
# DUT may halt on a firmware screen. Try cold reboot.
logging.info('Try cold reboot...')
self.cold_reboot()
@@ -1429,6 +1450,7 @@
Raises:
error.TestError: An error when the given step is not valid.
+ error.TestFail: Test failed in waiting DUT reboot.
"""
FAFT_STEP_KEYS = ('state_checker', 'userspace_action', 'reboot_action',
'firmware_action', 'install_deps_after_boot')
@@ -1461,7 +1483,11 @@
except AssertionError:
logging.info('wait_for_client() timed out.')
self.reset_client()
- raise
+ if self._trapped_in_recovery_reason:
+ raise error.TestFail('Trapped in the recovery reason: %d' %
+ self._trapped_in_recovery_reason)
+ else:
+ raise error.TestFail('Timed out waiting for DUT reboot.')
def run_faft_sequence(self):