Detect if PDU goes down

This aims at mitigating the time taken to bail out of
a running test if we have any PDUs that are down, by checking
their status before sending down any requests and maintaining
and making use of this state information through out the test
to quarantine APs that may be attached to this PDU. It makes
use of a global list that would be queried by all the threads
created across cartridges, across the runner for each spec, so
that no thread has to re-discover a broken PDU.

BUG=chromium:331914
TEST=ChaosPSK

Change-Id: If8d2df6d17634db2bfef7ec566f2d7d3e7442164
Reviewed-on: https://chromium-review.googlesource.com/184553
Reviewed-by: Bindu Mahadev <bmahadev@chromium.org>
Commit-Queue: Bindu Mahadev <bmahadev@chromium.org>
Tested-by: Bindu Mahadev <bmahadev@chromium.org>
11 files changed