More CTS Test Runner Fixes
Changing the initial status of tests now to fail, and
making InstrumentationCtsTestRunner explicitly report
when it has omitted a test due to profile or feature
requirements only.
More background on why:
Initially, the starting test status was set to PASS.
This seemed like a bug, because tests that did not
get a result back from the test runner would automatically
pass.
I noticed this when adding the profile/feature annotations,
so I changed the test status to NOT_EXECUTED. However,
I discovered that batch test runner would rerun any tests
that were not executed, so any tests that were discarded
by the annotations would be rerun infinitely..
So then...I changed the test status to start off as OMITTED,
so that discarded tests which don't produce any output would
get flagged as OMITTED and not cause infinite retrying.
However, this uncovered a whole class of tests that were
being omitted due to test errors (previously marked PASS even
though they were failing). This is because the tests would
fail to execute as a batch and then CTS as part of its algorithm
would run them individually again. However, due to an error
of being unable to load the test class, they would be marked
as PASS (before the OMITTED change), because there would be
output from the runner...
By setting the default status to FAIL, there won't be any tests
that get away as PASS due to getting no output from the runner.
Unfortunately, this means any tests that aren't executed or
thrown out by annotations, will be flagged. There seems to be
a couple of tests marked with @Suppress that will need to be
changed to @BrokenTest. There is probably a more involved fix
to fix these things, but its probably not worth it at this time.
The default status also cannot be ERROR, because CTS sees both
errors and failures as the same.
Change-Id: If1d561ee0e8c4919287469238d29c72c1ba0475c
4 files changed