pan/ltp-pan.c: add statistics about not fully tested testcases
Currently numbers of testcases in LTP will return TCONF when they are not
appropriate to run. But when users execute './runltp' to run the default
test suite towards an linux distribution or upstream kernel, if testcases
return TCONF, ultimately they will print TPASS to users, then users will
not know the real LTP test coverage.
NOTE: The overall test status would be TCONF even if just one test assertion
reported TCONF, which is not ideall but still far better than the
previous state.
Here we change this behaviour and show users testcases which return TCONF,
meaning that these testcases are not fully tested.
Here is a sample:
1. Before this patch, The output will be:
Test Start Time: Tue Jun 24 14:32:15 2014
-----------------------------------------
Testcase Result Exit Value
-------- ------ ----------
access01 PASS 0
access02 PASS 0
access03 PASS 0
getxattr01 PASS 0
getxattr02 PASS 0
access04 PASS 0
ext4-nsec-timestamps PASS 0
ext4-uninit-groups PASS 0
access05 PASS 0
access06 PASS 0
-----------------------------------------------
Total Tests: 10
Total Failures: 0
Kernel Version: 3.16.0-rc1+
Machine Architecture: x86_64
Hostname: localhost.localdomain
2. After this patch, the output will be:
Test Start Time: Tue Jun 24 14:28:27 2014
-----------------------------------------
Testcase Result Exit Value
-------- ------ ----------
access01 PASS 0
access02 PASS 0
access03 PASS 0
getxattr01 CONF 32
getxattr02 CONF 32
access04 PASS 0
ext4-nsec-timestamps CONF 32
ext4-uninit-groups CONF 32
access05 PASS 0
access06 PASS 0
-----------------------------------------------
Total Tests: 10
Total Skipped Tests: 4
Total Failures: 0
Kernel Version: 3.16.0-rc1+
Machine Architecture: x86_64
Hostname: localhost.localdomain
Signed-off-by: Xiaoguang Wang <wangxg.fnst@cn.fujitsu.com>
Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
5 files changed