| Functional and Stress Tests: Definition and Structure |
| ====================================================== |
| |
| This document describes what functional and stress tests are, how they |
| are created, and the structure that they follow in CVS. |
| |
| Functional Test Definition |
| -------------------------- |
| |
| Functional tests test the behavior of functional areas of the POSIX* |
| specification. They do not relate directly to lines in the POSIX |
| specifications, but rather test that the overall behavior specified by |
| the functional area is present. |
| |
| Stress Test Definition |
| ---------------------- |
| |
| Stress tests are designed to monitor how the system behaves when it is |
| taxed by excessively using the functional areas in the POSIX specification |
| or monitoring how the functional areas in the POSIX specification behave |
| when the system is taxed. |
| |
| Directory Structure |
| ------------------- |
| Functional and stress tests follow the same structure. |
| |
| The functional/ and stress directories off of posixtestsuite are where the |
| functional and stress tests, respectively, are stored. |
| |
| Each of these are further subdivided into POSIX area (Timers, Semaphores, |
| etc.). For example, functional/timers. |
| |
| Within the functional/<POSIX area> directory, you should create the |
| following files. |
| - Makefile - master Makefile which will make all tests within the area |
| |
| - run.sh - file which will run all functional tests. The run.sh can run |
| conformance tests, functional tests from a different area (i.e., |
| functional/threads/run.sh can call functional/timers/threadstests/test.c), |
| as well as functional tests in the current area. If there are tests that |
| apply to more than one area, they can go in either area and be called from |
| the other if needed. |
| Assume that the run.sh is called from the directory it is currently in |
| when writing path names to files. |
| |
| - assertions.xml - This file maps test descriptions to test cases as in |
| conformance tests. The grammar for functional/stress tests is: |
| <?xml version "1.0"?> |
| <!DOCTYPE assertions [ |
| <!ELEMENT assertion> |
| <!ATTLIST assertion |
| files CDATA #REQUIRED |
| tag CDATA #REQUIRED |
| > |
| ]> |
| Where the tag follows the same structure outlined in HOWTO_Assertions, and |
| the files="..." field tags a comma delimited list of files which are |
| used to test the assertion. For example, |
| <assertion id="1" files="threadone,threadtwo,threadthree" tag="pt:THR"> |
| Three threads can be run in parallel. |
| </assertion> |
| |
| - coverage.txt - This document contain information on the completion of |
| test cases for assertions as in the conformance tests. |
| Content is lines with content: |
| <assertion ID> <YES/NO - for completed/not completed> |
| |
| So long as the functional/stress tests follow these guidelines, they |
| can be built and run by the framework. How the directory is subdivided |
| is up to the area creator to define, but some suggestions have been: |
| - create directories corresponding to POSIX chapters (see .1-1990 or |
| .1-1996 specs) |
| - create a number corresponding to the assertion ID for each directory |
| |
| * POSIX (R) is a registered trademark of the IEEE |
| |
| Contributors: julie.n.fleischer REMOVE-THIS AT intel DOT com |
| ajosey REMOVE-THIS AT rdg DOT opengroup DOT org |
| geoffrey.r.gustafson REMOVE-THIS AT intel DOT com |