| Conformance Test: Definition and Structure |
| =========================================== |
| |
| This document describes what conformance tests are, how they are created, |
| and the structure they follow in CVS. |
| |
| Conformance Test Definition |
| --------------------------- |
| |
| Conformance tests test conformance to the POSIX* specification. They test |
| both that the header files are correct and that the behavior defined in |
| the specification is implemented. |
| |
| Directory Structure |
| ------------------- |
| |
| The directory structure for conformance tests is as follows. |
| |
| All tests are located in the conformance/ directory off of posixtestsuite/. |
| |
| Within this directory, there are the following three subdirectories: |
| |
| definitions/ - This directory contains tests for each *.h file in the POSIX |
| spec. Tests are taken from the Base Definitions document. |
| |
| interfaces/ - This directory contains tests for each function listed in the |
| POSIX spec. Tests are taken from the System Interfaces document. |
| |
| behavior/ - This directory contains tests from any document that do not |
| directly correlate to functions (as in interfaces/) or header files |
| (as in definitions/). |
| |
| The breakdown of these subdirectories is as follows: |
| |
| definitions/ |
| ----------- |
| The definitions/ directory contains one subdirectory per *.h file. Each |
| subdirectory should have the name *_h (i.e., the *.h file name with "_" |
| replacing "."). |
| |
| interfaces/ |
| ---------- |
| The interfaces directory contains one subdirectory per function listed |
| in the System Interfaces document. |
| |
| behavior/ |
| -------- |
| This directory is broken down by functional area in a manner similar to |
| functional and stress tests (see HOWTO_<TBD> - for now, see email discussions |
| on functional/stress test structure). |
| |
| Writing Tests |
| ------------- |
| |
| Within each lowest level directory (definitions/*_h for definitions, |
| interfaces/<function> for interfaces, and behavior/<functional area> for |
| behavior), an assertions.xml file is used to describe the tests, and tests |
| have the structure M-N.c or M-N.sh. More details are below. |
| |
| Speculative Tests |
| ----------------- |
| Tests which the POSIX spec uses the words "may" or "implementation-defined" |
| to define are considered speculative by PTS and are created in a |
| speculative/ directory off of the functional area directory. (For |
| example, mq_timedsend/speculative.) |
| These tests attempt to determine which of the implementation-defined |
| alternatives the current implementation implements. PTS_PASS is always |
| returned if the test is able to finish. |
| |
| assertions.xml |
| -------------- |
| This file contains a list of assertions to be tested, an ID for each |
| assertion, and descriptive attributes to be used in categorizing |
| assertions. For more information on creating and using these files, |
| see HOWTO_Assertions. |
| |
| Test Cases |
| ---------- |
| Test cases are numbered M-N.c or M-N.sh where M corresponds to the |
| assertion ID of the assertion being tested, and N is an ID for the specific |
| tests. (ex. assertion 8 may have test cases 8-1.c 8-2.c and 8-3.sh) |
| Generally, *.sh scripts are only used when the test case is trivial. |
| |
| The following documents may be helpful in test case creation: |
| |
| - HOWTO_DefinitionsTest - describes how to create definitions tests |
| - HOWTO_ResultCodes - describes the result codes test cases should return |
| - HOWTO_BoundaryTest - describes how to create boundary tests for functions |
| |
| * POSIX (R) is a registered trademark of the IEEE |
| |
| Contributors: rusty.lynch REMOVE-THIS AT intel DOT com |
| julie.n.fleischer REMOVE-THIS AT intel DOT com |