1 Conformance Test: Definition and Structure 2 =========================================== 3 4 This document describes what conformance tests are, how they are created, 5 and the structure they follow in CVS. 6 7 Conformance Test Definition 8 --------------------------- 9 10 Conformance tests test conformance to the POSIX* specification. They test 11 both that the header files are correct and that the behavior defined in 12 the specification is implemented. 13 14 Directory Structure 15 ------------------- 16 17 The directory structure for conformance tests is as follows. 18 19 All tests are located in the conformance/ directory off of posixtestsuite/. 20 21 Within this directory, there are the following three subdirectories: 22 23 definitions/ - This directory contains tests for each *.h file in the POSIX 24 spec. Tests are taken from the Base Definitions document. 25 26 interfaces/ - This directory contains tests for each function listed in the 27 POSIX spec. Tests are taken from the System Interfaces document. 28 29 behavior/ - This directory contains tests from any document that do not 30 directly correlate to functions (as in interfaces/) or header files 31 (as in definitions/). 32 33 The breakdown of these subdirectories is as follows: 34 35 definitions/ 36 ----------- 37 The definitions/ directory contains one subdirectory per *.h file. Each 38 subdirectory should have the name *_h (i.e., the *.h file name with "_" 39 replacing "."). 40 41 interfaces/ 42 ---------- 43 The interfaces directory contains one subdirectory per function listed 44 in the System Interfaces document. 45 46 behavior/ 47 -------- 48 This directory is broken down by functional area in a manner similar to 49 functional and stress tests (see HOWTO_<TBD> - for now, see email discussions 50 on functional/stress test structure). 51 52 Writing Tests 53 ------------- 54 55 Within each lowest level directory (definitions/*_h for definitions, 56 interfaces/<function> for interfaces, and behavior/<functional area> for 57 behavior), an assertions.xml file is used to describe the tests, and tests 58 have the structure M-N.c or M-N.sh. More details are below. 59 60 Speculative Tests 61 ----------------- 62 Tests which the POSIX spec uses the words "may" or "implementation-defined" 63 to define are considered speculative by PTS and are created in a 64 speculative/ directory off of the functional area directory. (For 65 example, mq_timedsend/speculative.) 66 These tests attempt to determine which of the implementation-defined 67 alternatives the current implementation implements. PTS_PASS is always 68 returned if the test is able to finish. 69 70 assertions.xml 71 -------------- 72 This file contains a list of assertions to be tested, an ID for each 73 assertion, and descriptive attributes to be used in categorizing 74 assertions. For more information on creating and using these files, 75 see HOWTO_Assertions. 76 77 Test Cases 78 ---------- 79 Test cases are numbered M-N.c or M-N.sh where M corresponds to the 80 assertion ID of the assertion being tested, and N is an ID for the specific 81 tests. (ex. assertion 8 may have test cases 8-1.c 8-2.c and 8-3.sh) 82 Generally, *.sh scripts are only used when the test case is trivial. 83 84 The following documents may be helpful in test case creation: 85 86 - HOWTO_DefinitionsTest - describes how to create definitions tests 87 - HOWTO_ResultCodes - describes the result codes test cases should return 88 - HOWTO_BoundaryTest - describes how to create boundary tests for functions 89 90 * POSIX (R) is a registered trademark of the IEEE 91 92 Contributors: rusty.lynch REMOVE-THIS AT intel DOT com 93 julie.n.fleischer REMOVE-THIS AT intel DOT com 94
