1 Functional and Stress Tests: Definition and Structure 2 ====================================================== 3 4 This document describes what functional and stress tests are, how they 5 are created, and the structure that they follow in CVS. 6 7 Functional Test Definition 8 -------------------------- 9 10 Functional tests test the behavior of functional areas of the POSIX* 11 specification. They do not relate directly to lines in the POSIX 12 specifications, but rather test that the overall behavior specified by 13 the functional area is present. 14 15 Stress Test Definition 16 ---------------------- 17 18 Stress tests are designed to monitor how the system behaves when it is 19 taxed by excessively using the functional areas in the POSIX specification 20 or monitoring how the functional areas in the POSIX specification behave 21 when the system is taxed. 22 23 Directory Structure 24 ------------------- 25 Functional and stress tests follow the same structure. 26 27 The functional/ and stress directories off of posixtestsuite are where the 28 functional and stress tests, respectively, are stored. 29 30 Each of these are further subdivided into POSIX area (Timers, Semaphores, 31 etc.). For example, functional/timers. 32 33 Within the functional/<POSIX area> directory, you should create the 34 following files. 35 - Makefile - master Makefile which will make all tests within the area 36 37 - run.sh - file which will run all functional tests. The run.sh can run 38 conformance tests, functional tests from a different area (i.e., 39 functional/threads/run.sh can call functional/timers/threadstests/test.c), 40 as well as functional tests in the current area. If there are tests that 41 apply to more than one area, they can go in either area and be called from 42 the other if needed. 43 Assume that the run.sh is called from the directory it is currently in 44 when writing path names to files. 45 46 - assertions.xml - This file maps test descriptions to test cases as in 47 conformance tests. The grammar for functional/stress tests is: 48 <?xml version "1.0"?> 49 <!DOCTYPE assertions [ 50 <!ELEMENT assertion> 51 <!ATTLIST assertion 52 files CDATA #REQUIRED 53 tag CDATA #REQUIRED 54 > 55 ]> 56 Where the tag follows the same structure outlined in HOWTO_Assertions, and 57 the files="..." field tags a comma delimited list of files which are 58 used to test the assertion. For example, 59 <assertion id="1" files="threadone,threadtwo,threadthree" tag="pt:THR"> 60 Three threads can be run in parallel. 61 </assertion> 62 63 - coverage.txt - This document contain information on the completion of 64 test cases for assertions as in the conformance tests. 65 Content is lines with content: 66 <assertion ID> <YES/NO - for completed/not completed> 67 68 So long as the functional/stress tests follow these guidelines, they 69 can be built and run by the framework. How the directory is subdivided 70 is up to the area creator to define, but some suggestions have been: 71 - create directories corresponding to POSIX chapters (see .1-1990 or 72 .1-1996 specs) 73 - create a number corresponding to the assertion ID for each directory 74 75 * POSIX (R) is a registered trademark of the IEEE 76 77 Contributors: julie.n.fleischer REMOVE-THIS AT intel DOT com 78 ajosey REMOVE-THIS AT rdg DOT opengroup DOT org 79 geoffrey.r.gustafson REMOVE-THIS AT intel DOT com 80
