1 tdc - Adding test cases for tdc 2 3 Author: Lucas Bates - lucasb (a] mojatatu.com 4 5 ADDING TEST CASES 6 ----------------- 7 8 User-defined tests should be added by defining a separate JSON file. This 9 will help prevent conflicts when updating the repository. Refer to 10 template.json for the required JSON format for test cases. 11 12 Include the 'id' field, but do not assign a value. Running tdc with the -i 13 option will generate a unique ID for that test case. 14 15 tdc will recursively search the 'tc-tests' subdirectory (or the 16 directories named with the -D option) for .json files. Any test case 17 files you create in these directories will automatically be included. 18 If you wish to store your custom test cases elsewhere, be sure to run 19 tdc with the -f argument and the path to your file, or the -D argument 20 and the path to your directory(ies). 21 22 Be aware of required escape characters in the JSON data - particularly 23 when defining the match pattern. Refer to the supplied json test files 24 for examples when in doubt. The match pattern is written in json, and 25 will be used by python. So the match pattern will be a python regular 26 expression, but should be written using json syntax. 27 28 29 TEST CASE STRUCTURE 30 ------------------- 31 32 Each test case has required data: 33 34 id: A unique alphanumeric value to identify a particular test case 35 name: Descriptive name that explains the command under test 36 category: A list of single-word descriptions covering what the command 37 under test is testing. Example: filter, actions, u32, gact, etc. 38 setup: The list of commands required to ensure the command under test 39 succeeds. For example: if testing a filter, the command to create 40 the qdisc would appear here. 41 This list can be empty. 42 Each command can be a string to be executed, or a list consisting 43 of a string which is a command to be executed, followed by 1 or 44 more acceptable exit codes for this command. 45 If only a string is given for the command, then an exit code of 0 46 will be expected. 47 cmdUnderTest: The tc command being tested itself. 48 expExitCode: The code returned by the command under test upon its termination. 49 tdc will compare this value against the actual returned value. 50 verifyCmd: The tc command to be run to verify successful execution. 51 For example: if the command under test creates a gact action, 52 verifyCmd should be "$TC actions show action gact" 53 matchPattern: A regular expression to be applied against the output of the 54 verifyCmd to prove the command under test succeeded. This pattern 55 should be as specific as possible so that a false positive is not 56 matched. 57 matchCount: How many times the regex in matchPattern should match. A value 58 of 0 is acceptable. 59 teardown: The list of commands to clean up after the test is completed. 60 The environment should be returned to the same state as when 61 this test was started: qdiscs deleted, actions flushed, etc. 62 This list can be empty. 63 Each command can be a string to be executed, or a list consisting 64 of a string which is a command to be executed, followed by 1 or 65 more acceptable exit codes for this command. 66 If only a string is given for the command, then an exit code of 0 67 will be expected. 68 69 70 SETUP/TEARDOWN ERRORS 71 --------------------- 72 73 If an error is detected during the setup/teardown process, execution of the 74 tests will immediately stop with an error message and the namespace in which 75 the tests are run will be destroyed. This is to prevent inaccurate results 76 in the test cases. tdc will output a series of TAP results for the skipped 77 tests. 78 79 Repeated failures of the setup/teardown may indicate a problem with the test 80 case, or possibly even a bug in one of the commands that are not being tested. 81 82 It's possible to include acceptable exit codes with the setup/teardown command 83 so that it doesn't halt the script for an error that doesn't matter. Turn the 84 individual command into a list, with the command being first, followed by all 85 acceptable exit codes for the command. 86 87 Example: 88 89 A pair of setup commands. The first can have exit code 0, 1 or 255, the 90 second must have exit code 0. 91 92 "setup": [ 93 [ 94 "$TC actions flush action gact", 95 0, 96 1, 97 255 98 ], 99 "$TC actions add action reclassify index 65536" 100 ], 101
