1 2 3 Linux Test Project HOWTO 4 5 10 October 2000 6 7 Nate Straz 8 9 Abstract 10 11 This document explains some of the more in depth topics of 12 the Linux Test Project and related testing issues. It does 13 not cover basic installation procedures. See the INSTALL 14 and README files in the tarball for that information. 15 16 1 Preface 17 18 This document was written to help bring the community up 19 to speed on the ins and outs of the Linux Test Project. 20 21 1.1 Copyright 22 23 Copyright (c) 2000 by SGI, Inc. 24 25 Please freely copy and distribute (sell or give away) this 26 document in any format. It's requested that corrections 27 and/or comments be forwarded to the document maintainer. 28 You may create a derivative work and distribute it provided 29 that you: 30 31 * Send your derivative work (in the most suitable format 32 such as sgml) to the LDP (Linux Documentation Project) 33 or the like for posting on the Internet. If not the LDP, 34 then let the LDP know where it is available. 35 36 * License the derivative work with this same license or use 37 GPL. Include a copyright notice and at least a pointer 38 to the license used. 39 40 * Give due credit to previous authors and major contributors. 41 42 If you're considering making a derived work other than a 43 translation, it's requested that you discuss your plans 44 with the current maintainer. 45 46 1.2 Disclaimer 47 48 Use the information in this document at your own risk. I 49 disavow any potential liability for the contents of this 50 document. Use of the concepts, examples, and/or other content 51 of this document is entirely at your own risk. 52 53 All copyrights are owned by their owners, unless specifically 54 noted otherwise. Use of a term in this document should 55 not be regarded as affecting the validity of any trademark 56 or service mark. 57 58 Naming of particular products or brands should not be seen 59 as endorsements. 60 61 You are strongly recommended to take a backup of your system 62 before major installation and backups at regular intervals. 63 64 2 Introduction 65 66 2.1 What is the Linux Test Project? 67 68 The Linux Test Project (LTP) is an effort to create a set 69 of tools and tests to verify the functionality and stability 70 of the Linux kernel. We hope this will support Linux development 71 by making unit testing more complete and minimizing user 72 impact by building a barrier to keep bugs from making it 73 to the user. 74 75 2.2 What is wrong with the current testing model? 76 77 The Linux development community utilizes two important (some 78 out argue most important) testing techniques in its normal 79 operations: Design and Code Inspections. The intent of LTP 80 is to support this by giving developers an ever growing 81 set of tools to help identify any operational problems in 82 their code that may be missed by human review. One of the 83 toughest categories of problems to catch with inspection 84 is that of interaction of features. With a continuously 85 improving set of tests and tools, developers can get an 86 indication of whether their changes may have broken some 87 other functionality. 88 89 There is no such thing as a perfect test base. It is only 90 useful it if keeps up with new and changing functionality, 91 and if it actually gets used. 92 93 2.3 Are you doing benchmarking? 94 95 Not at this time. We are more interested in functional, 96 regression, and stress testing the Linux kernel. Benchmarking 97 may be workable to compare the performance among kernel 98 versions. 99 100 2.4 Are you doing standards testing? 101 102 No, we are leaving that to the Linux Standards Base (LSB). 103 See the Linux Standards Base [http://www.linuxbase.org/||web site] 104 for more information. 105 106 3 Structure 107 108 The basic building block of the test project is a test case 109 that consists of a single action and a verification that 110 the action worked. The result of the test case is usually 111 restricted to PASS/FAIL. 112 113 A test program is a runnable program that contains one or 114 more test cases. Test programs often understand command 115 line options which alter their behavior. The options could 116 determine the amount of memory tested, the location of temporary 117 files, the type of network packet used, or any other useful 118 parameter. 119 120 Test tags are used to pair a unique identifier with a test 121 program and a set of command line options. Test tags are 122 the basis for test suites. 123 124 4 Writing Tests 125 126 Writing a test case is a lot easier than most people think. 127 Any code that you write to examine how a part of the kernel 128 works can be adapted into a test case. All that is needed 129 is a way to report the result of the action to the rest 130 of the world. There are several ways of doing this, some 131 more involved than others. 132 133 4.1 Exit Style Tests 134 135 Probably the simplest way of reporting the results of a test 136 case is the exit status of your program. If your test program 137 encounters unexpected or incorrect results, exit the test 138 program with a non-zero exit status, i.e. exit(1). Conversely, 139 if your program completes as expected, return a zero exit 140 status, i.e. exit(0). Any test driver should be able to 141 handle this type of error reporting. If a test program has 142 multiple test cases you won't know which test case failed, 143 but you will know the program that failed. 144 145 4.2 Formatted Output Tests 146 147 The next easiest way of reporting the results is to write 148 the results of each test case to standard output. This allows 149 for the testing results to be more understandable to both 150 the tester and the analysis tools. When the results are 151 written in a standard way, tools can be used to analyze 152 the results. 153 154 5 Testing Tools 155 156 The Linux Test Project has not yet decided on a "final" test 157 harness. We have provided a simple solution with ltp-pan to 158 make due until a complete solution has been found/created 159 that compliments the Linux kernel development process. 160 Several people have said we should use such and such a test 161 harness. Until we find we need a large complex test harness, 162 we will apply the KISS concept. 163 164 5.1 Ltp-pan 165 166 Ltp-pan is a simple test driver with the ability to keep track 167 of orphaned processes and capture test output. It works 168 by reading a list of test tags and command lines and runs 169 them. By default ltp-pan will select a command randomly from 170 the list of test tags, wait for it to finish. Through command 171 line options you can run through the entire list sequentially, 172 run n tests, keep n test running at all times, and buffer 173 test output. Ltp-pan can be nested to create very complex test 174 environments. 175 176 Ltp-pan uses an active file, also called a zoo file to keep track 177 of which tests are currently running. This file holds the 178 pid, tag, and a portion of the command line. When you start 179 ltp-pan it becomes a test tag in itself, thus it requires a 180 name for itself. Ltp-pan updates the active file to show which 181 test tags are currently running. When a test tag exits, 182 ltp-pan will overwrite the first character with a '#'. The active 183 file can be shared between multiple instances of ltp-pan so 184 you know which tests were running when the system crashes 185 by looking at one file. 186 187 A ltp-pan file contains a list of test tags for ltp-pan to run. The 188 format of a ltp-pan file is as follows: 189 190 testtag testprogram -o one -p two other command line options 191 192 # This is a comment. It is a good idea to describe the test 193 194 # tags in your ltp-pan file. Tests programs can have different 195 196 # behaviors depending on the command line options so it is 197 198 # helpful to describe what each test tag is meant to verify 199 or # provoke. 200 201 # Some more test cases 202 203 mm01 mmap001 -m 10000 204 205 # 40 Mb mmap() test. 206 207 # Creates a 10000 page mmap, touches all of the map, sync's 208 209 # it, and munmap()s it. 210 211 mm03 mmap001 -i 0 -I 1 -m 100 212 213 # repetitive mmapping test. 214 215 # Creates a one page map repetitively for one minute. 216 217 dup02 dup02 218 219 # Negative test for dup(2) with bad fd 220 221 kill09 kill09 222 223 # Basic test for kill(2) 224 225 fs-suite01 ltp-pan -e -a fs-suite01.zoo -n fs-suite01 -f runtest/fs 226 227 # run the entire set of file system tests 228 229 The test tags are simple identifiers, no spaces are allowed. 230 The test of the line is the program to run, which is done 231 using execvp(3). Lines starting with '#' are comments and 232 ignored by ltp-pan. It is a good practice to include descriptions 233 with your test tags so you can have a reminder what a certain 234 obscure test tag tries to do. 235 236 5.1.1 Examples 237 238 The most basic way to run ltp-pan is by passing the test program 239 and parameters on the command line. This will run the single 240 program once and wrap the output. 241 242 $ ltp-pan -a ltp.zoo -n tutor sleep 4 243 244 <<<test_start>>> 245 246 tag=cmdln stime=971450564 247 248 cmdline="sleep 4" 249 250 contacts="" 251 252 analysis=exit 253 254 initiation_status="ok" 255 256 <<<test_output>>> 257 258 <<<execution_status>>> 259 260 duration=103341903 termination_type=exited termination_id=0 261 corefile=no cutime=0 cstime=0 262 263 <<<test_end>>> 264 265 $ cat ltp.zoo 266 267 #9357,tutor,pan/ltp-pan -a ltp.zoo -n tutor sleep 4 268 269 #9358,cmdln,sleep 4 270 271 $ 272 273 How it works 274 275 This example shows the two parameters that are always required 276 by ltp-pan, the active file and a test tag for ltp-pan. The "sleep 277 4" on the end of the command line is a test program and 278 parameters that ltp-pan should run. This test is given the tag 279 "cmdln." Ltp-pan will run one test randomly, which ends 280 up being cmdln since it is the only test that we told ltp-pan 281 about. 282 283 In the active file, ltp.zoo, ltp-pan writes the pid, test tag, 284 and part of the command line for the currently running tests. 285 The command lines are truncated so each line will fit on 286 an 80 column display. When a test tag finishes, ltp-pan will 287 place a '#' at the beginning of the line to mark it as available. 288 Here you can see that cmdln and tutor, the name we gave 289 ltp-pan, ran to completion. If the computer hangs, you can read 290 this file to see which test programs were running. 291 292 We have run one test once. Let's do something a little more 293 exciting. Let's run one test several times, at the same 294 time. 295 296 $ ltp-pan -a ltp.zoo -n tutor -x 3 -s 3 -O /tmp sleep 1 297 298 <<<test_start>>> 299 300 tag=cmdln stime=971465653 301 302 cmdline="sleep 1" 303 304 contacts="" 305 306 analysis=exit 307 308 initiation_status="ok" 309 310 <<<test_output>>> 311 312 313 314 <<<execution_status>>> 315 316 duration=103326814 termination_type=exited termination_id=0 317 corefile=no 318 319 cutime=1 cstime=0 320 321 <<<test_end>>> 322 323 <<<test_start>>> 324 325 tag=cmdln stime=971465653 326 327 cmdline="sleep 1" 328 329 contacts="" 330 331 analysis=exit 332 333 initiation_status="ok" 334 335 <<<test_output>>> 336 337 338 339 <<<execution_status>>> 340 341 duration=103326814 termination_type=exited termination_id=0 342 corefile=no 343 344 cutime=0 cstime=1 345 346 <<<test_end>>> 347 348 <<<test_start>>> 349 350 tag=cmdln stime=971465653 351 352 cmdline="sleep 1" 353 354 contacts="" 355 356 analysis=exit 357 358 initiation_status="ok" 359 360 <<<test_output>>> 361 362 363 364 <<<execution_status>>> 365 366 duration=103326814 termination_type=exited termination_id=0 367 corefile=no 368 369 cutime=0 cstime=0 370 371 <<<test_end>>> 372 373 How it works 374 375 In this example we run another fake test from the command 376 line, but we run it three times (-s 3) and keep three test 377 tags active at the same time (-x 3). The -O parameter is 378 a directory where temporary files can be created to buffer 379 the output of each test tag. You can see in the output that 380 cmdln ran three times. If the -O option were omitted, your 381 test output would be mixed, making it almost worthless. 382 383 * Using a ltp-pan file to run multiple tests 384 385 * Nesting ltp-pan 386 387 For more information on ltp-pan see the man page doc/man1/ltp-pan.1. 388 389 5.2 Scanner 390 391 Ltp-scanner is a results analysis tool that understands the rts 392 style output which ltp-pan generates by default. It will produce 393 a table summarizing which tests passed and which failed. 394 395 6 To Do 396 397 There are a lot of things that still need to be done to make 398 this a complete kernel testing system. The following sections 399 will discuss some of the to do items in detail. 400 401 6.1 Configuration Analysis 402 403 While the number of configuration options for the Linux kernel 404 is seen as a strength to developers and users alike, it 405 is a curse to testers. To create a powerful automated testing 406 system, we need to be able to determine what the configuration 407 on the booted box is and then determine which tests should 408 be run on that box. 409 410 The Linux kernel has hundreds of configuration options that 411 can be set to compile the kernel. There are more options 412 that can be set when you boot the kernel and while it is 413 running. There are also many patches that can be applied 414 to the kernel to add functionality or change behavior. 415 416 6.2 Result Comparison 417 418 A lot of testing will be done in the life of the Linux Test 419 Project. Keeping track of the results from all the testing 420 will require some infrastructure. It would be nice to take 421 that output from a test machine, feed it to a program and 422 receive a list of items that broke since the last run on 423 that machine, or were fixed, or work on another test machine 424 but not on this one. 425 426 7 Contact information and updates 427 428 URL: http://ltp.sourceforge.net/ 429 430 mailing list: ltp (a] lists.linux.it 431 432 list archive: http://lists.linux.it/pipermail/ltp/ 433 434 Questions and comments should be sent to the LTP mailing 435 list at ltp (a] lists.linux.it. To subscribe, please go to 436 http://lists.linux.it/listinfo/ltp. 437 438 The source is also available via CVS. See the web site for 439 a web interface and check out instructions. 440 441 8 Glossary 442 443 Test IEEE/ANSI([footnote] Kit, Edward, Software Testing in the Real World: 444 Improving the Process. P. 82. ACM Press, 1995.) : 445 (i) An activity in which a system or component is executed 446 under specified conditions, the results are observed or 447 record, and an evaluation is made of some aspect of the 448 system or component. 449 (ii) A set of one or more test cases. 450 451 Test Case A test assertion with a single result that 452 is being verified. This allows designations such as PASS 453 or FAIL to be applied to a single bit of functionality. 454 A single test case may be one of many test cases for 455 testing the complete functionality of a system. 456 IEEE/ANSI: 457 (i)A set of test inputs, execution conditions, and expected 458 results developed for a particular objective. 459 (ii) The smallest entity that is always executed as a unit, 460 from beginning to end. 461 462 Test Driver A program that handles the execution of 463 test programs. It is responsible for starting the test 464 programs, capturing their output, and recording their 465 results. Ltp-pan is an example of a test driver. 466 467 Test Framework A mechanism for organizing a group of 468 tests. Frameworks may have complex or very simple API's, 469 drivers and result logging mechanisms. Examples of frameworks 470 are TETware and DejaGnu. 471 472 Test Harness A Test harness is the mechanism that connects 473 a test program to a test framework. It may be a specification 474 of exit codes, or a set of libraries for formatting messages 475 and determining exit codes. In TETware, the tet_result() 476 API is the test harness. 477 478 Test Program A single invokable program. A test program 479 can contain one or more test cases. The test harness's 480 API allows for reporting/analysis of the individual test 481 cases. 482 483 Test Suite A collection of tests programs, assertions, 484 cases grouped together under a framework. 485 486 Test Tag An identifier that corresponds to a command 487 line which runs a test. The tag is a single word that 488 matches a test program with a set of command line arguments. 489
