1 #LyX 1.1 created this file. For more info see http://www.lyx.org/ 2 \lyxformat 2.16 3 \textclass docbook 4 \begin_preamble 5 <!entity header system "header.sgml"> 6 \end_preamble 7 \language default 8 \inputencoding default 9 \fontscheme default 10 \graphics default 11 \paperfontsize default 12 \spacing single 13 \papersize Default 14 \paperpackage a4 15 \use_geometry 0 16 \use_amsmath 0 17 \paperorientation portrait 18 \secnumdepth 3 19 \tocdepth 3 20 \paragraph_separation indent 21 \defskip medskip 22 \quotes_language english 23 \quotes_times 2 24 \papercolumns 1 25 \papersides 1 26 \paperpagestyle default 27 28 \layout Title 29 \added_space_top vfill \added_space_bottom vfill 30 Linux Test Project HOWTO 31 \layout Date 32 33 10 October 2000 34 \layout Author 35 36 Nate Straz 37 \layout Abstract 38 39 This document explains some of the more in depth topics of the Linux Test 40 Project and related testing issues. 41 It does not cover basic installation procedures. 42 See the INSTALL and README files in the tarball for that information. 43 \layout Section 44 45 Preface 46 \layout Standard 47 48 This document was written to help bring the community up to speed on the 49 ins and outs of the Linux Test Project. 50 \layout Subsection 51 52 Copyright 53 \layout Standard 54 55 Copyright (c) 2000 by SGI, Inc. 56 57 \layout Standard 58 59 Please freely copy and distribute (sell or give away) this document in any 60 format. 61 It's requested that corrections and/or comments be fowarded to the document 62 maintainer. 63 You may create a derivative work and distribute it provided that you: 64 \layout Itemize 65 66 Send your derivative work (in the most suitable format such as sgml) to 67 the LDP (Linux Documentation Project) or the like for posting on the Internet. 68 If not the LDP, then let the LDP know where it is available. 69 70 \layout Itemize 71 72 License the derivative work with this same license or use GPL. 73 Include a copyright notice and at least a pointer to the license used. 74 75 \layout Itemize 76 77 Give due credit to previous authors and major contributors. 78 79 \layout Standard 80 81 If you're considering making a derived work other than a translation, it's 82 requested that you discuss your plans with the current maintainer. 83 84 \layout Subsection 85 86 Disclaimer 87 \layout Standard 88 89 Use the information in this document at your own risk. 90 I disavow any potential liability for the contents of this document. 91 Use of the concepts, examples, and/or other content of this document is 92 entirely at your own risk. 93 94 \layout Standard 95 96 All copyrights are owned by their owners, unless specifically noted otherwise. 97 Use of a term in this document should not be regarded as affecting the 98 validity of any trademark or service mark. 99 100 \layout Standard 101 102 Naming of particular products or brands should not be seen as endorsements. 103 104 \layout Standard 105 106 You are strongly recommended to take a backup of your system before major 107 installation and backups at regular intervals. 108 109 \layout Section 110 111 Introduction 112 \layout Subsection 113 114 What is the Linux Test Project? 115 \layout Standard 116 117 The Linux Test Project (LTP) is an effort to create a set of tools and tests 118 to verify the functionality and stability of the Linux kernel. 119 We hope this will support Linux development by making unit testing more 120 complete and minimizing user impact by building a barrier to keep bugs 121 from making it to the user. 122 123 \layout Subsection 124 125 What is wrong with the current testing model? 126 \layout Standard 127 128 The Linux development community utilizes two important (some out argue most 129 important) testing techniques in its normal operations: Design and Code 130 Inspections. 131 The intent of LTP is to support this by giving developers an ever growing 132 set of tools to help identify any operational problems in their code that 133 may be missed by human review. 134 One of the toughest categories of problems to catch with inspection is 135 that of interaction of features. 136 With a continuously improving set of tests and tools, developers can get 137 an indication of whether their changes may have broken some other functionality. 138 139 \layout Standard 140 141 There is no such thing as a perfect test base. 142 It is only useful it if keeps up with new and changing functionality, 143 and if it actually gets used. 144 145 \layout Subsection 146 147 Are you doing benchmarking? 148 \layout Standard 149 150 Not at this time. 151 We are more interested in functional, regression, and stress testing the 152 Linux kernel. 153 Benchmarking may be workable to compare the performance among kernel versions. 154 155 \layout Subsection 156 157 Are you doing standards testing? 158 \layout Standard 159 160 No, we are leaving that to the Linux Standards Base (LSB). 161 See the Linux Standards Base 162 \begin_inset LatexCommand \htmlurl[web site]{http://www.linuxbase.org/} 163 164 \end_inset 165 166 for more information. 167 168 \layout Section 169 170 Structure 171 \layout Standard 172 173 The basic building block of the test project is a 174 \series bold 175 test case 176 \series default 177 that consists of a single action and a verification that the action worked. 178 The result of the test case is usually restricted to PASS/FAIL. 179 180 \layout Standard 181 182 A 183 \series bold 184 test program 185 \series default 186 is a runnable program that contains one or more test cases. 187 Test programs often understand command line options which alter their behavior. 188 The options could determine the amount of memory tested, the location of 189 temporary files, the type of network packet used, or any other useful parameter. 190 \layout Standard 191 192 193 \series bold 194 Test tags 195 \series default 196 are used to pair a unique identifier with a test program and a set of command 197 line options. 198 Test tags are the basis for test suites. 199 \layout Section 200 201 Writing Tests 202 \layout Standard 203 204 Writing a test case is a lot easier than most people think. 205 Any code that you write to examine how a part of the kernel works can 206 be adapted into a test case. 207 All that is needed is a way to report the result of the action to the 208 rest of the world. 209 There are several ways of doing this, some more involved than others. 210 211 \layout Subsection 212 213 Exit Style Tests 214 \layout Standard 215 216 Probably the simplest way of reporting the results of a test case is the 217 exit status of your program. 218 If your test program encounters unexpected or incorrect results, exit 219 the test program with a non-zero exit status, i.e. 220 221 \family typewriter 222 exit(1) 223 \family default 224 . 225 Conversely, if your program completes as expected, return a zero exit status, 226 i.e. 227 228 \family typewriter 229 exit(0) 230 \family default 231 . 232 Any test driver should be able to handle this type of error reporting. 233 If a test program has multiple test cases you won't know which test case 234 failed, but you will know the program that failed. 235 236 \layout Subsection 237 238 Formatted Output Tests 239 \layout Standard 240 241 The next easiest way of reporting the results is to write the results of 242 each test case to standard output. 243 This allows for the testing results to be more understandable to both the 244 tester and the analysis tools. 245 When the results are written in a standard way, tools can be used to analyze 246 the results. 247 248 \layout Section 249 250 Testing Tools 251 \layout Standard 252 253 The Linux Test Project has not yet decided on a "final" test harness. 254 We have provided a simple solution with 255 \family typewriter 256 ltp-pan 257 \family default 258 to make due until a complete solution has been found/created that compliments 259 the Linux kernel development process. 260 Several people have said we should use such and such a test harness. 261 Until we find we need a large complex test harness, we will apply the KISS 262 concept. 263 264 \layout Subsection 265 266 Ltp-pan 267 \layout Standard 268 269 270 \family typewriter 271 ltp-pan 272 \family default 273 is a simple test driver with the ability to keep track of orphaned processes 274 and capture test output. 275 It works by reading a list of test tags and command lines and runs them. 276 By default ltp-pan will select a command randomly from the list of test tags, 277 wait for it to finish. 278 Through command line options you can run through the entire list sequentially, 279 run n tests, keep n test running at all times, and buffer test output. 280 Ltp-pan can be nested to create very complex test environments. 281 \layout Standard 282 283 Ltp-pan uses an 284 \emph on 285 active file 286 \emph default 287 , also called a 288 \emph on 289 zoo file 290 \emph default 291 to keep track of which tests are currently running. 292 This file holds the pid, tag, and a portion of the command line. 293 When you start ltp-pan it becomes a test tag in itself, thus it requires a 294 name for itself. 295 Ltp-pan updates the active file to show which test tags are currently running. 296 When a test tag exits, ltp-pan will overwrite the first character with a '#'. 297 The active file can be shared between multiple instances of ltp-pan so you 298 know which tests were running when the system crashes by looking at one 299 file. 300 301 \layout Standard 302 303 A 304 \emph on 305 ltp-pan file 306 \emph default 307 contains a list of test tags for ltp-pan to run. 308 The format of a ltp-pan file is as follows: 309 \layout Code 310 311 312 \latex no_latex 313 testtag testprogram -o one -p two other command line options 314 \layout Code 315 316 317 \latex no_latex 318 # This is a comment. 319 It is a good idea to describe the test 320 \layout Code 321 322 323 \latex no_latex 324 # tags in your ltp-pan file. 325 Tests programs can have different 326 \layout Code 327 328 329 \latex no_latex 330 # behaviors depending on the command line options so it is 331 \layout Code 332 333 334 \latex no_latex 335 # helpful to describe what each test tag is meant to verify or # provoke. 336 \layout Code 337 338 339 \latex no_latex 340 # Some more test cases 341 \layout Code 342 343 344 \latex no_latex 345 mm01 mmap001 -m 10000 346 \layout Code 347 348 349 \latex no_latex 350 # 40 Mb mmap() test. 351 \layout Code 352 353 354 \latex no_latex 355 # Creates a 10000 page mmap, touches all of the map, sync's 356 \layout Code 357 358 359 \latex no_latex 360 # it, and munmap()s it. 361 \layout Code 362 363 364 \latex no_latex 365 mm03 mmap001 -i 0 -I 1 -m 100 366 \layout Code 367 368 369 \latex no_latex 370 # repetitive mmapping test. 371 \layout Code 372 373 374 \latex no_latex 375 # Creates a one page map repetitively for one minute. 376 \layout Code 377 378 379 \latex no_latex 380 dup02 dup02 381 \layout Code 382 383 384 \latex no_latex 385 # Negative test for dup(2) with bad fd 386 \layout Code 387 388 389 \latex no_latex 390 kill09 kill09 391 \layout Code 392 393 394 \latex no_latex 395 # Basic test for kill(2) 396 \layout Code 397 398 399 \latex no_latex 400 fs-suite01 ltp-pan -e -a fs-suite01.zoo -n fs-suite01 -f runtest/fs 401 \layout Code 402 403 404 \latex no_latex 405 # run the entire set of file system tests 406 \layout Standard 407 408 The test tags are simple identifiers, no spaces are allowed. 409 The test of the line is the program to run, which is done using execvp(3). 410 Lines starting with '#' are comments and ignored by ltp-pan. 411 It is a good practice to include descriptions with your test tags so you 412 can have a reminder what a certain obscure test tag tries to do. 413 \layout Subsubsection 414 415 Examples 416 \layout Standard 417 418 The most basic way to run ltp-pan is by passing the test program and parameters 419 on the command line. 420 This will run the single program once and wrap the output. 421 422 \layout Code 423 424 425 \latex no_latex 426 $ ltp-pan -a ltp.zoo -n tutor sleep 4 427 \layout Code 428 429 430 \latex no_latex 431 <<<test_start>>> 432 \layout Code 433 434 435 \latex no_latex 436 tag=cmdln stime=971450564 437 \layout Code 438 439 440 \latex no_latex 441 cmdline="sleep 4" 442 \layout Code 443 444 445 \latex no_latex 446 contacts="" 447 \layout Code 448 449 450 \latex no_latex 451 analysis=exit 452 \layout Code 453 454 455 \latex no_latex 456 initiation_status="ok" 457 \layout Code 458 459 460 \latex no_latex 461 <<<test_output>>> 462 \layout Code 463 464 465 \latex no_latex 466 <<<execution_status>>> 467 \layout Code 468 469 470 \latex no_latex 471 duration=103341903 termination_type=exited termination_id=0 corefile=no 472 cutime=0 cstime=0 473 \layout Code 474 475 476 \latex no_latex 477 <<<test_end>>> 478 \layout Code 479 480 481 \latex no_latex 482 $ cat ltp.zoo 483 \layout Code 484 485 486 \latex no_latex 487 #9357,tutor,pan/ltp-pan -a ltp.zoo -n tutor sleep 4 488 \layout Code 489 490 491 \latex no_latex 492 #9358,cmdln,sleep 4 493 \layout Code 494 495 496 \latex no_latex 497 $ 498 \layout Paragraph 499 500 How it works 501 \layout Standard 502 503 This example shows the two parameters that are always required by ltp-pan, the 504 active file and a test tag for ltp-pan. 505 The 506 \begin_inset Quotes eld 507 \end_inset 508 509 sleep 4 510 \begin_inset Quotes erd 511 \end_inset 512 513 on the end of the command line is a test program and parameters that ltp-pan 514 should run. 515 This test is given the tag 516 \begin_inset Quotes eld 517 \end_inset 518 519 cmdln. 520 \begin_inset Quotes erd 521 \end_inset 522 523 Ltp-pan will run one test randomly, which ends up being cmdln since it is the 524 only test that we told ltp-pan about. 525 526 \layout Standard 527 528 In the active file, 529 \family typewriter 530 ltp.zoo 531 \family default 532 , ltp-pan writes the pid, test tag, and part of the command line for the currently 533 running tests. 534 The command lines are truncated so each line will fit on an 80 column display. 535 When a test tag finishes, ltp-pan will place a '#' at the beginning of the 536 line to mark it as available. 537 Here you can see that cmdln and tutor, the name we gave ltp-pan, ran to completion. 538 If the computer hangs, you can read this file to see which test programs 539 were running. 540 \layout Standard 541 542 We have run one test once. 543 Let's do something a little more exciting. 544 Let's run one test several times, at the same time. 545 546 \layout Code 547 548 549 \latex no_latex 550 $ ltp-pan -a ltp.zoo -n tutor -x 3 -s 3 -O /tmp sleep 1 551 \layout Code 552 553 554 \latex no_latex 555 <<<test_start>>> 556 \layout Code 557 558 559 \latex no_latex 560 tag=cmdln stime=971465653 561 \layout Code 562 563 564 \latex no_latex 565 cmdline="sleep 1" 566 \layout Code 567 568 569 \latex no_latex 570 contacts="" 571 \layout Code 572 573 574 \latex no_latex 575 analysis=exit 576 \layout Code 577 578 579 \latex no_latex 580 initiation_status="ok" 581 \layout Code 582 583 584 \latex no_latex 585 <<<test_output>>> 586 \layout Code 587 588 589 \latex no_latex 590 591 \layout Code 592 593 594 \latex no_latex 595 <<<execution_status>>> 596 \layout Code 597 598 599 \latex no_latex 600 duration=103326814 termination_type=exited termination_id=0 corefile=no 601 \layout Code 602 603 604 \latex no_latex 605 cutime=1 cstime=0 606 \layout Code 607 608 609 \latex no_latex 610 <<<test_end>>> 611 \layout Code 612 613 614 \latex no_latex 615 <<<test_start>>> 616 \layout Code 617 618 619 \latex no_latex 620 tag=cmdln stime=971465653 621 \layout Code 622 623 624 \latex no_latex 625 cmdline="sleep 1" 626 \layout Code 627 628 629 \latex no_latex 630 contacts="" 631 \layout Code 632 633 634 \latex no_latex 635 analysis=exit 636 \layout Code 637 638 639 \latex no_latex 640 initiation_status="ok" 641 \layout Code 642 643 644 \latex no_latex 645 <<<test_output>>> 646 \layout Code 647 648 649 \latex no_latex 650 651 \layout Code 652 653 654 \latex no_latex 655 <<<execution_status>>> 656 \layout Code 657 658 659 \latex no_latex 660 duration=103326814 termination_type=exited termination_id=0 corefile=no 661 \layout Code 662 663 664 \latex no_latex 665 cutime=0 cstime=1 666 \layout Code 667 668 669 \latex no_latex 670 <<<test_end>>> 671 \layout Code 672 673 674 \latex no_latex 675 <<<test_start>>> 676 \layout Code 677 678 679 \latex no_latex 680 tag=cmdln stime=971465653 681 \layout Code 682 683 684 \latex no_latex 685 cmdline="sleep 1" 686 \layout Code 687 688 689 \latex no_latex 690 contacts="" 691 \layout Code 692 693 694 \latex no_latex 695 analysis=exit 696 \layout Code 697 698 699 \latex no_latex 700 initiation_status="ok" 701 \layout Code 702 703 704 \latex no_latex 705 <<<test_output>>> 706 \layout Code 707 708 709 \latex no_latex 710 711 \layout Code 712 713 714 \latex no_latex 715 <<<execution_status>>> 716 \layout Code 717 718 719 \latex no_latex 720 duration=103326814 termination_type=exited termination_id=0 corefile=no 721 \layout Code 722 723 724 \latex no_latex 725 cutime=0 cstime=0 726 \layout Code 727 728 729 \latex no_latex 730 <<<test_end>>> 731 \layout Paragraph 732 733 How it works 734 \layout Standard 735 736 In this example we run another fake test from the command line, but we run 737 it three times (-s 3) and keep three test tags active at the same time 738 (-x 3). 739 The -O parameter is a directory where temporary files can be created to 740 buffer the output of each test tag. 741 You can see in the output that cmdln ran three times. 742 If the -O option were omitted, your test output would be mixed, making 743 it almost worthless. 744 745 \layout Itemize 746 747 Using a ltp-pan file to run multiple tests 748 \layout Itemize 749 750 Nesting ltp-pan 751 \layout Standard 752 753 For more information on ltp-pan see the man page 754 \family typewriter 755 doc/man1/ltp-pan.1 756 \family default 757 . 758 \layout Subsection 759 760 Scanner 761 \layout Standard 762 763 764 \family typewriter 765 Ltp-scanner 766 \family default 767 is a results analysis tool that understands the 768 \emph on 769 rts 770 \emph default 771 style output which 772 \family typewriter 773 ltp-pan 774 \family default 775 generates by default. 776 It will produce a table summarizing which tests passed and which failed. 777 778 \layout Subsection 779 780 The Quick-hitter Package 781 \layout Standard 782 783 Many of the tests released use the Quick-hitter test package to perform 784 tasks like create and move to a temporary directory, handle some common 785 command line parameters, loop, run in parallel, handle signals, and clean 786 up. 787 788 \layout Standard 789 790 There is an example test case, 791 \family typewriter 792 doc/examples/quickhit.c 793 \family default 794 , which shows how the quick-hitter package can be used. 795 The file is meant to be a supplement to the documentation, not a working 796 test case. 797 Use any of the tests in 798 \family typewriter 799 tests/ 800 \family default 801 as a template. 802 \layout Section 803 804 To Do 805 \layout Standard 806 807 There are a lot of things that still need to be done to make this a complete 808 kernel testing system. 809 The following sections will discuss some of the to do items in detail. 810 811 \layout Subsection 812 813 Configuration Analysis 814 \layout Standard 815 816 While the number of configuration options for the Linux kernel is seen as 817 a strength to developers and users alike, it is a curse to testers. 818 To create a powerful automated testing system, we need to be able to determine 819 what the configuration on the booted box is and then determine which tests 820 should be run on that box. 821 822 \layout Standard 823 824 The Linux kernel has hundreds of configuration options that can be set to 825 compile the kernel. 826 There are more options that can be set when you boot the kernel and while 827 it is running. 828 There are also many patches that can be applied to the kernel to add functiona 829 lity or change behavior. 830 831 \layout Subsection 832 833 Result Comparison 834 \layout Standard 835 836 A lot of testing will be done in the life of the Linux Test Project. 837 Keeping track of the results from all the testing will require some infrastruct 838 ure. 839 It would be nice to take that output from a test machine, feed it to a 840 program and receive a list of items that broke since the last run on that 841 machine, or were fixed, or work on another test machine but not on this 842 one. 843 844 \layout Section 845 846 Contact information and updates 847 \layout Literal 848 849 URL: http://ltp.sourceforge.net/ 850 \layout Literal 851 852 mailing list: ltp (a] lists.linux.it 853 \layout Literal 854 855 list archive: http://lists.linux.it/pipermail/ltp/ 856 \layout Standard 857 858 Questions and comments should be sent to the LTP mailing 859 list at ltp (a] lists.linux.it. To subscribe, please go to 860 http://lists.linux.it/pipermail/ltp/. 861 862 \layout Standard 863 864 The source is also available via CVS. 865 See the web site for a web interface and check out instructions. 866 867 \layout Section 868 869 Glossary 870 \layout Description 871 872 Test IEEE/ANSI 873 \begin_float footnote 874 \layout Standard 875 876 Kit, Edward, Software Testing in the Real World: Improving the Process. 877 P. 878 82. 879 ACM Press, 1995. 880 \end_float 881 : 882 \shape italic 883 884 \newline 885 886 \shape default 887 888 \shape italic 889 (i) 890 \shape default 891 An activity in which a system or component is executed under specified 892 conditions, the results are observed or record, and an evaluation is made 893 of some aspect of the system or component. 894 895 \shape italic 896 897 \newline 898 899 \shape default 900 901 \shape italic 902 (ii) 903 \shape default 904 A set of one or more test cases. 905 906 \layout Description 907 908 Test\SpecialChar ~ 909 Case A test assertion with a single result that is being verified. 910 This allows designations such as PASS or FAIL to be applied to a single 911 bit of functionality. 912 A single test case may be one of many test cases for testing the complete 913 functionality of a system. 914 915 \newline 916 IEEE/ANSI: 917 \shape italic 918 919 \newline 920 (i) 921 \shape default 922 A set of test inputs, execution conditions, and expected results developed 923 for a particular objective. 924 925 \shape italic 926 927 \newline 928 (ii) 929 \shape default 930 The smallest entity that is always executed as a unit, from beginning to 931 end. 932 933 \layout Description 934 935 Test\SpecialChar ~ 936 Driver A program that handles the execution of test programs. 937 It is responsible for starting the test programs, capturing their output, 938 and recording their results. 939 Ltp-pan is an example of a test driver. 940 \layout Description 941 942 Test\SpecialChar ~ 943 Framework A mechanism for organizing a group of tests. 944 Frameworks may have complex or very simple API's, drivers and result logging 945 mechanisms. 946 Examples of frameworks are TETware and DejaGnu. 947 948 \layout Description 949 950 Test\SpecialChar ~ 951 Harness A Test harness is the mechanism that connects a test program 952 to a test framework. 953 It may be a specification of exit codes, or a set of libraries for formatting 954 messages and determining exit codes. 955 In TETware, the tet_result() API is the test harness. 956 957 \layout Description 958 959 Test\SpecialChar ~ 960 Program A single invokable program. 961 A test program can contain one or more test cases. 962 The test harness's API allows for reporting/analysis of the individual 963 test cases. 964 965 \layout Description 966 967 Test\SpecialChar ~ 968 Suite A collection of tests programs, assertions, cases grouped together 969 under a framework. 970 971 \layout Description 972 973 Test\SpecialChar ~ 974 Tag An identifier that corresponds to a command line which runs a test. 975 The tag is a single word that matches a test program with a set of command 976 line arguments. 977 978 \the_end 979
