1 LTP Test Writing Guidelines 2 =========================== 3 4 This document describes LTP guidelines and LTP test interface and is intended 5 for anybody who want to write or modify a LTP testcase. It's not a definitive 6 guide and it's not, by any means, a substitute for common sense. 7 8 1. General Rules 9 ---------------- 10 11 1.1 Simplicity 12 ~~~~~~~~~~~~~~ 13 14 For all it's worth keep the testcases simple or better as simple as possible. 15 The kernel and libc are tricky beasts and the complexity imposed by their 16 interfaces is quite high. Concentrate on the interface you want to test and 17 follow the UNIX philosophy. It's a good idea to make the test as 18 self-contained as possible too (it should not depend on tools or libraries 19 that are not widely available). 20 21 Do not reinvent the wheel! 22 23 * Use LTP standard interface 24 * Do not add custom PASS/FAIL reporting functions 25 * Do not write Makefiles from scratch, 26 use LTP build system instead, etc. 27 * ... 28 29 1.2 Code duplication 30 ~~~~~~~~~~~~~~~~~~~~ 31 32 Copy & paste is a good servant but very poor master. If you are about to copy a 33 large part of the code from one testcase to another, think what would happen if 34 you find bug in the code that has been copied all around the tree. What about 35 moving it to a library instead? 36 37 The same goes for short but complicated parts, whenever you are about to copy & 38 paste a syscall wrapper that packs arguments accordingly to machine 39 architecture or similarly complicated code, put it into a header instead. 40 41 1.3 Coding style 42 ~~~~~~~~~~~~~~~~ 43 44 1.3.1 C coding style 45 ^^^^^^^^^^^^^^^^^^^^ 46 47 LTP adopted Linux kernel coding style. If you aren't familiar with its rules 48 locate 'linux/Documentation/CodingStyle' in the kernel sources and read it, 49 it's a well written introduction. 50 51 There is also a checkpatch (see 'linux/scripts/checkpatch.pl') script that can 52 be used to check your patches before the submission. 53 54 NOTE: If checkpatch does not report any problems, the code still may be wrong 55 as the tool only looks for common mistakes. 56 57 1.3.2 Shell coding style 58 ^^^^^^^^^^^^^^^^^^^^^^^^ 59 60 When writing testcases in shell write in *portable shell* only, it's a good 61 idea to try to run the test using alternative shell (alternative to bash, for 62 example dash) too. 63 64 *Portable shell* means Shell Command Language as defined by POSIX with a 65 exception of few widely used extensions, namely 'local' keyword used inside of 66 functions and '-o' and '-a' test parameters (that are marked as obsolete in 67 POSIX). 68 69 You can either try to run the testcases on Debian which has '/bin/sh' pointing 70 to 'dash' by default or install 'dash' on your favorite distribution and use 71 it to run the tests. If your distribution lacks 'dash' package you can always 72 compile it from http://gondor.apana.org.au/~herbert/dash/files/[source]. 73 74 Debian also has nice devscript 75 https://anonscm.debian.org/cgit/collab-maint/devscripts.git/tree/scripts/checkbashisms.pl[checkbashism.pl] 76 that can be used to check for non-portable shell code. 77 78 Here are some common sense style rules for shell 79 80 * Keep lines under 80 chars 81 82 * Use tabs for indentation 83 84 * Keep things simple, avoid unnecessary subshells 85 86 * Don't do confusing things (i.e. don't name your functions like common shell 87 commands, etc.) 88 89 * Quote variables 90 91 * Be consistent 92 93 1.4 Commenting code 94 ~~~~~~~~~~~~~~~~~~~ 95 96 Comments can sometimes save you day but they can easily do more harm than 97 good. There has been several cases where comments and actual implementation 98 were drifting slowly apart which yielded into API misuses and hard to find 99 bugs. Remember there is only one thing worse than no documentation, wrong 100 documentation. 101 102 Generally everybody should write code that is obvious (which unfortunately 103 isn't always possible). If there is a code that needs to be commented keep it 104 short and to the point. Never ever comment the obvious. 105 106 In case of LTP testcases it's customary to add a paragraph with highlevel test 107 description somewhere at the beginning of the file (usually right under the GPL 108 header). This helps other people to understand the overall goal of the test 109 before they dive into the technical details. 110 111 1.5 Backwards compatibility 112 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 113 114 LTP test should be as backward compatible as possible. Think of an enterprise 115 distributions with long term support (more than five years since the initial 116 release) or of an embedded platform that needs to use several years old 117 toolchain supplied by the manufacturer. 118 119 Therefore LTP test for more current features should be able to cope with older 120 systems. It should at least compile fine and if it's not appropriate for the 121 configuration it should return 'TCONF' (see test interface description below). 122 123 There are several types of checks we use: 124 125 The *configure script* is usually used to detect availability of a function 126 declarations in system headers. It's used to disable tests at compile time. 127 128 We also have runtime kernel version detection that can be used to disable 129 tests at runtime. 130 131 Checking the *errno* value is another type of runtime check. Most of the 132 syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented 133 or was disabled upon kernel compilation. 134 135 Sometimes it also makes sense to define a few macros instead of creating 136 configure test. One example are Linux specific POSIX clock ids in 137 'include/lapi/posix_clocks.h'. 138 139 1.6 Dealing with messed up legacy code 140 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 141 142 LTP contains a lot of old and messy code and we are cleaning it up as fast as 143 we can but despite the efforts there is still a lot. If you start modifying 144 old or a messed up testcase and your changes are more complicated than simple 145 typo fixes you should do a cleanup first (in a separate patch). It's easier to 146 review the changes if you separate the formatting fixes from the changes that 147 affects the test behavior. 148 149 The same goes for moving files. If you need a rename or move file do it in a 150 separate patch. 151 152 1.7 License 153 ~~~~~~~~~~~ 154 155 Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or 156 any later version). 157 158 2. Writing a testcase 159 --------------------- 160 161 2.1 LTP Structure 162 ~~~~~~~~~~~~~~~~~ 163 164 The structure of LTP is quite simple. Each test is a binary written either in 165 portable shell or C. The test gets a configuration via environment variables 166 and/or command line parameters, it prints additional information into the 167 stdout and reports overall success/failure via the exit value. 168 169 Tests are generally placed under the 'testcases/' directory. Everything that 170 is a syscall or (slightly confusingly) libc syscall wrapper goes under 171 'testcases/kernel/syscalls/'. Then there is 'testcases/open_posix_testsuite' 172 which is a well maintained fork of the upstream project that has been dead 173 since 2005 and also a number of directories with tests for more specific 174 features. 175 176 2.1.1 Runtest Files 177 ^^^^^^^^^^^^^^^^^^^ 178 179 The list of tests to be executed is stored in runtest files under the 180 'runtest/' directory. The default set of runtest files to be executed is 181 stored in 'scenario_groups/default'. When you add a test you should add 182 corresponding entries into some runtest file(s) as well. 183 184 For syscall tests (these placed under 'testcases/kernel/syscalls/') use 185 'runtest/syscalls' file, for kernel related tests for memory management we 186 have 'runtest/mm', etc. 187 188 IMPORTANT: The runtest files should have one entry per a test. Creating a 189 wrapper that runs all your tests and adding it as a single test 190 into runtest file is strongly discouraged. 191 192 2.1.2 Datafiles 193 ^^^^^^^^^^^^^^^ 194 195 If your test needs datafiles to work, these should be put into a subdirectory 196 named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to 197 do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the 198 'datafiles/Makefile'). 199 200 You can obtain path to datafiles via $TST_DATAROOT provided by test.sh 201 '$TST_DATAROOT/...' 202 or via C function 'tst_dataroot()' provided by libltp: 203 204 [source,c] 205 ------------------------------------------------------------------------------- 206 const char *dataroot = tst_dataroot(); 207 ------------------------------------------------------------------------------- 208 209 Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...', 210 but '$TST_DATAROOT' and 'tst_dataroot()' are preferred as these can be used 211 when running testcases directly in git tree as well as from install 212 location. 213 214 The path is constructed according to these rules: 215 216 1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID' 217 2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles' 218 (where '$STARTWD' is initial working directory as recorded by 'tst_tmpdir()') 219 3. else return '$CWD/datafiles' 220 221 See 'testcases/commands/file/' for example. 222 223 2.1.3 Subexecutables 224 ^^^^^^^^^^^^^^^^^^^^ 225 226 If you test needs to execute a binary, place it in the same directory as the 227 testcase and name the file starting with '${test_binary_name}_'. Once the 228 test is executed by the framework, the path to the directory with all LTP 229 binaries is added to the '$PATH' and you can execute it just by its name. 230 231 TIP: If you need to execute such test from the LTP tree, you can add path to 232 current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'. 233 234 2.2 Writing a test in C 235 ~~~~~~~~~~~~~~~~~~~~~~~ 236 237 2.2.1 Basic test structure 238 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 239 240 Let's start with an example, following code is a simple test for a 'getenv()'. 241 242 [source,c] 243 ------------------------------------------------------------------------------- 244 /* 245 * This is test for basic functionality of getenv(). 246 * 247 * - create an env variable and verify that getenv() can get get it 248 * - call getenv() with nonexisting variable name, check that it returns NULL 249 */ 250 251 #include "tst_test.h" 252 253 #define ENV1 "LTP_TEST_ENV" 254 #define ENV2 "LTP_TEST_THIS_DOES_NOT_EXIST" 255 #define ENV_VAL "val" 256 257 static void setup(void) 258 { 259 if (setenv(ENV1, ENV_VAL, 1)) 260 tst_brk(TBROK | TERRNO, "setenv() failed"); 261 } 262 263 static void test(void) 264 { 265 char *ret; 266 267 ret = getenv(ENV1); 268 269 if (!ret) { 270 tst_res(TFAIL, "getenv(" ENV1 ") = NULL"); 271 goto next; 272 } 273 274 if (!strcmp(ret, ENV_VAL)) { 275 tst_res(TPASS, "getenv(" ENV1 ") = '"ENV_VAL "'"); 276 } else { 277 tst_res(TFAIL, "getenv(" ENV1 ") = '%s', expected '" 278 ENV_VAL "'", ret); 279 } 280 281 next: 282 ret = getenv(ENV2); 283 284 if (ret) 285 tst_res(TFAIL, "getenv(" ENV2 ") = '%s'", ret); 286 else 287 tst_res(TPASS, "getenv(" ENV2 ") = NULL"); 288 } 289 290 static struct tst_test test = { 291 .test_all = test, 292 .setup = setup, 293 }; 294 ------------------------------------------------------------------------------- 295 296 Each test includes the 'tst_test.h' header and must define the 'struct 297 tst_test test' structure. 298 299 The overall test initialization is done in the 'setup()' function. 300 301 The overall cleanup is done in a 'cleanup()' function. Here 'cleanup()' is 302 omitted as the test does not have anything to clean up. If cleanup is set in 303 the test structure it's called on test exit just before the test library 304 cleanup. That especially means that cleanup can be called at any point in a 305 test execution. For example even when a test setup step has failed, therefore 306 the 'cleanup()' function must be able to cope with unfinished initialization, 307 and so on. 308 309 The test itself is done in the 'test()' function. The test function must work 310 fine if called in a loop. 311 312 There are two types of a test function pointers in the test structure. The 313 first one is a '.test_all' pointer that is used when test is implemented as a 314 single function. Then there is a '.test' function along with the number of 315 tests '.tcnt' that allows for more detailed result reporting. If the '.test' 316 pointer is set the function is called '.tcnt' times with an integer parameter 317 in range of [0, '.tcnt' - 1]. 318 319 IMPORTANT: Only one of '.test' and '.test_all' can be set at a time. 320 321 Each test has a default timeout set to 300s. The default timeout can be 322 overriden by setting '.timeout' in the test structure or by calling 323 'tst_set_timeout()' in the test 'setup()'. There are a few testcases whose run 324 time may vary arbitrarily, for these timeout can be disabled by setting it to 325 -1. 326 327 A word about the cleanup() callback 328 +++++++++++++++++++++++++++++++++++ 329 330 There are a few rules that needs to be followed in order to write correct 331 cleanup() callback. 332 333 1. Free only resources that were initialized. Keep in mind that callback can 334 be executed at any point in the test run. 335 336 2. Make sure to free resources in the reverse order they were 337 initialized. (Some of the steps may not depend on others and everything 338 will work if there were swapped but let's keep it in order.) 339 340 The first rule may seem complicated at first however, on the contrary, it's 341 quite easy. All you have to do is to keep track of what was already 342 initialized. For example file descriptors needs to be closed only if they were 343 assigned a valid file descriptor. For most of the things you need to create 344 extra flag that is set right after successful initialization though. Consider, 345 for example, test setup below. 346 347 [source,c] 348 ------------------------------------------------------------------------------- 349 static int fd0, fd1, mount_flag; 350 351 #define MNTPOINT "mntpoint" 352 #define FILE1 "mntpoint/file1" 353 #define FILE2 "mntpoint/file2" 354 355 static void setup(void) 356 { 357 SAFE_MKDIR(MNTPOINT, 0777); 358 SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL); 359 SAFE_MOUNT(tst_device->dev, MNTPOINT, tst_device->fs_type, 0, 0); 360 mount_flag = 1; 361 362 fd0 = SAFE_OPEN(cleanup, FILE1, O_CREAT | O_RDWR, 0666); 363 fd1 = SAFE_OPEN(cleanup, FILE2, O_CREAT | O_RDWR, 0666); 364 } 365 ------------------------------------------------------------------------------- 366 367 In this case the 'cleanup()' function may be invoked when any of the 'SAFE_*' 368 macros has failed and therefore must be able to work with unfinished 369 initialization as well. Since global variables are initialized to zero we can 370 just check that fd > 0 before we attempt to close it. The mount function 371 requires extra flag to be set after device was successfully mounted. 372 373 [source,c] 374 ------------------------------------------------------------------------------- 375 static void cleanup(void) 376 { 377 if (fd1 > 0) 378 SAFE_CLOSE(fd1); 379 380 if (fd0 > 0) 381 SAFE_CLOSE(fd0); 382 383 if (mount_flag && tst_umouont(MNTPOINT)) 384 tst_res(TWARN | TERRNO, "umount(%s)", MNTPOINT); 385 } 386 ------------------------------------------------------------------------------- 387 388 IMPORTANT: 'SAFE_MACROS()' used in cleanup *do not* exit the test. Failure 389 only produces a warning and the 'cleanup()' carries on. This is 390 intentional as we want to execute as much 'cleanup()' as possible. 391 392 WARNING: Calling tst_brk() in test 'cleanup()' does not exit the test as well 393 and 'TBROK' is converted to 'TWARN'. 394 395 NOTE: Creation and removal of the test temporary directory is handled in 396 the test library and the directory is removed recursively. Therefore 397 we do not have to remove files and directories in the test cleanup. 398 399 2.2.2 Basic test interface 400 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 401 402 [source,c] 403 ------------------------------------------------------------------------------- 404 void tst_res(int ttype, char *arg_fmt, ...); 405 ------------------------------------------------------------------------------- 406 407 Printf-like function to report test result, it's mostly used with ttype: 408 409 |============================== 410 | 'TPASS' | Test has passed. 411 | 'TFAIL' | Test has failed. 412 | 'TINFO' | General message. 413 |============================== 414 415 The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print 416 'errno', 'TEST_ERRNO' respectively. 417 418 [source,c] 419 ------------------------------------------------------------------------------- 420 void tst_brk(int ttype, char *arg_fmt, ...); 421 ------------------------------------------------------------------------------- 422 423 Printf-like function to report error and exit the test, it can be used with ttype: 424 425 |============================================================ 426 | 'TBROK' | Something has failed in test preparation phase. 427 | 'TCONF' | Test is not appropriate for current configuration 428 (syscall not implemented, unsupported arch, ...) 429 |============================================================ 430 431 The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print 432 'errno', 'TEST_ERRNO' respectively. 433 434 [source,c] 435 ------------------------------------------------------------------------------- 436 const char *tst_strsig(int sig); 437 ------------------------------------------------------------------------------- 438 439 Return the given signal number's corresponding string. 440 441 [source,c] 442 ------------------------------------------------------------------------------- 443 const char *tst_strerrno(int err); 444 ------------------------------------------------------------------------------- 445 446 Return the given errno number's corresponding string. Using this function to 447 translate 'errno' values to strings is preferred. You should not use the 448 'strerror()' function in the testcases. 449 450 [source,c] 451 ------------------------------------------------------------------------------- 452 const char *tst_strstatus(int status); 453 ------------------------------------------------------------------------------- 454 455 Returns string describing the status as returned by 'wait()'. 456 457 WARNING: This function is not thread safe. 458 459 [source,c] 460 ------------------------------------------------------------------------------- 461 void tst_set_timeout(unsigned int timeout); 462 ------------------------------------------------------------------------------- 463 464 Allows for setting timeout per test iteration dymanically in the test setup(), 465 the timeout is specified in seconds. There are a few testcases whose runtime 466 can vary arbitrarily, these can disable timeouts by setting it to -1. 467 468 2.2.3 Test temporary directory 469 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 470 471 If '.needs_tmpdir' is set to '1' in the 'struct tst_test' unique test 472 temporary is created and it's set as the test working directory. Tests *MUST 473 NOT* create temporary files outside that directory. 474 475 IMPORTANT: Close all file descriptors (that point to files in test temporary 476 directory, even the unlinked ones) either in the 'test()' function 477 or in the test 'cleanup()' otherwise the test may break temporary 478 directory removal on NFS (look for "NFS silly rename"). 479 480 [[2.2.4]] 481 2.2.4 Safe macros 482 ^^^^^^^^^^^^^^^^^ 483 484 Safe macros aim to simplify error checking in test preparation. Instead of 485 calling system API functions, checking for their return value and aborting the 486 test if the operation has failed, you just use corresponding safe macro. 487 488 Use them whenever it's possible. 489 490 Instead of writing: 491 492 [source,c] 493 ------------------------------------------------------------------------------- 494 fd = open("/dev/null", O_RDONLY); 495 if (fd < 0) 496 tst_brk(TBROK | TERRNO, "opening /dev/null failed"); 497 ------------------------------------------------------------------------------- 498 499 You write just: 500 501 [source,c] 502 ------------------------------------------------------------------------------- 503 fd = SAFE_OPEN("/dev/null", O_RDONLY); 504 ------------------------------------------------------------------------------- 505 506 IMPORTANT: The SAFE_CLOSE() function also sets the passed file descriptor to -1 507 after it's successfully closed. 508 509 They can also simplify reading and writing of sysfs files, you can, for 510 example, do: 511 512 [source,c] 513 ------------------------------------------------------------------------------- 514 SAFE_FILE_SCANF("/proc/sys/kernel/pid_max", "%lu", &pid_max); 515 ------------------------------------------------------------------------------- 516 517 See 'include/tst_safe_macros.h', 'include/tst_safe_stdio.h' and 518 'include/tst_safe_file_ops.h' and 'include/tst_safe_net.h' for a complete list. 519 520 2.2.5 Test specific command line options 521 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 522 523 [source,c] 524 ------------------------------------------------------------------------------- 525 struct tst_option { 526 char *optstr; 527 char **arg; 528 char *help; 529 }; 530 ------------------------------------------------------------------------------- 531 532 Test specific command line parameters can be passed with the 'NULL'-terminated 533 array of 'struct tst_option'. The 'optstr' is the command line option i.e. "o" 534 or "o:" if option has a parameter. Only short options are supported. The 'arg' 535 is where 'optarg' is stored upon match. If option has no parameter it's set to 536 non-'NULL' value if option was present. The 'help' is a short help string. 537 538 NOTE: The test parameters must not collide with common test parameters defined 539 in the library the currently used ones are +-i+, +-I+, +-C+, and +-h+. 540 541 [source,c] 542 ------------------------------------------------------------------------------- 543 int tst_parse_int(const char *str, int *val, int min, int max); 544 int tst_parse_float(const char *str, float *val, float min, float max); 545 ------------------------------------------------------------------------------- 546 547 Helpers for parsing the the strings returned in the 'struct tst_option'. 548 549 Both return zero on success and 'errno', mostly 'EINVAL' or 'ERANGE', on 550 failure. 551 552 Both functions are no-op if 'str' is 'NULL'. 553 554 The valid range for result includes both 'min' and 'max'. 555 556 .Example Usage 557 [source,c] 558 ------------------------------------------------------------------------------- 559 #include <limits.h> 560 #include "tst_test.h" 561 562 static char *str_threads; 563 static int threads = 10; 564 565 static struct tst_option options[] = { 566 {"t:", &str_threads, "Number of threads (default 10)"}, 567 ... 568 {NULL, NULL, NULL} 569 }; 570 571 static void setup(void) 572 { 573 if (tst_parse_int(str_threads, &threads, 1, INT_MAX)) 574 tst_brk(TBROK, "Invalid number of threads '%s'", str_threads); 575 576 ... 577 } 578 579 static void test_threads(void) 580 { 581 ... 582 583 for (i = 0; i < threads; i++) { 584 ... 585 } 586 587 ... 588 } 589 590 static struct tst_test test = { 591 ... 592 .options = options, 593 ... 594 }; 595 ------------------------------------------------------------------------------- 596 597 598 2.2.6 Runtime kernel version detection 599 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 600 601 Testcases for newly added kernel functionality require kernel newer than a 602 certain version to run. All you need to skip a test on older kernels is to 603 set the '.min_kver' string in the 'struct tst_test' to a minimal required 604 kernel version, e.g. '.min_kver = "2.6.30"'. 605 606 For more complicated operations such as skipping a test for a certain range 607 of kernel versions, following functions could be used: 608 609 [source,c] 610 ------------------------------------------------------------------------------- 611 int tst_kvercmp(int r1, int r2, int r3); 612 613 struct tst_kern_exv { 614 char *dist_name; 615 char *extra_ver; 616 }; 617 618 int tst_kvercmp2(int r1, int r2, int r3, struct tst_kern_exv *vers); 619 ------------------------------------------------------------------------------- 620 621 These two functions are intended for runtime kernel version detection. They 622 parse the output from 'uname()' and compare it to the passed values. 623 624 The return value is similar to the 'strcmp()' function, i.e. zero means equal, 625 negative value means that the kernel is older than than the expected value and 626 positive means that it's newer. 627 628 The second function 'tst_kvercmp2()' allows for specifying per-vendor table of 629 kernel versions as vendors typically backport fixes to their kernels and the 630 test may be relevant even if the kernel version does not suggests so. See 631 'testcases/kernel/syscalls/inotify/inotify04.c' for example usage. 632 633 WARNING: The shell 'tst_kvercmp' maps the result into unsigned integer - the 634 process exit value. 635 636 2.2.7 Fork()-ing 637 ^^^^^^^^^^^^^^^^ 638 639 Be wary that if the test forks and there were messages printed by the 640 'tst_*()' interfaces, the data may still be in libc/kernel buffers and these 641 *ARE NOT* flushed automatically. 642 643 This happens when 'stdout' gets redirected to a file. In this case, the 644 'stdout' is not line buffered, but block buffered. Hence after a fork content 645 of the buffers will be printed by the parent and each of the children. 646 647 To avoid that you should use 'SAFE_FORK()'. 648 649 IMPORTANT: You have to set the '.forks_child' flag in the test structure 650 if your testcase forks. 651 652 [[2.2.8]] 653 2.2.8 Doing the test in the child process 654 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 655 656 Results reported by 'tst_res()' are propagated to the parent test process via 657 block of shared memory. 658 659 Calling 'tst_brk()' causes child process to exit with non-zero exit value. 660 Which means that it's safe to use 'SAFE_*()' macros in the child processes as 661 well. 662 663 Children that outlive the 'test()' function execution are waited for in the 664 test library. Unclean child exit (killed by signal, non-zero exit value, etc.) 665 will cause the main test process to exit with 'tst_brk()', which especially 666 means that 'TBROK' propagated from a child process will cause the whole test 667 to exit with 'TBROK'. 668 669 If a test needs a child that segfaults or does anything else that cause it to 670 exit uncleanly all you need to do is to wait for such children from the 671 'test()' function so that it's reaped before the main test exits the 'test()' 672 function. 673 674 [source,c] 675 ------------------------------------------------------------------------------- 676 #include "tst_test.h" 677 678 void tst_reap_children(void); 679 ------------------------------------------------------------------------------- 680 681 The 'tst_reap_children()' function makes the process wait for all of its 682 children and exits with 'tst_brk(TBROK, ...)' if any of them returned 683 a non zero exit code. 684 685 2.2.9 Fork() and Parent-child synchronization 686 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 687 688 As LTP tests are written for Linux, most of the tests involve fork()-ing and 689 parent-child process synchronization. LTP includes a checkpoint library that 690 provides wait/wake futex based functions. 691 692 In order to use checkpoints the '.needs_checkpoints' flag in the 'struct 693 tst_test' must be set to '1', this causes the test library to initialize 694 checkpoints before the 'test()' function is called. 695 696 [source,c] 697 ------------------------------------------------------------------------------- 698 #include "tst_test.h" 699 700 TST_CHECKPOINT_WAIT(id) 701 702 TST_CHECKPOINT_WAIT2(id, msec_timeout) 703 704 TST_CHECKPOINT_WAKE(id) 705 706 TST_CHECKPOINT_WAKE2(id, nr_wake) 707 708 TST_CHECKPOINT_WAKE_AND_WAIT(id) 709 ------------------------------------------------------------------------------- 710 711 The checkpoint interface provides pair of wake and wait functions. The 'id' is 712 unsigned integer which specifies checkpoint to wake/wait for. As a matter of 713 fact it's an index to an array stored in a shared memory, so it starts on 714 '0' and there should be enough room for at least of hundred of them. 715 716 The 'TST_CHECKPOINT_WAIT()' and 'TST_CHECKPOINT_WAIT2()' suspends process 717 execution until it's woken up or until timeout is reached. 718 719 The 'TST_CHECKPOINT_WAKE()' wakes one process waiting on the checkpoint. 720 If no process is waiting the function retries until it success or until 721 timeout is reached. 722 723 If timeout has been reached process exits with appropriate error message (uses 724 'tst_brk()'). 725 726 The 'TST_CHECKPOINT_WAKE2()' does the same as 'TST_CHECKPOINT_WAKE()' but can 727 be used to wake precisely 'nr_wake' processes. 728 729 The 'TST_CHECKPOINT_WAKE_AND_WAIT()' is a shorthand for doing wake and then 730 immediately waiting on the same checkpoint. 731 732 Child processes created via 'SAFE_FORK()' are ready to use the checkpoint 733 synchronization functions, as they inherited the mapped page automatically. 734 735 Child processes started via 'exec()', or any other processes not forked from 736 the test process must initialize the checkpoint by calling 'tst_reinit()'. 737 738 For the details of the interface, look into the 'include/tst_checkpoint.h'. 739 740 [source,c] 741 ------------------------------------------------------------------------------- 742 #include "tst_test.h" 743 744 /* 745 * Waits for process state change. 746 * 747 * The state is one of the following: 748 * 749 * R - process is running 750 * S - process is sleeping 751 * D - process sleeping uninterruptibly 752 * Z - zombie process 753 * T - process is traced 754 */ 755 TST_PROCESS_STATE_WAIT(pid, state) 756 ------------------------------------------------------------------------------- 757 758 The 'TST_PROCESS_STATE_WAIT()' waits until process 'pid' is in requested 759 'state'. The call polls +/proc/pid/stat+ to get this information. 760 761 It's mostly used with state 'S' which means that process is sleeping in kernel 762 for example in 'pause()' or any other blocking syscall. 763 764 2.2.10 Signal handlers 765 ^^^^^^^^^^^^^^^^^^^^^^ 766 767 If you need to use signal handlers, keep the code short and simple. Don't 768 forget that the signal handler is called asynchronously and can interrupt the 769 code execution at any place. 770 771 This means that problems arise when global state is changed both from the test 772 code and signal handler, which will occasionally lead to: 773 774 * Data corruption (data gets into inconsistent state), this may happen, for 775 example, for any operations on 'FILE' objects. 776 777 * Deadlock, this happens, for example, if you call 'malloc(2)', 'free(2)', 778 etc. from both the test code and the signal handler at the same time since 779 'malloc' has global lock for it's internal data structures. (Be wary that 780 'malloc(2)' is used by the libc functions internally too.) 781 782 * Any other unreproducible and unexpected behavior. 783 784 Quite common mistake is to call 'exit(3)' from a signal handler. Note that this 785 function is not signal-async-safe as it flushes buffers, etc. If you need to 786 exit a test immediately from a signal handler use '_exit(2)' instead. 787 788 TIP: See 'man 7 signal' for the list of signal-async-safe functions. 789 790 If a signal handler sets a variable, its declaration must be 'volatile', 791 otherwise compiler may misoptimize the code. This is because the variable may 792 not be changed in the compiler code flow analysis. There is 'sig_atomic_t' 793 type defined in C99 but this one *DOES NOT* imply 'volatile' (it's just a 794 'typedef' to 'int'). So the correct type for a flag that is changed from a 795 signal handler is either 'volatile int' or 'volatile sig_atomic_t'. 796 797 2.2.11 Kernel Modules 798 ^^^^^^^^^^^^^^^^^^^^^ 799 800 There are certain cases where the test needs a kernel part and userspace part, 801 happily, LTP can build a kernel module and then insert it to the kernel on test 802 start for you. See 'testcases/kernel/device-drivers/block' for details. 803 804 2.2.11 Useful macros 805 ^^^^^^^^^^^^^^^^^^^^^ 806 807 [source,c] 808 ------------------------------------------------------------------------------- 809 ARRAY_SIZE(arr) 810 ------------------------------------------------------------------------------- 811 812 Returns the size of statically defined array, i.e. 813 '(sizeof(arr) / sizeof(*arr))' 814 815 [source,c] 816 ------------------------------------------------------------------------------- 817 LTP_ALIGN(x, a) 818 ------------------------------------------------------------------------------- 819 820 Aligns the x to be next multiple of a. The a must be power of 2. 821 822 2.2.12 Filesystem type detection 823 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 824 825 Some tests are known to fail on certain filesystems (you cannot swap on TMPFS, 826 there are unimplemented 'fcntl()' etc.). 827 828 If your test needs to be skipped on certain filesystems, use the interface 829 below: 830 831 [source,c] 832 ------------------------------------------------------------------------------- 833 #include "tst_test.h" 834 835 /* 836 * Unsupported only on NFS. 837 */ 838 if (tst_fs_type(".") == TST_NFS_MAGIC) 839 tst_brk(TCONF, "Test not supported on NFS filesystem"); 840 841 842 /* 843 * Unsupported on NFS, TMPFS and RAMFS 844 */ 845 long type; 846 847 switch ((type = tst_fs_type("."))) { 848 case TST_NFS_MAGIC: 849 case TST_TMPFS_MAGIC: 850 case TST_RAMFS_MAGIC: 851 tst_brk(TCONF, "Test not supported on %s filesystem", 852 tst_fs_type_name(type)); 853 break; 854 } 855 ------------------------------------------------------------------------------- 856 857 2.2.13 Thread-safety in the LTP library 858 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 859 860 It is safe to use library 'tst_res()' function in multi-threaded tests. 861 862 Only the main thread must return from the 'test()' function to the test 863 library and that must be done only after all threads that may call any library 864 function has been terminated. That especially means that threads that may call 865 'tst_brk()' must terminate before the execution of the 'test()' function 866 returns to the library. This is usually done by the main thread joining all 867 worker threads at the end of the 'test()' function. Note that the main thread 868 will never get to the library code in a case that 'tst_brk()' was called from 869 one of the threads since it will sleep at least in 'pthread_join()' on the 870 thread that called the 'tst_brk()' till 'exit()' is called by 'tst_brk()'. 871 872 The test-supplied cleanup function runs *concurrently* to the rest of the 873 threads in a case that cleanup was entered from 'tst_brk()'. Subsequent 874 threads entering 'tst_brk()' must be suspended or terminated at the start of 875 the the user supplied cleanup function. It may be necessary to stop or exit 876 the rest of the threads before the test cleans up as well. For example threads 877 that create new files should be stopped before temporary directory is be 878 removed. 879 880 Following code example shows thread safe cleanup function example using atomic 881 increment as a guard. The library calls its cleanup after the execution returns 882 from the user supplied cleanup and expects that only one thread returns from 883 the user supplied cleanup to the test library. 884 885 [source,c] 886 ------------------------------------------------------------------------------- 887 #include "tst_test.h" 888 889 static void cleanup(void) 890 { 891 static int flag; 892 893 if (tst_atomic_inc(&flag) != 1) 894 pthread_exit(NULL); 895 896 /* if needed stop the rest of the threads here */ 897 898 ... 899 900 /* then do cleanup work */ 901 902 ... 903 904 /* only one thread returns to the library */ 905 } 906 ------------------------------------------------------------------------------- 907 908 909 2.2.14 Testing with a block device 910 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 911 912 Some tests needs a block device (inotify tests, syscall 'EROFS' failures, 913 etc.). LTP library contains a code to prepare a testing device. 914 915 If '.needs_device' flag in the 'struct tst_test' is set the the 'tst_device' 916 structure is initialized with a path to a test device and default filesystem 917 to be used. 918 919 You can also request minimal device size in megabytes by setting 920 '.dev_min_size' the device is guaranteed to have at least the requested size 921 then. 922 923 If '.format_device' flag is set the device is formatted with a filesystem as 924 well. You can use '.dev_fs_type' to override the default filesystem type if 925 needed and pass additional options to mkfs via '.dev_fs_opts' and 926 '.dev_extra_opt' pointers. Note that '.format_device' implies '.needs_device' 927 there is no need to set both. 928 929 If '.mount_device' is set, the device is mounted at '.mntpoint' which is used 930 to pass a directory name that will be created and used as mount destination. 931 You can pass additional flags and data to the mount command via '.mnt_flags' 932 and '.mnt_data' pointers. Note that '.mount_device' implies '.needs_device' 933 and '.format_device' so there is no need to set the later two. 934 935 If '.needs_rofs' is set, read-only filesystem is mounted at '.mntpoint' this 936 one is supposed to be used for 'EROFS' tests. 937 938 If '.all_filesystems' is set the test function is executed for all supported 939 filesystems. Supported filesystems are detected based on existence of the 940 'mkfs.$fs' helper and on kernel support to mount it. For each supported 941 filesystem the 'tst_device.fs_type' is set to the currently tested fs type, if 942 '.format_device' is set the device is formatted as well, if '.mount_device' is 943 set it's mounted at '.mntpoint'. Also the test timeout is reset for each 944 execution of the test fuction. This flag is expected to be used for filesystem 945 related syscalls that are at least partly implemented in the filesystem 946 specific code e.g. fallocate(). 947 948 [source,c] 949 ------------------------------------------------------------------------------- 950 #include "tst_test.h" 951 952 struct tst_device { 953 const char *dev; 954 const char *fs_type; 955 }; 956 957 extern struct tst_device *tst_device; 958 959 int tst_umount(const char *path); 960 ------------------------------------------------------------------------------- 961 962 In case that 'LTP_DEV' is passed to the test in an environment, the library 963 checks that the file exists and that it's a block device, if 964 '.device_min_size' is set the device size is checked as well. If 'LTP_DEV' 965 wasn't set or if size requirements were not met a temporary file is created 966 and attached to a free loop device. 967 968 If there is no usable device and loop device couldn't be initialized the test 969 exits with 'TCONF'. 970 971 The 'tst_umount()' function works exactly as 'umount(2)' but retries several 972 times on 'EBUSY'. This is because various desktop daemons (gvfsd-trash is known 973 for that) may be stupid enough to probe all newly mounted filesystem which 974 results in 'umount(2)' failing with 'EBUSY'. 975 976 IMPORTANT: All testcases should use 'tst_umount()' instead of 'umount(2)' to 977 umount filesystems. 978 979 2.2.15 Formatting a device with a filesystem 980 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 981 982 [source,c] 983 ------------------------------------------------------------------------------- 984 #include "tst_test.h" 985 986 static void setup(void) 987 { 988 ... 989 SAFE_MKFS(tst_device->dev, tst_device->fs_type, NULL, NULL); 990 ... 991 } 992 ------------------------------------------------------------------------------- 993 994 This function takes a path to a device, filesystem type and an array of extra 995 options passed to mkfs. 996 997 The fs options 'fs_opts' should either be 'NULL' if there are none, or a 998 'NULL' terminated array of strings such as: 999 +const char *const opts[] = {"-b", "1024", NULL}+. 1000 1001 The extra option 'extra_opt' should either be 'NULL' if there is none, or a 1002 string such as '"102400"'; 'extra_opt' will be passed after device name. e.g: 1003 +mkfs -t ext4 -b 1024 /dev/sda1 102400+ in this case. 1004 1005 2.2.16 Verifying a filesystem's free space 1006 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1007 1008 Some tests have size requirements for the filesystem's free space. If these 1009 requirements are not satisfied, the tests should be skipped. 1010 1011 [source,c] 1012 ------------------------------------------------------------------------------- 1013 #include "tst_test.h" 1014 1015 int tst_fs_has_free(const char *path, unsigned int size, unsigned int mult); 1016 ------------------------------------------------------------------------------- 1017 1018 The 'tst_fs_has_free()' function returns 1 if there is enough space and 0 if 1019 there is not. 1020 1021 The 'path' is the pathname of any directory/file within a filesystem. 1022 1023 The 'mult' is a multiplier, one of 'TST_BYTES', 'TST_KB', 'TST_MB' or 'TST_GB'. 1024 1025 The required free space is calculated by 'size * mult', e.g. 1026 'tst_fs_has_free("/tmp/testfile", 64, TST_MB)' will return 1 if the 1027 filesystem, which '"/tmp/testfile"' is in, has 64MB free space at least, and 0 1028 if not. 1029 1030 2.2.17 Files, directories and fs limits 1031 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1032 1033 Some tests need to know the maximum count of links to a regular file or 1034 directory, such as 'rename(2)' or 'linkat(2)' to test 'EMLINK' error. 1035 1036 [source,c] 1037 ------------------------------------------------------------------------------- 1038 #include "tst_test.h" 1039 1040 int tst_fs_fill_hardlinks(const char *dir); 1041 ------------------------------------------------------------------------------- 1042 1043 Try to get maximum count of hard links to a regular file inside the 'dir'. 1044 1045 NOTE: This number depends on the filesystem 'dir' is on. 1046 1047 This function uses 'link(2)' to create hard links to a single file until it 1048 gets 'EMLINK' or creates 65535 links. If the limit is hit, the maximum number of 1049 hardlinks is returned and the 'dir' is filled with hardlinks in format 1050 "testfile%i", where i belongs to [0, limit) interval. If no limit is hit or if 1051 'link(2)' failed with 'ENOSPC' or 'EDQUOT', zero is returned and previously 1052 created files are removed. 1053 1054 [source,c] 1055 ------------------------------------------------------------------------------- 1056 #include "tst_test.h" 1057 1058 int tst_fs_fill_subdirs(const char *dir); 1059 ------------------------------------------------------------------------------- 1060 1061 Try to get maximum number of subdirectories in directory. 1062 1063 NOTE: This number depends on the filesystem 'dir' is on. For current kernel, 1064 subdir limit is not available for all filesystems (available for ext2, ext3, 1065 minix, sysv and more). If the test runs on some other filesystems, like ramfs, 1066 tmpfs, it will not even try to reach the limit and return 0. 1067 1068 This function uses 'mkdir(2)' to create directories in 'dir' until it gets 1069 'EMLINK' or creates 65535 directories. If the limit is hit, the maximum number 1070 of subdirectories is returned and the 'dir' is filled with subdirectories in 1071 format "testdir%i", where i belongs to [0, limit - 2) interval (because each 1072 newly created dir has two links already - the '.' and the link from parent 1073 dir). If no limit is hit or if 'mkdir(2)' failed with 'ENOSPC' or 'EDQUOT', 1074 zero is returned and previously created directories are removed. 1075 1076 [source,c] 1077 ------------------------------------------------------------------------------- 1078 #include "tst_test.h" 1079 1080 int tst_dir_is_empty(const char *dir, int verbose); 1081 ------------------------------------------------------------------------------- 1082 1083 Returns non-zero if directory is empty and zero otherwise. 1084 1085 Directory is considered empty if it contains only '.' and '..'. 1086 1087 2.2.18 Getting an unused PID number 1088 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1089 1090 Some tests require a 'PID', which is not used by the OS (does not belong to 1091 any process within it). For example, kill(2) should set errno to 'ESRCH' if 1092 it's passed such 'PID'. 1093 1094 [source,c] 1095 ------------------------------------------------------------------------------- 1096 #include "tst_test.h" 1097 1098 pid_t tst_get_unused_pid(void); 1099 ------------------------------------------------------------------------------- 1100 1101 Return a 'PID' value not used by the OS or any process within it. 1102 1103 [source,c] 1104 ------------------------------------------------------------------------------- 1105 #include "tst_test.h" 1106 1107 int tst_get_free_pids(void); 1108 ------------------------------------------------------------------------------- 1109 1110 Returns number of unused pids in the system. Note that this number may be 1111 different once the call returns and should be used only for rough estimates. 1112 1113 2.2.20 Running executables 1114 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 1115 1116 [source,c] 1117 ------------------------------------------------------------------------------- 1118 #include "tst_test.h" 1119 1120 int tst_run_cmd(const char *const argv[], 1121 const char *stdout_path, 1122 const char *stderr_path, 1123 int pass_exit_val); 1124 ------------------------------------------------------------------------------- 1125 1126 'tst_run_cmd' is a wrapper for 'vfork() + execvp()' which provides a way 1127 to execute an external program. 1128 1129 'argv[]' is a NULL-terminated array of strings starting with the program name 1130 which is followed by optional arguments. 1131 1132 A non-zero 'pass_exit_val' makes 'tst_run_cmd' return the program exit code to 1133 the caller. A zero for 'pass_exit_val' makes 'tst_run_cmd' exit the tests 1134 on failure. 1135 1136 In case that 'execvp()' has failed and the 'pass_exit_val' flag was set, the 1137 return value is '255' if 'execvp()' failed with 'ENOENT' and '254' otherwise. 1138 1139 'stdout_path' and 'stderr_path' determine where to redirect the program 1140 stdout and stderr I/O streams. 1141 1142 .Example 1143 [source,c] 1144 ------------------------------------------------------------------------------- 1145 #include "tst_test.h" 1146 1147 const char *const cmd[] = { "ls", "-l", NULL }; 1148 1149 ... 1150 /* Store output of 'ls -l' into log.txt */ 1151 tst_run_cmd(cmd, "log.txt", NULL, 0); 1152 ... 1153 ------------------------------------------------------------------------------- 1154 1155 2.2.21 Measuring elapsed time and helper functions 1156 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1157 1158 [source,c] 1159 ------------------------------------------------------------------------------- 1160 #include "tst_test.h" 1161 1162 void tst_timer_check(clockid_t clk_id); 1163 1164 void tst_timer_start(clockid_t clk_id); 1165 1166 void tst_timer_stop(void); 1167 1168 struct timespec tst_timer_elapsed(void); 1169 1170 long long tst_timer_elapsed_ms(void); 1171 1172 long long tst_timer_elapsed_us(void); 1173 ------------------------------------------------------------------------------- 1174 1175 The 'tst_timer_check()' function checks if specified 'clk_id' is suppored and 1176 exits the test with 'TCONF' otherwise. It's expected to be used in test 1177 'setup()' before any resources that needs to be cleaned up are initialized, 1178 hence it does not include a cleanup function parameter. 1179 1180 The 'tst_timer_start()' marks start time and stores the 'clk_id' for further 1181 use. 1182 1183 The 'tst_timer_stop()' marks the stop time using the same 'clk_id' as last 1184 call to 'tst_timer_start()'. 1185 1186 The 'tst_timer_elapsed*()' returns time difference between the timer start and 1187 last timer stop in several formats and units. 1188 1189 IMPORTANT: The timer functions use 'clock_gettime()' internally which needs to 1190 be linked with '-lrt' on older glibc. Please do not forget to add 1191 'LDLIBS+=-lrt' in Makefile. 1192 1193 [source,c] 1194 ------------------------------------------------------------------------------- 1195 long long tst_timespec_to_us(struct timespec t); 1196 long long tst_timespec_to_ms(struct timespec t); 1197 1198 struct timeval tst_us_to_timeval(long long us); 1199 struct timeval tst_ms_to_timeval(long long ms); 1200 1201 int tst_timespec_lt(struct timespec t1, struct timespec t2); 1202 1203 struct timespec tst_timespec_add_us(struct timespec t, long long us); 1204 1205 struct timespec tst_timespec_diff(struct timespec t1, struct timespec t2); 1206 long long tst_timespec_diff_us(struct timespec t1, struct timespec t2); 1207 long long tst_timespec_diff_ms(struct timespec t1, struct timespec t2); 1208 1209 struct timespec tst_timespec_abs_diff(struct timespec t1, struct timespec t2); 1210 long long tst_timespec_abs_diff_us(struct timespec t1, struct timespec t2); 1211 long long tst_timespec_abs_diff_ms(struct timespec t1, struct timespec t2); 1212 ------------------------------------------------------------------------------- 1213 1214 The first four functions are simple inline conversion functions. 1215 1216 The 'tst_timespec_lt()' function returns non-zero if 't1' is earlier than 1217 't2'. 1218 1219 The 'tst_timespec_add_us()' function adds 'us' microseconds to the timespec 1220 't'. The 'us' is expected to be positive. 1221 1222 The 'tst_timespec_diff*()' functions returns difference between two times, the 1223 't1' is expected to be later than 't2'. 1224 1225 The 'tst_timespec_abs_diff*()' functions returns absolute value of difference 1226 between two times. 1227 1228 NOTE: All conversions to ms and us rounds the value. 1229 1230 2.2.22 Datafiles 1231 ^^^^^^^^^^^^^^^^ 1232 1233 [source,c] 1234 ------------------------------------------------------------------------------- 1235 #include "tst_test.h" 1236 1237 static const char *const res_files[] = { 1238 "foo", 1239 "bar", 1240 NULL 1241 }; 1242 1243 static struct tst_test test = { 1244 ... 1245 .resource_files = res_files, 1246 ... 1247 } 1248 ------------------------------------------------------------------------------- 1249 1250 If the test needs additional files to be copied to the test temporary 1251 directory all you need to do is to list their filenames in the 1252 'NULL'-terminated array '.resource_files' in the tst_test structure. 1253 1254 When resource files is set test temporary directory is created automatically, 1255 there is need to set '.needs_tmpdir' as well. 1256 1257 The test library looks for datafiles first, these are either stored in a 1258 directory called +datafiles+ in the +$PWD+ at the start of the test or in 1259 +$LTPROOT/testcases/data/${test_binary_name}+. If the file is not found the 1260 library looks into +$LTPROOT/testcases/bin/+ and to +$PWD+ at the start of the 1261 test. This ensures that the testcases can copy the file(s) effortlessly both 1262 when test is started from the directory it was compiled in as well as when LTP 1263 was installed. 1264 1265 The file(s) are copied to the newly created test temporary directory which is 1266 set as the test working directory when the 'test()' functions is executed. 1267 1268 2.2.23 Code path tracing 1269 ^^^^^^^^^^^^^^^^^^^^^^^^ 1270 1271 'tst_res' is a macro, so on when you define a function in one file: 1272 1273 [source,c] 1274 ------------------------------------------------------------------------------- 1275 int do_action(int arg) 1276 { 1277 ... 1278 1279 if (ok) { 1280 tst_res(TPASS, "check passed"); 1281 return 0; 1282 } else { 1283 tst_res(TFAIL, "check failed"); 1284 return -1; 1285 } 1286 } 1287 ------------------------------------------------------------------------------- 1288 1289 and call it from another file, the file and line reported by 'tst_res' in this 1290 function will be from the former file. 1291 1292 'TST_TRACE' can make the analysis of such situations easier. It's a macro which 1293 inserts a call to 'tst_res(TINFO, ...)' in case its argument evaluates to 1294 non-zero. In this call to 'tst_res(TINFO, ...)' the file and line will be 1295 expanded using the actual location of 'TST_TRACE'. 1296 1297 For example, if this another file contains: 1298 1299 [source,c] 1300 ------------------------------------------------------------------------------- 1301 #include "tst_test.h" 1302 1303 if (TST_TRACE(do_action(arg))) { 1304 ... 1305 } 1306 ------------------------------------------------------------------------------- 1307 1308 the generated output may look similar to: 1309 1310 ------------------------------------------------------------------------------- 1311 common.h:9: FAIL: check failed 1312 test.c:8: INFO: do_action(arg) failed 1313 ------------------------------------------------------------------------------- 1314 1315 2.3 Writing a testcase in shell 1316 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1317 1318 LTP supports testcases to be written in a portable shell too. 1319 1320 There is a shell library modeled closely to the C interface at 1321 'testcases/lib/tst_test.sh'. 1322 1323 WARNING: All identifiers starting with TST_ or tst_ are reserved for the 1324 test library. 1325 1326 2.3.1 Basic test interface 1327 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 1328 1329 [source,sh] 1330 ------------------------------------------------------------------------------- 1331 #!/bin/sh 1332 # 1333 # This is a basic test for true shell buildin 1334 # 1335 1336 TST_TESTFUNC=do_test 1337 . tst_test.sh 1338 1339 do_test() 1340 { 1341 true 1342 ret=$? 1343 1344 if [ $ret -eq 0 ]; then 1345 tst_res TPASS "true returned 0" 1346 else 1347 tst_res TFAIL "true returned $ret" 1348 fi 1349 } 1350 1351 tst_run 1352 ------------------------------------------------------------------------------- 1353 1354 TIP: To execute this test the 'tst_test.sh' library must be in '$PATH'. If you 1355 are executing the test from a git checkout you can run it as 1356 'PATH="$PATH:../../lib" ./foo01.sh' 1357 1358 The shell library expects test setup, cleanup and the test function executing 1359 the test in the '$TST_SETUP', '$TST_CLEANUP' and '$TST_TESTFUNC' variables. 1360 1361 Both '$TST_SETUP' and '$TST_CLEANUP' are optional. 1362 1363 The '$TST_TESTFUNC' may be called several times if more than one test 1364 iteration was requested by passing right command line options to the test. 1365 1366 The '$TST_CLEANUP' may be called even in the middle of the setup and must be 1367 able to clean up correctly even in this situation. The easiest solution for 1368 this is to keep track of what was initialized and act accordingly in the 1369 cleanup. 1370 1371 Notice also the 'tst_run' function called at the end of the test that actually 1372 starts the test. 1373 1374 [source,sh] 1375 ------------------------------------------------------------------------------- 1376 #!/bin/sh 1377 # 1378 # Example test with tests in separate functions 1379 # 1380 1381 TST_TESTFUNC=test 1382 TST_CNT=2 1383 . tst_test.sh 1384 1385 test1() 1386 { 1387 tst_res TPASS "Test 1 passed" 1388 } 1389 1390 test2() 1391 { 1392 tst_res TPASS "Test 2 passed" 1393 } 1394 1395 tst_run 1396 ------------------------------------------------------------------------------- 1397 1398 If '$TST_CNT' is set, the test library looks if there are functions named 1399 '$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are 1400 found they are executed one by one. 1401 1402 [source,sh] 1403 ------------------------------------------------------------------------------- 1404 #!/bin/sh 1405 # 1406 # Example test with tests in a single function 1407 # 1408 1409 TST_TESTFUNC=do_test 1410 TST_CNT=2 1411 . tst_test.sh 1412 1413 do_test() 1414 { 1415 case $1 in 1416 1) tst_res TPASS "Test 1 passed";; 1417 2) tst_res TPASS "Test 2 passed";; 1418 esac 1419 } 1420 1421 tst_run 1422 ------------------------------------------------------------------------------- 1423 1424 Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., 1425 the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed 1426 to it in the '$1'. 1427 1428 2.3.2 Library variables 1429 ^^^^^^^^^^^^^^^^^^^^^^^ 1430 1431 Similarily to the C library various checks and preparations can be requested 1432 simply by setting right '$TST_NEEDS_FOO'. 1433 1434 [options="header"] 1435 |============================================================================= 1436 | Variable name | Action done 1437 | 'TST_NEEDS_ROOT' | Exit the test with 'TCONF' unless executed under root 1438 | 'TST_NEEDS_TMPDIR' | Create test temporary directory and cd into it. 1439 | 'TST_NEEDS_DEVICE' | Prepare test temporary device, the path to testing 1440 device is stored in '$TST_DEVICE' variable. 1441 | 'TST_NEEDS_CMDS' | String with command names that has to be present for 1442 the test (see below). 1443 | 'TST_NEEDS_MODULE' | Test module name needed for the test (see below). 1444 |============================================================================= 1445 1446 Checking for presence of commands 1447 +++++++++++++++++++++++++++++++++ 1448 1449 [source,sh] 1450 ------------------------------------------------------------------------------- 1451 #!/bin/sh 1452 1453 ... 1454 1455 TST_NEEDS_CMDS="modinfo modprobe" 1456 . tst_test.sh 1457 1458 ... 1459 1460 ------------------------------------------------------------------------------- 1461 1462 Setting '$TST_NEEDS_CMDS' to a string listing required commands will check for 1463 existence each of them and exits the test with 'TCONF' on first misssing. 1464 1465 Alternatively the 'tst_check_cmds()' function can be used to do the same on 1466 runtime, since sometimes we need to the check at runtime too. 1467 1468 Locating kernel modules 1469 +++++++++++++++++++++++ 1470 1471 The LTP build system can build kernel modules as well, setting 1472 '$TST_NEEDS_MODULE' to module name will cause to library to look for the 1473 module in a few possible paths. 1474 1475 If module was found the path to it will be stored into '$TST_MODPATH' 1476 variable, if module wasn't found the test will exit with 'TCONF'. 1477 1478 2.3.3 Optional command line parameters 1479 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1480 1481 [source,sh] 1482 ------------------------------------------------------------------------------- 1483 #!/bin/sh 1484 # 1485 # Optional test command line parameters 1486 # 1487 1488 TST_OPTS="af:" 1489 TST_USAGE=usage 1490 TST_PARSE_ARGS=parse_args 1491 TST_TESTFUNC=do_test 1492 1493 . tst_test.sh 1494 1495 ALTERNATIVE=0 1496 MODE="foo" 1497 1498 usage() 1499 { 1500 cat << EOF 1501 usage: $0 [-a] [-f <foo|bar>] 1502 1503 OPTIONS 1504 -a Enable support for alternative foo 1505 -f Specify foo or bar mode 1506 EOF 1507 } 1508 1509 parse_args() 1510 { 1511 case $1 in 1512 a) ALTERNATIVE=1 1513 f) MODE="$2" 1514 esac 1515 } 1516 1517 do_test() 1518 { 1519 ... 1520 } 1521 1522 tst_run 1523 ------------------------------------------------------------------------------- 1524 1525 The 'getopts' string for optional parameters is passed in the '$TST_OPTS' 1526 variable. There are a few default parameters that cannot be used by a test, 1527 these can be listed with passing help '-h' option to any test. 1528 1529 The function that prints the usage is passed in '$TST_USAGE', the help for 1530 the options implemented in the library is appended when usage is printed. 1531 1532 Lastly the fucntion '$PARSE_ARGS' is called with the option name in '$1' and, 1533 if option has argument, its value in '$2'. 1534 1535 [source,sh] 1536 ------------------------------------------------------------------------------- 1537 #!/bin/sh 1538 # 1539 # Optional test positional paramters 1540 # 1541 1542 TST_POS_ARGS=3 1543 TST_USAGE=usage 1544 TST_TESTFUNC=do_test 1545 1546 . tst_test.sh 1547 1548 usage() 1549 { 1550 cat << EOF 1551 usage: $0 [min] [max] [size] 1552 1553 EOF 1554 } 1555 1556 min="$1" 1557 max="$2" 1558 size="$3" 1559 1560 do_test() 1561 { 1562 ... 1563 } 1564 1565 tst_run 1566 ------------------------------------------------------------------------------- 1567 1568 You can also request a number of positional parameters by setting the 1569 '$TST_POS_ARGS' variable. If you do, these will be available as they were 1570 passed directly to the script in '$1', '$2', ..., '$n'. 1571 1572 2.3.4 Usefull library functions 1573 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1574 1575 Sleeping for subsecond intervals 1576 ++++++++++++++++++++++++++++++++ 1577 1578 Albeit there is a sleep command available basically everywhere not all 1579 implementations can support sleeping for less than one second. And most of the 1580 time sleeping for a second is too much. Therefore LTP includes 'tst_sleep' 1581 that can sleep for defined amount of seconds, milliseconds or microseconds. 1582 1583 [source,sh] 1584 ------------------------------------------------------------------------------- 1585 # sleep for 100 milliseconds 1586 tst_sleep 100ms 1587 ------------------------------------------------------------------------------- 1588 1589 Checking for integers 1590 +++++++++++++++++++++ 1591 1592 [source,sh] 1593 ------------------------------------------------------------------------------- 1594 # returns zero if passed an integer parameter, non-zero otherwise 1595 tst_is_int "$FOO" 1596 ------------------------------------------------------------------------------- 1597 1598 Obtaining random numbers 1599 ++++++++++++++++++++++++ 1600 1601 There is no '$RANDOM' in portable shell, use 'tst_random' instead. 1602 1603 [source,sh] 1604 ------------------------------------------------------------------------------- 1605 # get random integer between 0 and 1000 (including 0 and 1000) 1606 tst_random 0 1000 1607 ------------------------------------------------------------------------------- 1608 1609 Formatting device with a filesystem 1610 +++++++++++++++++++++++++++++++++++ 1611 1612 The 'tst_mkfs' helper will format device with the filesystem. 1613 1614 [source,sh] 1615 ------------------------------------------------------------------------------- 1616 # format test device with ext2 1617 tst_mkfs ext2 $TST_DEVICE 1618 ------------------------------------------------------------------------------- 1619 1620 Umounting filesystems 1621 +++++++++++++++++++++ 1622 1623 The 'tst_umount' helper is a safe way to umount a filesystem. 1624 1625 If the path passed to the function is not mounted (present in '/proc/mounts') 1626 it's noop. 1627 1628 Otherwise it retries to umount the filesystem a few times on a failure, which 1629 is a workaround since there are a daemons dumb enough to probe all newly 1630 mounted filesystems, which prevents them from umounting shortly after they 1631 were mounted. 1632 1633 Running commands as different user with 'su' 1634 ++++++++++++++++++++++++++++++++++++++++++++ 1635 1636 While some distributions retain paths added to +$PATH+ when doing 1637 +su user -c "command"+ this does not work at least in Debian. If you want to 1638 run LTP binaries as a different user you must use 'tst_su' instead which sets 1639 up +$PATH+ and the runs the command. 1640 1641 .Run test child binary as a test user 1642 [source,sh] 1643 ------------------------------------------------------------------------------- 1644 #!/bin/sh 1645 TCID=foo01 1646 . test.sh 1647 1648 tst_su testusr foo01_child 1649 if [ $? -ne 0 ]; then 1650 tst_resm TFAIL "foo failed" 1651 else 1652 tst_resm TPASS "foo passed" 1653 fi 1654 ------------------------------------------------------------------------------- 1655 1656 ROD and ROD_SILENT 1657 ++++++++++++++++++ 1658 1659 These functions supply the 'SAFE_MACROS' used in C although they work and are 1660 named differently. 1661 1662 [source,sh] 1663 ------------------------------------------------------------------------------- 1664 ROD_SILENT command arg1 arg2 ... 1665 1666 # is shorthand for: 1667 1668 command arg1 arg2 ... > /dev/null 2>&1 1669 if [ $? -ne 0 ]; then 1670 tst_brkm TBROK "..." 1671 fi 1672 1673 1674 ROD command arg1 arg2 ... 1675 1676 # is shorthand for: 1677 1678 ROD arg1 arg2 ... 1679 if [ $? -ne 0 ]; then 1680 tst_brkm TBROK "..." 1681 fi 1682 ------------------------------------------------------------------------------- 1683 1684 WARNING: Keep in mind that output redirection (to a file) happens in the 1685 caller rather than in the ROD function and cannot be checked for 1686 write errors by the ROD function. 1687 1688 As a matter of a fact doing +ROD echo a > /proc/cpuinfo+ would work just fine 1689 since the 'ROD' function will only get the +echo a+ part that will run just 1690 fine. 1691 1692 [source,sh] 1693 ------------------------------------------------------------------------------- 1694 # Redirect output to a file with ROD 1695 ROD echo foo \> bar 1696 ------------------------------------------------------------------------------- 1697 1698 Note the '>' is escaped with '\', this causes that the '>' and filename are 1699 passed to the 'ROD' function as parameters and the 'ROD' function contains 1700 code to split '$@' on '>' and redirects the output to the file. 1701 1702 EXPECT_PASS and EXPECT_FAIL 1703 +++++++++++++++++++++++++++ 1704 1705 [source,sh] 1706 ------------------------------------------------------------------------------- 1707 EXPECT_PASS command arg1 arg2 ... [ \> file ] 1708 EXPECT_FAIL command arg1 arg2 ... [ \> file ] 1709 ------------------------------------------------------------------------------- 1710 1711 'EXPECT_PASS' calls 'tst_resm TPASS' if the command exited with 0 exit code, 1712 and 'tst_resm TFAIL' otherwise. 'EXPECT_FAIL' does vice versa. 1713 1714 Output redirection rules are the same as for the 'ROD' function. In addition 1715 to that, 'EXPECT_FAIL' always redirects the command's stderr to '/dev/null'. 1716 1717 tst_kvcmp 1718 +++++++++ 1719 1720 This command compares the currently running kernel version given conditions 1721 with syntax similar to the shell test command. 1722 1723 [source,sh] 1724 ------------------------------------------------------------------------------- 1725 # Exit the test if kernel version is older or equal to 2.6.8 1726 if tst_kvcmp -le 2.6.8; then 1727 tst_brk TCONF "Kernel newer than 2.6.8 is needed" 1728 fi 1729 1730 # Exit the test if kernel is newer than 3.8 and older than 4.0.1 1731 if tst_kvcmp -gt 3.8 -a -lt 4.0.1; then 1732 tst_brk TCONF "Kernel must be older than 3.8 or newer than 4.0.1" 1733 fi 1734 ------------------------------------------------------------------------------- 1735 1736 [options="header"] 1737 |======================================================================= 1738 | expression | description 1739 | -eq kver | Returns true if kernel version is equal 1740 | -ne kver | Returns true if kernel version is not equal 1741 | -gt kver | Returns true if kernel version is greater 1742 | -ge kver | Returns true if kernel version is greater or equal 1743 | -lt kver | Returns true if kernel version is lesser 1744 | -le kver | Returns true if kernel version is lesser or equal 1745 | -a | Does logical and between two expressions 1746 | -o | Does logical or between two expressions 1747 |======================================================================= 1748 1749 The format for kernel version has to either be with one dot e.g. '2.6' or with 1750 two dots e.g. '4.8.1'. 1751 1752 .tst_fs_has_free 1753 [source,sh] 1754 ------------------------------------------------------------------------------- 1755 #!/bin/sh 1756 1757 ... 1758 1759 # whether current directory has 100MB free space at least. 1760 if ! tst_fs_has_free . 100MB; then 1761 tst_brkm TCONF "Not enough free space" 1762 fi 1763 1764 ... 1765 ------------------------------------------------------------------------------- 1766 1767 The 'tst_fs_has_free' shell interface returns 0 if the specified free space is 1768 satisfied, 1 if not, and 2 on error. 1769 1770 The second argument supports suffixes kB, MB and GB, the default unit is Byte. 1771 1772 .tst_retry 1773 [source,sh] 1774 ------------------------------------------------------------------------------- 1775 #!/bin/sh 1776 1777 ... 1778 1779 # Retry ping command three times 1780 tst_retry "ping -c 1 127.0.0.1" 1781 1782 if [ $? -ne 0 ]; then 1783 tst_resm TFAIL "Failed to ping 127.0.0.1" 1784 else 1785 tst_resm TPASS "Successfully pinged 127.0.0.1" 1786 fi 1787 1788 ... 1789 ------------------------------------------------------------------------------- 1790 1791 The 'tst_retry' function allows you to retry a command after waiting small 1792 amount of time until it succeeds or until given amount of retries has been 1793 reached (default is three attempts). 1794 1795 2.3.5 Restarting daemons 1796 ^^^^^^^^^^^^^^^^^^^^^^^^ 1797 1798 Restarting system daemons is a complicated task for two reasons. 1799 1800 * There are different init systems 1801 (SysV init, systemd, etc...) 1802 1803 * Daemon names are not unified between distributions 1804 (apache vs httpd, cron vs crond, various syslog variations) 1805 1806 To solve these problems LTP has 'testcases/lib/daemonlib.sh' library that 1807 provides functions to start/stop/query daemons as well as variables that store 1808 correct daemon name. 1809 1810 .Supported operations 1811 |============================================================================== 1812 | start_daemon() | Starts daemon, name is passed as first parameter. 1813 | stop_daemon() | Stops daemon, name is passed as first parameter. 1814 | restart_daemon() | Restarts daemon, name is passed as first parameter. 1815 | status_daemon() | Detect daemon status (exit code: 0: running, 1: not running). 1816 |============================================================================== 1817 1818 .Variables with detected names 1819 |============================================================================== 1820 | CROND_DAEMON | Cron daemon name (cron, crond). 1821 | SYSLOG_DAEMON | Syslog daemon name (syslog, syslog-ng, rsyslog). 1822 |============================================================================== 1823 1824 .Cron daemon restart example 1825 [source,sh] 1826 ------------------------------------------------------------------------------- 1827 #!/bin/sh 1828 # 1829 # Cron daemon restart example 1830 # 1831 TCID=cron01 1832 TST_COUNT=1 1833 . test.sh 1834 . daemonlib.sh 1835 1836 ... 1837 1838 restart_daemon $CROND_DAEMON 1839 1840 ... 1841 1842 tst_exit 1843 ------------------------------------------------------------------------------- 1844 1845 2.3.6 Access to the checkpoint interface 1846 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1847 1848 The shell library provides an implementation of the checkpoint interface 1849 compatible with the C version. All TST_CHECKPOINT_* functions are available. 1850 1851 In order to initialize checkpoints '$TST_NEEDS_CHECKPOINTS' must be set to '1' 1852 before the inclusion of 'test.sh': 1853 1854 [source,sh] 1855 ------------------------------------------------------------------------------- 1856 #!/bin/sh 1857 1858 TST_NEEDS_CHECKPOINTS=1 1859 . test.sh 1860 ------------------------------------------------------------------------------- 1861 1862 Since both the implementations are compatible, it's also possible to start 1863 a child binary process from a shell test and synchronize with it. This process 1864 must have checkpoints initialized by calling tst_reinit()'. 1865 1866 3. Common problems 1867 ------------------ 1868 1869 This chapter describes common problems/misuses and less obvious design patters 1870 (quirks) in UNIX interfaces. Read it carefully :) 1871 1872 3.1 umask() 1873 ~~~~~~~~~~~ 1874 1875 I've been hit by this one several times already... When you create files 1876 with 'open()' or 'creat()' etc, the mode specified as the last parameter *is 1877 not* the mode the file is created with. The mode depends on current 'umask()' 1878 settings which may clear some of the bits. If your test depends on specific 1879 file permissions you need either to change umask to 0 or 'chmod()' the file 1880 afterwards or use SAFE_TOUCH() that does the 'chmod()' for you. 1881 1882 3.2 access() 1883 ~~~~~~~~~~~ 1884 1885 If 'access(some_file, W_OK)' is executed by root, it will return success even 1886 if the file doesn't have write permission bits set (the same holds for R_OK 1887 too). For sysfs files you can use 'open()' as a workaround to check file 1888 read/write permissions. It might not work for other filesystems, for these you 1889 have to use 'stat()', 'lstat()' or 'fstat()'. 1890 1891 3.3 umount() EBUSY 1892 ~~~~~~~~~~~~~~~~~~ 1893 1894 Various desktop daemons (gvfsd-trash is known for that) may be stupid enough 1895 to probe all newly mounted filesystem which results in 'umount(2)' failing 1896 with 'EBUSY'; use 'tst_umount()' described in 2.2.19 that retries in this case 1897 instead of plain 'umount(2)'. 1898 1899 3.4 FILE buffers and fork() 1900 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1901 1902 Be vary that if a process calls 'fork(2)' the child process inherits open 1903 descriptors as well as copy of the parent memory so especially if there are 1904 any open 'FILE' buffers with a data in them they may be written both by the 1905 parent and children resulting in corrupted/duplicated data in the resulting 1906 files. 1907 1908 Also open 'FILE' streams are flushed and closed at 'exit(3)' so if your 1909 program works with 'FILE' streams, does 'fork(2)', and the child may end up 1910 calling 'exit(3)' you will likely end up with corrupted files. 1911 1912 The solution to this problem is either simply call 'fflush(NULL)' that flushes 1913 all open output 'FILE' streams just before doing 'fork(2)'. You may also use 1914 '_exit(2)' in child processes which does not flush 'FILE' buffers and also 1915 skips 'atexit(3)' callbacks. 1916 1917 4. Test Contribution Checklist 1918 ------------------------------ 1919 1920 1. Test compiles and runs fine (check with -i 10 too) 1921 2. Checkpatch does not report any errors 1922 3. The runtest entires are in place 1923 4. Test files are added into corresponding .gitignore files 1924 5. Patches apply over the latest git 1925 1926 1927 4.1 About .gitignore files 1928 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1929 1930 There are numerous '.gitignore' files in the LTP tree. Usually there is a 1931 '.gitignore' file per a group of tests. The reason for this setup is simple. 1932 It's easier to maintain a '.gitignore' file per directory with tests, rather 1933 than having single file in the project root directory. This way, we don't have 1934 to update all the gitignore files when moving directories, and they get deleted 1935 automatically when a directory with tests is removed. 1936