1 VTS LTP Workflow 2 ================ 3 4 A Guide to Developing LTP for VTS/Android 5 6 What is LTP? 7 ------------ 8 9 [Github](https://github.com/linux-test-project/ltp) (upstream) 10 11 LTP (Linux Test Project) is a suite of tests that covers both kernel interfaces 12 and userspace functionality (glibc, commonly used binaries, etc). For the 13 purposes of Android the userspace functionality testing is of less importance 14 and in fact much of it must be disabled, given the functionality is not 15 available in Android. 16 17 As of mid-2018 there are on the order of 900 tests executed in VTS. Most tests 18 are run in both 32-bit and 64-bit mode. Many more are available but currently 19 disabled due to either being broken or not applicable on Android. 20 21 How is LTP Run in VTS? 22 ---------------------- 23 24 The LTP source is located at external/ltp in the Android tree. This is not an 25 exact mirror of upstream, there are outstanding changes to LTP for it to work 26 on Android which have not yet been pushed upstream. In addition to the LTP 27 source there is also the VTS wrapper around it. This is located at 28 test/vts-testcase/kernel/ltp. Some noteworthy directories/files: 29 30 * `external/ltp/android/`: Contains Android-specific files, aside from Android.[bp, mk] at top level. 31 * `external/ltp/android/Android.ltp.mk`: Lists build rules for the LTP modules built under make. This file gets auto-generated by android/tools/gen_android_build.sh. 32 * `external/ltp/gen.bp`: Lists build rules for the LTP modules built under Soong. This file gets auto-generated by android/tools/gen_android_build.sh. 33 * `external/ltp/android/ltp_package_list.mk`: Lists all tests that will get pulled into VTS - VTS depends on this list. This file gets auto-generated by android/tools/gen_android_build.sh. 34 * `external/ltp/android/tools/disabled_tests.txt`: Lists tests which cannot or should not be compiled for Android. This file is read by gen_android_build.sh during LTP upgrades to produce *.mk files required to build LTP for Android. 35 * `external/ltp/testcases`: Source for LTP tests. Among the most important for the purposes of Treble are those in external/ltp/testcases/kernel/syscalls. 36 * `test/vts-testcase/kernel/ltp/configs/disabled_tests.py`: Any test listed here will not be run in VTS, either in the staging or stable set, despite being compiled. If a test does not appear here then it at least runs as part of the VtsKernelLtpStaging module. 37 * `test/vts-testcase/kernel/ltp/configs/stable_tests.py`: Any test listed here will run as part of the VtsKernelLtp (stable) module. 38 39 New tests are run in the staging set for some time (a couple weeks or more) to 40 ensure they are reliable prior to their being added to the stable set. 41 42 To run VTS LTP it must first be built. VTS is not device specific, you need not 43 compile it specifically for the device you wish to run it on, assuming it is 44 the same architecture. 45 46 * `. build/envsetup.sh` 47 * `lunch` 48 * `make -j vts` 49 50 Before running VTS ensure your host has the required prerequisites (this need 51 only be done once): 52 * `. test/vts/script/setup.sh` 53 54 Then open vts-tradefed and run the VTS stable set: 55 * `vts-tradefed` 56 * `vts-tf > run vts-kernel -m VtsKernelLtp` 57 58 If you regularly work with multiple devices it may be useful to specify the 59 specific device you wish to run VTS on via the serial number: 60 * `vts-tf > run vts-kernel -m VtsKernelLtp -s 000123456789` 61 62 Or we can run the staging set: 63 * `vts-tf > run vts-kernel -m VtsKernelLtpStaging` 64 65 Or a specific test within the stable set: 66 * `vts-tf > run vts-kernel -m VtsKernelLtp -t dio.dio13` 67 68 Maybe several: 69 * `vts-tf > run vts-kernel -m VtsKernelLtp -t dio.dio13,fs.stream04` 70 71 You may choose whether to specify bitness of the tests. This would run 4 tests - dio.dio13 72 both 32-bit and 64-bit, fs.stream04 64-bit, and fs.lftest01 32-bit: 73 * `vts-tf > run vts-kernel -m VtsKernelLtp -t dio.dio13,fs.stream04_64bit,fs.lftest01_32bit` 74 75 Running LTP Faster 76 ------------------ 77 78 Running LTP tests within VTS can be quite cumbersome, especially if you are 79 iterating a lot trying to debug something. Build and run LTP tests faster by 80 doing 81 82 * `external/ltp$ mma` 83 * `external/ltp$ adb sync data` 84 85 The test cases will be located at /data/nativetest{64,}/ltp/testcases/bin. 86 87 Sometimes you need to perform this step after syncing: 88 * `external/ltp$ make distclean` 89 Otherwise, build will fail. 90 91 In order to simulate the exact environment that VTS will be creating for each 92 of these tests, you can set the following environment variables before you run 93 the test. This is very useful if the test itself depends on some of these 94 variables to be set appropriately. 95 96 * `adb root && adb shell` 97 98 In the root shell on device: 99 * `mkdir -p /data/local/tmp/ltp/tmp/ltptemp` 100 * `mkdir -p /data/local/tmp/ltp/tmp/tmpbase` 101 * `mkdir -p /data/local/tmp/ltp/tmp/tmpdir` 102 * `restorecon -F -R /data/local/tmp/ltp` 103 * `export TMP=/data/local/tmp/ltp/tmp` 104 * `export LTPTMP=/data/local/tmp/ltp/tmp/ltptemp` 105 * `export LTP_DEV_FS_TYPE=ext4` 106 * `export TMPBASE=/data/local/tmp/ltp/tmp/tmpbase` 107 * `export TMPDIR=/data/local/tmp/ltp/tmp/tmpdir` 108 * `export LTPROOT=/data/local/tmp/ltp` 109 110 For running 64-bit tests: 111 * `export PATH=/data/nativetest64/ltp/testcases/bin:$PATH` 112 113 Or For running 32-bit tests: 114 * `export PATH=/data/nativetest/ltp/testcases/bin:$PATH` 115 116 How do I enable or disable tests from the LTP build? 117 ---------------------------------------------------- 118 119 Tests are disabled from the LTP build by adding them to 120 external/ltp/android/tools/disabled_tests.txt. Many tests have been added to 121 this file over time. Some of them are not applicable to Android and therefore 122 should not be built. Others were disabled here because they were failing at one 123 point in time for reasons unknown. 124 125 To make a change to what is built in LTP, add or remove an entry in this file, 126 and then update the Android-specific build files for LTP, mentioned above: 127 128 * `external/ltp/android/Android.ltp.mk`, for LTP modules built in make 129 * `external/ltp/gen.bp`, for LTP modules built in soong 130 * `external/ltp/android/ltp_package_list.mk`, which lists all LTP modules that get pulled into VTS 131 132 You can update these files manually, or you can run a script which regenerates 133 these files from scratch, external/ltp/android/tools/gen_android_build.sh. 134 135 How do I enable or disable tests from VTS LTP? 136 ---------------------------------------------- 137 138 In addition to whether modules are built in external/ltp, it is also necessary 139 to configure the VTS harness for LTP to determine which tests are in the stable 140 set, the staging set, or disabled. Note that being disabled in VTS does not 141 affect whether the test is built, but rather determines whether it is run at 142 all as part of the stable or staging sets. 143 144 The file test/vts-testcase/kernel/ltp/configs/stable_tests.py lists tests that 145 will run as part of VTS stable (VtsKernelLtp). The 146 test/vts-testcase/kernel/ltp/configs/disabled_tests.py file lists tests that 147 are runtime disabled in VTS. If a test is not listed in either of these files, 148 but is enabled in the build in external/ltp, then it is in the staging set 149 (VtsKernelLtpStaging). 150 151 When a test is enabled for the first time in VTS it should be in the staging 152 set. The behavior of the test will be observed over a period of time and ensure 153 the test is reliable. After a period of time (a week or two) it will be moved 154 to the stable set. 155 156 Tests that will never be relevant to Android should be disabled from the build 157 in external/ltp. Tests that are (hopefully) temporarily broken should be 158 runtime disabled in VTS. The staging and stable sets should normally all be 159 passing. If something is failing there it should either be fixed with priority 160 or disabled until it can be fixed. 161 162 How do I see or get notified about recent VTS LTP results? 163 ---------------------------------------------------------- 164 165 The internal portal at go/vts-web shows results for the continuous VTS testing 166 done on internal devices. By adding tests such as VtsKernelLtp or 167 VtsKernelLtpStaging to your favorites you will get email notifications whenever 168 failures are seen. Be warned, the testing iterates continuously and the 169 notification emails can generate a lot of traffic. 170 171 Test results are also gathered by Linaro and may be seen 172 [here](https://qa-reports.linaro.org/android-lkft/). 173 174 Help! The external/ltp build is failing! 175 ---------------------------------------- 176 177 Try doing a make distclean inside of external/ltp. If an upgrade to LTP has 178 recently merged or the build files were recently updated, stale files in 179 external/ltp can cause build failures. 180 181 What outstanding issues exist? 182 ------------------------------ 183 184 The hotlist for LTP bugs is [ltp-todo](https://buganizer.corp.google.com/hotlists/733268). 185 186 When you begin working on an LTP bug please assign the bug to yourself so that 187 others know it is being worked on. 188 189 Testing x86_64 190 -------------- 191 192 It is not advisable to run LTP tests directly on your host unless you are fully 193 aware of what the tests will do and are okay with it. These tests may 194 destabilize your box or cause data loss. If you need to run tests on an x86 195 platform and are unsure if they are safe you should run them in emulation, in a 196 virtualized environment, or on a dedicated development x86 platform. 197 198 Sending Fixes Upstream 199 ---------------------- 200 201 The mailing list for LTP is located 202 [here](https://lists.linux.it/listinfo/ltp). Some standard kernel guidelines 203 apply to sending patches; they should be checkpatch (scripts/checkpatch.pl in 204 the kernel repository) clean and sent in plain text in canonical patch format. 205 One easy way to do this is by using git format-patch and git send-email. 206 207 There is an #LTP channel on freenode. The maintainer Cyril Hrubis is there (his 208 nick is metan). 209 210 Merging Fixes 211 ------------------------ 212 213 When possible please merge fixes upstream first. Then cherrypick the change 214 onto aosp/master in external/ltp. 215 216
