1 VTS Kselftest Workflow
2 ======================
3
4 A Guide to Developing Kselftest for VTS/Android
5
6 What is Kselftest?
7 ------------------
8
9 Kselftest is a collection of tests included within the Linux kernel repository
10 at tools/testing/selftests. The tests cover a broad range of areas but total
11 coverage (as of 4.14 anyway) is very spotty. Many of the tests do not apply to
12 Android platforms.
13
14 As of mid 2018 there are 23 tests in kselftest which run on ARM in VTS.
15 Almost all of them get executed in both 32-bit and 64-bit mode. There are many
16 tests which are not enabled because they have dependencies which are not met on
17 Android platforms.
18
19 How is Kselftest Run in VTS?
20 ----------------------------
21
22 The Kselftest source is located at external/linux-kselftest in the Android
23 tree. It is a set of files (tools/testing/selftests/) copied into its own
24 repository from the Linux kernel repository. This is not an exact mirror of
25 upstream; there are outstanding changes to kselftest for it to work on Android
26 which have not yet been pushed upstream (contained in android/patches). In
27 addition to the kselftest source there is also the VTS wrapper around it. This
28 is located at test/vts-testcase/kernel/linux_kselftest. Some noteworthy
29 directories/files:
30
31 * `external/linux-kselftest/Android.bp`: Lists build rules for the kselftest modules built in Soong.
32 * `external/linux-kselftest/android/`: Contains Android-specific files, aside from Android.[bp, mk] at top level.
33 * `external/linux-kselftest/android/Android.kselftest.mk`: Lists build rules for the kselftest modules built under make.
34 * `external/linux-kselftest/android/kselftest_test_list.mk`: Lists all tests that will get pulled into VTS - VTS depends on this list.
35 * `external/linux-kselftest/android/README`: Lists details on why particular tests are not enabled in Android.
36 * `external/linux-kselftest/tools/testing/selftests`: Kselftest sources.
37 * `test/vts-testcase/kernel/linux_kselftest/kselftest_config.py`: Lists the tests included in the staging and stable VTS kselftest suites (VtsKernelLinuxKselftest, VtsKernelLinuxKselftestStaging). If a test is not listed here then it will not be run by VTS.
38
39 To run VTS kselftest it must first be built. VTS is not device specific, you
40 need not compile it specifically for the device you wish to run it on, assuming
41 it is the same architecture.
42 * `. build/envsetup.sh`
43 * `lunch`
44 * `make -j vts`
45
46 Before running VTS ensure your host has the required prerequisites (this need
47 only be done once):
48 * `. test/vts/script/setup.sh`
49
50 Then open vts-tradefed and run the VTS stable set:
51 * `vts-tradefed`
52 * `vts-tf > run vts-kernel -m VtsKernelLinuxKselftest`
53
54 If you regularly work with multiple devices it may be useful to specify the
55 specific device you wish to run VTS on via the serial number:
56 * `vts-tf > run vts-kernel -m VtsKernelLinuxKselftest -s 000123456789`
57
58 Or we can run the staging set:
59 * `vts-tf > run vts-kernel -m VtsKernelLinuxKselftestStaging`
60
61 Or a specific test within the stable set:
62 * `vts-tf > run vts-kernel -m VtsKernelLinuxKselftest -t net_socket`
63
64 Maybe several:
65 * `vts-tf > run vts-kernel -m VtsKernelLinuxKselftest -t net_socket,timers_posix_timers`
66
67 Running Kselftest Faster
68 ------------------------
69
70 Running kselftest tests within VTS can be quite cumbersome, especially if you
71 are iterating a lot trying to debug something. Build and run kselftest tests
72 faster by doing
73
74 * `external/linux-kselftest$ mma`
75 * `external/linux-kselftest$ adb sync data`
76
77 The test cases will be located at `/data/nativetest{64,}/linux-kselftest/`.
78
79 Build just the kselftest test you care about with `make -j kselftest_<testname>` (listed in `external/linux-kselftest/android/kselftest_test_list.mk`).
80
81 What Outstanding Issues Exist?
82 ------------------------------
83
84 The hotlist for kselftest bugs is
85 [kselftest-todo](https://buganizer.corp.google.com/hotlists/745928).
86
87 When you begin working on a kselftest bug please assign the bug to yourself so
88 that others know it is being worked on.
89
90 Testing x86_64
91 --------------
92
93 It is not advisable to run kselftest tests directly on your host unless you are fully
94 aware of what the tests will do and are okay with it. These tests may
95 destabilize your box or cause data loss. If you need to run tests on an x86
96 platform and are unsure if they are safe you should run them in emulation, in a
97 virtualized environment, or on a dedicated development x86 platform.
98
99 Sending Fixes Upstream
100 ----------------------
101
102 Kselftest is part of the upstream Linux kernel and as such you can use the
103 get_maintainers.pl script there to see who to send patches to. Heres an
104 example using a commit already upstream:
105
106 ```
107 smuckle@smuckle:~/repos/linux$ git show 352909b49ba | scripts/get_maintainer.pl
108 Shuah Khan <shuah (a] kernel.org> (maintainer:KERNEL SELFTEST FRAMEWORK,commit_signer:2/6=33%,authored:2/6=33%,added_lines:2/6=33%,removed_lines:3/6=50%)
109 Thomas Gleixner <tglx (a] linutronix.de> (commit_signer:2/6=33%)
110 Greg Kroah-Hartman <gregkh (a] linuxfoundation.org> (commit_signer:1/6=17%,authored:1/6=17%,added_lines:1/6=17%)
111 "Radim Krm" <rkrcmar (a] redhat.com> (commit_signer:1/6=17%)
112 Ingo Molnar <mingo (a] kernel.org> (commit_signer:1/6=17%,commit_signer:1/1=100%)
113 Andy Lutomirski <luto (a] kernel.org> (authored:2/6=33%,added_lines:2/6=33%,removed_lines:2/6=33%,authored:1/1=100%,added_lines:500/500=100%)
114 "Kirill A. Shutemov" <kirill.shutemov (a] linux.intel.com> (authored:1/6=17%,added_lines:1/6=17%,removed_lines:1/6=17%)
115 linux-kernel (a] vger.kernel.org (open list)
116 linux-kselftest (a] vger.kernel.org (open list:KERNEL SELFTEST FRAMEWORK)
117 ```
118
119 In summary patches should be sent to linux-kselftest (a] vger.kernel.org and
120 linux-kernel (a] vger.kernel.org, Shuah Khan (kselftest maintainer), and anyone
121 else who has touched the code in question. Standard Linux kernel coding style
122 and patch rules apply. They should be checkpatch (scripts/checkpatch.pl in the
123 kernel repository) clean and sent in plain text in canonical patch format. One
124 easy way to do this is by using git format-patch and git send-email.
125
126 Merging Fixes
127 -------------
128
129 When possible please merge fixes upstream first. Then cherrypick the change
130 onto aosp/master in external/linux-kselftest.
131
132 If your change cannot go upstream first for whatever reason, please commit a
133 patch for it into external/linux-kselftest/android/patches. This allows easier
134 tracking of the delta with upstream and streamlines upgrades to new kselftest
135 releases.
136
137 Updating Kselftest
138 ------------------
139
140 To merge in a new upstream version of kselftest:
141 1. Do a git merge of the upstream tag with the "ours" policy, dropping all upstream changes. Do not commit yet.
142 2. Delete tools/testing/selftests and replace it with a copy from the upstream kernel at the same tag as used above.
143 3. Apply the patches in android/patches/, resolving conflicts as required.
144 4. Test on all supported kernel versions, ensuring that any tests currently enabled in VTS do not generate new failures.
145 5. Commit the merge.
146
147 Notes on Individual Tests
148 -------------------------
149
150 ### bpf/
151
152 The bpf tests depend on the bpf and elf libraries, and several kernel
153 config options that are not required by Android.
154
155 ### cpufreq/
156
157 Cpufreq is not required by Android.
158
159 ### cpu-hotplug/
160
161 Cpu hotplug is not required by Android.
162
163 ### filesystems/dnotify_test.c
164
165 This test has an endless loop in it. It is not currently run as part of the
166 kselftest suite.
167
168 ### firmware/
169
170 These tests depend on a kernel module enabled by CONFIG_TEST_FIRMWARE, which is
171 not required by Android.
172
173 ### ftrace/
174
175 Ftrace is a debug feature and not required by Android.
176
177 ### gpio/
178
179 The gpio test requires a test kernel module and gpiolib, neither of which are
180 required by Android.
181
182 ### ipc/
183
184 The ipc test requires CONFIG_EXPERT and CONFIG_CHECKPOINT_RESTORE be enabled,
185 neither of which are required by Android.
186
187 ### kvm/
188
189 KVM is not required by Android.
190
191 ### lib/
192
193 The lib/ tests rely on kernel modules enabled by CONFIG_TEST_PRINTF,
194 CONFIG_TEST_BITMAP, and CONFIG_PRIME_NUMBERS. None of these are required by
195 Android.
196
197 ### locking/
198
199 The wait-wound mutex test depends on a kernel module enabled by
200 CONFIG_WW_MUTEX_SELFTEST which is not required by Android.
201
202 ### media_tests/
203
204 The media_tests depend on /dev/mediaX and /dev/videoX files which may not be
205 present. They also require manual inspection of dmesg and are not part of
206 the normal kselftest suite.
207
208 ### membarrier/
209
210 The membarrier test depends on CONFIG_EXPERT and CONFIG_MEMBARRIER being
211 enabled, neither of which are required by Android.
212
213 ### memfd/
214
215 These tests depend on FUSE which is not present in Android.
216
217 ### memory-hotplug/
218
219 Memory hotplug is not required by Android.
220
221 ### mount/
222
223 The mount test depends on CONFIG_USER_NS which is not required by Android.
224
225 ### mqueue/
226
227 These tests depend on -lrt and -lpopt which are not present in Android.
228
229 ### net/
230
231 The test_bpf.sh test requires a kernel module enabled by CONFIG_TEST_BPF.
232 The netdevice.sh test operates on an ethernet interface.
233 The psock_fanout test depends on CONFIG_BPF_SYSCALL which is not required on
234 Android.
235
236 ### netfilter/
237
238 These tests depend on CONFIG_NET_NS and CONFIG_NF_TABLES_INET, neither of which
239 are required by Android.
240
241 ### nsfs/
242
243 These tests depend on CONFIG_USER_NS and CONFIG_PID_NS, neither of which are
244 required by Android.
245
246 ### ntb/
247
248 NTB (non-transparent bridge) is not required by Android.
249
250 ### pstore/
251
252 Pstore is recommended, not required, for Android.
253
254 ### ptp/
255
256 PTP support is not required by Android.
257
258 ### rseq/
259
260 The rseq system call, enabled by CONFIG_RSEQ, is not required by Android.
261
262 ### sigaltstack/
263
264 This is a test for functionality that is not available in bionic (swapcontext()).
265
266 ### static_keys/
267
268 This test depends on a kernel module enabled by CONFIG_TEST_STATIC_KEYS. This
269 kernel module is not required by Android.
270
271 ### sync/
272
273 These tests verify the legacy sync API which is deprecated on Android.
274
275 ### sysctl
276
277 This test depends on a kernel module enabled by CONFIG_TEST_SYSCTL. This kernel
278 module is not required by Android.
279
280 ### timers/
281
282 The adjtick test adjusts the system time and takes 2 minutes.
283
284 The change_skew test takes too long to run as part of VTS.
285
286 The clocksource_switch test takes too long to run as part of VTS and also tries
287 all clocksources.
288
289 The freq_step test relies on clocks having a greater precision than what may be
290 available.
291
292 The leap-a-day test repeatedly modifies the system date in order to test leap
293 second behavior. This may disrupt the target environment.
294
295 The leapcrash test adjusts the system date and takes several minutes to run.
296
297 The mqueue_lat test relies on mq_* calls which are not present on Android.
298
299 The rtctest_setdate test modifies the RTC's time and date which may disrupt the
300 target environment.
301
302 The set-2038 test repeatedly sets the time in order to test edge cases. This
303 may disrupt the target enviornment. The test also takes a long time.
304
305 The set-tz test sets the timezone which may disrupt the target environment.
306
307 The skew_consistency test takes too long to run as part of VTS.
308
309 ### uevent/
310
311 This test depends on CONFIG_USER_NS which is not required by Android.
312
313 ### user/
314
315 This test depends on a kernel module enabled by CONFIG_TEST_USER_COPY. This
316 kernel module is not required by Android.
317
318 ### vm/
319
320 The hugepage-mmap, hugepage-shm, compaction, and map_hugetlb tests rely on
321 hugetlbfs/hugetlb page support which is not required by Android.
322
323 The userfaultfd test relies on the userfaultfd syscall which is not required by
324 Android.
325
326 ### watchdog/
327
328 The watchdog test depends on CONFIG_WATCHDOG which is not required by Android.
329
330 ### zram/
331
332 The zram test depends on CONFIG_ZSMALLOC and CONFIG_ZRAM which are not required
333 by Android.
334