1 # syzbot 2 3 `syzbot` system continuously fuzzes main Linux kernel branches and automatically 4 reports found bugs to kernel mailing lists. 5 [syzbot dashboard](https://syzkaller.appspot.com) shows current statuses of 6 bugs. All `syzbot`-reported bugs are also CCed to 7 [syzkaller-bugs mailing list](https://groups.google.com/forum/#!forum/syzkaller-bugs). 8 Direct all questions to `syzkaller (a] googlegroups.com`. 9 10 ## Bug status tracking 11 12 `syzbot` needs to know when a bug is fixed in order to (1) verify that it is 13 in fact fixed and (2) be able to report other similarly-looking crashes 14 (while a bug is considered open all similarly-looking crashes are merged into 15 the existing bug). To understand when a bug is fixed `syzbot` needs to know 16 what commit fixes the bug; once `syzbot` knows the commit it will track when 17 the commit reaches all kernel builds on all tracked branches. Only when the 18 commit reaches all builds, the bug is considered closed (new similarly-looking 19 crashes create a new bug). 20 21 ## Communication with syzbot 22 23 If you fix a bug reported by `syzbot`, please add the provided `Reported-by` 24 tag to the commit. You can also communicate with `syzbot` by replying to its 25 emails. The commands are: 26 27 - to attach a fixing commit to the bug (if you forgot to add `Reported-by` tag): 28 ``` 29 #syz fix: exact-commit-title 30 ```` 31 It's enough that the commit is merged into any tree or you are reasonably sure 32 about its final title, in particular, you don't need to wait for the commit to 33 be merged into upstream tree. `syzbot` only needs to know the title by which 34 it will appear in tested trees. In case of an error or a title change, you can 35 override the commit simply by sending another `#syz fix` command. 36 - to mark the bug as a duplicate of another `syzbot` bug: 37 ``` 38 #syz dup: exact-subject-of-another-report 39 ``` 40 - to undo a previous dup command and turn it into an independent bug again: 41 ``` 42 #syz undup 43 ``` 44 - to mark the bug as a one-off invalid report (e.g. induced by a previous memory corruption): 45 ``` 46 #syz invalid 47 ``` 48 **Note**: if the crash happens again, it will cause creation of a new bug report. 49 50 **Note**: all commands must start from beginning of the line. 51 52 **Note**: please keep `syzkaller-bugs (a] googlegroups.com` mailing list in CC. 53 It serves as a history of what happened with each bug report. 54 55 ## Testing patches 56 57 `syzbot` can test patches for bugs *with reproducers*. This can be used for 58 testing of fix patches, or just for debugging (i.e. adding additional checks to 59 code and testing with them), or to check if the bug still happens. To test on 60 a particular git tree and branch reply with: 61 ``` 62 #syz test: git://repo/address.git branch 63 ``` 64 or alternatively, to test on exact commit reply with: 65 ``` 66 #syz test: git://repo/address.git commit-hash 67 ``` 68 If you also provide a patch with the email, `syzbot` will apply it on top of the 69 tree before testing. The patch can be provided either inline in email text or as 70 a text attachment (which is more reliable if your email client messes with 71 whitespaces). 72 73 If you don't provide a patch, `syzbot` will test the tree as is. 74 This is useful if this is your own tree which already contains the patch, 75 or to check if the bug is already fixed by some recent commit. 76 77 After sending an email you should get a reply email with results within an hour. 78 79 Note: you may send the request only to `syzbot` email address, as patches sent 80 to some mailing lists (e.g. netdev, netfilter-devel) will trigger patchwork. 81 82 ## syzkaller reproducers 83 84 `syzbot` aims at providing stand-alone C reproducers for all reported bugs. 85 However, sometimes it can't extract a reproducer at all, or can only extract a 86 syzkaller reproducer. syzkaller reproducers are programs in a special syzkaller 87 notation and they can be executed on the target system with a little bit more 88 effort. See [this](https://github.com/google/syzkaller/blob/master/docs/executing_syzkaller_programs.md) 89 for instructions. 90 91 A syskaller program can also give you an idea as to what syscalls with what 92 arguments were executed (note that some calls can actually be executed in 93 parallel). 94 95 A syzkaller program can be converted to an almost equivalent C source using `syz-prog2c` utility. `syz-prog2c` has lots of flags in common with [syz-execprog](https://github.com/google/syzkaller/blob/master/docs/executing_syzkaller_programs.md), e.g. `-threaded`/`-collide` which control if the syscalls are executed sequentially or in parallel. An example invocation: 96 97 ``` 98 syz-prog2c -prog repro.syz.txt -threaded -collide -repeat -procs=8 -sandbox=namespace -tun -tmpdir -waitrepeat 99 ``` 100 101 However, note that if `syzbot` did not provide a C reproducer, it wasn't able to trigger the bug using the C program (though, it can be just because the bug is triggered by a subtle race condition). 102 103 ## Crash does not reproduce? 104 105 If the provided reproducer does not work for you, most likely it is related to the 106 fact that you have slightly different setup than `syzbot`. `syzbot` has obtained 107 the provided crash report on the provided reproducer on a freshly-booted 108 machine, so the reproducer worked for it somehow. 109 110 `syzbot` uses GCE VMs for testing, but *usually* it is not important. 111 112 If the reproducer exits quickly, try to run it several times, or in a loop. 113 There can be some races involved. 114 115 Exact compilers used by `syzbot` can be found here: 116 - [gcc 7.1.1 20170620](https://storage.googleapis.com/syzkaller/gcc-7.tar.gz) (245MB) 117 - [gcc 8.0.1 20180301](https://storage.googleapis.com/syzkaller/gcc-8.0.1-20180301.tar.gz) (286MB) 118 - [gcc 8.0.1 20180412](https://storage.googleapis.com/syzkaller/gcc-8.0.1-20180412.tar.gz) (33MB) 119 - [clang 7.0.0 (trunk 329060)](https://storage.googleapis.com/syzkaller/clang-kmsan-329060.tar.gz) (44MB) 120 - [clang 7.0.0 (trunk 334104)](https://storage.googleapis.com/syzkaller/clang-kmsan-334104.tar.gz) (44MB) 121 122 A qemu-suitable Debian/wheezy image can be found [here](https://storage.googleapis.com/syzkaller/wheezy.img) (1GB, compression somehow breaks it), root ssh key for it is [here](https://storage.googleapis.com/syzkaller/wheezy.img.key). 123 A reference `qemu` command line to run it is as follows: 124 ``` 125 qemu-system-x86_64 -hda wheezy.img -net user,hostfwd=tcp::10022-:22 \ 126 -net nic -nographic -enable-kvm -m 2G -smp 4 -cpu host \ 127 -kernel arch/x86/boot/bzImage \ 128 -append "root=/dev/sda console=ttyS0 earlyprintk=serial rodata=n \ 129 oops=panic panic_on_warn=1 panic=86400 kvm-intel.nested=1" 130 ``` 131 And then you can ssh into it using: 132 ``` 133 ssh -p 10022 -i wheezy.id_rsa root@localhost 134 ``` 135 136 ## No reproducer at all? 137 138 Reproducers are best-effort. `syzbot` always tries to create reproducers, and 139 once it has one it adds it to the bug. If there is no reproducer referenced in a 140 bug, a reproducer does not exist. There are multiple reasons why `syzbot` can 141 fail to create a reproducer: some crashes are caused by subtle races and are 142 very hard to reproduce in general; some crashes are caused by global accumulated 143 state in kernel (e.g. lockdep reports); some crashes are caused by 144 non-reproducible coincidences (e.g. an integer `0x12345` happened to reference an 145 existing IPC object) and there is long tail of other reasons. 146 147 ## Moderation queue 148 149 Bugs with reproducers are automatically reported to kernel mailing lists. 150 Bugs without reproducers are first staged in moderation queue to filter out 151 invalid, unactionable or duplicate reports. Staged bugs are shown on dashboard 152 in [moderation](https://syzkaller.appspot.com/#upstream-moderation2) section 153 and mailed to 154 [syzkaller-upstream-moderation](https://groups.google.com/forum/#!forum/syzkaller-upstream-moderation) 155 mailing list. Staged bugs accept all commands supported for reported bugs 156 (`fix`, `dup`, `invalid`) with a restriction that reported and staged bugs 157 can't be `dup`-ed onto each other in any direction. Additionally, staged bugs 158 accept upstream command: 159 ``` 160 #syz upstream 161 ``` 162 which sends the bug to kernel mailing lists. 163 164 ## KMSAN bugs 165 166 `KMSAN` is a dynamic, compiler-based tool (similar to `KASAN`) that detects 167 uses of uninitialized values. As compared to (now deleted) `KMEMCHECK` which 168 simply detected loads of non-stored-to memory, `KMSAN` tracks precise 169 propagation of uninitialized values through memory and registers and only flags 170 actual eventual uses of uninitialized values. For example, `KMSAN` will detect 171 a branch on or a `copy_to_user()` of values that transitively come from 172 uninitialized memory created by heap/stack allocations. This ensures 173 /theoretical/ absense of both false positives and false negatives (with some 174 implementation limitations of course). 175 176 `KMSAN` is not upstream yet, though, we want to upstream it later. For now, 177 it lives in [github.com/google/kmsan](https://github.com/google/kmsan) and is 178 based on a reasonably fresh upstream tree. As the result, any patch testing 179 requests for `KMSAN` bugs need to go to `KMSAN` tree 180 (`https://github.com/google/kmsan.git` repo, `master` branch). Also note that 181 `KMSAN` requires `clang` compiler. 182 183 Report explanation. The first call trace points to the `use` of the uninit value 184 (which is usually a branching or copying it to userspace). Then there are 0 or 185 more "Uninit was stored to memory at:" stacks which denote how the unint value 186 travelled through memory. Finally there is a "Uninit was created at:" 187 section which points either to a heap allocation or a stack variable which 188 is the original source of uninitialized-ness. 189 190 ## No custom patches 191 192 While `syzbot` can test patches that fix bugs, it does not support applying 193 custom patches during fuzzing. It always tests vanilla unmodified git trees. 194 There are several reasons for this: 195 196 - custom patches may not apply tomorrow 197 - custom patches may not apply to all of the tested git trees 198 - it's hard to communicate exact state of the code with bug reports (not just hash anymore) 199 - line numbers won't match in reports (which always brings suspecion as to the quality of reports) 200 - custom patches can also introduce bugs, and even if they don't a developer may (rightfully) 201 question validity of and may not want to spend time on reports obtained 202 with a number of out-of-tree patches 203 - order of patch application generatelly matters, and at some point patches 204 need to be removed, there is nobody to manage this 205 206 We've experimented with application of custom patches in the past and it lead 207 to unrecoverable mess. If you want `syzbot` to pick up patches sooner, 208 ask tree maintainers for priority handling. 209 210 ## Kernel configs 211 212 Kernel configs, sysctls and command line arguments that `syzbot` uses are available in [/dashboard/config](/dashboard/config). 213 214 ## Is syzbot code available? 215 216 Yes, it is [here](https://github.com/google/syzkaller/tree/master/dashboard/app). 217