Home | History | Annotate | Download | only in docs
      1 # syzbot
      2 
      3 `syzbot` system continuously fuzzes main Linux kernel branches and automatically
      4 reports found bugs to kernel mailing lists.
      5 [syzbot dashboard](https://syzkaller.appspot.com) shows current statuses of
      6 bugs. All `syzbot`-reported bugs are also CCed to
      7 [syzkaller-bugs mailing list](https://groups.google.com/forum/#!forum/syzkaller-bugs).
      8 Direct all questions to `syzkaller (a] googlegroups.com`.
      9 
     10 ## Bug status tracking
     11 
     12 `syzbot` needs to know when a bug is fixed in order to (1) verify that it is
     13 in fact fixed and (2) be able to report other similarly-looking crashes
     14 (while a bug is considered open all similarly-looking crashes are merged into
     15 the existing bug). To understand when a bug is fixed `syzbot` needs to know
     16 what commit fixes the bug; once `syzbot` knows the commit it will track when
     17 the commit reaches all kernel builds on all tracked branches. Only when the
     18 commit reaches all builds, the bug is considered closed (new similarly-looking
     19 crashes create a new bug).
     20 
     21 ## Communication with syzbot
     22 
     23 If you fix a bug reported by `syzbot`, please add the provided `Reported-by`
     24 tag to the commit. You can also communicate with `syzbot` by replying to its
     25 emails. The commands are:
     26 
     27 - to attach a fixing commit to the bug (if you forgot to add `Reported-by` tag):
     28 ```
     29 #syz fix: exact-commit-title
     30 ````
     31 It's enough that the commit is merged into any tree or you are reasonably sure
     32 about its final title, in particular, you don't need to wait for the commit to
     33 be merged into upstream tree. `syzbot` only needs to know the title by which
     34 it will appear in tested trees. In case of an error or a title change, you can
     35 override the commit simply by sending another `#syz fix` command.
     36 - to mark the bug as a duplicate of another `syzbot` bug:
     37 ```
     38 #syz dup: exact-subject-of-another-report
     39 ```
     40 - to undo a previous dup command and turn it into an independent bug again:
     41 ```
     42 #syz undup
     43 ```
     44 - to mark the bug as a one-off invalid report (e.g. induced by a previous memory corruption):
     45 ```
     46 #syz invalid
     47 ```
     48 **Note**: if the crash happens again, it will cause creation of a new bug report.
     49 
     50 **Note**: all commands must start from beginning of the line.
     51 
     52 **Note**: please keep `syzkaller-bugs (a] googlegroups.com` mailing list in CC.
     53 It serves as a history of what happened with each bug report.
     54 
     55 ## Testing patches
     56 
     57 `syzbot` can test patches for bugs *with reproducers*. This can be used for
     58 testing of fix patches, or just for debugging (i.e. adding additional checks to
     59 code and testing with them), or to check if the bug still happens. To test on
     60 a particular git tree and branch reply with:
     61 ```
     62 #syz test: git://repo/address.git branch
     63 ```
     64 or alternatively, to test on exact commit reply with:
     65 ```
     66 #syz test: git://repo/address.git commit-hash
     67 ```
     68 If you also provide a patch with the email, `syzbot` will apply it on top of the
     69 tree before testing. The patch can be provided either inline in email text or as
     70 a text attachment (which is more reliable if your email client messes with
     71 whitespaces).
     72 
     73 If you don't provide a patch, `syzbot` will test the tree as is.
     74 This is useful if this is your own tree which already contains the patch,
     75 or to check if the bug is already fixed by some recent commit.
     76 
     77 After sending an email you should get a reply email with results within an hour.
     78 
     79 Note: you may send the request only to `syzbot` email address, as patches sent
     80 to some mailing lists (e.g. netdev, netfilter-devel) will trigger patchwork.
     81 
     82 ## syzkaller reproducers
     83 
     84 `syzbot` aims at providing stand-alone C reproducers for all reported bugs.
     85 However, sometimes it can't extract a reproducer at all, or can only extract a
     86 syzkaller reproducer. syzkaller reproducers are programs in a special syzkaller
     87 notation and they can be executed on the target system with a little bit more
     88 effort. See [this](https://github.com/google/syzkaller/blob/master/docs/executing_syzkaller_programs.md)
     89 for instructions.
     90 
     91 A syskaller program can also give you an idea as to what syscalls with what
     92 arguments were executed (note that some calls can actually be executed in
     93 parallel).
     94 
     95 A syzkaller program can be converted to an almost equivalent C source using `syz-prog2c` utility. `syz-prog2c` has lots of flags in common with [syz-execprog](https://github.com/google/syzkaller/blob/master/docs/executing_syzkaller_programs.md), e.g. `-threaded`/`-collide` which control if the syscalls are executed sequentially or in parallel. An example invocation:
     96 
     97 ```
     98 syz-prog2c -prog repro.syz.txt -threaded -collide -repeat -procs=8 -sandbox=namespace -tun -tmpdir -waitrepeat
     99 ```
    100 
    101 However, note that if `syzbot` did not provide a C reproducer, it wasn't able to trigger the bug using the C program (though, it can be just because the bug is triggered by a subtle race condition).
    102 
    103 ## Crash does not reproduce?
    104 
    105 If the provided reproducer does not work for you, most likely it is related to the
    106 fact that you have slightly different setup than `syzbot`. `syzbot` has obtained
    107 the provided crash report on the provided reproducer on a freshly-booted
    108 machine, so the reproducer worked for it somehow.
    109 
    110 `syzbot` uses GCE VMs for testing, but *usually* it is not important.
    111 
    112 If the reproducer exits quickly, try to run it several times, or in a loop.
    113 There can be some races involved.
    114 
    115 Exact compilers used by `syzbot` can be found here:
    116 - [gcc 7.1.1 20170620](https://storage.googleapis.com/syzkaller/gcc-7.tar.gz) (245MB)
    117 - [gcc 8.0.1 20180301](https://storage.googleapis.com/syzkaller/gcc-8.0.1-20180301.tar.gz) (286MB)
    118 - [gcc 8.0.1 20180412](https://storage.googleapis.com/syzkaller/gcc-8.0.1-20180412.tar.gz) (33MB)
    119 - [clang 7.0.0 (trunk 329060)](https://storage.googleapis.com/syzkaller/clang-kmsan-329060.tar.gz) (44MB)
    120 - [clang 7.0.0 (trunk 334104)](https://storage.googleapis.com/syzkaller/clang-kmsan-334104.tar.gz) (44MB)
    121 
    122 A qemu-suitable Debian/wheezy image can be found [here](https://storage.googleapis.com/syzkaller/wheezy.img) (1GB, compression somehow breaks it), root ssh key for it is [here](https://storage.googleapis.com/syzkaller/wheezy.img.key).
    123 A reference `qemu` command line to run it is as follows:
    124 ```
    125 qemu-system-x86_64 -hda wheezy.img -net user,hostfwd=tcp::10022-:22 \
    126     -net nic -nographic -enable-kvm -m 2G -smp 4 -cpu host \
    127     -kernel arch/x86/boot/bzImage \
    128     -append "root=/dev/sda console=ttyS0 earlyprintk=serial rodata=n \
    129       oops=panic panic_on_warn=1 panic=86400 kvm-intel.nested=1"
    130 ```
    131 And then you can ssh into it using:
    132 ```
    133 ssh -p 10022 -i wheezy.id_rsa root@localhost
    134 ```
    135 
    136 ## No reproducer at all?
    137 
    138 Reproducers are best-effort. `syzbot` always tries to create reproducers, and
    139 once it has one it adds it to the bug. If there is no reproducer referenced in a
    140 bug, a reproducer does not exist. There are multiple reasons why `syzbot` can
    141 fail to create a reproducer: some crashes are caused by subtle races and are
    142 very hard to reproduce in general; some crashes are caused by global accumulated
    143 state in kernel (e.g. lockdep reports); some crashes are caused by
    144 non-reproducible coincidences (e.g. an integer `0x12345` happened to reference an
    145 existing IPC object) and there is long tail of other reasons.
    146 
    147 ## Moderation queue
    148 
    149 Bugs with reproducers are automatically reported to kernel mailing lists.
    150 Bugs without reproducers are first staged in moderation queue to filter out
    151 invalid, unactionable or duplicate reports. Staged bugs are shown on dashboard
    152 in [moderation](https://syzkaller.appspot.com/#upstream-moderation2) section
    153 and mailed to
    154 [syzkaller-upstream-moderation](https://groups.google.com/forum/#!forum/syzkaller-upstream-moderation)
    155 mailing list. Staged bugs accept all commands supported for reported bugs
    156 (`fix`, `dup`, `invalid`) with a restriction that reported and staged bugs
    157 can't be `dup`-ed onto each other in any direction. Additionally, staged bugs
    158 accept upstream command:
    159 ```
    160 #syz upstream
    161 ```
    162 which sends the bug to kernel mailing lists.
    163 
    164 ## KMSAN bugs
    165 
    166 `KMSAN` is a dynamic, compiler-based tool (similar to `KASAN`) that detects
    167 uses of uninitialized values. As compared to (now deleted) `KMEMCHECK` which
    168 simply detected loads of non-stored-to memory, `KMSAN` tracks precise
    169 propagation of uninitialized values through memory and registers and only flags
    170 actual eventual uses of uninitialized values. For example, `KMSAN` will detect
    171 a branch on or a `copy_to_user()` of values that transitively come from
    172 uninitialized memory created by heap/stack allocations. This ensures
    173 /theoretical/ absense of both false positives and false negatives (with some
    174 implementation limitations of course).
    175 
    176 `KMSAN` is not upstream yet, though, we want to upstream it later. For now,
    177 it lives in [github.com/google/kmsan](https://github.com/google/kmsan) and is
    178 based on a reasonably fresh upstream tree. As the result, any patch testing
    179 requests for `KMSAN` bugs need to go to `KMSAN` tree
    180 (`https://github.com/google/kmsan.git` repo, `master` branch). Also note that
    181 `KMSAN` requires `clang` compiler.
    182 
    183 Report explanation. The first call trace points to the `use` of the uninit value
    184 (which is usually a branching or copying it to userspace). Then there are 0 or
    185 more "Uninit was stored to memory at:" stacks which denote how the unint value
    186 travelled through memory. Finally there is a "Uninit was created at:"
    187 section which points either to a heap allocation or a stack variable which
    188 is the original source of uninitialized-ness.
    189 
    190 ## No custom patches
    191 
    192 While `syzbot` can test patches that fix bugs, it does not support applying
    193 custom patches during fuzzing. It always tests vanilla unmodified git trees.
    194 There are several reasons for this:
    195 
    196 - custom patches may not apply tomorrow
    197 - custom patches may not apply to all of the tested git trees
    198 - it's hard to communicate exact state of the code with bug reports (not just hash anymore)
    199 - line numbers won't match in reports (which always brings suspecion as to the quality of reports)
    200 - custom patches can also introduce bugs, and even if they don't a developer may (rightfully)
    201   question validity of and may not want to spend time on reports obtained
    202   with a number of out-of-tree patches
    203 - order of patch application generatelly matters, and at some point patches
    204   need to be removed, there is nobody to manage this
    205 
    206 We've experimented with application of custom patches in the past and it lead
    207 to unrecoverable mess. If you want `syzbot` to pick up patches sooner,
    208 ask tree maintainers for priority handling.
    209 
    210 ## Kernel configs
    211 
    212 Kernel configs, sysctls and command line arguments that `syzbot` uses are available in [/dashboard/config](/dashboard/config).
    213 
    214 ## Is syzbot code available?
    215 
    216 Yes, it is [here](https://github.com/google/syzkaller/tree/master/dashboard/app).
    217