1 JFuzz 2 ===== 3 4 JFuzz is a tool for generating random programs with the objective 5 of fuzz testing the ART infrastructure. Each randomly generated program 6 can be run under various modes of execution, such as using the interpreter, 7 using the optimizing compiler, using an external reference implementation, 8 or using various target architectures. Any difference between the outputs 9 (**divergence**) may indicate a bug in one of the execution modes. 10 11 JFuzz can be combined with DexFuzz to get multi-layered fuzz testing. 12 13 How to run JFuzz 14 ================ 15 16 jfuzz [-s seed] [-d expr-depth] [-l stmt-length] 17 [-i if-nest] [-n loop-nest] [-v] [-h] 18 19 where 20 21 -s : defines a deterministic random seed 22 (randomized using time by default) 23 -d : defines a fuzzing depth for expressions 24 (higher values yield deeper expressions) 25 -l : defines a fuzzing length for statement lists 26 (higher values yield longer statement sequences) 27 -i : defines a fuzzing nest for if/switch statements 28 (higher values yield deeper nested conditionals) 29 -n : defines a fuzzing nest for for/while/do-while loops 30 (higher values yield deeper nested loops) 31 -t : defines a fuzzing nest for try-catch-finally blocks 32 (higher values yield deeper nested try-catch-finally blocks) 33 -v : prints version number and exits 34 -h : prints help and exits 35 36 The current version of JFuzz sends all output to stdout, and uses 37 a fixed testing class named Test. So a typical test run looks as follows. 38 39 jfuzz > Test.java 40 jack -cp ${JACK_CLASSPATH} --output-dex . Test.java 41 art -classpath classes.dex Test 42 43 How to start JFuzz testing 44 ========================== 45 46 run_jfuzz_test.py 47 [--num_tests=NUM_TESTS] 48 [--device=DEVICE] 49 [--mode1=MODE] [--mode2=MODE] 50 [--report_script=SCRIPT] 51 [--jfuzz_arg=ARG] 52 [--true_divergence] 53 [--dexer=DEXER] 54 [--debug_info] 55 56 where 57 58 --num_tests : number of tests to run (10000 by default) 59 --device : target device serial number (passed to adb -s) 60 --mode1 : m1 61 --mode2 : m2, with m1 != m2, and values one of 62 ri = reference implementation on host (default for m1) 63 hint = Art interpreter on host 64 hopt = Art optimizing on host (default for m2) 65 tint = Art interpreter on target 66 topt = Art optimizing on target 67 --report_script : path to script called for each divergence 68 --jfuzz_arg : argument for jfuzz 69 --true_divergence : don't bisect timeout divergences 70 --dexer=DEXER : use either dx, d8, or jack to obtain dex files 71 --debug_info : include debugging info 72 73 How to start JFuzz nightly testing 74 ================================== 75 76 run_jfuzz_test_nightly.py 77 [--num_proc NUM_PROC] 78 79 where 80 81 --num_proc : number of run_jfuzz_test.py instances to run (8 by default) 82 83 Remaining arguments are passed to run\_jfuzz_test.py. 84 85 How to start J/DexFuzz testing (multi-layered) 86 ============================================== 87 88 run_dex_fuzz_test.py 89 [--num_tests=NUM_TESTS] 90 [--num_inputs=NUM_INPUTS] 91 [--device=DEVICE] 92 [--dexer=DEXER] 93 [--debug_info] 94 95 where 96 97 --num_tests : number of tests to run (10000 by default) 98 --num_inputs : number of JFuzz programs to generate 99 --device : target device serial number (passed to adb -s) 100 --dexer=DEXER : use either dx, d8, or jack to obtain dex files 101 --debug_info : include debugging info 102 103 Background 104 ========== 105 106 Although test suites are extremely useful to validate the correctness of a 107 system and to ensure that no regressions occur, any test suite is necessarily 108 finite in size and scope. Tests typically focus on validating particular 109 features by means of code sequences most programmers would expect. Regression 110 tests often use slightly less idiomatic code sequences, since they reflect 111 problems that were not anticipated originally, but occurred in the field. 112 Still, any test suite leaves the developer wondering whether undetected bugs 113 and flaws still linger in the system. 114 115 Over the years, fuzz testing has gained popularity as a testing technique for 116 discovering such lingering bugs, including bugs that can bring down a system 117 in an unexpected way. Fuzzing refers to feeding a large amount of random data 118 as input to a system in an attempt to find bugs or make it crash. Generation- 119 based fuzz testing constructs random, but properly formatted input data. 120 Mutation-based fuzz testing applies small random changes to existing inputs 121 in order to detect shortcomings in a system. Profile-guided or coverage-guided 122 fuzzing adds a direction to the way these random changes are applied. Multi- 123 layered approaches generate random inputs that are subsequently mutated at 124 various stages of execution. 125 126 The randomness of fuzz testing implies that the size and scope of testing is no 127 longer bounded. Every new run can potentially discover bugs and crashes that were 128 hereto undetected. 129