1 DexFuzz 2 ======= 3 4 DexFuzz is primarily a tool for fuzzing DEX files. Fuzzing is the introduction of 5 subtle changes ("mutations") to a file to produce a new test case. These test cases 6 can be used to test the various modes of execution available to ART (Interpreter, 7 Quick compiler, Optimizing compiler) to check for bugs in these modes of execution. 8 This is done by differential testing - each test file is executed with each mode of 9 execution, and any differences between the resulting outputs may be an indication of 10 a bug in one of the modes. 11 12 For a wider overview of DexFuzz, see: 13 14 http://community.arm.com/groups/android-community/blog/2014/11/26/the-art-of-fuzz-testing 15 16 In typical operation, you provide DexFuzz with a set of DEX files that are the "seeds" 17 for mutation - e.g. some tests taken from the ART test suite - and point it at an 18 ADB-connected Android device, and it will fuzz these seed files, and execute the 19 resulting new tests on the Android device. 20 21 How to run DexFuzz 22 ================== 23 24 DexFuzz can run its test programs on either an ADB-connected device, or a host-build of 25 ART locally. 26 27 Execution on an ADB-connected device 28 ------------------------------------ 29 30 1. Build dexfuzz with mmma tools/dexfuzz from within art/. 31 2. Make sure you have an Android device connected via ADB, that is capable of 32 having DEX files pushed to it and executed with the dalvikvm command. 33 3. Make sure you're in the Android build environment! 34 (That is, . build/envsetup.sh && lunch) 35 4. Create a new directory, and place some DEX files in here. These are the seed files 36 that are mutated to form new tests. 37 5. Create a directory on your device that mutated test files can be pushed to and 38 executed in, using dalvikvm. For example, /data/art-test/ 39 6. If you currently have multiple devices connected via ADB, find out the name of 40 your device using "adb devices -l". 41 7. Run this command: 42 43 dexfuzz --inputs=<seeds dir> --execute --repeat=<attempts> \ 44 --dump-output <combination of ISA(s) and and backend(s)> 45 46 You MUST specify one of the following ISAs: 47 --arm 48 --arm64 49 --x86 50 --x86_64 51 --mips 52 --mips64 53 54 And also at least two of the following backends: 55 --interpreter 56 --quick 57 --optimizing 58 59 Note that if you wanted to test both ARM and ARM64 on an ARM64 device, you can use 60 --allarm. Also in this case only one backend is needed, if i.e., you wanted to test 61 ARM Quick Backend vs. ARM64 Quick Backend. 62 63 Some legal examples: 64 --arm --quick --optimizing 65 --x86 --quick --optimizing --interpreter 66 --allarm --quick 67 68 Add in --device=<device name, e.g. device:generic> if you want to specify a device. 69 Add in --execute-dir=<dir on device> if you want to specify an execution directory. 70 (The default is /data/art-test/) 71 72 Host Execution 73 -------------- 74 75 DexFuzz now supports execution on your host machine. 76 Follow steps 1, 3, 4, and 7 as above, but also observe the following: 77 - instead of specifying an ISA, use --host 78 - ANDROID_DATA must be set, pointing to a location where dex2oat will place 79 OAT files after compilation. 80 - Files will always be executed in the same directory where you are executing DexFuzz. 81 82 Fuzzer Operation 83 ---------------- 84 85 As the fuzzer works, you'll see output like: 86 87 |-----------------------------------------------------------------| 88 |Iterations|VerifyFail|MutateFail|Timed Out |Successful|Divergence| 89 |-----------------------------------------------------------------| 90 | 48 | 37 | 4 | 0 | 6 | 1 | 91 92 Iterations - number of attempts we've made to mutate DEX files. 93 VerifyFail - the number of mutated files that ended up failing to verify, either 94 on the host, or the target. 95 MutateFail - because mutation is a random process, and has attempt thresholds to 96 avoid attempting to mutate a file indefinitely, it is possible that 97 an attempt to mutate a file doesn't actually mutate it. This counts 98 those occurrences. 99 Timed Out - mutated files that timed out for one or more backends. 100 Current timeouts are: 101 Quick - 5 seconds 102 Optimizing - 5 seconds 103 Intepreter - 30 seconds 104 (use --short-timeouts to set all backends to 2 seconds.) 105 Successful - mutated files that executed and all backends agreed on the resulting 106 output. NB: if all backends crashed with the same output, this would 107 be considered a success - proper detection of crashes is still to come. 108 Divergence - mutated files that executed and some backend disagreed about the 109 resulting output. Divergent programs are run multiple times with a 110 single backend, to check if they diverge from themselves, and these are 111 not included in the count. If multiple architectures are being used 112 (ARM/ARM64), and the divergences align with different architectures, 113 these are also not included in the count. 114 115 8. Check report.log for the full report, including input file and RNG seed for each 116 test program. This allows you to recreate a bad program with, e.g.: 117 118 dexfuzz --input=<input file> --seed=<seed value> 119 120 Check dexfuzz --help for the full list of options. 121 122 NOTE: DEX files with unicode strings are not fully supported yet, and DEX files with 123 JNI elements are not supported at all currently. 124 125 Mutation Likelihoods 126 ==================== 127 128 Each bytecode mutation has a chance out of 100% of firing. Following is the listing 129 of each mutation's probability. If you wish to easily adjust these values, copy 130 these values into a file called likelihoods.txt, and run dexfuzz with 131 --likelihoods=likelihoods.txt. 132 133 ArithOpChanger 75 134 BranchShifter 30 135 CmpBiasChanger 30 136 ConstantValueChanger 70 137 ConversionRepeater 50 138 FieldFlagChanger 40 139 InstructionDeleter 40 140 InstructionDuplicator 80 141 InstructionSwapper 80 142 NewMethodCaller 10 143 NonsenseStringPrinter 10 144 PoolIndexChanger 30 145 RandomInstructionGenerator 30 146 SwitchBranchShifter 30 147 TryBlockShifter 40 148 ValuePrinter 40 149 VRegChanger 60 150
