1 DexFuzz 2 ======= 3 4 DexFuzz is primarily a tool for fuzzing DEX files. Fuzzing is the introduction of 5 subtle changes ("mutations") to a file to produce a new test case. These test cases 6 can be used to test the various modes of execution available to ART (Interpreter, 7 Optimizing compiler) to check for bugs in these modes of execution. 8 This is done by differential testing - each test file is executed with each mode of 9 execution, and any differences between the resulting outputs may be an indication of 10 a bug in one of the modes. 11 12 For a wider overview of DexFuzz, see: 13 14 http://community.arm.com/groups/android-community/blog/2014/11/26/the-art-of-fuzz-testing 15 16 In typical operation, you provide DexFuzz with a set of DEX files that are the "seeds" 17 for mutation - e.g. some tests taken from the ART test suite - and point it at an 18 ADB-connected Android device, and it will fuzz these seed files, and execute the 19 resulting new tests on the Android device. 20 21 How to run DexFuzz 22 ================== 23 24 DexFuzz can run its test programs on either an ADB-connected device, or a host-build of 25 ART locally. 26 27 Execution on an ADB-connected device 28 ------------------------------------ 29 30 1. Build dexfuzz with mmma tools/dexfuzz from within art/. 31 2. Make sure you have an Android device connected via ADB, that is capable of 32 having DEX files pushed to it and executed with the dalvikvm command. 33 3. Make sure you're in the Android build environment! 34 (That is, . build/envsetup.sh && lunch) 35 4. Create a new directory, and place some DEX files in here. These are the seed files 36 that are mutated to form new tests. 37 5. Create a directory on your device that mutated test files can be pushed to and 38 executed in, using dalvikvm. For example, /data/art-test/ 39 6. If you currently have multiple devices connected via ADB, find out the name of 40 your device using "adb devices -l". 41 7. Run this command: 42 43 dexfuzz --inputs=<seeds dir> --execute --repeat=<attempts> \ 44 --dump-output <combination of ISA(s) and and backend(s)> 45 46 You MUST specify one of the following ISAs: 47 --arm 48 --arm64 49 --x86 50 --x86_64 51 --mips 52 --mips64 53 54 And also at least two of the following backends: 55 --interpreter 56 --optimizing 57 58 Note that if you wanted to test both ARM and ARM64 on an ARM64 device, you can use 59 --allarm. Also in this case only one backend is needed, if i.e., you wanted to test 60 ARM Optimizing Backend vs. ARM64 Optimizing Backend. 61 62 Some legal examples: 63 --arm --optimizing --interpreter 64 --x86 --optimizing --interpreter 65 --allarm --optimizing 66 67 Add in --device=<device name, e.g. device:generic> if you want to specify a device. 68 Add in --execute-dir=<dir on device> if you want to specify an execution directory. 69 (The default is /data/art-test/) 70 71 Host Execution 72 -------------- 73 74 DexFuzz now supports execution on your host machine. 75 Follow steps 1, 3, 4, and 7 as above, but also observe the following: 76 - instead of specifying an ISA, use --host 77 - ANDROID_DATA must be set, pointing to a location where dex2oat will place 78 OAT files after compilation. 79 - Files will always be executed in the same directory where you are executing DexFuzz. 80 81 Fuzzer Operation 82 ---------------- 83 84 As the fuzzer works, you'll see output like: 85 86 |-----------------------------------------------------------------| 87 |Iterations|VerifyFail|MutateFail|Timed Out |Successful|Divergence| 88 |-----------------------------------------------------------------| 89 | 48 | 37 | 4 | 0 | 6 | 1 | 90 91 Iterations - number of attempts we've made to mutate DEX files. 92 VerifyFail - the number of mutated files that ended up failing to verify, either 93 on the host, or the target. 94 MutateFail - because mutation is a random process, and has attempt thresholds to 95 avoid attempting to mutate a file indefinitely, it is possible that 96 an attempt to mutate a file doesn't actually mutate it. This counts 97 those occurrences. 98 Timed Out - mutated files that timed out for one or more backends. 99 Current timeouts are: 100 Optimizing - 5 seconds 101 Interpreter - 30 seconds 102 (use --short-timeouts to set all backends to 2 seconds.) 103 Successful - mutated files that executed and all backends agreed on the resulting 104 output. NB: if all backends crashed with the same output, this would 105 be considered a success - proper detection of crashes is still to come. 106 Divergence - mutated files that executed and some backend disagreed about the 107 resulting output. Divergent programs are run multiple times with a 108 single backend, to check if they diverge from themselves, and these are 109 not included in the count. If multiple architectures are being used 110 (ARM/ARM64), and the divergences align with different architectures, 111 these are also not included in the count. 112 113 8. Check report.log for the full report, including input file and RNG seed for each 114 test program. This allows you to recreate a bad program with, e.g.: 115 116 dexfuzz --input=<input file> --seed=<seed value> 117 118 Check dexfuzz --help for the full list of options. 119 120 NOTE: DEX files with unicode strings are not fully supported yet, and DEX files with 121 JNI elements are not supported at all currently. 122 123 Mutation Likelihoods 124 ==================== 125 126 Each bytecode mutation has a chance out of 100% of firing. Following is the listing 127 of each mutation's probability. If you wish to easily adjust these values, copy 128 these values into a file called likelihoods.txt, and run dexfuzz with 129 --likelihoods=likelihoods.txt. 130 131 ArithOpChanger 75 132 BranchShifter 30 133 CmpBiasChanger 30 134 ConstantValueChanger 70 135 ConversionRepeater 50 136 FieldFlagChanger 40 137 InstructionDeleter 40 138 InstructionDuplicator 80 139 InstructionSwapper 80 140 NewMethodCaller 10 141 NonsenseStringPrinter 10 142 PoolIndexChanger 30 143 RandomInstructionGenerator 30 144 SwitchBranchShifter 30 145 TryBlockShifter 40 146 ValuePrinter 40 147 VRegChanger 60 148
