1 Android Clang/LLVM 2 ================== 3 4 Platform Projects 5 ----------------- 6 7 #### external/llvm 8 **Branch**: *aosp/dev* 9 10 * This branch tracks LLVM upstream directly and contains our Android-specific 11 patches that do not exist in upstream. 12 13 **Branch**: *aosp/master* 14 15 * This branch receives updates from the *aosp/dev* branch as a squashed single 16 commit. 17 Any patches submitted here need to be replayed in *aosp/dev* as well for 18 consistency with future rebases. 19 20 #### external/clang 21 **Branch**: *aosp/dev* 22 23 * This branch tracks Clang upstream directly and contains our Android-specific 24 patches that do not exist in upstream. 25 26 **Branch**: *aosp/master* 27 28 * This branch receives updates from the *aosp/dev* branch as a squashed single 29 commit. 30 Any patches submitted here need to be replayed in *aosp/dev* as well for 31 consistency with future rebases. 32 33 #### external/compiler-rt 34 **Branch**: *aosp/dev* 35 36 * This branch tracks compiler-rt upstream directly and contains our 37 Android-specific patches that do not exist in upstream. 38 39 **Branch**: *aosp/master* 40 41 * This branch receives updates from the *aosp/dev* branch as a squashed single 42 commit. 43 Any patches submitted here need to be replayed in *aosp/dev* as well for 44 consistency with future rebases. 45 46 47 **Note**: Similar branching strategies can be used for **external/libcxx** and 48 **external/libcxxabi**, but these projects are less divergent and may not need 49 special LLVM rebase behavior. 50 51 52 Development Flow 53 ---------------- 54 55 Rebases take place in the **external/** *aosp/dev* branches by merging 56 upstream CLs directly. 57 Conflicts are resolved manually and then a patch is produced to adapt 58 Android.mk files (as well as to add/delete any updated source files). 59 Prebuilts are then generated using these projects and placed into the proper 60 **prebuilts/clang/host** subprojects. 61 62 For example, these projects for LLVM 3.8 would be: 63 64 prebuilts/clang/host/linux-x86/3.8 65 prebuilts/clang/host/darwin-x86/3.8 66 prebuilts/clang/host/windows-x86/3.8 67 68 In order to prepare for the actual rebase (including updating dependent 69 projects), we will copy each **external/** *aosp/dev* project to its 70 corresponding **external/** *aosp/master* project as a squashed single CL. 71 This makes rollbacks simpler, and it also reduces churn on the Android build 72 servers. 73 This also has the side effect of not spamming non-Android @google.com 74 committers to upstream LLVM projects, since their commits will be batched up 75 into a single copy commit on each tracked **external/** project. 76 77 Prebuilts for llvm-rs-cc and bcc\_compat also need to be generated for 78 prebuilts/sdk. 79 This is done by running **frameworks/rs/update\_rs\_prebuilts.sh** on both Mac 80 and Linux. 81 After this completes, the **prebuilts/sdk** project will have a prepared 82 branch/CL that can be uploaded for review/commit. 83 84 85 Fixing Bugs 86 ----------- 87 88 If we find a host-side bug that needs to be fixed, it may trigger an update of 89 the host prebuilts (i.e. rebase). 90 Device-side fixes can be pushed directly to **external/** *aosp/master* and then 91 copied to **external/** *aosp/dev* to speed up the process (assuming that it 92 doesnt affect the host builds). 93 94 95 Looking at Upstream 96 ------------------- 97 98 Here are the remotes to add in your **external/** projects. You can synchronize 99 them with `git fetch upstream`. 100 101 102 Clang: git remote add upstream http://llvm.org/git/clang.git 103 LLVM: git remote add upstream http://llvm.org/git/llvm.git 104 compiler-rt: git remote add upstream http://llvm.org/git/compiler-rt.git 105 106 107 LLVM 3.8 era rebase process 108 --------------------------- 109 110 Loop over llvm, clang, compiler-rt (in this order): 111 112 1. We are working from a separate untracked/merged branch called *aosp/dev*, 113 so we cant really use repo start. 114 115 git branch -D working_dev 116 git checkout -b working_dev aosp/dev 117 118 2. **OPTIONAL FIXUPS**. 119 These aren't really necessary if you remember to always keep *aosp/dev* and 120 *aosp/master* synchronized otherwise, but very often someone will forget to 121 merge back a change. 122 123 1. Grab the squashed commit that went into *aosp/master* and mark it 124 committed to *aosp/dev* too. 125 126 **Note**: If there were changes to *aosp/master* before the squashed 127 commit, grab those changes (using step 2), before applying this step, 128 and finally repeat step 2 for changes after the squashed commit. 129 130 git branch -D clean_master 131 git checkout -b clean_master <SHA_FOR_SQUASH> 132 git checkout working_dev 133 git merge -s ours clean_master 134 git push aosp refs/heads/working_dev:refs/heads/dev 135 git branch -D clean_master 136 137 2. Grab all outstanding changes that went into *aosp/master* and put them 138 into *aosp/dev* too. 139 140 git branch -D clean_master 141 git checkout -b clean_master aosp/master 142 git checkout working_dev 143 git merge clean_master 144 git push aosp refs/heads/working_dev:refs/heads/dev 145 git branch -D clean_master 146 147 3. Merge the upstream branch. 148 Use `git log upstream/master` to browse upstream commits and find a SHA. 149 150 git merge <upstream_sha> 151 152 4. Fix conflicts. 153 154 5. Update build rules and commit that patch on top. 155 156 6. Test everything before pushing. 157 158 7. Submit your work to *aosp/dev*. 159 160 git push aosp refs/heads/working_dev:refs/heads/dev 161 162 8. Squash your work for *aosp/master*. 163 164 repo start update_38 . 165 git merge --squash working_dev 166 git commit -a 167 repo upload . 168 169 9. Test everything before submitting the patch from the previous step. 170 171 10. Grab the squashed commit and replay it in *aosp/dev*. 172 173 repo sync . 174 git remote update 175 git branch -D clean_master 176 git checkout -b clean_master aosp/master 177 git checkout working_dev 178 179 Use `-s ours` to ensure that we skip the squashed set of changes. 180 If/when we forget this, we have to do it later. 181 182 git merge -s ours clean_master 183 git push aosp refs/heads/working_dev:refs/heads/dev 184 git branch -D clean_master 185 186 11. Clean up after our working branch. 187 188 git checkout --detach 189 git branch -D working_dev 190 191 This works better because we can keep full history in *aosp/dev*, while 192 maintaining easy reverts/commits through *aosp/master*. 193 194 Generating new prebuilts 195 ------------------------ 196 197 1. Build all of llvm/clang/compiler-rt. 198 199 cd $ANDROID_BUILD_TOP/ 200 mmma external/llvm external/clang external/compiler-rt -j48 201 202 cd $ANDROID_BUILD_TOP/prebuilts/clang/host/linux-x86/3.8 203 ./build_toolchain.sh 204 ./build_profile_rt.sh 205 206 2. Copy over our new prebuilts. 207 Note that these were built with the previous clang. 208 209 ./update.sh 210 211 3. Now we need to actually rebuild everything with the new clang. 212 This includes clang itself! 213 214 rm -rf out/ 215 ./build_toolchain.sh 216 ./build_profile_rt.sh 217 218 4. Now we need to copy over our actual self-hosted prebuilts. 219 220 ./update.sh 221 222 5. Rebuild/test everything one more time to ensure correctness. 223 There may be necessary fixups here, to handle .ll reading or other projects 224 where new warnings/errors are firing. 225 226 m -j48 checkbuild 227 228 6. Upload the changes produced in **prebuilts/clang/host**. 229 This may entail more than a simple `git commit -a`, so look at `git status` 230 before finally uploading/committing. 231 232 repo start updated_toolchain . 233 git commit -a 234 repo upload --cbr . 235 236 7. Update RenderScript prebuilts. 237 238 cd $ANDROID_BUILD_TOP/frameworks/rs 239 ./update_rs_prebuilts.sh 240 241 8. The prebuilts get copied to **prebuilts/sdk**, so we must upload the 242 relevant bits from there. 243 244 cd $ANDROID_BUILD_TOP/prebuilts/sdk 245 git commit -a 246 repo upload . 247 248 9. Submit CLs. 249 250 251 Steps/Checklist for testing: 252 ---------------------------- 253 254 1. Do a Clang-only build and ensure it is not broken: 255 256 USE_CLANG_PLATFORM_BUILD=true m -j48 257 258 2. Go to **external/llvm** and run `./android_test.sh` (no known failures 259 as of 2015-10-08). 260 3. Ensure successful build for all architectures: 32- and 64- bit ARM, x86 and 261 Mips. 262 4. Run 32- and 64- bit RenderScript CTS at least for ARM and AArch64. 263 5. Test RenderScript apps: RsTest, ImageProcessing, and finally 264 RSTest\_Compatlib in compatibility mode. 265 6. Test old APKs with rebased tools: grab the above apps from a different tree 266 (i.e. without the rebase), push them to a device with the rebased tools, and 267 test. 268 This ensures that the rebased BitcodeReader can read the output of old 269 BitcodeWriters. 270 7. Test new APKs on an old device: test freshly built APKs for 271 RSTest\_V{11,14,16}, and ImageProcessing\_2 on an old device (say Manta) and 272 ensure they pass. 273 This ensures that the rebase did not break the 2.9 and 3.2 BitcodeWriters. 274 8. Run ART host tests. 275 This was broken by a rebase once, and worth testing after every rebase. 276 277 croot && cd art && mma -j40 test-art-host 278 279 9. Run ART device tests. 280 281 croot && cd art && mma -j4 test-art-device 282 283 284 Checklist for CLs 285 ----------------- 286 287 The following projects will almost always have CLs as a part of the rebase. 288 Depending on the changes in LLVM, there might be updates to other projects as 289 well. 290 291 * External projects 292 293 * **external/clang** 294 * **external/compiler-rt** 295 * **external/llvm** 296 * **frameworks/compile/mclinker** 297 298 * RenderScript projects 299 300 * **frameworks/compile/libbcc** 301 * **frameworks/compile/slang** 302 * **frameworks/rs** 303 304 * Prebuilts 305 306 * **prebuilts/clang/host/darwin-x86/** 307 * **prebuilts/clang/host/linux-x86/** 308 * **prebuilts/clang/host/windows-x86/** 309 * **prebuilts/sdk** 310 311 * CTS tests 312 313 * **cts/tests/tests/renderscript** 314 * **cts/tests/tests/renderscriptlegacy** 315 * **cts/tests/tests/rscpp** 316 317
