1 Bionic comes with a set of 'clean' Linux kernel headers that can safely be 2 included by userland applications and libraries without fear of hideous 3 conflicts. for more information why this is needed, see the "RATIONALE" 4 section at the end of this document. 5 6 these clean headers are automatically generated by several scripts located 7 in the 'bionic/kernel/tools' directory, which process a set of original 8 and unmodified kernel headers in order to get rid of many annoying 9 declarations and constructs that usually result in compilation failure. 10 11 the 'clean headers' only contain type and macro definitions, with the 12 exception of a couple static inline functions used for performance 13 reason (e.g. optimized CPU-specific byte-swapping routines) 14 15 they can be included from C++, or when compiling code in strict ANSI mode. 16 they can be also included before or after any Bionic C library header. 17 18 the generation process works as follows: 19 20 * 'external/kernel-headers/original/' 21 contains a set of kernel headers as normally found in the 'include' 22 directory of a normal Linux kernel source tree. note that this should 23 only contain the files that are really needed by Android (use 24 'find_headers.py' to find these automatically). 25 26 * 'bionic/libc/kernel/common' 27 contains the non-arch-specific clean headers and directories 28 (e.g. linux, asm-generic and mtd) 29 30 * 'bionic/libc/kernel/arch-arm/' 31 contains the ARM-specific directory tree of clean headers. 32 33 * 'bionic/libc/kernel/arch-arm/asm' 34 contains the real ARM-specific headers 35 36 * 'bionic/libc/kernel/arch-x86' 37 'bionic/libc/kernel/arch-x86/asm' 38 similarly contains all headers and symlinks to be used on x86 39 40 * 'bionic/libc/kernel/tools' contains various Python and shell scripts used 41 to manage and re-generate the headers 42 43 the tools you can use are: 44 45 * tools/find_users.py 46 scans a list of source files or directories and prints which ones do 47 include Linux headers. 48 49 * tools/find_headers.py 50 scans a list of source files or directories and recursively finds all 51 the original kernel headers they need. 52 53 * tools/clean_header.py 54 prints the clean version of a given kernel header. with the -u option, 55 this will also update the corresponding clean header file if its 56 content has changed. you can also process more than one file with -u 57 58 * tools/update_all.py 59 automatically update all clean headers from the content of 60 'external/kernel-headers/original'. this is the script you're likely going to 61 run whenever you update the original headers. 62 63 NOTE: 64 if ANDROID_PRODUCT_OUT is defined in your environment, both 'clean_header.py' 65 and 'update_all.py' will automatically issue "p4 add/edit/delete" commands 66 appropriately to reflect the changes being made. 67 68 you will need to "p4 submit" manually though... 69 70 71 HOW TO BUILD BIONIC AND OTHER PROGRAMS WITH THE CLEAN HEADERS: 72 ============================================================== 73 74 add bionic/kernel/common and bionic/kernel/arch-<yourarch> to your C 75 include path. that should be enough. Note that Bionic will not compile properly 76 if you don't. 77 78 79 HOW TO SUPPORT ANOTHER ARCHITECTURE: 80 ==================================== 81 82 see the content of tools/defaults.py, you will need to make a few updates 83 here: 84 85 - add a new item to the 'kernel_archs' list of supported architectures 86 87 - add a proper definition for 'kernel_known_<arch>_statics' with 88 relevant definitions. 89 90 - update 'kernel_known_statics' to map "<arch>" to 91 'kernel_known_<arch>_statics' 92 93 then, add the new architecture-specific headers to original/asm-<arch>. 94 (please ensure that these are really needed, e.g. with tools/find_headers.py) 95 96 finally, run tools/update_all.py 97 98 99 100 HOW TO UPDATE THE HEADERS WHEN NEEDED: 101 ====================================== 102 103 IMPORTANT IMPORTANT: 104 105 WHEN UPDATING THE HEADERS, ALWAYS CHECK THAT THE NEW CLEAN HEADERS DO 106 NOT BREAK THE KERNEL <-> USER ABI, FOR EXAMPLE BY CHANGING THE SIZE 107 OF A GIVEN TYPE. THIS TASK CANNOT BE EASILY AUTOMATED AT THE MOMENT 108 109 copy any updated kernel header into the corresponding location under 110 'bionic/kernel/original'. 111 112 for any new kernel header you want to add, first run tools/find_headers.py to be 113 sure that it is really needed by the Android sources. then add it to 114 'bionic/kernel/original' 115 116 then, run tools/update_all.py to re-run the auto-cleaning 117 118 119 120 HOW THE CLEANUP PROCESS WORKS: 121 ============================== 122 123 this section describes the action performed by the cleanup program(s) when they 124 process the original kernel headers into clean ones: 125 126 1. Optimize well-known macros (e.g. __KERNEL__, __KERNEL_STRICT_NAMES) 127 128 this pass gets rid of everything that is guarded by a well-known macro 129 definition. this means that a block like 130 131 #ifdef __KERNEL__ 132 .... 133 #endif 134 135 will be totally omitted from the output. the optimizer is smart enough to 136 handle all complex C-preprocessor conditional expression appropriately. 137 this means that, for example: 138 139 #if defined(__KERNEL__) || defined(FOO) 140 ... 141 #endif 142 143 will be transformed into: 144 145 #ifdef FOO 146 ... 147 #endif 148 149 see tools/defaults.py for the list of well-known macros used in this pass, 150 in case you need to update it in the future. 151 152 note that this also remove any reference to a kernel-specific configuration 153 macro like CONFIG_FOO from the clean headers. 154 155 156 2. remove variable and function declarations: 157 158 this pass scans non-directive text and only keeps things that look like a 159 typedef/struct/union/enum declaration. this allows to get rid of any variable 160 or function declaration that should only be used within the kernel anyway 161 (and which normally *should* be guarded in a #ifdef __KERNEL__ ... #endif 162 block, if the kernel writers were not so messy) 163 164 there are however a few exceptions: it is seldom useful to keep the definition 165 of some static inline functions performing very simple operations. a good 166 example is the optimized 32-bit byte-swap function found in 167 arch-arm/asm/byteorder.h 168 169 the list of exceptions is in tools/defaults.py in case you need to update it 170 in the future. 171 172 note that we do *not* remove macro definitions, including these macro that 173 perform a call to one of these kernel-header functions, or even define other 174 functions. we consider it safe since userland applications have no business 175 using them anyway. 176 177 178 3. whitespace cleanup: 179 180 the final pass remove any comments and empty lines from the final headers. 181 182 183 4. add a standard disclaimer: 184 185 prepended to each generated header, contains a message like 186 "do not edit directly - file was auto-generated by ...." 187 188 189 RATIONALE: 190 ========== 191 192 OVERVIEW OF THE CURRENT KERNEL HEADER MESS: 193 ------------------------------------------- 194 195 The original kernel headers are not easily usable from userland applications. 196 they contain many declarations and construct that will result in a compilation 197 failure or even worse, incorrect behaviour. for example: 198 199 - some headers try to define Posix types (e.g. size_t, ssize_t) that can 200 conflict with the corresponding definitions provided by your C library. 201 202 - some headers use constructs that cannot be compiled in ANSI C mode. 203 204 - some headers use constructs do not compile with C++ at all. 205 206 - some headers contain invalid "legacy" definitions for the benefit of old 207 C libraries (e.g. glibc5) but result in incorrect behaviour if used 208 directly. 209 210 e.g. gid_t being defined in <linux/types.h> as a 16-bit type while the 211 kernel uses 32-bit ids. this results in problems when getgroups() or 212 setgroups() are called, since they operate on gid_t arrays. 213 214 unfortunately, these headers are also the only source of some really extensive 215 constant and type definitions that are required by userland applications. 216 think any library/program that need to access ALSA, or Video4Linux, or 217 anything related to a specific device or Linux-specific system interface 218 (e.g. IOCTLS, etc...) 219 220 As a consequence, every Linux distribution provides a set of patched kernel 221 headers to be used by userland applications (which installs in 222 /usr/include/linux/, /usr/include/asm/, etc...). these are manually maintained 223 by distribution packagers, and generated either manually or with various 224 scripts. these headers are also tailored to GNU LibC and cannot be reused 225 easily by Bionic. 226 227 for a really long period, the kernel authors have stated that they don't want 228 to fix the problem, even when someone proposed a patch to start cleaning the 229 official headers. from their point of view this is purely a library author 230 problem. 231 232 fortunately, enlightnment happened, and the kernel now provides a way to 233 install a set of "user-friendly" headers that are generated from the official 234 ones by stripping the __KERNEL__ protected declarations. 235 236 unfortunately, this is not enough for Bionic because the result still contains 237 a few broken declarations that are difficult to route around. (see below for 238 a little bit of details). 239 240 we plan to be able to support these kernel-generated user-land headers in the 241 future, but the priority on this issue is very low. 242 243 244 WHAT WE DO: 245 ----------- 246 247 so we're doomed to repeat the same effort than anyone else. the big difference 248 here is that we want to automate as much as possible the generation of the 249 clean headers to easily support additional architectures in the future, 250 and keep current with upstream changes in the header definitions with the 251 least possible hassle. 252 253 of course, this is only a race to the bottom. the kernel maintainers still 254 feel free to randomly break the structure of their headers (e.g. moving the 255 location of some files) occasionally, so we'll need to keep up with that by 256 updating our build script/original headers as these cases happen. 257 258 what we do is keep a set of "original" kernel headers, and process them 259 automatically to generate a set of "clean" headers that can be used from 260 userland and the C library. 261 262 note that the "original" headers can be tweaked a little to avoid some subtle 263 issues. for example: 264 265 - when the location of various USB-related headers changes in the kernel 266 source tree, we want to keep them at the same location in our generated 267 headers (there is no reason to break the userland API for something 268 like that). 269 270 - sometimes, we prefer to take certain things out of blocks guarded by a 271 #ifdef __KERNEL__ .. #endif. for example, on recent kernels <linux/wireless.h> 272 only includes <linux/if.h> when in kernel mode. we make it available to 273 userland as well since some code out there assumes that this is the case. 274 275 - sometimes, the header is simply incorrect (e.g. it uses a type without 276 including the header that defines it before-hand) 277 278
