1 Bionic comes with a set of 'clean' Linux kernel headers that can safely be 2 included by userland applications and libraries without fear of hideous 3 conflicts. for more information why this is needed, see the "RATIONALE" 4 section at the end of this document. 5 6 these clean headers are automatically generated by several scripts located 7 in the 'bionic/kernel/tools' directory, which process a set of original 8 and unmodified kernel headers in order to get rid of many annoying 9 declarations and constructs that usually result in compilation failure. 10 11 the 'clean headers' only contain type and macro definitions, with the 12 exception of a couple static inline functions used for performance 13 reason (e.g. optimized CPU-specific byte-swapping routines) 14 15 they can be included from C++, or when compiling code in strict ANSI mode. 16 they can be also included before or after any Bionic C library header. 17 18 the generation process works as follows: 19 20 * 'bionic/kernel/original/' 21 contains a set of kernel headers as normally found in the 'include' 22 directory of a normal Linux kernel source tree. note that this should 23 only contain the files that are really needed by Android (use 24 'find_headers.py' to find these automatically). 25 26 * 'bionic/kernel/common' 27 contains the non-arch-specific clean headers and directories 28 (e.g. linux, asm-generic and mtd) 29 30 *'bionic/kernel/arch-arm/' 31 contains the ARM-specific directory tree of clean headers. 32 33 * 'bionic/kernel/arch-arm/asm' 34 contains the real ARM-specific headers 35 36 * 'bionic/kernel/arch-x86' 37 similarly contains all headers and symlinks to be used on x86 38 39 * 'bionic/kernel/tools' contains various Python and shell scripts used 40 to manage and re-generate the headers 41 42 the tools you can use are: 43 44 * tools/find_users.py 45 scans a list of source files or directories and prints which ones do 46 include Linux headers. 47 48 * tools/find_headers.py 49 scans a list of source files or directories and recursively finds all 50 the original kernel headers they need. 51 52 * tools/clean_header.py 53 prints the clean version of a given kernel header. with the -u option, 54 this will also update the corresponding clean header file if its 55 content has changed. you can also process more than one file with -u 56 57 * tools/update_all.py 58 automatically update all clean headers from the content of 59 'bionic/kernel/original'. this is the script you're likely going to 60 run whenever you update the original headers. 61 62 NOTE: 63 if ANDROID_PRODUCT_OUT is defined in your environment, both 'clean_header.py' 64 and 'update_all.py' will automatically issue "p4 add/edit/delete" commands 65 appropriately to reflect the changes being made. 66 67 you will need to "p4 submit" manually though... 68 69 70 HOW TO BUILD BIONIC AND OTHER PROGRAMS WITH THE CLEAN HEADERS: 71 ============================================================== 72 73 add bionic/kernel/common and bionic/kernel/arch-<yourarch> to your C 74 include path. that should be enough. Note that Bionic will not compile properly 75 if you don't. 76 77 78 HOW TO SUPPORT ANOTHER ARCHITECTURE: 79 ==================================== 80 81 see the content of tools/defaults.py, you will need to make a few updates 82 here: 83 84 - add a new item to the 'kernel_archs' list of supported architectures 85 86 - add a proper definition for 'kernel_known_<arch>_statics' with 87 relevant definitions. 88 89 - update 'kernel_known_statics' to map "<arch>" to 90 'kernel_known_<arch>_statics' 91 92 then, add the new architecture-specific headers to original/asm-<arch>. 93 (please ensure that these are really needed, e.g. with tools/find_headers.py) 94 95 finally, run tools/update_all.py 96 97 98 99 HOW TO UPDATE THE HEADERS WHEN NEEDED: 100 ====================================== 101 102 IMPORTANT IMPORTANT: 103 104 WHEN UPDATING THE HEADERS, ALWAYS CHECK THAT THE NEW CLEAN HEADERS DO 105 NOT BREAK THE KERNEL <-> USER ABI, FOR EXAMPLE BY CHANGING THE SIZE 106 OF A GIVEN TYPE. THIS TASK CANNOT BE EASILY AUTOMATED AT THE MOMENT 107 108 copy any updated kernel header into the corresponding location under 109 'bionic/kernel/original'. 110 111 for any new kernel header you want to add, first run tools/find_headers.py to be 112 sure that it is really needed by the Android sources. then add it to 113 'bionic/kernel/original' 114 115 then, run tools/update_all.py to re-run the auto-cleaning 116 117 118 119 HOW THE CLEANUP PROCESS WORKS: 120 ============================== 121 122 this section describes the action performed by the cleanup program(s) when they 123 process the original kernel headers into clean ones: 124 125 1. Optimize well-known macros (e.g. __KERNEL__, __KERNEL_STRICT_NAMES) 126 127 this pass gets rid of everything that is guarded by a well-known macro 128 definition. this means that a block like 129 130 #ifdef __KERNEL__ 131 .... 132 #endif 133 134 will be totally omitted from the output. the optimizer is smart enough to 135 handle all complex C-preprocessor conditional expression appropriately. 136 this means that, for example: 137 138 #if defined(__KERNEL__) || defined(FOO) 139 ... 140 #endif 141 142 will be transformed into: 143 144 #ifdef FOO 145 ... 146 #endif 147 148 see tools/defaults.py for the list of well-known macros used in this pass, 149 in case you need to update it in the future. 150 151 note that this also remove any reference to a kernel-specific configuration 152 macro like CONFIG_FOO from the clean headers. 153 154 155 2. remove variable and function declarations: 156 157 this pass scans non-directive text and only keeps things that look like a 158 typedef/struct/union/enum declaration. this allows to get rid of any variable 159 or function declaration that should only be used within the kernel anyway 160 (and which normally *should* be guarded in a #ifdef __KERNEL__ ... #endif 161 block, if the kernel writers were not so messy) 162 163 there are however a few exceptions: it is seldom useful to keep the definition 164 of some static inline functions performing very simple operations. a good 165 example is the optimized 32-bit byte-swap function found in 166 arch-arm/asm/byteorder.h 167 168 the list of exceptions is in tools/defaults.py in case you need to update it 169 in the future. 170 171 note that we do *not* remove macro definitions, including these macro that 172 perform a call to one of these kernel-header functions, or even define other 173 functions. we consider it safe since userland applications have no business 174 using them anyway. 175 176 177 3. whitespace cleanup: 178 179 the final pass remove any comments and empty lines from the final headers. 180 181 182 4. add a standard disclaimer: 183 184 prepended to each generated header, contains a message like 185 "do not edit directly - file was auto-generated by ...." 186 187 188 RATIONALE: 189 ========== 190 191 OVERVIEW OF THE CURRENT KERNEL HEADER MESS: 192 ------------------------------------------- 193 194 The original kernel headers are not easily usable from userland applications. 195 they contain many declarations and construct that will result in a compilation 196 failure or even worse, incorrect behaviour. for example: 197 198 - some headers try to define Posix types (e.g. size_t, ssize_t) that can 199 conflict with the corresponding definitions provided by your C library. 200 201 - some headers use constructs that cannot be compiled in ANSI C mode. 202 203 - some headers use constructs do not compile with C++ at all. 204 205 - some headers contain invalid "legacy" definitions for the benefit of old 206 C libraries (e.g. glibc5) but result in incorrect behaviour if used 207 directly. 208 209 e.g. gid_t being defined in <linux/types.h> as a 16-bit type while the 210 kernel uses 32-bit ids. this results in problems when getgroups() or 211 setgroups() are called, since they operate on gid_t arrays. 212 213 unfortunately, these headers are also the only source of some really extensive 214 constant and type definitions that are required by userland applications. 215 think any library/program that need to access ALSA, or Video4Linux, or 216 anything related to a specific device or Linux-specific system interface 217 (e.g. IOCTLS, etc...) 218 219 As a consequence, every Linux distribution provides a set of patched kernel 220 headers to be used by userland applications (which installs in 221 /usr/include/linux/, /usr/include/asm/, etc...). these are manually maintained 222 by distribution packagers, and generated either manually or with various 223 scripts. these headers are also tailored to GNU LibC and cannot be reused 224 easily by Bionic. 225 226 for a really long period, the kernel authors have stated that they don't want 227 to fix the problem, even when someone proposed a patch to start cleaning the 228 official headers. from their point of view this is purely a library author 229 problem. 230 231 fortunately, enlightnment happened, and the kernel now provides a way to 232 install a set of "user-friendly" headers that are generated from the official 233 ones by stripping the __KERNEL__ protected declarations. 234 235 unfortunately, this is not enough for Bionic because the result still contains 236 a few broken declarations that are difficult to route around. (see below for 237 a little bit of details). 238 239 we plan to be able to support these kernel-generated user-land headers in the 240 future, but the priority on this issue is very low. 241 242 243 WHAT WE DO: 244 ----------- 245 246 so we're doomed to repeat the same effort than anyone else. the big difference 247 here is that we want to automate as much as possible the generation of the 248 clean headers to easily support additional architectures in the future, 249 and keep current with upstream changes in the header definitions with the 250 least possible hassle. 251 252 of course, this is only a race to the bottom. the kernel maintainers still 253 feel free to randomly break the structure of their headers (e.g. moving the 254 location of some files) occasionally, so we'll need to keep up with that by 255 updating our build script/original headers as these cases happen. 256 257 what we do is keep a set of "original" kernel headers, and process them 258 automatically to generate a set of "clean" headers that can be used from 259 userland and the C library. 260 261 note that the "original" headers can be tweaked a little to avoid some subtle 262 issues. for example: 263 264 - when the location of various USB-related headers changes in the kernel 265 source tree, we want to keep them at the same location in our generated 266 headers (there is no reason to break the userland API for something 267 like that). 268 269 - sometimes, we prefer to take certain things out of blocks guarded by a 270 #ifdef __KERNEL__ .. #endif. for example, on recent kernels <linux/wireless.h> 271 only includes <linux/if.h> when in kernel mode. we make it available to 272 userland as well since some code out there assumes that this is the case. 273 274 - sometimes, the header is simply incorrect (e.g. it uses a type without 275 including the header that defines it before-hand) 276 277