1 Bionic comes with a set of 'clean' Linux kernel headers that can safely be 2 included by userland applications and libraries without fear of hideous 3 conflicts. for more information why this is needed, see the "RATIONALE" 4 section at the end of this document. 5 6 these clean headers are automatically generated by several scripts located 7 in the 'bionic/kernel/tools' directory, which process a set of original 8 and unmodified kernel headers in order to get rid of many annoying 9 declarations and constructs that usually result in compilation failure. 10 11 the 'clean headers' only contain type and macro definitions, with the 12 exception of a couple static inline functions used for performance 13 reason (e.g. optimized CPU-specific byte-swapping routines) 14 15 they can be included from C++, or when compiling code in strict ANSI mode. 16 they can be also included before or after any Bionic C library header. 17 18 the generation process works as follows: 19 20 * 'external/kernel-headers/original/' 21 contains a set of kernel headers as normally found in the 'include' 22 directory of a normal Linux kernel source tree. note that this should 23 only contain the files that are really needed by Android (use 24 'find_headers.py' to find these automatically). 25 26 * 'bionic/libc/kernel/common' 27 contains the non-arch-specific clean headers and directories 28 (e.g. linux, asm-generic and mtd) 29 30 * 'bionic/libc/kernel/arch-arm/' 31 contains the ARM-specific directory tree of clean headers. 32 33 * 'bionic/libc/kernel/arch-arm/asm' 34 contains the real ARM-specific headers 35 36 * 'bionic/libc/kernel/arch-x86' 37 'bionic/libc/kernel/arch-x86/asm' 38 similarly contains all headers and symlinks to be used on x86 39 40 * 'bionic/libc/kernel/tools' contains various Python and shell scripts used 41 to manage and re-generate the headers 42 43 the tools you can use are: 44 45 * tools/find_users.py 46 scans a list of source files or directories and prints which ones do 47 include Linux headers. 48 49 * tools/find_headers.py 50 scans a list of source files or directories and recursively finds all 51 the original kernel headers they need. 52 53 * tools/clean_header.py 54 prints the clean version of a given kernel header. with the -u option, 55 this will also update the corresponding clean header file if its 56 content has changed. you can also process more than one file with -u 57 58 * tools/update_all.py 59 automatically update all clean headers from the content of 60 'external/kernel-headers/original'. this is the script you're likely going to 61 run whenever you update the original headers. 62 63 64 HOW TO BUILD BIONIC AND OTHER PROGRAMS WITH THE CLEAN HEADERS: 65 ============================================================== 66 67 add bionic/kernel/common and bionic/kernel/arch-<yourarch> to your C 68 include path. that should be enough. Note that Bionic will not compile properly 69 if you don't. 70 71 72 HOW TO SUPPORT ANOTHER ARCHITECTURE: 73 ==================================== 74 75 see the content of tools/defaults.py, you will need to make a few updates 76 here: 77 78 - add a new item to the 'kernel_archs' list of supported architectures 79 80 - add a proper definition for 'kernel_known_<arch>_statics' with 81 relevant definitions. 82 83 - update 'kernel_known_statics' to map "<arch>" to 84 'kernel_known_<arch>_statics' 85 86 then, add the new architecture-specific headers to original/asm-<arch>. 87 (please ensure that these are really needed, e.g. with tools/find_headers.py) 88 89 finally, run tools/update_all.py 90 91 92 93 HOW TO UPDATE THE HEADERS WHEN NEEDED: 94 ====================================== 95 96 IMPORTANT IMPORTANT: 97 98 WHEN UPDATING THE HEADERS, ALWAYS CHECK THAT THE NEW CLEAN HEADERS DO 99 NOT BREAK THE KERNEL <-> USER ABI, FOR EXAMPLE BY CHANGING THE SIZE 100 OF A GIVEN TYPE. THIS TASK CANNOT BE EASILY AUTOMATED AT THE MOMENT 101 102 copy any updated kernel header into the corresponding location under 103 'bionic/kernel/original'. 104 105 for any new kernel header you want to add, first run tools/find_headers.py to be 106 sure that it is really needed by the Android sources. then add it to 107 'bionic/kernel/original' 108 109 then, run tools/update_all.py to re-run the auto-cleaning 110 111 112 113 HOW THE CLEANUP PROCESS WORKS: 114 ============================== 115 116 this section describes the action performed by the cleanup program(s) when they 117 process the original kernel headers into clean ones: 118 119 1. Optimize well-known macros (e.g. __KERNEL__, __KERNEL_STRICT_NAMES) 120 121 this pass gets rid of everything that is guarded by a well-known macro 122 definition. this means that a block like 123 124 #ifdef __KERNEL__ 125 .... 126 #endif 127 128 will be totally omitted from the output. the optimizer is smart enough to 129 handle all complex C-preprocessor conditional expression appropriately. 130 this means that, for example: 131 132 #if defined(__KERNEL__) || defined(FOO) 133 ... 134 #endif 135 136 will be transformed into: 137 138 #ifdef FOO 139 ... 140 #endif 141 142 see tools/defaults.py for the list of well-known macros used in this pass, 143 in case you need to update it in the future. 144 145 note that this also remove any reference to a kernel-specific configuration 146 macro like CONFIG_FOO from the clean headers. 147 148 149 2. remove variable and function declarations: 150 151 this pass scans non-directive text and only keeps things that look like a 152 typedef/struct/union/enum declaration. this allows to get rid of any variable 153 or function declaration that should only be used within the kernel anyway 154 (and which normally *should* be guarded in a #ifdef __KERNEL__ ... #endif 155 block, if the kernel writers were not so messy) 156 157 there are however a few exceptions: it is seldom useful to keep the definition 158 of some static inline functions performing very simple operations. a good 159 example is the optimized 32-bit byte-swap function found in 160 arch-arm/asm/byteorder.h 161 162 the list of exceptions is in tools/defaults.py in case you need to update it 163 in the future. 164 165 note that we do *not* remove macro definitions, including these macro that 166 perform a call to one of these kernel-header functions, or even define other 167 functions. we consider it safe since userland applications have no business 168 using them anyway. 169 170 171 3. whitespace cleanup: 172 173 the final pass remove any comments and empty lines from the final headers. 174 175 176 4. add a standard disclaimer: 177 178 prepended to each generated header, contains a message like 179 "do not edit directly - file was auto-generated by ...." 180 181 182 RATIONALE: 183 ========== 184 185 OVERVIEW OF THE CURRENT KERNEL HEADER MESS: 186 ------------------------------------------- 187 188 The original kernel headers are not easily usable from userland applications. 189 they contain many declarations and construct that will result in a compilation 190 failure or even worse, incorrect behaviour. for example: 191 192 - some headers try to define Posix types (e.g. size_t, ssize_t) that can 193 conflict with the corresponding definitions provided by your C library. 194 195 - some headers use constructs that cannot be compiled in ANSI C mode. 196 197 - some headers use constructs do not compile with C++ at all. 198 199 - some headers contain invalid "legacy" definitions for the benefit of old 200 C libraries (e.g. glibc5) but result in incorrect behaviour if used 201 directly. 202 203 e.g. gid_t being defined in <linux/types.h> as a 16-bit type while the 204 kernel uses 32-bit ids. this results in problems when getgroups() or 205 setgroups() are called, since they operate on gid_t arrays. 206 207 unfortunately, these headers are also the only source of some really extensive 208 constant and type definitions that are required by userland applications. 209 think any library/program that need to access ALSA, or Video4Linux, or 210 anything related to a specific device or Linux-specific system interface 211 (e.g. IOCTLS, etc...) 212 213 As a consequence, every Linux distribution provides a set of patched kernel 214 headers to be used by userland applications (which installs in 215 /usr/include/linux/, /usr/include/asm/, etc...). these are manually maintained 216 by distribution packagers, and generated either manually or with various 217 scripts. these headers are also tailored to GNU LibC and cannot be reused 218 easily by Bionic. 219 220 for a really long period, the kernel authors have stated that they don't want 221 to fix the problem, even when someone proposed a patch to start cleaning the 222 official headers. from their point of view this is purely a library author 223 problem. 224 225 fortunately, enlightnment happened, and the kernel now provides a way to 226 install a set of "user-friendly" headers that are generated from the official 227 ones by stripping the __KERNEL__ protected declarations. 228 229 unfortunately, this is not enough for Bionic because the result still contains 230 a few broken declarations that are difficult to route around. (see below for 231 a little bit of details). 232 233 we plan to be able to support these kernel-generated user-land headers in the 234 future, but the priority on this issue is very low. 235 236 237 WHAT WE DO: 238 ----------- 239 240 so we're doomed to repeat the same effort than anyone else. the big difference 241 here is that we want to automate as much as possible the generation of the 242 clean headers to easily support additional architectures in the future, 243 and keep current with upstream changes in the header definitions with the 244 least possible hassle. 245 246 of course, this is only a race to the bottom. the kernel maintainers still 247 feel free to randomly break the structure of their headers (e.g. moving the 248 location of some files) occasionally, so we'll need to keep up with that by 249 updating our build script/original headers as these cases happen. 250 251 what we do is keep a set of "original" kernel headers, and process them 252 automatically to generate a set of "clean" headers that can be used from 253 userland and the C library. 254 255 note that the "original" headers can be tweaked a little to avoid some subtle 256 issues. for example: 257 258 - when the location of various USB-related headers changes in the kernel 259 source tree, we want to keep them at the same location in our generated 260 headers (there is no reason to break the userland API for something 261 like that). 262 263 - sometimes, we prefer to take certain things out of blocks guarded by a 264 #ifdef __KERNEL__ .. #endif. for example, on recent kernels <linux/wireless.h> 265 only includes <linux/if.h> when in kernel mode. we make it available to 266 userland as well since some code out there assumes that this is the case. 267 268 - sometimes, the header is simply incorrect (e.g. it uses a type without 269 including the header that defines it before-hand) 270 271
