Home | History | Annotate | only in /bionic/libc/kernel
Up to higher level directory
NameDateSize
arch-arm/15-Nov-2011
arch-sh/15-Nov-2011
arch-x86/15-Nov-2011
common/15-Nov-2011
README.TXT15-Nov-201110.4K
tools/15-Nov-2011

README.TXT

      1 Bionic comes with a set of 'clean' Linux kernel headers that can safely be
      2 included by userland applications and libraries without fear of hideous
      3 conflicts. for more information why this is needed, see the "RATIONALE"
      4 section at the end of this document.
      5 
      6 these clean headers are automatically generated by several scripts located
      7 in the 'bionic/kernel/tools' directory, which process a set of original
      8 and unmodified kernel headers in order to get rid of many annoying
      9 declarations and constructs that usually result in compilation failure.
     10 
     11 the 'clean headers' only contain type and macro definitions, with the
     12 exception of a couple static inline functions used for performance
     13 reason (e.g. optimized CPU-specific byte-swapping routines)
     14 
     15 they can be included from C++, or when compiling code in strict ANSI mode.
     16 they can be also included before or after any Bionic C library header.
     17 
     18 the generation process works as follows:
     19 
     20   * 'bionic/kernel/original/'
     21     contains a set of kernel headers as normally found in the 'include'
     22     directory of a normal Linux kernel source tree. note that this should
     23     only contain the files that are really needed by Android (use
     24     'find_headers.py' to find these automatically).
     25 
     26   * 'bionic/kernel/common'
     27     contains the non-arch-specific clean headers and directories
     28     (e.g. linux, asm-generic and mtd)
     29 
     30   *'bionic/kernel/arch-arm/'
     31     contains the ARM-specific directory tree of clean headers.
     32 
     33   * 'bionic/kernel/arch-arm/asm'
     34     contains the real ARM-specific headers
     35 
     36   * 'bionic/kernel/arch-x86'
     37     similarly contains all headers and symlinks to be used on x86
     38 
     39   * 'bionic/kernel/tools' contains various Python and shell scripts used
     40     to manage and re-generate the headers
     41 
     42 the tools you can use are:
     43 
     44   * tools/find_users.py
     45     scans a list of source files or directories and prints which ones do
     46     include Linux headers.
     47 
     48   * tools/find_headers.py
     49     scans a list of source files or directories and recursively finds all
     50     the original kernel headers they need.
     51 
     52   * tools/clean_header.py
     53     prints the clean version of a given kernel header. with the -u option,
     54     this will also update the corresponding clean header file if its
     55     content has changed. you can also process more than one file with -u
     56 
     57   * tools/update_all.py
     58     automatically update all clean headers from the content of 
     59     'bionic/kernel/original'. this is the script you're likely going to 
     60     run whenever you update the original headers.
     61 
     62 NOTE:
     63   if ANDROID_PRODUCT_OUT is defined in your environment, both 'clean_header.py'
     64   and 'update_all.py' will automatically issue "p4 add/edit/delete" commands
     65   appropriately to reflect the changes being made.
     66 
     67   you will need to "p4 submit" manually though...
     68 
     69 
     70 HOW TO BUILD BIONIC AND OTHER PROGRAMS WITH THE CLEAN HEADERS:
     71 ==============================================================
     72 
     73 add bionic/kernel/common and bionic/kernel/arch-<yourarch> to your C
     74 include path. that should be enough. Note that Bionic will not compile properly 
     75 if you don't.
     76 
     77 
     78 HOW TO SUPPORT ANOTHER ARCHITECTURE:
     79 ====================================
     80 
     81 see the content of tools/defaults.py, you will need to make a few updates
     82 here:
     83 
     84   - add a new item to the 'kernel_archs' list of supported architectures
     85 
     86   - add a proper definition for 'kernel_known_<arch>_statics' with
     87     relevant definitions.
     88 
     89   - update 'kernel_known_statics' to map "<arch>" to
     90     'kernel_known_<arch>_statics'
     91 
     92 then, add the new architecture-specific headers to original/asm-<arch>.
     93 (please ensure that these are really needed, e.g. with tools/find_headers.py)
     94 
     95 finally, run tools/update_all.py
     96 
     97 
     98 
     99 HOW TO UPDATE THE HEADERS WHEN NEEDED:
    100 ======================================
    101 
    102 IMPORTANT IMPORTANT:
    103 
    104   WHEN UPDATING THE HEADERS, ALWAYS CHECK THAT THE NEW CLEAN HEADERS DO
    105   NOT BREAK THE KERNEL <-> USER ABI, FOR EXAMPLE BY CHANGING THE SIZE
    106   OF A GIVEN TYPE. THIS TASK CANNOT BE EASILY AUTOMATED AT THE MOMENT
    107 
    108 copy any updated kernel header into the corresponding location under
    109 'bionic/kernel/original'.
    110 
    111 for any new kernel header you want to add, first run tools/find_headers.py to be
    112 sure that it is really needed by the Android sources. then add it to
    113 'bionic/kernel/original'
    114 
    115 then, run tools/update_all.py to re-run the auto-cleaning
    116 
    117 
    118 
    119 HOW THE CLEANUP PROCESS WORKS:
    120 ==============================
    121 
    122 this section describes the action performed by the cleanup program(s) when they
    123 process the original kernel headers into clean ones:
    124 
    125 1. Optimize well-known macros (e.g. __KERNEL__, __KERNEL_STRICT_NAMES)
    126 
    127     this pass gets rid of everything that is guarded by a well-known macro
    128     definition. this means that a block like
    129 
    130        #ifdef __KERNEL__
    131        ....
    132        #endif
    133 
    134     will be totally omitted from the output. the optimizer is smart enough to
    135     handle all complex C-preprocessor conditional expression appropriately.
    136     this means that, for example:
    137 
    138        #if defined(__KERNEL__) || defined(FOO)
    139        ...
    140        #endif
    141 
    142     will be transformed into:
    143 
    144        #ifdef FOO
    145        ...
    146        #endif
    147 
    148     see tools/defaults.py for the list of well-known macros used in this pass,
    149     in case you need to update it in the future.
    150 
    151     note that this also remove any reference to a kernel-specific configuration
    152     macro like CONFIG_FOO from the clean headers.
    153 
    154 
    155 2. remove variable and function declarations:
    156 
    157   this pass scans non-directive text and only keeps things that look like a
    158   typedef/struct/union/enum declaration. this allows to get rid of any variable
    159   or function declaration that should only be used within the kernel anyway
    160   (and which normally *should* be guarded in a #ifdef __KERNEL__ ... #endif
    161   block, if the kernel writers were not so messy)
    162 
    163   there are however a few exceptions: it is seldom useful to keep the definition
    164   of some static inline functions performing very simple operations. a good
    165   example is the optimized 32-bit byte-swap function found in
    166   arch-arm/asm/byteorder.h
    167 
    168   the list of exceptions is in tools/defaults.py in case you need to update it
    169   in the future.
    170 
    171   note that we do *not* remove macro definitions, including these macro that
    172   perform a call to one of these kernel-header functions, or even define other
    173   functions. we consider it safe since userland applications have no business
    174   using them anyway.
    175 
    176 
    177 3. whitespace cleanup:
    178 
    179   the final pass remove any comments and empty lines from the final headers.
    180 
    181 
    182 4. add a standard disclaimer:
    183 
    184   prepended to each generated header, contains a message like
    185   "do not edit directly - file was auto-generated by ...."
    186 
    187 
    188 RATIONALE:
    189 ==========
    190 
    191 OVERVIEW OF THE CURRENT KERNEL HEADER MESS:
    192 -------------------------------------------
    193 
    194 The original kernel headers are not easily usable from userland applications.
    195 they contain many declarations and construct that will result in a compilation
    196 failure or even worse, incorrect behaviour. for example:
    197 
    198 - some headers try to define Posix types (e.g. size_t, ssize_t) that can
    199   conflict with the corresponding definitions provided by your C library.
    200 
    201 - some headers use constructs that cannot be compiled in ANSI C mode.
    202 
    203 - some headers use constructs do not compile with C++ at all.
    204 
    205 - some headers contain invalid "legacy" definitions for the benefit of old
    206   C libraries (e.g. glibc5) but result in incorrect behaviour if used
    207   directly.
    208 
    209   e.g. gid_t being defined in <linux/types.h> as a 16-bit type while the
    210   kernel uses 32-bit ids. this results in problems when getgroups() or
    211   setgroups() are called, since they operate on gid_t arrays.
    212 
    213 unfortunately, these headers are also the only source of some really extensive
    214 constant and type definitions that are required by userland applications.
    215 think any library/program that need to access ALSA, or Video4Linux, or
    216 anything related to a specific device or Linux-specific system interface
    217 (e.g. IOCTLS, etc...)
    218 
    219 As a consequence, every Linux distribution provides a set of patched kernel
    220 headers to be used by userland applications (which installs in
    221 /usr/include/linux/, /usr/include/asm/, etc...). these are manually maintained
    222 by distribution packagers, and generated either manually or with various
    223 scripts. these headers are also tailored to GNU LibC and cannot be reused
    224 easily by Bionic.
    225 
    226 for a really long period, the kernel authors have stated that they don't want
    227 to fix the problem, even when someone proposed a patch to start cleaning the
    228 official headers. from their point of view this is purely a library author
    229 problem.
    230 
    231 fortunately, enlightnment happened, and the kernel now provides a way to
    232 install a set of "user-friendly" headers that are generated from the official
    233 ones by stripping the __KERNEL__ protected declarations.
    234 
    235 unfortunately, this is not enough for Bionic because the result still contains
    236 a few broken declarations that are difficult to route around. (see below for
    237 a little bit of details).
    238 
    239 we plan to be able to support these kernel-generated user-land headers in the
    240 future, but the priority on this issue is very low.
    241 
    242 
    243 WHAT WE DO:
    244 -----------
    245 
    246 so we're doomed to repeat the same effort than anyone else. the big difference
    247 here is that we want to automate as much as possible the generation of the
    248 clean headers to easily support additional architectures in the future,
    249 and keep current with upstream changes in the header definitions with the
    250 least possible hassle.
    251 
    252 of course, this is only a race to the bottom. the kernel maintainers still
    253 feel free to randomly break the structure of their headers (e.g. moving the
    254 location of some files) occasionally, so we'll need to keep up with that by
    255 updating our build script/original headers as these cases happen.
    256 
    257 what we do is keep a set of "original" kernel headers, and process them
    258 automatically to generate a set of "clean" headers that can be used from
    259 userland and the C library.
    260 
    261 note that the "original" headers can be tweaked a little to avoid some subtle
    262 issues. for example:
    263 
    264 - when the location of various USB-related headers changes in the kernel
    265   source tree, we want to keep them at the same location in our generated
    266   headers (there is no reason to break the userland API for something
    267   like that).
    268 
    269 - sometimes, we prefer to take certain things out of blocks guarded by a
    270   #ifdef __KERNEL__ .. #endif. for example, on recent kernels <linux/wireless.h>
    271   only includes <linux/if.h> when in kernel mode. we make it available to
    272   userland as well since some code out there assumes that this is the case.
    273 
    274 - sometimes, the header is simply incorrect (e.g. it uses a type without
    275   including the header that defines it before-hand)
    276 
    277