The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 1 | Bionic comes with a set of 'clean' Linux kernel headers that can safely be |
| 2 | included by userland applications and libraries without fear of hideous |
| 3 | conflicts. for more information why this is needed, see the "RATIONALE" |
| 4 | section at the end of this document. |
| 5 | |
| 6 | these clean headers are automatically generated by several scripts located |
| 7 | in the 'bionic/kernel/tools' directory, which process a set of original |
| 8 | and unmodified kernel headers in order to get rid of many annoying |
| 9 | declarations and constructs that usually result in compilation failure. |
| 10 | |
| 11 | the 'clean headers' only contain type and macro definitions, with the |
| 12 | exception of a couple static inline functions used for performance |
| 13 | reason (e.g. optimized CPU-specific byte-swapping routines) |
| 14 | |
| 15 | they can be included from C++, or when compiling code in strict ANSI mode. |
| 16 | they can be also included before or after any Bionic C library header. |
| 17 | |
| 18 | the generation process works as follows: |
| 19 | |
Andrew Hsieh | 126601d | 2012-03-23 23:07:36 +0800 | [diff] [blame] | 20 | * 'external/kernel-headers/original/' |
The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 21 | contains a set of kernel headers as normally found in the 'include' |
| 22 | directory of a normal Linux kernel source tree. note that this should |
| 23 | only contain the files that are really needed by Android (use |
| 24 | 'find_headers.py' to find these automatically). |
| 25 | |
Andrew Hsieh | 126601d | 2012-03-23 23:07:36 +0800 | [diff] [blame] | 26 | * 'bionic/libc/kernel/common' |
The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 27 | contains the non-arch-specific clean headers and directories |
| 28 | (e.g. linux, asm-generic and mtd) |
| 29 | |
Andrew Hsieh | 126601d | 2012-03-23 23:07:36 +0800 | [diff] [blame] | 30 | * 'bionic/libc/kernel/arch-arm/' |
The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 31 | contains the ARM-specific directory tree of clean headers. |
| 32 | |
Andrew Hsieh | 126601d | 2012-03-23 23:07:36 +0800 | [diff] [blame] | 33 | * 'bionic/libc/kernel/arch-arm/asm' |
The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 34 | contains the real ARM-specific headers |
| 35 | |
Andrew Hsieh | 126601d | 2012-03-23 23:07:36 +0800 | [diff] [blame] | 36 | * 'bionic/libc/kernel/arch-x86' |
| 37 | 'bionic/libc/kernel/arch-x86/asm' |
The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 38 | similarly contains all headers and symlinks to be used on x86 |
| 39 | |
Andrew Hsieh | 126601d | 2012-03-23 23:07:36 +0800 | [diff] [blame] | 40 | * 'bionic/libc/kernel/tools' contains various Python and shell scripts used |
The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 41 | to manage and re-generate the headers |
| 42 | |
| 43 | the tools you can use are: |
| 44 | |
| 45 | * tools/find_users.py |
| 46 | scans a list of source files or directories and prints which ones do |
| 47 | include Linux headers. |
| 48 | |
| 49 | * tools/find_headers.py |
| 50 | scans a list of source files or directories and recursively finds all |
| 51 | the original kernel headers they need. |
| 52 | |
| 53 | * tools/clean_header.py |
| 54 | prints the clean version of a given kernel header. with the -u option, |
| 55 | this will also update the corresponding clean header file if its |
| 56 | content has changed. you can also process more than one file with -u |
| 57 | |
| 58 | * tools/update_all.py |
| 59 | automatically update all clean headers from the content of |
Glenn Kasten | c61f990 | 2011-12-19 11:27:50 -0800 | [diff] [blame] | 60 | 'external/kernel-headers/original'. this is the script you're likely going to |
The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 61 | run whenever you update the original headers. |
| 62 | |
The Android Open Source Project | 1dc9e47 | 2009-03-03 19:28:35 -0800 | [diff] [blame] | 63 | |
| 64 | HOW TO BUILD BIONIC AND OTHER PROGRAMS WITH THE CLEAN HEADERS: |
| 65 | ============================================================== |
| 66 | |
| 67 | add bionic/kernel/common and bionic/kernel/arch-<yourarch> to your C |
| 68 | include path. that should be enough. Note that Bionic will not compile properly |
| 69 | if you don't. |
| 70 | |
| 71 | |
| 72 | HOW TO SUPPORT ANOTHER ARCHITECTURE: |
| 73 | ==================================== |
| 74 | |
| 75 | see the content of tools/defaults.py, you will need to make a few updates |
| 76 | here: |
| 77 | |
| 78 | - add a new item to the 'kernel_archs' list of supported architectures |
| 79 | |
| 80 | - add a proper definition for 'kernel_known_<arch>_statics' with |
| 81 | relevant definitions. |
| 82 | |
| 83 | - update 'kernel_known_statics' to map "<arch>" to |
| 84 | 'kernel_known_<arch>_statics' |
| 85 | |
| 86 | then, add the new architecture-specific headers to original/asm-<arch>. |
| 87 | (please ensure that these are really needed, e.g. with tools/find_headers.py) |
| 88 | |
| 89 | finally, run tools/update_all.py |
| 90 | |
| 91 | |
| 92 | |
| 93 | HOW TO UPDATE THE HEADERS WHEN NEEDED: |
| 94 | ====================================== |
| 95 | |
| 96 | IMPORTANT IMPORTANT: |
| 97 | |
| 98 | WHEN UPDATING THE HEADERS, ALWAYS CHECK THAT THE NEW CLEAN HEADERS DO |
| 99 | NOT BREAK THE KERNEL <-> USER ABI, FOR EXAMPLE BY CHANGING THE SIZE |
| 100 | OF A GIVEN TYPE. THIS TASK CANNOT BE EASILY AUTOMATED AT THE MOMENT |
| 101 | |
| 102 | copy any updated kernel header into the corresponding location under |
| 103 | 'bionic/kernel/original'. |
| 104 | |
| 105 | for any new kernel header you want to add, first run tools/find_headers.py to be |
| 106 | sure that it is really needed by the Android sources. then add it to |
| 107 | 'bionic/kernel/original' |
| 108 | |
| 109 | then, run tools/update_all.py to re-run the auto-cleaning |
| 110 | |
| 111 | |
| 112 | |
| 113 | HOW THE CLEANUP PROCESS WORKS: |
| 114 | ============================== |
| 115 | |
| 116 | this section describes the action performed by the cleanup program(s) when they |
| 117 | process the original kernel headers into clean ones: |
| 118 | |
| 119 | 1. Optimize well-known macros (e.g. __KERNEL__, __KERNEL_STRICT_NAMES) |
| 120 | |
| 121 | this pass gets rid of everything that is guarded by a well-known macro |
| 122 | definition. this means that a block like |
| 123 | |
| 124 | #ifdef __KERNEL__ |
| 125 | .... |
| 126 | #endif |
| 127 | |
| 128 | will be totally omitted from the output. the optimizer is smart enough to |
| 129 | handle all complex C-preprocessor conditional expression appropriately. |
| 130 | this means that, for example: |
| 131 | |
| 132 | #if defined(__KERNEL__) || defined(FOO) |
| 133 | ... |
| 134 | #endif |
| 135 | |
| 136 | will be transformed into: |
| 137 | |
| 138 | #ifdef FOO |
| 139 | ... |
| 140 | #endif |
| 141 | |
| 142 | see tools/defaults.py for the list of well-known macros used in this pass, |
| 143 | in case you need to update it in the future. |
| 144 | |
| 145 | note that this also remove any reference to a kernel-specific configuration |
| 146 | macro like CONFIG_FOO from the clean headers. |
| 147 | |
| 148 | |
| 149 | 2. remove variable and function declarations: |
| 150 | |
| 151 | this pass scans non-directive text and only keeps things that look like a |
| 152 | typedef/struct/union/enum declaration. this allows to get rid of any variable |
| 153 | or function declaration that should only be used within the kernel anyway |
| 154 | (and which normally *should* be guarded in a #ifdef __KERNEL__ ... #endif |
| 155 | block, if the kernel writers were not so messy) |
| 156 | |
| 157 | there are however a few exceptions: it is seldom useful to keep the definition |
| 158 | of some static inline functions performing very simple operations. a good |
| 159 | example is the optimized 32-bit byte-swap function found in |
| 160 | arch-arm/asm/byteorder.h |
| 161 | |
| 162 | the list of exceptions is in tools/defaults.py in case you need to update it |
| 163 | in the future. |
| 164 | |
| 165 | note that we do *not* remove macro definitions, including these macro that |
| 166 | perform a call to one of these kernel-header functions, or even define other |
| 167 | functions. we consider it safe since userland applications have no business |
| 168 | using them anyway. |
| 169 | |
| 170 | |
| 171 | 3. whitespace cleanup: |
| 172 | |
| 173 | the final pass remove any comments and empty lines from the final headers. |
| 174 | |
| 175 | |
| 176 | 4. add a standard disclaimer: |
| 177 | |
| 178 | prepended to each generated header, contains a message like |
| 179 | "do not edit directly - file was auto-generated by ...." |
| 180 | |
| 181 | |
| 182 | RATIONALE: |
| 183 | ========== |
| 184 | |
| 185 | OVERVIEW OF THE CURRENT KERNEL HEADER MESS: |
| 186 | ------------------------------------------- |
| 187 | |
| 188 | The original kernel headers are not easily usable from userland applications. |
| 189 | they contain many declarations and construct that will result in a compilation |
| 190 | failure or even worse, incorrect behaviour. for example: |
| 191 | |
| 192 | - some headers try to define Posix types (e.g. size_t, ssize_t) that can |
| 193 | conflict with the corresponding definitions provided by your C library. |
| 194 | |
| 195 | - some headers use constructs that cannot be compiled in ANSI C mode. |
| 196 | |
| 197 | - some headers use constructs do not compile with C++ at all. |
| 198 | |
| 199 | - some headers contain invalid "legacy" definitions for the benefit of old |
| 200 | C libraries (e.g. glibc5) but result in incorrect behaviour if used |
| 201 | directly. |
| 202 | |
| 203 | e.g. gid_t being defined in <linux/types.h> as a 16-bit type while the |
| 204 | kernel uses 32-bit ids. this results in problems when getgroups() or |
| 205 | setgroups() are called, since they operate on gid_t arrays. |
| 206 | |
| 207 | unfortunately, these headers are also the only source of some really extensive |
| 208 | constant and type definitions that are required by userland applications. |
| 209 | think any library/program that need to access ALSA, or Video4Linux, or |
| 210 | anything related to a specific device or Linux-specific system interface |
| 211 | (e.g. IOCTLS, etc...) |
| 212 | |
| 213 | As a consequence, every Linux distribution provides a set of patched kernel |
| 214 | headers to be used by userland applications (which installs in |
| 215 | /usr/include/linux/, /usr/include/asm/, etc...). these are manually maintained |
| 216 | by distribution packagers, and generated either manually or with various |
| 217 | scripts. these headers are also tailored to GNU LibC and cannot be reused |
| 218 | easily by Bionic. |
| 219 | |
| 220 | for a really long period, the kernel authors have stated that they don't want |
| 221 | to fix the problem, even when someone proposed a patch to start cleaning the |
| 222 | official headers. from their point of view this is purely a library author |
| 223 | problem. |
| 224 | |
| 225 | fortunately, enlightnment happened, and the kernel now provides a way to |
| 226 | install a set of "user-friendly" headers that are generated from the official |
| 227 | ones by stripping the __KERNEL__ protected declarations. |
| 228 | |
| 229 | unfortunately, this is not enough for Bionic because the result still contains |
| 230 | a few broken declarations that are difficult to route around. (see below for |
| 231 | a little bit of details). |
| 232 | |
| 233 | we plan to be able to support these kernel-generated user-land headers in the |
| 234 | future, but the priority on this issue is very low. |
| 235 | |
| 236 | |
| 237 | WHAT WE DO: |
| 238 | ----------- |
| 239 | |
| 240 | so we're doomed to repeat the same effort than anyone else. the big difference |
| 241 | here is that we want to automate as much as possible the generation of the |
| 242 | clean headers to easily support additional architectures in the future, |
| 243 | and keep current with upstream changes in the header definitions with the |
| 244 | least possible hassle. |
| 245 | |
| 246 | of course, this is only a race to the bottom. the kernel maintainers still |
| 247 | feel free to randomly break the structure of their headers (e.g. moving the |
| 248 | location of some files) occasionally, so we'll need to keep up with that by |
| 249 | updating our build script/original headers as these cases happen. |
| 250 | |
| 251 | what we do is keep a set of "original" kernel headers, and process them |
| 252 | automatically to generate a set of "clean" headers that can be used from |
| 253 | userland and the C library. |
| 254 | |
| 255 | note that the "original" headers can be tweaked a little to avoid some subtle |
| 256 | issues. for example: |
| 257 | |
| 258 | - when the location of various USB-related headers changes in the kernel |
| 259 | source tree, we want to keep them at the same location in our generated |
| 260 | headers (there is no reason to break the userland API for something |
| 261 | like that). |
| 262 | |
| 263 | - sometimes, we prefer to take certain things out of blocks guarded by a |
| 264 | #ifdef __KERNEL__ .. #endif. for example, on recent kernels <linux/wireless.h> |
| 265 | only includes <linux/if.h> when in kernel mode. we make it available to |
| 266 | userland as well since some code out there assumes that this is the case. |
| 267 | |
| 268 | - sometimes, the header is simply incorrect (e.g. it uses a type without |
| 269 | including the header that defines it before-hand) |
| 270 | |