1 <html><head><title>toybox source code walkthrough</title></head> 2 <!--#include file="header.html" --> 3 4 <p><h1><a name="style" /><a href="#style">Code style</a></h1></p> 5 6 <p>The primary goal of toybox is _simple_ code. Keeping the code small is 7 second, with speed and lots of features coming in somewhere after that. 8 (For more on that, see the <a href=design.html>design</a> page.)</p> 9 10 <p>A simple implementation usually takes up fewer lines of source code, 11 meaning more code can fit on the screen at once, meaning the programmer can 12 see more of it on the screen and thus keep more if in their head at once. 13 This helps code auditing and thus reduces bugs. That said, sometimes being 14 more explicit is preferable to being clever enough to outsmart yourself: 15 don't be so terse your code is unreadable.</p> 16 17 <p>Toybox has an actual coding style guide over on 18 <a href=design.html#codestyle>the design page</a>, but in general we just 19 want the code to be consistent.</p> 20 21 <p><h1><a name="building" /><a href="#building">Building Toybox</a></h1></p> 22 23 <p>Toybox is configured using the Kconfig language pioneered by the Linux 24 kernel, and adopted by many other projects (uClibc, OpenEmbedded, etc). 25 This generates a ".config" file containing the selected options, which 26 controls which features are included when compiling toybox.</p> 27 28 <p>Each configuration option has a default value. The defaults indicate the 29 "maximum sane configuration", I.E. if the feature defaults to "n" then it 30 either isn't complete or is a special-purpose option (such as debugging 31 code) that isn't intended for general purpose use.</p> 32 33 <p>For a more compact human-editable version .config files, you can use the 34 <a href=http://landley.net/aboriginal/FAQ.html#dev_miniconfig>miniconfig</a> 35 format.</p> 36 37 <p>The standard build invocation is:</p> 38 39 <ul> 40 <li>make defconfig #(or menuconfig)</li> 41 <li>make</li> 42 <li>make install</li> 43 </ul> 44 45 <p>Type "make help" to see all available build options.</p> 46 47 <p>The file "configure" contains a number of environment variable definitions 48 which influence the build, such as specifying which compiler to use or where 49 to install the resulting binaries. This file is included by the build, but 50 accepts existing definitions of the environment variables, so it may be sourced 51 or modified by the developer before building and the definitions exported 52 to the environment will take precedence.</p> 53 54 <p>(To clarify: ".config" lists the features selected by defconfig/menuconfig, 55 I.E. "what to build", and "configure" describes the build and installation 56 environment, I.E. "how to build it".)</p> 57 58 <p><h1><a name="running"><a href="#running">Running a command</a></h1></p> 59 60 <h2>main</h2> 61 62 <p>The toybox main() function is at the end of main.c at the top level. It has 63 two possible codepaths, only one of which is configured into any given build 64 of toybox.</p> 65 66 <p>If CONFIG_SINGLE is selected, toybox is configured to contain only a single 67 command, so most of the normal setup can be skipped. In this case the 68 multiplexer isn't used, instead main() calls toy_singleinit() (also in main.c) 69 to set up global state and parse command line arguments, calls the command's 70 main function out of toy_list (in the CONFIG_SINGLE case the array has a single entry, no need to search), and if the function returns instead of exiting 71 it flushes stdout (detecting error) and returns toys.exitval.</p> 72 73 <p>When CONFIG_SINGLE is not selected, main() uses basename() to find the 74 name it was run as, shifts its argument list one to the right so it lines up 75 with where the multiplexer function expects it, and calls toybox_main(). This 76 leverages the multiplexer command's infrastructure to find and run the 77 appropriate command. (A command name starting with "toybox" will 78 recursively call toybox_main(); you can go "./toybox toybox toybox toybox ls" 79 if you want to...)</p> 80 81 <h2>toybox_main</h2> 82 83 <p>The toybox_main() function is also in main,c. It handles a possible 84 --help option ("toybox --help ls"), prints the list of available commands if no 85 arguments were provided to the multiplexer (or with full path names if any 86 other option is provided before a command name, ala "toybox --list"). 87 Otherwise it calls toy_exec() on its argument list.</p> 88 89 <p>Note that the multiplexer is the first entry in toy_list (the rest of the 90 list is sorted alphabetically to allow binary search), so toybox_main can 91 cheat and just grab the first entry to quickly set up its context without 92 searching. Since all command names go through the multiplexer at least once 93 in the non-TOYBOX_SINGLE case, this avoids a redundant search of 94 the list.</p> 95 96 <p>The toy_exec() function is also in main.c. It performs toy_find() to 97 perform a binary search on the toy_list array to look up the command's 98 entry by name and saves it in the global variable which, calls toy_init() 99 to parse command line arguments and set up global state (using which->options), 100 and calls the appropriate command's main() function (which->toy_main). On 101 return it flushes all pending ansi FILE * I/O, detects if stdout had an 102 error, and then calls xexit() (which uses toys.exitval).</p> 103 104 <p><h1><a name="infrastructure" /><a href="#infrastructure">Infrastructure</a></h1></p> 105 106 <p>The toybox source code is in following directories:</p> 107 <ul> 108 <li>The <a href="#top">top level directory</a> contains the file main.c (were 109 execution starts), the header file toys.h (included by every command), and 110 other global infrastructure.</li> 111 <li>The <a href="#lib">lib directory</a> contains common functions shared by 112 multiple commands:</li> 113 <ul> 114 <li><a href="#lib_lib">lib/lib.c</a></li> 115 <li><a href="#lib_xwrap">lib/xwrap.c</a></li> 116 <li><a href="#lib_llist">lib/llist.c</a></li> 117 <li><a href="#lib_args">lib/args.c</a></li> 118 <li><a href="#lib_dirtree">lib/dirtree.c</a></li> 119 </ul> 120 <li>The <a href="#toys">toys directory</a> contains the C files implementating 121 each command. Currently it contains five subdirectories categorizing the 122 commands: posix, lsb, other, example, and pending.</li> 123 <li>The <a href="#scripts">scripts directory</a> contains the build and 124 test infrastructure.</li> 125 <li>The <a href="#kconfig">kconfig directory</a> contains the configuration 126 infrastructure implementing menuconfig (copied from the Linux kernel).</li> 127 <li>The <a href="#generated">generated directory</a> contains intermediate 128 files generated from other parts of the source code.</li> 129 </ul> 130 131 <a name="adding" /> 132 <p><h1><a href="#adding">Adding a new command</a></h1></p> 133 <p>To add a new command to toybox, add a C file implementing that command to 134 one of the subdirectories under the toys directory. No other files need to 135 be modified; the build extracts all the information it needs (such as command 136 line arguments) from specially formatted comments and macros in the C file. 137 (See the description of the <a href="#generated">"generated" directory</a> 138 for details.)</p> 139 140 <p>Currently there are five subdirectories under "toys", one for commands 141 defined by the POSIX standard, one for commands defined by the Linux Standard 142 Base, an "other" directory for commands not covered by an obvious standard, 143 a directory of example commands (templates to use when starting new commands), 144 and a "pending" directory of commands that need further review/cleanup 145 before moving to one of the other directories (run these at your own risk, 146 cleanup patches welcome). 147 These directories are just for developer convenience sorting the commands, 148 the directories are otherwise functionally identical. To add a new category, 149 create the appropriate directory with a README file in it whose first line 150 is the description menuconfig should use for the directory.)</p> 151 152 <p>An easy way to start a new command is copy the file "toys/example/hello.c" 153 to the name of the new command, and modify this copy to implement the new 154 command (more or less by turning every instance of "hello" into the 155 name of your command, updating the command line arguments, globals, and 156 help data, and then filling out its "main" function with code that does 157 something interesting).</p> 158 159 <p>You could also start with "toys/example/skeleton.c", which provides a lot 160 more example code (showing several variants of command line option 161 parsing, how to implement multiple commands in the same file, and so on). 162 But usually it's just more stuff to delete.</p> 163 164 <p>Here's a checklist of steps to turn hello.c into another command:</p> 165 166 <ul> 167 <li><p>First "cp toys/example/hello.c toys/other/yourcommand.c" and open 168 the new file in your preferred text editor.</p> 169 <ul><li><p>Note that the 170 name of the new file is significant: it's the name of the new command you're 171 adding to toybox. The build includes all *.c files under toys/*/ whose 172 names are a case insensitive match for an enabled config symbol. So 173 toys/posix/cat.c only gets included if you have "CAT=y" in ".config".</p></li> 174 </ul></p></li> 175 176 <li><p>Change the one line comment at the top of the file (currently 177 "hello.c - A hello world program") to describe your new file.</p></li> 178 179 <li><p>Change the copyright notice to your name, email, and the current 180 year.</p></li> 181 182 <li><p>Give a URL to the relevant standards document, where applicable. 183 (Sample links to SUSv4 and LSB are provided, feel free to link to other 184 documentation or standards as appropriate.)</p></li> 185 186 <li><p>Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line. 187 The NEWTOY macro fills out this command's <a href="#toy_list">toy_list</a> 188 structure. The arguments to the NEWTOY macro are:</p> 189 190 <ol> 191 <li><p>the name used to run your command</p></li> 192 <li><p>the command line argument <a href="#lib_args">option parsing string</a> (0 if none)</p></li> 193 <li><p>a bitfield of TOYFLAG values 194 (defined in toys.h) providing additional information such as where your 195 command should be installed on a running system, whether to blank umask 196 before running, whether or not the command must run as root (and thus should 197 retain root access if installed SUID), and so on.</p></li> 198 </ol> 199 </li> 200 201 <li><p>Change the kconfig data (from "config YOURCOMMAND" to the end of the 202 comment block) to supply your command's configuration and help 203 information. The uppper case config symbols are used by menuconfig, and are 204 also what the CFG_ and USE_() macros are generated from (see [TODO]). The 205 help information here is used by menuconfig, and also by the "help" command to 206 describe your new command. (See [TODO] for details.) By convention, 207 unfinished commands default to "n" and finished commands default to "y", 208 so "make defconfig" selects all finished commands. (Note, "finished" means 209 "ready to be used", not that it'll never change again.)<p> 210 211 <p>Each help block should start with a "usage: yourcommand" line explaining 212 any command line arguments added by this config option. The "help" command 213 outputs this text, and scripts/config2help.c in the build infrastructure 214 collates these usage lines for commands with multiple configuration 215 options when producing generated/help.h.</p> 216 </li> 217 218 <li><p>Change the "#define FOR_hello" line to "#define FOR_yourcommand" right 219 before the "#include <toys.h>". (This selects the appropriate FLAG_ macros and 220 does a "#define TT this.yourcommand" so you can access the global variables 221 out of the space-saving union of structures. If you aren't using any command 222 flag bits and aren't defining a GLOBAL block, you can delete this line.)</p></li> 223 224 <li><p>Update the GLOBALS() macro to contain your command's global 225 variables. If your command has no global variables, delete this macro.</p> 226 227 <p>Variables in the GLOBALS() block are are stored in a space saving 228 <a href="#toy_union">union of structures</a> format, which may be accessed 229 using the TT macro as if TT were a global structure (so TT.membername). 230 If you specified two-character command line arguments in 231 NEWTOY(), the first few global variables will be initialized by the automatic 232 argument parsing logic, and the type and order of these variables must 233 correspond to the arguments specified in NEWTOY(). 234 (See <a href="#lib_args">lib/args.c</a> for details.)</p></li> 235 236 <li><p>Rename hello_main() to yourcommand_main(). This is the main() function 237 where execution of your command starts. Your command line options are 238 already sorted into this.optflags, this.optargs, this.optc, and the GLOBALS() 239 as appropriate by the time this function is called. (See 240 <a href="#lib_args">get_optflags()</a> for details.)</p></li> 241 242 <li><p>Switch on TOYBOX_DEBUG in menuconfig (toybox global settings menu) 243 the first time you build and run your new command. If anything is wrong 244 with your option string, that will give you error messages.</p> 245 246 <p>Otherwise it'll just segfault without 247 explanation when it falls off the end because it didn't find a matching 248 end parantheses for a longopt, or you put a nonexistent option in a square 249 bracket grouping... Since these kind of errors can only be caused by a 250 developer, not by end users, we don't normally want runtime checks for 251 them. Once you're happy with your option string, you can switch TOYBOX_DEBUG 252 back off.</p></li> 253 </ul> 254 255 <a name="headers" /><h2><a href="#headers">Headers.</a></h2> 256 257 <p>Commands generally don't have their own headers. If it's common code 258 it can live in lib/, if it isn't put it in the command's .c file. (The line 259 between implementing multiple commands in a C file via OLDTOY() to share 260 infrastructure and moving that shared infrastructure to lib/ is a judgement 261 call. Try to figure out which is simplest.)</p> 262 263 <p>The top level toys.h should #include all the standard (posix) headers 264 that any command uses. (Partly this is friendly to ccache and partly this 265 makes the command implementations shorter.) Individual commands should only 266 need to include nonstandard headers that might prevent that command from 267 building in some context we'd care about (and thus requiring that command to 268 be disabled to avoid a build break).</p> 269 270 <p>Target-specific stuff (differences between compiler versions, libc versions, 271 or operating systems) should be confined to lib/portability.h and 272 lib/portability.c. (There's even some minimal compile-time environment probing 273 that writes data to generated/portability.h, see scripts/genconfig.sh.)</p> 274 275 <p>Only include linux/*.h headers from individual commands (not from other 276 headers), and only if you really need to. Data that varies per architecture 277 is a good reason to include a header. If you just need a couple constants 278 that haven't changed since the 1990's, it's ok to #define them yourself or 279 just use the constant inline with a comment explaining what it is. (A 280 #define that's only used once isn't really helping.)</p> 281 282 <p><a name="top" /><h1><a href="#top">Top level directory.</a></h1></p> 283 284 <p>This directory contains global infrastructure.</p> 285 286 <h3>toys.h</h3> 287 <p>Each command #includes "toys.h" as part of its standard prolog. It 288 may "#define FOR_commandname" before doing so to get some extra entries 289 specific to this command.</p> 290 291 <p>This file sucks in most of the commonly used standard #includes, so 292 individual files can just #include "toys.h" and not have to worry about 293 stdargs.h and so on. Individual commands still need to #include 294 special-purpose headers that may not be present on all systems (and thus would 295 prevent toybox from building that command on such a system with that command 296 enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab 297 support (mntent.h and sys/mount.h), and so on.</p> 298 299 <p>The toys.h header also defines structures for most of the global variables 300 provided to each command by toybox_main(). These are described in 301 detail in the description for main.c, where they are initialized.</p> 302 303 <p>The global variables are grouped into structures (and a union) for space 304 savings, to more easily track the amount of memory consumed by them, 305 so that they may be automatically cleared/initialized as needed, and so 306 that access to global variables is more easily distinguished from access to 307 local variables.</p> 308 309 <h3>main.c</h3> 310 <p>Contains the main() function where execution starts, plus 311 common infrastructure to initialize global variables and select which command 312 to run. The "toybox" multiplexer command also lives here. (This is the 313 only command defined outside of the toys directory.)</p> 314 315 <p>Execution starts in main() which trims any path off of the first command 316 name and calls toybox_main(), which calls toy_exec(), which calls toy_find() 317 and toy_init() before calling the appropriate command's function from 318 toy_list[] (via toys.which->toy_main()). 319 If the command is "toybox", execution recurses into toybox_main(), otherwise 320 the call goes to the appropriate commandname_main() from a C file in the toys 321 directory.</p> 322 323 <p>The following global variables are defined in main.c:</p> 324 <ul> 325 <a name="toy_list" /> 326 <li><p><b>struct toy_list toy_list[]</b> - array describing all the 327 commands currently configured into toybox. The first entry (toy_list[0]) is 328 for the "toybox" multiplexer command, which runs all the other built-in commands 329 without symlinks by using its first argument as the name of the command to 330 run and the rest as that command's argument list (ala "./toybox echo hello"). 331 The remaining entries are the commands in alphabetical order (for efficient 332 binary search).</p> 333 334 <p>This is a read-only array initialized at compile time by 335 defining macros and #including generated/newtoys.h.</p> 336 337 <p>Members of struct toy_list (defined in "toys.h") include:</p> 338 <ul> 339 <li><p>char *<b>name</b> - the name of this command.</p></li> 340 <li><p>void (*<b>toy_main</b>)(void) - function pointer to run this 341 command.</p></li> 342 <li><p>char *<b>options</b> - command line option string (used by 343 get_optflags() in lib/args.c to intialize toys.optflags, toys.optargs, and 344 entries in the toy's GLOBALS struct). When this is NULL, no option 345 parsing is done before calling toy_main().</p></li> 346 <li><p>int <b>flags</b> - Behavior flags for this command. The following flags are currently understood:</p> 347 348 <ul> 349 <li><b>TOYFLAG_USR</b> - Install this command under /usr</li> 350 <li><b>TOYFLAG_BIN</b> - Install this command under /bin</li> 351 <li><b>TOYFLAG_SBIN</b> - Install this command under /sbin</li> 352 <li><b>TOYFLAG_NOFORK</b> - This command can be used as a shell builtin.</li> 353 <li><b>TOYFLAG_UMASK</b> - Call umask(0) before running this command.</li> 354 <li><b>TOYFLAG_STAYROOT</b> - Don't drop permissions for this command if toybox is installed SUID root.</li> 355 <li><b>TOYFLAG_NEEDROOT</b> - This command cannot function unless run with root access.</li> 356 </ul> 357 <br> 358 359 <p>These flags are combined with | (or). For example, to install a command 360 in /usr/bin, or together TOYFLAG_USR|TOYFLAG_BIN.</p> 361 </ul> 362 </li> 363 364 <li><p><b>struct toy_context toys</b> - global structure containing information 365 common to all commands, initializd by toy_init() and defined in "toys.h". 366 Members of this structure include:</p> 367 <ul> 368 <li><p>struct toy_list *<b>which</b> - a pointer to this command's toy_list 369 structure. Mostly used to grab the name of the running command 370 (toys->which.name).</p> 371 </li> 372 <li><p>int <b>exitval</b> - Exit value of this command. Defaults to zero. The 373 error_exit() functions will return 1 if this is zero, otherwise they'll 374 return this value.</p></li> 375 <li><p>char **<b>argv</b> - "raw" command line options, I.E. the original 376 unmodified string array passed in to main(). Note that modifying this changes 377 "ps" output, and is not recommended. This array is null terminated; a NULL 378 entry indicates the end of the array.</p> 379 <p>Most commands don't use this field, instead the use optargs, optflags, 380 and the fields in the GLOBALS struct initialized by get_optflags().</p> 381 </li> 382 <li><p>unsigned <b>optflags</b> - Command line option flags, set by 383 <a href="#lib_args">get_optflags()</a>. Indicates which of the command line options listed in 384 toys->which.options occurred this time.</p> 385 386 <p>The rightmost command line argument listed in toys->which.options sets bit 387 1, the next one sets bit 2, and so on. This means the bits are set in the same 388 order the binary digits would be listed if typed out as a string. For example, 389 the option string "abcd" would parse the command line "-c" to set optflags to 2, 390 "-a" would set optflags to 8, and "-bd" would set optflags to 6 (4|2).</p> 391 392 <p>Only letters are relevant to optflags. In the string "a*b:c#d", d=1, c=2, 393 b=4, a=8. Punctuation after a letter initializes global variables at the 394 start of the GLOBALS() block (see <a href="#toy_union">union toy_union this</a> 395 for details).</p> 396 397 <p>The build infrastructure creates FLAG_ macros for each option letter, 398 corresponding to the bit position, so you can check (toys.optflags & FLAG_x) 399 to see if a flag was specified. (The correct set of FLAG_ macros is selected 400 by defining FOR_mycommand before #including toys.h. The macros live in 401 toys/globals.h which is generated by scripts/make.sh.)</p> 402 403 <p>For more information on option parsing, see <a href="#lib_args">get_optflags()</a>.</p> 404 405 </li> 406 <li><p>char **<b>optargs</b> - Null terminated array of arguments left over 407 after get_optflags() removed all the ones it understood. Note: optarg[0] is 408 the first argument, not the command name. Use toys.which->name for the command 409 name.</p></li> 410 <li><p>int <b>optc</b> - Optarg count, equivalent to argc but for 411 optargs[].<p></li> 412 </ul> 413 414 <a name="toy_union" /> 415 <li><p><b>union toy_union this</b> - Union of structures containing each 416 command's global variables.</p> 417 418 <p>Global variables are useful: they reduce the overhead of passing extra 419 command line arguments between functions, they conveniently start prezeroed to 420 save initialization costs, and the command line argument parsing infrastructure 421 can also initialize global variables with its results.</p> 422 423 <p>But since each toybox process can only run one command at a time, allocating 424 space for global variables belonging to other commands you aren't currently 425 running would be wasteful.</p> 426 427 <p>Toybox handles this by encapsulating each command's global variables in 428 a structure, and declaring a union of those structures with a single global 429 instance (called "this"). The GLOBALS() macro contains the global 430 variables that should go in the current command's global structure. Each 431 variable can then be accessed as "this.commandname.varname". 432 If you #defined FOR_commandname before including toys.h, the macro TT is 433 #defined to this.commandname so the variable can then be accessed as 434 "TT.variable". See toys/hello.c for an example.</p> 435 436 <p>A command that needs global variables should declare a structure to 437 contain them all, and add that structure to this union. A command should never 438 declare global variables outside of this, because such global variables would 439 allocate memory when running other commands that don't use those global 440 variables.</p> 441 442 <p>The first few fields of this structure can be intialized by <a href="#lib_args">get_optargs()</a>, 443 as specified by the options field off this command's toy_list entry. See 444 the get_optargs() description in lib/args.c for details.</p> 445 </li> 446 447 <li><b>char toybuf[4096]</b> - a common scratch space buffer so 448 commands don't need to allocate their own. Any command is free to use this, 449 and it should never be directly referenced by functions in lib/ (although 450 commands are free to pass toybuf in to a library function as an argument).</li> 451 </ul> 452 453 <p>The following functions are defined in main.c:</p> 454 <ul> 455 <li><p>struct toy_list *<b>toy_find</b>(char *name) - Return the toy_list 456 structure for this command name, or NULL if not found.</p></li> 457 <li><p>void <b>toy_init</b>(struct toy_list *which, char *argv[]) - fill out 458 the global toys structure, calling get_optargs() if necessary.</p></li> 459 <li><p>void <b>toy_exec</b>(char *argv[]) - Run a built-in command with 460 arguments.</p> 461 <p>Calls toy_find() on argv[0] (which must be just a command name 462 without path). Returns if it can't find this command, otherwise calls 463 toy_init(), toys->which.toy_main(), and exit() instead of returning.</p> 464 465 <p>Use the library function xexec() to fall back to external executables 466 in $PATH if toy_exec() can't find a built-in command. Note that toy_exec() 467 does not strip paths before searching for a command, so "./command" will 468 never match an internal command.</li> 469 470 <li><p>void <b>toybox_main</b>(void) - the main function for the multiplexer 471 command (I.E. "toybox"). Given a command name as its first argument, calls 472 toy_exec() on its arguments. With no arguments, it lists available commands. 473 If the first argument starts with "-" it lists each command with its default 474 install path prepended.</p></li> 475 476 </ul> 477 478 <h3>Config.in</h3> 479 480 <p>Top level configuration file in a stylized variant of 481 <a href=http://kernel.org/doc/Documentation/kbuild/kconfig-language.txt>kconfig</a> format. Includes generated/Config.in.</p> 482 483 <p>These files are directly used by "make menuconfig" to select which commands 484 to build into toybox (thus generating a .config file), and by 485 scripts/config2help.py to create generated/help.h.</p> 486 487 <a name="generated" /> 488 <h1><a href="#generated">Temporary files:</a></h1> 489 490 <p>There is one temporary file in the top level source directory:</p> 491 <ul> 492 <li><p><b>.config</b> - Configuration file generated by kconfig, indicating 493 which commands (and options to commands) are currently enabled. Used 494 to make generated/config.h and determine which toys/*/*.c files to build.</p> 495 496 <p>You can create a human readable "miniconfig" version of this file using 497 <a href=http://landley.net/aboriginal/new_platform.html#miniconfig>these 498 instructions</a>.</p> 499 </li> 500 </ul> 501 502 <p><h2>Directory generated/</h2></p> 503 504 <p>The remaining temporary files live in the "generated/" directory, 505 which is for files generated at build time from other source files.</p> 506 507 <ul> 508 <li><p><b>generated/Config.in</b> - Kconfig entries for each command, included 509 from the top level Config.in. The help text here is used to generate 510 help.h.</p> 511 512 <p>Each command has a configuration entry with an upper case version of 513 the command name. Options to commands start with the command 514 name followed by an underscore and the option name. Global options are attached 515 to the "toybox" command, and thus use the prefix "TOYBOX_". This organization 516 is used by scripts/cfg2files to select which toys/*/*.c files to compile for a 517 given .config.</p> 518 </li> 519 520 <li><p><b>generated/config.h</b> - list of CFG_SYMBOL and USE_SYMBOL() macros, 521 generated from .config by a sed invocation in scripts/make.sh.</p> 522 523 <p>CFG_SYMBOL is a comple time constant set to 1 for enabled symbols and 0 for 524 disabled symbols. This allows the use of normal if() statements to remove 525 code at compile time via the optimizer's dead code elimination (which removes 526 from the binary any code that cannot be reached). This saves space without 527 cluttering the code with #ifdefs or leading to configuration dependent build 528 breaks. (See the 1992 Usenix paper 529 <a href=http://doc.cat-v.org/henry_spencer/ifdef_considered_harmful.pdf>#ifdef 530 Considered Harmful</a> for more information.)</p> 531 532 <p>When you can't entirely avoid an #ifdef, the USE_SYMBOL(code) macro 533 provides a less intrusive alternative, evaluating to the code in parentheses 534 when the symbol is enabled, and nothing when the symbol is disabled. This 535 is most commonly used around NEWTOY() declarations (so only the enabled 536 commands show up in toy_list), and in option strings. This can also be used 537 for things like varargs or structure members which can't always be 538 eliminated by a simple test on CFG_SYMBOL. Remember, unlike CFG_SYMBOL 539 this is really just a variant of #ifdef, and can still result in configuration 540 dependent build breaks. Use with caution.</p> 541 </li> 542 543 <li><p><b>generated/flags.h</b> - FLAG_? macros indicating which command 544 line options were seen. The option parsing in lib/args.c sets bits in 545 toys.optflags, which can be tested by anding with the appropriate FLAG_ 546 macro. (Bare longopts, which have no corresponding short option, will 547 have the longopt name after FLAG_. All others use the single letter short 548 option.)</p> 549 550 <p>To get the appropriate macros for your command, #define FOR_commandname 551 before #including toys.h. To switch macro sets (because you have an OLDTOY() 552 with different options in the same .c file), #define CLEANUP_oldcommand 553 and also #define FOR_newcommand, then #include "generated/flags.h" to switch. 554 </p> 555 </li> 556 557 <li><p><b>generated/globals.h</b> - 558 Declares structures to hold the contents of each command's GLOBALS(), 559 and combines them into "global_union this". (Yes, the name was 560 chosen to piss off C++ developers who think that C 561 is merely a subset of C++, not a language in its own right.)</p> 562 563 <p>The union reuses the same memory for each command's global struct: 564 since only one command's globals are in use at any given time, collapsing 565 them together saves space. The headers #define TT to the appropriate 566 "this.commandname", so you can refer to the current command's global 567 variables out of "this" as TT.variablename.</p> 568 569 <p>The globals start zeroed, and the first few are filled out by the 570 lib/args.c argument parsing code called from main.c.</p> 571 </li> 572 573 <li><p><b>toys/help.h</b> - Help strings for use by the "help" command and 574 --help options. This file #defines a help_symbolname string for each 575 symbolname, but only the symbolnames matching command names get used 576 by show_help() in lib/help.c to display help for commands.</p> 577 578 <p>This file is created by scripts/make.sh, which compiles scripts/config2help.c 579 into the binary generated/config2help, and then runs it against the top 580 level .config and Config.in files to extract the help text from each config 581 entry and collate together dependent options.</p> 582 583 <p>This file contains help text for all commands, regardless of current 584 configuration, but only the ones currently enabled in the .config file 585 wind up in the help_data[] array, and only the enabled dependent options 586 have their help text added to the command they depend on.</p> 587 </li> 588 589 <li><p><b>generated/newtoys.h</b> - 590 All the NEWTOY() and OLDTOY() macros from toys/*/*.c. The "toybox" multiplexer 591 is the first entry, the rest are in alphabetical order. Each line should be 592 inside an appropriate USE_ macro, so code that #includes this file only sees 593 the currently enabled commands.</p> 594 595 <p>By #definining NEWTOY() to various things before #including this file, 596 it may be used to create function prototypes (in toys.h), initialize the 597 help_data array (in lib/help.c), initialize the toy_list array (in main.c, 598 the alphabetical order lets toy_find() do a binary search, the exception to 599 the alphabetical order lets it use the multiplexer without searching), and so 600 on. (It's even used to initialize the NEED_OPTIONS macro, which produces a 1 601 or 0 for each command using command line option parsing, which is ORed together 602 to allow compile-time dead code elimination to remove the whole of 603 lib/args.c if nothing currently enabled is using it.)<p> 604 605 <p>Each NEWTOY and OLDTOY macro contains the command name, command line 606 option string (telling lib/args.c how to parse command line options for 607 this command), recommended install location, and miscelaneous data such 608 as whether this command should retain root permissions if installed suid.</p> 609 </li> 610 611 <li><p><b>toys/oldtoys.h</b> - Macros with the command line option parsing 612 string for each NEWTOY. This allows an OLDTOY that's just an alias for an 613 existing command to refer to the existing option string instead of 614 having to repeat it.</p> 615 </li> 616 </ul> 617 618 <a name="lib"> 619 <h2>Directory lib/</h2> 620 621 <p>TODO: document lots more here.</p> 622 623 <p>lib: getmountlist(), error_msg/error_exit, xmalloc(), 624 strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(), 625 itoa().</p> 626 627 628 629 <a name="lib_xwrap"><h3>lib/xwrap.c</h3> 630 631 <p>Functions prefixed with the letter x call perror_exit() when they hit 632 errors, to eliminate common error checking. This prints an error message 633 and the strerror() string for the errno encountered.</p> 634 635 <p>You can intercept this exit by assigning a setjmp/longjmp buffer to 636 toys.rebound (set it back to zero to restore the default behavior). 637 If you do this, cleaning up resource leaks is your problem.</p> 638 639 <ul> 640 <li><b>void xstrncpy(char *dest, char *src, size_t size)</b></li> 641 <li><b>void xexit(void)</b></li> 642 <li><b>void *xmalloc(size_t size)</b></li> 643 <li><b>void *xzalloc(size_t size)</b></li> 644 <li><b>void *xrealloc(void *ptr, size_t size)</b></li> 645 <li><b>char *xstrndup(char *s, size_t n)</b></li> 646 <li><b>char *xstrdup(char *s)</b></li> 647 <li><b>char *xmprintf(char *format, ...)</b></li> 648 <li><b>void xprintf(char *format, ...)</b></li> 649 <li><b>void xputs(char *s)</b></li> 650 <li><b>void xputc(char c)</b></li> 651 <li><b>void xflush(void)</b></li> 652 <li><b>pid_t xfork(void)</b></li> 653 <li><b>void xexec_optargs(int skip)</b></li> 654 <li><b>void xexec(char **argv)</b></li> 655 <li><b>pid_t xpopen(char **argv, int *pipes)</b></li> 656 <li><b>int xpclose(pid_t pid, int *pipes)</b></li> 657 <li><b>void xaccess(char *path, int flags)</b></li> 658 <li><b>void xunlink(char *path)</b></li> 659 <li><p><b>int xcreate(char *path, int flags, int mode)<br /> 660 int xopen(char *path, int flags)</b></p> 661 662 <p>The xopen() and xcreate() functions open an existing file (exiting if 663 it's not there) and create a new file (exiting if it can't).</p> 664 665 <p>They default to O_CLOEXEC so the filehandles aren't passed on to child 666 processes. Feed in O_CLOEXEC to disable this.</p> 667 </li> 668 <li><p><b>void xclose(int fd)</b></p> 669 670 <p>Because NFS is broken, and won't necessarily perform the requested 671 operation (and report the error) until you close the file. Of course, this 672 being NFS, it's not guaranteed to report the error there either, but it 673 _can_.</p> 674 675 <p>Nothing else ever reports an error on close, everywhere else it's just a 676 VFS operation freeing some resources. NFS is _special_, in a way that 677 other network filesystems like smbfs and v9fs aren't..</p> 678 </li> 679 <li><b>int xdup(int fd)</b></li> 680 <li><p><b>size_t xread(int fd, void *buf, size_t len)</b></p> 681 682 <p>Can return 0, but not -1.</p> 683 </li> 684 <li><p><b>void xreadall(int fd, void *buf, size_t len)</b></p> 685 686 <p>Reads the entire len-sized buffer, retrying to complete short 687 reads. Exits if it can't get enough data.</p></li> 688 689 <li><p><b>void xwrite(int fd, void *buf, size_t len)</b></p> 690 691 <p>Retries short writes, exits if can't write the entire buffer.</p></li> 692 693 <li><b>off_t xlseek(int fd, off_t offset, int whence)</b></li> 694 <li><b>char *xgetcwd(void)</b></li> 695 <li><b>void xstat(char *path, struct stat *st)</b></li> 696 <li><p><b>char *xabspath(char *path, int exact) </b></p> 697 698 <p>After several years of 699 <a href=http://landley.net/notes-2007.html#18-06-2007>wrestling</a> 700 <a href=http://landley.net/notes-2008.html#19-01-2008>with</a> realpath(), 701 I broke down and <a href=http://landley.net/notes-2012.html#20-11-2012>wrote 702 my own</a> implementation that doesn't use the one in libc. As I explained: 703 704 <blockquote><p>If the path ends with a broken link, 705 readlink -f should show where the link points to, not where the broken link 706 lives. (The point of readlink -f is "if I write here, where would it attempt 707 to create a file".) The problem is, realpath() returns NULL for a path ending 708 with a broken link, and I can't beat different behavior out of code locked 709 away in libc.</p></blockquote> 710 711 <p> 712 </li> 713 <li><b>void xchdir(char *path)</b></li> 714 <li><b>void xchroot(char *path)</b></li> 715 716 <li><p><b>struct passwd *xgetpwuid(uid_t uid)<br /> 717 struct group *xgetgrgid(gid_t gid)<br /> 718 struct passwd *xgetpwnam(char *name)</b></p> 719 720 <p></p> 721 </li> 722 723 724 725 <li><b>void xsetuser(struct passwd *pwd)</b></li> 726 <li><b>char *xreadlink(char *name)</b></li> 727 <li><b>char *xreadfile(char *name, char *buf, off_t len)</b></li> 728 <li><b>int xioctl(int fd, int request, void *data)</b></li> 729 <li><b>void xpidfile(char *name)</b></li> 730 <li><b>void xsendfile(int in, int out)</b></li> 731 <li><b>long xparsetime(char *arg, long units, long *fraction)</b></li> 732 <li><b>void xregcomp(regex_t *preg, char *regex, int cflags)</b></li> 733 </ul> 734 735 <a name="lib_lib"><h3>lib/lib.c</h3> 736 <p>Eight gazillion common functions:</p> 737 738 <ul> 739 <li><b>void verror_msg(char *msg, int err, va_list va)</b></li> 740 <li><b>void error_msg(char *msg, ...)</b></li> 741 <li><b>void perror_msg(char *msg, ...)</b></li> 742 <li><b>void error_exit(char *msg, ...)</b></li> 743 <li><b>void perror_exit(char *msg, ...)</b></li> 744 <li><b>ssize_t readall(int fd, void *buf, size_t len)</b></li> 745 <li><b>ssize_t writeall(int fd, void *buf, size_t len)</b></li> 746 <li><b>off_t lskip(int fd, off_t offset)</b></li> 747 <li><b>int mkpathat(int atfd, char *dir, mode_t lastmode, int flags)</b></li> 748 <li><b>struct string_list **splitpath(char *path, struct string_list **list)</b></li> 749 <li><b>struct string_list *find_in_path(char *path, char *filename)</b></li> 750 <li><b>long atolx(char *numstr)</b></li> 751 <li><b>long atolx_range(char *numstr, long low, long high)</b></li> 752 <li><b>int numlen(long l)</b></li> 753 <li><b>int stridx(char *haystack, char needle)</b></li> 754 <li><b>int strstart(char **a, char *b)</b></li> 755 <li><b>off_t fdlength(int fd)</b></li> 756 <li><b>char *readfile(char *name, char *ibuf, off_t len)</b></li> 757 <li><b>void msleep(long miliseconds)</b></li> 758 <li><b>int64_t peek_le(void *ptr, unsigned size)</b></li> 759 <li><b>int64_t peek_be(void *ptr, unsigned size)</b></li> 760 <li><b>int64_t peek(void *ptr, unsigned size)</b></li> 761 <li><b>void poke(void *ptr, uint64_t val, int size)</b></li> 762 <li><b>void loopfiles_rw(char **argv, int flags, int permissions, int failok,</b></li> 763 <li><b>void loopfiles(char **argv, void (*function)(int fd, char *name))</b></li> 764 <li><b>char *get_rawline(int fd, long *plen, char end)</b></li> 765 <li><b>char *get_line(int fd)</b></li> 766 <li><b>int wfchmodat(int fd, char *name, mode_t mode)</b></li> 767 <li><b>static void tempfile_handler(int i)</b></li> 768 <li><b>int copy_tempfile(int fdin, char *name, char **tempname)</b></li> 769 <li><b>void delete_tempfile(int fdin, int fdout, char **tempname)</b></li> 770 <li><b>void replace_tempfile(int fdin, int fdout, char **tempname)</b></li> 771 <li><b>void crc_init(unsigned int *crc_table, int little_endian)</b></li> 772 <li><b>int terminal_size(unsigned *xx, unsigned *yy)</b></li> 773 <li><b>int yesno(char *prompt, int def)</b></li> 774 <li><b>void generic_signal(int sig)</b></li> 775 <li><b>void sigatexit(void *handler)</b></li> 776 <li><b>int sig_to_num(char *pidstr)</b></li> 777 <li><b>char *num_to_sig(int sig)</b></li> 778 <li><b>mode_t string_to_mode(char *modestr, mode_t mode)</b></li> 779 <li><b>void mode_to_string(mode_t mode, char *buf)</b></li> 780 <li><b>void names_to_pid(char **names, int (*callback)(pid_t pid, char *name))</b></li> 781 <li><b>int human_readable(char *buf, unsigned long long num)</b></li> 782 </ul> 783 784 <h3>lib/portability.h</h3> 785 786 <p>This file is automatically included from the top of toys.h, and smooths 787 over differences between platforms (hardware targets, compilers, C libraries, 788 operating systems, etc).</p> 789 790 <p>This file provides SWAP macros (SWAP_BE16(x) and SWAP_LE32(x) and so on).</p> 791 792 <p>A macro like SWAP_LE32(x) means "The value in x is stored as a little 793 endian 32 bit value, so perform the translation to/from whatever the native 794 32-bit format is". You do the swap once on the way in, and once on the way 795 out. If your target is already little endian, the macro is a NOP.</p> 796 797 <p>The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions. 798 In each case, the name of the macro refers to the _external_ representation, 799 and converts to/from whatever your native representation happens to be (which 800 can vary depending on what you're currently compiling for).</p> 801 802 <a name="lib_llist"><h3>lib/llist.c</h3> 803 804 <p>Some generic single and doubly linked list functions, which take 805 advantage of a couple properties of C:</p> 806 807 <ul> 808 <li><p>Structure elements are laid out in memory in the order listed, and 809 the first element has no padding. This means you can always treat (typecast) 810 a pointer to a structure as a pointer to the first element of the structure, 811 even if you don't know anything about the data following it.</p></li> 812 813 <li><p>An array of length zero at the end of a structure adds no space 814 to the sizeof() the structure, but if you calculate how much extra space 815 you want when you malloc() the structure it will be available at the end. 816 Since C has no bounds checking, this means each struct can have one variable 817 length array.</p></li> 818 </ul> 819 820 <p>Toybox's list structures always have their <b>next</b> pointer as 821 the first entry of each struct, and singly linked lists end with a NULL pointer. 822 This allows generic code to traverse such lists without knowing anything 823 else about the specific structs composing them: if your pointer isn't NULL 824 typecast it to void ** and dereference once to get the next entry.</p> 825 826 <p><b>lib/lib.h</b> defines three structure types:</p> 827 <ul> 828 <li><p><b>struct string_list</b> - stores a single string (<b>char str[0]</b>), 829 memory for which is allocated as part of the node. (I.E. llist_traverse(list, 830 free); can clean up after this type of list.)</p></li> 831 832 <li><p><b>struct arg_list</b> - stores a pointer to a single string 833 (<b>char *arg</b>) which is stored in a separate chunk of memory.</p></li> 834 835 <li><p><b>struct double_list</b> - has a second pointer (<b>struct double_list 836 *prev</b> along with a <b>char *data</b> for payload.</p></li> 837 </ul> 838 839 <b>List Functions</b> 840 841 <ul> 842 <li><p>void *<b>llist_pop</b>(void **list) - advances through a list ala 843 <b>node = llist_pop(&list);</b> This doesn't modify the list contents, 844 but does advance the pointer you feed it (which is why you pass the _address_ 845 of that pointer, not the pointer itself).</p></li> 846 847 <li><p>void <b>llist_traverse</b>(void *list, void (*using)(void *data)) - 848 iterate through a list calling a function on each node.</p></li> 849 850 <li><p>struct double_list *<b>dlist_add</b>(struct double_list **llist, char *data) 851 - append an entry to a circular linked list. 852 This function allocates a new struct double_list wrapper and returns the 853 pointer to the new entry (which you can usually ignore since it's llist->prev, 854 but if llist was NULL you need it). The argument is the ->data field for the 855 new node.</p></li> 856 <ul><li><p>void <b>dlist_add_nomalloc</b>(struct double_list **llist, 857 struct double_list *new) - append existing struct double_list to 858 list, does not allocate anything.</p></li></ul> 859 </ul> 860 861 <b>List code trivia questions:</b> 862 863 <ul> 864 <li><p><b>Why do arg_list and double_list contain a char * payload instead of 865 a void *?</b> - Because you always have to typecast a void * to use it, and 866 typecasting a char * does no harm. Since strings are the most common 867 payload, and doing math on the pointer ala 868 "(type *)(ptr+sizeof(thing)+sizeof(otherthing))" requires ptr to be char * 869 anyway (at least according to the C standard), defaulting to char * saves 870 a typecast.</p> 871 </li> 872 873 <li><p><b>Why do the names ->str, ->arg, and ->data differ?</b> - To force 874 you to keep track of which one you're using, calling free(node->str) would 875 be bad, and _failing_ to free(node->arg) leaks memory.</p></li> 876 877 <li><p><b>Why does llist_pop() take a void * instead of void **?</b> - 878 because the stupid compiler complains about "type punned pointers" when 879 you typecast and dereference on the same line, 880 due to insane FSF developers hardwiring limitations of their optimizer 881 into gcc's warning system. Since C automatically typecasts any other 882 pointer type to and from void *, the current code works fine. It's sad that it 883 won't warn you if you forget the &, but the code crashes pretty quickly in 884 that case.</p></li> 885 886 <li><p><b>How do I assemble a singly-linked-list in order?</b> - use 887 a double_list, dlist_add() your entries, and then break the circle with 888 <b>list->prev->next = NULL;</b> when done.</li> 889 </ul> 890 891 <a name="lib_args"><h3>lib/args.c</h3> 892 893 <p>Toybox's main.c automatically parses command line options before calling the 894 command's main function. Option parsing starts in get_optflags(), which stores 895 results in the global structures "toys" (optflags and optargs) and "this".</p> 896 897 <p>The option parsing infrastructure stores a bitfield in toys.optflags to 898 indicate which options the current command line contained, and defines FLAG 899 macros code can use to check whether each argument's bit is set. Arguments 900 attached to those options are saved into the command's global structure 901 ("this"). Any remaining command line arguments are collected together into 902 the null-terminated array toys.optargs, with the length in toys.optc. (Note 903 that toys.optargs does not contain the current command name at position zero, 904 use "toys.which->name" for that.) The raw command line arguments get_optflags() 905 parsed are retained unmodified in toys.argv[].</p> 906 907 <p>Toybox's option parsing logic is controlled by an "optflags" string, using 908 a format reminiscent of getopt's optargs but with several important differences. 909 Toybox does not use the getopt() 910 function out of the C library, get_optflags() is an independent implementation 911 which doesn't permute the original arguments (and thus doesn't change how the 912 command is displayed in ps and top), and has many features not present in 913 libc optargs() (such as the ability to describe long options in the same string 914 as normal options).</p> 915 916 <p>Each command's NEWTOY() macro has an optflags string as its middle argument, 917 which sets toy_list.options for that command to tell get_optflags() what 918 command line arguments to look for, and what to do with them. 919 If a command has no option 920 definition string (I.E. the argument is NULL), option parsing is skipped 921 for that command, which must look at the raw data in toys.argv to parse its 922 own arguments. (If no currently enabled command uses option parsing, 923 get_optflags() is optimized out of the resulting binary by the compiler's 924 --gc-sections option.)</p> 925 926 <p>You don't have to free the option strings, which point into the environment 927 space (I.E. the string data is not copied). A TOYFLAG_NOFORK command 928 that uses the linked list type "*" should free the list objects but not 929 the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not 930 NOFORK, exit() will free all the malloced data anyway unless you want 931 to implement a CONFIG_TOYBOX_FREE cleanup for it.)</p> 932 933 <h4>Optflags format string</h4> 934 935 <p>Note: the optflags option description string format is much more 936 concisely described by a large comment at the top of lib/args.c.</p> 937 938 <p>The general theory is that letters set optflags, and punctuation describes 939 other actions the option parsing logic should take.</p> 940 941 <p>For example, suppose the command line <b>command -b fruit -d walrus -a 42</b> 942 is parsed using the optflags string "<b>a#b:c:d</b>". (I.E. 943 toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d", 944 "walrus", "-a", "42"]). When get_optflags() returns, the following data is 945 available to command_main(): 946 947 <ul> 948 <li><p>In <b>struct toys</b>: 949 <ul> 950 <li>toys.optflags = 13; // FLAG_a = 8 | FLAG_b = 4 | FLAG_d = 1</li> 951 <li>toys.optargs[0] = "walrus"; // leftover argument</li> 952 <li>toys.optargs[1] = NULL; // end of list</li> 953 <li>toys.optc = 1; // there was 1 leftover argument</li> 954 <li>toys.argv[] = {"-b", "fruit", "-d", "walrus", "-a", "42"}; // The original command line arguments 955 </ul> 956 <p></li> 957 958 <li><p>In <b>union this</b> (treated as <b>long this[]</b>): 959 <ul> 960 <li>this[0] = NULL; // -c didn't get an argument this time, so get_optflags() didn't change it and toys_init() zeroed "this" during setup.)</li> 961 <li>this[1] = (long)"fruit"; // argument to -b</li> 962 <li>this[2] = 42; // argument to -a</li> 963 </ul> 964 </p></li> 965 </ul> 966 967 <p>If the command's globals are:</p> 968 969 <blockquote><pre> 970 GLOBALS( 971 char *c; 972 char *b; 973 long a; 974 ) 975 </pre></blockquote> 976 977 <p>That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember, 978 each entry that receives an argument must be a long or pointer, to line up 979 with the array position. Right to left in the optflags string corresponds to 980 top to bottom in GLOBALS().</p> 981 982 <p>Put globals not filled out by the option parsing logic at the end of the 983 GLOBALS block. Common practice is to list the options one per line (to 984 make the ordering explicit, first to last in globals corresponds to right 985 to left in the option string), then leave a blank line before any non-option 986 globals.</p> 987 988 <p><b>long toys.optflags</b></p> 989 990 <p>Each option in the optflags string corresponds to a bit position in 991 toys.optflags, with the same value as a corresponding binary digit. The 992 rightmost argument is (1<<0), the next to last is (1<<1) and so on. If 993 the option isn't encountered while parsing argv[], its bit remains 0.</p> 994 995 <p>Each option -x has a FLAG_x macro for the command letter. Bare --longopts 996 with no corresponding short option have a FLAG_longopt macro for the long 997 optionname. Commands enable these macros by #defining FOR_commandname before 998 #including <toys.h>. When multiple commands are implemented in the same 999 source file, you can switch flag contexts later in the file by 1000 #defining CLEANUP_oldcommand and #defining FOR_newcommand, then 1001 #including <generated/flags.h>.</p> 1002 1003 <p>Options disabled in the current configuration (wrapped in 1004 a USE_BLAH() macro for a CONFIG_BLAH that's switched off) have their 1005 corresponding FLAG macro set to zero, so code checking them ala 1006 if (toys.optargs & FLAG_x) gets optimized out via dead code elimination. 1007 #defining FORCE_FLAGS when switching flag context disables this 1008 behavior: the flag is never zero even if the config is disabled. This 1009 allows code shared between multiple commands to use the same flag 1010 values, as long as the common flags match up right to left in both option 1011 strings.</p> 1012 1013 <p>For example, 1014 the optflags string "abcd" would parse the command line argument "-c" to set 1015 optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to 1016 6 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8). To check if -c 1017 was encountered, code could test: if (toys.optflags & FLAG_c) printf("yup"); 1018 (See the toys/examples directory for more.)</p> 1019 1020 <p>Only letters are relevant to optflags, punctuation is skipped: in the 1021 string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter 1022 usually indicate that the option takes an argument.</p> 1023 1024 <p>Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is 1025 the amount a long would have on 32-bit platforms anyway; 64 bit code on 1026 32 bit platforms is too expensive to require in common code used by almost 1027 all commands.) Bit positions beyond the 1<<31 aren't recorded, but 1028 parsing higher options can still set global variables.</p> 1029 1030 <p><b>Automatically setting global variables from arguments (union this)</b></p> 1031 1032 <p>The following punctuation characters may be appended to an optflags 1033 argument letter, indicating the option takes an additional argument:</p> 1034 1035 <ul> 1036 <li><b>:</b> - plus a string argument, keep most recent if more than one.</li> 1037 <li><b>*</b> - plus a string argument, appended to a linked list.</li> 1038 <li><b>@</b> - plus an occurrence counter (stored in a long)</li> 1039 <li><b>#</b> - plus a signed long argument. 1040 <li><b>-</b> - plus a signed long argument defaulting to negative (start argument with + to force a positive value).</li> 1041 <li><b>.</b> - plus a floating point argument (if CFG_TOYBOX_FLOAT).</li> 1042 <ul>The following can be appended to a float or double: 1043 <li><b><123</b> - error if argument is less than this</li> 1044 <li><b>>123</b> - error if argument is greater than this</li> 1045 <li><b>=123</b> - default value if argument not supplied</li> 1046 </ul> 1047 </ul> 1048 1049 <p><b>GLOBALS</b></p> 1050 1051 <p>Options which have an argument fill in the corresponding slot in the global 1052 union "this" (see generated/globals.h), treating it as an array of longs 1053 with the rightmost saved in this[0]. As described above, using "a*b:c#d", 1054 "-c 42" would set this[0] = 42; and "-b 42" would set this[1] = "42"; each 1055 slot is left NULL if the corresponding argument is not encountered.</p> 1056 1057 <p>This behavior is useful because the LP64 standard ensures long and pointer 1058 are the same size. C99 guarantees structure members will occur in memory 1059 in the same order they're declared, and that padding won't be inserted between 1060 consecutive variables of register size. Thus the first few entries can 1061 be longs or pointers corresponding to the saved arguments.</p> 1062 1063 <p>See toys/example/*.c for longer examples of parsing options into the 1064 GLOBALS block.</p> 1065 1066 <p><b>char *toys.optargs[]</b></p> 1067 1068 <p>Command line arguments in argv[] which are not consumed by option parsing 1069 (I.E. not recognized either as -flags or arguments to -flags) will be copied 1070 to toys.optargs[], with the length of that array in toys.optc. 1071 (When toys.optc is 0, no unrecognized command line arguments remain.) 1072 The order of entries is preserved, and as with argv[] this new array is also 1073 terminated by a NULL entry.</p> 1074 1075 <p>Option parsing can require a minimum or maximum number of optargs left 1076 over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the 1077 start of the optflags string.</p> 1078 1079 <p>The special argument "--" terminates option parsing, storing all remaining 1080 arguments in optargs. The "--" itself is consumed.</p> 1081 1082 <p><b>Other optflags control characters</b></p> 1083 1084 <p>The following characters may occur at the start of each command's 1085 optflags string, before any options that would set a bit in toys.optflags:</p> 1086 1087 <ul> 1088 <li><b>^</b> - stop at first nonoption argument (for nice, xargs...)</li> 1089 <li><b>?</b> - allow unknown arguments (pass non-option arguments starting 1090 with - through to optargs instead of erroring out).</li> 1091 <li><b>&</b> - the first argument has imaginary dash (ala tar/ps. If given twice, all arguments have imaginary dash.)</li> 1092 <li><b><</b> - must be followed by a decimal digit indicating at least this many leftover arguments are needed in optargs (default 0)</li> 1093 <li><b>></b> - must be followed by a decimal digit indicating at most this many leftover arguments allowed (default MAX_INT)</li> 1094 </ul> 1095 1096 <p>The following characters may be appended to an option character, but do 1097 not by themselves indicate an extra argument should be saved in this[]. 1098 (Technically any character not recognized as a control character sets an 1099 optflag, but letters are never control characters.)</p> 1100 1101 <ul> 1102 <li><b>^</b> - stop parsing options after encountering this option, everything else goes into optargs.</li> 1103 <li><b>|</b> - this option is required. If more than one marked, only one is required.</li> 1104 </ul> 1105 1106 <p>The following may be appended to a float or double:</p> 1107 1108 <ul> 1109 <li><b><123</b> - error if argument is less than this</li> 1110 <li><b>>123</b> - error if argument is greater than this</li> 1111 <li><b>=123</b> - default value if argument not supplied</li> 1112 </ul> 1113 1114 <p>Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT 1115 is enabled. (Otherwise the code to determine where floating point constants 1116 end drops out. When disabled, it can reserve a global data slot for the 1117 argument so offsets won't change, but will never fill it out.) You can handle 1118 this by using the USE_BLAH() macros with C string concatenation, ala:</p> 1119 1120 <blockquote>"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"</blockquote> 1121 1122 <p><b>--longopts</b></p> 1123 1124 <p>The optflags string can contain long options, which are enclosed in 1125 parentheses. They may be appended to an existing option character, in 1126 which case the --longopt is a synonym for that option, ala "a:(--fred)" 1127 which understands "-a blah" or "--fred blah" as synonyms.</p> 1128 1129 <p>Longopts may also appear before any other options in the optflags string, 1130 in which case they have no corresponding short argument, but instead set 1131 their own bit based on position. So for "(walrus)#(blah)xy:z", "command 1132 --walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8) 1133 and would assign this[1] = 42;</p> 1134 1135 <p>A short option may have multiple longopt synonyms, "a(one)(two)", but 1136 each "bare longopt" (ala "(one)(two)abc" before any option characters) 1137 always sets its own bit (although you can group them with +X).</p> 1138 1139 <p>Only bare longopts have a FLAG_ macro with the longopt name 1140 (ala --fred would #define FLAG_fred). Other longopts use the short 1141 option's FLAG macro to test the toys.optflags bit.</p> 1142 1143 <p>Options with a semicolon ";" after their data type can only set their 1144 corresponding GLOBALS() entry via "--longopt=value". For example, option 1145 string "x(boing): y" would set TT.x if it saw "--boing=value", but would 1146 treat "--boing value" as setting FLAG_x in toys.optargs, leaving TT.x NULL, 1147 and keeping "value" in toys.optargs[]. (This lets "ls --color" and 1148 "ls --color=auto" both work.)</p> 1149 1150 <p><b>[groups]</b></p> 1151 1152 <p>At the end of the option string, square bracket groups can define 1153 relationships between existing options. (This only applies to short 1154 options, bare --longopts can't participate.)</p> 1155 1156 <p>The first character of the group defines the type, the remaining 1157 characters are options it applies to:</p> 1158 1159 <ul> 1160 <li><b>-</b> - Exclusive, switch off all others in this group.</li> 1161 <li><b>+</b> - Inclusive, switch on all others in this group.</li> 1162 <li><b>!</b> - Error, fail if more than one defined.</li> 1163 </ul> 1164 1165 <p>So "abc[-abc]" means -ab = -b, -ba = -a, -abc = -c. "abc[+abc]" 1166 means -ab=-abc, -c=-abc, and "abc[!abc] means -ab calls error_exit("no -b 1167 with -a"). Note that [-] groups clear the GLOBALS option slot of 1168 options they're switching back off, but [+] won't set options it didn't see 1169 (just the optflags).</p> 1170 1171 <p><b>whitespace</b></p> 1172 1173 <p>Arguments may occur with or without a space (I.E. "-a 42" or "-a42"). 1174 The command line argument "-abc" may be interepreted many different ways: 1175 the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4 1176 and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves 1177 "c" as the argument to -b.</p> 1178 1179 <p>Note that & changes whitespace handling, so that the command line 1180 "tar cvfCj outfile.tar.bz2 topdir filename" is parsed the same as 1181 "tar filename -c -v -j -f outfile.tar.bz2 -C topdir". Note that "tar -cvfCj 1182 one two three" would equal "tar -c -v -f Cj one two three". (This matches 1183 historical usage.)</p> 1184 1185 <p>Appending a space to the option in the option string ("a: b") makes it 1186 require a space, I.E. "-ab" is interpreted as "-a" "-b". That way "kill -stop" 1187 differs from "kill -s top".</p> 1188 1189 <p>Appending ; to a longopt in the option string makes its argument optional, 1190 and only settable with =, so in ls "(color):;" can accept "ls --color" and 1191 "ls --color=auto" without complaining that the first has no argument.</p> 1192 1193 <a name="lib_dirtree"><h3>lib/dirtree.c</h3> 1194 1195 <p>The directory tree traversal code should be sufficiently generic 1196 that commands never need to use readdir(), scandir(), or the fts.h family 1197 of functions.</p> 1198 1199 <p>These functions do not call chdir() or rely on PATH_MAX. Instead they 1200 use openat() and friends, using one filehandle per directory level to 1201 recurseinto subdirectories. (I.E. they can descend 1000 directories deep 1202 if setrlimit(RLIMIT_NOFILE) allows enough open filehandles, and the default 1203 in /proc/self/limits is generally 1024.)</p> 1204 1205 <p>The basic dirtree functions are:</p> 1206 1207 <ul> 1208 <li><p><b>dirtree_read(char *path, int (*callback)(struct dirtree node))</b> - 1209 recursively read directories, either applying callback() or returning 1210 a tree of struct dirtree if callback is NULL.</p></li> 1211 1212 <li><p><b>dirtree_path(struct dirtree *node, int *plen)</b> - malloc() a 1213 string containing the path from the root of this tree to this node. If 1214 plen isn't NULL then *plen is how many extra bytes to malloc at the end 1215 of string.</p></li> 1216 1217 <li><p><b>dirtree_parentfd(struct dirtree *node)</b> - return fd of 1218 containing directory, for use with openat() and such.</p></li> 1219 </ul> 1220 1221 <p>The <b>dirtree_read()</b> function takes two arguments, a starting path for 1222 the root of the tree, and a callback function. The callback takes a 1223 <b>struct dirtree *</b> (from lib/lib.h) as its argument. If the callback is 1224 NULL, the traversal uses a default callback (dirtree_notdotdot()) which 1225 recursively assembles a tree of struct dirtree nodes for all files under 1226 this directory and subdirectories (filtering out "." and ".." entries), 1227 after which dirtree_read() returns the pointer to the root node of this 1228 snapshot tree.</p> 1229 1230 <p>Otherwise the callback() is called on each entry in the directory, 1231 with struct dirtree * as its argument. This includes the initial 1232 node created by dirtree_read() at the top of the tree.</p> 1233 1234 <p><b>struct dirtree</b></p> 1235 1236 <p>Each struct dirtree node contains <b>char name[]</b> and <b>struct stat 1237 st</b> entries describing a file, plus a <b>char *symlink</b> 1238 which is NULL for non-symlinks.</p> 1239 1240 <p>During a callback function, the <b>int data</b> field of directory nodes 1241 contains a dirfd (for use with the openat() family of functions). This is 1242 generally used by calling dirtree_parentfd() on the callback's node argument. 1243 For symlinks, data contains the length of the symlink string. On the second 1244 callback from DIRTREE_COMEAGAIN (depth-first traversal) data = -1 for 1245 all nodes (that's how you can tell it's the second callback).</p> 1246 1247 <p>Users of this code may put anything they like into the <b>long extra</b> 1248 field. For example, "cp" and "mv" use this to store a dirfd for the destination 1249 directory (and use DIRTREE_COMEAGAIN to get the second callback so they can 1250 close(node->extra) to avoid running out of filehandles). 1251 This field is not directly used by the dirtree code, and 1252 thanks to LP64 it's large enough to store a typecast pointer to an 1253 arbitrary struct.</p> 1254 1255 <p>The return value of the callback combines flags (with boolean or) to tell 1256 the traversal infrastructure how to behave:</p> 1257 1258 <ul> 1259 <li><p><b>DIRTREE_SAVE</b> - Save this node, assembling a tree. (Without 1260 this the struct dirtree is freed after the callback returns. Filtering out 1261 siblings is fine, but discarding a parent while keeping its child leaks 1262 memory.)</p></li> 1263 <li><p><b>DIRTREE_ABORT</b> - Do not examine any more entries in this 1264 directory. (Does not propagate up tree: to abort entire traversal, 1265 return DIRTREE_ABORT from parent callbacks too.)</p></li> 1266 <li><p><b>DIRTREE_RECURSE</b> - Examine directory contents. Ignored for 1267 non-directory entries. The remaining flags only take effect when 1268 recursing into the children of a directory.</p></li> 1269 <li><p><b>DIRTREE_COMEAGAIN</b> - Call the callback a second time after 1270 examining all directory contents, allowing depth-first traversal. 1271 On the second call, dirtree->data = -1.</p></li> 1272 <li><p><b>DIRTREE_SYMFOLLOW</b> - follow symlinks when populating children's 1273 <b>struct stat st</b> (by feeding a nonzero value to the symfollow argument of 1274 dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to 1275 directories as directories. (Avoiding infinite recursion is the callback's 1276 problem: the non-NULL dirtree->symlink can still distinguish between 1277 them.)</p></li> 1278 </ul> 1279 1280 <p>Each struct dirtree contains three pointers (next, parent, and child) 1281 to other struct dirtree.</p> 1282 1283 <p>The <b>parent</b> pointer indicates the directory 1284 containing this entry; even when not assembling a persistent tree of 1285 nodes the parent entries remain live up to the root of the tree while 1286 child nodes are active. At the top of the tree the parent pointer is 1287 NULL, meaning the node's name[] is either an absolute path or relative 1288 to cwd. The function dirtree_parentfd() gets the directory file descriptor 1289 for use with openat() and friends, returning AT_FDCWD at the top of tree.</p> 1290 1291 <p>The <b>child</b> pointer points to the first node of the list of contents of 1292 this directory. If the directory contains no files, or the entry isn't 1293 a directory, child is NULL.</p> 1294 1295 <p>The <b>next</b> pointer indicates sibling nodes in the same directory as this 1296 node, and since it's the first entry in the struct the llist.c traversal 1297 mechanisms work to iterate over sibling nodes. Each dirtree node is a 1298 single malloc() (even char *symlink points to memory at the end of the node), 1299 so llist_free() works but its callback must descend into child nodes (freeing 1300 a tree, not just a linked list), plus whatever the user stored in extra.</p> 1301 1302 <p>The <b>dirtree_read</b>() function is a simple wrapper, calling <b>dirtree_add_node</b>() 1303 to create a root node relative to the current directory, then calling 1304 <b>handle_callback</b>() on that node (which recurses as instructed by the callback 1305 return flags). Some commands (such as chgrp) bypass this wrapper, for example 1306 to control whether or not to follow symlinks to the root node; symlinks 1307 listed on the command line are often treated differently than symlinks 1308 encountered during recursive directory traversal). 1309 1310 <p>The ls command not only bypasses the wrapper, but never returns 1311 <b>DIRTREE_RECURSE</b> from the callback, instead calling <b>dirtree_recurse</b>() manually 1312 from elsewhere in the program. This gives ls -lR manual control 1313 of traversal order, which is neither depth first nor breadth first but 1314 instead a sort of FIFO order requried by the ls standard.</p> 1315 1316 <a name="toys"> 1317 <h1><a href="#toys">Directory toys/</a></h1> 1318 1319 <p>This directory contains command implementations. Each command is a single 1320 self-contained file. Adding a new command involves adding a single 1321 file, and removing a command involves removing that file. Commands use 1322 shared infrastructure from the lib/ and generated/ directories.</p> 1323 1324 <p>Currently there are five subdirectories under "toys/" containing "posix" 1325 commands described in POSIX-2008, "lsb" commands described in the Linux 1326 Standard Base 4.1, "other" commands not described by either standard, 1327 "pending" commands awaiting cleanup (which default to "n" in menuconfig 1328 because they don't necessarily work right yet), and "example" code showing 1329 how toybox infrastructure works and providing template/skeleton files to 1330 start new commands.</p> 1331 1332 <p>The only difference directory location makes is which menu the command 1333 shows up in during "make menuconfig", the directories are otherwise identical. 1334 Note that the commands exist within a single namespace at runtime, so you can't 1335 have the same command in multiple subdirectories. (The build tries to fail 1336 informatively when you do that.)</p> 1337 1338 <p>There is one more sub-menus in "make menuconfig" containing global 1339 configuration options for toybox. This menu is defined in the top level 1340 Config.in.</p> 1341 1342 <p>See <a href="#adding">adding a new command</a> for details on the 1343 layout of a command file.</p> 1344 1345 <h2>Directory scripts/</h2> 1346 1347 <p>Build infrastructure. The makefile calls scripts/make.sh for "make" 1348 and scripts/install.sh for "make install".</p> 1349 1350 <p>There's also a test suite, "make test" calls make/test.sh, which runs all 1351 the tests in make/test/*. You can run individual tests via 1352 "scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run 1353 that test against the host implementation instead of the toybox one.</p> 1354 1355 <h3>scripts/cfg2files.sh</h3> 1356 1357 <p>Run .config through this filter to get a list of enabled commands, which 1358 is turned into a list of files in toys via a sed invocation in the top level 1359 Makefile. 1360 </p> 1361 1362 <h2>Directory kconfig/</h2> 1363 1364 <p>Menuconfig infrastructure copied from the Linux kernel. See the 1365 Linux kernel's Documentation/kbuild/kconfig-language.txt</p> 1366 1367 <!-- todo 1368 1369 Better OLDTOY and multiple command explanation. From Config.in: 1370 1371 <p>A command with multiple names (or multiple similar commands implemented in 1372 the same .c file) should have config symbols prefixed with the name of their 1373 C file. I.E. config symbol prefixes are NEWTOY() names. If OLDTOY() names 1374 have config symbols they must be options (symbols with an underscore and 1375 suffix) to the NEWTOY() name. (See generated/toylist.h)</p> 1376 --> 1377 1378 <!--#include file="footer.html" --> 1379
