1 <html><head><title>toybox source code walkthrough</title></head> 2 <!--#include file="header.html" --> 3 4 <p><h1><a name="style" /><a href="#style">Code style</a></h1></p> 5 6 <p>The primary goal of toybox is _simple_ code. Keeping the code small is 7 second, with speed and lots of features coming in somewhere after that. 8 (For more on that, see the <a href=design.html>design</a> page.)</p> 9 10 <p>A simple implementation usually takes up fewer lines of source code, 11 meaning more code can fit on the screen at once, meaning the programmer can 12 see more of it on the screen and thus keep more if in their head at once. 13 This helps code auditing and thus reduces bugs. That said, sometimes being 14 more explicit is preferable to being clever enough to outsmart yourself: 15 don't be so terse your code is unreadable.</p> 16 17 <p>Toybox has an actual coding style guide over on 18 <a href=design.html#codestyle>the design page</a>, but in general we just 19 want the code to be consistent.</p> 20 21 <p><h1><a name="building" /><a href="#building">Building Toybox</a></h1></p> 22 23 <p>Toybox is configured using the Kconfig language pioneered by the Linux 24 kernel, and adopted by many other projects (uClibc, OpenEmbedded, etc). 25 This generates a ".config" file containing the selected options, which 26 controls which features are included when compiling toybox.</p> 27 28 <p>Each configuration option has a default value. The defaults indicate the 29 "maximum sane configuration", I.E. if the feature defaults to "n" then it 30 either isn't complete or is a special-purpose option (such as debugging 31 code) that isn't intended for general purpose use.</p> 32 33 <p>For a more compact human-editable version .config files, you can use the 34 <a href=http://landley.net/aboriginal/FAQ.html#dev_miniconfig>miniconfig</a> 35 format.</p> 36 37 <p>The standard build invocation is:</p> 38 39 <ul> 40 <li>make defconfig #(or menuconfig)</li> 41 <li>make</li> 42 <li>make install</li> 43 </ul> 44 45 <p>Type "make help" to see all available build options.</p> 46 47 <p>The file "configure" contains a number of environment variable definitions 48 which influence the build, such as specifying which compiler to use or where 49 to install the resulting binaries. This file is included by the build, but 50 accepts existing definitions of the environment variables, so it may be sourced 51 or modified by the developer before building and the definitions exported 52 to the environment will take precedence.</p> 53 54 <p>(To clarify: ".config" lists the features selected by defconfig/menuconfig, 55 I.E. "what to build", and "configure" describes the build and installation 56 environment, I.E. "how to build it".)</p> 57 58 <p>By default "make install" puts files in /usr/toybox. Adding this to the 59 $PATH is up to you. The environment variable $PREFIX can change the 60 install location, ala "PREFIX=/usr/local/bin make install".</p> 61 62 <p>If you need an unstripped (debug) version of any of these binaries, 63 look in generated/unstripped.</p> 64 65 <p><h1><a name="running"><a href="#running">Running a command</a></h1></p> 66 67 <h2>main</h2> 68 69 <p>The toybox main() function is at the end of main.c at the top level. It has 70 two possible codepaths, only one of which is configured into any given build 71 of toybox.</p> 72 73 <p>If CONFIG_SINGLE is selected, toybox is configured to contain only a single 74 command, so most of the normal setup can be skipped. In this case the 75 multiplexer isn't used, instead main() calls toy_singleinit() (also in main.c) 76 to set up global state and parse command line arguments, calls the command's 77 main function out of toy_list (in the CONFIG_SINGLE case the array has a single entry, no need to search), and if the function returns instead of exiting 78 it flushes stdout (detecting error) and returns toys.exitval.</p> 79 80 <p>When CONFIG_SINGLE is not selected, main() uses basename() to find the 81 name it was run as, shifts its argument list one to the right so it lines up 82 with where the multiplexer function expects it, and calls toybox_main(). This 83 leverages the multiplexer command's infrastructure to find and run the 84 appropriate command. (A command name starting with "toybox" will 85 recursively call toybox_main(); you can go "./toybox toybox toybox toybox ls" 86 if you want to...)</p> 87 88 <h2>toybox_main</h2> 89 90 <p>The toybox_main() function is also in main,c. It handles a possible 91 --help option ("toybox --help ls"), prints the list of available commands if no 92 arguments were provided to the multiplexer (or with full path names if any 93 other option is provided before a command name, ala "toybox --list"). 94 Otherwise it calls toy_exec() on its argument list.</p> 95 96 <p>Note that the multiplexer is the first entry in toy_list (the rest of the 97 list is sorted alphabetically to allow binary search), so toybox_main can 98 cheat and just grab the first entry to quickly set up its context without 99 searching. Since all command names go through the multiplexer at least once 100 in the non-TOYBOX_SINGLE case, this avoids a redundant search of 101 the list.</p> 102 103 <p>The toy_exec() function is also in main.c. It performs toy_find() to 104 perform a binary search on the toy_list array to look up the command's 105 entry by name and saves it in the global variable which, calls toy_init() 106 to parse command line arguments and set up global state (using which->options), 107 and calls the appropriate command's main() function (which->toy_main). On 108 return it flushes all pending ansi FILE * I/O, detects if stdout had an 109 error, and then calls xexit() (which uses toys.exitval).</p> 110 111 <p><h1><a name="infrastructure" /><a href="#infrastructure">Infrastructure</a></h1></p> 112 113 <p>The toybox source code is in following directories:</p> 114 <ul> 115 <li>The <a href="#top">top level directory</a> contains the file main.c (were 116 execution starts), the header file toys.h (included by every command), and 117 other global infrastructure.</li> 118 <li>The <a href="#lib">lib directory</a> contains common functions shared by 119 multiple commands:</li> 120 <ul> 121 <li><a href="#lib_lib">lib/lib.c</a></li> 122 <li><a href="#lib_xwrap">lib/xwrap.c</a></li> 123 <li><a href="#lib_llist">lib/llist.c</a></li> 124 <li><a href="#lib_args">lib/args.c</a></li> 125 <li><a href="#lib_dirtree">lib/dirtree.c</a></li> 126 </ul> 127 <li>The <a href="#toys">toys directory</a> contains the C files implementating 128 each command. Currently it contains five subdirectories categorizing the 129 commands: posix, lsb, other, example, and pending.</li> 130 <li>The <a href="#scripts">scripts directory</a> contains the build and 131 test infrastructure.</li> 132 <li>The <a href="#kconfig">kconfig directory</a> contains the configuration 133 infrastructure implementing menuconfig (copied from the Linux kernel).</li> 134 <li>The <a href="#generated">generated directory</a> contains intermediate 135 files generated from other parts of the source code.</li> 136 </ul> 137 138 <a name="adding" /> 139 <p><h1><a href="#adding">Adding a new command</a></h1></p> 140 <p>To add a new command to toybox, add a C file implementing that command to 141 one of the subdirectories under the toys directory. No other files need to 142 be modified; the build extracts all the information it needs (such as command 143 line arguments) from specially formatted comments and macros in the C file. 144 (See the description of the <a href="#generated">"generated" directory</a> 145 for details.)</p> 146 147 <p>Currently there are five subdirectories under "toys", one for commands 148 defined by the POSIX standard, one for commands defined by the Linux Standard 149 Base, an "other" directory for commands not covered by an obvious standard, 150 a directory of example commands (templates to use when starting new commands), 151 and a "pending" directory of commands that need further review/cleanup 152 before moving to one of the other directories (run these at your own risk, 153 cleanup patches welcome). 154 These directories are just for developer convenience sorting the commands, 155 the directories are otherwise functionally identical. To add a new category, 156 create the appropriate directory with a README file in it whose first line 157 is the description menuconfig should use for the directory.)</p> 158 159 <p>An easy way to start a new command is copy the file "toys/example/hello.c" 160 to the name of the new command, and modify this copy to implement the new 161 command (more or less by turning every instance of "hello" into the 162 name of your command, updating the command line arguments, globals, and 163 help data, and then filling out its "main" function with code that does 164 something interesting).</p> 165 166 <p>You could also start with "toys/example/skeleton.c", which provides a lot 167 more example code (showing several variants of command line option 168 parsing, how to implement multiple commands in the same file, and so on). 169 But usually it's just more stuff to delete.</p> 170 171 <p>Here's a checklist of steps to turn hello.c into another command:</p> 172 173 <ul> 174 <li><p>First "cp toys/example/hello.c toys/other/yourcommand.c" and open 175 the new file in your preferred text editor.</p> 176 <ul><li><p>Note that the 177 name of the new file is significant: it's the name of the new command you're 178 adding to toybox. The build includes all *.c files under toys/*/ whose 179 names are a case insensitive match for an enabled config symbol. So 180 toys/posix/cat.c only gets included if you have "CAT=y" in ".config".</p></li> 181 </ul></p></li> 182 183 <li><p>Change the one line comment at the top of the file (currently 184 "hello.c - A hello world program") to describe your new file.</p></li> 185 186 <li><p>Change the copyright notice to your name, email, and the current 187 year.</p></li> 188 189 <li><p>Give a URL to the relevant standards document, where applicable. 190 (Sample links to SUSv4 and LSB are provided, feel free to link to other 191 documentation or standards as appropriate.)</p></li> 192 193 <li><p>Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line. 194 The NEWTOY macro fills out this command's <a href="#toy_list">toy_list</a> 195 structure. The arguments to the NEWTOY macro are:</p> 196 197 <ol> 198 <li><p>the name used to run your command</p></li> 199 <li><p>the command line argument <a href="#lib_args">option parsing string</a> (0 if none)</p></li> 200 <li><p>a bitfield of TOYFLAG values 201 (defined in toys.h) providing additional information such as where your 202 command should be installed on a running system, whether to blank umask 203 before running, whether or not the command must run as root (and thus should 204 retain root access if installed SUID), and so on.</p></li> 205 </ol> 206 </li> 207 208 <li><p>Change the kconfig data (from "config YOURCOMMAND" to the end of the 209 comment block) to supply your command's configuration and help 210 information. The uppper case config symbols are used by menuconfig, and are 211 also what the CFG_ and USE_() macros are generated from (see [TODO]). The 212 help information here is used by menuconfig, and also by the "help" command to 213 describe your new command. (See [TODO] for details.) By convention, 214 unfinished commands default to "n" and finished commands default to "y", 215 so "make defconfig" selects all finished commands. (Note, "finished" means 216 "ready to be used", not that it'll never change again.)<p> 217 218 <p>Each help block should start with a "usage: yourcommand" line explaining 219 any command line arguments added by this config option. The "help" command 220 outputs this text, and scripts/config2help.c in the build infrastructure 221 collates these usage lines for commands with multiple configuration 222 options when producing generated/help.h.</p> 223 </li> 224 225 <li><p>Change the "#define FOR_hello" line to "#define FOR_yourcommand" right 226 before the "#include <toys.h>". (This selects the appropriate FLAG_ macros and 227 does a "#define TT this.yourcommand" so you can access the global variables 228 out of the space-saving union of structures. If you aren't using any command 229 flag bits and aren't defining a GLOBAL block, you can delete this line.)</p></li> 230 231 <li><p>Update the GLOBALS() macro to contain your command's global 232 variables. If your command has no global variables, delete this macro.</p> 233 234 <p>Variables in the GLOBALS() block are are stored in a space saving 235 <a href="#toy_union">union of structures</a> format, which may be accessed 236 using the TT macro as if TT were a global structure (so TT.membername). 237 If you specified two-character command line arguments in 238 NEWTOY(), the first few global variables will be initialized by the automatic 239 argument parsing logic, and the type and order of these variables must 240 correspond to the arguments specified in NEWTOY(). 241 (See <a href="#lib_args">lib/args.c</a> for details.)</p></li> 242 243 <li><p>Rename hello_main() to yourcommand_main(). This is the main() function 244 where execution of your command starts. Your command line options are 245 already sorted into this.optflags, this.optargs, this.optc, and the GLOBALS() 246 as appropriate by the time this function is called. (See 247 <a href="#lib_args">get_optflags()</a> for details.)</p></li> 248 249 <li><p>Switch on TOYBOX_DEBUG in menuconfig (toybox global settings menu) 250 the first time you build and run your new command. If anything is wrong 251 with your option string, that will give you error messages.</p> 252 253 <p>Otherwise it'll just segfault without 254 explanation when it falls off the end because it didn't find a matching 255 end parantheses for a longopt, or you put a nonexistent option in a square 256 bracket grouping... Since these kind of errors can only be caused by a 257 developer, not by end users, we don't normally want runtime checks for 258 them. Once you're happy with your option string, you can switch TOYBOX_DEBUG 259 back off.</p></li> 260 </ul> 261 262 <a name="headers" /><h2><a href="#headers">Headers.</a></h2> 263 264 <p>Commands generally don't have their own headers. If it's common code 265 it can live in lib/, if it isn't put it in the command's .c file. (The line 266 between implementing multiple commands in a C file via OLDTOY() to share 267 infrastructure and moving that shared infrastructure to lib/ is a judgement 268 call. Try to figure out which is simplest.)</p> 269 270 <p>The top level toys.h should #include all the standard (posix) headers 271 that any command uses. (Partly this is friendly to ccache and partly this 272 makes the command implementations shorter.) Individual commands should only 273 need to include nonstandard headers that might prevent that command from 274 building in some context we'd care about (and thus requiring that command to 275 be disabled to avoid a build break).</p> 276 277 <p>Target-specific stuff (differences between compiler versions, libc versions, 278 or operating systems) should be confined to lib/portability.h and 279 lib/portability.c. (There's even some minimal compile-time environment probing 280 that writes data to generated/portability.h, see scripts/genconfig.sh.)</p> 281 282 <p>Only include linux/*.h headers from individual commands (not from other 283 headers), and only if you really need to. Data that varies per architecture 284 is a good reason to include a header. If you just need a couple constants 285 that haven't changed since the 1990's, it's ok to #define them yourself or 286 just use the constant inline with a comment explaining what it is. (A 287 #define that's only used once isn't really helping.)</p> 288 289 <p><a name="top" /><h1><a href="#top">Top level directory.</a></h1></p> 290 291 <p>This directory contains global infrastructure.</p> 292 293 <h3>toys.h</h3> 294 <p>Each command #includes "toys.h" as part of its standard prolog. It 295 may "#define FOR_commandname" before doing so to get some extra entries 296 specific to this command.</p> 297 298 <p>This file sucks in most of the commonly used standard #includes, so 299 individual files can just #include "toys.h" and not have to worry about 300 stdargs.h and so on. Individual commands still need to #include 301 special-purpose headers that may not be present on all systems (and thus would 302 prevent toybox from building that command on such a system with that command 303 enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab 304 support (mntent.h and sys/mount.h), and so on.</p> 305 306 <p>The toys.h header also defines structures for most of the global variables 307 provided to each command by toybox_main(). These are described in 308 detail in the description for main.c, where they are initialized.</p> 309 310 <p>The global variables are grouped into structures (and a union) for space 311 savings, to more easily track the amount of memory consumed by them, 312 so that they may be automatically cleared/initialized as needed, and so 313 that access to global variables is more easily distinguished from access to 314 local variables.</p> 315 316 <h3>main.c</h3> 317 <p>Contains the main() function where execution starts, plus 318 common infrastructure to initialize global variables and select which command 319 to run. The "toybox" multiplexer command also lives here. (This is the 320 only command defined outside of the toys directory.)</p> 321 322 <p>Execution starts in main() which trims any path off of the first command 323 name and calls toybox_main(), which calls toy_exec(), which calls toy_find() 324 and toy_init() before calling the appropriate command's function from 325 toy_list[] (via toys.which->toy_main()). 326 If the command is "toybox", execution recurses into toybox_main(), otherwise 327 the call goes to the appropriate commandname_main() from a C file in the toys 328 directory.</p> 329 330 <p>The following global variables are defined in main.c:</p> 331 <ul> 332 <a name="toy_list" /> 333 <li><p><b>struct toy_list toy_list[]</b> - array describing all the 334 commands currently configured into toybox. The first entry (toy_list[0]) is 335 for the "toybox" multiplexer command, which runs all the other built-in commands 336 without symlinks by using its first argument as the name of the command to 337 run and the rest as that command's argument list (ala "./toybox echo hello"). 338 The remaining entries are the commands in alphabetical order (for efficient 339 binary search).</p> 340 341 <p>This is a read-only array initialized at compile time by 342 defining macros and #including generated/newtoys.h.</p> 343 344 <p>Members of struct toy_list (defined in "toys.h") include:</p> 345 <ul> 346 <li><p>char *<b>name</b> - the name of this command.</p></li> 347 <li><p>void (*<b>toy_main</b>)(void) - function pointer to run this 348 command.</p></li> 349 <li><p>char *<b>options</b> - command line option string (used by 350 get_optflags() in lib/args.c to intialize toys.optflags, toys.optargs, and 351 entries in the toy's GLOBALS struct). When this is NULL, no option 352 parsing is done before calling toy_main().</p></li> 353 <li><p>int <b>flags</b> - Behavior flags for this command. The following flags are currently understood:</p> 354 355 <ul> 356 <li><b>TOYFLAG_USR</b> - Install this command under /usr</li> 357 <li><b>TOYFLAG_BIN</b> - Install this command under /bin</li> 358 <li><b>TOYFLAG_SBIN</b> - Install this command under /sbin</li> 359 <li><b>TOYFLAG_NOFORK</b> - This command can be used as a shell builtin.</li> 360 <li><b>TOYFLAG_UMASK</b> - Call umask(0) before running this command.</li> 361 <li><b>TOYFLAG_STAYROOT</b> - Don't drop permissions for this command if toybox is installed SUID root.</li> 362 <li><b>TOYFLAG_NEEDROOT</b> - This command cannot function unless run with root access.</li> 363 </ul> 364 <br> 365 366 <p>These flags are combined with | (or). For example, to install a command 367 in /usr/bin, or together TOYFLAG_USR|TOYFLAG_BIN.</p> 368 </ul> 369 </li> 370 371 <li><p><b>struct toy_context toys</b> - global structure containing information 372 common to all commands, initializd by toy_init() and defined in "toys.h". 373 Members of this structure include:</p> 374 <ul> 375 <li><p>struct toy_list *<b>which</b> - a pointer to this command's toy_list 376 structure. Mostly used to grab the name of the running command 377 (toys->which.name).</p> 378 </li> 379 <li><p>int <b>exitval</b> - Exit value of this command. Defaults to zero. The 380 error_exit() functions will return 1 if this is zero, otherwise they'll 381 return this value.</p></li> 382 <li><p>char **<b>argv</b> - "raw" command line options, I.E. the original 383 unmodified string array passed in to main(). Note that modifying this changes 384 "ps" output, and is not recommended. This array is null terminated; a NULL 385 entry indicates the end of the array.</p> 386 <p>Most commands don't use this field, instead the use optargs, optflags, 387 and the fields in the GLOBALS struct initialized by get_optflags().</p> 388 </li> 389 <li><p>unsigned <b>optflags</b> - Command line option flags, set by 390 <a href="#lib_args">get_optflags()</a>. Indicates which of the command line options listed in 391 toys->which.options occurred this time.</p> 392 393 <p>The rightmost command line argument listed in toys->which.options sets bit 394 1, the next one sets bit 2, and so on. This means the bits are set in the same 395 order the binary digits would be listed if typed out as a string. For example, 396 the option string "abcd" would parse the command line "-c" to set optflags to 2, 397 "-a" would set optflags to 8, and "-bd" would set optflags to 6 (4|2).</p> 398 399 <p>Only letters are relevant to optflags. In the string "a*b:c#d", d=1, c=2, 400 b=4, a=8. Punctuation after a letter initializes global variables at the 401 start of the GLOBALS() block (see <a href="#toy_union">union toy_union this</a> 402 for details).</p> 403 404 <p>The build infrastructure creates FLAG_ macros for each option letter, 405 corresponding to the bit position, so you can check (toys.optflags & FLAG_x) 406 to see if a flag was specified. (The correct set of FLAG_ macros is selected 407 by defining FOR_mycommand before #including toys.h. The macros live in 408 toys/globals.h which is generated by scripts/make.sh.)</p> 409 410 <p>For more information on option parsing, see <a href="#lib_args">get_optflags()</a>.</p> 411 412 </li> 413 <li><p>char **<b>optargs</b> - Null terminated array of arguments left over 414 after get_optflags() removed all the ones it understood. Note: optarg[0] is 415 the first argument, not the command name. Use toys.which->name for the command 416 name.</p></li> 417 <li><p>int <b>optc</b> - Optarg count, equivalent to argc but for 418 optargs[].<p></li> 419 </ul> 420 421 <a name="toy_union" /> 422 <li><p><b>union toy_union this</b> - Union of structures containing each 423 command's global variables.</p> 424 425 <p>Global variables are useful: they reduce the overhead of passing extra 426 command line arguments between functions, they conveniently start prezeroed to 427 save initialization costs, and the command line argument parsing infrastructure 428 can also initialize global variables with its results.</p> 429 430 <p>But since each toybox process can only run one command at a time, allocating 431 space for global variables belonging to other commands you aren't currently 432 running would be wasteful.</p> 433 434 <p>Toybox handles this by encapsulating each command's global variables in 435 a structure, and declaring a union of those structures with a single global 436 instance (called "this"). The GLOBALS() macro contains the global 437 variables that should go in the current command's global structure. Each 438 variable can then be accessed as "this.commandname.varname". 439 If you #defined FOR_commandname before including toys.h, the macro TT is 440 #defined to this.commandname so the variable can then be accessed as 441 "TT.variable". See toys/hello.c for an example.</p> 442 443 <p>A command that needs global variables should declare a structure to 444 contain them all, and add that structure to this union. A command should never 445 declare global variables outside of this, because such global variables would 446 allocate memory when running other commands that don't use those global 447 variables.</p> 448 449 <p>The first few fields of this structure can be intialized by <a href="#lib_args">get_optargs()</a>, 450 as specified by the options field off this command's toy_list entry. See 451 the get_optargs() description in lib/args.c for details.</p> 452 </li> 453 454 <li><b>char toybuf[4096]</b> - a common scratch space buffer guaranteed 455 to start zeroed, so commands don't need to allocate/initialize their own. 456 Any command is free to use this, and it should never be directly referenced 457 by functions in lib/ (although commands are free to pass toybuf in to a 458 library function as an argument).</li> 459 460 <li><b>char libbuf[4096]</b> - like toybuf, but for use by common code in 461 lib/*.c. Commands should never directly reference libbuf, and library 462 could should nnever directly reference toybuf.</li> 463 </ul> 464 465 <p>The following functions are defined in main.c:</p> 466 <ul> 467 <li><p>struct toy_list *<b>toy_find</b>(char *name) - Return the toy_list 468 structure for this command name, or NULL if not found.</p></li> 469 <li><p>void <b>toy_init</b>(struct toy_list *which, char *argv[]) - fill out 470 the global toys structure, calling get_optargs() if necessary.</p></li> 471 <li><p>void <b>toy_exec</b>(char *argv[]) - Run a built-in command with 472 arguments.</p> 473 <p>Calls toy_find() on argv[0] (which must be just a command name 474 without path). Returns if it can't find this command, otherwise calls 475 toy_init(), toys->which.toy_main(), and exit() instead of returning.</p> 476 477 <p>Use the library function xexec() to fall back to external executables 478 in $PATH if toy_exec() can't find a built-in command. Note that toy_exec() 479 does not strip paths before searching for a command, so "./command" will 480 never match an internal command.</li> 481 482 <li><p>void <b>toybox_main</b>(void) - the main function for the multiplexer 483 command (I.E. "toybox"). Given a command name as its first argument, calls 484 toy_exec() on its arguments. With no arguments, it lists available commands. 485 If the first argument starts with "-" it lists each command with its default 486 install path prepended.</p></li> 487 488 </ul> 489 490 <h3>Config.in</h3> 491 492 <p>Top level configuration file in a stylized variant of 493 <a href=http://kernel.org/doc/Documentation/kbuild/kconfig-language.txt>kconfig</a> format. Includes generated/Config.in.</p> 494 495 <p>These files are directly used by "make menuconfig" to select which commands 496 to build into toybox (thus generating a .config file), and by 497 scripts/config2help.py to create generated/help.h.</p> 498 499 <a name="generated" /> 500 <h1><a href="#generated">Temporary files:</a></h1> 501 502 <p>There is one temporary file in the top level source directory:</p> 503 <ul> 504 <li><p><b>.config</b> - Configuration file generated by kconfig, indicating 505 which commands (and options to commands) are currently enabled. Used 506 to make generated/config.h and determine which toys/*/*.c files to build.</p> 507 508 <p>You can create a human readable "miniconfig" version of this file using 509 <a href=http://landley.net/aboriginal/new_platform.html#miniconfig>these 510 instructions</a>.</p> 511 </li> 512 </ul> 513 514 <p><h2>Directory generated/</h2></p> 515 516 <p>The remaining temporary files live in the "generated/" directory, 517 which is for files generated at build time from other source files.</p> 518 519 <ul> 520 <li><p><b>generated/Config.in</b> - Kconfig entries for each command, included 521 from the top level Config.in. The help text here is used to generate 522 help.h.</p> 523 524 <p>Each command has a configuration entry with an upper case version of 525 the command name. Options to commands start with the command 526 name followed by an underscore and the option name. Global options are attached 527 to the "toybox" command, and thus use the prefix "TOYBOX_". This organization 528 is used by scripts/cfg2files to select which toys/*/*.c files to compile for a 529 given .config.</p> 530 </li> 531 532 <li><p><b>generated/config.h</b> - list of CFG_SYMBOL and USE_SYMBOL() macros, 533 generated from .config by a sed invocation in scripts/make.sh.</p> 534 535 <p>CFG_SYMBOL is a comple time constant set to 1 for enabled symbols and 0 for 536 disabled symbols. This allows the use of normal if() statements to remove 537 code at compile time via the optimizer's dead code elimination (which removes 538 from the binary any code that cannot be reached). This saves space without 539 cluttering the code with #ifdefs or leading to configuration dependent build 540 breaks. (See the 1992 Usenix paper 541 <a href=http://doc.cat-v.org/henry_spencer/ifdef_considered_harmful.pdf>#ifdef 542 Considered Harmful</a> for more information.)</p> 543 544 <p>When you can't entirely avoid an #ifdef, the USE_SYMBOL(code) macro 545 provides a less intrusive alternative, evaluating to the code in parentheses 546 when the symbol is enabled, and nothing when the symbol is disabled. This 547 is most commonly used around NEWTOY() declarations (so only the enabled 548 commands show up in toy_list), and in option strings. This can also be used 549 for things like varargs or structure members which can't always be 550 eliminated by a simple test on CFG_SYMBOL. Remember, unlike CFG_SYMBOL 551 this is really just a variant of #ifdef, and can still result in configuration 552 dependent build breaks. Use with caution.</p> 553 </li> 554 555 <li><p><b>generated/flags.h</b> - FLAG_? macros indicating which command 556 line options were seen. The option parsing in lib/args.c sets bits in 557 toys.optflags, which can be tested by anding with the appropriate FLAG_ 558 macro. (Bare longopts, which have no corresponding short option, will 559 have the longopt name after FLAG_. All others use the single letter short 560 option.)</p> 561 562 <p>To get the appropriate macros for your command, #define FOR_commandname 563 before #including toys.h. To switch macro sets (because you have an OLDTOY() 564 with different options in the same .c file), #define CLEANUP_oldcommand 565 and also #define FOR_newcommand, then #include "generated/flags.h" to switch. 566 </p> 567 </li> 568 569 <li><p><b>generated/globals.h</b> - 570 Declares structures to hold the contents of each command's GLOBALS(), 571 and combines them into "global_union this". (Yes, the name was 572 chosen to piss off C++ developers who think that C 573 is merely a subset of C++, not a language in its own right.)</p> 574 575 <p>The union reuses the same memory for each command's global struct: 576 since only one command's globals are in use at any given time, collapsing 577 them together saves space. The headers #define TT to the appropriate 578 "this.commandname", so you can refer to the current command's global 579 variables out of "this" as TT.variablename.</p> 580 581 <p>The globals start zeroed, and the first few are filled out by the 582 lib/args.c argument parsing code called from main.c.</p> 583 </li> 584 585 <li><p><b>toys/help.h</b> - Help strings for use by the "help" command and 586 --help options. This file #defines a help_symbolname string for each 587 symbolname, but only the symbolnames matching command names get used 588 by show_help() in lib/help.c to display help for commands.</p> 589 590 <p>This file is created by scripts/make.sh, which compiles scripts/config2help.c 591 into the binary generated/config2help, and then runs it against the top 592 level .config and Config.in files to extract the help text from each config 593 entry and collate together dependent options.</p> 594 595 <p>This file contains help text for all commands, regardless of current 596 configuration, but only the ones currently enabled in the .config file 597 wind up in the help_data[] array, and only the enabled dependent options 598 have their help text added to the command they depend on.</p> 599 </li> 600 601 <li><p><b>generated/newtoys.h</b> - 602 All the NEWTOY() and OLDTOY() macros from toys/*/*.c. The "toybox" multiplexer 603 is the first entry, the rest are in alphabetical order. Each line should be 604 inside an appropriate USE_ macro, so code that #includes this file only sees 605 the currently enabled commands.</p> 606 607 <p>By #definining NEWTOY() to various things before #including this file, 608 it may be used to create function prototypes (in toys.h), initialize the 609 help_data array (in lib/help.c), initialize the toy_list array (in main.c, 610 the alphabetical order lets toy_find() do a binary search, the exception to 611 the alphabetical order lets it use the multiplexer without searching), and so 612 on. (It's even used to initialize the NEED_OPTIONS macro, which produces a 1 613 or 0 for each command using command line option parsing, which is ORed together 614 to allow compile-time dead code elimination to remove the whole of 615 lib/args.c if nothing currently enabled is using it.)<p> 616 617 <p>Each NEWTOY and OLDTOY macro contains the command name, command line 618 option string (telling lib/args.c how to parse command line options for 619 this command), recommended install location, and miscelaneous data such 620 as whether this command should retain root permissions if installed suid.</p> 621 </li> 622 623 <li><p><b>toys/oldtoys.h</b> - Macros with the command line option parsing 624 string for each NEWTOY. This allows an OLDTOY that's just an alias for an 625 existing command to refer to the existing option string instead of 626 having to repeat it.</p> 627 </li> 628 </ul> 629 630 <a name="lib"> 631 <h2>Directory lib/</h2> 632 633 <p>TODO: document lots more here.</p> 634 635 <p>lib: getmountlist(), error_msg/error_exit, xmalloc(), 636 strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(), 637 itoa().</p> 638 639 640 641 <a name="lib_xwrap"><h3>lib/xwrap.c</h3> 642 643 <p>Functions prefixed with the letter x call perror_exit() when they hit 644 errors, to eliminate common error checking. This prints an error message 645 and the strerror() string for the errno encountered.</p> 646 647 <p>We replaced exit(), _exit(), and atexit() with xexit(), _xexit(), and 648 sigatexit(). This gives _xexit() the option to siglongjmp(toys.rebound, 1) 649 instead of exiting, lets xexit() report stdout flush failures to stderr 650 and change the exit code to indicate error, lets our toys.exit function 651 change happen for signal exit paths and lets us remove the functions 652 after we've called them.</p> 653 654 <p>You can intercept our exit by assigning a setjmp/longjmp buffer to 655 toys.rebound (set it back to zero to restore the default behavior). 656 If you do this, cleaning up resource leaks is your problem.</p> 657 658 <ul> 659 <li><b>void xstrncpy(char *dest, char *src, size_t size)</b></li> 660 <li><p><b><p>void _xexit(void)</b></p> 661 <p>Calls siglongjmp(toys.rebound, 1), or else _exit(toys.exitval). This 662 lets you ignore errors with the NO_EXIT() macro wrapper, or intercept 663 them with WOULD_EXIT().</p> 664 <li><b><p>void xexit(void)</b></p> 665 <p>Calls toys.xexit functions (if any) and flushes stdout/stderr (reporting 666 failure to write to stdout both to stderr and in the exit code), then 667 calls _xexit().</p> 668 </li> 669 <li><b>void *xmalloc(size_t size)</b></li> 670 <li><b>void *xzalloc(size_t size)</b></li> 671 <li><b>void *xrealloc(void *ptr, size_t size)</b></li> 672 <li><b>char *xstrndup(char *s, size_t n)</b></li> 673 <li><b>char *xstrdup(char *s)</b></li> 674 <li><b>char *xmprintf(char *format, ...)</b></li> 675 <li><b>void xprintf(char *format, ...)</b></li> 676 <li><b>void xputs(char *s)</b></li> 677 <li><b>void xputc(char c)</b></li> 678 <li><b>void xflush(void)</b></li> 679 <li><b>pid_t xfork(void)</b></li> 680 <li><b>void xexec_optargs(int skip)</b></li> 681 <li><b>void xexec(char **argv)</b></li> 682 <li><b>pid_t xpopen(char **argv, int *pipes)</b></li> 683 <li><b>int xpclose(pid_t pid, int *pipes)</b></li> 684 <li><b>void xaccess(char *path, int flags)</b></li> 685 <li><b>void xunlink(char *path)</b></li> 686 <li><p><b>int xcreate(char *path, int flags, int mode)<br /> 687 int xopen(char *path, int flags)</b></p> 688 689 <p>The xopen() and xcreate() functions open an existing file (exiting if 690 it's not there) and create a new file (exiting if it can't).</p> 691 692 <p>They default to O_CLOEXEC so the filehandles aren't passed on to child 693 processes. Feed in O_CLOEXEC to disable this.</p> 694 </li> 695 <li><p><b>void xclose(int fd)</b></p> 696 697 <p>Because NFS is broken, and won't necessarily perform the requested 698 operation (and report the error) until you close the file. Of course, this 699 being NFS, it's not guaranteed to report the error there either, but it 700 _can_.</p> 701 702 <p>Nothing else ever reports an error on close, everywhere else it's just a 703 VFS operation freeing some resources. NFS is _special_, in a way that 704 other network filesystems like smbfs and v9fs aren't..</p> 705 </li> 706 <li><b>int xdup(int fd)</b></li> 707 <li><p><b>size_t xread(int fd, void *buf, size_t len)</b></p> 708 709 <p>Can return 0, but not -1.</p> 710 </li> 711 <li><p><b>void xreadall(int fd, void *buf, size_t len)</b></p> 712 713 <p>Reads the entire len-sized buffer, retrying to complete short 714 reads. Exits if it can't get enough data.</p></li> 715 716 <li><p><b>void xwrite(int fd, void *buf, size_t len)</b></p> 717 718 <p>Retries short writes, exits if can't write the entire buffer.</p></li> 719 720 <li><b>off_t xlseek(int fd, off_t offset, int whence)</b></li> 721 <li><b>char *xgetcwd(void)</b></li> 722 <li><b>void xstat(char *path, struct stat *st)</b></li> 723 <li><p><b>char *xabspath(char *path, int exact) </b></p> 724 725 <p>After several years of 726 <a href=http://landley.net/notes-2007.html#18-06-2007>wrestling</a> 727 <a href=http://landley.net/notes-2008.html#19-01-2008>with</a> realpath(), 728 I broke down and <a href=http://landley.net/notes-2012.html#20-11-2012>wrote 729 my own</a> implementation that doesn't use the one in libc. As I explained: 730 731 <blockquote><p>If the path ends with a broken link, 732 readlink -f should show where the link points to, not where the broken link 733 lives. (The point of readlink -f is "if I write here, where would it attempt 734 to create a file".) The problem is, realpath() returns NULL for a path ending 735 with a broken link, and I can't beat different behavior out of code locked 736 away in libc.</p></blockquote> 737 738 <p> 739 </li> 740 <li><b>void xchdir(char *path)</b></li> 741 <li><b>void xchroot(char *path)</b></li> 742 743 <li><p><b>struct passwd *xgetpwuid(uid_t uid)<br /> 744 struct group *xgetgrgid(gid_t gid)<br /> 745 struct passwd *xgetpwnam(char *name)</b></p> 746 </li> 747 748 <li><b>void xsetuser(struct passwd *pwd)</b></li> 749 <li><b>char *xreadlink(char *name)</b></li> 750 <li><b>char *xreadfile(char *name, char *buf, off_t len)</b></li> 751 <li><b>int xioctl(int fd, int request, void *data)</b></li> 752 <li><b>void xpidfile(char *name)</b></li> 753 <li><b>void xsendfile(int in, int out)</b></li> 754 <li><b>long xparsetime(char *arg, long units, long *fraction)</b></li> 755 <li><b>void xregcomp(regex_t *preg, char *regex, int cflags)</b></li> 756 </ul> 757 758 <a name="lib_lib"><h3>lib/lib.c</h3> 759 <p>Eight gazillion common functions, see lib/lib.h for the moment:</p> 760 761 <h3>lib/portability.h</h3> 762 763 <p>This file is automatically included from the top of toys.h, and smooths 764 over differences between platforms (hardware targets, compilers, C libraries, 765 operating systems, etc).</p> 766 767 <p>This file provides SWAP macros (SWAP_BE16(x) and SWAP_LE32(x) and so on).</p> 768 769 <p>A macro like SWAP_LE32(x) means "The value in x is stored as a little 770 endian 32 bit value, so perform the translation to/from whatever the native 771 32-bit format is". You do the swap once on the way in, and once on the way 772 out. If your target is already little endian, the macro is a NOP.</p> 773 774 <p>The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions. 775 In each case, the name of the macro refers to the _external_ representation, 776 and converts to/from whatever your native representation happens to be (which 777 can vary depending on what you're currently compiling for).</p> 778 779 <a name="lib_llist"><h3>lib/llist.c</h3> 780 781 <p>Some generic single and doubly linked list functions, which take 782 advantage of a couple properties of C:</p> 783 784 <ul> 785 <li><p>Structure elements are laid out in memory in the order listed, and 786 the first element has no padding. This means you can always treat (typecast) 787 a pointer to a structure as a pointer to the first element of the structure, 788 even if you don't know anything about the data following it.</p></li> 789 790 <li><p>An array of length zero at the end of a structure adds no space 791 to the sizeof() the structure, but if you calculate how much extra space 792 you want when you malloc() the structure it will be available at the end. 793 Since C has no bounds checking, this means each struct can have one variable 794 length array.</p></li> 795 </ul> 796 797 <p>Toybox's list structures always have their <b>next</b> pointer as 798 the first entry of each struct, and singly linked lists end with a NULL pointer. 799 This allows generic code to traverse such lists without knowing anything 800 else about the specific structs composing them: if your pointer isn't NULL 801 typecast it to void ** and dereference once to get the next entry.</p> 802 803 <p><b>lib/lib.h</b> defines three structure types:</p> 804 <ul> 805 <li><p><b>struct string_list</b> - stores a single string (<b>char str[0]</b>), 806 memory for which is allocated as part of the node. (I.E. llist_traverse(list, 807 free); can clean up after this type of list.)</p></li> 808 809 <li><p><b>struct arg_list</b> - stores a pointer to a single string 810 (<b>char *arg</b>) which is stored in a separate chunk of memory.</p></li> 811 812 <li><p><b>struct double_list</b> - has a second pointer (<b>struct double_list 813 *prev</b> along with a <b>char *data</b> for payload.</p></li> 814 </ul> 815 816 <b>List Functions</b> 817 818 <ul> 819 <li><p>void *<b>llist_pop</b>(void **list) - advances through a list ala 820 <b>node = llist_pop(&list);</b> This doesn't modify the list contents, 821 but does advance the pointer you feed it (which is why you pass the _address_ 822 of that pointer, not the pointer itself).</p></li> 823 824 <li><p>void <b>llist_traverse</b>(void *list, void (*using)(void *data)) - 825 iterate through a list calling a function on each node.</p></li> 826 827 <li><p>struct double_list *<b>dlist_add</b>(struct double_list **llist, char *data) 828 - append an entry to a circular linked list. 829 This function allocates a new struct double_list wrapper and returns the 830 pointer to the new entry (which you can usually ignore since it's llist->prev, 831 but if llist was NULL you need it). The argument is the ->data field for the 832 new node.</p></li> 833 <ul><li><p>void <b>dlist_add_nomalloc</b>(struct double_list **llist, 834 struct double_list *new) - append existing struct double_list to 835 list, does not allocate anything.</p></li></ul> 836 </ul> 837 838 <b>List code trivia questions:</b> 839 840 <ul> 841 <li><p><b>Why do arg_list and double_list contain a char * payload instead of 842 a void *?</b> - Because you always have to typecast a void * to use it, and 843 typecasting a char * does no harm. Since strings are the most common 844 payload, and doing math on the pointer ala 845 "(type *)(ptr+sizeof(thing)+sizeof(otherthing))" requires ptr to be char * 846 anyway (at least according to the C standard), defaulting to char * saves 847 a typecast.</p> 848 </li> 849 850 <li><p><b>Why do the names ->str, ->arg, and ->data differ?</b> - To force 851 you to keep track of which one you're using, calling free(node->str) would 852 be bad, and _failing_ to free(node->arg) leaks memory.</p></li> 853 854 <li><p><b>Why does llist_pop() take a void * instead of void **?</b> - 855 because the stupid compiler complains about "type punned pointers" when 856 you typecast and dereference on the same line, 857 due to insane FSF developers hardwiring limitations of their optimizer 858 into gcc's warning system. Since C automatically typecasts any other 859 pointer type to and from void *, the current code works fine. It's sad that it 860 won't warn you if you forget the &, but the code crashes pretty quickly in 861 that case.</p></li> 862 863 <li><p><b>How do I assemble a singly-linked-list in order?</b> - use 864 a double_list, dlist_add() your entries, and then call dlist_terminate(list) 865 to break the circle when done (turning the last ->next and the first ->prev 866 into NULLs).</p> 867 </ul> 868 869 <a name="lib_args"><h3>lib/args.c</h3> 870 871 <p>Toybox's main.c automatically parses command line options before calling the 872 command's main function. Option parsing starts in get_optflags(), which stores 873 results in the global structures "toys" (optflags and optargs) and "this".</p> 874 875 <p>The option parsing infrastructure stores a bitfield in toys.optflags to 876 indicate which options the current command line contained, and defines FLAG 877 macros code can use to check whether each argument's bit is set. Arguments 878 attached to those options are saved into the command's global structure 879 ("this"). Any remaining command line arguments are collected together into 880 the null-terminated array toys.optargs, with the length in toys.optc. (Note 881 that toys.optargs does not contain the current command name at position zero, 882 use "toys.which->name" for that.) The raw command line arguments get_optflags() 883 parsed are retained unmodified in toys.argv[].</p> 884 885 <p>Toybox's option parsing logic is controlled by an "optflags" string, using 886 a format reminiscent of getopt's optargs but with several important differences. 887 Toybox does not use the getopt() 888 function out of the C library, get_optflags() is an independent implementation 889 which doesn't permute the original arguments (and thus doesn't change how the 890 command is displayed in ps and top), and has many features not present in 891 libc optargs() (such as the ability to describe long options in the same string 892 as normal options).</p> 893 894 <p>Each command's NEWTOY() macro has an optflags string as its middle argument, 895 which sets toy_list.options for that command to tell get_optflags() what 896 command line arguments to look for, and what to do with them. 897 If a command has no option 898 definition string (I.E. the argument is NULL), option parsing is skipped 899 for that command, which must look at the raw data in toys.argv to parse its 900 own arguments. (If no currently enabled command uses option parsing, 901 get_optflags() is optimized out of the resulting binary by the compiler's 902 --gc-sections option.)</p> 903 904 <p>You don't have to free the option strings, which point into the environment 905 space (I.E. the string data is not copied). A TOYFLAG_NOFORK command 906 that uses the linked list type "*" should free the list objects but not 907 the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not 908 NOFORK, exit() will free all the malloced data anyway unless you want 909 to implement a CONFIG_TOYBOX_FREE cleanup for it.)</p> 910 911 <h4>Optflags format string</h4> 912 913 <p>Note: the optflags option description string format is much more 914 concisely described by a large comment at the top of lib/args.c.</p> 915 916 <p>The general theory is that letters set optflags, and punctuation describes 917 other actions the option parsing logic should take.</p> 918 919 <p>For example, suppose the command line <b>command -b fruit -d walrus -a 42</b> 920 is parsed using the optflags string "<b>a#b:c:d</b>". (I.E. 921 toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d", 922 "walrus", "-a", "42"]). When get_optflags() returns, the following data is 923 available to command_main(): 924 925 <ul> 926 <li><p>In <b>struct toys</b>: 927 <ul> 928 <li>toys.optflags = 13; // FLAG_a = 8 | FLAG_b = 4 | FLAG_d = 1</li> 929 <li>toys.optargs[0] = "walrus"; // leftover argument</li> 930 <li>toys.optargs[1] = NULL; // end of list</li> 931 <li>toys.optc = 1; // there was 1 leftover argument</li> 932 <li>toys.argv[] = {"-b", "fruit", "-d", "walrus", "-a", "42"}; // The original command line arguments 933 </ul> 934 <p></li> 935 936 <li><p>In <b>union this</b> (treated as <b>long this[]</b>): 937 <ul> 938 <li>this[0] = NULL; // -c didn't get an argument this time, so get_optflags() didn't change it and toys_init() zeroed "this" during setup.)</li> 939 <li>this[1] = (long)"fruit"; // argument to -b</li> 940 <li>this[2] = 42; // argument to -a</li> 941 </ul> 942 </p></li> 943 </ul> 944 945 <p>If the command's globals are:</p> 946 947 <blockquote><pre> 948 GLOBALS( 949 char *c; 950 char *b; 951 long a; 952 ) 953 </pre></blockquote> 954 955 <p>That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember, 956 each entry that receives an argument must be a long or pointer, to line up 957 with the array position. Right to left in the optflags string corresponds to 958 top to bottom in GLOBALS().</p> 959 960 <p>Put globals not filled out by the option parsing logic at the end of the 961 GLOBALS block. Common practice is to list the options one per line (to 962 make the ordering explicit, first to last in globals corresponds to right 963 to left in the option string), then leave a blank line before any non-option 964 globals.</p> 965 966 <p><b>long toys.optflags</b></p> 967 968 <p>Each option in the optflags string corresponds to a bit position in 969 toys.optflags, with the same value as a corresponding binary digit. The 970 rightmost argument is (1<<0), the next to last is (1<<1) and so on. If 971 the option isn't encountered while parsing argv[], its bit remains 0.</p> 972 973 <p>Each option -x has a FLAG_x macro for the command letter. Bare --longopts 974 with no corresponding short option have a FLAG_longopt macro for the long 975 optionname. Commands enable these macros by #defining FOR_commandname before 976 #including <toys.h>. When multiple commands are implemented in the same 977 source file, you can switch flag contexts later in the file by 978 #defining CLEANUP_oldcommand and #defining FOR_newcommand, then 979 #including <generated/flags.h>.</p> 980 981 <p>Options disabled in the current configuration (wrapped in 982 a USE_BLAH() macro for a CONFIG_BLAH that's switched off) have their 983 corresponding FLAG macro set to zero, so code checking them ala 984 if (toys.optargs & FLAG_x) gets optimized out via dead code elimination. 985 #defining FORCE_FLAGS when switching flag context disables this 986 behavior: the flag is never zero even if the config is disabled. This 987 allows code shared between multiple commands to use the same flag 988 values, as long as the common flags match up right to left in both option 989 strings.</p> 990 991 <p>For example, 992 the optflags string "abcd" would parse the command line argument "-c" to set 993 optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to 994 6 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8). To check if -c 995 was encountered, code could test: if (toys.optflags & FLAG_c) printf("yup"); 996 (See the toys/examples directory for more.)</p> 997 998 <p>Only letters are relevant to optflags, punctuation is skipped: in the 999 string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter 1000 usually indicate that the option takes an argument.</p> 1001 1002 <p>Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is 1003 the amount a long would have on 32-bit platforms anyway; 64 bit code on 1004 32 bit platforms is too expensive to require in common code used by almost 1005 all commands.) Bit positions beyond the 1<<31 aren't recorded, but 1006 parsing higher options can still set global variables.</p> 1007 1008 <p><b>Automatically setting global variables from arguments (union this)</b></p> 1009 1010 <p>The following punctuation characters may be appended to an optflags 1011 argument letter, indicating the option takes an additional argument:</p> 1012 1013 <ul> 1014 <li><b>:</b> - plus a string argument, keep most recent if more than one.</li> 1015 <li><b>*</b> - plus a string argument, appended to a linked list.</li> 1016 <li><b>@</b> - plus an occurrence counter (stored in a long)</li> 1017 <li><b>#</b> - plus a signed long argument. 1018 <li><b>-</b> - plus a signed long argument defaulting to negative (start argument with + to force a positive value).</li> 1019 <li><b>.</b> - plus a floating point argument (if CFG_TOYBOX_FLOAT).</li> 1020 <ul>The following can be appended to a float or double: 1021 <li><b><123</b> - error if argument is less than this</li> 1022 <li><b>>123</b> - error if argument is greater than this</li> 1023 <li><b>=123</b> - default value if argument not supplied</li> 1024 </ul> 1025 </ul> 1026 1027 <p><b>GLOBALS</b></p> 1028 1029 <p>Options which have an argument fill in the corresponding slot in the global 1030 union "this" (see generated/globals.h), treating it as an array of longs 1031 with the rightmost saved in this[0]. As described above, using "a*b:c#d", 1032 "-c 42" would set this[0] = 42; and "-b 42" would set this[1] = "42"; each 1033 slot is left NULL if the corresponding argument is not encountered.</p> 1034 1035 <p>This behavior is useful because the LP64 standard ensures long and pointer 1036 are the same size. C99 guarantees structure members will occur in memory 1037 in the same order they're declared, and that padding won't be inserted between 1038 consecutive variables of register size. Thus the first few entries can 1039 be longs or pointers corresponding to the saved arguments.</p> 1040 1041 <p>The main downside is that numeric arguments ("#" and "-" format) 1042 are limited to +- 2 billion on 32 bit platforms (the "truncate -s 8G" 1043 problem), because long is only 64 bits on 64 bit hosts, so the capabilities 1044 of some tools differ when built in 32 bit vs 64 bit mode. Fixing this 1045 kind of ugly and even embedded designs are slowly moving to 64 bits, 1046 so our current plan is to document the problem and wait it out. (If 1047 "x32 mode" and similar becomes popular enough, we may revisit this 1048 decision.)</p> 1049 1050 <p>See toys/example/*.c for longer examples of parsing options into the 1051 GLOBALS block.</p> 1052 1053 <p><b>char *toys.optargs[]</b></p> 1054 1055 <p>Command line arguments in argv[] which are not consumed by option parsing 1056 (I.E. not recognized either as -flags or arguments to -flags) will be copied 1057 to toys.optargs[], with the length of that array in toys.optc. 1058 (When toys.optc is 0, no unrecognized command line arguments remain.) 1059 The order of entries is preserved, and as with argv[] this new array is also 1060 terminated by a NULL entry.</p> 1061 1062 <p>Option parsing can require a minimum or maximum number of optargs left 1063 over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the 1064 start of the optflags string.</p> 1065 1066 <p>The special argument "--" terminates option parsing, storing all remaining 1067 arguments in optargs. The "--" itself is consumed.</p> 1068 1069 <p><b>Other optflags control characters</b></p> 1070 1071 <p>The following characters may occur at the start of each command's 1072 optflags string, before any options that would set a bit in toys.optflags:</p> 1073 1074 <ul> 1075 <li><b>^</b> - stop at first nonoption argument (for nice, xargs...)</li> 1076 <li><b>?</b> - allow unknown arguments (pass non-option arguments starting 1077 with - through to optargs instead of erroring out).</li> 1078 <li><b>&</b> - the first argument has imaginary dash (ala tar/ps. If given twice, all arguments have imaginary dash.)</li> 1079 <li><b><</b> - must be followed by a decimal digit indicating at least this many leftover arguments are needed in optargs (default 0)</li> 1080 <li><b>></b> - must be followed by a decimal digit indicating at most this many leftover arguments allowed (default MAX_INT)</li> 1081 </ul> 1082 1083 <p>The following characters may be appended to an option character, but do 1084 not by themselves indicate an extra argument should be saved in this[]. 1085 (Technically any character not recognized as a control character sets an 1086 optflag, but letters are never control characters.)</p> 1087 1088 <ul> 1089 <li><b>^</b> - stop parsing options after encountering this option, everything else goes into optargs.</li> 1090 <li><b>|</b> - this option is required. If more than one marked, only one is required.</li> 1091 </ul> 1092 1093 <p>The following may be appended to a float or double:</p> 1094 1095 <ul> 1096 <li><b><123</b> - error if argument is less than this</li> 1097 <li><b>>123</b> - error if argument is greater than this</li> 1098 <li><b>=123</b> - default value if argument not supplied</li> 1099 </ul> 1100 1101 <p>Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT 1102 is enabled. (Otherwise the code to determine where floating point constants 1103 end drops out. When disabled, it can reserve a global data slot for the 1104 argument so offsets won't change, but will never fill it out.) You can handle 1105 this by using the USE_BLAH() macros with C string concatenation, ala:</p> 1106 1107 <blockquote>"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"</blockquote> 1108 1109 <p><b>--longopts</b></p> 1110 1111 <p>The optflags string can contain long options, which are enclosed in 1112 parentheses. They may be appended to an existing option character, in 1113 which case the --longopt is a synonym for that option, ala "a:(--fred)" 1114 which understands "-a blah" or "--fred blah" as synonyms.</p> 1115 1116 <p>Longopts may also appear before any other options in the optflags string, 1117 in which case they have no corresponding short argument, but instead set 1118 their own bit based on position. So for "(walrus)#(blah)xy:z", "command 1119 --walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8) 1120 and would assign this[1] = 42;</p> 1121 1122 <p>A short option may have multiple longopt synonyms, "a(one)(two)", but 1123 each "bare longopt" (ala "(one)(two)abc" before any option characters) 1124 always sets its own bit (although you can group them with +X).</p> 1125 1126 <p>Only bare longopts have a FLAG_ macro with the longopt name 1127 (ala --fred would #define FLAG_fred). Other longopts use the short 1128 option's FLAG macro to test the toys.optflags bit.</p> 1129 1130 <p>Options with a semicolon ";" after their data type can only set their 1131 corresponding GLOBALS() entry via "--longopt=value". For example, option 1132 string "x(boing): y" would set TT.x if it saw "--boing=value", but would 1133 treat "--boing value" as setting FLAG_x in toys.optargs, leaving TT.x NULL, 1134 and keeping "value" in toys.optargs[]. (This lets "ls --color" and 1135 "ls --color=auto" both work.)</p> 1136 1137 <p><b>[groups]</b></p> 1138 1139 <p>At the end of the option string, square bracket groups can define 1140 relationships between existing options. (This only applies to short 1141 options, bare --longopts can't participate.)</p> 1142 1143 <p>The first character of the group defines the type, the remaining 1144 characters are options it applies to:</p> 1145 1146 <ul> 1147 <li><b>-</b> - Exclusive, switch off all others in this group.</li> 1148 <li><b>+</b> - Inclusive, switch on all others in this group.</li> 1149 <li><b>!</b> - Error, fail if more than one defined.</li> 1150 </ul> 1151 1152 <p>So "abc[-abc]" means -ab = -b, -ba = -a, -abc = -c. "abc[+abc]" 1153 means -ab=-abc, -c=-abc, and "abc[!abc] means -ab calls error_exit("no -b 1154 with -a"). Note that [-] groups clear the GLOBALS option slot of 1155 options they're switching back off, but [+] won't set options it didn't see 1156 (just the optflags).</p> 1157 1158 <p><b>whitespace</b></p> 1159 1160 <p>Arguments may occur with or without a space (I.E. "-a 42" or "-a42"). 1161 The command line argument "-abc" may be interepreted many different ways: 1162 the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4 1163 and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves 1164 "c" as the argument to -b.</p> 1165 1166 <p>Note that & changes whitespace handling, so that the command line 1167 "tar cvfCj outfile.tar.bz2 topdir filename" is parsed the same as 1168 "tar filename -c -v -j -f outfile.tar.bz2 -C topdir". Note that "tar -cvfCj 1169 one two three" would equal "tar -c -v -f Cj one two three". (This matches 1170 historical usage.)</p> 1171 1172 <p>Appending a space to the option in the option string ("a: b") makes it 1173 require a space, I.E. "-ab" is interpreted as "-a" "-b". That way "kill -stop" 1174 differs from "kill -s top".</p> 1175 1176 <p>Appending ; to a longopt in the option string makes its argument optional, 1177 and only settable with =, so in ls "(color):;" can accept "ls --color" and 1178 "ls --color=auto" without complaining that the first has no argument.</p> 1179 1180 <a name="lib_dirtree"><h3>lib/dirtree.c</h3> 1181 1182 <p>The directory tree traversal code should be sufficiently generic 1183 that commands never need to use readdir(), scandir(), or the fts.h family 1184 of functions.</p> 1185 1186 <p>These functions do not call chdir() or rely on PATH_MAX. Instead they 1187 use openat() and friends, using one filehandle per directory level to 1188 recurse into subdirectories. (I.E. they can descend 1000 directories deep 1189 if setrlimit(RLIMIT_NOFILE) allows enough open filehandles, and the default 1190 in /proc/self/limits is generally 1024.)</p> 1191 1192 <p>There are two main ways to use dirtree: 1) assemble a tree of nodes 1193 representing a snapshot of directory state and traverse them using the 1194 ->next and ->child pointers, or 2) traverse the tree calling a callback 1195 function on each entry, and freeing its node afterwards. (You can also 1196 combine the two, using the callback as a filter to determine which nodes 1197 to keep.)</p> 1198 1199 <p>The basic dirtree functions are:</p> 1200 1201 <ul> 1202 <li><p><b>struct dirtree *dirtree_read(char *path, int (*callback)(struct 1203 dirtree node))</b> - recursively read files and directories, calling 1204 callback() on each, and returning a tree of saved nodes (if any). 1205 If path doesn't exist, returns DIRTREE_ABORTVAL. If callback is NULL, 1206 returns a single node at that path.</p> 1207 1208 <li><p><b>dirtree_notdotdot(struct dirtree *new)</b> - standard callback 1209 which discards "." and ".." entries and returns DIRTREE_SAVE|DIRTREE_RECURSE 1210 for everything else. Used directly, this assembles a snapshot tree of 1211 the contents of this directory and its subdirectories 1212 to be processed after dirtree_read() returns (by traversing the 1213 struct dirtree's ->next and ->child pointers from the returned root node).</p> 1214 1215 <li><p><b>dirtree_path(struct dirtree *node, int *plen)</b> - malloc() a 1216 string containing the path from the root of this tree to this node. If 1217 plen isn't NULL then *plen is how many extra bytes to malloc at the end 1218 of string.</p></li> 1219 1220 <li><p><b>dirtree_parentfd(struct dirtree *node)</b> - return fd of 1221 directory containing this node, for use with openat() and such.</p></li> 1222 </ul> 1223 1224 <p>The <b>dirtree_read()</b> function is the standard way to start 1225 directory traversal. It takes two arguments: a starting path for 1226 the root of the tree, and a callback function. The callback() is called 1227 on each directory entry, its argument is a fully populated 1228 <b>struct dirtree *</b> (from lib/lib.h) describing the node, and its 1229 return value tells the dirtree infrastructure what to do next.</p> 1230 1231 <p>(There's also a three argument version, 1232 <b>dirtree_flagread(char *path, int flags, int (*callback)(struct 1233 dirtree node))</b>, which lets you apply flags like DIRTREE_SYMFOLLOW and 1234 DIRTREE_SHUTUP to reading the top node, but this only affects the top node. 1235 Child nodes use the flags returned by callback().</p> 1236 1237 <p><b>struct dirtree</b></p> 1238 1239 <p>Each struct dirtree node contains <b>char name[]</b> and <b>struct stat 1240 st</b> entries describing a file, plus a <b>char *symlink</b> 1241 which is NULL for non-symlinks.</p> 1242 1243 <p>During a callback function, the <b>int dirfd</b> field of directory nodes 1244 contains a directory file descriptor (for use with the openat() family of 1245 functions). This isn't usually used directly, intstead call dirtree_parentfd() 1246 on the callback's node argument. The <b>char again</a> field is 0 for the 1247 first callback on a node, and 1 on the second callback (triggered by returning 1248 DIRTREE_COMEAGAIN on a directory, made after all children have been processed). 1249 </p> 1250 1251 <p>Users of this code may put anything they like into the <b>long extra</b> 1252 field. For example, "cp" and "mv" use this to store a dirfd for the destination 1253 directory (and use DIRTREE_COMEAGAIN to get the second callback so they can 1254 close(node->extra) to avoid running out of filehandles). 1255 This field is not directly used by the dirtree code, and 1256 thanks to LP64 it's large enough to store a typecast pointer to an 1257 arbitrary struct.</p> 1258 1259 <p>The return value of the callback combines flags (with boolean or) to tell 1260 the traversal infrastructure how to behave:</p> 1261 1262 <ul> 1263 <li><p><b>DIRTREE_SAVE</b> - Save this node, assembling a tree. (Without 1264 this the struct dirtree is freed after the callback returns. Filtering out 1265 siblings is fine, but discarding a parent while keeping its child leaks 1266 memory.)</p></li> 1267 <li><p><b>DIRTREE_ABORT</b> - Do not examine any more entries in this 1268 directory. (Does not propagate up tree: to abort entire traversal, 1269 return DIRTREE_ABORT from parent callbacks too.)</p></li> 1270 <li><p><b>DIRTREE_RECURSE</b> - Examine directory contents. Ignored for 1271 non-directory entries. The remaining flags only take effect when 1272 recursing into the children of a directory.</p></li> 1273 <li><p><b>DIRTREE_COMEAGAIN</b> - Call the callback on this node a second time 1274 after examining all directory contents, allowing depth-first traversal. 1275 On the second call, dirtree->again is nonzero.</p></li> 1276 <li><p><b>DIRTREE_SYMFOLLOW</b> - follow symlinks when populating children's 1277 <b>struct stat st</b> (by feeding a nonzero value to the symfollow argument of 1278 dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to 1279 directories as directories. (Avoiding infinite recursion is the callback's 1280 problem: the non-NULL dirtree->symlink can still distinguish between 1281 them. The "find" command follows ->parent up the tree to the root node 1282 each time, checking to make sure that stat's dev and inode pair don't 1283 match any ancestors.)</p></li> 1284 </ul> 1285 1286 <p>Each struct dirtree contains three pointers (next, parent, and child) 1287 to other struct dirtree.</p> 1288 1289 <p>The <b>parent</b> pointer indicates the directory 1290 containing this entry; even when not assembling a persistent tree of 1291 nodes the parent entries remain live up to the root of the tree while 1292 child nodes are active. At the top of the tree the parent pointer is 1293 NULL, meaning the node's name[] is either an absolute path or relative 1294 to cwd. The function dirtree_parentfd() gets the directory file descriptor 1295 for use with openat() and friends, returning AT_FDCWD at the top of tree.</p> 1296 1297 <p>The <b>child</b> pointer points to the first node of the list of contents of 1298 this directory. If the directory contains no files, or the entry isn't 1299 a directory, child is NULL.</p> 1300 1301 <p>The <b>next</b> pointer indicates sibling nodes in the same directory as this 1302 node, and since it's the first entry in the struct the llist.c traversal 1303 mechanisms work to iterate over sibling nodes. Each dirtree node is a 1304 single malloc() (even char *symlink points to memory at the end of the node), 1305 so llist_free() works but its callback must descend into child nodes (freeing 1306 a tree, not just a linked list), plus whatever the user stored in extra.</p> 1307 1308 <p>The <b>dirtree_flagread</b>() function is a simple wrapper, calling <b>dirtree_add_node</b>() 1309 to create a root node relative to the current directory, then calling 1310 <b>dirtree_handle_callback</b>() on that node (which recurses as instructed by the callback 1311 return flags). The flags argument primarily lets you 1312 control whether or not to follow symlinks to the root node; symlinks 1313 listed on the command line are often treated differently than symlinks 1314 encountered during recursive directory traversal. 1315 1316 <p>The ls command not only bypasses this wrapper, but never returns 1317 <b>DIRTREE_RECURSE</b> from the callback, instead calling <b>dirtree_recurse</b>() manually 1318 from elsewhere in the program. This gives ls -lR manual control 1319 of traversal order, which is neither depth first nor breadth first but 1320 instead a sort of FIFO order requried by the ls standard.</p> 1321 1322 <a name="toys"> 1323 <h1><a href="#toys">Directory toys/</a></h1> 1324 1325 <p>This directory contains command implementations. Each command is a single 1326 self-contained file. Adding a new command involves adding a single 1327 file, and removing a command involves removing that file. Commands use 1328 shared infrastructure from the lib/ and generated/ directories.</p> 1329 1330 <p>Currently there are five subdirectories under "toys/" containing "posix" 1331 commands described in POSIX-2008, "lsb" commands described in the Linux 1332 Standard Base 4.1, "other" commands not described by either standard, 1333 "pending" commands awaiting cleanup (which default to "n" in menuconfig 1334 because they don't necessarily work right yet), and "example" code showing 1335 how toybox infrastructure works and providing template/skeleton files to 1336 start new commands.</p> 1337 1338 <p>The only difference directory location makes is which menu the command 1339 shows up in during "make menuconfig", the directories are otherwise identical. 1340 Note that the commands exist within a single namespace at runtime, so you can't 1341 have the same command in multiple subdirectories. (The build tries to fail 1342 informatively when you do that.)</p> 1343 1344 <p>There is one more sub-menus in "make menuconfig" containing global 1345 configuration options for toybox. This menu is defined in the top level 1346 Config.in.</p> 1347 1348 <p>See <a href="#adding">adding a new command</a> for details on the 1349 layout of a command file.</p> 1350 1351 <h2>Directory scripts/</h2> 1352 1353 <p>Build infrastructure. The makefile calls scripts/make.sh for "make" 1354 and scripts/install.sh for "make install".</p> 1355 1356 <p>There's also a test suite, "make test" calls make/test.sh, which runs all 1357 the tests in make/test/*. You can run individual tests via 1358 "scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run 1359 that test against the host implementation instead of the toybox one.</p> 1360 1361 <h3>scripts/cfg2files.sh</h3> 1362 1363 <p>Run .config through this filter to get a list of enabled commands, which 1364 is turned into a list of files in toys via a sed invocation in the top level 1365 Makefile. 1366 </p> 1367 1368 <h2>Directory kconfig/</h2> 1369 1370 <p>Menuconfig infrastructure copied from the Linux kernel. See the 1371 Linux kernel's Documentation/kbuild/kconfig-language.txt</p> 1372 1373 <!-- todo 1374 1375 Better OLDTOY and multiple command explanation. From Config.in: 1376 1377 <p>A command with multiple names (or multiple similar commands implemented in 1378 the same .c file) should have config symbols prefixed with the name of their 1379 C file. I.E. config symbol prefixes are NEWTOY() names. If OLDTOY() names 1380 have config symbols they must be options (symbols with an underscore and 1381 suffix) to the NEWTOY() name. (See generated/toylist.h)</p> 1382 --> 1383 1384 <!--#include file="footer.html" --> 1385