blob: a5ffa15544940768aca5eed1cd2a733fd328c10b [file] [log] [blame]
Rob Landley349ff522014-01-04 13:09:42 -06001<html><head><title>toybox source code walkthrough</title></head>
Rob Landley4e68de12007-12-13 07:00:27 -06002<!--#include file="header.html" -->
3
Rob Landley27048342013-08-18 14:24:59 -05004<p><h1><a name="style" /><a href="#style">Code style</a></h1></p>
Rob Landleye7c9a6d2012-02-28 06:34:09 -06005
6<p>The primary goal of toybox is _simple_ code. Keeping the code small is
Rob Landleyed6ed622012-03-06 20:49:03 -06007second, with speed and lots of features coming in somewhere after that.
8(For more on that, see the <a href=design.html>design</a> page.)</p>
Rob Landleye7c9a6d2012-02-28 06:34:09 -06009
10<p>A simple implementation usually takes up fewer lines of source code,
11meaning more code can fit on the screen at once, meaning the programmer can
12see more of it on the screen and thus keep more if in their head at once.
Rob Landleyed6ed622012-03-06 20:49:03 -060013This helps code auditing and thus reduces bugs. That said, sometimes being
14more explicit is preferable to being clever enough to outsmart yourself:
15don't be so terse your code is unreadable.</p>
Rob Landley5a0660f2007-12-27 21:36:44 -060016
Rob Landley7aa651a2012-11-13 17:14:08 -060017<p>Toybox source uses two spaces per indentation level, and wraps at 80
18columns.</p>
Rob Landley5a0660f2007-12-27 21:36:44 -060019
20<p>Gotos are allowed for error handling, and for breaking out of
21nested loops. In general, a goto should only jump forward (not back), and
22should either jump to the end of an outer loop, or to error handling code
23at the end of the function. Goto labels are never indented: they override the
24block structure of the file. Putting them at the left edge makes them easy
25to spot as overrides to the normal flow of control, which they are.</p>
26
Rob Landley27048342013-08-18 14:24:59 -050027<p><h1><a name="building" /><a href="#building">Building Toybox</a></h1></p>
Rob Landleye7c9a6d2012-02-28 06:34:09 -060028
29<p>Toybox is configured using the Kconfig language pioneered by the Linux
30kernel, and adopted by many other projects (uClibc, OpenEmbedded, etc).
31This generates a ".config" file containing the selected options, which
Rob Landley7aa651a2012-11-13 17:14:08 -060032controls which features are included when compiling toybox.</p>
Rob Landleye7c9a6d2012-02-28 06:34:09 -060033
34<p>Each configuration option has a default value. The defaults indicate the
35"maximum sane configuration", I.E. if the feature defaults to "n" then it
36either isn't complete or is a special-purpose option (such as debugging
37code) that isn't intended for general purpose use.</p>
38
39<p>The standard build invocation is:</p>
40
41<ul>
42<li>make defconfig #(or menuconfig)</li>
43<li>make</li>
44<li>make install</li>
45</ul>
46
47<p>Type "make help" to see all available build options.</p>
48
49<p>The file "configure" contains a number of environment variable definitions
50which influence the build, such as specifying which compiler to use or where
51to install the resulting binaries. This file is included by the build, but
52accepts existing definitions of the environment variables, so it may be sourced
53or modified by the developer before building and the definitions exported
54to the environment will take precedence.</p>
55
56<p>(To clarify: "configure" describes the build and installation environment,
57".config" lists the features selected by defconfig/menuconfig.)</p>
Rob Landley6882ee82008-02-12 18:41:34 -060058
Rob Landley27048342013-08-18 14:24:59 -050059<p><h1><a name="running"><a href="#running">Running a command</a></h1></p>
60
61<h2>main</h2>
62
63<p>The toybox main() function is at the end of main.c at the top level. It has
64two possible codepaths, only one of which is configured into any given build
65of toybox.</p>
66
67<p>If CONFIG_SINGLE is selected, toybox is configured to contain only a single
68command, so most of the normal setup can be skipped. In this case the
69multiplexer isn't used, instead main() calls toy_singleinit() (also in main.c)
70to set up global state and parse command line arguments, calls the command's
71main function out of toy_list (in the CONFIG_SINGLE case the array has a single entry, no need to search), and if the function returns instead of exiting
72it flushes stdout (detecting error) and returns toys.exitval.</p>
73
74<p>When CONFIG_SINGLE is not selected, main() uses basename() to find the
75name it was run as, shifts its argument list one to the right so it lines up
76with where the multiplexer function expects it, and calls toybox_main(). This
77leverages the multiplexer command's infrastructure to find and run the
78appropriate command. (A command name starting with "toybox" will
79recursively call toybox_main(); you can go "./toybox toybox toybox toybox ls"
80if you want to...)</p>
81
82<h2>toybox_main</h2>
83
84<p>The toybox_main() function is also in main,c. It handles a possible
85--help option ("toybox --help ls"), prints the list of available commands if no
86arguments were provided to the multiplexer (or with full path names if any
87other option is provided before a command name, ala "toybox --list").
88Otherwise it calls toy_exec() on its argument list.</p>
89
90<p>Note that the multiplexer is the first entry in toy_list (the rest of the
91list is sorted alphabetically to allow binary search), so toybox_main can
92cheat and just grab the first entry to quickly set up its context without
93searching. Since all command names go through the multiplexer at least once
94in the non-TOYBOX_SINGLE case, this avoids a redundant search of
95the list.</p>
96
97<p>The toy_exec() function is also in main.c. It performs toy_find() to
98perform a binary search on the toy_list array to look up the command's
99entry by name and saves it in the global variable which, calls toy_init()
100to parse command line arguments and set up global state (using which->options),
101and calls the appropriate command's main() function (which->toy_main). On
102return it flushes all pending ansi FILE * I/O, detects if stdout had an
103error, and then calls xexit() (which uses toys.exitval).</p>
104
105<p><h1><a name="infrastructure" /><a href="#infrastructure">Infrastructure</a></h1></p>
Rob Landley4e68de12007-12-13 07:00:27 -0600106
Rob Landley7c04f012008-01-20 19:00:16 -0600107<p>The toybox source code is in following directories:</p>
108<ul>
109<li>The <a href="#top">top level directory</a> contains the file main.c (were
110execution starts), the header file toys.h (included by every command), and
111other global infrastructure.</li>
112<li>The <a href="#lib">lib directory</a> contains common functions shared by
Rob Landleyc8d0da52012-07-15 17:47:08 -0500113multiple commands:</li>
114<ul>
115<li><a href="#lib_lib">lib/lib.c</a></li>
116<li><a href="#lib_llist">lib/llist.c</a></li>
117<li><a href="#lib_args">lib/args.c</a></li>
118<li><a href="#lib_dirtree">lib/dirtree.c</a></li>
119</ul>
Rob Landley7c04f012008-01-20 19:00:16 -0600120<li>The <a href="#toys">toys directory</a> contains the C files implementating
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500121each command. Currently it contains three subdirectories:
122posix, lsb, and other.</li>
Rob Landley7c04f012008-01-20 19:00:16 -0600123<li>The <a href="#scripts">scripts directory</a> contains the build and
124test infrastructure.</li>
125<li>The <a href="#kconfig">kconfig directory</a> contains the configuration
126infrastructure implementing menuconfig (copied from the Linux kernel).</li>
127<li>The <a href="#generated">generated directory</a> contains intermediate
128files generated from other parts of the source code.</li>
129</ul>
Rob Landley4e68de12007-12-13 07:00:27 -0600130
Rob Landleybbe500e2012-02-26 21:53:15 -0600131<a name="adding" />
Rob Landley7eaf4f52014-04-09 08:30:09 -0500132<p><h1><a href="#adding">Adding a new command</a></h1></p>
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500133<p>To add a new command to toybox, add a C file implementing that command under
Rob Landley7c04f012008-01-20 19:00:16 -0600134the toys directory. No other files need to be modified; the build extracts
Rob Landley6882ee82008-02-12 18:41:34 -0600135all the information it needs (such as command line arguments) from specially
Rob Landley7c04f012008-01-20 19:00:16 -0600136formatted comments and macros in the C file. (See the description of the
Rob Landleye7c9a6d2012-02-28 06:34:09 -0600137<a href="#generated">"generated" directory</a> for details.)</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600138
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500139<p>Currently there are three subdirectories under "toys", one for commands
140defined by the POSIX standard, one for commands defined by the Linux Standard
141Base, and one for all other commands. (This is just for developer convenience
142sorting them, the directories are otherwise functionally identical.)</p>
143
144<p>An easy way to start a new command is copy the file "toys/other/hello.c" to
Rob Landley7c04f012008-01-20 19:00:16 -0600145the name of the new command, and modify this copy to implement the new command.
Rob Landley6882ee82008-02-12 18:41:34 -0600146This file is an example command meant to be used as a "skeleton" for
Rob Landley7c04f012008-01-20 19:00:16 -0600147new commands (more or less by turning every instance of "hello" into the
148name of your command, updating the command line arguments, globals, and
149help data, and then filling out its "main" function with code that does
Rob Landley6882ee82008-02-12 18:41:34 -0600150something interesting). It provides examples of all the build infrastructure
151(including optional elements like command line argument parsing and global
152variables that a "hello world" program doesn't strictly need).</p>
Rob Landley7c04f012008-01-20 19:00:16 -0600153
154<p>Here's a checklist of steps to turn hello.c into another command:</p>
155
156<ul>
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500157<li><p>First "cd toys/other" and "cp hello.c yourcommand.c". Note that the name
Rob Landley7c04f012008-01-20 19:00:16 -0600158of this file is significant, it's the name of the new command you're adding
159to toybox. Open your new file in your favorite editor.</p></li>
160
161<li><p>Change the one line comment at the top of the file (currently
162"hello.c - A hello world program") to describe your new file.</p></li>
163
164<li><p>Change the copyright notice to your name, email, and the current
165year.</p></li>
166
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500167<li><p>Give a URL to the relevant standards document, where applicable.
168(Sample links to SUSv4 and LSB are provided, feel free to link to other
169documentation or standards as appropriate.)</p></li>
Rob Landley7c04f012008-01-20 19:00:16 -0600170
Rob Landley66a69d92012-01-16 01:44:17 -0600171<li><p>Update the USE_YOURCOMMAND(NEWTOY(yourcommand,"blah",0)) line.
172The NEWTOY macro fills out this command's <a href="#toy_list">toy_list</a>
173structure. The arguments to the NEWTOY macro are:</p>
174
175<ol>
176<li><p>the name used to run your command</p></li>
177<li><p>the command line argument <a href="#lib_args">option parsing string</a> (NULL if none)</p></li>
178<li><p>a bitfield of TOYFLAG values
179(defined in toys.h) providing additional information such as where your
180command should be installed on a running system, whether to blank umask
181before running, whether or not the command must run as root (and thus should
182retain root access if installed SUID), and so on.</p></li>
183</ol>
184</li>
Rob Landley7c04f012008-01-20 19:00:16 -0600185
186<li><p>Change the kconfig data (from "config YOURCOMMAND" to the end of the
187comment block) to supply your command's configuration and help
188information. The uppper case config symbols are used by menuconfig, and are
189also what the CFG_ and USE_() macros are generated from (see [TODO]). The
190help information here is used by menuconfig, and also by the "help" command to
191describe your new command. (See [TODO] for details.) By convention,
Rob Landley66a69d92012-01-16 01:44:17 -0600192unfinished commands default to "n" and finished commands default to "y",
193so "make defconfig" selects all finished commands. (Note, "finished" means
194"ready to be used", not that it'll never change again.)<p>
195
196<p>Each help block should start with a "usage: yourcommand" line explaining
197any command line arguments added by this config option. The "help" command
198outputs this text, and scripts/config2help.c in the build infrastructure
199collates these usage lines for commands with multiple configuration
200options when producing generated/help.h.</p>
201</li>
Rob Landley7c04f012008-01-20 19:00:16 -0600202
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500203<li><p>Change the "#define FOR_hello" line to "#define FOR_yourcommand" right
204before the "#include <toys.h>". (This selects the appropriate FLAG_ macros and
205does a "#define TT this.yourcommand" so you can access the global variables
206out of the space-saving union of structures. If you aren't using any command
207flag bits and aren't defining a GLOBAL block, you can delete this line.)</p></li>
Rob Landley7c04f012008-01-20 19:00:16 -0600208
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500209<li><p>Update the GLOBALS() macro to contain your command's global
210variables. If your command has no global variables, delete this macro.</p>
211
212<p>Variables in the GLOBALS() block are are stored in a space saving
213<a href="#toy_union">union of structures</a> format, which may be accessed
214using the TT macro as if TT were a global structure (so TT.membername).
215If you specified two-character command line arguments in
216NEWTOY(), the first few global variables will be initialized by the automatic
217argument parsing logic, and the type and order of these variables must
218correspond to the arguments specified in NEWTOY().
219(See <a href="#lib_args">lib/args.c</a> for details.)</p></li>
Rob Landley7c04f012008-01-20 19:00:16 -0600220
221<li><p>Rename hello_main() to yourcommand_main(). This is the main() function
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500222where execution of your command starts. Your command line options are
223already sorted into this.optflags, this.optargs, this.optc, and the GLOBALS()
224as appropriate by the time this function is called. (See
225<a href="#lib_args">get_optflags()</a> for details.</p></li>
Rob Landley7c04f012008-01-20 19:00:16 -0600226</ul>
227
Rob Landley7eaf4f52014-04-09 08:30:09 -0500228<a name="headers" /><h2><a href="#headers">Headers.</a></h2>
Rob Landley85a32412013-12-27 06:53:15 -0600229
230<p>Commands generally don't have their own headers. If it's common code
231it can live in lib/, if it isn't put it in the command's .c file. (The line
232between implementing multiple commands in a C file via OLDTOY() to share
233infrastructure and moving that shared infrastructure to lib/ is a judgement
234call. Try to figure out which is simplest.)</p>
235
236<p>The top level toys.h should #include all the standard (posix) headers
237that any command uses. (Partly this is friendly to ccache and partly this
238makes the command implementations shorter.) Individual commands should only
239need to include nonstandard headers that might prevent that command from
240building in some context we'd care about (and thus requiring that command to
241be disabled to avoid a build break).</p>
242
243<p>Target-specific stuff (differences between compiler versions, libc versions,
244or operating systems) should be confined to lib/portability.h and
245lib/portability.c. (There's even some minimal compile-time environment probing
246that writes data to generated/portability.h, see scripts/genconfig.sh.)</p>
247
248<p>Only include linux/*.h headers from individual commands (not from other
249headers), and only if you really need to. Data that varies per architecture
250is a good reason to include a header. If you just need a couple constants
251that haven't changed since the 1990's, it's ok to #define them yourself or
252just use the constant inline with a comment explaining what it is. (A
253#define that's only used once isn't really helping.)</p>
254
Rob Landley7eaf4f52014-04-09 08:30:09 -0500255<p><a name="top" /><h1><a href="#top">Top level directory.</a></h1></p>
Rob Landley7c04f012008-01-20 19:00:16 -0600256
Rob Landley66a69d92012-01-16 01:44:17 -0600257<p>This directory contains global infrastructure.</p>
258
259<h3>toys.h</h3>
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500260<p>Each command #includes "toys.h" as part of its standard prolog. It
261may "#define FOR_commandname" before doing so to get some extra entries
262specific to this command.</p>
Rob Landley66a69d92012-01-16 01:44:17 -0600263
264<p>This file sucks in most of the commonly used standard #includes, so
265individual files can just #include "toys.h" and not have to worry about
266stdargs.h and so on. Individual commands still need to #include
267special-purpose headers that may not be present on all systems (and thus would
268prevent toybox from building that command on such a system with that command
269enabled). Examples include regex support, any "linux/" or "asm/" headers, mtab
270support (mntent.h and sys/mount.h), and so on.</p>
271
272<p>The toys.h header also defines structures for most of the global variables
273provided to each command by toybox_main(). These are described in
274detail in the description for main.c, where they are initialized.</p>
275
276<p>The global variables are grouped into structures (and a union) for space
277savings, to more easily track the amount of memory consumed by them,
278so that they may be automatically cleared/initialized as needed, and so
279that access to global variables is more easily distinguished from access to
280local variables.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600281
282<h3>main.c</h3>
283<p>Contains the main() function where execution starts, plus
284common infrastructure to initialize global variables and select which command
Rob Landley6882ee82008-02-12 18:41:34 -0600285to run. The "toybox" multiplexer command also lives here. (This is the
Rob Landley7c04f012008-01-20 19:00:16 -0600286only command defined outside of the toys directory.)</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600287
Rob Landley6882ee82008-02-12 18:41:34 -0600288<p>Execution starts in main() which trims any path off of the first command
289name and calls toybox_main(), which calls toy_exec(), which calls toy_find()
Rob Landley66a69d92012-01-16 01:44:17 -0600290and toy_init() before calling the appropriate command's function from
291toy_list[] (via toys.which->toy_main()).
Rob Landley6882ee82008-02-12 18:41:34 -0600292If the command is "toybox", execution recurses into toybox_main(), otherwise
293the call goes to the appropriate commandname_main() from a C file in the toys
294directory.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600295
Rob Landley6882ee82008-02-12 18:41:34 -0600296<p>The following global variables are defined in main.c:</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600297<ul>
Rob Landley66a69d92012-01-16 01:44:17 -0600298<a name="toy_list" />
299<li><p><b>struct toy_list toy_list[]</b> - array describing all the
Rob Landley4e68de12007-12-13 07:00:27 -0600300commands currently configured into toybox. The first entry (toy_list[0]) is
301for the "toybox" multiplexer command, which runs all the other built-in commands
302without symlinks by using its first argument as the name of the command to
303run and the rest as that command's argument list (ala "./toybox echo hello").
304The remaining entries are the commands in alphabetical order (for efficient
305binary search).</p>
306
307<p>This is a read-only array initialized at compile time by
Rob Landley6882ee82008-02-12 18:41:34 -0600308defining macros and #including generated/newtoys.h.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600309
Rob Landley66a69d92012-01-16 01:44:17 -0600310<p>Members of struct toy_list (defined in "toys.h") include:</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600311<ul>
312<li><p>char *<b>name</b> - the name of this command.</p></li>
313<li><p>void (*<b>toy_main</b>)(void) - function pointer to run this
314command.</p></li>
315<li><p>char *<b>options</b> - command line option string (used by
316get_optflags() in lib/args.c to intialize toys.optflags, toys.optargs, and
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500317entries in the toy's GLOBALS struct). When this is NULL, no option
Rob Landley66a69d92012-01-16 01:44:17 -0600318parsing is done before calling toy_main().</p></li>
Rob Landley6882ee82008-02-12 18:41:34 -0600319<li><p>int <b>flags</b> - Behavior flags for this command. The following flags are currently understood:</p>
320
321<ul>
322<li><b>TOYFLAG_USR</b> - Install this command under /usr</li>
323<li><b>TOYFLAG_BIN</b> - Install this command under /bin</li>
324<li><b>TOYFLAG_SBIN</b> - Install this command under /sbin</li>
325<li><b>TOYFLAG_NOFORK</b> - This command can be used as a shell builtin.</li>
326<li><b>TOYFLAG_UMASK</b> - Call umask(0) before running this command.</li>
Rob Landley66a69d92012-01-16 01:44:17 -0600327<li><b>TOYFLAG_STAYROOT</b> - Don't drop permissions for this command if toybox is installed SUID root.</li>
328<li><b>TOYFLAG_NEEDROOT</b> - This command cannot function unless run with root access.</li>
Rob Landley6882ee82008-02-12 18:41:34 -0600329</ul>
330<br>
331
332<p>These flags are combined with | (or). For example, to install a command
333in /usr/bin, or together TOYFLAG_USR|TOYFLAG_BIN.</p>
334</ul>
Rob Landley4e68de12007-12-13 07:00:27 -0600335</li>
336
Rob Landley66a69d92012-01-16 01:44:17 -0600337<li><p><b>struct toy_context toys</b> - global structure containing information
338common to all commands, initializd by toy_init() and defined in "toys.h".
339Members of this structure include:</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600340<ul>
341<li><p>struct toy_list *<b>which</b> - a pointer to this command's toy_list
342structure. Mostly used to grab the name of the running command
343(toys->which.name).</p>
344</li>
345<li><p>int <b>exitval</b> - Exit value of this command. Defaults to zero. The
346error_exit() functions will return 1 if this is zero, otherwise they'll
347return this value.</p></li>
348<li><p>char **<b>argv</b> - "raw" command line options, I.E. the original
349unmodified string array passed in to main(). Note that modifying this changes
Rob Landley66a69d92012-01-16 01:44:17 -0600350"ps" output, and is not recommended. This array is null terminated; a NULL
351entry indicates the end of the array.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600352<p>Most commands don't use this field, instead the use optargs, optflags,
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500353and the fields in the GLOBALS struct initialized by get_optflags().</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600354</li>
355<li><p>unsigned <b>optflags</b> - Command line option flags, set by
Rob Landley66a69d92012-01-16 01:44:17 -0600356<a href="#lib_args">get_optflags()</a>. Indicates which of the command line options listed in
Rob Landley6882ee82008-02-12 18:41:34 -0600357toys->which.options occurred this time.</p>
358
359<p>The rightmost command line argument listed in toys->which.options sets bit
3601, the next one sets bit 2, and so on. This means the bits are set in the same
361order the binary digits would be listed if typed out as a string. For example,
362the option string "abcd" would parse the command line "-c" to set optflags to 2,
363"-a" would set optflags to 8, and "-bd" would set optflags to 6 (4|2).</p>
364
365<p>Only letters are relevant to optflags. In the string "a*b:c#d", d=1, c=2,
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500366b=4, a=8. Punctuation after a letter initializes global variables at the
367start of the GLOBALS() block (see <a href="#toy_union">union toy_union this</a>
368for details).</p>
369
370<p>The build infrastructure creates FLAG_ macros for each option letter,
371corresponding to the bit position, so you can check (toys.optflags & FLAG_x)
372to see if a flag was specified. (The correct set of FLAG_ macros is selected
373by defining FOR_mycommand before #including toys.h. The macros live in
374toys/globals.h which is generated by scripts/make.sh.)</p>
Rob Landley6882ee82008-02-12 18:41:34 -0600375
Rob Landley66a69d92012-01-16 01:44:17 -0600376<p>For more information on option parsing, see <a href="#lib_args">get_optflags()</a>.</p>
Rob Landley6882ee82008-02-12 18:41:34 -0600377
378</li>
Rob Landley4e68de12007-12-13 07:00:27 -0600379<li><p>char **<b>optargs</b> - Null terminated array of arguments left over
380after get_optflags() removed all the ones it understood. Note: optarg[0] is
381the first argument, not the command name. Use toys.which->name for the command
382name.</p></li>
Rob Landley6882ee82008-02-12 18:41:34 -0600383<li><p>int <b>optc</b> - Optarg count, equivalent to argc but for
384optargs[].<p></li>
Rob Landley4e68de12007-12-13 07:00:27 -0600385<li><p>int <b>exithelp</b> - Whether error_exit() should print a usage message
386via help_main() before exiting. (True during option parsing, defaults to
387false afterwards.)</p></li>
Rob Landley66a69d92012-01-16 01:44:17 -0600388</ul>
Rob Landley4e68de12007-12-13 07:00:27 -0600389
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500390<a name="toy_union" />
Rob Landley66a69d92012-01-16 01:44:17 -0600391<li><p><b>union toy_union this</b> - Union of structures containing each
Rob Landley4e68de12007-12-13 07:00:27 -0600392command's global variables.</p>
393
Rob Landley6882ee82008-02-12 18:41:34 -0600394<p>Global variables are useful: they reduce the overhead of passing extra
395command line arguments between functions, they conveniently start prezeroed to
396save initialization costs, and the command line argument parsing infrastructure
397can also initialize global variables with its results.</p>
398
399<p>But since each toybox process can only run one command at a time, allocating
400space for global variables belonging to other commands you aren't currently
401running would be wasteful.</p>
402
403<p>Toybox handles this by encapsulating each command's global variables in
Rob Landley66a69d92012-01-16 01:44:17 -0600404a structure, and declaring a union of those structures with a single global
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500405instance (called "this"). The GLOBALS() macro contains the global
Rob Landley66a69d92012-01-16 01:44:17 -0600406variables that should go in the current command's global structure. Each
407variable can then be accessed as "this.commandname.varname".
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500408If you #defined FOR_commandname before including toys.h, the macro TT is
409#defined to this.commandname so the variable can then be accessed as
410"TT.variable". See toys/hello.c for an example.</p>
Rob Landley6882ee82008-02-12 18:41:34 -0600411
Rob Landley66a69d92012-01-16 01:44:17 -0600412<p>A command that needs global variables should declare a structure to
Rob Landley4e68de12007-12-13 07:00:27 -0600413contain them all, and add that structure to this union. A command should never
414declare global variables outside of this, because such global variables would
415allocate memory when running other commands that don't use those global
416variables.</p>
417
Rob Landley66a69d92012-01-16 01:44:17 -0600418<p>The first few fields of this structure can be intialized by <a href="#lib_args">get_optargs()</a>,
Rob Landley4e68de12007-12-13 07:00:27 -0600419as specified by the options field off this command's toy_list entry. See
420the get_optargs() description in lib/args.c for details.</p>
421</li>
422
Rob Landley81b899d2007-12-18 02:02:47 -0600423<li><b>char toybuf[4096]</b> - a common scratch space buffer so
Rob Landley4e68de12007-12-13 07:00:27 -0600424commands don't need to allocate their own. Any command is free to use this,
425and it should never be directly referenced by functions in lib/ (although
426commands are free to pass toybuf in to a library function as an argument).</li>
427</ul>
428
Rob Landley6882ee82008-02-12 18:41:34 -0600429<p>The following functions are defined in main.c:</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600430<ul>
431<li><p>struct toy_list *<b>toy_find</b>(char *name) - Return the toy_list
432structure for this command name, or NULL if not found.</p></li>
Rob Landley81b899d2007-12-18 02:02:47 -0600433<li><p>void <b>toy_init</b>(struct toy_list *which, char *argv[]) - fill out
434the global toys structure, calling get_optargs() if necessary.</p></li>
Rob Landley6882ee82008-02-12 18:41:34 -0600435<li><p>void <b>toy_exec</b>(char *argv[]) - Run a built-in command with
436arguments.</p>
437<p>Calls toy_find() on argv[0] (which must be just a command name
Rob Landley4e68de12007-12-13 07:00:27 -0600438without path). Returns if it can't find this command, otherwise calls
Rob Landley6882ee82008-02-12 18:41:34 -0600439toy_init(), toys->which.toy_main(), and exit() instead of returning.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600440
Rob Landley6882ee82008-02-12 18:41:34 -0600441<p>Use the library function xexec() to fall back to external executables
442in $PATH if toy_exec() can't find a built-in command. Note that toy_exec()
443does not strip paths before searching for a command, so "./command" will
444never match an internal command.</li>
445
446<li><p>void <b>toybox_main</b>(void) - the main function for the multiplexer
447command (I.E. "toybox"). Given a command name as its first argument, calls
448toy_exec() on its arguments. With no arguments, it lists available commands.
449If the first argument starts with "-" it lists each command with its default
450install path prepended.</p></li>
Rob Landley4e68de12007-12-13 07:00:27 -0600451
452</ul>
453
454<h3>Config.in</h3>
455
456<p>Top level configuration file in a stylized variant of
Rob Landley6882ee82008-02-12 18:41:34 -0600457<a href=http://kernel.org/doc/Documentation/kbuild/kconfig-language.txt>kconfig</a> format. Includes generated/Config.in.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600458
459<p>These files are directly used by "make menuconfig" to select which commands
460to build into toybox (thus generating a .config file), and by
Rob Landley6882ee82008-02-12 18:41:34 -0600461scripts/config2help.py to create generated/help.h.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600462
Rob Landley7eaf4f52014-04-09 08:30:09 -0500463<a name="generated" />
464<h1><a href="#generated">Temporary files:</a></h1>
Rob Landley4e68de12007-12-13 07:00:27 -0600465
Rob Landley6882ee82008-02-12 18:41:34 -0600466<p>There is one temporary file in the top level source directory:</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600467<ul>
468<li><p><b>.config</b> - Configuration file generated by kconfig, indicating
469which commands (and options to commands) are currently enabled. Used
Rob Landley7eaf4f52014-04-09 08:30:09 -0500470to make generated/config.h and determine which toys/*/*.c files to build.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600471
Rob Landley6882ee82008-02-12 18:41:34 -0600472<p>You can create a human readable "miniconfig" version of this file using
Rob Landley66a69d92012-01-16 01:44:17 -0600473<a href=http://landley.net/aboriginal/new_platform.html#miniconfig>these
Rob Landley6882ee82008-02-12 18:41:34 -0600474instructions</a>.</p>
475</li>
476</ul>
477
Rob Landley7eaf4f52014-04-09 08:30:09 -0500478<p><h2>Directory generated/</h2></p>
479
480<p>The remaining temporary files live in the "generated/" directory,
481which is for files generated at build time from other source files.</p>
Rob Landley6882ee82008-02-12 18:41:34 -0600482
483<ul>
Rob Landley7eaf4f52014-04-09 08:30:09 -0500484<li><p><b>generated/Config.in</b> - Included from the top level Config.in,
485contains one or more configuration entries for each command.</p>
486
487<p>Each command has a configuration entry with an upper case version of
488the command name. Options to commands start with the command
489name followed by an underscore and the option name. Global options are attached
490to the "toybox" command, and thus use the prefix "TOYBOX_". This organization
491is used by scripts/cfg2files to select which toys/*/*.c files to compile for a
492given .config.</p>
493
494<p>A command with multiple names (or multiple similar commands implemented in
495the same .c file) should have config symbols prefixed with the name of their
496C file. I.E. config symbol prefixes are NEWTOY() names. If OLDTOY() names
497have config symbols they must be options (symbols with an underscore and
498suffix) to the NEWTOY() name. (See generated/toylist.h)</p>
499</li>
500
Rob Landley6882ee82008-02-12 18:41:34 -0600501<li><p><b>generated/config.h</b> - list of CFG_SYMBOL and USE_SYMBOL() macros,
Rob Landley4e68de12007-12-13 07:00:27 -0600502generated from .config by a sed invocation in the top level Makefile.</p>
503
504<p>CFG_SYMBOL is a comple time constant set to 1 for enabled symbols and 0 for
Rob Landley7eaf4f52014-04-09 08:30:09 -0500505disabled symbols. This allows the use of normal if() statements to remove
Rob Landley6882ee82008-02-12 18:41:34 -0600506code at compile time via the optimizer's dead code elimination (which removes
Rob Landley7eaf4f52014-04-09 08:30:09 -0500507from the binary any code that cannot be reached). This saves space without
Rob Landley4e68de12007-12-13 07:00:27 -0600508cluttering the code with #ifdefs or leading to configuration dependent build
Rob Landley7eaf4f52014-04-09 08:30:09 -0500509breaks. (See the 1992 Usenix paper
Rob Landleyb6063de2012-01-29 13:54:13 -0600510<a href=http://doc.cat-v.org/henry_spencer/ifdef_considered_harmful.pdf>#ifdef
Rob Landley4e68de12007-12-13 07:00:27 -0600511Considered Harmful</a> for more information.)</p>
512
513<p>USE_SYMBOL(code) evaluates to the code in parentheses when the symbol
Rob Landley7eaf4f52014-04-09 08:30:09 -0500514is enabled, and nothing when the symbol is disabled. This can be used
Rob Landley4e68de12007-12-13 07:00:27 -0600515for things like varargs or variable declarations which can't always be
Rob Landley7eaf4f52014-04-09 08:30:09 -0500516eliminated by a simple test on CFG_SYMBOL. Note that
Rob Landley4e68de12007-12-13 07:00:27 -0600517(unlike CFG_SYMBOL) this is really just a variant of #ifdef, and can
Rob Landley7eaf4f52014-04-09 08:30:09 -0500518still result in configuration dependent build breaks. Use with caution.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600519</li>
Rob Landley4e68de12007-12-13 07:00:27 -0600520
Rob Landley7eaf4f52014-04-09 08:30:09 -0500521<li><p><b>generated/flags.h</b> - FLAG_? macros indicating which command
522line options were seen. The option parsing in lib/args.c sets bits in
523toys.optflags, which can be tested by anding with the appropriate FLAG_
524macro. (Bare longopts, which have no corresponding short option, will
525have the longopt name after FLAG_. All others use the single letter short
526option.)</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600527
Rob Landley7eaf4f52014-04-09 08:30:09 -0500528<p>To get the appropriate macros for your command, #define FOR_commandname
529before #including toys.h. To switch macro sets (because you have an OLDTOY()
530with different options in the same .c file), #define CLEANUP_oldcommand
531and also #define FOR_newcommand, then #include "generated/flags.h" to switch.
532</p>
533</li>
Rob Landley4e68de12007-12-13 07:00:27 -0600534
Rob Landley7eaf4f52014-04-09 08:30:09 -0500535<li><p><b>generated/globals.h</b> -
536Declares structures to hold the contents of each command's GLOBALS(),
537and combines them into "global_union this". (Yes, the name was
538chosen to piss off C++ developers who think that C
539is merely a subset of C++, not a language in its own right.)</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600540
Rob Landley7eaf4f52014-04-09 08:30:09 -0500541<p>The union reuses the same memory for each command's global struct:
542since only one command's globals are in use at any given time, collapsing
543them together saves space. The headers #define TT to the appropriate
544"this.commandname", so you can refer to the current command's global
545variables out of "this" as TT.variablename.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600546
Rob Landley7eaf4f52014-04-09 08:30:09 -0500547<p>The globals start zeroed, and the first few are filled out by the
548lib/args.c argument parsing code called from main.c.</p>
549</li>
Rob Landley4e68de12007-12-13 07:00:27 -0600550
Rob Landley7eaf4f52014-04-09 08:30:09 -0500551<li><p><b>toys/help.h</b> -
552#defines two help text strings for each command: a single line
Rob Landley4e68de12007-12-13 07:00:27 -0600553command_help and an additinal command_help_long. This is used by help_main()
554in toys/help.c to display help for commands.</p>
555
Rob Landley7eaf4f52014-04-09 08:30:09 -0500556<p>This file is created by scripts/make.sh, which compiles scripts/config2help.c
557into the binary generated/config2help, and then runs it against the top
558level .config and Config.in files to extract the help text from each config
559entry and collate together dependent options.</p>
Rob Landley4e68de12007-12-13 07:00:27 -0600560
Rob Landley7eaf4f52014-04-09 08:30:09 -0500561<p>This file contains help text for all commands, regardless of current
562configuration, but only the ones currently enabled in the .config file
563wind up in the help_data[] array, and only the enabled dependent options
564have their help text added to the command they depend on.</p>
565</li>
566
567<li><p><b>generated/newtoys.h</b> -
568All the NEWTOY() and OLDTOY() macros in alphabetical order,
569each of which should be inside the appropriate USE_ macro. (Ok, not _quite_
570alphabetical orer: the "toybox" multiplexer is always the first entry.)</p>
571
572<p>By #definining NEWTOY() to various things before #including this file,
573it may be used to create function prototypes (in toys.h), initialize the
574toy_list array (in main.c, the alphabetical order lets toy_find() do a
575binary search), initialize the help_data array (in lib/help.c), and so on.
576(It's even used to initialize the NEED_OPTIONS macro, which is has a 1 or 0
577for each command using command line option parsing, ORed together.
578This allows compile-time dead code elimination to remove the whole of
579lib/args.c if nothing currently enabled is using it.)<p>
580
581<p>Each NEWTOY and OLDTOY macro contains the command name, command line
582option string (telling lib/args.c how to parse command line options for
583this command), recommended install location, and miscelaneous data such
584as whether this command should retain root permissions if installed suid.</p>
585</li>
586
587<li><p><b>toys/oldtoys.h</b> - Macros with the command line option parsing
588string for each NEWTOY. This allows an OLDTOY that's just an alias for an
589existing command to refer to the existing option string instead of
590having to repeat it.</p>
591</li>
592</ul>
Rob Landley4e68de12007-12-13 07:00:27 -0600593
Rob Landley137bf342012-03-09 08:33:57 -0600594<a name="lib">
Rob Landley81b899d2007-12-18 02:02:47 -0600595<h2>Directory lib/</h2>
Rob Landley4e68de12007-12-13 07:00:27 -0600596
Rob Landley137bf342012-03-09 08:33:57 -0600597<p>TODO: document lots more here.</p>
598
Rob Landleyc8d0da52012-07-15 17:47:08 -0500599<p>lib: getmountlist(), error_msg/error_exit, xmalloc(),
Rob Landley7c04f012008-01-20 19:00:16 -0600600strlcpy(), xexec(), xopen()/xread(), xgetcwd(), xabspath(), find_in_path(),
601itoa().</p>
602
Rob Landley137bf342012-03-09 08:33:57 -0600603<h3>lib/portability.h</h3>
604
605<p>This file is automatically included from the top of toys.h, and smooths
606over differences between platforms (hardware targets, compilers, C libraries,
607operating systems, etc).</p>
608
609<p>This file provides SWAP macros (SWAP_BE16(x) and SWAP_LE32(x) and so on).</p>
610
611<p>A macro like SWAP_LE32(x) means "The value in x is stored as a little
612endian 32 bit value, so perform the translation to/from whatever the native
61332-bit format is". You do the swap once on the way in, and once on the way
614out. If your target is already little endian, the macro is a NOP.</p>
615
616<p>The SWAP macros come in BE and LE each with 16, 32, and 64 bit versions.
617In each case, the name of the macro refers to the _external_ representation,
618and converts to/from whatever your native representation happens to be (which
619can vary depending on what you're currently compiling for).</p>
620
Rob Landleyc8d0da52012-07-15 17:47:08 -0500621<a name="lib_llist"><h3>lib/llist.c</h3>
622
623<p>Some generic single and doubly linked list functions, which take
624advantage of a couple properties of C:</p>
625
626<ul>
627<li><p>Structure elements are laid out in memory in the order listed, and
628the first element has no padding. This means you can always treat (typecast)
629a pointer to a structure as a pointer to the first element of the structure,
630even if you don't know anything about the data following it.</p></li>
631
632<li><p>An array of length zero at the end of a structure adds no space
633to the sizeof() the structure, but if you calculate how much extra space
634you want when you malloc() the structure it will be available at the end.
635Since C has no bounds checking, this means each struct can have one variable
636length array.</p></li>
637</ul>
638
639<p>Toybox's list structures always have their <b>next</b> pointer as
640the first entry of each struct, and singly linked lists end with a NULL pointer.
641This allows generic code to traverse such lists without knowing anything
642else about the specific structs composing them: if your pointer isn't NULL
643typecast it to void ** and dereference once to get the next entry.</p>
644
645<p><b>lib/lib.h</b> defines three structure types:</p>
646<ul>
647<li><p><b>struct string_list</b> - stores a single string (<b>char str[0]</b>),
648memory for which is allocated as part of the node. (I.E. llist_traverse(list,
649free); can clean up after this type of list.)</p></li>
650
651<li><p><b>struct arg_list</b> - stores a pointer to a single string
652(<b>char *arg</b>) which is stored in a separate chunk of memory.</p></li>
653
654<li><p><b>struct double_list</b> - has a second pointer (<b>struct double_list
655*prev</b> along with a <b>char *data</b> for payload.</p></li>
656</ul>
657
658<b>List Functions</b>
659
660<ul>
661<li><p>void *<b>llist_pop</b>(void **list) - advances through a list ala
662<b>node = llist_pop(&list);</b> This doesn't modify the list contents,
663but does advance the pointer you feed it (which is why you pass the _address_
664of that pointer, not the pointer itself).</p></li>
665
666<li><p>void <b>llist_traverse</b>(void *list, void (*using)(void *data)) -
667iterate through a list calling a function on each node.</p></li>
668
669<li><p>struct double_list *<b>dlist_add</b>(struct double_list **llist, char *data)
670- append an entry to a circular linked list.
671This function allocates a new struct double_list wrapper and returns the
672pointer to the new entry (which you can usually ignore since it's llist->prev,
673but if llist was NULL you need it). The argument is the ->data field for the
674new node.</p></li>
675<ul><li><p>void <b>dlist_add_nomalloc</b>(struct double_list **llist,
676struct double_list *new) - append existing struct double_list to
677list, does not allocate anything.</p></li></ul>
678</ul>
679
Rob Landley7eaf4f52014-04-09 08:30:09 -0500680<b>List code trivia questions:</b>
Rob Landleyc8d0da52012-07-15 17:47:08 -0500681
682<ul>
683<li><p><b>Why do arg_list and double_list contain a char * payload instead of
684a void *?</b> - Because you always have to typecast a void * to use it, and
Rob Landley7eaf4f52014-04-09 08:30:09 -0500685typecasting a char * does no harm. Since strings are the most common
686payload, and doing math on the pointer ala
687"(type *)(ptr+sizeof(thing)+sizeof(otherthing))" requires ptr to be char *
688anyway (at least according to the C standard), defaulting to char * saves
689a typecast.</p>
Rob Landleyc8d0da52012-07-15 17:47:08 -0500690</li>
691
692<li><p><b>Why do the names ->str, ->arg, and ->data differ?</b> - To force
693you to keep track of which one you're using, calling free(node->str) would
694be bad, and _failing_ to free(node->arg) leaks memory.</p></li>
695
696<li><p><b>Why does llist_pop() take a void * instead of void **?</b> -
697because the stupid compiler complains about "type punned pointers" when
Rob Landley7eaf4f52014-04-09 08:30:09 -0500698you typecast and dereference on the same line,
Rob Landleyc8d0da52012-07-15 17:47:08 -0500699due to insane FSF developers hardwiring limitations of their optimizer
700into gcc's warning system. Since C automatically typecasts any other
Rob Landley7eaf4f52014-04-09 08:30:09 -0500701pointer type to and from void *, the current code works fine. It's sad that it
Rob Landleyc8d0da52012-07-15 17:47:08 -0500702won't warn you if you forget the &, but the code crashes pretty quickly in
703that case.</p></li>
704
705<li><p><b>How do I assemble a singly-linked-list in order?</b> - use
706a double_list, dlist_add() your entries, and then break the circle with
707<b>list->prev->next = NULL;</b> when done.</li>
708</ul>
709
Rob Landley66a69d92012-01-16 01:44:17 -0600710<a name="lib_args"><h3>lib/args.c</h3>
Rob Landley7c04f012008-01-20 19:00:16 -0600711
Rob Landley66a69d92012-01-16 01:44:17 -0600712<p>Toybox's main.c automatically parses command line options before calling the
713command's main function. Option parsing starts in get_optflags(), which stores
714results in the global structures "toys" (optflags and optargs) and "this".</p>
715
716<p>The option parsing infrastructure stores a bitfield in toys.optflags to
717indicate which options the current command line contained. Arguments
718attached to those options are saved into the command's global structure
719("this"). Any remaining command line arguments are collected together into
720the null-terminated array toys.optargs, with the length in toys.optc. (Note
721that toys.optargs does not contain the current command name at position zero,
722use "toys.which->name" for that.) The raw command line arguments get_optflags()
723parsed are retained unmodified in toys.argv[].</p>
724
725<p>Toybox's option parsing logic is controlled by an "optflags" string, using
726a format reminiscent of getopt's optargs but has several important differences.
727Toybox does not use the getopt()
728function out of the C library, get_optflags() is an independent implementation
729which doesn't permute the original arguments (and thus doesn't change how the
730command is displayed in ps and top), and has many features not present in
731libc optargs() (such as the ability to describe long options in the same string
732as normal options).</p>
733
734<p>Each command's NEWTOY() macro has an optflags string as its middle argument,
735which sets toy_list.options for that command to tell get_optflags() what
736command line arguments to look for, and what to do with them.
737If a command has no option
738definition string (I.E. the argument is NULL), option parsing is skipped
739for that command, which must look at the raw data in toys.argv to parse its
740own arguments. (If no currently enabled command uses option parsing,
741get_optflags() is optimized out of the resulting binary by the compiler's
742--gc-sections option.)</p>
743
744<p>You don't have to free the option strings, which point into the environment
745space (I.E. the string data is not copied). A TOYFLAG_NOFORK command
746that uses the linked list type "*" should free the list objects but not
747the data they point to, via "llist_free(TT.mylist, NULL);". (If it's not
748NOFORK, exit() will free all the malloced data anyway unless you want
749to implement a CONFIG_TOYBOX_FREE cleanup for it.)</p>
750
751<h4>Optflags format string</h4>
752
753<p>Note: the optflags option description string format is much more
754concisely described by a large comment at the top of lib/args.c.</p>
755
756<p>The general theory is that letters set optflags, and punctuation describes
757other actions the option parsing logic should take.</p>
758
759<p>For example, suppose the command line <b>command -b fruit -d walrus -a 42</b>
760is parsed using the optflags string "<b>a#b:c:d</b>". (I.E.
761toys.which->options="a#b:c:d" and argv = ["command", "-b", "fruit", "-d",
762"walrus", "-a", "42"]). When get_optflags() returns, the following data is
763available to command_main():
764
765<ul>
766<li><p>In <b>struct toys</b>:
767<ul>
768<li>toys.optflags = 13; // -a = 8 | -b = 4 | -d = 1</li>
769<li>toys.optargs[0] = "walrus"; // leftover argument</li>
770<li>toys.optargs[1] = NULL; // end of list</li>
Rob Landleyb911d4d2013-09-21 14:27:26 -0500771<li>toys.optc = 1; // there was 1 leftover argument</li>
Rob Landley66a69d92012-01-16 01:44:17 -0600772<li>toys.argv[] = {"-b", "fruit", "-d", "walrus", "-a", "42"}; // The original command line arguments
773</ul>
774<p></li>
775
776<li><p>In <b>union this</b> (treated as <b>long this[]</b>):
777<ul>
778<li>this[0] = NULL; // -c didn't get an argument this time, so get_optflags() didn't change it and toys_init() zeroed "this" during setup.)</li>
779<li>this[1] = (long)"fruit"; // argument to -b</li>
780<li>this[2] = 42; // argument to -a</li>
781</ul>
782</p></li>
783</ul>
784
785<p>If the command's globals are:</p>
786
787<blockquote><pre>
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500788GLOBALS(
Rob Landley66a69d92012-01-16 01:44:17 -0600789 char *c;
790 char *b;
791 long a;
792)
Rob Landley66a69d92012-01-16 01:44:17 -0600793</pre></blockquote>
794<p>That would mean TT.c == NULL, TT.b == "fruit", and TT.a == 42. (Remember,
795each entry that receives an argument must be a long or pointer, to line up
796with the array position. Right to left in the optflags string corresponds to
Rob Landleyc0e56ed2012-10-08 00:02:30 -0500797top to bottom in GLOBALS().</p>
Rob Landley66a69d92012-01-16 01:44:17 -0600798
Rob Landleyb911d4d2013-09-21 14:27:26 -0500799<p>Put globals not filled out by the option parsing logic at the end of the
800GLOBALS block. Common practice is to list the options one per line (to
801make the ordering explicit, first to last in globals corresponds to right
802to left in the option string), then leave a blank line before any non-option
803globals.</p>
804
Rob Landley66a69d92012-01-16 01:44:17 -0600805<p><b>long toys.optflags</b></p>
806
807<p>Each option in the optflags string corresponds to a bit position in
808toys.optflags, with the same value as a corresponding binary digit. The
809rightmost argument is (1<<0), the next to last is (1<<1) and so on. If
Rob Landleyb4a0efa2012-02-06 21:15:19 -0600810the option isn't encountered while parsing argv[], its bit remains 0.</p>
811
812<p>For example,
Rob Landley66a69d92012-01-16 01:44:17 -0600813the optflags string "abcd" would parse the command line argument "-c" to set
814optflags to 2, "-a" would set optflags to 8, "-bd" would set optflags to
8156 (I.E. 4|2), and "-a -c" would set optflags to 10 (2|8).</p>
816
817<p>Only letters are relevant to optflags, punctuation is skipped: in the
818string "a*b:c#d", d=1, c=2, b=4, a=8. The punctuation after a letter
819usually indicate that the option takes an argument.</p>
820
Rob Landleyb4a0efa2012-02-06 21:15:19 -0600821<p>Since toys.optflags is an unsigned int, it only stores 32 bits. (Which is
822the amount a long would have on 32-bit platforms anyway; 64 bit code on
82332 bit platforms is too expensive to require in common code used by almost
824all commands.) Bit positions beyond the 1<<31 aren't recorded, but
825parsing higher options can still set global variables.</p>
826
Rob Landley66a69d92012-01-16 01:44:17 -0600827<p><b>Automatically setting global variables from arguments (union this)</b></p>
828
829<p>The following punctuation characters may be appended to an optflags
830argument letter, indicating the option takes an additional argument:</p>
831
832<ul>
833<li><b>:</b> - plus a string argument, keep most recent if more than one.</li>
834<li><b>*</b> - plus a string argument, appended to a linked list.</li>
Rob Landley66a69d92012-01-16 01:44:17 -0600835<li><b>@</b> - plus an occurrence counter (stored in a long)</li>
Rob Landleyb6063de2012-01-29 13:54:13 -0600836<li><b>#</b> - plus a signed long argument.
Rob Landleyb911d4d2013-09-21 14:27:26 -0500837<li><b>-</b> - plus a signed long argument defaulting to negative (start argument with + to force a positive value).</li>
Rob Landleyb6063de2012-01-29 13:54:13 -0600838<li><b>.</b> - plus a floating point argument (if CFG_TOYBOX_FLOAT).</li>
839<ul>The following can be appended to a float or double:
840<li><b>&lt;123</b> - error if argument is less than this</li>
841<li><b>&gt;123</b> - error if argument is greater than this</li>
842<li><b>=123</b> - default value if argument not supplied</li>
843</ul>
Rob Landley66a69d92012-01-16 01:44:17 -0600844</ul>
845
Rob Landleyb911d4d2013-09-21 14:27:26 -0500846<p>A note about "." and CFG_TOYBOX_FLOAT: option parsing only understands <>=
847after . when CFG_TOYBOX_FLOAT
848is enabled. (Otherwise the code to determine where floating point constants
849end drops out; it requires floating point). When disabled, it can reserve a
850global data slot for the argument (so offsets won't change in your
851GLOBALS[] block), but will never fill it out. You can handle
852this by using the USE_BLAH() macros with C string concatenation, ala:
853"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"</p>
854
855<p><b>GLOBALS</b></p>
Rob Landley66a69d92012-01-16 01:44:17 -0600856
857<p>Options which have an argument fill in the corresponding slot in the global
858union "this" (see generated/globals.h), treating it as an array of longs
Rob Landleyb911d4d2013-09-21 14:27:26 -0500859with the rightmost saved in this[0]. As described above, using "a*b:c#d",
860"-c 42" would set this[0] = 42; and "-b 42" would set this[1] = "42"; each
861slot is left NULL if the corresponding argument is not encountered.</p>
Rob Landley66a69d92012-01-16 01:44:17 -0600862
863<p>This behavior is useful because the LP64 standard ensures long and pointer
Rob Landleyb4a0efa2012-02-06 21:15:19 -0600864are the same size. C99 guarantees structure members will occur in memory
865in the same order they're declared, and that padding won't be inserted between
Rob Landley66a69d92012-01-16 01:44:17 -0600866consecutive variables of register size. Thus the first few entries can
867be longs or pointers corresponding to the saved arguments.</p>
868
Rob Landleyb911d4d2013-09-21 14:27:26 -0500869<p>See toys/other/hello.c for a longer example of parsing options into the
870GLOBALS block.</p>
871
Rob Landley66a69d92012-01-16 01:44:17 -0600872<p><b>char *toys.optargs[]</b></p>
873
874<p>Command line arguments in argv[] which are not consumed by option parsing
875(I.E. not recognized either as -flags or arguments to -flags) will be copied
876to toys.optargs[], with the length of that array in toys.optc.
877(When toys.optc is 0, no unrecognized command line arguments remain.)
878The order of entries is preserved, and as with argv[] this new array is also
879terminated by a NULL entry.</p>
880
881<p>Option parsing can require a minimum or maximum number of optargs left
882over, by adding "<1" (read "at least one") or ">9" ("at most nine") to the
883start of the optflags string.</p>
884
885<p>The special argument "--" terminates option parsing, storing all remaining
886arguments in optargs. The "--" itself is consumed.</p>
887
888<p><b>Other optflags control characters</b></p>
889
890<p>The following characters may occur at the start of each command's
891optflags string, before any options that would set a bit in toys.optflags:</p>
892
893<ul>
894<li><b>^</b> - stop at first nonoption argument (for nice, xargs...)</li>
895<li><b>?</b> - allow unknown arguments (pass non-option arguments starting
896with - through to optargs instead of erroring out).</li>
897<li><b>&amp;</b> - the first argument has imaginary dash (ala tar/ps. If given twice, all arguments have imaginary dash.)</li>
898<li><b>&lt;</b> - must be followed by a decimal digit indicating at least this many leftover arguments are needed in optargs (default 0)</li>
899<li><b>&gt;</b> - must be followed by a decimal digit indicating at most this many leftover arguments allowed (default MAX_INT)</li>
900</ul>
901
902<p>The following characters may be appended to an option character, but do
903not by themselves indicate an extra argument should be saved in this[].
904(Technically any character not recognized as a control character sets an
905optflag, but letters are never control characters.)</p>
906
907<ul>
908<li><b>^</b> - stop parsing options after encountering this option, everything else goes into optargs.</li>
909<li><b>|</b> - this option is required. If more than one marked, only one is required.</li>
Rob Landley66a69d92012-01-16 01:44:17 -0600910</ul>
911
Rob Landleyb6063de2012-01-29 13:54:13 -0600912<p>The following may be appended to a float or double:</p>
913
914<ul>
915<li><b>&lt;123</b> - error if argument is less than this</li>
916<li><b>&gt;123</b> - error if argument is greater than this</li>
917<li><b>=123</b> - default value if argument not supplied</li>
918</ul>
919
920<p>Option parsing only understands <>= after . when CFG_TOYBOX_FLOAT
921is enabled. (Otherwise the code to determine where floating point constants
922end drops out. When disabled, it can reserve a global data slot for the
923argument so offsets won't change, but will never fill it out.). You can handle
924this by using the USE_BLAH() macros with C string concatenation, ala:</p>
925
926<blockquote>"abc." USE_TOYBOX_FLOAT("<1.23>4.56=7.89") "def"</blockquote>
927
Rob Landley66a69d92012-01-16 01:44:17 -0600928<p><b>--longopts</b></p>
929
930<p>The optflags string can contain long options, which are enclosed in
931parentheses. They may be appended to an existing option character, in
932which case the --longopt is a synonym for that option, ala "a:(--fred)"
933which understands "-a blah" or "--fred blah" as synonyms.</p>
934
935<p>Longopts may also appear before any other options in the optflags string,
936in which case they have no corresponding short argument, but instead set
937their own bit based on position. So for "(walrus)#(blah)xy:z" "command
938--walrus 42" would set toys.optflags = 16 (-z = 1, -y = 2, -x = 4, --blah = 8)
939and would assign this[1] = 42;</p>
940
941<p>A short option may have multiple longopt synonyms, "a(one)(two)", but
942each "bare longopt" (ala "(one)(two)abc" before any option characters)
943always sets its own bit (although you can group them with +X).</p>
Rob Landley7c04f012008-01-20 19:00:16 -0600944
Rob Landleyb911d4d2013-09-21 14:27:26 -0500945<p><b>[groups]</b></p>
946
947<p>At the end of the option string, square bracket groups can define
948relationships between existing options. (This only applies to short
949options, bare --longopts can't participate.)</p>
950
951<p>The first character of the group defines the type, the remaining
952characters are options it applies to:</p>
953
954<ul>
955<li><b>-</b> - Exclusive, switch off all others in this group.</li>
956<li><b>+</b> - Inclusive, switch on all others in this group.</li>
957<li><b>!</b> - Error, fail if more than one defined.</li>
958</ul>
959
960<p>So "abc[-abc]" means -ab = -b, -ba = -a, -abc = -c. "abc[+abc]"
961means -ab=-abc, -c=-abc, and "abc[!abc] means -ab calls error_exit("no -b
962with -a"). Note that [-] groups clear the GLOBALS option slot of
963options they're switching back off, but [+] won't set options it didn't see
964(just the optflags).</p>
965
966<p><b>whitespace</b></p>
967
968<p>Arguments may occur with or without a space (I.E. "-a 42" or "-a42").
969The command line argument "-abc" may be interepreted many different ways:
970the optflags string "cba" sets toys.optflags = 7, "c:ba" sets toys.optflags=4
971and saves "ba" as the argument to -c, and "cb:a" sets optflags to 6 and saves
972"c" as the argument to -b.</p>
973
974<p>Note that &amp; changes whitespace handling, so that the command line
975"tar cvfCj outfile.tar.bz2 topdir filename" is parsed the same as
976"tar filename -c -v -j -f outfile.tar.bz2 -C topdir". Note that "tar -cvfCj
977one two three" would equal "tar -c -v -f Cj one two three". (This matches
978historical usage.)</p>
979
980<p>Appending a space to the option in the option string ("a: b") makes it
981require a space, I.E. "-ab" is interpreted as "-a" "-b". That way "kill -stop"
982differs from "kill -s top".</p>
983
984<p>Appending ; to a longopt in the option string makes its argument optional,
985and only settable with =, so in ls "(color):;" can accept "ls --color" and
986"ls --color=auto" without complaining that the first has no argument.</p>
987
Rob Landleyc8d0da52012-07-15 17:47:08 -0500988<a name="lib_dirtree"><h3>lib/dirtree.c</h3>
989
990<p>The directory tree traversal code should be sufficiently generic
991that commands never need to use readdir(), scandir(), or the fts.h family
992of functions.</p>
993
994<p>These functions do not call chdir() or rely on PATH_MAX. Instead they
995use openat() and friends, using one filehandle per directory level to
996recurseinto subdirectories. (I.E. they can descend 1000 directories deep
997if setrlimit(RLIMIT_NOFILE) allows enough open filehandles, and the default
998in /proc/self/limits is generally 1024.)</p>
999
1000<p>The basic dirtree functions are:</p>
1001
1002<ul>
1003<li><p><b>dirtree_read(char *path, int (*callback)(struct dirtree node))</b> -
1004recursively read directories, either applying callback() or returning
1005a tree of struct dirtree if callback is NULL.</p></li>
1006
1007<li><p><b>dirtree_path(struct dirtree *node, int *plen)</b> - malloc() a
1008string containing the path from the root of this tree to this node. If
1009plen isn't NULL then *plen is how many extra bytes to malloc at the end
1010of string.</p></li>
1011
1012<li><p><b>dirtree_parentfd(struct dirtree *node)</b> - return fd of
1013containing directory, for use with openat() and such.</p></li>
1014</ul>
1015
1016<p>The <b>dirtree_read()</b> function takes two arguments, a starting path for
1017the root of the tree, and a callback function. The callback takes a
1018<b>struct dirtree *</b> (from lib/lib.h) as its argument. If the callback is
1019NULL, the traversal uses a default callback (dirtree_notdotdot()) which
1020recursively assembles a tree of struct dirtree nodes for all files under
1021this directory and subdirectories (filtering out "." and ".." entries),
1022after which dirtree_read() returns the pointer to the root node of this
1023snapshot tree.</p>
1024
1025<p>Otherwise the callback() is called on each entry in the directory,
1026with struct dirtree * as its argument. This includes the initial
1027node created by dirtree_read() at the top of the tree.</p>
1028
1029<p><b>struct dirtree</b></p>
1030
1031<p>Each struct dirtree node contains <b>char name[]</b> and <b>struct stat
1032st</b> entries describing a file, plus a <b>char *symlink</b>
1033which is NULL for non-symlinks.</p>
1034
1035<p>During a callback function, the <b>int data</b> field of directory nodes
1036contains a dirfd (for use with the openat() family of functions). This is
1037generally used by calling dirtree_parentfd() on the callback's node argument.
1038For symlinks, data contains the length of the symlink string. On the second
1039callback from DIRTREE_COMEAGAIN (depth-first traversal) data = -1 for
1040all nodes (that's how you can tell it's the second callback).</p>
1041
1042<p>Users of this code may put anything they like into the <b>long extra</b>
1043field. For example, "cp" and "mv" use this to store a dirfd for the destination
1044directory (and use DIRTREE_COMEAGAIN to get the second callback so they can
1045close(node->extra) to avoid running out of filehandles).
1046This field is not directly used by the dirtree code, and
1047thanks to LP64 it's large enough to store a typecast pointer to an
1048arbitrary struct.</p>
1049
1050<p>The return value of the callback combines flags (with boolean or) to tell
1051the traversal infrastructure how to behave:</p>
1052
1053<ul>
1054<li><p><b>DIRTREE_SAVE</b> - Save this node, assembling a tree. (Without
1055this the struct dirtree is freed after the callback returns. Filtering out
1056siblings is fine, but discarding a parent while keeping its child leaks
1057memory.)</p></li>
1058<li><p><b>DIRTREE_ABORT</b> - Do not examine any more entries in this
1059directory. (Does not propagate up tree: to abort entire traversal,
1060return DIRTREE_ABORT from parent callbacks too.)</p></li>
1061<li><p><b>DIRTREE_RECURSE</b> - Examine directory contents. Ignored for
1062non-directory entries. The remaining flags only take effect when
1063recursing into the children of a directory.</p></li>
1064<li><p><b>DIRTREE_COMEAGAIN</b> - Call the callback a second time after
1065examining all directory contents, allowing depth-first traversal.
1066On the second call, dirtree->data = -1.</p></li>
1067<li><p><b>DIRTREE_SYMFOLLOW</b> - follow symlinks when populating children's
1068<b>struct stat st</b> (by feeding a nonzero value to the symfollow argument of
1069dirtree_add_node()), which means DIRTREE_RECURSE treats symlinks to
1070directories as directories. (Avoiding infinite recursion is the callback's
1071problem: the non-NULL dirtree->symlink can still distinguish between
1072them.)</p></li>
1073</ul>
1074
1075<p>Each struct dirtree contains three pointers (next, parent, and child)
1076to other struct dirtree.</p>
1077
1078<p>The <b>parent</b> pointer indicates the directory
1079containing this entry; even when not assembling a persistent tree of
1080nodes the parent entries remain live up to the root of the tree while
1081child nodes are active. At the top of the tree the parent pointer is
1082NULL, meaning the node's name[] is either an absolute path or relative
1083to cwd. The function dirtree_parentfd() gets the directory file descriptor
1084for use with openat() and friends, returning AT_FDCWD at the top of tree.</p>
1085
1086<p>The <b>child</b> pointer points to the first node of the list of contents of
1087this directory. If the directory contains no files, or the entry isn't
1088a directory, child is NULL.</p>
1089
1090<p>The <b>next</b> pointer indicates sibling nodes in the same directory as this
1091node, and since it's the first entry in the struct the llist.c traversal
1092mechanisms work to iterate over sibling nodes. Each dirtree node is a
1093single malloc() (even char *symlink points to memory at the end of the node),
1094so llist_free() works but its callback must descend into child nodes (freeing
1095a tree, not just a linked list), plus whatever the user stored in extra.</p>
1096
1097<p>The <b>dirtree_read</b>() function is a simple wrapper, calling <b>dirtree_add_node</b>()
1098to create a root node relative to the current directory, then calling
1099<b>handle_callback</b>() on that node (which recurses as instructed by the callback
1100return flags). Some commands (such as chgrp) bypass this wrapper, for example
1101to control whether or not to follow symlinks to the root node; symlinks
1102listed on the command line are often treated differently than symlinks
1103encountered during recursive directory traversal).
1104
1105<p>The ls command not only bypasses the wrapper, but never returns
1106<b>DIRTREE_RECURSE</b> from the callback, instead calling <b>dirtree_recurse</b>() manually
1107from elsewhere in the program. This gives ls -lR manual control
1108of traversal order, which is neither depth first nor breadth first but
1109instead a sort of FIFO order requried by the ls standard.</p>
1110
Rob Landley7eaf4f52014-04-09 08:30:09 -05001111<a name="toys">
1112<h1><a href="#toys">Directory toys/</a></h1>
Rob Landleyc0e56ed2012-10-08 00:02:30 -05001113
1114<p>This directory contains command implementations. Each command is a single
1115self-contained file. Adding a new command involves adding a single
1116file, and removing a command involves removing that file. Commands use
1117shared infrastructure from the lib/ and generated/ directories.</p>
1118
1119<p>Currently there are three subdirectories under "toys/" containing commands
1120described in POSIX-2008, the Linux Standard Base 4.1, or "other". The only
1121difference this makes is which menu the command shows up in during "make
1122menuconfig", the directories are otherwise identical. Note that they commands
1123exist within a single namespace at runtime, so you can't have the same
1124command in multiple subdirectories.</p>
1125
1126<p>(There are actually four sub-menus in "make menuconfig", the fourth
1127contains global configuration options for toybox, and lives in Config.in at
1128the top level.)</p>
1129
1130<p>See <a href="#adding">adding a new command</a> for details on the
1131layout of a command file.</p>
1132
Rob Landley81b899d2007-12-18 02:02:47 -06001133<h2>Directory scripts/</h2>
Rob Landley4e68de12007-12-13 07:00:27 -06001134
Rob Landley1f4f41a2012-10-08 21:31:07 -05001135<p>Build infrastructure. The makefile calls scripts/make.sh for "make"
1136and scripts/install.sh for "make install".</p>
1137
1138<p>There's also a test suite, "make test" calls make/test.sh, which runs all
1139the tests in make/test/*. You can run individual tests via
1140"scripts/test.sh command", or "TEST_HOST=1 scripts/test.sh command" to run
1141that test against the host implementation instead of the toybox one.</p>
1142
Rob Landley4e68de12007-12-13 07:00:27 -06001143<h3>scripts/cfg2files.sh</h3>
1144
1145<p>Run .config through this filter to get a list of enabled commands, which
1146is turned into a list of files in toys via a sed invocation in the top level
1147Makefile.
1148</p>
1149
Rob Landley81b899d2007-12-18 02:02:47 -06001150<h2>Directory kconfig/</h2>
Rob Landley4e68de12007-12-13 07:00:27 -06001151
1152<p>Menuconfig infrastructure copied from the Linux kernel. See the
1153Linux kernel's Documentation/kbuild/kconfig-language.txt</p>
1154
Rob Landley66a69d92012-01-16 01:44:17 -06001155<a name="generated">
1156<h2>Directory generated/</h2>
1157
1158<p>All the files in this directory except the README are generated by the
1159build. (See scripts/make.sh)</p>
1160
1161<ul>
1162<li><p><b>config.h</b> - CFG_COMMAND and USE_COMMAND() macros set by menuconfig via .config.</p></li>
1163
1164<li><p><b>Config.in</b> - Kconfig entries for each command. Included by top level Config.in. The help text in here is used to generated help.h</p></li>
1165
1166<li><p><b>help.h</b> - Help text strings for use by "help" command. Building
1167this file requires python on the host system, so the prebuilt file is shipped
1168in the build tarball to avoid requiring python to build toybox.</p></li>
1169
1170<li><p><b>newtoys.h</b> - List of NEWTOY() or OLDTOY() macros for all available
1171commands. Associates command_main() functions with command names, provides
1172option string for command line parsing (<a href="#lib_args">see lib/args.c</a>),
1173specifies where to install each command and whether toysh should fork before
1174calling it.</p></li>
1175</ul>
1176
1177<p>Everything in this directory is a derivative file produced from something
1178else. The entire directory is deleted by "make distclean".</p>
Rob Landley4e68de12007-12-13 07:00:27 -06001179<!--#include file="footer.html" -->