Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 1 | <!--#include file="header.html" --> |
| 2 | <title>Toybox Roadmap</title> |
| 3 | |
| 4 | <h2>Goals and use cases</h2> |
| 5 | |
| 6 | <p>We have several potential use cases for a new set of command line |
| 7 | utilities, and are using those to determine which commands to implement |
| 8 | for Toybox's 1.0 release.</p> |
| 9 | |
| 10 | <p>Our current candidate list combines the commands toybox already implements, |
| 11 | the development environment command list, the toolbox standard commands, |
| 12 | various vendor configurations of busybox, a selected subset of the SUSv4 |
| 13 | standard, a couple of the less-insane bits of LSB, a few outright requests, |
| 14 | plus additional to-be-determined shell functionality.</p> |
| 15 | |
| 16 | <p>See the <a href=status.html>status page</a> for the combined list |
| 17 | and progress towards implementing it.</p> |
| 18 | |
| 19 | <hr /> |
| 20 | <a name="standards"> |
| 21 | <h2>Use case: standards compliance.</h2> |
| 22 | |
Rob Landley | 571b070 | 2012-11-13 16:13:45 -0600 | [diff] [blame] | 23 | <h3><a name=susv4 /><a href="#susv4">POSIX-2008/SUSv4</a></h3> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 24 | <p>The best standards are the kind that describe reality, rather than |
| 25 | attempting to impose a new one. (I.E. a good standard should document, not |
| 26 | legislate.)</p> |
| 27 | |
| 28 | <p>The kind of standards which describe existing reality tend to be approved by |
| 29 | more than one standards body, such ANSI and ISO both approving C. That's why |
| 30 | the IEEE POSIX committee's 2008 standard, the Single Unix Specification version |
| 31 | 4, and the Open Group Base Specification edition 7 are all the same standard |
| 32 | from three sources.</p> |
| 33 | |
| 34 | <p>The <a href="http://opengroup.org/onlinepubs/9699919799/idx/utilities.html">"utilities" |
| 35 | section</a> |
| 36 | of these standards is devoted to the unix command line, and are the best such |
| 37 | standard for our purposes. (My earlier work on BusyBox was implemented with |
| 38 | regard to SUSv3, an earlier version of this standard.)</p> |
| 39 | |
| 40 | <h3>Problems with the standard</h3> |
| 41 | |
| 42 | <p>Unfortunately, these standards describe a subset of reality, lacking any |
| 43 | mention of commands such as init, login, or mount required to actually boot a |
| 44 | system. It provides ipcrm and ipcs, but not ipcmk, so you can use System V IPC |
| 45 | resources but not create them.</p> |
| 46 | |
| 47 | <p>These standards also contain a large number of commands that are |
| 48 | inappropriate for toybox to implement in its 1.0 release. (Perhaps some of |
| 49 | these could be reintroduced in later releases, but not now.)</p> |
| 50 | |
| 51 | <p>Starting with the full "utilities" list, we first remove generally obsolete |
| 52 | commands (compess ed ex pr uncompress uccp uustat uux), commands for the |
| 53 | pre-CVS "SCCS" source control system (admin delta get prs rmdel sact sccs unget |
| 54 | val what), fortran support (asa fort77), and batch processing support (batch |
| 55 | qalter qdel qhold qmove qmsg qrerun qrls qselect qsig qstat qsub).</p> |
| 56 | |
| 57 | <p>Some commands are for a compiler toolchain (ar c99 cflow ctags cxref gencat |
| 58 | iconv lex m4 make nm strings strip tsort yacc), which is outside of toybox's |
| 59 | mandate and should be supplied externally. (Again, some of these may be |
| 60 | revisited later, but not for toybox 1.0.)</p> |
| 61 | |
| 62 | <p>Some commands are part of a command shell, and cannot be implemented as |
| 63 | separate executables (alias bg cd command fc fg getopts hash jobs kill read |
| 64 | type ulimit umask unalias wait). These may be revisited as part of a built-in |
| 65 | toybox shell, but are not exported into $PATH via symlinks. (If you fork a |
| 66 | child process and have it "cd" then exit, you've accomplished nothing.)</p> |
| 67 | |
| 68 | <p>A few other commands are judgement calls, providing command-line |
| 69 | internationalization support (iconv locale localedef), System V inter-process |
| 70 | communication (ipcrm ipcs), and cross-tty communication from the minicomputer |
| 71 | days (talk mesg write). The "pax" utility was supplanted by tar, "mailx" is |
| 72 | a command line email client, and "lp" submits files for printing to... what |
| 73 | exactly? (cups?) The standard defines crontab but not crond.</p> |
| 74 | |
| 75 | <p>Removing all of that leaves the following commands, which toybox should |
| 76 | implement:</p> |
| 77 | |
| 78 | <blockquote><b> |
| 79 | <span id=posix> |
| 80 | at awk basename bc cal cat chgrp chmod chown cksum cmp comm cp |
| 81 | csplit cut date dd df diff dirname du echo env expand expr false file find |
| 82 | fold fuser getconf grep head id join kill link ln logger logname ls man |
| 83 | mkdir mkfifo more mv newgrp nice nl nohup od paste patch pathchk printf ps |
| 84 | pwd renice rm rmdir sed sh sleep sort split stty tabs tail tee test time |
| 85 | touch tput tr true tty uname unexpand uniq unlink uudecode uuencode vi wc |
| 86 | who xargs zcat |
| 87 | </span> |
| 88 | </b></blockquote> |
| 89 | |
Rob Landley | 571b070 | 2012-11-13 16:13:45 -0600 | [diff] [blame] | 90 | <h3><a name=sigh /><a href="#sigh">Linux Standard Base</a></h3> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 91 | |
| 92 | <p>One attempt to supplement POSIX towards an actual usable system was the |
| 93 | Linux Standard Base. Unfortunately, the quality of this "standard" is |
| 94 | fairly low.</p> |
| 95 | |
| 96 | <p>POSIX allowed its standards process to be compromised |
| 97 | by leaving things out, thus allowing IBM mainframes and Windows NT to drive |
| 98 | a truck through the holes and declare themselves compilant. But it means what |
| 99 | they DID standardize tends to be respected.</p> |
| 100 | |
| 101 | <p>The Linux Standard Base's failure mode is different, they respond to |
| 102 | pressure by including special-case crap, such as allowing Red Hat to shoehorn |
| 103 | RPM on the standard even though all sorts of distros (from Debian to Slackware) |
| 104 | don't use it and probably never will. This means anything in the LSB is |
| 105 | at best a suggestion: arbitrary portions of this standard are widely |
| 106 | ignored.</p> |
| 107 | |
| 108 | <p>The LSB does specify a <a href=http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/cmdbehav.html>list of command line |
| 109 | utilities</a>:</p> |
| 110 | |
| 111 | <blockquote><b> |
| 112 | ar at awk batch bc chfn chsh col cpio crontab df dmesg du echo egrep |
| 113 | fgrep file fuser gettext grep groupadd groupdel groupmod groups |
| 114 | gunzip gzip hostname install install_initd ipcrm ipcs killall lpr ls |
| 115 | lsb_release m4 md5sum mknod mktemp more mount msgfmt newgrp od passwd |
| 116 | patch pidof remove_initd renice sed sendmail seq sh shutdown su sync |
| 117 | tar umount useradd userdel usermod xargs zcat |
| 118 | </b></blockquote> |
| 119 | |
| 120 | <p>Where posix specifies one of those commands, LSB's deltas tend to be |
| 121 | accomodations for broken tool versions which aren't up to date with the |
| 122 | standard yet. (See <a href=http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/more.html>more</a> and <a href=http://refspecs.linuxfoundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/xargs.html>xargs</a> |
| 123 | for examples.)</p> |
| 124 | |
| 125 | <p>Since we've already committed to using our own judgement to skip bits of |
| 126 | POSIX, and LSB's "judgement" in this regard is purely bug workarounds to declare |
| 127 | various legacy tool implementations "compliant", this means we're mostly |
| 128 | interested in the set of tools that aren't specified in posix at all.</p> |
| 129 | |
| 130 | <p>Of these, gettext and msgfmt are internationalization, install_initd and |
| 131 | remove_initd aren't present on ubuntu 10.04, lpr is out of scope, and |
| 132 | lsb_release is a distro issue (it's a nice command, but the output of |
| 133 | lsb_release -a is the name and version number of the linux distro you're |
| 134 | running, which toybox doesn't know).</p> |
| 135 | |
| 136 | <p>This leaves:</p> |
| 137 | |
| 138 | <blockquote><b> |
| 139 | <span id=lsb> |
| 140 | chfn chsh dmesg egrep fgrep groupadd groupdel groupmod groups |
| 141 | gunzip gzip hostname install killall md5sum |
| 142 | mknod mktemp mount passwd pidof sendmail seq shutdown |
| 143 | su sync tar umount useradd userdel usermod zcat |
| 144 | </span> |
| 145 | </b></blockquote> |
| 146 | |
| 147 | <hr /> |
| 148 | <a name="dev_env"> |
Rob Landley | 571b070 | 2012-11-13 16:13:45 -0600 | [diff] [blame] | 149 | <h2><a href="#dev_env">Use case: provide a self-hosting development environment</a></h2> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 150 | |
| 151 | <p>The following commands are enough to build the Aboriginal Linux development |
| 152 | environment, boot it to a shell prompt, and build Linux From Scratch 6.8 under |
| 153 | it. (Aboriginal Linux currently uses BusyBox for this, thus provides a |
| 154 | drop-in test environment for toybox. We install both implementations side |
| 155 | by side, redirecting the symlinks a command at a time until the older |
| 156 | package is no longer used, and can be removed.)</p> |
| 157 | |
| 158 | <p>This use case includes running init scripts and other shell scripts, running |
| 159 | configure, make, and install in each package, and providing basic command line |
| 160 | facilities such as a text editor. (It does not include a compiler toolchain or |
| 161 | C library, those are outside the scope of this project.)</p> |
| 162 | |
| 163 | <blockquote><b> |
| 164 | <span id=development> |
| 165 | bzcat cat cp dirname echo env patch rmdir sha1sum sleep sort sync |
| 166 | true uname wc which yes zcat |
| 167 | awk basename bzip2 chmod chown cmp cut date dd diff |
| 168 | egrep expr find grep gzip head hostname id install ln ls |
| 169 | mkdir mktemp mv od readlink rm sed sh tail tar touch tr uniq |
| 170 | wget whoami xargs chgrp comm gunzip less logname man split |
| 171 | tee test time bunzip2 chgrp chroot comm cpio dmesg |
| 172 | dnsdomainname ftpd ftpget ftpput gunzip ifconfig init less |
| 173 | logname losetup man mdev mount mountpoint nc pgrep pkill |
| 174 | pwd route split stat switch_root tac umount vi |
| 175 | </span> |
| 176 | </b></blockquote> |
| 177 | |
| 178 | <p>Note: Aboriginal Linux installs bash 2.05b as #!/bin/sh and its scripts |
| 179 | require bash extensions not present in shells such as busybox ash. |
| 180 | This means that toysh needs to supply several bash extensions _and_ work |
| 181 | when called under the name "bash".</p> |
| 182 | |
| 183 | <hr /> |
Rob Landley | 571b070 | 2012-11-13 16:13:45 -0600 | [diff] [blame] | 184 | <h2><a name=android /><a href="#android">Use case: Replacing Android Toolbox</a></h2> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 185 | |
Rob Landley | fdc10c9 | 2012-10-16 17:09:30 -0500 | [diff] [blame] | 186 | <p>Android has a policy against GPL in userspace, so even though BusyBox |
| 187 | predates Android by many years, they couldn't use it. Instead they grabbed |
| 188 | an old version of ash and implemented their own command line utility set |
| 189 | called "toolbox".</p> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 190 | |
Rob Landley | fdc10c9 | 2012-10-16 17:09:30 -0500 | [diff] [blame] | 191 | <p>Toolbox doesn't have its own repository, instead it's part of Android's |
| 192 | <a href=https://android.googlesource.com/platform/system/core>system/core |
| 193 | git repository</a> (this analysis looked at commit 51ccef27cab58).</p> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 194 | |
Rob Landley | fdc10c9 | 2012-10-16 17:09:30 -0500 | [diff] [blame] | 195 | <h3>Toolbox commands:</h3> |
| 196 | |
| 197 | <p>According to core/toolbox/Android.mk the toolbox directory builds the |
| 198 | following commands:</p> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 199 | |
| 200 | <blockquote><b> |
Rob Landley | fdc10c9 | 2012-10-16 17:09:30 -0500 | [diff] [blame] | 201 | ls mount cat ps kill ln insmod rmmod lsmod ifconfig setconsole |
| 202 | rm mkdir rmdir reboot getevent sendevent date wipe sync umount |
| 203 | start stop notify cmp dmesg route hd dd df getprop setprop watchprops |
| 204 | log sleep renice printenv smd chmod chown newfs_msdos netstat ioctl |
| 205 | mv schedtop top iftop id uptime vmstat nandread ionice touch lsof md5 r |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 206 | </b></blockquote> |
| 207 | |
Rob Landley | fdc10c9 | 2012-10-16 17:09:30 -0500 | [diff] [blame] | 208 | <p>If selinux is enabled, you also get:</p> |
| 209 | <blockquote><b> |
| 210 | getenforce setenforce chcon restorecon runcon getsebool setsebool load_policy |
| 211 | </b></blockquote> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 212 | |
Rob Landley | fdc10c9 | 2012-10-16 17:09:30 -0500 | [diff] [blame] | 213 | <p>The Android.mk file also refers to dynarray.c and toolbox.c as library |
| 214 | code. This leaves the following apparently unused C files in toolbox/*.c, each |
| 215 | of which has a command_main() function and seems to implement a standalone |
| 216 | command:</p> |
| 217 | |
| 218 | <blockquote><b> |
| 219 | alarm exists lsusb readtty rotatefb setkey syren |
| 220 | </b></blockquote> |
| 221 | |
| 222 | <h3>Command shell (ash)</h3> |
| 223 | |
| 224 | <p>The core/sh subdirectory contains a fork of ash 1.17, and sucks in |
| 225 | liblinenoise to provide command line history/editing.</p> |
| 226 | |
| 227 | <h3>Other Android core commands</h3> |
| 228 | |
| 229 | <p>Other than the toolbox and sh directories, the currently interesting |
| 230 | subdirectories in the core repository are fs_mgr, gpttool, init, |
| 231 | logcat, logwrapper, mkbootimg, netcfg, run-as, and sdcard.</p> |
| 232 | |
| 233 | <ul> |
| 234 | <li><b>fs_mgr</b> - subset of mount</li> |
| 235 | <li><b>gpttool</b> - subset of fdisk</li> |
| 236 | <li><b>init</b> - Android's PID 1</li> |
| 237 | <li><b>logcat</b> - read android log format</li> |
| 238 | <li><b>logwrapper</b> - redirect stdio to android log</li> |
| 239 | <li><b>mkbootimg</b> - create signed boot image</li> |
| 240 | <li><b>netcfg</b> - network configuration (sucks in libnetutils)</li> |
| 241 | <li><b>run-as</b> - subset of sudo</li> |
| 242 | <li><b>sdcard</b> - FUSE wrapper to squash UID/GID/permissions to what FAT supports.</li> |
| 243 | </ul> |
| 244 | |
| 245 | <p>Almost all of these reinvent an existing wheel with less functionality and a |
| 246 | different user interface. We may want to provide that interface, but |
| 247 | implementing the full commands (mount, fdisk, init, ifconfig with dhcp, |
| 248 | and sudo) come first.</p> |
| 249 | |
| 250 | <p>Although logcat/logwrapper also reinvent a wheel, Android did so in the |
| 251 | kernel and these provide an interface to that.</p> |
| 252 | |
| 253 | <p>Also, gpttool and mkbootimg are install tools, and sdcard looks like a |
| 254 | testing tool. These aren't a priority if android wants to use its own |
| 255 | bespoke code to install itself.</p> |
| 256 | |
| 257 | <h3>Analysis</h3> |
| 258 | |
| 259 | <p>For reference, combining everything listed above, we get:</p> |
| 260 | |
| 261 | <blockquote><b> |
| 262 | alarm ash cat chcon chmod chown cmp date dd df dmesg exists fs_mgr getenforce |
| 263 | getevent getprop getsebool gpttool hd id ifconfig iftop init insmod ioctl |
| 264 | ionice kill ln load_policy log logcat logwrapper ls lsmod lsof lsusb md5 |
| 265 | mkbootimg mkdir mount mv nandread netcfg netstat newfs_msdos notify printenv |
| 266 | ps r readtty reboot renice restorecon rm rmdir rmmod rotatefb route run-as |
| 267 | runcon schedtop sdcard sendevent setconsole setenforce setkey setprop setsebool |
| 268 | sleep smd start stop sync syren top touch umount uptime vmstat watchprops wipe |
| 269 | </b></blockquote> |
| 270 | |
| 271 | <p>We may eventually implement all of that, but for toybox 1.0 we need to |
| 272 | focus a bit. For our first pass, let's ignore selinux, strip out the "unlisted" |
| 273 | commands except lsusb, and grab just logcat and logwrapper from the "core" |
| 274 | commands (since the rest have some full/standard version providing that |
| 275 | functionality, which we can implement a shim interface for later).</p> |
| 276 | |
| 277 | <p>This means toybox should implement:</p> |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 278 | <blockquote><b> |
| 279 | <span id=toolbox> |
Rob Landley | fdc10c9 | 2012-10-16 17:09:30 -0500 | [diff] [blame] | 280 | cat chmod chown cmp date dd df dmesg getevent getprop hd id ifconfig iftop |
| 281 | insmod ioctl ionice kill ln log logcat logwrapper ls lsmod lsof lsusb md5 mkdir |
| 282 | mount mv nandread |
| 283 | netstat newfs_msdos notify printenv ps r reboot renice rm rmdir rmmod route |
| 284 | schedtop sendevent setconsole setprop sleep smd start stop sync top touch |
| 285 | umount uptime vmstat watchprops wipe |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 286 | </span> |
Rob Landley | fdc10c9 | 2012-10-16 17:09:30 -0500 | [diff] [blame] | 287 | <b></blockquote> |
| 288 | |
| 289 | <p>The following Toolbox commands are already covered in previous |
| 290 | sections of this analysis:</p> |
| 291 | |
| 292 | <blockquote><b> |
| 293 | cat chmod chown cmp date dd df dmesg id ifconfig insmod kill ln ls lsmod |
| 294 | mkdir mount mv ps renice rm rmdir rmmod route sleep sync top touch umount |
| 295 | </b></blockquote> |
| 296 | |
| 297 | <p>Which leaves the following commands as new from Toolbox:</p> |
| 298 | |
| 299 | <blockquote><b> |
| 300 | getevent getprop hd iftop ioctl ionice log lsof nandread netstat |
| 301 | newfs_msdos notify printenv r reboot schedtop sendevent setconsole |
| 302 | setprop smd start stop top uptime vmstat watchprops wipe |
Rob Landley | 8f90d3a | 2012-07-21 23:58:40 -0500 | [diff] [blame] | 303 | </b></blockquote> |
| 304 | |
| 305 | <hr /> |
| 306 | <h2>Requests:</h2> |
| 307 | |
| 308 | <p>The following additional commands have been requested by various users:</p> |
| 309 | <blockquote><b> |
| 310 | <span id=request> |
| 311 | freeramdisk getty halt hexdump hwclock klogd modprobe ping ping6 pivot_root |
| 312 | poweroff sfdisk sudo syslogd taskset telnet telnetd tracepath traceroute unzip |
| 313 | usleep vconfig zip free login modinfo unshare netcat help |
| 314 | </span> |
| 315 | </b></blockquote> |
| 316 | |
| 317 | <!-- #include "footer.html" --> |
| 318 | |