1 # $NetBSD: TOUR,v 1.8 1996/10/16 14:24:56 christos Exp $ 2 # @(#)TOUR 8.1 (Berkeley) 5/31/93 3 4 NOTE -- This is the original TOUR paper distributed with ash and 5 does not represent the current state of the shell. It is provided anyway 6 since it provides helpful information for how the shell is structured, 7 but be warned that things have changed -- the current shell is 8 still under development. 9 10 ================================================================ 11 12 A Tour through Ash 13 14 Copyright 1989 by Kenneth Almquist. 15 16 17 DIRECTORIES: The subdirectory bltin contains commands which can 18 be compiled stand-alone. The rest of the source is in the main 19 ash directory. 20 21 SOURCE CODE GENERATORS: Files whose names begin with "mk" are 22 programs that generate source code. A complete list of these 23 programs is: 24 25 program intput files generates 26 ------- ------------ --------- 27 mkbuiltins builtins builtins.h builtins.c 28 mkinit *.c init.c 29 mknodes nodetypes nodes.h nodes.c 30 mksignames - signames.h signames.c 31 mksyntax - syntax.h syntax.c 32 mktokens - token.h 33 bltin/mkexpr unary_op binary_op operators.h operators.c 34 35 There are undoubtedly too many of these. Mkinit searches all the 36 C source files for entries looking like: 37 38 INIT { 39 x = 1; /* executed during initialization */ 40 } 41 42 RESET { 43 x = 2; /* executed when the shell does a longjmp 44 back to the main command loop */ 45 } 46 47 SHELLPROC { 48 x = 3; /* executed when the shell runs a shell procedure */ 49 } 50 51 It pulls this code out into routines which are when particular 52 events occur. The intent is to improve modularity by isolating 53 the information about which modules need to be explicitly 54 initialized/reset within the modules themselves. 55 56 Mkinit recognizes several constructs for placing declarations in 57 the init.c file. 58 INCLUDE "file.h" 59 includes a file. The storage class MKINIT makes a declaration 60 available in the init.c file, for example: 61 MKINIT int funcnest; /* depth of function calls */ 62 MKINIT alone on a line introduces a structure or union declara- 63 tion: 64 MKINIT 65 struct redirtab { 66 short renamed[10]; 67 }; 68 Preprocessor #define statements are copied to init.c without any 69 special action to request this. 70 71 INDENTATION: The ash source is indented in multiples of six 72 spaces. The only study that I have heard of on the subject con- 73 cluded that the optimal amount to indent is in the range of four 74 to six spaces. I use six spaces since it is not too big a jump 75 from the widely used eight spaces. If you really hate six space 76 indentation, use the adjind (source included) program to change 77 it to something else. 78 79 EXCEPTIONS: Code for dealing with exceptions appears in 80 exceptions.c. The C language doesn't include exception handling, 81 so I implement it using setjmp and longjmp. The global variable 82 exception contains the type of exception. EXERROR is raised by 83 calling error. EXINT is an interrupt. EXSHELLPROC is an excep- 84 tion which is raised when a shell procedure is invoked. The pur- 85 pose of EXSHELLPROC is to perform the cleanup actions associated 86 with other exceptions. After these cleanup actions, the shell 87 can interpret a shell procedure itself without exec'ing a new 88 copy of the shell. 89 90 INTERRUPTS: In an interactive shell, an interrupt will cause an 91 EXINT exception to return to the main command loop. (Exception: 92 EXINT is not raised if the user traps interrupts using the trap 93 command.) The INTOFF and INTON macros (defined in exception.h) 94 provide uninterruptable critical sections. Between the execution 95 of INTOFF and the execution of INTON, interrupt signals will be 96 held for later delivery. INTOFF and INTON can be nested. 97 98 MEMALLOC.C: Memalloc.c defines versions of malloc and realloc 99 which call error when there is no memory left. It also defines a 100 stack oriented memory allocation scheme. Allocating off a stack 101 is probably more efficient than allocation using malloc, but the 102 big advantage is that when an exception occurs all we have to do 103 to free up the memory in use at the time of the exception is to 104 restore the stack pointer. The stack is implemented using a 105 linked list of blocks. 106 107 STPUTC: If the stack were contiguous, it would be easy to store 108 strings on the stack without knowing in advance how long the 109 string was going to be: 110 p = stackptr; 111 *p++ = c; /* repeated as many times as needed */ 112 stackptr = p; 113 The folloing three macros (defined in memalloc.h) perform these 114 operations, but grow the stack if you run off the end: 115 STARTSTACKSTR(p); 116 STPUTC(c, p); /* repeated as many times as needed */ 117 grabstackstr(p); 118 119 We now start a top-down look at the code: 120 121 MAIN.C: The main routine performs some initialization, executes 122 the user's profile if necessary, and calls cmdloop. Cmdloop is 123 repeatedly parses and executes commands. 124 125 OPTIONS.C: This file contains the option processing code. It is 126 called from main to parse the shell arguments when the shell is 127 invoked, and it also contains the set builtin. The -i and -j op- 128 tions (the latter turns on job control) require changes in signal 129 handling. The routines setjobctl (in jobs.c) and setinteractive 130 (in trap.c) are called to handle changes to these options. 131 132 PARSING: The parser code is all in parser.c. A recursive des- 133 cent parser is used. Syntax tables (generated by mksyntax) are 134 used to classify characters during lexical analysis. There are 135 three tables: one for normal use, one for use when inside single 136 quotes, and one for use when inside double quotes. The tables 137 are machine dependent because they are indexed by character vari- 138 ables and the range of a char varies from machine to machine. 139 140 PARSE OUTPUT: The output of the parser consists of a tree of 141 nodes. The various types of nodes are defined in the file node- 142 types. 143 144 Nodes of type NARG are used to represent both words and the con- 145 tents of here documents. An early version of ash kept the con- 146 tents of here documents in temporary files, but keeping here do- 147 cuments in memory typically results in significantly better per- 148 formance. It would have been nice to make it an option to use 149 temporary files for here documents, for the benefit of small 150 machines, but the code to keep track of when to delete the tem- 151 porary files was complex and I never fixed all the bugs in it. 152 (AT&T has been maintaining the Bourne shell for more than ten 153 years, and to the best of my knowledge they still haven't gotten 154 it to handle temporary files correctly in obscure cases.) 155 156 The text field of a NARG structure points to the text of the 157 word. The text consists of ordinary characters and a number of 158 special codes defined in parser.h. The special codes are: 159 160 CTLVAR Variable substitution 161 CTLENDVAR End of variable substitution 162 CTLBACKQ Command substitution 163 CTLBACKQ|CTLQUOTE Command substitution inside double quotes 164 CTLESC Escape next character 165 166 A variable substitution contains the following elements: 167 168 CTLVAR type name '=' [ alternative-text CTLENDVAR ] 169 170 The type field is a single character specifying the type of sub- 171 stitution. The possible types are: 172 173 VSNORMAL $var 174 VSMINUS ${var-text} 175 VSMINUS|VSNUL ${var:-text} 176 VSPLUS ${var+text} 177 VSPLUS|VSNUL ${var:+text} 178 VSQUESTION ${var?text} 179 VSQUESTION|VSNUL ${var:?text} 180 VSASSIGN ${var=text} 181 VSASSIGN|VSNUL ${var=text} 182 183 In addition, the type field will have the VSQUOTE flag set if the 184 variable is enclosed in double quotes. The name of the variable 185 comes next, terminated by an equals sign. If the type is not 186 VSNORMAL, then the text field in the substitution follows, ter- 187 minated by a CTLENDVAR byte. 188 189 Commands in back quotes are parsed and stored in a linked list. 190 The locations of these commands in the string are indicated by 191 CTLBACKQ and CTLBACKQ+CTLQUOTE characters, depending upon whether 192 the back quotes were enclosed in double quotes. 193 194 The character CTLESC escapes the next character, so that in case 195 any of the CTL characters mentioned above appear in the input, 196 they can be passed through transparently. CTLESC is also used to 197 escape '*', '?', '[', and '!' characters which were quoted by the 198 user and thus should not be used for file name generation. 199 200 CTLESC characters have proved to be particularly tricky to get 201 right. In the case of here documents which are not subject to 202 variable and command substitution, the parser doesn't insert any 203 CTLESC characters to begin with (so the contents of the text 204 field can be written without any processing). Other here docu- 205 ments, and words which are not subject to splitting and file name 206 generation, have the CTLESC characters removed during the vari- 207 able and command substitution phase. Words which are subject 208 splitting and file name generation have the CTLESC characters re- 209 moved as part of the file name phase. 210 211 EXECUTION: Command execution is handled by the following files: 212 eval.c The top level routines. 213 redir.c Code to handle redirection of input and output. 214 jobs.c Code to handle forking, waiting, and job control. 215 exec.c Code to to path searches and the actual exec sys call. 216 expand.c Code to evaluate arguments. 217 var.c Maintains the variable symbol table. Called from expand.c. 218 219 EVAL.C: Evaltree recursively executes a parse tree. The exit 220 status is returned in the global variable exitstatus. The alter- 221 native entry evalbackcmd is called to evaluate commands in back 222 quotes. It saves the result in memory if the command is a buil- 223 tin; otherwise it forks off a child to execute the command and 224 connects the standard output of the child to a pipe. 225 226 JOBS.C: To create a process, you call makejob to return a job 227 structure, and then call forkshell (passing the job structure as 228 an argument) to create the process. Waitforjob waits for a job 229 to complete. These routines take care of process groups if job 230 control is defined. 231 232 REDIR.C: Ash allows file descriptors to be redirected and then 233 restored without forking off a child process. This is accom- 234 plished by duplicating the original file descriptors. The redir- 235 tab structure records where the file descriptors have be dupli- 236 cated to. 237 238 EXEC.C: The routine find_command locates a command, and enters 239 the command in the hash table if it is not already there. The 240 third argument specifies whether it is to print an error message 241 if the command is not found. (When a pipeline is set up, 242 find_command is called for all the commands in the pipeline be- 243 fore any forking is done, so to get the commands into the hash 244 table of the parent process. But to make command hashing as 245 transparent as possible, we silently ignore errors at that point 246 and only print error messages if the command cannot be found 247 later.) 248 249 The routine shellexec is the interface to the exec system call. 250 251 EXPAND.C: Arguments are processed in three passes. The first 252 (performed by the routine argstr) performs variable and command 253 substitution. The second (ifsbreakup) performs word splitting 254 and the third (expandmeta) performs file name generation. If the 255 "/u" directory is simulated, then when "/u/username" is replaced 256 by the user's home directory, the flag "didudir" is set. This 257 tells the cd command that it should print out the directory name, 258 just as it would if the "/u" directory were implemented using 259 symbolic links. 260 261 VAR.C: Variables are stored in a hash table. Probably we should 262 switch to extensible hashing. The variable name is stored in the 263 same string as the value (using the format "name=value") so that 264 no string copying is needed to create the environment of a com- 265 mand. Variables which the shell references internally are preal- 266 located so that the shell can reference the values of these vari- 267 ables without doing a lookup. 268 269 When a program is run, the code in eval.c sticks any environment 270 variables which precede the command (as in "PATH=xxx command") in 271 the variable table as the simplest way to strip duplicates, and 272 then calls "environment" to get the value of the environment. 273 There are two consequences of this. First, if an assignment to 274 PATH precedes the command, the value of PATH before the assign- 275 ment must be remembered and passed to shellexec. Second, if the 276 program turns out to be a shell procedure, the strings from the 277 environment variables which preceded the command must be pulled 278 out of the table and replaced with strings obtained from malloc, 279 since the former will automatically be freed when the stack (see 280 the entry on memalloc.c) is emptied. 281 282 BUILTIN COMMANDS: The procedures for handling these are scat- 283 tered throughout the code, depending on which location appears 284 most appropriate. They can be recognized because their names al- 285 ways end in "cmd". The mapping from names to procedures is 286 specified in the file builtins, which is processed by the mkbuil- 287 tins command. 288 289 A builtin command is invoked with argc and argv set up like a 290 normal program. A builtin command is allowed to overwrite its 291 arguments. Builtin routines can call nextopt to do option pars- 292 ing. This is kind of like getopt, but you don't pass argc and 293 argv to it. Builtin routines can also call error. This routine 294 normally terminates the shell (or returns to the main command 295 loop if the shell is interactive), but when called from a builtin 296 command it causes the builtin command to terminate with an exit 297 status of 2. 298 299 The directory bltins contains commands which can be compiled in- 300 dependently but can also be built into the shell for efficiency 301 reasons. The makefile in this directory compiles these programs 302 in the normal fashion (so that they can be run regardless of 303 whether the invoker is ash), but also creates a library named 304 bltinlib.a which can be linked with ash. The header file bltin.h 305 takes care of most of the differences between the ash and the 306 stand-alone environment. The user should call the main routine 307 "main", and #define main to be the name of the routine to use 308 when the program is linked into ash. This #define should appear 309 before bltin.h is included; bltin.h will #undef main if the pro- 310 gram is to be compiled stand-alone. 311 312 CD.C: This file defines the cd and pwd builtins. The pwd com- 313 mand runs /bin/pwd the first time it is invoked (unless the user 314 has already done a cd to an absolute pathname), but then 315 remembers the current directory and updates it when the cd com- 316 mand is run, so subsequent pwd commands run very fast. The main 317 complication in the cd command is in the docd command, which 318 resolves symbolic links into actual names and informs the user 319 where the user ended up if he crossed a symbolic link. 320 321 SIGNALS: Trap.c implements the trap command. The routine set- 322 signal figures out what action should be taken when a signal is 323 received and invokes the signal system call to set the signal ac- 324 tion appropriately. When a signal that a user has set a trap for 325 is caught, the routine "onsig" sets a flag. The routine dotrap 326 is called at appropriate points to actually handle the signal. 327 When an interrupt is caught and no trap has been set for that 328 signal, the routine "onint" in error.c is called. 329 330 OUTPUT: Ash uses it's own output routines. There are three out- 331 put structures allocated. "Output" represents the standard out- 332 put, "errout" the standard error, and "memout" contains output 333 which is to be stored in memory. This last is used when a buil- 334 tin command appears in backquotes, to allow its output to be col- 335 lected without doing any I/O through the UNIX operating system. 336 The variables out1 and out2 normally point to output and errout, 337 respectively, but they are set to point to memout when appropri- 338 ate inside backquotes. 339 340 INPUT: The basic input routine is pgetc, which reads from the 341 current input file. There is a stack of input files; the current 342 input file is the top file on this stack. The code allows the 343 input to come from a string rather than a file. (This is for the 344 -c option and the "." and eval builtin commands.) The global 345 variable plinno is saved and restored when files are pushed and 346 popped from the stack. The parser routines store the number of 347 the current line in this variable. 348 349 DEBUGGING: If DEBUG is defined in shell.h, then the shell will 350 write debugging information to the file $HOME/trace. Most of 351 this is done using the TRACE macro, which takes a set of printf 352 arguments inside two sets of parenthesis. Example: 353 "TRACE(("n=%d0, n))". The double parenthesis are necessary be- 354 cause the preprocessor can't handle functions with a variable 355 number of arguments. Defining DEBUG also causes the shell to 356 generate a core dump if it is sent a quit signal. The tracing 357 code is in show.c. 358
