Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 1 | \section{\module{shlex} --- |
Fred Drake | 184e836 | 1999-05-11 15:14:15 +0000 | [diff] [blame] | 2 | Simple lexical analysis} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 3 | |
| 4 | \declaremodule{standard}{shlex} |
Fred Drake | 39cddb7 | 1999-01-12 19:22:11 +0000 | [diff] [blame] | 5 | \modulesynopsis{Simple lexical analysis for \UNIX{} shell-like languages.} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 6 | \moduleauthor{Eric S. Raymond}{esr@snark.thyrsus.com} |
| 7 | \sectionauthor{Eric S. Raymond}{esr@snark.thyrsus.com} |
| 8 | |
Fred Drake | 292b9eb | 1998-12-22 18:40:50 +0000 | [diff] [blame] | 9 | \versionadded{1.5.2} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 10 | |
| 11 | The \class{shlex} class makes it easy to write lexical analyzers for |
| 12 | simple syntaxes resembling that of the \UNIX{} shell. This will often |
| 13 | be useful for writing minilanguages, e.g.\ in run control files for |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 14 | Python applications. |
| 15 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 16 | \begin{classdesc}{shlex}{\optional{stream\optional{, file}}} |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 17 | A \class{shlex} instance or subclass instance is a lexical analyzer |
| 18 | object. The initialization argument, if present, specifies where to |
| 19 | read characters from. It must be a file- or stream-like object with |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 20 | \method{read()} and \method{readline()} methods. If no argument is given, |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 21 | input will be taken from \code{sys.stdin}. The second optional |
| 22 | argument is a filename string, which sets the initial value of the |
| 23 | \member{infile} member. If the stream argument is omitted or |
Thomas Wouters | f831663 | 2000-07-16 19:01:10 +0000 | [diff] [blame] | 24 | equal to \code{sys.stdin}, this second argument defaults to ``stdin''. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 25 | \end{classdesc} |
| 26 | |
Fred Drake | 184e836 | 1999-05-11 15:14:15 +0000 | [diff] [blame] | 27 | |
| 28 | \begin{seealso} |
| 29 | \seemodule{ConfigParser}{Parser for configuration files similar to the |
| 30 | Windows \file{.ini} files.} |
| 31 | \end{seealso} |
| 32 | |
| 33 | |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 34 | \subsection{shlex Objects \label{shlex-objects}} |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 35 | |
| 36 | A \class{shlex} instance has the following methods: |
| 37 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 38 | |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 39 | \begin{methoddesc}{get_token}{} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 40 | Return a token. If tokens have been stacked using |
| 41 | \method{push_token()}, pop a token off the stack. Otherwise, read one |
| 42 | from the input stream. If reading encounters an immediate |
| 43 | end-of-file, an empty string is returned. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 44 | \end{methoddesc} |
| 45 | |
| 46 | \begin{methoddesc}{push_token}{str} |
| 47 | Push the argument onto the token stack. |
| 48 | \end{methoddesc} |
| 49 | |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 50 | \begin{methoddesc}{read_token}{} |
| 51 | Read a raw token. Ignore the pushback stack, and do not interpret source |
| 52 | requests. (This is not ordinarily a useful entry point, and is |
| 53 | documented here only for the sake of completeness.) |
| 54 | \end{methoddesc} |
| 55 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 56 | \begin{methoddesc}{sourcehook}{filename} |
| 57 | When \class{shlex} detects a source request (see |
| 58 | \member{source} below) this method is given the following token as |
| 59 | argument, and expected to return a tuple consisting of a filename and |
| 60 | an open file-like object. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 61 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 62 | Normally, this method first strips any quotes off the argument. If |
| 63 | the result is an absolute pathname, or there was no previous source |
| 64 | request in effect, or the previous source was a stream |
| 65 | (e.g. \code{sys.stdin}), the result is left alone. Otherwise, if the |
| 66 | result is a relative pathname, the directory part of the name of the |
| 67 | file immediately before it on the source inclusion stack is prepended |
| 68 | (this behavior is like the way the C preprocessor handles |
Eric S. Raymond | bd1a489 | 2001-01-16 14:18:55 +0000 | [diff] [blame^] | 69 | \code{\#include "file.h"}). |
| 70 | |
| 71 | The result of the manipulations is treated as a filename, and returned |
| 72 | as the first component of the tuple, with |
| 73 | \function{open()} called on it to yield the second component. (Note: |
| 74 | this is the reverse of the order of arguments in instance initialization!) |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 75 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 76 | This hook is exposed so that you can use it to implement directory |
| 77 | search paths, addition of file extensions, and other namespace hacks. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 78 | There is no corresponding `close' hook, but a shlex instance will call |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 79 | the \method{close()} method of the sourced input stream when it |
| 80 | returns \EOF. |
Eric S. Raymond | bd1a489 | 2001-01-16 14:18:55 +0000 | [diff] [blame^] | 81 | |
| 82 | For more explicit control of source stacking, use the next two |
| 83 | methods. |
| 84 | \end{methoddesc} |
| 85 | |
| 86 | \begin{methoddesc}{push_source}{stream\optional{, filename}} |
| 87 | Push an input source stream onto the input stack. If the filename |
| 88 | argument is specified it will later be available for use in error |
| 89 | messages. This is the same method used internally by the |
| 90 | \method{sourcehook} method. (New in 2.1) |
| 91 | \end{methoddesc} |
| 92 | |
| 93 | \begin{methoddesc}{pop_source}{}} |
| 94 | Pop the last-pushed input source from the input stack. |
| 95 | This is the same method used internally when the lexer reaches |
| 96 | \EOF on a stacked input stream. (New in 2.1) |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 97 | \end{methoddesc} |
| 98 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 99 | \begin{methoddesc}{error_leader}{\optional{file\optional{, line}}} |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 100 | This method generates an error message leader in the format of a |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 101 | \UNIX{} C compiler error label; the format is '"\%s", line \%d: ', |
| 102 | where the \samp{\%s} is replaced with the name of the current source |
| 103 | file and the \samp{\%d} with the current input line number (the |
| 104 | optional arguments can be used to override these). |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 105 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 106 | This convenience is provided to encourage \module{shlex} users to |
| 107 | generate error messages in the standard, parseable format understood |
| 108 | by Emacs and other \UNIX{} tools. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 109 | \end{methoddesc} |
| 110 | |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 111 | Instances of \class{shlex} subclasses have some public instance |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 112 | variables which either control lexical analysis or can be used for |
| 113 | debugging: |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 114 | |
| 115 | \begin{memberdesc}{commenters} |
| 116 | The string of characters that are recognized as comment beginners. |
| 117 | All characters from the comment beginner to end of line are ignored. |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 118 | Includes just \character{\#} by default. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 119 | \end{memberdesc} |
| 120 | |
| 121 | \begin{memberdesc}{wordchars} |
| 122 | The string of characters that will accumulate into multi-character |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 123 | tokens. By default, includes all \ASCII{} alphanumerics and |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 124 | underscore. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 125 | \end{memberdesc} |
| 126 | |
| 127 | \begin{memberdesc}{whitespace} |
| 128 | Characters that will be considered whitespace and skipped. Whitespace |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 129 | bounds tokens. By default, includes space, tab, linefeed and |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 130 | carriage-return. |
| 131 | \end{memberdesc} |
| 132 | |
| 133 | \begin{memberdesc}{quotes} |
| 134 | Characters that will be considered string quotes. The token |
| 135 | accumulates until the same quote is encountered again (thus, different |
Fred Drake | 184e836 | 1999-05-11 15:14:15 +0000 | [diff] [blame] | 136 | quote types protect each other as in the shell.) By default, includes |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 137 | \ASCII{} single and double quotes. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 138 | \end{memberdesc} |
| 139 | |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 140 | \begin{memberdesc}{infile} |
| 141 | The name of the current input file, as initially set at class |
| 142 | instantiation time or stacked by later source requests. It may |
| 143 | be useful to examine this when constructing error messages. |
| 144 | \end{memberdesc} |
| 145 | |
| 146 | \begin{memberdesc}{instream} |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 147 | The input stream from which this \class{shlex} instance is reading |
| 148 | characters. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 149 | \end{memberdesc} |
| 150 | |
| 151 | \begin{memberdesc}{source} |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 152 | This member is \code{None} by default. If you assign a string to it, |
| 153 | that string will be recognized as a lexical-level inclusion request |
| 154 | similar to the \samp{source} keyword in various shells. That is, the |
| 155 | immediately following token will opened as a filename and input taken |
| 156 | from that stream until \EOF, at which point the \method{close()} |
| 157 | method of that stream will be called and the input source will again |
| 158 | become the original input stream. Source requests may be stacked any |
| 159 | number of levels deep. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 160 | \end{memberdesc} |
| 161 | |
| 162 | \begin{memberdesc}{debug} |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 163 | If this member is numeric and \code{1} or more, a \class{shlex} |
| 164 | instance will print verbose progress output on its behavior. If you |
| 165 | need to use this, you can read the module source code to learn the |
| 166 | details. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 167 | \end{memberdesc} |
| 168 | |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 169 | Note that any character not declared to be a word character, |
| 170 | whitespace, or a quote will be returned as a single-character token. |
| 171 | |
| 172 | Quote and comment characters are not recognized within words. Thus, |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 173 | the bare words \samp{ain't} and \samp{ain\#t} would be returned as single |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 174 | tokens by the default parser. |
| 175 | |
| 176 | \begin{memberdesc}{lineno} |
| 177 | Source line number (count of newlines seen so far plus one). |
| 178 | \end{memberdesc} |
| 179 | |
| 180 | \begin{memberdesc}{token} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 181 | The token buffer. It may be useful to examine this when catching |
| 182 | exceptions. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 183 | \end{memberdesc} |