Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 1 | \section{\module{shlex} --- |
Fred Drake | 184e836 | 1999-05-11 15:14:15 +0000 | [diff] [blame] | 2 | Simple lexical analysis} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 3 | |
| 4 | \declaremodule{standard}{shlex} |
Fred Drake | c116b82 | 2001-05-09 15:50:17 +0000 | [diff] [blame] | 5 | \modulesynopsis{Simple lexical analysis for \UNIX\ shell-like languages.} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 6 | \moduleauthor{Eric S. Raymond}{esr@snark.thyrsus.com} |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 7 | \moduleauthor{Gustavo Niemeyer}{niemeyer@conectiva.com} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 8 | \sectionauthor{Eric S. Raymond}{esr@snark.thyrsus.com} |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 9 | \sectionauthor{Gustavo Niemeyer}{niemeyer@conectiva.com} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 10 | |
Fred Drake | 292b9eb | 1998-12-22 18:40:50 +0000 | [diff] [blame] | 11 | \versionadded{1.5.2} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 12 | |
| 13 | The \class{shlex} class makes it easy to write lexical analyzers for |
| 14 | simple syntaxes resembling that of the \UNIX{} shell. This will often |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 15 | be useful for writing minilanguages, (for example, in run control |
| 16 | files for Python applications) or for parsing quoted strings. |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 17 | |
| 18 | The \module{shlex} module defines the following functions: |
| 19 | |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 20 | \begin{funcdesc}{split}{s\optional{, comments}} |
Gustavo Niemeyer | 48f3dcc | 2003-04-20 01:57:03 +0000 | [diff] [blame] | 21 | Split the string \var{s} using shell-like syntax. If \var{comments} is |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 22 | \constant{False} (the default), the parsing of comments in the given |
| 23 | string will be disabled (setting the \member{commenters} member of the |
| 24 | \class{shlex} instance to the empty string). This function operates |
| 25 | in \POSIX{} mode. |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 26 | \versionadded{2.3} |
| 27 | \end{funcdesc} |
| 28 | |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 29 | The \module{shlex} module defines the following class: |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 30 | |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 31 | \begin{classdesc}{shlex}{\optional{instream\optional{, |
| 32 | infile\optional{, posix}}}} |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 33 | A \class{shlex} instance or subclass instance is a lexical analyzer |
| 34 | object. The initialization argument, if present, specifies where to |
| 35 | read characters from. It must be a file-/stream-like object with |
| 36 | \method{read()} and \method{readline()} methods, or a string (strings |
| 37 | are accepted since Python 2.3). If no argument is given, input will be |
| 38 | taken from \code{sys.stdin}. The second optional argument is a filename |
| 39 | string, which sets the initial value of the \member{infile} member. If |
| 40 | the \var{instream} argument is omitted or equal to \code{sys.stdin}, |
| 41 | this second argument defaults to ``stdin''. The \var{posix} argument |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 42 | was introduced in Python 2.3, and defines the operational mode. When |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 43 | \var{posix} is not true (default), the \class{shlex} instance will |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 44 | operate in compatibility mode. When operating in \POSIX{} mode, |
| 45 | \class{shlex} will try to be as close as possible to the \POSIX{} shell |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 46 | parsing rules. See section~\ref{shlex-objects}. |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 47 | \end{classdesc} |
| 48 | |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 49 | \begin{seealso} |
| 50 | \seemodule{ConfigParser}{Parser for configuration files similar to the |
| 51 | Windows \file{.ini} files.} |
| 52 | \end{seealso} |
| 53 | |
| 54 | |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 55 | \subsection{shlex Objects \label{shlex-objects}} |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 56 | |
| 57 | A \class{shlex} instance has the following methods: |
| 58 | |
| 59 | \begin{methoddesc}{get_token}{} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 60 | Return a token. If tokens have been stacked using |
| 61 | \method{push_token()}, pop a token off the stack. Otherwise, read one |
| 62 | from the input stream. If reading encounters an immediate |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 63 | end-of-file, \member{self.eof} is returned (the empty string (\code{''}) |
| 64 | in non-\POSIX{} mode, and \code{None} in \POSIX{} mode). |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 65 | \end{methoddesc} |
| 66 | |
| 67 | \begin{methoddesc}{push_token}{str} |
| 68 | Push the argument onto the token stack. |
| 69 | \end{methoddesc} |
| 70 | |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 71 | \begin{methoddesc}{read_token}{} |
| 72 | Read a raw token. Ignore the pushback stack, and do not interpret source |
| 73 | requests. (This is not ordinarily a useful entry point, and is |
| 74 | documented here only for the sake of completeness.) |
| 75 | \end{methoddesc} |
| 76 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 77 | \begin{methoddesc}{sourcehook}{filename} |
| 78 | When \class{shlex} detects a source request (see |
| 79 | \member{source} below) this method is given the following token as |
| 80 | argument, and expected to return a tuple consisting of a filename and |
| 81 | an open file-like object. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 82 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 83 | Normally, this method first strips any quotes off the argument. If |
| 84 | the result is an absolute pathname, or there was no previous source |
| 85 | request in effect, or the previous source was a stream |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 86 | (such as \code{sys.stdin}), the result is left alone. Otherwise, if the |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 87 | result is a relative pathname, the directory part of the name of the |
| 88 | file immediately before it on the source inclusion stack is prepended |
| 89 | (this behavior is like the way the C preprocessor handles |
Eric S. Raymond | bd1a489 | 2001-01-16 14:18:55 +0000 | [diff] [blame] | 90 | \code{\#include "file.h"}). |
| 91 | |
| 92 | The result of the manipulations is treated as a filename, and returned |
| 93 | as the first component of the tuple, with |
| 94 | \function{open()} called on it to yield the second component. (Note: |
| 95 | this is the reverse of the order of arguments in instance initialization!) |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 96 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 97 | This hook is exposed so that you can use it to implement directory |
| 98 | search paths, addition of file extensions, and other namespace hacks. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 99 | There is no corresponding `close' hook, but a shlex instance will call |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 100 | the \method{close()} method of the sourced input stream when it |
| 101 | returns \EOF. |
Eric S. Raymond | bd1a489 | 2001-01-16 14:18:55 +0000 | [diff] [blame] | 102 | |
Fred Drake | 25be193 | 2001-01-16 20:52:41 +0000 | [diff] [blame] | 103 | For more explicit control of source stacking, use the |
| 104 | \method{push_source()} and \method{pop_source()} methods. |
Eric S. Raymond | bd1a489 | 2001-01-16 14:18:55 +0000 | [diff] [blame] | 105 | \end{methoddesc} |
| 106 | |
| 107 | \begin{methoddesc}{push_source}{stream\optional{, filename}} |
| 108 | Push an input source stream onto the input stack. If the filename |
| 109 | argument is specified it will later be available for use in error |
| 110 | messages. This is the same method used internally by the |
Fred Drake | 25be193 | 2001-01-16 20:52:41 +0000 | [diff] [blame] | 111 | \method{sourcehook} method. |
| 112 | \versionadded{2.1} |
Eric S. Raymond | bd1a489 | 2001-01-16 14:18:55 +0000 | [diff] [blame] | 113 | \end{methoddesc} |
| 114 | |
Fred Drake | 25be193 | 2001-01-16 20:52:41 +0000 | [diff] [blame] | 115 | \begin{methoddesc}{pop_source}{} |
Eric S. Raymond | bd1a489 | 2001-01-16 14:18:55 +0000 | [diff] [blame] | 116 | Pop the last-pushed input source from the input stack. |
| 117 | This is the same method used internally when the lexer reaches |
Raymond Hettinger | b67449d | 2003-09-08 18:52:18 +0000 | [diff] [blame] | 118 | \EOF{} on a stacked input stream. |
Fred Drake | 25be193 | 2001-01-16 20:52:41 +0000 | [diff] [blame] | 119 | \versionadded{2.1} |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 120 | \end{methoddesc} |
| 121 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 122 | \begin{methoddesc}{error_leader}{\optional{file\optional{, line}}} |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 123 | This method generates an error message leader in the format of a |
Fred Drake | 25be193 | 2001-01-16 20:52:41 +0000 | [diff] [blame] | 124 | \UNIX{} C compiler error label; the format is \code{'"\%s", line \%d: '}, |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 125 | where the \samp{\%s} is replaced with the name of the current source |
| 126 | file and the \samp{\%d} with the current input line number (the |
| 127 | optional arguments can be used to override these). |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 128 | |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 129 | This convenience is provided to encourage \module{shlex} users to |
| 130 | generate error messages in the standard, parseable format understood |
| 131 | by Emacs and other \UNIX{} tools. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 132 | \end{methoddesc} |
| 133 | |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 134 | Instances of \class{shlex} subclasses have some public instance |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 135 | variables which either control lexical analysis or can be used for |
| 136 | debugging: |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 137 | |
| 138 | \begin{memberdesc}{commenters} |
| 139 | The string of characters that are recognized as comment beginners. |
| 140 | All characters from the comment beginner to end of line are ignored. |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 141 | Includes just \character{\#} by default. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 142 | \end{memberdesc} |
| 143 | |
| 144 | \begin{memberdesc}{wordchars} |
| 145 | The string of characters that will accumulate into multi-character |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 146 | tokens. By default, includes all \ASCII{} alphanumerics and |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 147 | underscore. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 148 | \end{memberdesc} |
| 149 | |
| 150 | \begin{memberdesc}{whitespace} |
| 151 | Characters that will be considered whitespace and skipped. Whitespace |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 152 | bounds tokens. By default, includes space, tab, linefeed and |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 153 | carriage-return. |
| 154 | \end{memberdesc} |
| 155 | |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 156 | \begin{memberdesc}{escape} |
| 157 | Characters that will be considered as escape. This will be only used |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 158 | in \POSIX{} mode, and includes just \character{\textbackslash} by default. |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 159 | \versionadded{2.3} |
| 160 | \end{memberdesc} |
| 161 | |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 162 | \begin{memberdesc}{quotes} |
| 163 | Characters that will be considered string quotes. The token |
| 164 | accumulates until the same quote is encountered again (thus, different |
Fred Drake | 184e836 | 1999-05-11 15:14:15 +0000 | [diff] [blame] | 165 | quote types protect each other as in the shell.) By default, includes |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 166 | \ASCII{} single and double quotes. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 167 | \end{memberdesc} |
| 168 | |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 169 | \begin{memberdesc}{escapedquotes} |
| 170 | Characters in \member{quotes} that will interpret escape characters |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 171 | defined in \member{escape}. This is only used in \POSIX{} mode, and |
| 172 | includes just \character{"} by default. |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 173 | \versionadded{2.3} |
| 174 | \end{memberdesc} |
| 175 | |
| 176 | \begin{memberdesc}{whitespace_split} |
Neal Norwitz | 10cf218 | 2003-04-17 23:09:08 +0000 | [diff] [blame] | 177 | If \code{True}, tokens will only be split in whitespaces. This is useful, for |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 178 | example, for parsing command lines with \class{shlex}, getting tokens |
| 179 | in a similar way to shell arguments. |
| 180 | \versionadded{2.3} |
| 181 | \end{memberdesc} |
| 182 | |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 183 | \begin{memberdesc}{infile} |
| 184 | The name of the current input file, as initially set at class |
| 185 | instantiation time or stacked by later source requests. It may |
| 186 | be useful to examine this when constructing error messages. |
| 187 | \end{memberdesc} |
| 188 | |
| 189 | \begin{memberdesc}{instream} |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 190 | The input stream from which this \class{shlex} instance is reading |
| 191 | characters. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 192 | \end{memberdesc} |
| 193 | |
| 194 | \begin{memberdesc}{source} |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 195 | This member is \code{None} by default. If you assign a string to it, |
| 196 | that string will be recognized as a lexical-level inclusion request |
| 197 | similar to the \samp{source} keyword in various shells. That is, the |
| 198 | immediately following token will opened as a filename and input taken |
| 199 | from that stream until \EOF, at which point the \method{close()} |
| 200 | method of that stream will be called and the input source will again |
| 201 | become the original input stream. Source requests may be stacked any |
| 202 | number of levels deep. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 203 | \end{memberdesc} |
| 204 | |
| 205 | \begin{memberdesc}{debug} |
Fred Drake | 52dc76c | 2000-07-03 09:56:23 +0000 | [diff] [blame] | 206 | If this member is numeric and \code{1} or more, a \class{shlex} |
| 207 | instance will print verbose progress output on its behavior. If you |
| 208 | need to use this, you can read the module source code to learn the |
| 209 | details. |
Guido van Rossum | d67ddbb | 2000-05-01 20:14:47 +0000 | [diff] [blame] | 210 | \end{memberdesc} |
| 211 | |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 212 | \begin{memberdesc}{lineno} |
| 213 | Source line number (count of newlines seen so far plus one). |
| 214 | \end{memberdesc} |
| 215 | |
| 216 | \begin{memberdesc}{token} |
Fred Drake | 1189fa9 | 1998-12-22 18:24:13 +0000 | [diff] [blame] | 217 | The token buffer. It may be useful to examine this when catching |
| 218 | exceptions. |
Guido van Rossum | 5e97c9d | 1998-12-22 05:18:24 +0000 | [diff] [blame] | 219 | \end{memberdesc} |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 220 | |
| 221 | \begin{memberdesc}{eof} |
| 222 | Token used to determine end of file. This will be set to the empty |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 223 | string (\code{''}), in non-\POSIX{} mode, and to \code{None} in |
| 224 | \POSIX{} mode. |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 225 | \versionadded{2.3} |
| 226 | \end{memberdesc} |
| 227 | |
| 228 | \subsection{Parsing Rules\label{shlex-parsing-rules}} |
| 229 | |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 230 | When operating in non-\POSIX{} mode, \class{shlex} will try to obey to |
| 231 | the following rules. |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 232 | |
| 233 | \begin{itemize} |
| 234 | \item Quote characters are not recognized within words |
| 235 | (\code{Do"Not"Separate} is parsed as the single word |
| 236 | \code{Do"Not"Separate}); |
| 237 | \item Escape characters are not recognized; |
| 238 | \item Enclosing characters in quotes preserve the literal value of |
| 239 | all characters within the quotes; |
| 240 | \item Closing quotes separate words (\code{"Do"Separate} is parsed |
| 241 | as \code{"Do"} and \code{Separate}); |
| 242 | \item If \member{whitespace_split} is \code{False}, any character not |
| 243 | declared to be a word character, whitespace, or a quote will be |
| 244 | returned as a single-character token. If it is \code{True}, |
| 245 | \class{shlex} will only split words in whitespaces; |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 246 | \item EOF is signaled with an empty string (\code{''}); |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 247 | \item It's not possible to parse empty strings, even if quoted. |
| 248 | \end{itemize} |
| 249 | |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 250 | When operating in \POSIX{} mode, \class{shlex} will try to obey to the |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 251 | following parsing rules. |
| 252 | |
| 253 | \begin{itemize} |
| 254 | \item Quotes are stripped out, and do not separate words |
| 255 | (\code{"Do"Not"Separate"} is parsed as the single word |
| 256 | \code{DoNotSeparate}); |
| 257 | \item Non-quoted escape characters (e.g. \character{\textbackslash}) |
| 258 | preserve the literal value of the next character that follows; |
| 259 | \item Enclosing characters in quotes which are not part of |
| 260 | \member{escapedquotes} (e.g. \character{'}) preserve the literal |
| 261 | value of all characters within the quotes; |
| 262 | \item Enclosing characters in quotes which are part of |
| 263 | \member{escapedquotes} (e.g. \character{"}) preserves the literal |
| 264 | value of all characters within the quotes, with the exception of |
| 265 | the characters mentioned in \member{escape}. The escape characters |
| 266 | retain its special meaning only when followed by the quote in use, |
| 267 | or the escape character itself. Otherwise the escape character |
| 268 | will be considered a normal character. |
Fred Drake | af78512 | 2003-12-31 05:18:46 +0000 | [diff] [blame^] | 269 | \item EOF is signaled with a \constant{None} value; |
Fred Drake | aa3b5d2 | 2003-04-17 21:49:04 +0000 | [diff] [blame] | 270 | \item Quoted empty strings (\code{''}) are allowed; |
Gustavo Niemeyer | 68d8cef | 2003-04-17 21:31:33 +0000 | [diff] [blame] | 271 | \end{itemize} |
| 272 | |