Eric Raymond:
(1) Added and documented the capability for shlex to handle
lexical-level inclusion and a stack of input sources. Also, the input
stream member is now documented, and the constructor takes an optional
source-filename. The class provides facilities to generate error
messages that track file and line number.
(2) Add a convenience function to generate C-compiler style error
leaders.
diff --git a/Doc/lib/libshlex.tex b/Doc/lib/libshlex.tex
index 174a39f..a070908 100644
--- a/Doc/lib/libshlex.tex
+++ b/Doc/lib/libshlex.tex
@@ -13,12 +13,15 @@
be useful for writing minilanguages, e.g.\ in run control files for
Python applications.
-\begin{classdesc}{shlex}{\optional{stream}}
+\begin{classdesc}{shlex}{\optional{stream}, \optional{file}}
A \class{shlex} instance or subclass instance is a lexical analyzer
object. The initialization argument, if present, specifies where to
read characters from. It must be a file- or stream-like object with
\method{read()} and \method{readline()} methods. If no argument is given,
-input will be taken from \code{sys.stdin}.
+input will be taken from \code{sys.stdin}. The second optional
+argument is a filename string, which sets the initial value of the
+\member{infile} member. If the stream argument is omitted or
+equal to \code{sys.stdin}, this second argument defauilts to ``stdin''.
\end{classdesc}
@@ -43,6 +46,38 @@
Push the argument onto the token stack.
\end{methoddesc}
+\begin{methoddesc}{read_token}{}
+Read a raw token. Ignore the pushback stack, and do not interpret source
+requests. (This is not ordinarily a useful entry point, and is
+documented here only for the sake of completeness.)
+\end{methoddesc}
+
+\begin{methoddesc}{openhook}{filename}
+When shlex detects a source request (see \member{source} below)
+this method is given the following token as argument, and expected to
+return a tuple consisting of a filename and an opened stream object.
+
+Normally, this method just strips any quotes off the argument and
+treats it as a filename, calling \code{open()} on it. It is exposed so that
+you can use it to implement directory search paths, addition of
+file extensions, and other namespace hacks.
+
+There is no corresponding `close' hook, but a shlex instance will call
+the \code{close()} method of the sourced input stream when it returns EOF.
+\end{methoddesc}
+
+\begin{methoddesc}{error_leader}{\optional{file}, \optional{line}}
+This method generates an error message leader in the format of a
+Unix C compiler error label; the format is '"\%s", line \%d: ',
+where the \%s is replaced with the name of the current source file and
+the \%d with the current input line number (the optional arguments
+can be used to override these).
+
+This convenience is provided to encourage shlex users to generate
+error messages in the standard, parseable format understood by Emacs
+and other Unix tools.
+\end{methoddesc}
+
Instances of \class{shlex} subclasses have some public instance
variables which either control lexical analysis or can be used
for debugging:
@@ -72,6 +107,33 @@
\ASCII{} single and double quotes.
\end{memberdesc}
+\begin{memberdesc}{infile}
+The name of the current input file, as initially set at class
+instantiation time or stacked by later source requests. It may
+be useful to examine this when constructing error messages.
+\end{memberdesc}
+
+\begin{memberdesc}{instream}
+The input stream from which this shlex instance is reading characters.
+\end{memberdesc}
+
+\begin{memberdesc}{source}
+This member is None by default. If you assign a string to it, that
+string will be recognized as a lexical-level inclusion request similar
+to the `source' keyword in various shells. That is, the immediately
+following token will opened as a filename and input taken from that
+stream until EOF, at which point the \code{close()} method of that
+stream will be called and the input source will again become the
+original input stream. Source requests may be stacked any number of
+levels deep.
+\end{memberdesc}
+
+\begin{memberdesc}{debug}
+If this member is numeric and 1 or more, a shlex instance will print
+verbose progress output on its behavior. If you need to use this,
+you can read the module source code to learn the details.
+\end{memberdesc}
+
Note that any character not declared to be a word character,
whitespace, or a quote will be returned as a single-character token.