Initial revision
diff --git a/Doc/ref/ref2.tex b/Doc/ref/ref2.tex
new file mode 100644
index 0000000..250bd2e
--- /dev/null
+++ b/Doc/ref/ref2.tex
@@ -0,0 +1,349 @@
+\chapter{Lexical analysis}
+
+A Python program is read by a {\em parser}. Input to the parser is a
+stream of {\em tokens}, generated by the {\em lexical analyzer}. This
+chapter describes how the lexical analyzer breaks a file into tokens.
+\index{lexical analysis}
+\index{parser}
+\index{token}
+
+\section{Line structure}
+
+A Python program is divided in a number of logical lines. The end of
+a logical line is represented by the token NEWLINE. Statements cannot
+cross logical line boundaries except where NEWLINE is allowed by the
+syntax (e.g. between statements in compound statements).
+\index{line structure}
+\index{logical line}
+\index{NEWLINE token}
+
+\subsection{Comments}
+
+A comment starts with a hash character (\verb\#\) that is not part of
+a string literal, and ends at the end of the physical line. A comment
+always signifies the end of the logical line. Comments are ignored by
+the syntax.
+\index{comment}
+\index{logical line}
+\index{physical line}
+\index{hash character}
+
+\subsection{Line joining}
+
+Two or more physical lines may be joined into logical lines using
+backslash characters (\verb/\/), as follows: when a physical line ends
+in a backslash that is not part of a string literal or comment, it is
+joined with the following forming a single logical line, deleting the
+backslash and the following end-of-line character. For example:
+\index{physical line}
+\index{line joining}
+\index{backslash character}
+%
+\begin{verbatim}
+month_names = ['Januari', 'Februari', 'Maart', \
+ 'April', 'Mei', 'Juni', \
+ 'Juli', 'Augustus', 'September', \
+ 'Oktober', 'November', 'December']
+\end{verbatim}
+
+\subsection{Blank lines}
+
+A logical line that contains only spaces, tabs, and possibly a
+comment, is ignored (i.e., no NEWLINE token is generated), except that
+during interactive input of statements, an entirely blank logical line
+terminates a multi-line statement.
+\index{blank line}
+
+\subsection{Indentation}
+
+Leading whitespace (spaces and tabs) at the beginning of a logical
+line is used to compute the indentation level of the line, which in
+turn is used to determine the grouping of statements.
+\index{indentation}
+\index{whitespace}
+\index{leading whitespace}
+\index{space}
+\index{tab}
+\index{grouping}
+\index{statement grouping}
+
+First, tabs are replaced (from left to right) by one to eight spaces
+such that the total number of characters up to there is a multiple of
+eight (this is intended to be the same rule as used by {\UNIX}). The
+total number of spaces preceding the first non-blank character then
+determines the line's indentation. Indentation cannot be split over
+multiple physical lines using backslashes.
+
+The indentation levels of consecutive lines are used to generate
+INDENT and DEDENT tokens, using a stack, as follows.
+\index{INDENT token}
+\index{DEDENT token}
+
+Before the first line of the file is read, a single zero is pushed on
+the stack; this will never be popped off again. The numbers pushed on
+the stack will always be strictly increasing from bottom to top. At
+the beginning of each logical line, the line's indentation level is
+compared to the top of the stack. If it is equal, nothing happens.
+If it is larger, it is pushed on the stack, and one INDENT token is
+generated. If it is smaller, it {\em must} be one of the numbers
+occurring on the stack; all numbers on the stack that are larger are
+popped off, and for each number popped off a DEDENT token is
+generated. At the end of the file, a DEDENT token is generated for
+each number remaining on the stack that is larger than zero.
+
+Here is an example of a correctly (though confusingly) indented piece
+of Python code:
+
+\begin{verbatim}
+def perm(l):
+ # Compute the list of all permutations of l
+
+ if len(l) <= 1:
+ return [l]
+ r = []
+ for i in range(len(l)):
+ s = l[:i] + l[i+1:]
+ p = perm(s)
+ for x in p:
+ r.append(l[i:i+1] + x)
+ return r
+\end{verbatim}
+
+The following example shows various indentation errors:
+
+\begin{verbatim}
+ def perm(l): # error: first line indented
+ for i in range(len(l)): # error: not indented
+ s = l[:i] + l[i+1:]
+ p = perm(l[:i] + l[i+1:]) # error: unexpected indent
+ for x in p:
+ r.append(l[i:i+1] + x)
+ return r # error: inconsistent dedent
+\end{verbatim}
+
+(Actually, the first three errors are detected by the parser; only the
+last error is found by the lexical analyzer --- the indentation of
+\verb\return r\ does not match a level popped off the stack.)
+
+\section{Other tokens}
+
+Besides NEWLINE, INDENT and DEDENT, the following categories of tokens
+exist: identifiers, keywords, literals, operators, and delimiters.
+Spaces and tabs are not tokens, but serve to delimit tokens. Where
+ambiguity exists, a token comprises the longest possible string that
+forms a legal token, when read from left to right.
+
+\section{Identifiers}
+
+Identifiers (also referred to as names) are described by the following
+lexical definitions:
+\index{identifier}
+\index{name}
+
+\begin{verbatim}
+identifier: (letter|"_") (letter|digit|"_")*
+letter: lowercase | uppercase
+lowercase: "a"..."z"
+uppercase: "A"..."Z"
+digit: "0"..."9"
+\end{verbatim}
+
+Identifiers are unlimited in length. Case is significant.
+
+\subsection{Keywords}
+
+The following identifiers are used as reserved words, or {\em
+keywords} of the language, and cannot be used as ordinary
+identifiers. They must be spelled exactly as written here:
+\index{keyword}
+\index{reserved word}
+
+\begin{verbatim}
+and del for in print
+break elif from is raise
+class else global not return
+continue except if or try
+def finally import pass while
+\end{verbatim}
+
+% # This Python program sorts and formats the above table
+% import string
+% l = []
+% try:
+% while 1:
+% l = l + string.split(raw_input())
+% except EOFError:
+% pass
+% l.sort()
+% for i in range((len(l)+4)/5):
+% for j in range(i, len(l), 5):
+% print string.ljust(l[j], 10),
+% print
+
+\section{Literals} \label{literals}
+
+Literals are notations for constant values of some built-in types.
+\index{literal}
+\index{constant}
+
+\subsection{String literals}
+
+String literals are described by the following lexical definitions:
+\index{string literal}
+
+\begin{verbatim}
+stringliteral: "'" stringitem* "'"
+stringitem: stringchar | escapeseq
+stringchar: <any ASCII character except newline or "\" or "'">
+escapeseq: "'" <any ASCII character except newline>
+\end{verbatim}
+\index{ASCII}
+
+String literals cannot span physical line boundaries. Escape
+sequences in strings are actually interpreted according to rules
+similar to those used by Standard C. The recognized escape sequences
+are:
+\index{physical line}
+\index{escape sequence}
+\index{Standard C}
+\index{C}
+
+\begin{center}
+\begin{tabular}{|l|l|}
+\hline
+\verb/\\/ & Backslash (\verb/\/) \\
+\verb/\'/ & Single quote (\verb/'/) \\
+\verb/\a/ & ASCII Bell (BEL) \\
+\verb/\b/ & ASCII Backspace (BS) \\
+%\verb/\E/ & ASCII Escape (ESC) \\
+\verb/\f/ & ASCII Formfeed (FF) \\
+\verb/\n/ & ASCII Linefeed (LF) \\
+\verb/\r/ & ASCII Carriage Return (CR) \\
+\verb/\t/ & ASCII Horizontal Tab (TAB) \\
+\verb/\v/ & ASCII Vertical Tab (VT) \\
+\verb/\/{\em ooo} & ASCII character with octal value {\em ooo} \\
+\verb/\x/{\em xx...} & ASCII character with hex value {\em xx...} \\
+\hline
+\end{tabular}
+\end{center}
+\index{ASCII}
+
+In strict compatibility with Standard C, up to three octal digits are
+accepted, but an unlimited number of hex digits is taken to be part of
+the hex escape (and then the lower 8 bits of the resulting hex number
+are used in all current implementations...).
+
+All unrecognized escape sequences are left in the string unchanged,
+i.e., {\em the backslash is left in the string.} (This behavior is
+useful when debugging: if an escape sequence is mistyped, the
+resulting output is more easily recognized as broken. It also helps a
+great deal for string literals used as regular expressions or
+otherwise passed to other modules that do their own escape handling.)
+\index{unrecognized escape sequence}
+
+\subsection{Numeric literals}
+
+There are three types of numeric literals: plain integers, long
+integers, and floating point numbers.
+\index{number}
+\index{numeric literal}
+\index{integer literal}
+\index{plain integer literal}
+\index{long integer literal}
+\index{floating point literal}
+\index{hexadecimal literal}
+\index{octal literal}
+\index{decimal literal}
+
+Integer and long integer literals are described by the following
+lexical definitions:
+
+\begin{verbatim}
+longinteger: integer ("l"|"L")
+integer: decimalinteger | octinteger | hexinteger
+decimalinteger: nonzerodigit digit* | "0"
+octinteger: "0" octdigit+
+hexinteger: "0" ("x"|"X") hexdigit+
+
+nonzerodigit: "1"..."9"
+octdigit: "0"..."7"
+hexdigit: digit|"a"..."f"|"A"..."F"
+\end{verbatim}
+
+Although both lower case `l' and upper case `L' are allowed as suffix
+for long integers, it is strongly recommended to always use `L', since
+the letter `l' looks too much like the digit `1'.
+
+Plain integer decimal literals must be at most $2^{31} - 1$ (i.e., the
+largest positive integer, assuming 32-bit arithmetic). Plain octal and
+hexadecimal literals may be as large as $2^{32} - 1$, but values
+larger than $2^{31} - 1$ are converted to a negative value by
+subtracting $2^{32}$. There is no limit for long integer literals.
+
+Some examples of plain and long integer literals:
+
+\begin{verbatim}
+7 2147483647 0177 0x80000000
+3L 79228162514264337593543950336L 0377L 0x100000000L
+\end{verbatim}
+
+Floating point literals are described by the following lexical
+definitions:
+
+\begin{verbatim}
+floatnumber: pointfloat | exponentfloat
+pointfloat: [intpart] fraction | intpart "."
+exponentfloat: (intpart | pointfloat) exponent
+intpart: digit+
+fraction: "." digit+
+exponent: ("e"|"E") ["+"|"-"] digit+
+\end{verbatim}
+
+The allowed range of floating point literals is
+implementation-dependent.
+
+Some examples of floating point literals:
+
+\begin{verbatim}
+3.14 10. .001 1e100 3.14e-10
+\end{verbatim}
+
+Note that numeric literals do not include a sign; a phrase like
+\verb\-1\ is actually an expression composed of the operator
+\verb\-\ and the literal \verb\1\.
+
+\section{Operators}
+
+The following tokens are operators:
+\index{operators}
+
+\begin{verbatim}
++ - * / %
+<< >> & | ^ ~
+< == > <= <> != >=
+\end{verbatim}
+
+The comparison operators \verb\<>\ and \verb\!=\ are alternate
+spellings of the same operator.
+
+\section{Delimiters}
+
+The following tokens serve as delimiters or otherwise have a special
+meaning:
+\index{delimiters}
+
+\begin{verbatim}
+( ) [ ] { }
+; , : . ` =
+\end{verbatim}
+
+The following printing ASCII characters are not used in Python. Their
+occurrence outside string literals and comments is an unconditional
+error:
+\index{ASCII}
+
+\begin{verbatim}
+@ $ " ?
+\end{verbatim}
+
+They may be used by future versions of the language though!