Python/compile.txt

Developer Notes for Python Compiler
===================================

Table of Contents
-----------------

- Scope
    Defines the limits of the change
- Parse Trees
    Describes the local (Python) concept
- Abstract Syntax Trees (AST)
    Describes the AST technology used
- Parse Tree to AST
    Defines the transform approach
- Control Flow Graphs
    Defines the creation of "basic blocks"
- AST to CFG to Bytecode
    Tracks the flow from AST to bytecode
- Code Objects
    Pointer to making bytecode "executable"
- Modified Files
    Files added/modified/removed from CPython compiler
- ToDo
    Work yet remaining (before complete)
- References
    Academic and technical references to technology used.


Scope
-----

Historically (through 2.4), compilation from source code to bytecode
involved two steps:

1. Parse the source code into a parse tree (Parser/pgen.c)
2. Emit bytecode based on the parse tree (Python/compile.c)

Historically, this is not how a standard compiler works.  The usual
steps for compilation are:

1. Parse source code into a parse tree (Parser/pgen.c)
2. Transform parse tree into an Abstract Syntax Tree (Python/ast.c)
3. Transform AST into a Control Flow Graph (Python/newcompile.c)
4. Emit bytecode based on the Control Flow Graph (Python/newcompile.c)

Starting with Python 2.5, the above steps are now used.  This change
was done to simplify compilation by breaking it into three steps.
The purpose of this document is to outline how the lattter three steps
of the process works.

This document does not touch on how parsing works beyond what is needed
to explain what is needed for compilation.  It is also not exhaustive
in terms of the how the entire system works.  You will most likely need
to read some source to have an exact understanding of all details.


Parse Trees
-----------

Python's parser is an LL(1) parser mostly based off of the
implementation laid out in the Dragon Book [Aho86]_.

The grammar file for Python can be found in Grammar/Grammar with the
numeric value of grammar rules are stored in Include/graminit.h.  The
numeric values for types of tokens (literal tokens, such as ``:``,
numbers, etc.) are kept in Include/token.h).  The parse tree made up of
``node *`` structs (as defined in Include/node.h).

Querying data from the node structs can be done with the following
macros (which are all defined in Include/token.h):

- ``CHILD(node *, int)``
	Returns the nth child of the node using zero-offset indexing
- ``RCHILD(node *, int)``
	Returns the nth child of the node from the right side; use
	negative numbers!
- ``NCH(node *)``
	Number of children the node has
- ``STR(node *)``
	String representation of the node; e.g., will return ``:`` for a
	COLON token
- ``TYPE(node *)``
	The type of node as specified in ``Include/graminit.h``
- ``REQ(node *, TYPE)``
	Assert that the node is the type that is expected
- ``LINENO(node *)``
	retrieve the line number of the source code that led to the
	creation of the parse rule; defined in Python/ast.c

To tie all of this example, consider the rule for 'while'::

  while_stmt: 'while' test ':' suite ['else' ':' suite]

The node representing this will have ``TYPE(node) == while_stmt`` and
the number of children can be 4 or 7 depending on if there is an 'else'
statement.  To access what should be the first ':' and require it be an
actual ':' token, `(REQ(CHILD(node, 2), COLON)``.


Abstract Syntax Trees (AST)
---------------------------

The abstract syntax tree (AST) is a high-level representation of the
program structure without the necessity of containing the source code;
it can be thought of a abstract representation of the source code.  The
specification of the AST nodes is specified using the Zephyr Abstract
Syntax Definition Language (ASDL) [Wang97]_.

The definition of the AST nodes for Python is found in the file
Parser/Python.asdl .

Each AST node (representing statements, expressions, and several
specialized types, like list comprehensions and exception handlers) is
defined by the ASDL.  Most definitions in the AST correspond to a
particular source construct, such as an 'if' statement or an attribute
lookup.  The definition is independent of its realization in any
particular programming language.

The following fragment of the Python ASDL construct demonstrates the
approach and syntax::

  module Python
  {
	stmt = FunctionDef(identifier name, arguments args, stmt* body,
			    expr* decorators)
	      | Return(expr? value) | Yield(expr value)
	      attributes (int lineno)
  }

The preceding example describes three different kinds of statements;
function definitions, return statements, and yield statements.  All
three kinds are considered of type stmt as shown by '|' separating the
various kinds.  They all take arguments of various kinds and amounts.

Modifiers on the argument type specify the number of values needed; '?'
means it is optional, '*' means 0 or more, no modifier means only one
value for the argument and it is required.  FunctionDef, for instance,
takes an identifier for the name, 'arguments' for args, zero or more
stmt arguments for 'body', and zero or more expr arguments for
'decorators'.

Do notice that something like 'arguments', which is a node type, is
represented as a single AST node and not as a sequence of nodes as with
stmt as one might expect.  

All three kinds also have an 'attributes' argument; this is shown by the
fact that 'attributes' lacks a '|' before it.

The statement definitions above generate the following C structure type::

  typedef struct _stmt *stmt_ty;

  struct _stmt {
        enum { FunctionDef_kind=1, Return_kind=2, Yield_kind=3 } kind;
        union {
                struct {
                        identifier name;
                        arguments_ty args;
                        asdl_seq *body;
                } FunctionDef;
                
                struct {
                        expr_ty value;
                } Return;
                
                struct {
                        expr_ty value;
                } Yield;
        } v;
        int lineno;
   }

Also generated are a series of constructor functions that allocate (in
this case) a stmt_ty struct with the appropriate initialization.  The
'kind' field specifies which component of the union is initialized.  The
FunctionDef() constructor function sets 'kind' to FunctionDef_kind and
initializes the 'name', 'args', 'body', and 'attributes' fields.

*** NOTE: if you make a change here that can affect the output of bytecode that
is already in existence, make sure to delete your old .py(c|o) files!  Running
``find . -name '*.py[co]' -exec rm -f {} ';'`` should do the trick.


Parse Tree to AST
-----------------

The AST is generated from the parse tree in (see Python/ast.c) using the
function::

  mod_ty PyAST_FromNode(const node *n);

The function begins a tree walk of the parse tree, creating various AST
nodes as it goes along.  It does this by allocating all new nodes it
needs, calling the proper AST node creation functions for any required
supporting functions, and connecting them as needed.

Do realize that there is no automated nor symbolic connection between
the grammar specification and the nodes in the parse tree.  No help is
directly provided by the parse tree as in yacc.

For instance, one must keep track of
which node in the parse tree one is working with (e.g., if you are
working with an 'if' statement you need to watch out for the ':' token
to find the end of the conditional).  No help is directly provided by
the parse tree as in yacc.

The functions called to generate AST nodes from the parse tree all have
the name ast_for_xx where xx is what the grammar rule that the function
handles (alias_for_import_name is the exception to this).  These in turn
call the constructor functions as defined by the ASDL grammar and
contained in Python/Python-ast.c (which was generated by
Parser/asdl_c.py) to create the nodes of the AST.  This all leads to a
sequence of AST nodes stored in asdl_seq structs.


Function and macros for creating and using ``asdl_seq *`` types as found
in Python/asdl.c and Include/asdl.h:

- ``asdl_seq_new(int)``
	Allocate memory for an asdl_seq for length 'size'
- ``asdl_seq_free(asdl_seq *)``
	Free asdl_seq struct
- ``asdl_seq_GET(asdl_seq *seq, int pos)``
	Get item held at 'pos'
- ``asdl_seq_SET(asdl_seq *seq, int pos, void *val)``
	Set 'pos' in 'seq' to 'val'
- ``asdl_seq_APPEND(asdl_seq *seq, void *val)``
	Set the end of 'seq' to 'val'
- ``asdl_seq_LEN(asdl_seq *)``
	Return the length of 'seq'

If you are working with statements, you must also worry about keeping
track of what line number generated the statement.  Currently the line
number is passed as the last parameter to each stmt_ty function.


Control Flow Graphs
-------------------

A control flow graph (often referenced by its acronym, CFG) is a
directed graph that models the flow of a program using basic blocks that
contain the intermediate representation (abbreviated "IR", and in this
case is Python bytecode) within the blocks.  Basic blocks themselves are
a block of IR that has a single entry point but possibly multiple exit
points.  The single entry point is the key to basic blocks; it all has
to do with jumps.  An entry point is the target of something that
changes control flow (such as a function call or a jump) while exit
points are instructions that would change the flow of the program (such
as jumps and 'return' statements).  What this means is that a basic
block is a chunk of code that starts at the entry point and runs to an
exit point or the end of the block.
  
As an example, consider an 'if' statement with an 'else' block.  The
guard on the 'if' is a basic block which is pointed to by the basic
block containing the code leading to the 'if' statement.  The 'if'
statement block contains jumps (which are exit points) to the true body
of the 'if' and the 'else' body (which may be NULL), each of which are
their own basic blocks.  Both of those blocks in turn point to the
basic block representing the code following the entire 'if' statement.

CFGs are usually one step away from final code output.  Code is directly
generated from the basic blocks (with jump targets adjusted based on the
output order) by doing a post-order depth-first search on the CFG
following the edges.


AST to CFG to Bytecode
----------------------

With the AST created, the next step is to create the CFG. The first step
is to convert the AST to Python bytecode without having jump targets
resolved to specific offsets (this is calculated when the CFG goes to
final bytecode). Essentially, this transforms the AST into Python
bytecode with control flow represented by the edges of the CFG.

Conversion is done in two passes.  The first creates the namespace
(variables can be classified as local, free/cell for closures, or
global).  With that done, the second pass essentially flattens the CFG
into a list and calculates jump offsets for final output of bytecode.

The conversion process is initiated by a call to the function in
Python/newcompile.c::

  PyCodeObject * PyAST_Compile(mod_ty, const char *, PyCompilerFlags);

This function does both the conversion of the AST to a CFG and
outputting final bytecode from the CFG.  The AST to CFG step is handled
mostly by the two functions called by PyAST_Compile()::

  struct symtable * PySymtable_Build(mod_ty, const char *,
					PyFutureFeatures);
  PyCodeObject * compiler_mod(struct compiler *, mod_ty);

The former is in Python/symtable.c while the latter is in
Python/newcompile.c .

PySymtable_Build() begins by entering the starting code block for the
AST (passed-in) and then calling the proper symtable_visit_xx function
(with xx being the AST node type).  Next, the AST tree is walked with
the various code blocks that delineate the reach of a local variable
as blocks are entered and exited::

  static int symtable_enter_block(struct symtable *, identifier,
				    block_ty, void *, int);
  static int symtable_exit_block(struct symtable *, void *);

Once the symbol table is created, it is time for CFG creation, whose
code is in Python/newcompile.c .  This is handled by several functions
that break the task down by various AST node types.  The functions are
all named compiler_visit_xx where xx is the name of the node type (such
as stmt, expr, etc.).  Each function receives a ``struct compiler *``
and xx_ty where xx is the AST node type.  Typically these functions
consist of a large 'switch' statement, branching based on the kind of
node type passed to it.  Simple things are handled inline in the
'switch' statement with more complex transformations farmed out to other
functions named compiler_xx with xx being a descriptive name of what is
being handled.

When transforming an arbitrary AST node, use the VISIT macro::

  VISIT(struct compiler *, <node type>, <AST node>);

The appropriate compiler_visit_xx function is called, based on the value
passed in for <node type> (so ``VISIT(c, expr, node)`` calls
``compiler_visit_expr(c, node)``).  The VISIT_SEQ macro is very similar,
 but is called on AST node sequences (those values that were created as
arguments to a node that used the '*' modifier).  There is also
VISIT_SLICE just for handling slices::

  VISIT_SLICE(struct compiler *, slice_ty, expr_context_ty);

Emission of bytecode is handled by the following macros:

- ``ADDOP(struct compiler *c, int op)``
    add 'op' as an opcode
- ``ADDOP_I(struct compiler *c, int op, int oparg)``
    add 'op' with an 'oparg' argument
- ``ADDOP_O(struct compiler *c, int op, PyObject *type, PyObject *obj)``
    add 'op' with the proper argument based on the position of obj in
    'type', but with no handling of mangled names; used for when you
    need to do named lookups of objects such as globals, consts, or
    parameters where name mangling is not possible and the scope of the
    name is known
- ``ADDOP_NAME(struct compiler *, int, PyObject *, PyObject *)``
    just like ADDOP_O, but name mangling is also handled; used for
    attribute loading or importing based on name
- ``ADDOP_JABS(struct compiling *c, int op, basicblock b)``
    create an absolute jump to the basic block 'b'
- ``ADDOP_JREL(struct compiling *c, int op, basicblock b)``
    create a relative jump to the basic block 'b'

Several helper functions that will emit bytecode and are named
compiler_xx() where xx is what the function helps with (list, boolop
 etc.).  A rather useful one is::

  static int compiler_nameop(struct compiler *, identifier,
				expr_context_ty);

This function looks up the scope of a variable and, based on the
expression context, emits the proper opcode to load, store, or delete
the variable.

As for handling the line number on which a statement is defined, is
handled by compiler_visit_stmt() and thus is not a worry.

In addition to emitting bytecode based on the AST node, handling the
creation of basic blocks must be done.  Below are the macros and
functions used for managing basic blocks:

- ``NEW_BLOCK(struct compiler *)``
    create block and set it as current
- ``NEXT_BLOCK(struct compiler *)``
    basically NEW_BLOCK() plus jump from current block
- ``compiler_new_block(struct compiler *)``
    create a block but don't use it (used for generating jumps)

Once the CFG is created, it must be flattened and then final emission of
bytecode occurs.  Flattening is handled using a post-order depth-first
search.  Once flattened, jump offsets are backpatched based on the
flattening and then a PyCodeObject file is created.  All of this is
handled by calling::

  PyCodeObject * assemble(struct compiler *, int);

*** NOTE: if you make a change here that can affect the output of bytecode that
is already in existence, make sure to delete your old .py(c|o) files!  Running
``find . -name '*.py[co]' -exec rm -f {} ';'`` should do the trick.


Code Objects
------------

In the end, one ends up with a PyCodeObject which is defined in
Include/code.h .  And with that you now have executable Python bytecode!


Modified Files
--------------

+ Parser/

    - Python.asdl
        ASDL syntax file

    - asdl.py
        "An implementation of the Zephyr Abstract Syntax Definition
        Language."  Uses SPARK_ to parse the ASDL files.

    - asdl_c.py
        "Generate C code from an ASDL description."  Generates
	../Python/Python-ast.c and ../Include/Python-ast.h .

    - spark.py
        SPARK_ parser generator

+ Python/

    - Python-ast.c
        Creates C structs corresponding to the ASDL types.  Also
	contains code for marshaling AST nodes (core ASDL types have
	marshaling code in asdl.c).  "File automatically generated by
	../Parser/asdl_c.py".

    - asdl.c
        Contains code to handle the ASDL sequence type.  Also has code
        to handle marshalling the core ASDL types, such as number and
        identifier.  used by Python-ast.c for marshaling AST nodes.

    - ast.c
        Converts Python's parse tree into the abstract syntax tree.

    - compile.txt
        This file.

    - newcompile.c
        New version of compile.c that handles the emitting of bytecode.

    - symtable.c
	Generates symbol table from AST.
	

+ Include/

    - Python-ast.h
        Contains the actual definitions of the C structs as generated by
        ../Python/Python-ast.c .
        "Automatically generated by ../Parser/asdl_c.py".

    - asdl.h
        Header for the corresponding ../Python/ast.c .

    - ast.h
        Declares PyAST_FromNode() external (from ../Python/ast.c).

    - code.h
	Header file for ../Objects/codeobject.c; contains definition of
	PyCodeObject.

    - symtable.h
	Header for ../Python/symtable.c .  struct symtable and
	PySTEntryObject are defined here.

+ Objects/

    - codeobject.c
	Contains PyCodeObject-related code (originally in
	../Python/compile.c).


ToDo
----
*** NOTE: all bugs and patches should be filed on SF under the group
	    "AST" for easy searching.  It also does not hurt to put
	    "[AST]" at the beginning of the subject line of the tracker
	    item.

+ Stdlib support
    - AST->Python access?
    - rewrite compiler package to mirror AST structure?
+ Documentation
    - flesh out this doc
	* byte stream output
	* explanation of how the symbol table pass works
	* code object (PyCodeObject)
+ Universal
    - make sure entire test suite passes
    - fix memory leaks
    - make sure return types are properly checked for errors
    - no gcc warnings

References
----------

.. [Aho86] Alfred V. Aho, Ravi Sethi, Jeffrey D. Ullman.
   `Compilers: Principles, Techniques, and Tools`,
   http://www.amazon.com/exec/obidos/tg/detail/-/0201100886/104-0162389-6419108

.. [Wang97]  Daniel C. Wang, Andrew W. Appel, Jeff L. Korn, and Chris
   S. Serra.  `The Zephyr Abstract Syntax Description Language.`_
   In Proceedings of the Conference on Domain-Specific Languages, pp.
   213--227, 1997.

.. _The Zephyr Abstract Syntax Description Language.:
   http://www.cs.princeton.edu/~danwang/Papers/dsl97/dsl97.html

.. _SPARK: http://pages.cpsc.ucalgary.ca/~aycock/spark/