IronPython Architecture

 

Contents

1.  Introduction

2.  Compiler

2.1.  Units of compilation

2.1.1.  Snippets of code

2.1.2.  Python modules

2.2.  Phases

2.2.1.  Parser

2.2.2.  AST trees

2.2.3.  Name binding

2.2.4.  Code generation

2.2.5.  Caching of generated assemblies

3.  Basic operations

3.1.  del <name>

4.  Functions

4.1.  Function declaration

4.2.  Function calls

4.3.  Nested functions

5.  Type system

5.1.  Python built-in types

5.2.  Python user types

5.2.1.  Class declaration

5.2.2.   Instance objects of Python user types

5.2.3.   Nested class declarations

5.3.  CLI types

6.  Built-in exceptions

 

1. Introduction

IronPython is the implementation of the Python language targeting the CLI. This document describes the basic details of the implementation, and the corresponding class names in the source code.

 

The engine is implemented in C# and is in IronPython.dll. It mainly consists of a compiler, and the runtime implementation of operations.

2. Compiler

The compiler (IronPython.Compiler.*) compiles Python source code into CLI assemblies using Reflection.Emit and dynamic methods. The compiler generates fully verifiable CIL code.

2.1.  Units of compilation

The unit of compilation in IronPython can be one of the following.  Compilation entry points are all in IronPython.Compiler.Generation.OutputGenerator.

2.1.1. Snippets of code

Snippets (IronPython.Hosting.CompiledCode) are small pieces of code. They are used for the following scenarios:

 

2.1.2.  Python modules

When a Python module is imported, it is compiled into a single assembly with a class (inheriting from IronPython.Compiler.CompiledModule) corresponding to the module's name. The class will have a method Initialize() contains the IL for all the top-level code of the module.

2.2.  Phases

The compiler consists of the following phases.

2.2.1.  Parser

The hand-written parser IronPython.Compiler.Parser generates abstract-syntax trees (ASTs).

2.2.2.  AST trees

The parser generates abstract-syntax trees (ASTs). The top level object when parsing a module is a ModuleDef. This contains statements (IronPython.Compiler.Ast.Statement) with expressions (IronPython.Compiler.Ast.Expression) underneath them. Names (IronPython.Runtime.SymbolId) are symbols used in the Python code.

2.2.3.  Name binding

Names (IronPython.Runtime.SymbolId) are unqualified symbols used in the source. For example, in the Python code:

 

def foo(a, b):

    result = a + b.attr + bar()

    return result

 

foo, a, b, bar, and result are unqualified symbols. These need to be bound to specific entities (IronPython.Compiler.Generation.Slot) using the Python lookup rules. This is done using namespaces (IronPython.Compiler.Generation.Namespace). Namespaces can be nested to get lexical scoping.

2.2.4.  Code generation

After all the names in the ASTs are bound, every statement recursively generates code for itself. The entry points here are in the AST classes themselves.

2.2.5. Caching of generated assemblies

Assemblies generated for Python script modules can be saved to disk next to the script as an EXE with the same name as the Python script. This is enabled using "ipy.exe -X:SaveAssemblies". The easiest way to understand how the generated code works is by disassembling the cached assemblies using ildasm.exe.

3.  Basic operations

Most basic operations are compiled into calls to functions in the IronPython.Runtime.Operations.Ops class. For example, the Python source code "a + b" results in IL to push a and b on the IL stack and call Ops.Add(object x, object y).

3.1.  del <name>

All names are to be implemented as dictionaries in Python. However, for performance and better integration with the CLI, IronPython implements many names as CLI entities, which cannot be deleted once they are created. For eg. module global variables are implemented as CLI static variables of the __main__ class in the assembly representing the Python module.

 

Hence, to implement "del", an alternate scheme is used. We just assign an instance of the type IronPython.Runtime.Uninitialized to represent that the Slot has been deleted. Any access to the name first checks if it is holding an Uninitialized, which means that it should virtually not exist.

4.  Functions

4.1.  Function declaration

Function declarations are statements in Python, and may be nested in other statement blocks just like any other statement. They are represented by IronPython.Compiler.Ast.FuncDef. When IronPython sees a function declaration, it generates a CLI function to represent the function body. The function declaration statement, FuncDef.Emit generates code to create a Function object that refers to the CLI function containing the code for the body.

4.2.  Function calls

Function calls are implemented by calling one of the Ops.Call overload methods, and passing in an instance of IronPython.Runtime.Calls.FastCallable as the first argument. Ops.Call then handles argument passing, and dispatces to the delegate contained in the Function object.

4.3.  Nested functions

Nested functions or closures are implemented by generating a CLI class which stores the environment of the enclosing method. The enclosing function initializes a local variable of the environment type, and creates a delegate that binds to the environment local variable, and the function body of the nested function.

5.  Type system

IronPython builds on top of the CLI type system to provide seamless integration with the CLI. Python "object" is represented by System.Object, and System.Object is the base of the type hierarchy.

5.1.  Python builtin types

These are implemented as CLI types in the engine. For eg, "list" is implemented by IronPython.Runtime.List. These types are handled the same as other CLI types, see below.

5.2.  Python user types

Every Python user type is represented by an instance of IronPython.Runtime.Types.UserType. The instance maintains information about the base types, the class attributes, etc.

5.2.1. Class declaration

Class declaration is a statement in Python, just like any other statement. IronPython converts it into a call to UserType.MakeClass passing in the list of base types. UserType.MakeClass creates an instance of UserType and initializes it with the information about the type. It also looks for an existing instance type that can be used for instances of the new type as described below, and creates a new one if necessary

5.2.2.  Instance objects of Python user types

Instance objects of Python user types are represented by instances of a generated CLI type (created by IronPython.Compiler.Generation.NewTypeMaker) whose __class__ member field points to the UserType. The same generated instance type can be used for instances of multiple Python types, with the __class__ member field used to distinguish between them.

 

Different types need to be generated only if the CLI-specific properties of the Python type are different. For eg, instances of all Python types inheriting from a given CLI type, say System.IO.Stream, will be represented by a generated instance type which inherits from System.IO.Stream. This is required for integration with the CLI so that the instances can be passed to CLI methods which expect System.IO.Stream as an argument. A single CLI types is used for all Python subtypes so that Python functionality like dynamically changing the type of an instance or changing the __bases__ of a type can be supported as well as possible in a way similar to Python's new-style classes.

5.2.3.   Nested class declarations

This is similar to Nested functions

5.3.  CLI types

IronPython code can seamlessly access CLI types. Information about CLI types (System.Type) is represented by IronPython.Runtime.Types.ReflectedType, and provides the functionality of the Python "type".

 

Note that since the builtin types are really just CLI types implemented by the engine, they too are represented by ReflectedType.

6.  Builtin exceptions

Builtin Python exception are implemented by throwing a corresponding CLI exception. The catch clauses in the Python code check for the CLI exception type, and then wrap it in an instance of IronPython.Runtime.Exceptions.PythonException. This allows Python code to catch exceptions thrown by CLI code.