Tutorials and examples

Table of contents

Introduction

This page lists some snippets of Nemerle code you may want to look at. If you have any example to be submitted here -- please edit this page or contact us.


Short tutorials for writing example applications

The tutorials gathered here are ment to be used as easy introduction to the the language, by stepping through writing process of small applications.

There is also Grokking Nemerle -- a "book" covering most of the language.

Snippets

You can find various examples of Nemerle code in snippets (http://nemerle.org/svn/nemerle/trunk/snippets/) directory in Nemerle source tree. You are welcomed to transform them into tutorials. Examples include:

These are also bundled with the source distribution.

Larger examples

There are also several more advanced examples there:

Sokoban solver

Sokoban (http://nemerle.org/svn/nemerle/trunk/snippets/sokoban/) -- Sokoban solver by Bartosz Podlejski

Nemerle Course programs

Examples done by students during the Nemerle course at the CS Institute in the Wroclaw University.

ICFP'2004 Programming Contest entry

In June 2004 there was a contest (http://www.cis.upenn.edu/proj/plclub/contest/ants.html) to create finite state automata controlling the ants. Our entry is now available here (http://nemerle.org/svn/nemerle/trunk/snippets/ants-icfp2004/).

The Great Language Shootout Examples

These (http://nemerle.org/svn/nemerle/trunk/snippets/shootout/) are the few examples ported from the Win32 Computer Language Shootout (http://dada.perl.it/shootout/). We didn't submit them yet, looking for volunteers to port more. When we have enough examples, we will also send them to the recently linked renewed shootout (http://shootout.alioth.debian.org/).

ntrace

This simple program (http://nemerle.org/svn/nemerle/trunk/snippets/ntrace/) will trace memory leaks in programs written in C. By Jacek Śliwerski.

Sioux HTTP server

A simplistic HTTP server (http://nemerle.org/svn/nemerle/trunk/snippets/sioux/). The main idea behind it is to serve web applications written in Nemerle. It already served as a subscription server for the XVIII-th FIT (a local conference) and later for the CSL 2004 (http://csl04.ii.uni.wroc.pl/).

External projects created using Nemerle

RiDL - Compiler tools

RiDL (http://sourceforge.net/projects/ridl/) is a set of tools to simplify building compilers using Nemerle. It includes a lexical analyzer generator and a parser generator.

It is created by Kojo Adams (http://sourceforge.net/users/samanpa/) and uses BSD Licence (http://sourceforge.net/softwaremap/trove_list.php?form_cat=187).

Speagram

Speagram (http://www.speagram.org/index.html) is an eager purely functional language with a strong type system and unusual syntax resembling natural language.

First Tutorial

Table of contents

Introduction

This tutorial will describe basics of programming in Nemerle. We assume the reader is familiar with C#, Java or C++.

Simple examples

This section lists simple example that look almost the same as in C# (or Java, or C++).

The very-first example

 class Hello
 {
   static Main () : void
   {
     System.Console.WriteLine ("Hello world!");
   }
 }

As you can see the only difference between Nemerle and C# version of this program is that we write method's return type at the right, after colon. There are some reasons to do it this way, we will explain them later.

To run this example:

For people unfamiliar with C#: the entry point of execution is the static Main method in some class. System.Console.WriteLine is a call to a function from the .NET framework.

The adder

We will now write a very simple program to read and add two numbers.

 /*  Our second example.  
    This is a comment. */ 
 
 using System;
 
 //  This is also a comment
 
 public class Adder      //  As in C sharp we can mark class public.
 {
   public static Main () : void         //  And method too.
   {
     /*  Read two lines, convert them to integers and return their
       sum. */ 
     Console.WriteLine ("The sum is {0}",
                        //  System.Int32.Parse converts string into integer.
                        Int32.Parse (Console.ReadLine ()) +
                        Int32.Parse (Console.ReadLine ()));
   }
 }

The using declaration imports identifiers from specified namespace, so they can be used without prefix. Unlike in C# it can also import members from classes, not only from namespaces. For example:

 using System;
 using System.Console;
 
 public class Adder
 {
   public static Main () : void
   {
     WriteLine ("The sum is {0}",
                Int32.Parse (ReadLine ()) +
                Int32.Parse (ReadLine ()));
   }
 }

As you can see for both lines we first read them and then convert to integers. We can factor this into a method:

 using System;
 
 public class Adder
 {
   //  It is private by default.
   static ReadInteger () : int
   {
     Int32.Parse (Console.ReadLine ())
   }
 
   public static Main () : void
   {
     def x = ReadInteger (); //  Variable definition.
     def y = ReadInteger ();
     //  Use standard .NET function for formatting output.
     Console.WriteLine ("{0} + {1} = {2}", x, y, x + y);
   }
 }

The method definition looks the way we would expect from Main example. However in Main itself we define two values: x and y. This is done using the def keyword. As you can see we do not write type of variable at the place of definition. The compiler sees that ReadInteger returns an int and therefore type of x has to also be int. This is called type inference.

In this example we see no gain from using def instead of int as you would do in C# (both are 3 characters long :-). However in most cases type names are far longer:

FooBarQuxxFactory fact = new FooBarQuxxFactory (); // C#
def fact = FooBarQuxxFactory (); // Nemerle

As you can see we also do not use the new keyword.

Counting lines in file

 class LineCounter
 {
   public static Main () : void
   {
     //  Open a file.
     def sr = System.IO.StreamReader ("SomeFile.txt");   //  (1)
     mutable line_no = 0;                               //  (2)
     mutable line = sr.ReadLine ();
     while (line != null) {              //  (3)
       System.Console.WriteLine (line);
       line_no = line_no + 1;            //  (4)
       line = sr.ReadLine ();
     };                                  //  (5)
     System.Console.WriteLine ("Line count: {0}", line_no);
   }
 }

There are few remarkable things about this example. First one is the very important difference between lines marked (1) and (2).

In (1) we define an immutable value sr. Immutable means it cannot be changed once it is defined. This may first seem odd, but it is quite often that variables are never changed, once created. The def statement is there to mark this intent.

In (2) we define a mutable value. This is the same thing as variable in C#, It has to be initialized before use and can be changed later. The assignment operator in Nemerle is called =, it works much like in C#. We see its use in (4).

In (3) we see the while loop. It works much like in C#. We also have do ... while loops in Nemerle.

Functional examples

This section will introduce some more functional features of Nemerle.

Functional programming (FP) is style in which you do not modify state of the machine with instructions, but rather evaluate functions yielding new and new values. That is entire program is just one big expression. In purely functional language (Haskell being main example) you cannot modify any objects once they are created (there is no assignment operator, like = in Nemerle). There are no loops, just recursive functions.


Nemerle does not force you to use FP. However you can use it whenever you find it necessary. Some algorithms have very natural representation when written in functional style -- for example functional languages are very good at manipulation of tree-like data structures (like XML, in fact XSLT can be thought of as a functional language).

However at the beginning we will use functional style to write simple programs, that could be written in imperative style without much problems.

We will be using terms method and function interchangeably.

Rewriting line counter without the loop

We will now rewrite our previous example not to use loops, just a recursive function. It will get longer, but we will fix that soon.

 class LineCounterWithoutLoop
 {
   public static Main () : void
   {
     def sr = System.IO.StreamReader ("SomeFile.txt");
     mutable line_no = 0;
 
     def read_lines () : void {            //  (1)
       def line = sr.ReadLine ();
       when (line != null) {               //  (2)
         System.Console.WriteLine (line);  //  (3)
         line_no = line_no + 1;            //  (4)
         read_lines ()                     //  (5)
       }
     };
 
     read_lines ();      //  (6)
 
     System.Console.WriteLine ("Line count: {0}", line_no); //  (7)
   }
 }

In (1) we define a nested method called read_lines inside the Main method. The method is there to simulate the while loop from our previous example. It takes no parameters and returns void value.

(2) If line wasn't null (i.e. it was not the last line), (3) we write the line we just read, (4) increase the line number, and finally (5) call ourself to read rest of the lines. The when expression is explained below.

Next (6) we call read_lines for the first time, and finally (7) print the line count.

The read_lines will get called as many times as there are lines in the file. As you can see this is the same as the while loop, just expressed in a slightly different way. It is very important to grok this concept of writing loops as recursion, in order to program functionally in Nemerle.

If you are concerned about performance of this form of writing loops -- fear you not. When function body ends with call to another function -- no new stack frame is created. It is called a tail call. Thanks to it the example above is as efficient as the while loop we seen before.

In Nemerle the if expression always need to have the else clause. It's done this way to avoid stupid bugs with dangling-else:

// C#, misleading indentation hides real code meaning
if (foo)
   if (bar)
     m1 ();
else
   m2 ();

If you do not want the else clause, use when expression, as seen in the example. There is also unless expression, equivalent to when with the condition negated.

Rewriting line counter without mutable values

Our previous aim of rewriting line counter removed the loop and one mutable value. However one mutable value has left, so we cannot say the example is written functionally. We will now kill it.

 class FunctionalLineCounter
 {
   public static Main () : void
   {
     def sr = System.IO.StreamReader ("SomeFile.txt");
     def read_lines (line_no : int) : int {   //  (1)
       def line = sr.ReadLine ();
       if (line == null)      //  (2)
         line_no              //  (3)
       else {
         System.Console.WriteLine (line);  //  (4)
         read_lines (line_no + 1)          //  (5)
       }
     };
 
     System.Console.WriteLine ("Line count: {0}", read_lines (0)); //  (6)
   }
 }

In (1) we again define nested method called read_lines. However this time it takes one integer parameter -- the current line number. It returns number of lines in entire file.

(2) If line we just read is null (that was last line), we (3) return the current line number as number of lines in entire file. As you can see there is no return statement. The return value of method is its last expression.

(4) Otherwise (it was not last line) we write the line we just read. Next (5) we call ourself to read the next line. We need to increase line number, since it is next line what we will be reading. Note that as a return value from this invocation of read_lines we return what the next invocation of read_lines returned. It in turn returns what the next invocation returned and so on, until, at the end of file, we reach (3), and final line count is returned through each invocation of read_lines.

In (6) we call the read_lines nested method, with initial line number of 0 to read the file and print out line count.

Type inference

We have already seen type inference used to guess types of values defined with def or mutable. It can be also used to guess type of function parameters and return type. Try removing the : int constraints from line marked (1) in our previous example.

Type inference only works for nested functions. Type annotations are required in top-level methods (that is methods defined in classes, not in other methods). This is design decision, that is here not to change external interfaces by accident.

It is sometimes quote hard to tell the type of parameter, from just looking how it is used. For example consider:

 class HarderInference
 {
   static Main () : int {
     def f (x) {
       x.Length
     };
     f ("foo");
   }
 }

When compiling the f method we cannot tell if x is a string or array or something else. Nevertheless, we can tell it later (looking at f invocation) and Nemerle type inference engine does it.

If function with incomplete type information was not used or its type was ambiguous, compiler would refuse to compile it.

More info

Now, once you read through all this, please move to the Grokking Nemerle tutorial, that is much more complete. You can also have a look at The Reference Manual if you are tough.

Second Tutorial

We will write some example code, so you can see some common code patterns when writing programs in Nemerle.

Small app with array and Hashtable

Let us create an array and print its elements walking through it in loop.

using System.Console;
 
module MyApp
{
   Main () : void
   {
     // create an array containing strings (array [string])
     def elements = array ["{0}", "a {0} a", "{0}", "b {0}", "b {0}"];
 
     // iterate through elements of array
     for (mutable i = 0; i < elements.Length; i++)
       // print current value of `i' with using format from array
       WriteLine (elements [i], i);
   }
}

The keys stored in elements array looks a little bit strange, but we will use WriteLine method as a good example of processing key together with value associated to it.

Counting unique elements

We will now make the collection of keys unique. For this purpose we will use generic dictionary type, which in Nemerle is called Nemerle.Collections.Hashtable.

using Nemerle.Collections;
using System.Console;
 
module MyApp
{
   Main () : void
   {
     // create an array containing strings (array [string])
     def elements = array ["{0}", "a {0} a", "{0}", "b {0}", "b {0}"];
 
     def hash = Hashtable ();
 
     // count the number of occurrences of each unique element
     foreach (e in elements)
     {
       unless (hash.Contains (e))
         hash[e] = 0;
         
       hash[e] += 1;
     }
 
     // print each element with number indicating its total count
     foreach (keypair in hash) {
       WriteLine (keypair.Key, keypair.Value);
     }
   }
}

For each entry in hashtable we store the number of how many times it occured.

What about functional programming?

Ok, iterating over collection with foreach is nice, but it is still *too long*!

// print each element with number indicating its total count
foreach (keypair in hash)
  WriteLine (keypair.Key, keypair.Value);
 
// pass matching overload of WriteLine to iteration method
hash.Iter (WriteLine)

Hashtable have the special Iter overload, which takes a generic two-argument function, which will be called for every element stored in collection. The declaration of Iter looks like this:

class Hashtable ['a, 'b] 
{
   public Iter (f : 'a * 'b -> void) : void 
   { ... }
...
}

and after substituting 'a and 'b with actual types in our hash, we just use it as Iter : (string * int -> void) -> void, which takes string * int -> void function as the only argument.

And one more way of iterating over a Hashtable is to query KeyValuePairs property and use pairs of key and value in foreach statement:

foreach( (key, value) in hash.KeyValuePairs)
  WriteLine(key, value);

Gtk text editor (tutorial)

Table of contents

Simple text editor in GTK#

In this tutorial we will see step by step how could you build a simple text editor in Nemerle using Gtk Sharp.

Part I creating a window

First we must create an application and display a window, which will be the base of our editor.

 
using System;
using Gtk;
 
class MainWindow : Window
{
	public this()
	{
                // set caption of window
		base ("Very Simple Editor");
                // resize windows to some reasonable shape
		SetSizeRequest (300,200);
	}
}
 
module SimpleEditor
{ 
	Main() : void
	{
		Application.Init();
		def win = MainWindow();
 
                // exit application when editor window is deleted
                win.DeleteEvent += fun (_) { Application.Quit () };
 
                win.ShowAll ();
		Application.Run();
	}
}

SimpleEditor is a module with Main method - an entry point of our application. It is executed first when program is run and creates editor window. MainWindow is quite simple now, but we will add things to it in a moment.

One more thing to note is win.DeleteEvent, which is a signal triggered when window is destroyed (for example user clicks close button). We specify what should happen then, by attaching an anonymous function to the event. Here, we will simply quit application.

Part II adding text input section

Now we need to add the area to the window, where actual text can we written by user.

We modify MainWindow class to hold the TextView object.

 
class MainWindow : Window
{
        /// text input area of our window
	input : TextView;
 
	public this()
	{
                // set caption of window
		base ("Very Simple Editor");
                // resize windows to some reasonable shape
		SetSizeRequest (300,200);
 
		def scroll = ScrolledWindow ();		
		input = TextView ();
		scroll.Add (input);
		
                // place scrolledwin inside our main window
           	Add (scroll);
	}
}

In constructor of our window we must choose how the text input will be positioned. We use Gtk container to encapsulate TextView into ScrolledWindow. This will allow long texts entered into text view to be easily viewable.

Add (scroll) adds the ScrolledWindow as part of our text editor window. So, till now we have created working text input.


Part III a simple menu

But we still want to perform some more advanced operations in our editor. Ha, we will need a menu. Insted of adding scroll to the main window, we will create a menu and put it into vertical box together with text view.

 
  def menu = MenuBar ();
  def mi = NMenuItem ("File");
  menu.Append(mi);		
		
  def vbox = VBox ();
  vbox.PackStart (menu, false, false, 0u);
  vbox.PackStart (scroll, true, true, 0u);
 
  // place vertical box inside our main window
  Add (vbox);

In last instruction we are adding entire vertical box into main window.

The NMenuItem is a class, which we define below, so we can extend it and add some actions to it. Here is the initial code we will work with.

 
class NMenuItem : MenuItem
{
	public name : string;
	
	public this(l : string) 
	{
		base(l);
		name = l;		
	}
}

For now it only set the caption of base MenuItem.


Part IV adding submenus

Now we want our only menu item to have some submenus for the tasks available in the editor.

We add a property to NMenuItem, which allows easy adding of child menus:

 
// class NMenuItem : MenuItem 
// { 
 
// this property allows us to set submenus of this menu item
public SubmenuList : list [NMenuItem]
{ 
  set
  { 
    def sub = Menu();
    foreach (submenu in value) sub.Append (x);
    this.Submenu = sub;
  }
}
 
// }

Well, it simply iterate through supplied list and add each of its elements to the current instance's submenu.

We use this property in constructor of MainWindow:

 
// def menu = MenuBar ();
// def mi = NMenuItem ("File");
mi.SubmenuList = 
[
  NMenuItem ("Open"),
  NMenuItem ("Save as...")
];
// menu.Append(mi);		


Part V adding tasks to the submenus

Finally, we will attach some action to the items of our editor's menu. We will allow opening and saving files from the editor.

First, we add second constructor to NMenuItem, which will accept action function to be perfomed after choosing given menu item.

 
public this(l : string, e : object * EventArgs -> void) 
{
  base(l);
  name = l;		
  this.Activated += e;
}

This constructor does the same things as previous one, but also attach given function to Activated event of current menu item.

Now we define a function, which will handle opening and saving files by our editor. We add it to MainWindow:

 
// handler of opening and saving files
OnMenuFile (i : object, _ : EventArgs) : void
{
  def mi = i :> NMenuItem;
  def fs = FileSelection (mi.name);
		
  when (fs.Run () == ResponseType.Ok :> int) match (mi.name)
  {
    | "Open" =>
       def stream = IO.StreamReader (fs.Filename);
       input.Buffer.Text = stream.ReadToEnd();
 
    | "Save as..." =>
       def s = IO.StreamWriter(fs.Filename);
       s.Write(input.Buffer.Text);
       s.Close();
 
    | _ => ();
  };
  fs.Hide();
}

This method creates a window for selecting files (Gtk FileSelection) and checks from which menu item it was called. Here we match on name of NMenuItem, but you could also use something like enums to distinguish various menu items (or attach specific handler to each of them).

Actions performed for opening and saving files are quite simple - we use Buffer.Text properties of TextView to get/set contents of editor input.

Last thing to do is to actually create menu items using handlers. We change previous code to:

 
def menu = MenuBar ();
def mi = NMenuItem ("File");
mi.SubmenuList = 
[
   NMenuItem ("Open", OnMenuFile),
   NMenuItem ("Save as...", OnMenuFile)
];
menu.Append(mi);

Result

The result of these several steps can be accessed here (http://nemerle.org/svn/nemerle/trunk/snippets/gtk/vsedit/vsedit.n) and it looks like this:

 
using System;
using Gtk;
 
class NMenuItem : MenuItem
{
	public name : string;
	
	public this(l : string) 
	{
		base(l);
		name = l;		
	}
 
	public this(l : string, e : object * EventArgs -> void) 
	{
		base(l);
		name = l;		
		this.Activated += e;
	}
 
 
        // this property allows us to set submenus of this menu item
	public SubmenuList : list [NMenuItem]
	{ 
		set
		{ 
			def sub = Menu();
                        foreach (submenu in value) sub.Append (submenu);
			this.Submenu = sub;
		}
	}
}
 
 
class MainWindow : Window
{
        /// text input area of our window
	input : TextView;
 
	public this()
	{
                // set caption of window
		base ("Very Simple Editor");
                // resize windows to some reasonable shape
		SetSizeRequest (300,200);
 
		def scroll = ScrolledWindow ();		
		input = TextView ();
		scroll.Add (input);
 
		def menu = MenuBar ();
		def mi = NMenuItem ("File");
		mi.SubmenuList = 
		[
			NMenuItem ("Open", OnMenuFile),
			NMenuItem ("Save as...", OnMenuFile)
		];
		menu.Append(mi);		
		
		def vbox = VBox ();
		vbox.PackStart (menu, false, false, 0u);
	        vbox.PackStart (scroll, true, true, 0u);
 
		// place vertical box inside our main window
		Add (vbox);
	}
 
        // handler of opening and saving files
	OnMenuFile (i : object, _ : EventArgs) : void
	{
		def mi = i :> NMenuItem;
		def fs = FileSelection (mi.name);
		
		when (fs.Run () == ResponseType.Ok :> int) match (mi.name)
		{
			| "Open" =>
                                def stream = IO.StreamReader (fs.Filename);
				input.Buffer.Text = stream.ReadToEnd();
 
			| "Save as..." =>
				def s = IO.StreamWriter(fs.Filename);
				s.Write(input.Buffer.Text);
				s.Close();
			| _ => ();
		};
		fs.Hide();
	}
}
 
module SimpleEditor
{ 
	Main() : void
	{
		Application.Init();
		def win = MainWindow();
 
                // exit application when editor window is deleted
                win.DeleteEvent += fun (_) { Application.Quit () };
 
                win.ShowAll ();
		Application.Run();
	}
}

Macros tutorial

Nemerle type-safe macros

Table of contents

What exactly is a macro?

Basically every macro is a function, which takes a fragment of code as parameter(s) and returns some other code. On the highest level of abstraction it doesn't matter if parameters are function calls, type definitions or just a sequence of assignments. The most important fact is that they are not common objects (e.g. instances of some types, like integer numbers), but their internal representation in the compiler (i.e. syntax trees).

A macro is defined in the program just like any other function, using common Nemerle syntax. The only difference is the structure of the data it operates on and the way how it is used (executed at compile-time).

A macro, once created, can be used to process some parts of the code. It's done by calling it with block(s) of code as parameter(s). This operation is in most cases indistinguishable from a common function call (like f(1)), so a programmer using a macro would not be confused by unknown syntax. The main concept of our design is to make the usage of macros as transparent as possible. From the user point of view, it is not important if particular parameters are passed to a function, which would process them at the compile-time and insert some new code in their place, or to an ordinary one.

Defining a new macro

Writing the macro is as simple as writing the common function. It looks the same except it is preceded by a keyword macro and it lives at toplevel (not inside any class). This will make the compiler know about how to use defined method (i.e. run it at the compile-time in every place where it is used).

Macros can take zero (if we just want to generate new code) or more parameters. They are all elements of the language grammar, so their type is limited to the set of defined syntax objects. The same holds for a return value of a macro.

Example:

macro generate_expression ()
{
  MyModule.compute_some_expression ();
}

This example macro does not take any parameters and is used in the code by simply writing generate_expression ();. The most important is the difference between generate_expression and compute_some_expression - the first one is a function executed by the compiler during compilation, while the latter is just some common function that must return syntax tree of expressions (which is here returned and inserted into program code by generate_expression).

Compiling a simplest macro

In order to create and use a macro you have to write a library, which will contain its executable form. You simply create a new file mymacro.n, which can contain for example

macro m () {
  Nemerle.IO.printf ("compile-time\n");
  <[ Nemerle.IO.printf ("run-time\n") ]>;
}

and compile it with command

ncc -r Nemerle.Compiler.dll -t:dll mymacro.n -o mymacro.dll

Now you can use m() in any program, like here

module M {
  public Main () : void {
    m ();
  }
}

You must add a reference to mymacro.dll during compilation of this program. It might look like

ncc -r mymacro.dll myprog.n -o myprog.exe

Exercise

Write a macro, which, when used, should slow down the compilation by 5 seconds (use System.Timers namespace) and print the version of the operating system used to compile program (use System.Environment namespace).

Operating on syntax trees

Definition of function compute_some_expression might look like:

using Nemerle.Compiler.Parsetree;
 
module MyModule 
{
  public mutable debug_on : bool;
 
  public compute_some_expression () : PExpr 
  {
    if (debug_on) 
      <[ System.Console.WriteLine ("Hello, I'm debug message") ]>
    else
      <[ () ]>
  }
}

The examples above show a macro, which conditionally inlines expression printing a message. It's not quite useful yet, but it has introduced the meaning of compile-time computations and also some new syntax used only in writing macros and functions operating on syntax trees. We have written here the <[ ... ]> constructor to build a syntax tree of expression (e.g. '()').

Quotation operator

<[ ... ]> is used to both construction and decomposition of syntax trees. Those operations are similar to quotation of code. Simply, everything which is written inside <[ ... ]>, corresponds to its own syntax tree. It can be any valid Nemerle code, so a programmer does not have to learn internal representation of syntax trees in the compiler.

macro print_date (at_compile_time)
{                   
  match (at_compile_time) {
    | <[ true ]> => MyModule.print_compilation_time ()
    | _ => <[ WriteLine (DateTime.Now.ToString ()) ]>
  }
}

The quotation alone allows using only constant expressions, which is insufficient for most tasks. For example, to write function print_compilation_time we must be able to create an expression based on a value known at the compile-time. In next sections we introduce the rest of macros' syntax to operate on general syntax trees.

Matching subexpressions

When we want to decompose some large code (or more precisely, its syntax tree), we must bind its smaller parts to variables. Then we can process them recursively or just use them in an arbitrary way to construct the result.

We can operate on entire subexpressions by writing $( ... ) or $ID inside the quotation operator <[ ... ]>. This means binding the value of ID or the interior of parenthesized expression to the part of syntax tree described by corresponding quotation.

macro for (init, cond, change, body)
{
  <[ 
    $init;
    def loop () : void {
      if ($cond) { $body; $change; loop() } 
      else ()
    };
    loop ()
  ]>
}

The above macro defines function for, which is similar to the loop known from C. It can be used like this

for (mutable i = 0, i < 10, i++, printf ("%d", i))

Later we show how to extend the language syntax to make the syntax of for exactly as in C.

Base elements of grammar

Sometimes quoted expressions have literals inside of them (like strings, integers, etc.) and we want to operate on their value, not on their syntax trees. It is possible, because they are constant expressions and their runtime value is known at the compile-time.

Let's consider the previously used function print_compilation_time.

using System;
using Nemerle.Compiler.Parsetree;
 
module MyModule {
  public print_compilation_time () : PExpr
  {                   
    <[ System.Console.WriteLine ($(DateTime.Now.ToString () : string)) ]>
  }
}

Here we see some new extension of splicing syntax where we create a syntax tree of string literal from a known value. It is done by adding : string inside the $(...) construct. One can think about it as of enforcing the type of spliced expression to a literal (similar to common Nemerle type enforcement), but in the matter of fact something more is happening here - a real value is lifted to its representation as syntax tree of a literal.

Other types of literals (int, bool, float, char) are treated the same. This notation can be used also in pattern matching. We can match constant values in expressions this way.

There is also a similar schema for splicing and matching variables of a given name. $(v : name) denotes a variable, whose name is contained by object v (of special type Name). There are some good reasons for encapsulating a real identifier within this object.

Constructs with variable amount of elements

You might have noticed, that Nemerle has a few grammar elements, which are composed of a list of subexpressions. For example, a sequence of expressions enclosed with { .. } braces may contain zero or more elements.

When splicing values of some expressions, we would like to decompose or compose such constructs in a general way - i.e. obtain all expressions in a given sequence. It is natural to think about them as if a list of expressions and to bind this list to some variable in meta-language. It is done with special syntax ..:

mutable exps = [ <[ printf ("%d ", x) ]>, <[ printf ("%d ", y) ]> ];
exps = <[ def x = 1 ]> :: <[ def y = 2 ]> :: exps;
<[ {.. $exps } ]>

We have used { .. $exps } here to create the sequence of expressions from list exps : list<Expr>;. A similar syntax is used to splice the content of tuples (( .. $elist )) and other constructs, like array []:

using Nemerle.Collections;
 
macro castedarray (e) {
 match (e) {
  | <[ array [.. $elements ] ]> =>
     def casted = List.Map (elements, fun (x) { <[ ($x :> object) ]> });
     <[ array [.. $casted] ]>
  | _ => e
 }
}

If the exact number of expressions in tuple/sequence is known during writing the quotation, then it can be expressed with

<[ $e_1; $e_2; $e_3; x = 2; f () ]>

The .. syntax is used when there are e_i : Expr for 1 <= i <= n.

Exercise

Write a macro rotate, which takes two parameters: a pair of floating point numbers (describing a point in 2D space) and an angle (in radians). The macro should return a new pair -- a point rotated by the given angle. The macro should use as much information as is available at the compile-time, e.g. if all numbers supplied are constant, then only the final result should be inlined, otherwise the result must be computed at runtime.

Adding new syntax to the compiler

After we have written the for macro, we would like the compiler to understand some changes to its syntax. Especially the C-like notation

for (mutable i = 0; i < n; --i) {
  sum += i;
  Nemerle.IO.printf ("%d\n", sum);
}

In order to achieve that, we have to define which tokens and grammar elements may form a call of for macro. We do that by changing its header to

macro for (init, cond, change, body)
syntax ("for", "(", init, ";", cond, ";", change, ")", body)

The syntax keyword is used here to define a list of elements forming the syntax of the macro call. The first token must always be an unique identifier (from now on it is treated as a special keyword triggering parsing of defined sequence). It is followed by tokens composed of operators or identifiers passed as string literals or names of parameters of macro. Each parameter must occur exactly once.

Parsing of syntax rule is straightforward - tokens from input program must match those from definition, parameters are parsed according to their type. Default type of a parameter is Expr, which is just an ordinary expression (consult Nemerle grammar in <link>reference.html</link>). All allowed parameter types will be described in the extended version of reference manual corresponding to macros.

Exercise

Add a new syntactic construct forpermutation to your program. It should be defined as the macro

macro forp (i, n : int, m : int, body)

and introduce syntax, which allows writing the following program

mutable i = 0;
forpermutation (i in 3 to 10) Nemerle.IO.printf ("%d\n", i)

It should create a random permutation p of numbers x_j, m <= x_j <= n at the compile-time. Then generate the code executing body of the loop n - m + 1 times, preceding each of them with assignment of permutation element to i.

Macros in custom attributes

Executing macros on type declarations

Nemerle macros are simply plugins to the compiler. We decided not to restrict them only to operations on expressions, but allow them to transform almost any part of program.

Macros can be used within custom attributes written near methods, type declarations, method parameters, fields, etc. They are executed with those entities passed as their parameters.


As an example, let us take a look at Serializable macro. Its usage looks like this:

[Serializable]
class S {
  public this (v : int, m : S) { a = v; my = m; }
  my : S;
  a : int;
}

From now on, S has additional method Serialize and it implements interface ISerializable. We can use it in our code like this

def s = S (4, S (5, null));
s.Serialize ();

And the output is

<a>4</a>
<my>
  <a>5</a>
  <my>
    <null/>
  </my>
</my>

The macro modifies type S at compile-time and adds some code to it. Also inheritance relation of given class is changed, by making it implement interface ISerializable

public interface ISerializable {
  Serialize () : void;
}

Manipulating type declarations

In general, macros placed in attributes can do many transformations and analysis of program objects passed to them. To see Serializable macro's internals and discuss some design issues, let's go into its code.

[Nemerle.MacroUsage (Nemerle.MacroPhase.BeforeInheritance, Nemerle.MacroTargets.Class,
                     Inherited = true)]
macro Serializable (t : TypeBuilder)
{
  t.AddImplementedInterface (<[ ISerializable ]>)
}

First we have to add interface, which given type is about to implement. But more important thing is the phase modifier BeforeInheritance in macro's custom attribute. In general, we separate three stages of execution for attribute macros. BeforeInheritance specifies that the macro will be able to change subtyping information of the class it operates on.

So, we have added interface to our type, we now have to create Serialize () method.

[Nemerle.MacroUsage (Nemerle.MacroPhase.WithTypedMembers, Nemerle.MacroTargets.Class,
                     Inherited = true)]
macro Serializable (t : TypeBuilder)
{
  /// here we list its fields and choose only those, which are not derived
  /// or static
  def fields = t.GetFields (BindingFlags.Instance | BindingFlags.Public %|
                            BindingFlags.NonPublic | BindingFlags.DeclaredOnly);
 
  /// now create list of expressions which will print object's data  
  mutable serializers = [];
 
  /// traverse through fields, taking their type constructors  
  foreach (x : IField in fields) {
    def tc = x.GetMemType ().TypeInfo;
    def nm = Macros.UseSiteSymbol (x.Name);
    if (tc != null)
      if (tc.IsValueType)
        /// we can safely print value types as strings        
        serializers = <[
                         printf ("<%s>", $(x.Name : string));
                         System.Console.Write ($(nm : name));
                         printf ("</%s>\n", $(x.Name : string));
                       ]>
                       :: serializers
      else
        /// we can try to check, if type of given field also implements ISerializable
        if (x.GetMemType ().Require (<[ ttype: ISerializable ]>))
          serializers = <[
                           printf ("<%s>\n", $(x.Name : string));      
                           if ($(nm : name) != null)
                             $(nm : name).Serialize ()
                           else
                             printf ("<null/>\n");
                           printf ("</%s>\n", $(x.Name : string));
                         ]>
                         :: serializers
        else
          /// and finally, we encounter case when there is no easy way to serialize 
          /// given field
          Message.FatalError ("field `" + x.Name + "' cannot be serialized")
    else
      Message.FatalError ("field `" + x.Name + "' cannot be serialized")
  };
  // after analyzing fields, we create method in our type, to execute created
  // expressions
  t.Define (<[ decl: public Serialize () : void
                     implements ISerializable.Serialize {
                       .. $serializers
                     }
            ]>);
}


Execution stages

Analysing object-oriented hierarchy and class members is a separate pass of the compilation. First it creates inheritance relation between classes, so we know exactly all base types of given type. After that every member inside of them (methods, fields, etc.) is being analysed and added to the hierarchy and its type annotations are resolved. After that also the rules regarding implemented interface methods are checked.

For the needs of macros we have decided to distinguish three moments in this pass at which they can operate on elements of class hierarchy. Every macro can be annotated with a stage, at which it should be executed.

Parameters of attribute macros

Every executed attribute macro operates on some element of class hierarchy, so it must be supplied with an additional parameter describing the object, on which macro was placed. This way it can easily query for properties of that element and use compiler's API to reflect or change the context in which it was defined.

For example a method macro declaration would be

[Nemerle.MacroUsage (Nemerle.MacroPhase.WithTypedMembers,
                     Nemerle.MacroTargets.Method)]
macro MethodMacro (t : TypeBuilder, f : MethodBuilder, expr)
{
  // use 't' and 'f' to query or change class-level elements
  // of program
}

Macro is annotated with additional attributes specifying respectively the stage in which macro will be executed and the macro target.

The available parameters contain references to class hierarchy elements that given macro operates on. They are automatically supplied by compiler and they vary on the target and stage of given macro. Here is a little table specifying valid parameters for each stage and target of attribute macro.


Attribute macro targets and parameters
MacroTarget MacroPhase.BeforeInheritance MacroPhase.BeforeTypedMembers MacroPhase.WithTypedMembers
Class TypeBuilder TypeBuilder TypeBuilder
Method TypeBuilder, ParsedMethod TypeBuilder, ParsedMethod TypeBuilder, MethodBuilder
Field TypeBuilder, ParsedField TypeBuilder, ParsedField TypeBuilder, FieldBuilder
Property TypeBuilder, ParsedProperty TypeBuilder, ParsedProperty TypeBuilder, PropertyBuilder
Event TypeBuilder, ParsedEvent TypeBuilder, ParsedEvent TypeBuilder, EventBuilder
Parameter TypeBuilder, ParsedMethod, ParsedParameter TypeBuilder, ParsedMethod, ParsedParameter TypeBuilder, MethodBuilder, ParameterBuilder
Assembly (not implemented) (none) (none) (none)


The intuition is that every macro has parameter holding its target and additionally objects containing it (like TypeBuilder is available in most of the attribute macros).

After those implicitly available parameters there come standard parameters explicitly supplied by user. They are the same as for expression level macros.

Reference to more advanced aspects

Hygiene and alpha-renaming of identifiers

Problem with names capture

Identifiers in quoted code (object code) must be treated in a special way, because we usually do not know in which scope they would appear. Especially they should not mix with variables with the same names from the macro-use site.


Consider the following macro defining a local function f

macro identity (e) { <[ def f (x) { x }; f($e) ]> }

Calling it with identity (f(1)) might generate confusing code like

def f (x) { x }; f (f (1))

To preserve names capture, all macro generated variables should be renamed to their unique counterparts, like in

def f_42 (x_43) { x_43 }; f_42 (f (1))

Hygiene of macros

The idea of separating variables introduced by a macro from those defined in the plain code (or other macros) is called `hygiene' after Lisp and Scheme languages. In Nemerle we define it as putting identifiers created during a single macro execution into a unique namespace. Variables from different namespaces cannot bind to each other.


In other words, a macro cannot create identifiers capturing any external variables or visible outside of its own generated code. This means, that there is no need to care about locally used names.


The Hygiene is obtained by encapsulating identifiers in special Name class. The compiler uses it to distinguish names from different macro executions and scopes (for details of implementation consult paper about macros (http://nemerle.org/metaprogramming.pdf)). Variables with appropriate information are created automatically by quotation.

def definition = <[ def y = 4 ]>;
<[ def x = 5; $definition; x + y ]>

When a macro creates the above code, identifiers y and x are tagged with the same unique mark. Now they cannot be captured by any external variables (with a different mark). We operate on the Name class, when the quoted code is composed or decomposed and we use <[ $(x : name) ]> construct. Here x is bound to am object of type Name, which we can use in other place to create exactly the same identifier.

An identifier can be also created by calling method Macros.NewSymbol(), which returns Name with an unique identifier, tagged with a current mark.

def x = Macros.NewSymbol ();
<[ def $(x : name) = 5; $(x : name) + 4 ]>

Controlled breaking hygiene

Sometimes it is useful to generate identifiers, which bind to variables visible in place where a macro is used. For example one of macro's parameters is a string with some identifiers inside. If we want to use these as real identifiers, then we need to break automatic hygiene. It is especially useful in embedding domain-specific languages, which reference symbols from the original program.

As an example consider a Nemerle.IO.sprint (string literal) macro (which have the syntax shortcut $"some text $id "). It searches given string literal for $var and creates a code concatenating text before and after $var to the value of var.ToString ().

def x = 3;
System.Console.WriteLine ($"My value of x is $x and I'm happy");

expands to

def x = 3;
System.Console.WriteLine ({ 
  def sb = System.Text.StringBuilder ("My value of x is "); 
  sb.Append (x.ToString ()); 
  sb.Append (" and I'm happy"); 
  sb.ToString () 
});

Breaking of hygiene is necessary here, because we generate code (reference to x), which need to have the same context as variables from invocation place of macro.

To make given name bind to the symbols from macro usesite, we use Nemerle.Macros.UseSiteSymbol (name : string) : Name function, or special splicing target usesite in quotations. Their use would be like in this simplified implementation of macro

macro sprint (lit : string) 
{
  def (prefix, symbol, suffix) = Helper.ExtractDollars (lit);
  def varname = Nemerle.Macros.UseSiteSymbol (symbol);
  <[ 
    def sb = System.Text.StringBuilder ($(prefix : string)); 
    sb.Append ($(varname : name).ToString ()); 
    // or alternatively  $(symbol : usesite)
    sb.Append ($(suffix : string)); 
    sb.ToString () 
  ]>
}

Note that this operations is 'safe', that is it changes context of variable to the place where macro invocation was created (see paper for more details).

Unhygienic variables

Sometimes it is useful to completely break hygiene, where programmer only want to experiment with new ideas. From our experience, it is often hard to reason about correct contexts for variables, especially when writing class level macros. In this case it is useful to be able to easily break hygine.

Nemerle provides it with <[ $("id" : dyn) ]> construct. It makes produced variable break hygiene rules and always bind to the nearest definition with the same name.

ASP.NET (tutorial)

Preparations

In order to use Nemerle in your ASP.NET (http://asp.net) pages you first need to add following line

<compiler language="n;nemerle;Nemerle" extension=".n" warningLevel="1"  compilerOptions=""                        
   type="Nemerle.Compiler.NemerleCodeProvider, Nemerle.Compiler, Version=0.2.10.0, Culture=neutral, PublicKeyToken=5291d186334f6101" 
  />

in <configuration><system.web><compilation>... to your machine.config file (tested on mono, where this file can be found in /etc/mono/1.0/)

Example page

Now you can create test.aspx with following content:

<%@ language=Nemerle %>
 
  <script runat="server">
    Page_Load(_ : object, _ : EventArgs) : void {
      Message.Text = $"You last accessed this page at: $(DateTime.Now)";
    }
 
    EnterBtn_Click(_ : object, _ : EventArgs) : void 
    {
      Message.Text = $"Hi $(Name.Text), welcome to ASP.NET!";
    }
  </script>
 
<html>
      <form runat=server>
 
          <font face="Verdana"> 
 
             Please enter your name: <asp:textbox id="Name" runat=server/> 
                                     <asp:button text="Enter" Onclick="EnterBtn_Click" runat=server/>
 
             <p>
               <asp:label id="Message"  runat=server/>
             </p>
 
          </font>
 
       </form>
</html>

And everything should work fine. This scenario was tested with xsp and mod_mono on mono 1.1.6, please let us know how it works on MS.NET + ISS.

Remoting (tutorial)

This is a simple example that shows how to realize remoting in Nemerle. Essentially this is translation of the example presented in the MonoHandbook (http://monohandbook.monoforge.com/monkeyguide/remoting/introduction.html).


Remoting allow clients to access objects in the server. And thus it is a good abstraction to distribute computing.

In this example both Server and Client share the definition of ServerObjects. The server program opens a port for conection and publish a ServerObject. Later the client program connects himself to the server port and access, remotely, the published objects and their methods.

The example shows the basic remoting usage, managing objects by reference and by value (through serialization).

Table of contents

Client

Save the following code in the file "Client.n".

using System;
using System.Net;
using System.Runtime.Remoting;
using System.Runtime.Remoting.Channels;
using System.Runtime.Remoting.Channels.Tcp;
 
namespace RemotingTest
{
    class RemotingClient
    {
        static Main () :void
        {
            def ch = TcpChannel (0);
            ChannelServices.RegisterChannel (ch);
            
            // This gets the object from the server (a list of ServerObject)
 
            Console.WriteLine ("Getting instance ...");
            def remOb = Activator.GetObject (typeof (ServerList),
                                             "tcp://localhost:1122/test.rem");
                                             
            def list = remOb :>ServerList;
            
            // These are remote calls that return references to remote objects
 
            Console.WriteLine ("Creating two remote items...");
            def item1 = list.NewItem ("Created at server 1") :ServerObject;
            item1.SetValue (111); // another call made to the remote object
            def item2 = list.NewItem ("Created at server 2") :ServerObject;
            item2.SetValue (222);
            Console.WriteLine ();
 
 
            // Two objects created in this client app
            
            Console.WriteLine ("Creating two client items...");
            def item3 = ServerObject ("Created at client 1");
            item3.SetValue (333);
            def item4 = ServerObject ("Created at client 2");
            item4.SetValue (444);
            Console.WriteLine ();
 
            // Object references passed to the remote list
            
            Console.WriteLine ("Adding items...");
            list.Add (item3);
            list.Add (item4);
            Console.WriteLine ();
 
 
            // This sums all values of the ServerObjects in the list. The server
	    // makes a remote call to this client to get the value of the
	    // objects created locally
            Console.WriteLine ("Processing items...");
            list.ProcessItems ();
            Console.WriteLine ();
            
            // Passing some complex info as parameter and return value
            
            Console.WriteLine ("Setting complex data...");
            def cd = ComplexData (AnEnum.D, array ["some" :object, 22, "info"]);
            def res = list.SetComplexData (cd) :ComplexData;
            Console.WriteLine ("This is the result:");
            res.Dump ();
            Console.WriteLine ();
 
            list.Clear ();
            Console.WriteLine ("Done.");
 
            ch.StopListening (null);
 
        }
    }
}

Server

Save the following code in the file "Server.n".

using System;
using System.Runtime.Remoting;
using System.Runtime.Remoting.Channels;
using System.Runtime.Remoting.Channels.Tcp;
 
namespace RemotingTest
{
    public class RemotingServer
    {
        static Main () :int
        {
            Console.WriteLine ("Starting Server");
            def ch = TcpChannel (1122);
            ChannelServices.RegisterChannel (ch);
            
            def ser = ServerList ();
            _ = RemotingServices.Marshal (ser, "test.rem");
            
            Console.WriteLine ("Server Running ...");
            _ = Console.ReadLine ();
            
            ch.StopListening (null);
            
            0;
        }
    }
}

ServerObjects

Save the following code in the file "ServerObjects.n".

using System;
using System.Runtime.Remoting;
using System.Collections;
 
namespace RemotingTest
{
    // A list of ServerObject instances
    public class ServerList :MarshalByRefObject
    {
        values : ArrayList;
        
        public this ()
        {
            values = ArrayList ();
        }
        
        public Add (v :ServerObject) :void
        {
            _ = values.Add (v);
            System.Console.WriteLine ("Added " + v.Name);
            
        }
 
        public ProcessItems () :void
        {
            System.Console.WriteLine ("Processing...");
            
            mutable total = 0;
            foreach (ob in values) total += (ob :> ServerObject).GetValue ();
           
            System.Console.WriteLine ("Total: " + total.ToString());
        }
 
        public Clear () :void
        {
            values.Clear ();
        }
 
        public  NewItem (name :string) :ServerObject
        {
            def obj = ServerObject (name);
            Add (obj);
            obj;
        }
 
        public SetComplexData (data :ComplexData) :ComplexData 
        {
            System.Console.WriteLine ("Showing content of ComplexData");
            data.Dump ();
            data;
        }
    }
    
 
    // A remotable object
    public class ServerObject :MarshalByRefObject
    {
        mutable _value :int;
        _name :string;
 
        public this (name :string)
        {
            _name = name;
        }
        
        public Name :string 
        {
            get { _name; }
        }
 
        public SetValue (v :int) :void
        {
            System.Console.WriteLine ("ServerObject " + _name + ": setting " + v.ToString() );
            _value = v;
        }
        
        public GetValue () :int
        {
            System.Console.WriteLine ("ServerObject " + _name + ": getting " + _value.ToString());
            _value;
        }
    }
    
 
 
    public enum AnEnum { |A |B |C |D |E };
 
    [Serializable] 
    public class ComplexData
    {
        public Val :AnEnum;
        public Info : array [object];
 
        public this ( va:AnEnum, info : array [object]) 
        {
            Info = info;
            Val = va;
        }
        
        public  Dump () :void
        {
            System.Console.WriteLine ("Content:");
            System.Console.WriteLine ("Val: " + Val.ToString());
            foreach (ob in Info) System.Console.WriteLine ("Array item: " + ob.ToString());
        }
    }
}

Compilation

Save the following code in the file "Makefile".

all:
	ncc -t:library ServerObjects.n -o ServerObjects.dll
	ncc -r:ServerObjects.dll -r:System.Runtime.Remoting.dll Client.n -o Client.exe
	ncc -r:ServerObjects.dll -r:System.Runtime.Remoting.dll Server.n -o Server.exe

Run "make" to compile the programs. Then execute the "Server.exe" in a console and "Client.exe" in another one.

Serializing Variants

By default custom attribues on variants are not passed to its variant options.

Thus using

[Serializable] 
public variant AnEnum { |A |B |C |D |E };

instead of the proposed enum (which is directly serializable) will create an System.Runtime.Serialization.SerializationException error - you must explicitly declare each element of the variant as Serializable.

[Serializable] 
public variant AnEnum { 
	[Serializable] | A 
	[Serializable] | B 
	[Serializable] | C { x : string; }
	[Serializable] | D { x : int; }
	[Serializable] | E 
};

Using a macro to mark all options with attribute

It is possible to use a macro Nemerle.MarkOptions to realize this in a compact way. For a little insight of how it works, there is its a full source, which fortunately is short.

using Nemerle.Compiler;
using Nemerle.Compiler.Parsetree;
 
[Nemerle.MacroUsage (Nemerle.MacroPhase.BeforeInheritance,
                     Nemerle.MacroTargets.Class)]
macro MarkOptions (t : TypeBuilder, attribute)
{
  // iterate through members of this type and select only variant options
  foreach (ClassMember.TypeDeclaration
            (TopDeclaration.VariantOption as vo) in t.GetParsedMembers ())
  {
    // add given custom attribute to this variant option
    vo.AddCustomAttribute (attribute)
  }
}

Now you can use

[Serializable] 
[Nemerle.MarkOptions (Serializable)]
public variant AnEnum { 
	| A 
	| B 
	| C { x : string; }
	| D { y : int; }
	| E 
}

Side note Remember that if you use variant instead of enum, the expression

AnEnum.B

has to be changed to

AnEnum.B ()
This is for keeping consistency with
AnEnum.C ("Ala")

PInvoke (tutorial)

You can take advantage of native platform libraries from within Nemerle programs. The syntax is very similar to C#'s and other .NET languages. Here is the simplest examlple:

using System;
using System.Runtime.InteropServices;
 
class PlatformInvokeTest
{
    [DllImport("msvcrt.dll")]
    public extern static puts(c : string) : int;
 
    [DllImport("msvcrt.dll")]
    internal extern static _flushall() : int;
    
    public static Main() : void
    {
        _ = puts("Test");
        _ = _flushall();
    }
}

As you can see we use DllImport attribute, which comes from System.Runtime.InteropServices namespace. Every method marked with this attribute should also have extern modifier. The concept is that the implementation of given method is substituted by call to unmanaged method from library specified inside DllImport attribute.

So, in above example, during execution of Main method first the puts function from msvcrt.dll will be called (printing a text to the standard output) and after that _flushall (making sure that all contents of buffer are printed).

For more see P/Invoke tutorial (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/csref/html/vcwlkplatforminvoketutorial.asp) from MSDN.

Macros tutorial

Nemerle type-safe macros

Table of contents

What exactly is a macro?

Basically every macro is a function, which takes a fragment of code as parameter(s) and returns some other code. On the highest level of abstraction it doesn't matter if parameters are function calls, type definitions or just a sequence of assignments. The most important fact is that they are not common objects (e.g. instances of some types, like integer numbers), but their internal representation in the compiler (i.e. syntax trees).

A macro is defined in the program just like any other function, using common Nemerle syntax. The only difference is the structure of the data it operates on and the way how it is used (executed at compile-time).

A macro, once created, can be used to process some parts of the code. It's done by calling it with block(s) of code as parameter(s). This operation is in most cases indistinguishable from a common function call (like f(1)), so a programmer using a macro would not be confused by unknown syntax. The main concept of our design is to make the usage of macros as transparent as possible. From the user point of view, it is not important if particular parameters are passed to a function, which would process them at the compile-time and insert some new code in their place, or to an ordinary one.

Defining a new macro

Writing the macro is as simple as writing the common function. It looks the same except it is preceded by a keyword macro and it lives at toplevel (not inside any class). This will make the compiler know about how to use defined method (i.e. run it at the compile-time in every place where it is used).

Macros can take zero (if we just want to generate new code) or more parameters. They are all elements of the language grammar, so their type is limited to the set of defined syntax objects. The same holds for a return value of a macro.

Example:

macro generate_expression ()
{
  MyModule.compute_some_expression ();
}

This example macro does not take any parameters and is used in the code by simply writing generate_expression ();. The most important is the difference between generate_expression and compute_some_expression - the first one is a function executed by the compiler during compilation, while the latter is just some common function that must return syntax tree of expressions (which is here returned and inserted into program code by generate_expression).

Compiling a simplest macro

In order to create and use a macro you have to write a library, which will contain its executable form. You simply create a new file mymacro.n, which can contain for example

macro m () {
  Nemerle.IO.printf ("compile-time\n");
  <[ Nemerle.IO.printf ("run-time\n") ]>;
}

and compile it with command

ncc -r Nemerle.Compiler.dll -t:dll mymacro.n -o mymacro.dll

Now you can use m() in any program, like here

module M {
  public Main () : void {
    m ();
  }
}

You must add a reference to mymacro.dll during compilation of this program. It might look like

ncc -r mymacro.dll myprog.n -o myprog.exe

Exercise

Write a macro, which, when used, should slow down the compilation by 5 seconds (use System.Timers namespace) and print the version of the operating system used to compile program (use System.Environment namespace).

Operating on syntax trees

Definition of function compute_some_expression might look like:

using Nemerle.Compiler.Parsetree;
 
module MyModule 
{
  public mutable debug_on : bool;
 
  public compute_some_expression () : PExpr 
  {
    if (debug_on) 
      <[ System.Console.WriteLine ("Hello, I'm debug message") ]>
    else
      <[ () ]>
  }
}

The examples above show a macro, which conditionally inlines expression printing a message. It's not quite useful yet, but it has introduced the meaning of compile-time computations and also some new syntax used only in writing macros and functions operating on syntax trees. We have written here the <[ ... ]> constructor to build a syntax tree of expression (e.g. '()').

Quotation operator

<[ ... ]> is used to both construction and decomposition of syntax trees. Those operations are similar to quotation of code. Simply, everything which is written inside <[ ... ]>, corresponds to its own syntax tree. It can be any valid Nemerle code, so a programmer does not have to learn internal representation of syntax trees in the compiler.

macro print_date (at_compile_time)
{                   
  match (at_compile_time) {
    | <[ true ]> => MyModule.print_compilation_time ()
    | _ => <[ WriteLine (DateTime.Now.ToString ()) ]>
  }
}

The quotation alone allows using only constant expressions, which is insufficient for most tasks. For example, to write function print_compilation_time we must be able to create an expression based on a value known at the compile-time. In next sections we introduce the rest of macros' syntax to operate on general syntax trees.

Matching subexpressions

When we want to decompose some large code (or more precisely, its syntax tree), we must bind its smaller parts to variables. Then we can process them recursively or just use them in an arbitrary way to construct the result.

We can operate on entire subexpressions by writing $( ... ) or $ID inside the quotation operator <[ ... ]>. This means binding the value of ID or the interior of parenthesized expression to the part of syntax tree described by corresponding quotation.

macro for (init, cond, change, body)
{
  <[ 
    $init;
    def loop () : void {
      if ($cond) { $body; $change; loop() } 
      else ()
    };
    loop ()
  ]>
}

The above macro defines function for, which is similar to the loop known from C. It can be used like this

for (mutable i = 0, i < 10, i++, printf ("%d", i))

Later we show how to extend the language syntax to make the syntax of for exactly as in C.

Base elements of grammar

Sometimes quoted expressions have literals inside of them (like strings, integers, etc.) and we want to operate on their value, not on their syntax trees. It is possible, because they are constant expressions and their runtime value is known at the compile-time.

Let's consider the previously used function print_compilation_time.

using System;
using Nemerle.Compiler.Parsetree;
 
module MyModule {
  public print_compilation_time () : PExpr
  {                   
    <[ System.Console.WriteLine ($(DateTime.Now.ToString () : string)) ]>
  }
}

Here we see some new extension of splicing syntax where we create a syntax tree of string literal from a known value. It is done by adding : string inside the $(...) construct. One can think about it as of enforcing the type of spliced expression to a literal (similar to common Nemerle type enforcement), but in the matter of fact something more is happening here - a real value is lifted to its representation as syntax tree of a literal.

Other types of literals (int, bool, float, char) are treated the same. This notation can be used also in pattern matching. We can match constant values in expressions this way.

There is also a similar schema for splicing and matching variables of a given name. $(v : name) denotes a variable, whose name is contained by object v (of special type Name). There are some good reasons for encapsulating a real identifier within this object.

Constructs with variable amount of elements

You might have noticed, that Nemerle has a few grammar elements, which are composed of a list of subexpressions. For example, a sequence of expressions enclosed with { .. } braces may contain zero or more elements.

When splicing values of some expressions, we would like to decompose or compose such constructs in a general way - i.e. obtain all expressions in a given sequence. It is natural to think about them as if a list of expressions and to bind this list to some variable in meta-language. It is done with special syntax ..:

mutable exps = [ <[ printf ("%d ", x) ]>, <[ printf ("%d ", y) ]> ];
exps = <[ def x = 1 ]> :: <[ def y = 2 ]> :: exps;
<[ {.. $exps } ]>

We have used { .. $exps } here to create the sequence of expressions from list exps : list<Expr>;. A similar syntax is used to splice the content of tuples (( .. $elist )) and other constructs, like array []:

using Nemerle.Collections;
 
macro castedarray (e) {
 match (e) {
  | <[ array [.. $elements ] ]> =>
     def casted = List.Map (elements, fun (x) { <[ ($x :> object) ]> });
     <[ array [.. $casted] ]>
  | _ => e
 }
}

If the exact number of expressions in tuple/sequence is known during writing the quotation, then it can be expressed with

<[ $e_1; $e_2; $e_3; x = 2; f () ]>

The .. syntax is used when there are e_i : Expr for 1 <= i <= n.

Exercise

Write a macro rotate, which takes two parameters: a pair of floating point numbers (describing a point in 2D space) and an angle (in radians). The macro should return a new pair -- a point rotated by the given angle. The macro should use as much information as is available at the compile-time, e.g. if all numbers supplied are constant, then only the final result should be inlined, otherwise the result must be computed at runtime.

Adding new syntax to the compiler

After we have written the for macro, we would like the compiler to understand some changes to its syntax. Especially the C-like notation

for (mutable i = 0; i < n; --i) {
  sum += i;
  Nemerle.IO.printf ("%d\n", sum);
}

In order to achieve that, we have to define which tokens and grammar elements may form a call of for macro. We do that by changing its header to

macro for (init, cond, change, body)
syntax ("for", "(", init, ";", cond, ";", change, ")", body)

The syntax keyword is used here to define a list of elements forming the syntax of the macro call. The first token must always be an unique identifier (from now on it is treated as a special keyword triggering parsing of defined sequence). It is followed by tokens composed of operators or identifiers passed as string literals or names of parameters of macro. Each parameter must occur exactly once.

Parsing of syntax rule is straightforward - tokens from input program must match those from definition, parameters are parsed according to their type. Default type of a parameter is Expr, which is just an ordinary expression (consult Nemerle grammar in <link>reference.html</link>). All allowed parameter types will be described in the extended version of reference manual corresponding to macros.

Exercise

Add a new syntactic construct forpermutation to your program. It should be defined as the macro

macro forp (i, n : int, m : int, body)

and introduce syntax, which allows writing the following program

mutable i = 0;
forpermutation (i in 3 to 10) Nemerle.IO.printf ("%d\n", i)

It should create a random permutation p of numbers x_j, m <= x_j <= n at the compile-time. Then generate the code executing body of the loop n - m + 1 times, preceding each of them with assignment of permutation element to i.

Macros in custom attributes

Executing macros on type declarations

Nemerle macros are simply plugins to the compiler. We decided not to restrict them only to operations on expressions, but allow them to transform almost any part of program.

Macros can be used within custom attributes written near methods, type declarations, method parameters, fields, etc. They are executed with those entities passed as their parameters.


As an example, let us take a look at Serializable macro. Its usage looks like this:

[Serializable]
class S {
  public this (v : int, m : S) { a = v; my = m; }
  my : S;
  a : int;
}

From now on, S has additional method Serialize and it implements interface ISerializable. We can use it in our code like this

def s = S (4, S (5, null));
s.Serialize ();

And the output is

<a>4</a>
<my>
  <a>5</a>
  <my>
    <null/>
  </my>
</my>

The macro modifies type S at compile-time and adds some code to it. Also inheritance relation of given class is changed, by making it implement interface ISerializable

public interface ISerializable {
  Serialize () : void;
}

Manipulating type declarations

In general, macros placed in attributes can do many transformations and analysis of program objects passed to them. To see Serializable macro's internals and discuss some design issues, let's go into its code.

[Nemerle.MacroUsage (Nemerle.MacroPhase.BeforeInheritance, Nemerle.MacroTargets.Class,
                     Inherited = true)]
macro Serializable (t : TypeBuilder)
{
  t.AddImplementedInterface (<[ ISerializable ]>)
}

First we have to add interface, which given type is about to implement. But more important thing is the phase modifier BeforeInheritance in macro's custom attribute. In general, we separate three stages of execution for attribute macros. BeforeInheritance specifies that the macro will be able to change subtyping information of the class it operates on.

So, we have added interface to our type, we now have to create Serialize () method.

[Nemerle.MacroUsage (Nemerle.MacroPhase.WithTypedMembers, Nemerle.MacroTargets.Class,
                     Inherited = true)]
macro Serializable (t : TypeBuilder)
{
  /// here we list its fields and choose only those, which are not derived
  /// or static
  def fields = t.GetFields (BindingFlags.Instance | BindingFlags.Public %|
                            BindingFlags.NonPublic | BindingFlags.DeclaredOnly);
 
  /// now create list of expressions which will print object's data  
  mutable serializers = [];
 
  /// traverse through fields, taking their type constructors  
  foreach (x : IField in fields) {
    def tc = x.GetMemType ().TypeInfo;
    def nm = Macros.UseSiteSymbol (x.Name);
    if (tc != null)
      if (tc.IsValueType)
        /// we can safely print value types as strings        
        serializers = <[
                         printf ("<%s>", $(x.Name : string));
                         System.Console.Write ($(nm : name));
                         printf ("</%s>\n", $(x.Name : string));
                       ]>
                       :: serializers
      else
        /// we can try to check, if type of given field also implements ISerializable
        if (x.GetMemType ().Require (<[ ttype: ISerializable ]>))
          serializers = <[
                           printf ("<%s>\n", $(x.Name : string));      
                           if ($(nm : name) != null)
                             $(nm : name).Serialize ()
                           else
                             printf ("<null/>\n");
                           printf ("</%s>\n", $(x.Name : string));
                         ]>
                         :: serializers
        else
          /// and finally, we encounter case when there is no easy way to serialize 
          /// given field
          Message.FatalError ("field `" + x.Name + "' cannot be serialized")
    else
      Message.FatalError ("field `" + x.Name + "' cannot be serialized")
  };
  // after analyzing fields, we create method in our type, to execute created
  // expressions
  t.Define (<[ decl: public Serialize () : void
                     implements ISerializable.Serialize {
                       .. $serializers
                     }
            ]>);
}


Execution stages

Analysing object-oriented hierarchy and class members is a separate pass of the compilation. First it creates inheritance relation between classes, so we know exactly all base types of given type. After that every member inside of them (methods, fields, etc.) is being analysed and added to the hierarchy and its type annotations are resolved. After that also the rules regarding implemented interface methods are checked.

For the needs of macros we have decided to distinguish three moments in this pass at which they can operate on elements of class hierarchy. Every macro can be annotated with a stage, at which it should be executed.

Parameters of attribute macros

Every executed attribute macro operates on some element of class hierarchy, so it must be supplied with an additional parameter describing the object, on which macro was placed. This way it can easily query for properties of that element and use compiler's API to reflect or change the context in which it was defined.

For example a method macro declaration would be

[Nemerle.MacroUsage (Nemerle.MacroPhase.WithTypedMembers,
                     Nemerle.MacroTargets.Method)]
macro MethodMacro (t : TypeBuilder, f : MethodBuilder, expr)
{
  // use 't' and 'f' to query or change class-level elements
  // of program
}

Macro is annotated with additional attributes specifying respectively the stage in which macro will be executed and the macro target.

The available parameters contain references to class hierarchy elements that given macro operates on. They are automatically supplied by compiler and they vary on the target and stage of given macro. Here is a little table specifying valid parameters for each stage and target of attribute macro.


Attribute macro targets and parameters
MacroTarget MacroPhase.BeforeInheritance MacroPhase.BeforeTypedMembers MacroPhase.WithTypedMembers
Class TypeBuilder TypeBuilder TypeBuilder
Method TypeBuilder, ParsedMethod TypeBuilder, ParsedMethod TypeBuilder, MethodBuilder
Field TypeBuilder, ParsedField TypeBuilder, ParsedField TypeBuilder, FieldBuilder
Property TypeBuilder, ParsedProperty TypeBuilder, ParsedProperty TypeBuilder, PropertyBuilder
Event TypeBuilder, ParsedEvent TypeBuilder, ParsedEvent TypeBuilder, EventBuilder
Parameter TypeBuilder, ParsedMethod, ParsedParameter TypeBuilder, ParsedMethod, ParsedParameter TypeBuilder, MethodBuilder, ParameterBuilder
Assembly (not implemented) (none) (none) (none)


The intuition is that every macro has parameter holding its target and additionally objects containing it (like TypeBuilder is available in most of the attribute macros).

After those implicitly available parameters there come standard parameters explicitly supplied by user. They are the same as for expression level macros.

Reference to more advanced aspects

Hygiene and alpha-renaming of identifiers

Problem with names capture

Identifiers in quoted code (object code) must be treated in a special way, because we usually do not know in which scope they would appear. Especially they should not mix with variables with the same names from the macro-use site.


Consider the following macro defining a local function f

macro identity (e) { <[ def f (x) { x }; f($e) ]> }

Calling it with identity (f(1)) might generate confusing code like

def f (x) { x }; f (f (1))

To preserve names capture, all macro generated variables should be renamed to their unique counterparts, like in

def f_42 (x_43) { x_43 }; f_42 (f (1))

Hygiene of macros

The idea of separating variables introduced by a macro from those defined in the plain code (or other macros) is called `hygiene' after Lisp and Scheme languages. In Nemerle we define it as putting identifiers created during a single macro execution into a unique namespace. Variables from different namespaces cannot bind to each other.


In other words, a macro cannot create identifiers capturing any external variables or visible outside of its own generated code. This means, that there is no need to care about locally used names.


The Hygiene is obtained by encapsulating identifiers in special Name class. The compiler uses it to distinguish names from different macro executions and scopes (for details of implementation consult paper about macros (http://nemerle.org/metaprogramming.pdf)). Variables with appropriate information are created automatically by quotation.

def definition = <[ def y = 4 ]>;
<[ def x = 5; $definition; x + y ]>

When a macro creates the above code, identifiers y and x are tagged with the same unique mark. Now they cannot be captured by any external variables (with a different mark). We operate on the Name class, when the quoted code is composed or decomposed and we use <[ $(x : name) ]> construct. Here x is bound to am object of type Name, which we can use in other place to create exactly the same identifier.

An identifier can be also created by calling method Macros.NewSymbol(), which returns Name with an unique identifier, tagged with a current mark.

def x = Macros.NewSymbol ();
<[ def $(x : name) = 5; $(x : name) + 4 ]>

Controlled breaking hygiene

Sometimes it is useful to generate identifiers, which bind to variables visible in place where a macro is used. For example one of macro's parameters is a string with some identifiers inside. If we want to use these as real identifiers, then we need to break automatic hygiene. It is especially useful in embedding domain-specific languages, which reference symbols from the original program.

As an example consider a Nemerle.IO.sprint (string literal) macro (which have the syntax shortcut $"some text $id "). It searches given string literal for $var and creates a code concatenating text before and after $var to the value of var.ToString ().

def x = 3;
System.Console.WriteLine ($"My value of x is $x and I'm happy");

expands to

def x = 3;
System.Console.WriteLine ({ 
  def sb = System.Text.StringBuilder ("My value of x is "); 
  sb.Append (x.ToString ()); 
  sb.Append (" and I'm happy"); 
  sb.ToString () 
});

Breaking of hygiene is necessary here, because we generate code (reference to x), which need to have the same context as variables from invocation place of macro.

To make given name bind to the symbols from macro usesite, we use Nemerle.Macros.UseSiteSymbol (name : string) : Name function, or special splicing target usesite in quotations. Their use would be like in this simplified implementation of macro

macro sprint (lit : string) 
{
  def (prefix, symbol, suffix) = Helper.ExtractDollars (lit);
  def varname = Nemerle.Macros.UseSiteSymbol (symbol);
  <[ 
    def sb = System.Text.StringBuilder ($(prefix : string)); 
    sb.Append ($(varname : name).ToString ()); 
    // or alternatively  $(symbol : usesite)
    sb.Append ($(suffix : string)); 
    sb.ToString () 
  ]>
}

Note that this operations is 'safe', that is it changes context of variable to the place where macro invocation was created (see paper for more details).

Unhygienic variables

Sometimes it is useful to completely break hygiene, where programmer only want to experiment with new ideas. From our experience, it is often hard to reason about correct contexts for variables, especially when writing class level macros. In this case it is useful to be able to easily break hygine.

Nemerle provides it with <[ $("id" : dyn) ]> construct. It makes produced variable break hygiene rules and always bind to the nearest definition with the same name.