PyPy
PyPy[summary-2006-03-15]

This Week in PyPy X

Introduction

If a week can be a long time in politics, it can be even longer in open source documentation... it seems that writing one summary a week isn't really sustainable, so we're going to try to scale back to an activity report every two weeks or so. This report covers roughly the last month of development.

PyCon

The end of February was naturally dominated by PyCon in Texas which was attended by several core developers. We gave three well-received talks, talked to a great many interested people and held a successful sprint, giving several newcomers their first taste of hacking on PyPy.

You can read more about this in the sprint report.

The Logic Sprint

Last week there was a PyPy-sprint at the UCL in Louvain-la-Neuve. Its main topic was logic and constraint programming in PyPy. Two of the sprint days were devoted to discussions with some researchers at the UCL who are heavily involved with the Oz programming language. The discussions were fruitful and very stimulating. For more details see the logic sprint report

The Logic Object Space

One of the more visible results of the sprint is the Logic Object Space, a very small (in terms of lines of code) object space that implements dataflow variables similar to what Oz has in Python. For examples on how it can be used, see the files uthread.py.

Garbage Collection

This week Michael with the help of Carl, Samuele and Armin managed for the first time to compile the pypy interpreter together with our own garbage collector written in Python. Performance is not that great as yet, about 40 times slower than CPython or about 7 times slower than with the Boehm collector. Given that we haven't tuned our garbage collector for performance at all, this isn't too bad.

This required quite a bit of work and first we refactored the way the existing two garbage collectors (reference counting and using the Boehm collector) are implemented in the C backend. Before, they were implemented by inserting C code into appropriate places in the generated code using string formatting. This had several disadvantages: %-formatting C code is obviously not helpful for other backends, and it makes writing code that reasons about the GC operations really very hard. We rewrote this to use a "GCTransformer" to instead insert new operations into the graphs that have equivalent functionality.

We wrote two transfomers that re-implemented the reference counting and Boehm policies, and a new "FrameworkGCTransformer" that inserts calls to functions defined by one of the garbage collectors written by Carl for his Summer of Code project. The final piece of the puzzle was a solution to the perennial "finding roots" problem, where we went for the exceedingly ugly and inefficient approach of keeping our own stack of root pointers.

And then we banged our heads against the usual obscure bugs and finally made it work. It turns out that debugging C that is generated from Python code that manipulates memory addresses directly is hard on the brain... we need to get back to being able to run everything on the memory simulator.

JIT Status

The JIT did not progress much recently. Nevertheless, it seems useful to try to give an overview of the current situation and where we are going next.

Our current "state of the art" in the area is represented by pypy.jit.test.test_hint_timeshift.test_arith_plus_minus(). It is a really tiny interpreter which gets turned into a compiler. There is a very draftish description of what is going on in:

http://codespeak.net/pypy/dist/pypy/doc/jit.html

Current areas of work:

  • the hint-rtyping actually produces a lot of bookkeeping code; we will have to reduce this amount in the future. More worryingly, it doesn't handle calls at all at the moment. We need to rethink and document how exactly the code the hint-rtyper produces should look like before we can start working in this direction.
  • in the last few days, Samuele and Arre worked on virtual structures (blue variables) for the hint-rtyper. Together with Armin they fixed some remaining bugs in the hint-annotator. Hint-rtyper support is still very preliminary and already hitting some head-scratching questions.