PyPy
PyPy[buildtool]

PyPyBuilder

What is this?

PyPyBuilder is an application that allows people to build PyPy instances on demand. If you have a nice idle machine connected to the Internet, and don't mind us 'borrowing' it every once in a while, you can start up the client script (in bin/client) and have the server send compile jobs to your machine. If someone requests a build of PyPy that is not already available on the PyPy website, and your machine is capable of making such a build, the server may ask your machine to create it. If enough people participate, with diverse enough machines, a 'build farm' is created.

Quick usage instructions

For the impatient, that just want to get started, some quick instructions.

First you'll need to have a checkout of the 'buildtool' package, that can be found here:

https://codespeak.net/svn/pypy/build/buildtool

To start a compilation, run (from the buildtool root directory):

$ ./bin/startcompile.py [options] <email address>

where the options can be found by using --help, and the email address will be used to send mail to once the compilation is finished.

To start a build server, to participate in the build farm, do:

$ ./bin/buildserver.py

That's it for the compilation script and build server, if you have your own project and want to set up your own meta server, you'll have to be a bit more patient and read the details below...

Components

The application consists of 3 main components: a meta server component, a client component that handles compilations (let's call this a 'build server') and a small client component to start compile jobs (which we'll call 'requesting clients' for now).

The server waits for build server to register, and for compile job requests. When participating clients register, they pass the server information about what compilations the system can handle (system info), and a set of options to use for compilation (compile info).

When now a requesting client requests a compilation job, the server checks whether a suitable binary is already available based on the system and compile info, and if so returns that. If there isn't one, the server walks through a list of connected participating clients to see if one of them can handle the job, and if so dispatches the compilation. If there's no participating client to handle the job, it gets queued until there is.

If a client crashes during compilation, the build is restarted, or error information is sent to the logs and requesting client, depending on the type of error. As long as no compilation error occurs (read: on disconnects, system errors, etc.) compilation will be retried until a build is available.

Once a build is available, the server will send an email to all clients waiting for the build (it could be that more than one person asked for some build at the same time!).

Configuration

There are several aspects to configuration on this system. Of course, for the meta server, build server and startcompile components there is configuration for the host and port to connect to, and there is some additional configuration for things like which mailhost to use (only applies to the server), but also there is configuration data passed around to determine what client is picked, and what the client needs to compile exactly.

Config file

The host/port configuration etc. can be found in the file 'config.py' in the build tool dir. There are several things that can be configured here, mostly related to what application to build, and where to build it. Please read the file carefully when setting up a new build network, or when participating for compilation, because certain items (e.g. the svnpath_to_url function, or the client_checkers) can make the system a lot less secure when not configured properly.

Note that all client-related configuration is done from command-line switches, so the configuration file is supposed to be changed on a per-project basis: unless you have specific needs, use a test version of the build tool, or are working on another project than PyPy, you will not want to modify the it.

System configuration

This information is used by the client and startcompile components. On the participating clients this information is retrieved by querying the system, on the requesting clients the system values are used by default, but may be overridden (so a requesting client running an x86 can still request PPC builds, for instance). The clients compare their own system config to that of a build request, and will (should) refuse a build if it can not be executed because of incompatibilities.

Compilation configuration

The third form of configuration is that of the to-be-built application itself, its compilation arguments. This configuration is only provided by the requesting clients, build servers can examine the information and refuse a compilation based on this configuration (just like with the system config, see 'client_checkers' in 'config.py'). Compilation configuration can be controlled using command-line arguments (use 'bin/startcompile.py --help' for an overview).

Build tool options

Yet another part of the configuration are the options that are used by the startcompile.py script itself: the user can specify what SVN path (relative to a certain base path) and what Subversion revision is desired. The revision can either be specified exactly, or as a range of versions.

Installation

Build Server

Installing the system should not be required: just run './bin/buildserver' to start. Note that it depends on the py lib (as does the rest of PyPy).

When starting a build server with PyPy's default configuration, it will connect to a meta server we have running in codespeak.net.

Meta Server

Also for the server there's no real setup required, and again there's a dependency on the py lib. Starting it is done by running './bin/metaserver'.

Running a compile job

Again installation is not required, just run './bin/startcompile.py [options] <email>' (see --help for the options) to start. Again, you need to have the py lib installed.

Normally the codespeak.net meta server will be used when this script is issued.

Using the build tool for other projects

The code for the build tool is meant to be generic. Using it for other projects than PyPy (for which it was originally written) is relatively straight-forward: just change the configuration, and implement a build client script (probably highly resembling bin/buildserver.py).

Note that there is a test project in 'tool/build/testproject' that can serve as an example.

Prerequisities

Your project can use the build tool if:

  • it can be built from Python

    Of course this is a rather vague requirement: theoretically _anything_ can be built from Python; it's just a matter of integrating it into the tool properly... A project that can entirely be built from Python code (like PyPy) is easier to integrate than something that is built from the command line, though (although implementing that won't be very hard either, see the test project for instance).

  • it is located in Subversion

    The build tool makes very little hard-coded assumptions, but having code in Subversion is one of them. There are several locations in the code where SVN is assumed: the command line options (see build tool options), the server (which checks SVN urls for validity, and converts HEAD revision requests to actual revision ids) and and build client (which checks out the data) all make this assumption, changing to a different revision control system is currently not easy and unsupported (but who knows what the future will bring).

  • it uses PyPy's config mechanism

    PyPy has a very nice, generic configuration mechanism (essentially wrapper OptionParser stuff) that makes dealing with fragmented configuration and command-line options a lot easier. This mechanism is used by the build tool: it assumes configuration is provided in this format. If your project uses this configuration mechanism already, you can provide the root Config object from config.compile_config; if not it should be fairly straight- forward to wrap your existing configuration with the PyPy stuff.

Basically that's it: if your project is stored in SVN, and you don't mind using Python a bit, it shouldn't be too hard to get things going (note that more documentation about this subject will follow in the future).

Web Front-End

To examine the status of the meta server, connected build servers and build requests, there is a web server available. This can be started using './bin/webserver' and uses port 8080 by default (override in config.py).

The web server presents a number of different pages:

  • / and /metaserverstatus - meta server status

    this displays a small list of information about the meta server, such as the amount of connected build servers, the amount of builds available, the amount of waiting clients, etc.

  • /buildservers - connected build servers

    this page contains a list of all connected build servers, system information and what build they're currently working on (if any)

  • /builds - a list of builds

    here you'll find a list of all builds, both done and in-progress and queued ones, with links to the details pages, the date they were requested and their status

  • /build/<id> - build details

    the 'build' (virtual) directory contains pages of information for each build - each of those pages displays status information, time requested, time started and finished (if appropriate), links to the zip and logs, and system and compile information

There's a build tool status web server for the meta server on codespeak.net available at http://codespeak.net/pypy/buildstatus/.

More info

For more information, bug reports, patches, etc., please send an email to guido@merlinux.de.