Some random thoughts about automatic (or not) distribution layer.
What I want to achieve is to make clean approach to perform distribution mechanism with virtually any distribution heuristic.
First (simplest) step is to allow user to write RPython programs with some kind of remote control over program execution. For start I would suggest using RMI (Remote Method Invocation) and remote object access (in case of low level it would be struct access). For the simplicity it will make some sense to target high-level platform at the beggining (CLI platform seems like obvious choice), which provides more primitives for performing such operations. To make attempt easier, I'll provide some subset of type system to be serializable which can go as parameters to such a call.
I take advantage of several assumptions:
The second step is to allow some heuristic to live and change calls to RMI calls. This should follow some assumptions (which may vary, regarding implementation):
The biggest step here is to provide JIT integration into distribution system. This should allow to make it really usefull (probably compile-time distribution will not work for example for whole Python interpreter, because of too huge granularity). This is quite unclear for me how to do that (JIT is not complete and I don't know too much about it). Probably we take JIT information about graphs and try to feed it to heuristic in some way to change the calls into RMI.
Most problems are to make mechanism working efficiently, so: