RFC: Viper: yet another python implementation

[Martijn Faassen]

John (Max) Skaller <skaller at maxtal.com.au> wrote:
> This is a request for comments on Viper, Yet Another implementation
> of Python. I'm building Viperi, the interpreter,  and I'm interested in
> comments on three issues:

> 	1) compatability with Python
> 	2) Extensions
> 	3) Optimisation

> Viperi is being built as a prelude to implementation of an optimising
> compiler, Viperc. Unlike previous attempts at a compiler, Viperc will do
> type inference, inlining, and special case analysis,
> in an attempt to optimise generated code.

It sounds like my (vaporware) Swallow idea. The difference with Viper
is that Viper apparently is going to do type inference, while Swallow
would (at least in the first stages) make do with nonintrusive type
declarations, somewhat like this:

__types__ = { 
  foo : functionType(args={a: intType, b: intType}, returns=listType(intType)),
  bar : functionType(args={}, returns=None)
}

def foo(a, b):
   return [a, b]

def bar():
   print "hello world"

But I forget the exact details right now. :) I did get a very simple Python
program going that could figure out if two complex types in the declaration
were the same. The idea is to go with a generic type approach somewhat
similar to C++ templates (but easier :). For instance, you'd say:

listType(intType) and Swallow would emit C code for a list of integers.

The idea is also to specify the types of local variables.

In the end, the only types that would need inferring with such a system
would be the type of empty lists and dictionaries:

__types__ = { foo : functionType(..., locals={ bar: listType(intType }) }

a = [] + [1, 2]

The first [] would have no type yet, but it would be fairly straightforward
to figure it out from the expression (I hope!)
 
> The interpreter aims to be compatible with 'pure python':
> python not using external C modules, and not fiddling
> implementation details too much. Some of the fiddles
> will work, and some will not. 

This is a good idea, as pure Python will often still be the development
language (while Swallow/Viper would only be invoked to compile and
optimize things later on, at least in the early stages).

> Compatibility:
> 	1) exceptions: both string and class exceptions are supported,
> 	the class exceptions are client defined in exceptions.py

> 	2) rebinding module variables after importing is not permitted
> 	3) None is a keyword.
> 	4) range is a keyword

Why this? Perhaps instead you could make 'range' and some other builtin
functions 'special functions'. The idea is that like module variables
(and some other set of things perhaps) they cannot be rebinded to something
else, so that the optimizer can do its stuff.

> 	5) What restrictions can you live with?

> Extensions:
> 	1) full slicing semantics
> 	2) optional values in dictionaries (Defaults to None)
> 	3) 'in' applies to dictionaries 'as if a sequence of keys were used'
> 	4) lexically scoped functions (including lambdas)
> 	5) optionally typed function parameters

> 	6) what do you want?

I wouldn't aim for *any* extensions to start with. In order for such a
project to succeed you want Viper code to run in Python and vice versa, 
unless there's serious messing with internals. At least make it possible
to turn these extensions *off* so they'll be errors.

[snip optimisation discussion]

Hm, I wish you luck, especially with the type inference...

Does viperc compile Python to C code? If so, try to minimize the amount of
C code that's emitted, if at all possible.

> 	Generally, compiler optimisations
> will go hand in hand with restrictions -- to optimise
> something, information is required, and that will
> require giving up some dynamism.

So you could do with Swallow's "type system" (big word for nothing there),
right? 

> 	It is difficult to optimise individual
> modules. It is a different story to optimise
> a whole program, where _every_ call
> to a function can be traced. Of course,
> this may involve restricting what 'exec' can do,
> and it may involve other restrictions
> (such as not  changing any module bindings
> after loading).

Hm, but the possibility to optimise individual modules is what attracts
many people to this idea. This way, people can stay away from writing
C modules and write them in Python instead. The Swallow cycle I had in
mind was something like this:

  * Write Python module

  * Need to make it faster (at least some functions)

  * Supply full type information for any local variables, arguments, etc.
    Anything *called* by the functions should also be Swallowable (or already
    supported by Swallow).

  * Swallow now optimises these functions by translating them to C, and
    you can use them in Python (just like C extension modules).

For classes you'd need some extra support (and more restrictions) but that
was phase 2. My advice is to start as small and unambitious as possible
for such an ambitious project. :)

Regards,

Martijn