[SciPy-dev] Is Python being Dylanized?

[eric]

Hey Prabhu,

My sense is that user supplied type information mainly benefits classes.  I'm
not convinced we need strong typing for functions yet -- the only place I can
think of where we absolutely need it is for recursion.  In all other cases, it
appears that typing can be inferred at run time.  The two benefits that strong
typing does provide is smaller footprint (we can pre-compile multiple extenstion
functions within the same module), and possibly some safety.  I am not a
language weenie, so I could be missing something.  This is just based off of my
experience with weave.

>From Pat and your comments, the place where typing becomes more helpful is with
classes.  I haven't look at this much, so I'll go along with it for now.

>   (1) Define a language that is a subset of Python with a well defined
>   set of supported features.  Lets call this High Performance Python
>   or HPP (hpp) for short.

Right now, subset is the empty set. :-)

>
>     (a) A few special details like type information should be
>     supported and allowed for in hpp.
>
>     (b) These details can also be passed in via some special data
>     attribute or method.  Something like so:
>
> class Foo:
>     def __init__(self):
>         self.x = 1.0
>         self.y = 2.0
>         self.name = "Foo"
>         ...
>
>     def func(self):
>         return self.x*self.y
>
> classdata = {'x': 'FloatType', 'y': 'FloatType', 'name': 'FloatType'}
> classmember = {'__init__': {'args': None},
>                'func': {'args': None, 'return': 'FloatType'}}
>
> f = Foo()
> fa = weave.accelerate(Foo, classdata, classmember)

looks reasonable.

>   I think you get what I mean.  The idea is that if someone defines a
>   class like this its easy for weave to deal with it.  Expecting the
>   user to do this is not so bad because afterall they are going to get
>   pretty solid acceleration.  Also, they would do this after
>   experimenting with their code and design.  So the type information
>   does not get in their way when they are experimenting/prototyping.
>   So we get out of their hair and they out of ours.  Effectively they
>   can rapidly prototype it and then type it. :) This would make life
>   easy for everyone.  As you would note this is very similar to
>   Dylan's optional typing.  The above approach also does not rely on
>   changing Python in anyway.


>
>   (2) By defining HPP clearly it becomes easy for a user to know
>   exactly what language features are accelerated and what are not.

Good idea, but right now we're a looong way from specifying it.  The current
framework is still being developed.  Once we've gotten it up and running and
understand what is gonna be hard, we can start thinking about defining the
limits.

>   Therefore with a little training a person can become an adept at
>   writing HPP code and thereby maximize their productivity.  I think
>   this is an important step.

>   (3) As weave.accelerate improves and more core Python language
>   features are supported the HPP language set can be changed.

Sure

>   (4) Additionally if a tool like PyChecker were adapted to HPP then a
>   person could easily figure out if a class is HPP'able or not and
>   remedy the situation.

+1.  Very good idea.

>   (5) If its not too hard, then maybe those functions that the user
>   does not provide type information will remain unaccelerated.
>
> Clearly the above should work for builtin types.  So how do we support
> classes, inheritance etc.?  Maybe the type information can be used to
> abstract an interface.  Afterall, by defining a class with type
> information all necessary information is available.  Once that is done
> if a person defines a member function that is passed a base class then
> any derived class can also be supported because the basic interface is
> supported.  This should work by simply creating a class hierarchy in
> the wrapped C++ code that is generated.  I wonder if I'm merely
> re-inventing C++ here?  Maybe all that is needed is to somehow parse
> Python and generate equivalent C++ code underneath.  Essentially,
> Pythonic-C++.  In such a case, if we get a reasonable amount of C++
> supported HPP should be highly useable and technically it could be as
> fast as C++.
>
> As regards types changing over time, it should be possible to deal
> with them if the user is capable of anticipating them and specifying
> them in advance.  Like:
>
> def sqrt(x):
>   # do stuff
>
> weave.accelerate(sqrt, {'Float', 'Int', 'Matrix',... })
>
> where Matrix is some already accelerated Python class.
>
> Some of you might think that its too much expecting the user to
> provide type information for everything in advance.  But usually, the
> user has to do it anyway when they use C++/C/Fortran.  And IMHO, its
> not very hard adding type information after the code is reasonably
> mature.
>
> As I said before maybe these are all completely crazy and impossible
> ideas.  My apologies if they are.  If they are not then it looks like
> HPP should be pretty feasible and very exciting.

I don't think they are impossible, but I also do not have a handle on what the
possibilities with classes are yet.  Currently, I'm most interested in getting
Pat's framework more mature so that it is easy for people to modify and
experiment with.  This will mostly be done in the context of functions because
weave (the old weave) doesn't support classes yet.

I'm glad to see others are thinking about all this stuff!

see ya,
eric