From faassen at infrae.com  Mon Aug  2 14:04:25 2004
From: faassen at infrae.com (Martijn Faassen)
Date: Mon, 02 Aug 2004 14:04:25 +0200
Subject: [pypy-dev] iron python was released
In-Reply-To: <20040730152446.GD20259@solar.trillke>
References: <20040730152446.GD20259@solar.trillke>
Message-ID: <410E2DC9.2070300@infrae.com>

holger krekel wrote:
> Hi pypy-dev, 
> 
> just in case you missed it (i know i did so far), IronPython
> has been released: http://www.ironpython.com/ on 28th of
> July.  It is an (incomplete)  port of Python to the Common
> Lanuage Runtime (CLR) of Microsoft's .NET.  And the author, Jim
> Hugunin, will begin to work for Microsoft on 2nd of August
> (read more about it on the web site above). 
> 
> If anyone has tried it out already feel free to share your
> impressions. So far, I only skimmed the source code (c#) a 
> bit because it doesn't seem to be easily runnable on linux. 

I got it to run fairly easily on friday:

apt-get install mono

mono bin/IronPythonConsole.exe

or:

mono bin/IronPythonConsole.exe <some code>

The CLR is cool that way. :)

Regards,

Martijn


From roeland.rengelink at chello.nl  Wed Aug 11 18:51:13 2004
From: roeland.rengelink at chello.nl (Roeland Rengelink)
Date: Wed, 11 Aug 2004 18:51:13 +0200
Subject: [pypy-dev] testfailure on test_unicode.py in trivial object space
Message-ID: <411A4E81.2040003@chello.nl>

Hi,

I just checked out the source code and ran the unittest as per 
instructions on

http://codespeak.net/pypy/index.cgi?source

i.e.

python pypy/pypy/testall.py

(Note that there is a typo in the filename there (should be 
test_all.py), and that this runs the tests in trivial objectspace.)

There is a test failure in

objspace/std/test/test_unicodestring.py

which is reproduced by

python py.py -T
PyPy in TrivialObjSpace on top of Python 2.3.3
 >>>> u'a' == 'a'
||Traceback (interpreter-level):
...
Traceback (application-level):
   File "<inline>", line 1 in ?
'a' == u'a'
(application-level) TypeError: unicode.__cmp__(x,y) requires y to be a 
'unicode', not a 'str'

This works correctly in standard object space.

As far as I can tell pypy -T tries u'a'.__cmp__('a'), which raises the 
exception, while CPython ends up doing 'a'.__eq__(str(u'a'))

I do think that pypy approaches this more rationally in standard object 
space (i.e., implementing __eq__ and __lt__ for unicode, rather than 
__cmp__, as CPython does). So it's probably a good idea to accept the 
exception in trivila object space, rather than trying to emulate the 
convoluted rules for comparison in CPython.

It would probably be a good idea to skip the test in trivial object 
space (see patch).

Cheers,

Roeland Rengelink

Index: test_unicodestring.py
===================================================================
--- test_unicodestring.py       (revision 5925)
+++ test_unicodestring.py       (working copy)
@@ -5,13 +5,17 @@
  from pypy.tool import testit


-class TestUnicodeString(testit.AppTestCase):
+class TestUnicodeStringStdOnly(testit.AppTestCase):
+    def setUp(self):
+         self.space = testit.objspace('std')
+
      def test_compares(self):
          self.assertEqual(u'a', 'a')
          self.assertEqual('a', u'a')
          self.assertNotEqual(u'a', 'b')
          self.assertNotEqual('a', u'b')

+class TestUnicodeString(testit.AppTestCase):
      def test_addition(self):
          def check(a, b):
              self.assertEqual(a, b)



From arigo at tunes.org  Thu Aug 12 17:42:02 2004
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 12 Aug 2004 16:42:02 +0100
Subject: [pypy-dev] testfailure on test_unicode.py in trivial object space
In-Reply-To: <411A4E81.2040003@chello.nl>
References: <411A4E81.2040003@chello.nl>
Message-ID: <20040812154202.GA19484@vicky.ecs.soton.ac.uk>

Hello Roeland,

On Wed, Aug 11, 2004 at 06:51:13PM +0200, Roeland Rengelink wrote:
> PyPy in TrivialObjSpace on top of Python 2.3.3
> >>>> u'a' == 'a'
> ||Traceback (interpreter-level):
> ...

> It would probably be a good idea to skip the test in trivial object 
> space (see patch).

Definitely.  Patch applied -- thanks.


A bient?t,

Armin


From mwh at python.net  Thu Aug 12 19:41:50 2004
From: mwh at python.net (Michael Hudson)
Date: Thu, 12 Aug 2004 18:41:50 +0100
Subject: [pypy-dev] Re: testfailure on test_unicode.py in trivial object
	space
References: <411A4E81.2040003@chello.nl>
	<20040812154202.GA19484@vicky.ecs.soton.ac.uk>
Message-ID: <2macx0xnhd.fsf@starship.python.net>

Armin Rigo <arigo at tunes.org> writes:

> Hello Roeland,
>
> On Wed, Aug 11, 2004 at 06:51:13PM +0200, Roeland Rengelink wrote:
>> PyPy in TrivialObjSpace on top of Python 2.3.3
>> >>>> u'a' == 'a'
>> ||Traceback (interpreter-level):
>> ...
>
>> It would probably be a good idea to skip the test in trivial object 
>> space (see patch).
>
> Definitely.  Patch applied -- thanks.

I should have done this sooner, it's my fault and I've known about the
trivial object space failure for some time, sorry.  When is it time to
make the standard object space the default?

Cheers,
mwh

-- 
  I have gathered a posie of other men's flowers, and nothing but
  the thread that binds them is my own.                   -- Montaigne



From arigo at tunes.org  Thu Aug 12 20:35:02 2004
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 12 Aug 2004 19:35:02 +0100
Subject: [pypy-dev] Re: testfailure on test_unicode.py in trivial object
	space
In-Reply-To: <2macx0xnhd.fsf@starship.python.net>
References: <411A4E81.2040003@chello.nl>
	<20040812154202.GA19484@vicky.ecs.soton.ac.uk>
	<2macx0xnhd.fsf@starship.python.net>
Message-ID: <20040812183502.GA26024@vicky.ecs.soton.ac.uk>

Hello Michael,

On Thu, Aug 12, 2004 at 06:41:50PM +0100, Michael Hudson wrote:
> When is it time to make the standard object space the default?

Right now would be a great plan.


Armin


From tismer at stackless.com  Fri Aug 13 01:12:47 2004
From: tismer at stackless.com (Christian Tismer)
Date: Fri, 13 Aug 2004 01:12:47 +0200
Subject: [pypy-dev] Prothon on PyPy? (was:Prothon on CPython intrepreter?)
In-Reply-To: <001a01c480a6$37911d00$d801a8c0@MarkVaio>
References: <001a01c480a6$37911d00$d801a8c0@MarkVaio>
Message-ID: <411BF96F.7050806@stackless.com>

Mark Hahn wrote:

> Christian Tismer wrote:
> 
> 
>>Prothon is different enough to be an other target language. 
>>This would enhance our ideas and the implementations, because 
>>if we have to support different concepts, we will find better 
>>building blocks and better abstraction.
>>
>>Without having talked to the rest of the group, I am
>>pretty sure that we all would be glad to open a Prothon 
>>branch in PyPy.
> 
> 
> I flattered once again. Feel free.  
> 
> What would this require me to do?

Joining the group, learn svn, create a sub-project,
discuss it, find out what we are trying to implement,
find out how Prothon's needs can be built in.
On the latter, I think this will not be that much.
You need a new object space and a different interpreter.
At least I think it makes sense for you to talk in
pypy-dev.

> Would I be paid some of those euros? :-)

I guess no, not in this round, unless someone drops off
whom we need to replace, but you can join the sponsored
group when we apply for the next funding. Well, you would need
to be European, or employed by a European firm or institution.
We are trying to get Tim Peters in for that, for instance.

> Seriously though, I'd be happy to help out however I can.

It is less about helping out, but more like getting something.
You might just enjoy to have prothon as another target,
and it will run on whatever implementation framework we build.
Java is quite likely, .net is possible.

PyPy-ers, I'd be interested to see Mark in our project, at
least on the list.

ciao - chris
-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Mission Impossible 5oftware  :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34  home +49 30 802 86 56  mobile +49 173 24 18 776
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/



From mark at prothon.org  Fri Aug 13 05:29:11 2004
From: mark at prothon.org (Mark Hahn)
Date: Thu, 12 Aug 2004 20:29:11 -0700
Subject: [pypy-dev] Re: Prothon on PyPy? (was:Prothon on CPython
	intrepreter?)
References: <001a01c480a6$37911d00$d801a8c0@MarkVaio>
	<411BF96F.7050806@stackless.com>
Message-ID: <mkj5bdxm5syf$.gm3suv5qz7mp.dlg@40tude.net>

On Fri, 13 Aug 2004 01:12:47 +0200, Christian Tismer wrote:

>>>Without having talked to the rest of the group, I am
>>>pretty sure that we all would be glad to open a Prothon 
>>>branch in PyPy.

I've read everything I can about PyPy and checked out the repository.  I
haven't started looking at code yet.  PyPy looks quite exciting.  The
object space seperation looks very compatible with the .Net CLR. 

I was thinking I would implement Prothon on .Net CLR first and then next
year start seriously looking at PyPy, but now I've got this crazy idea in
my head to use PyPy to implement Prothon on top of the CLR.

I know the real PyPy with all its goals is a long way out, but I don't need
all the goals PyPy does.  Instead of an automatic translation to C using
your flow-graph tool, I could do a manual port to c# in the beginning. Or
maybe I'll take longer and actually work on the auto-translation to C#.
Also I would be starting full-time now, not later.

It would be ironic to have the first implementation of PyPy not be Python
and not on CPython.  It could be kind of a test run for the real PyPy.

Maybe when I wake up in the morning I will come to my senses and get over
this crazy idea.



From mark at hahnca.com  Fri Aug 13 02:40:57 2004
From: mark at hahnca.com (Mark Hahn)
Date: Thu, 12 Aug 2004 17:40:57 -0700
Subject: [pypy-dev] RE: Prothon on PyPy? (was:Prothon on CPython
	intrepreter?)
In-Reply-To: <411BF96F.7050806@stackless.com>
Message-ID: <001601c480ce$332a3940$0d01a8c0@MarkVaio>

Christian Tismer wrote:

> Mark Hahn wrote:
> > What would this require me to do?
> 
> Joining the group, 

I've joined pypy-dev if that is what you mean.  Do you know of any mirror
for pypy-dev that will let me use a newsreader with it?  Gmane only seems to
have python.pypy.

> learn svn, create a sub-project,

No problem.  Prothon has used SVN from the beginning.  Ben Collins, the
Prothon co-founder, is one of the SVN guys.  I'll wait to create the
sub-project until I get my feet wet.

> discuss it, find out what we are trying to implement,
> find out how Prothon's needs can be built in.
> On the latter, I think this will not be that much.

Will do.  I will read some archives and lurk for a while.  I may be a pain
in the a** in the beginning with my questions.

> You need a new object space and a different interpreter.

We will find that out soon enough.

> At least I think it makes sense for you to talk in
> pypy-dev.

That I am good at.  Do I have to sprechen Deutsche? :-)



From mwh at python.net  Fri Aug 13 12:30:25 2004
From: mwh at python.net (Michael Hudson)
Date: Fri, 13 Aug 2004 11:30:25 +0100
Subject: [pypy-dev] Re: testfailure on test_unicode.py in trivial object
	space
References: <411A4E81.2040003@chello.nl>
	<20040812154202.GA19484@vicky.ecs.soton.ac.uk>
	<2macx0xnhd.fsf@starship.python.net>
	<20040812183502.GA26024@vicky.ecs.soton.ac.uk>
Message-ID: <2m1xibxrcu.fsf@starship.python.net>

Armin Rigo <arigo at tunes.org> writes:

> Hello Michael,
>
> On Thu, Aug 12, 2004 at 06:41:50PM +0100, Michael Hudson wrote:
>> When is it time to make the standard object space the default?
>
> Right now would be a great plan.

Done.

Cheers,
mwh

-- 
  ... so the notion that it is meaningful to pass pointers to memory
  objects into which any random function may write random values
  without having a clue where they point, has _not_ been debunked as
  the sheer idiocy it really is.        -- Erik Naggum, comp.lang.lisp



From jacob at strakt.com  Fri Aug 13 13:12:01 2004
From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Fri, 13 Aug 2004 13:12:01 +0200
Subject: [pypy-dev] The pycore project
Message-ID: <200408131312.01835.jacob@strakt.com>

This seems to be another Python interpreter project to watch for useful 
synergies:

http://webpages.charter.net/allanms/2004/08/you-dont-tug-on-supermans-cape.html

The link was on yesterdays Daily Python URL.

Jacob


From jacob at strakt.com  Fri Aug 13 13:32:25 2004
From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Fri, 13 Aug 2004 13:32:25 +0200
Subject: [pypy-dev] Another project that may give useful input: libjit
Message-ID: <200408131332.25295.jacob@strakt.com>

This may be of some use:
http://www.southern-storm.com.au/libjit.html

Jacob


From hpk at trillke.net  Fri Aug 13 13:37:58 2004
From: hpk at trillke.net (holger krekel)
Date: Fri, 13 Aug 2004 13:37:58 +0200
Subject: Recoding C libs in Python (was: Re: [pypy-dev] The pycore project)
In-Reply-To: <200408131312.01835.jacob@strakt.com>
References: <200408131312.01835.jacob@strakt.com>
Message-ID: <20040813113758.GD5735@solar.trillke>

[Jacob Hall?n Fri, Aug 13, 2004 at 01:12:01PM +0200]
> This seems to be another Python interpreter project to watch for useful 
> synergies:
> 
> http://webpages.charter.net/allanms/2004/08/you-dont-tug-on-supermans-cape.html
> 
> The link was on yesterdays Daily Python URL.

Interesting (although there is no code).  

This reminds me of the task of reimplementing C-coded
libraries in Python which seems to be needed for a couple of
projects (Jython, IronPython, this new pycore one) and
certainly for PyPy even more so as we are approaching
low-level translations.  

So i guess we should think about organizing this effort in a
pypy-independent way i.e. it should be equally accessible to
all projects/interested parties and have a code base that can
be deployed within all the implementations.  What do you
think? Anyone interested to head/contribute to such an effort? 

cheers, 

    Holger



From tismer at stackless.com  Fri Aug 13 14:18:08 2004
From: tismer at stackless.com (Christian Tismer)
Date: Fri, 13 Aug 2004 14:18:08 +0200
Subject: [pypy-dev] Re: [Prothon-user] Re: Prothon on PyPy?
In-Reply-To: <mkj5bdxm5syf$.gm3suv5qz7mp.dlg@40tude.net>
References: <001a01c480a6$37911d00$d801a8c0@MarkVaio>	<411BF96F.7050806@stackless.com>
	<mkj5bdxm5syf$.gm3suv5qz7mp.dlg@40tude.net>
Message-ID: <411CB180.7030804@stackless.com>

Mark Hahn wrote:

...

> I know the real PyPy with all its goals is a long way out, but I don't need
> all the goals PyPy does.  Instead of an automatic translation to C using
> your flow-graph tool, I could do a manual port to c# in the beginning. Or
> maybe I'll take longer and actually work on the auto-translation to C#.
> Also I would be starting full-time now, not later.

Sure. Any useful fast path makes sense. You can also
try to use the C translation and put a C# transformation
on top of it, or modify it. If that leads to a C# code
generator for the project, it will be a win.
Or you do a direct manual thing first, to get things flying.
After all, having more supported target engines and more
target languages is good for the project.

> It would be ironic to have the first implementation of PyPy not be Python
> and not on CPython.  It could be kind of a test run for the real PyPy.

No problem if we start with IronicPython. :-)

> Maybe when I wake up in the morning I will come to my senses and get over
> this crazy idea.

I wish you a very good sleep -- chris
-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Mission Impossible 5oftware  :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34  home +49 30 802 86 56  mobile +49 173 24 18 776
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/


From rxe at ukshells.co.uk  Fri Aug 13 14:25:19 2004
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Fri, 13 Aug 2004 13:25:19 +0100 (BST)
Subject: [pypy-dev] Sprint in Dublin
Message-ID: <Pine.LNX.4.60.0408131211070.22564@echo.ukshells.co.uk>



Hi everyone!

I've been talking to a few people about organising a room in the local 
college for a sprint in Dublin for early October.  This is not a 
announcement that a sprint will be happening, just would like to get a 
feel for the interest level incase I am pursuing this in vain. 
Preliminary dates would be Oct 4th-11th. Also, if anyone who can assist me 
in organising - feel free to drop me a line. Thanks!

Cheers,
Richard



From mwh at python.net  Fri Aug 13 14:52:53 2004
From: mwh at python.net (Michael Hudson)
Date: Fri, 13 Aug 2004 13:52:53 +0100
Subject: [pypy-dev] Re: Sprint in Dublin
References: <Pine.LNX.4.60.0408131211070.22564@echo.ukshells.co.uk>
Message-ID: <2mpt5vw66y.fsf@starship.python.net>

Richard Emslie <rxe at ukshells.co.uk> writes:

> Hi everyone!
>
> I've been talking to a few people about organising a room in the local
> college for a sprint in Dublin for early October.

Hmm, the location suits me very well (Ryanair fly Bristol->Dublin, I
think) but the dates do not.  Unfortunately, I'm not really sure when
would be better -- later in October is more likely to work for me,
though.

Cheers,
mwh

-- 
  Arrrrgh, the braindamage!  It's not unlike the massively
  non-brilliant decision to use the period in abbreviations
  as well as a sentence terminator.  Had these people no
  imagination at _all_?                 -- Erik Naggum, comp.lang.lisp



From jacob at strakt.com  Fri Aug 13 14:57:11 2004
From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Fri, 13 Aug 2004 14:57:11 +0200
Subject: [pypy-dev] Re: Sprint in Dublin
In-Reply-To: <2mpt5vw66y.fsf@starship.python.net>
References: <Pine.LNX.4.60.0408131211070.22564@echo.ukshells.co.uk>
	<2mpt5vw66y.fsf@starship.python.net>
Message-ID: <200408131457.11240.jacob@strakt.com>

On fredag 13 augusti 2004 14.52, Michael Hudson wrote:
> Richard Emslie <rxe at ukshells.co.uk> writes:
> > Hi everyone!
> >
> > I've been talking to a few people about organising a room in the local
> > college for a sprint in Dublin for early October.
>
> Hmm, the location suits me very well (Ryanair fly Bristol->Dublin, I
> think) but the dates do not.  Unfortunately, I'm not really sure when
> would be better -- later in October is more likely to work for me,
> though.

The same goes for Laura and me. Early parts of October we are already booked, 
while having a sprint in Dublin would be very nice.

Jacob


From hpk at trillke.net  Fri Aug 13 15:06:16 2004
From: hpk at trillke.net (holger krekel)
Date: Fri, 13 Aug 2004 15:06:16 +0200
Subject: [pypy-dev] Re: [pypy-sprint] Sprint in Dublin
In-Reply-To: <Pine.LNX.4.60.0408131211070.22564@echo.ukshells.co.uk>
References: <Pine.LNX.4.60.0408131211070.22564@echo.ukshells.co.uk>
Message-ID: <20040813130616.GG5735@solar.trillke>

Hi Richard, 

[Richard Emslie Fri, Aug 13, 2004 at 01:25:19PM +0100]
> Hi everyone!
> 
> I've been talking to a few people about organising a room in the local 
> college for a sprint in Dublin for early October.  This is not a 
> announcement that a sprint will be happening, just would like to get a 
> feel for the interest level incase I am pursuing this in vain. 

I am definitely interested but you already knew this :-) 

> Preliminary dates would be Oct 4th-11th.

fine for me ASFAIK. Later is probably ok, too. 

> Also, if anyone who can assist me 
> in organising - feel free to drop me a line. Thanks!

I should be able to help organising.  Note that i wrote 
a checklist for some earlier sprints which helps to not
forget things:

  http://codespeak.net/pypy/index.cgi?doc/sprintinfo/checklist.html

which gets generated out of trunk/pypy/doc/sprintinfo/checklist.txt 
so feel free to enhance/correct the document. 

cheers, 

    Holger


From lac at strakt.com  Fri Aug 13 16:00:07 2004
From: lac at strakt.com (Laura Creighton)
Date: Fri, 13 Aug 2004 16:00:07 +0200
Subject: [pypy-dev] Re: [pypy-sprint] Sprint in Dublin 
In-Reply-To: Message from Richard Emslie <rxe@ukshells.co.uk> of "Fri,
	13 Aug 2004 13:25:19 BST."
	<Pine.LNX.4.60.0408131211070.22564@echo.ukshells.co.uk> 
References: <Pine.LNX.4.60.0408131211070.22564@echo.ukshells.co.uk> 
Message-ID: <200408131400.i7DE07uV030035@ratthing-b246.strakt.com>

Actually, it is early November that Jacob and I have already booked.

Laura, who thinks that a Sprint in Dublin would be very nice indeed.


From arigo at tunes.org  Fri Aug 13 16:41:37 2004
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 13 Aug 2004 15:41:37 +0100
Subject: [pypy-dev] Re: Sprint in Dublin
In-Reply-To: <2mpt5vw66y.fsf@starship.python.net>
References: <Pine.LNX.4.60.0408131211070.22564@echo.ukshells.co.uk>
	<2mpt5vw66y.fsf@starship.python.net>
Message-ID: <20040813144137.GA2478@vicky.ecs.soton.ac.uk>

Hi Richard!

[Richard]
> > I've been talking to a few people about organising a room in the local
> > college for a sprint in Dublin for early October.

Nice !  I can also volunteer to help organize things.  If it is of any help I
could show up in Dublin 1-2 days in advance (I've never been there, though.)

[Michael]
> Hmm, the location suits me very well (Ryanair fly Bristol->Dublin, I
> think) but the dates do not.  Unfortunately, I'm not really sure when
> would be better -- later in October is more likely to work for me,
> though.

It would be a shame that you could not attend the closest sprint so far!
I vote for whatever sprint dates suit Michael.


Looking forward to see you,

Armin


From anthony at interlink.com.au  Fri Aug 13 16:50:54 2004
From: anthony at interlink.com.au (Anthony Baxter)
Date: Sat, 14 Aug 2004 00:50:54 +1000
Subject: [pypy-dev] Re: Recoding C libs in Python
In-Reply-To: <20040813113758.GD5735@solar.trillke>
References: <200408131312.01835.jacob@strakt.com>
	<20040813113758.GD5735@solar.trillke>
Message-ID: <411CD54E.90601@interlink.com.au>

holger krekel wrote:
> This reminds me of the task of reimplementing C-coded
> libraries in Python which seems to be needed for a couple of
> projects (Jython, IronPython, this new pycore one) and
> certainly for PyPy even more so as we are approaching
> low-level translations.  
> 
> So i guess we should think about organizing this effort in a
> pypy-independent way i.e. it should be equally accessible to
> all projects/interested parties and have a code base that can
> be deployed within all the implementations.  What do you
> think? Anyone interested to head/contribute to such an effort? 

Here's a quick breakdown of the modules in Modules/, broken
up into rough groups. Hope it's useful - probably worth putting
on a wiki page so that people can correct this grouping, and
annotate it appropriately. For instance, there's a horde of
strange little SGI-specific modules - I might have missed one
or two.


Possibly doable
---------------

These ones are probably achievable, although you might think
otherwise.

_bisectmodule - already exists
_codecsmodule
_csv - already exists?
_heapqmodule - already exists
_hotshot
_localemodule
_randommodule
_sre - already exists?
_weakref
arraymodule
audioop
binascii
cPickle - already exists
cStringIO - already exists
cgensupport
cmathmodule
collectionsmodule
cryptmodule
datetimemodule - already exists
gcmodule
imageop
imgfile
itertoolsmodule
mathmodule
md5module
operator
parsermodule
rgbimgmodule
sgimodule
shamodule
stropmodule
structmodule
symtablemodule
syslogmodule
timingmodule
unicodedata
yuvconvert
zipimport

Deprecated
----------

These modules are deprecated

regexmodule
regexpr
rotormodule
mpzmodule

Links in an external library
----------------------------

These are all wrappers around external C libraries. Rewriting them
from scratch is really not practical.

_bsddb
_curses_panel
_cursesmodule
_ssl
_tkinter
almodule
bsddbmodule
bz2module
cdmodule
clmodule
dbmmodule
flmodule
fmmodule
gdbmmodule
glmodule
nismodule
puremodule
pyexpat
readline (replace with pyrepl?)
svmodule
termios
tkappinit
zlibmodule

System specific
---------------

These are modules that are wrappers around system calls
and the like. Rewriting them in pure Python will still
need a way to invoke the underlying calls - and will probably
be different for each platform (pypy, pycore, ironpython, &c).
A lot of these are also a mass of #ifdefs for various broken
operating systems.

dlmodule
errnomodule
fcntlmodule
fpectlmodule
fpetestmodule
grpmodule
linuxaudiodev
mmapmodule
ossaudiodev
posixmodule
pwdmodule
resource
selectmodule
signalmodule
socketmodule
sunaudiodev
threadmodule
timemodule



From lac at strakt.com  Fri Aug 13 17:08:05 2004
From: lac at strakt.com (Laura Creighton)
Date: Fri, 13 Aug 2004 17:08:05 +0200
Subject: [pypy-dev] Ever since we nuked (nearly all of) our spam in the
	pypy-dev archive
Message-ID: <200408131508.i7DF85jJ030369@ratthing-b246.strakt.com>

all of our links to articles in the pypy-dev archive have been broken.
I'm fixing them now.  Right now I am in doc/devel/testdesign.txt, and
I have found a dangling ReST reference:

after: Command line tool test_all
but before: New unit testing framework (not ready yet)

There is this thing: 

_mail: http://codespeak.net/pipermail/pypy-dev/2003q2/000789.html

This is a note from Armin saying he likes the Oscon Paper, which doesn't
seem to fit there at all.  But nothing refers to mail ... does anybody
remember writing this, what it was for, and what I should do with it?

I also changed the text to reflect the fact that the Standard Object
Space is what you get when you don't specify what you are running ...

Laura


From fuf at mageo.cz  Fri Aug 13 17:38:18 2004
From: fuf at mageo.cz (Michal Vitecek)
Date: Fri, 13 Aug 2004 17:38:18 +0200
Subject: [pypy-dev] Re: Recoding C libs in Python
In-Reply-To: <411CD54E.90601@interlink.com.au>
References: <200408131312.01835.jacob@strakt.com>
	<20040813113758.GD5735@solar.trillke>
	<411CD54E.90601@interlink.com.au>
Message-ID: <20040813153818.GA29068@foof.i3.cz>

Anthony Baxter wrote:
>Here's a quick breakdown of the modules in Modules/, broken
>up into rough groups. Hope it's useful - probably worth putting
>on a wiki page so that people can correct this grouping, and
>annotate it appropriately. For instance, there's a horde of
>strange little SGI-specific modules - I might have missed one
>or two.
>
>
>Possibly doable
>---------------

 i would add 'copy' module as well - the C implementation exists (cCopy)
 but is buggy.

-- 
		fuf		(fuf at mageo.cz)


From lac at strakt.com  Fri Aug 13 18:26:50 2004
From: lac at strakt.com (Laura Creighton)
Date: Fri, 13 Aug 2004 18:26:50 +0200
Subject: [pypy-dev] Re: Recoding C libs in Python 
In-Reply-To: Message from Anthony Baxter <anthony@interlink.com.au> 
	of "Sat, 14 Aug 2004 00:50:54 +1000." <411CD54E.90601@interlink.com.au> 
References: <200408131312.01835.jacob@strakt.com>
	<20040813113758.GD5735@solar.trillke>
	<411CD54E.90601@interlink.com.au> 
Message-ID: <200408131626.i7DGQo08030645@ratthing-b246.strakt.com>

In a message of Sat, 14 Aug 2004 00:50:54 +1000, Anthony Baxter writes:
>holger krekel wrote:
>> This reminds me of the task of reimplementing C-coded
>> libraries in Python which seems to be needed for a couple of
>> projects (Jython, IronPython, this new pycore one) and
>> certainly for PyPy even more so as we are approaching
>> low-level translations.  
>> 
>> So i guess we should think about organizing this effort in a
>> pypy-independent way i.e. it should be equally accessible to
>> all projects/interested parties and have a code base that can
>> be deployed within all the implementations.  What do you
>> think? Anyone interested to head/contribute to such an effort? 
>
>Here's a quick breakdown of the modules in Modules/, broken
>up into rough groups. Hope it's useful - probably worth putting
>on a wiki page so that people can correct this grouping, and
>annotate it appropriately. For instance, there's a horde of
>strange little SGI-specific modules - I might have missed one
>or two.

<snip>

Thank you very much.  The list now lives at
http://codespeak.net/pypy/index.cgi?doc/cmodules.html and you can
edit it the usual way as <your pypy trunk>/doc/cmodules.txt

Hmm, should we make a real wiki page of this so that people from other
projects don't need to get our codebase to edit?  

Laura


From arigo at tunes.org  Fri Aug 13 18:36:01 2004
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 13 Aug 2004 17:36:01 +0100
Subject: [pypy-dev] Ever since we nuked (nearly all of) our spam in the
	pypy-dev archive
In-Reply-To: <200408131508.i7DF85jJ030369@ratthing-b246.strakt.com>
References: <200408131508.i7DF85jJ030369@ratthing-b246.strakt.com>
Message-ID: <20040813163601.GA2412@vicky.ecs.soton.ac.uk>

Hello Laura,

On Fri, Aug 13, 2004 at 05:08:05PM +0200, Laura Creighton wrote:
> There is this thing: 
> 
> _mail: http://codespeak.net/pipermail/pypy-dev/2003q2/000789.html
> 
> This is a note from Armin saying he likes the Oscon Paper, which doesn't
> seem to fit there at all.  But nothing refers to mail ... does anybody
> remember writing this, what it was for, and what I should do with it?

Sorry cannot help there specifically, apart from noting that most of the rest
of testdesign.txt doesn't apply any more, since we are heading for the simpler
'assert'-based scheme.

Which reminds me to ask you how ready for PyPy you think the test conversion
tool is.  If I remember correctly the point still holding us was different:
how to actually run app-level tests.  I don't remember where we left the
question.  In fact, pypy/tool/pypyutest.py looks like the solution (which we
hacked in the post-EuroPython days) but I don't remember how complete that is.


A bient?t,

Armin.


From lac at strakt.com  Fri Aug 13 18:55:06 2004
From: lac at strakt.com (Laura Creighton)
Date: Fri, 13 Aug 2004 18:55:06 +0200
Subject: [pypy-dev] Ever since we nuked (nearly all of) our spam in the
	pypy-dev archive 
In-Reply-To: Message from Armin Rigo <arigo@tunes.org> of "Fri,
	13 Aug 2004 17:36:01 BST."
	<20040813163601.GA2412@vicky.ecs.soton.ac.uk> 
References: <200408131508.i7DF85jJ030369@ratthing-b246.strakt.com>
	<20040813163601.GA2412@vicky.ecs.soton.ac.uk> 
Message-ID: <200408131655.i7DGt6ZC030740@ratthing-b246.strakt.com>

In a message of Fri, 13 Aug 2004 17:36:01 BST, Armin Rigo writes:

>Which reminds me to ask you how ready for PyPy you think the test convers
>ion
>tool is.  If I remember correctly the point still holding us was differen
>t:
>how to actually run app-level tests.  I don't remember where we left the
>question.  In fact, pypy/tool/pypyutest.py looks like the solution (which
> we
>hacked in the post-EuroPython days) but I don't remember how complete tha
>t is.
>
>
>A bient?t,
>
>Armin.

The tool works great.  It converted our whole tree without error. I
tried it on the source trees of a few other projects as well, and
haven't found any errors in more than a month, I think.  (Actually, I
commented out the part that does assertRaises, since I don't have a
'raises' yet.)  What I think is left to do is to remove the lines that
say 'import unittest' and substitute whatever magic you need to put in
the file to get the new test framework -- assuming there is any.  

I don't know how complete pypy/tool/pypyutest.py is, either.

Laura


From mark at prothon.org  Sat Aug 14 21:52:19 2004
From: mark at prothon.org (Mark Hahn)
Date: Sat, 14 Aug 2004 12:52:19 -0700
Subject: [pypy-dev] Re: [Prothon-user] Re: Prothon on PyPy?
References: <001a01c480a6$37911d00$d801a8c0@MarkVaio>	<411BF96F.7050806@stackless.com>
	<mkj5bdxm5syf$.gm3suv5qz7mp.dlg@40tude.net>
	<411CB180.7030804@stackless.com>
Message-ID: <6b1eqt6dv8l3.tlku3yru7ymq.dlg@40tude.net>

On Fri, 13 Aug 2004 14:18:08 +0200, Christian Tismer wrote:

> Mark Hahn wrote:
> 
> ...
> 
>> I know the real PyPy with all its goals is a long way out, but I don't need
>> all the goals PyPy does.  Instead of an automatic translation to C using
>> your flow-graph tool, I could do a manual port to c# in the beginning. Or
>> maybe I'll take longer and actually work on the auto-translation to C#.
>> Also I would be starting full-time now, not later.
> 
> Sure. Any useful fast path makes sense. You can also
> try to use the C translation and put a C# transformation
> on top of it, or modify it. If that leads to a C# code
> generator for the project, it will be a win.
> Or you do a direct manual thing first, to get things flying.
> After all, having more supported target engines and more
> target languages is good for the project.
> 
>> It would be ironic to have the first implementation of PyPy not be Python
>> and not on CPython.  It could be kind of a test run for the real PyPy.
> 
> No problem if we start with IronicPython. :-)
> 
>> Maybe when I wake up in the morning I will come to my senses and get over
>> this crazy idea.
> 
> I wish you a very good sleep -- chris

After thinking about it I realized I need more experience with the CLR and
C# before I could tackle the PyPy generator.  I'm going ahead with the
Prothon implementation on the CLR but I'll keep PyPy in mind.  Every so
often I'll step back and revisit the issue.

Obviously there would be a core implementation hand-written in C#.  I'll
try to see how small that could be as I write this.  Maybe I could even
structure my current code that way.  Maybe I could even think like a code
generator as I write my code.



From arigo at tunes.org  Sun Aug 15 23:33:05 2004
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 15 Aug 2004 22:33:05 +0100
Subject: [pypy-dev] translate_pypy.py
Message-ID: <20040815213305.GA19055@vicky.ecs.soton.ac.uk>

Hi,

I've played a bit with interactive graph browsing in Pygame.  Try out the new
version of src/goal/translate_pypy.py.  It analyzes a bit of PyPy (still not
sure why only this bit), then display the the call graph (where nodes are
functions) and the class hierarchy.  Clicking on a function gives you the
function's usual flow graph.  Clicking on a class shows you the inferred
attributes.

The goal of the recent changes were also to make it practical to browse large
graphs.  We should be able not to get lost now.  The main method of navigation
is to click on a node or a connecting edge, which goes there.  Use the left
arrow key to come back.  If you wish to, as before, you can also click'n'drag
(left button to zoom, right button to scroll).


A bient?t,

Armin.


From arigo at tunes.org  Tue Aug 17 21:21:45 2004
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 17 Aug 2004 20:21:45 +0100
Subject: [pypy-dev] genpyrex -> genc ?
Message-ID: <20040817192145.GA8095@vicky.ecs.soton.ac.uk>

Hello,

Perhaps generating Pyrex code isn't such a great idea.  The great idea was to
use CPython as an intermediate target.  But writing C extension modules in C  
instead of in Pyrex is quite possibly much cleaner, because: 
 
* very basic C code is sufficient; 
 
* removing yet another intermediate step might not be a bad idea for clarity;
  
* reference counting is easy because we know exactly when variables get out of
  scope: when they reach the end of a basic block and are not sent to the next
  block;
  
* we don't have to work around various Pyrex restrictions (or hack Pyrex to
  lift them);
  
* most importantly, the whole mess with the "class Op" in genpyrex.py can 
  probably be completely omitted, because we can use C macros and generate
  just a succession of OP_XXX(a,b,c) where XXX is the operation name and
  a,b,c are the arguments.  Mapping the operations to actual C code is done
  once and for all in a C header file with macro definitions.

A quick-test (borring from genpyrex.py) is checked in a branch in
http://www.codespeak.net/svn/pypy/branch/pypy-genc/ producing (not yet
compilable) C code that looks like the attached example. It looks much like
assembler code.  It's quite readable and easy to map to the original flow
graph, too.

Not sure what to do now.  Does this look like a good idea?


A bient?t,

Armin.
-------------- next part --------------

/************************************************************/
 /***  Generic C header section                            ***/

#include <Python.h>

#define FREE(x)  Py_DECREF(x)


/************************************************************/
 /***  The rest is produced by genc.py                     ***/

/* forward declarations */
static PyObject* is_perfect_number__e653614(PyObject* v1);

/************************************************************/

PyObject* is_perfect_number__e653614(PyObject* v1)
{
	PyObject* v1;
	PyObject* v3;
	PyObject* v26;
	PyObject* v27;
	PyObject* v66;
	PyObject* v67;
	PyObject* v68;
	PyObject* v69;
	PyObject* v70;
	PyObject* v107;
	PyObject* v108;
	PyObject* v109;
	PyObject* v110;
	PyObject* v111;
	PyObject* v73;
	PyObject* v74;
	PyObject* v75;
	PyObject* v76;
	PyObject* v77;
	PyObject* v78;
	PyObject* v79;
	PyObject* v80;
	PyObject* v2;
	PyObject* v112;
	PyObject* v113;
	PyObject* v114;
	PyObject* v115;
	PyObject* v116;
	PyObject* v83;
	PyObject* v84;
	PyObject* v85;
	PyObject* v117;
	PyObject* v118;
	PyObject* v119;
	PyObject* v120;
	PyObject* v121;
	PyObject* v122;
	PyObject* v123;
	PyObject* v124;
	PyObject* v91;
	PyObject* v92;
	PyObject* v93;
	PyObject* v94;
	PyObject* v62;
	PyObject* v63;
	PyObject* v64;
	PyObject* v65;
	PyObject* v125;
	PyObject* v126;
	PyObject* v127;
	PyObject* v128;
	PyObject* v129;
	PyObject* v130;
	PyObject* v131;
	PyObject* v132;
	PyObject* v103;

  block0:
	v3 = v1;
	goto block1;

  block1:
	v26 = v3;
	v27 = (1);
	goto block2;

  block2:
	v66 = v26;
	v67 = v27;
	v68 = (0);
	goto block3;

  block3:
	OP_LT(v67, v66, v69)
	OP_IS_TRUE(v69, v70)
	CASE(v70, False) {
		v107 = v66;
		v108 = v67;
		v109 = v68;
		v110 = v69;
		v111 = v70;
		goto block4;
	}
	ELSE(v70, True) {
		v112 = v66;
		v113 = v67;
		v114 = v68;
		v115 = v69;
		v116 = v70;
		goto block8;
	}
  block4:
	FREE(v111)
	v73 = v107;
	v74 = v108;
	v75 = v109;
	v76 = v110;
	goto block5;

  block5:
	FREE(v76)
	v77 = v73;
	v78 = v74;
	v79 = v75;
	goto block6;

  block6:
	OP_EQ(v77, v79, v80)
	FREE(v77)
	FREE(v78)
	FREE(v79)
	v2 = v80;
	goto block7;

  block7:
	return v2;

  block8:
	OP_MOD(v112, v113, v83)
	OP_EQ(v83, (0), v84)
	OP_IS_TRUE(v84, v85)
	CASE(v85, False) {
		v117 = v112;
		v118 = v113;
		v119 = v114;
		v120 = v115;
		v121 = v116;
		v122 = v83;
		v123 = v84;
		v124 = v85;
		goto block9;
	}
	ELSE(v85, True) {
		v125 = v112;
		v126 = v113;
		v127 = v114;
		v128 = v115;
		v129 = v116;
		v130 = v83;
		v131 = v84;
		v132 = v85;
		goto block12;
	}
  block9:
	FREE(v120)
	FREE(v121)
	FREE(v122)
	FREE(v124)
	v91 = v117;
	v92 = v118;
	v93 = v119;
	v94 = v123;
	goto block10;

  block10:
	FREE(v94)
	v62 = v91;
	v63 = v92;
	v64 = v93;
	goto block11;

  block11:
	OP_INPLACE_ADD(v63, (1), v65)
	FREE(v63)
	v66 = v62;
	v67 = v65;
	v68 = v64;
	goto block3;

  block12:
	OP_INPLACE_ADD(v127, v126, v103)
	FREE(v127)
	FREE(v128)
	FREE(v129)
	FREE(v130)
	FREE(v131)
	FREE(v132)
	v62 = v125;
	v63 = v126;
	v64 = v103;
	goto block11;

}

/************************************************************/

static PyObject* c_is_perfect_number(PyObject* self, PyObject* args)
{

	PyObject* a0;
	if (!PyArg_ParseTuple(args, "O", &a0))
		return NULL;
	return is_perfect_number__e653614(a0);

}


static PyMethodDef is_perfect_numberMethods[] = {
	{"is_perfect_number", (PyCFunction)c_is_perfect_number, METH_VARARGS},
	{NULL, NULL}
};


void initis_perfect_number(void)
{
	Py_InitModule("is_perfect_number", is_perfect_numberMethods);
}


From rxe at ukshells.co.uk  Wed Aug 18 00:32:32 2004
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Tue, 17 Aug 2004 23:32:32 +0100 (BST)
Subject: [pypy-dev] genpyrex -> genc ?
In-Reply-To: <20040817192145.GA8095@vicky.ecs.soton.ac.uk>
References: <20040817192145.GA8095@vicky.ecs.soton.ac.uk>
Message-ID: <Pine.LNX.4.60.0408172316210.22759@echo.ukshells.co.uk>


Hi Armin,

On Tue, 17 Aug 2004, Armin Rigo wrote:

> Hello,
>
> Perhaps generating Pyrex code isn't such a great idea.  The great idea was to
> use CPython as an intermediate target.  But writing C extension modules in C
> instead of in Pyrex is quite possibly much cleaner, because:
>
> * very basic C code is sufficient;
>
> * removing yet another intermediate step might not be a bad idea for clarity;
>
> * reference counting is easy because we know exactly when variables get out of
>  scope: when they reach the end of a basic block and are not sent to the next
>  block;
>
> * we don't have to work around various Pyrex restrictions (or hack Pyrex to
>  lift them);
>
> * most importantly, the whole mess with the "class Op" in genpyrex.py can
>  probably be completely omitted, because we can use C macros and generate
>  just a succession of OP_XXX(a,b,c) where XXX is the operation name and
>  a,b,c are the arguments.  Mapping the operations to actual C code is done
>  once and for all in a C header file with macro definitions.
>
> A quick-test (borring from genpyrex.py) is checked in a branch in
> http://www.codespeak.net/svn/pypy/branch/pypy-genc/ producing (not yet
> compilable) C code that looks like the attached example. It looks much like
> assembler code.  It's quite readable and easy to map to the original flow
> graph, too.
>
> Not sure what to do now.  Does this look like a good idea?
>

Not much to add.  It is far easier to understand the C (CPython) code than 
the Pyrex as much closer to basic blocks.  Wasn't looking forward to 
having to dive into the Pyrex source code to try figure out what went 
wrong, when it does go wrong.  :-) Anyways you said that already.  +1 from 
me.

BTW New graphdisplay stuff is v. cool.  Would be nice if we could get a 
popup with the generated C code for that block when mouse lingers over it, 
somehow.

Just a silly thought off the top of my head - we could generate very basic 
c++ code (pretty much the same as the very basic c code) but with 
reference counted c++ pointers / wrappers and use of C scoping rules 
instead of PyObjects, and using the c++ exceptions mechanism to do pypy 
exceptions.  That probably needs some expanding / rationalization, but you 
get the idea (I might have a play at the weekend.)

Cheers,
Richard


From arigo at tunes.org  Wed Aug 18 10:38:30 2004
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 18 Aug 2004 09:38:30 +0100
Subject: [pypy-dev] genpyrex -> genc ?
In-Reply-To: <Pine.LNX.4.60.0408172316210.22759@echo.ukshells.co.uk>
References: <20040817192145.GA8095@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408172316210.22759@echo.ukshells.co.uk>
Message-ID: <20040818083830.GA21440@vicky.ecs.soton.ac.uk>

Hello Richard,

On Tue, Aug 17, 2004 at 11:32:32PM +0100, Richard Emslie wrote:
> Not much to add.  It is far easier to understand the C (CPython) code than 
> the Pyrex as much closer to basic blocks.

That was my impression too.

> BTW New graphdisplay stuff is v. cool.  Would be nice if we could get a 
> popup with the generated C code for that block when mouse lingers over it, 
> somehow.

Yes, this would be useful in particular when the generated C code starts to 
depend on annotations (i.e. be typed).  My plan to do that is to write type
suffix in the macro calls, like:

   OP_ADD_iii(v5, v6, v7)

where the "i" mean that the annotator has found the corresponding variable to
be an integer.  Then in genc.h we'd have several macros:

#define OP_ADD_iii(x,y,res)  res=x+y;
#define OP_ADD_ooo(x,y,res)  res=PyNumber_Add(x,y);

(Plus some error checking.)

> Just a silly thought off the top of my head - we could generate very basic 
> c++ code (pretty much the same as the very basic c code) but with 
> reference counted c++ pointers / wrappers and use of C scoping rules 
> instead of PyObjects, and using the c++ exceptions mechanism to do pypy 
> exceptions.

Could be interesting to try.  It might be not obvious to use the C++ scoping
rules to get the refcounting effect that we can get now (see example code
below), which is to FREE() precisely the variables that are active at the
current position -- Pyrex takes a very crude approach: initialize all
variables with NULL, and at the end Py_XDECREF() them all.  By constrast
genc.py only ever uses Py_DECREF() on variables that are known to be alive.  
Knowing which variables are alive is extremely simple in our flow model:
that's all input variables of the block, plus the "result" variables of the
operations above the one that failed.  Moreover when a block exits into
another block we can just transfer the references from the output variables to
the next block's input variables with no Py_XXCREF() at all -- this is the
part that is difficult to do with automatic scope-based refcounting in C++.

The exception part is interesting too, though we might also think along the
lines of setjmp/longjmp to do them in C at some point.  But it is probable
(and to hope that) on some platforms where setjmp is expensive, C++ compilers
can produce better code.  So C++ is worth a try, even if we only use a very
small number of non-C features.  The idea is the avoid to gradually
re-introduce the Pyrex problems if we target C++: convoluting the generated
source code to force it into some high-level concept of C++ when just plain C
is a reasonable alternative.

Musing aloud, a few targets that would probably be worth a try (or more):

* directly assembler code ((c) mwh),

* the brand new Mini C compiler written in Python, without the C front-end,
  targetting its intermediate representation,

* LLVM,

* Lisp again -- but not the current gencl.py based on the obfuscated
  genpyrex.py, but a minimal translator only a few pages long that just
  produces calls to Lisp macros (like genc.py currently does for C).
  After all Lisp macros are *really* powerful and we'll have no problem
  compiling complex type-dependent code with them.  But these macros should
  be defined in Lisp instead of gencl.py doing the dirty work.

* Parrot, a JVM class file, .NET CLR, Psyco's ivm (low-level interpreter),
  etc.


A bient?t,

Armin.


From hpk at trillke.net  Wed Aug 18 13:18:39 2004
From: hpk at trillke.net (holger krekel)
Date: Wed, 18 Aug 2004 13:18:39 +0200
Subject: [pypy-dev] genpyrex -> genc ?
In-Reply-To: <20040817192145.GA8095@vicky.ecs.soton.ac.uk>
References: <20040817192145.GA8095@vicky.ecs.soton.ac.uk>
Message-ID: <20040818111839.GV27481@solar.trillke>

Hi Armin, 

[Armin Rigo Tue, Aug 17, 2004 at 08:21:45PM +0100]
> Perhaps generating Pyrex code isn't such a great idea.

i'd dare to say it was a good idea at the time because
it worked and allowed us to test/debug/hack things quickly. 

> The great idea was to use CPython as an intermediate target.

yes, that was at the core. 

> But writing C extension modules in C  instead of in Pyrex is
> quite possibly much cleaner [...] 

It should be much cleaner in the end, although it seems you
are currently avoiding to generate e.g. C-int <-> PyInt style
code, not caring about Exceptions.  At least i liked Pyrex for 
helping with all this neccessary "fluff" as well as integration
into CPython so that it was easy to test the newly generated code. 
This is of course all possible to do ourself and it's probably
a good time to do it. 

> A quick-test (borring from genpyrex.py) is checked in a branch in
> http://www.codespeak.net/svn/pypy/branch/pypy-genc/ producing (not yet
> compilable) C code ... 

Yes, i hope it doesn't take too much time to actually generate 
C that doesn't result in Segfaults all the time :-) 

>It looks much like assembler code.  It's quite readable and
>easy to map to the original flow graph, too.

Yes, that's nice. 
 
> Not sure what to do now.  Does this look like a good idea?

try to get it to actually work and integrated into the tests? 

cheers, 

    holger


From radeex at gmail.com  Thu Aug 19 00:37:58 2004
From: radeex at gmail.com (Christopher Armstrong)
Date: Wed, 18 Aug 2004 18:37:58 -0400
Subject: [pypy-dev] PyPy for restricted execution Python
Message-ID: <60ed19d40408181537411f4824@mail.gmail.com>

I want to have a function along these lines:

safeEval(code, ns={'name': obj}, operations=['import', 'def', 'for', 'while'])

It'll execute 'code', with only access to the names in 'ns' dict, and
only the operations listed in the 'operations' list. This function
would be called from a regular CPython interpreter, and safeEval would
be an interface to PyPy.

My end goal is to have potentially hundreds of *totally* untrusted
scripts running concurrently; they shouldn't be able to stomp on each
other or on the "main" application code. CPU and memory restriction
are a big issue, but I'm only concerned with namespace and operation
restriction at the moment. The core of any restricted system needs to
be able to start code in an absolutely empty environment, not allowing
anything at all. Then I can add operations and functions necessary for
my application after I do the Hard Work of auditing each one ;)

This is the same reason I was recently working (well, somewhat.. ;) on
PLT-Spy[1], an implementation of Python on PLT-Scheme. It compiled the
Python to Scheme code which has calls to a Python runtime implemented
for PLT. The Python runtime does 99% of its work by calling out to
libpython, i.e., CPython. So, this let me have my cake and eat it too:
I could take advantage of PLT's awesome restricted-execution support,
*and* I could use CPython without reimplementing the whole runtime.

But it wasn't _that_ simple: we have to modify some fundamental parts
of CPython to interoperate with PLT. The garbage collection and
namespace functions, in particular. I'm hoping I can entirely avoid C
code by using PyPy and still be able to use the stock Python runtime.

PyPy is confusing to me so far, though. I've worked with several
implementations of small languages, including a tiny lisp meant for
restricted execution, and even an implementation of Python (PLT-Spy).
I can't seem to get my brain integrated with the PyPy codebase, yet,
but I'm not sure where to start. So, given that explanation of my
goal, where should I start? What might I have to change in PyPy? How
can I get an analog of pypy.interpreter.main.run_string that lets me
restrict what's available to the code?

Thanks for any help :-)

1: http://plt-spy.sourceforge.net/

-- 
 Twisted | Christopher Armstrong: International Man of Twistery
  Radix  |          Release Manager,  Twisted Project
---------+            http://radix.twistedmatrix.com


From hpk at trillke.net  Thu Aug 19 09:46:32 2004
From: hpk at trillke.net (holger krekel)
Date: Thu, 19 Aug 2004 09:46:32 +0200
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <60ed19d40408181537411f4824@mail.gmail.com>
References: <60ed19d40408181537411f4824@mail.gmail.com>
Message-ID: <20040819074632.GR27481@solar.trillke>

Hi Christopher, 

[Christopher Armstrong Wed, Aug 18, 2004 at 06:37:58PM -0400]
> I want to have a function along these lines:
> 
> safeEval(code, ns={'name': obj}, operations=['import', 'def', 'for', 'while'])
> 
> It'll execute 'code', with only access to the names in 'ns' dict, and
> only the operations listed in the 'operations' list. This function
> would be called from a regular CPython interpreter, and safeEval would
> be an interface to PyPy.

Note that your 'operations' seem to be syntactical constructs and not 
operations in a bytecode or objectspace sense.  If you really want to
forbid certain syntactical constructs you probably need to look
at the abstract syntax tree of the code string. Are you sure
you want to restrict something on the syntactical level?  Or
do you rather need to e.g. restrict what can be imported? The
latter might be done by providing your own bytecode 
and/or builtin-implementations within PyPy. 

And when you expose the above 'obj' what do you expect with
respect to attribute-accesses?  You can go a long way just 
having access to a plain python object. 

> My end goal is to have potentially hundreds of *totally* untrusted
> scripts running concurrently; they shouldn't be able to stomp on each
> other or on the "main" application code. CPU and memory restriction
> are a big issue, but I'm only concerned with namespace and operation
> restriction at the moment. The core of any restricted system needs to
> be able to start code in an absolutely empty environment, not allowing
> anything at all. Then I can add operations and functions necessary for
> my application after I do the Hard Work of auditing each one ;)

This would be an interesting one-week sprint topic at some point :-) 
It seems the basic approaches are controling name
visibility and/or controling object accesses. Both require 
careful thinking about all the objects reachable through 
attribute accesses. 

Maybe other can chime in here so that we can think together
about what the best approach for a restricted Python within 
PyPy might be.  So far it hasn't been anybodies "special interest". 

> This is the same reason I was recently working (well, somewhat.. ;) on
> PLT-Spy[1], an implementation of Python on PLT-Scheme. It compiled the
> Python to Scheme code which has calls to a Python runtime implemented
> for PLT. The Python runtime does 99% of its work by calling out to
> libpython, i.e., CPython. So, this let me have my cake and eat it too:
> I could take advantage of PLT's awesome restricted-execution support,
> *and* I could use CPython without reimplementing the whole runtime.
> 
> But it wasn't _that_ simple: we have to modify some fundamental parts
> of CPython to interoperate with PLT. The garbage collection and
> namespace functions, in particular. I'm hoping I can entirely avoid C
> code by using PyPy and still be able to use the stock Python runtime.

PyPy should generally allow you to avoid C. 

But I am not sure you have a misconception here.  While our
intermediate goal is to generate code runnable on top of the
CPython runtime PyPy aims to be a self-contained
implementation.  Do you need the stock CPython runtime for
anything in particular or would you be happy with a
self-contained Python implementation allowing restricted
execution? 
 
> PyPy is confusing to me so far, though. I've worked with several
> implementations of small languages, including a tiny lisp meant for
> restricted execution, and even an implementation of Python (PLT-Spy).
> I can't seem to get my brain integrated with the PyPy codebase, yet,
> but I'm not sure where to start. So, given that explanation of my
> goal, where should I start? What might I have to change in PyPy? How
> can I get an analog of pypy.interpreter.main.run_string that lets me
> restrict what's available to the code?

Possible starting points include the builtin module (modules/__builtin__*.py), 
the parser (see my first para, though), or the bytecode implementations 
(interpreter/pyopcode.py) or ... coming to a sprint which might take place 
between 9th and 19th of October in Dublin :-) 

cheers, 

    holger


From radeex at gmail.com  Thu Aug 19 10:15:14 2004
From: radeex at gmail.com (Christopher Armstrong)
Date: Thu, 19 Aug 2004 04:15:14 -0400
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <20040819074632.GR27481@solar.trillke>
References: <60ed19d40408181537411f4824@mail.gmail.com>
	<20040819074632.GR27481@solar.trillke>
Message-ID: <60ed19d4040819011558ca3ea2@mail.gmail.com>

Thanks for the reply, Holger!

On Thu, 19 Aug 2004 09:46:32 +0200, holger krekel <hpk at trillke.net> wrote:
> Hi Christopher,
> 
> [Christopher Armstrong Wed, Aug 18, 2004 at 06:37:58PM -0400]
> > safeEval(code, ns={'name': obj}, operations=['import', 'def', 'for', 'while'])
> 
> Note that your 'operations' seem to be syntactical constructs and not
> operations in a bytecode or objectspace sense. [...] Are you sure
> you want to restrict something on the syntactical level?  Or
> do you rather need to e.g. restrict what can be imported? 

Oops. In PLT-Spy, even those syntactical structures got compiled into
simple scheme function calls, so I'm used to thinking of them in that
way ;) I doubt it's *absolutely* necessary for me to totally restrict
those syntactic operations, but I think it would be cool to be able to
do that (it fits with the "absolutely inert environment as a
starting-point" principle).

> And when you expose the above 'obj' what do you expect with
> respect to attribute-accesses?  You can go a long way just
> having access to a plain python object.

Well, yes, I'm going to have to painstakingly audit every single
function, object, or whatever I make accessible to the code to make
sure it can't give access to any *other* objects I don't want the user
to have.  But are you saying there are inherent insecurities in the
python object system that give access to things I won't want the user
to access? Hopefully PyPy will be flexible enough for me to create
simpler versions of the objects, or maybe I can write something that
wraps objects and only has accessors for certain attributes (e.g.,
__call__) to be accessed by the restricted code. Horray capabilities
:-)

> But I am not sure you have a misconception here.  While our
> intermediate goal is to generate code runnable on top of the
> CPython runtime PyPy aims to be a self-contained
> implementation.  Do you need the stock CPython runtime for
> anything in particular or would you be happy with a
> self-contained Python implementation allowing restricted
> execution?

I'm self-centered ;) I think it would be a more expedient route to my
own goals to do it the way I suggested, as it looks like there's a
long way to go have PyPy be a total CPython replacement. If, otoh, you
guys make it usable in that regard by the time I want to actually
apply this, I'd be absolutely thrilled to use PyPy for all of my code
:-)

> Possible starting points include the builtin module (modules/__builtin__*.py),
> the parser (see my first para, though), or the bytecode implementations
> (interpreter/pyopcode.py)

Thanks! I'll look into these when I'm off work.

> or ... coming to a sprint which might take place
> between 9th and 19th of October in Dublin :-)

:-( I'm in the middle of a move and a new full-time job (that has no
interest in me researching this subject ;) so I doubt I'll have the
time to go to any sprints within the next year, but who knows! Maybe I
can organize one in Australia ;-)

-- 
 Twisted | Christopher Armstrong: International Man of Twistery
  Radix  |          Release Manager,  Twisted Project
---------+            http://radix.twistedmatrix.com


From hpk at trillke.net  Thu Aug 19 10:49:09 2004
From: hpk at trillke.net (holger krekel)
Date: Thu, 19 Aug 2004 10:49:09 +0200
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <60ed19d4040819011558ca3ea2@mail.gmail.com>
References: <60ed19d40408181537411f4824@mail.gmail.com>
	<20040819074632.GR27481@solar.trillke>
	<60ed19d4040819011558ca3ea2@mail.gmail.com>
Message-ID: <20040819084909.GV27481@solar.trillke>

Hi Christopher, 

[Christopher Armstrong Thu, Aug 19, 2004 at 04:15:14AM -0400]
> On Thu, 19 Aug 2004 09:46:32 +0200, holger krekel <hpk at trillke.net> wrote:
> > [Christopher Armstrong Wed, Aug 18, 2004 at 06:37:58PM -0400]
> > >
> > > safeEval(code, ns={'name': obj}, operations=['import', 'def', 'for', 'while'])
> > And when you expose the above 'obj' what do you expect with
> > respect to attribute-accesses?  You can go a long way just
> > having access to a plain python object.
> Well, yes, I'm going to have to painstakingly audit every single
> function, object, or whatever I make accessible to the code to make
> sure it can't give access to any *other* objects I don't want the user
> to have.  But are you saying there are inherent insecurities in the
> python object system that give access to things I won't want the user
> to access? Hopefully PyPy will be flexible enough for me to create
> simpler versions of the objects, or maybe I can write something that
> wraps objects and only has accessors for certain attributes (e.g.,
> __call__) to be accessed by the restricted code. Horray capabilities
> :-)

I think the challenge is to find an easy scheme for doing
restricted execution in Python.  So far it seems that there is
a lot of complexity involved.

The specific goal of disallowing File/IO operations and
restricting memory/cpu usage seems interesting enough to look
for a specific solution and not for some general scheme of
dis/allowing object accesses.  

One basic idea is to be able to allow/disallow the usage of code
objects according to in which module they live.  We then
wouldn't disallow "import socket" or any other import which
leads to importing socket but we would say "when i execute
this function f() here i don't want to execute code that comes
from the module 'socket'".  (Of course you can also have a positive
list of allowed modules etc.pp.). 

So we would have to keep exact references where code
originates/was build (mostly 'co_filename')  and we would need
the interpreter to check for these references and some
policies at certain points.  This seems easy enough to do in
PyPy.  

> > But I am not sure you have a misconception here.  While our
> > intermediate goal is to generate code runnable on top of the
> > CPython runtime PyPy aims to be a self-contained
> > implementation.  Do you need the stock CPython runtime for
> > anything in particular or would you be happy with a
> > self-contained Python implementation allowing restricted
> > execution?
> 
> I'm self-centered ;) I think it would be a more expedient route to my
> own goals to do it the way I suggested, as it looks like there's a
> long way to go have PyPy be a total CPython replacement. If, otoh, you
> guys make it usable in that regard by the time I want to actually
> apply this, I'd be absolutely thrilled to use PyPy for all of my code
> :-)

We'll see.  In some ways PyPy is not that far away from being
self-contained.  FYI, we really have only been hacking for ca.
7 sprint weeks so far plus some extra weeks spent by a few
developers.  
 
> > or ... coming to a sprint which might take place
> > between 9th and 19th of October in Dublin :-)
> 
> :-( I'm in the middle of a move and a new full-time job (that has no
> interest in me researching this subject ;) so I doubt I'll have the
> time to go to any sprints within the next year, but who knows! Maybe I
> can organize one in Australia ;-)

Uh oh, i see.  Well then good luck at your new job, anyway. It's not
even related to Twisted? 

cheers, 

    Holger


From radeex at gmail.com  Thu Aug 19 11:22:22 2004
From: radeex at gmail.com (Christopher Armstrong)
Date: Thu, 19 Aug 2004 05:22:22 -0400
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <20040819084909.GV27481@solar.trillke>
References: <60ed19d40408181537411f4824@mail.gmail.com>
	<20040819074632.GR27481@solar.trillke>
	<60ed19d4040819011558ca3ea2@mail.gmail.com>
	<20040819084909.GV27481@solar.trillke>
Message-ID: <60ed19d404081902225abc084e@mail.gmail.com>

Yo Homey,

On Thu, 19 Aug 2004 10:49:09 +0200, holger krekel <hpk at trillke.net> wrote:
> I think the challenge is to find an easy scheme for doing
> restricted execution in Python.  So far it seems that there is
> a lot of complexity involved.

Yes, doing it in an easy way (without the painstaking auditing) is
indeed a challenge, and not really one I'm interested in. Of course, a
lot of the auditing could be shared; i.e., I could come up with a
whitelist that's generally useful and secure in pretty much any
context and so other people wouldn't have to re-audit these functions.
However, the only truly "easy" way is blacklisting (at least this is
how it seems most people think when they think of restricted
execution), but I find blacklisting totally inadequate.

> The specific goal of disallowing File/IO operations and
> restricting memory/cpu usage seems interesting enough to look
> for a specific solution and not for some general scheme of
> dis/allowing object accesses.

*shrug*. Really, what applications is that useful for? Pretty much all
blacklisting schemes I've seen (e.g. disallowing File/IO operations)
have either been boring or still allow access to things I don't want
users to access.

> One basic idea is to be able to allow/disallow the usage of code
> objects according to in which module they live.  We then
> wouldn't disallow "import socket" or any other import which
> leads to importing socket but we would say "when i execute
> this function f() here i don't want to execute code that comes
> from the module 'socket'".  (Of course you can also have a positive
> list of allowed modules etc.pp.).
>
> So we would have to keep exact references where code
> originates/was build (mostly 'co_filename')  and we would need
> the interpreter to check for these references and some
> policies at certain points.  This seems easy enough to do in
> PyPy.

Hmm, strange. Well, I still don't think it's relevant to my case; I
doubt I will allow access to *any* stdlib modules in my particular
use-case (which I will explain further down) for restricted-execution.
 I will only put very application-specific things in my whitelist.

> We'll see.  In some ways PyPy is not that far away from being
> self-contained.  FYI, we really have only been hacking for ca.
> 7 sprint weeks so far plus some extra weeks spent by a few
> developers.

I hope that whatever hacking I do for restricted execution would also
improve PyPy as a general purpose CPython replacement.

> > :-( I'm in the middle of a move and a new full-time job (that has no
> > interest in me researching this subject ;) so I doubt I'll have the
> > time to go to any sprints within the next year, but who knows! Maybe I
> > can organize one in Australia ;-)
> 
> Uh oh, i see.  Well then good luck at your new job, anyway. It's not
> even related to Twisted?

Oh, it involves Python, Twisted, and lots of other things I'm pretty
involved with, but unfortunately, not games or restricted execution
;-( I plan on putting almost all of my free time into those two goals
over the coming years, though.

I guess I might as well explain what I really want to do with this
restricted execution: as many do, I want to deploy it in the context
of a multiplayer distributed virtual world where untrusted users
(i.e., free accounts) can add custom behavior to their avatar or the
objects they create. SecondLife[1] has done this in an amazing way (in
that it's the most technologically advanced I've seen today), except
the language they use is crap,  it's not open source, and there are
still many flaws[2]. Other older text-based systems (LambdaMOO, IIRC?)
offer scripting to untrusted users as well.

In a slightly different application, the users themselves wouldn't be
able to script their objects, but server administrators (in a
distributed VW) will. If you want a more focused *game* virtual world,
it's pretty impractical to let users script their own stuff (well,
maybe in more limited degrees...). However, it could still be very
useful for distributed worlds (in a network and social sense). For
example, you have server A and server B, run by Bob and Jane... They
want a portal between their servers, and they want to allow users to
bring objects developed by Bob and Jane to each other's server. But
Bob doesn't want users on Jane's machine bringing an Ultimate Sword of
Immense Kitten Slaying onto his server (or an Ultimate Bomb of L33t
/etc/shadow reading, for that matter). So he'll run her code in his
restricted execution environment so they can't read his /etc/shadow or
do more than 5000 points of damage.

1: http://secondlife.com/
2: mostly server-crashing bugs, but I'm sure there are some that can
be used more maliciously.

-- 
 Twisted | Christopher Armstrong: International Man of Twistery
  Radix  |          Release Manager,  Twisted Project
---------+            http://radix.twistedmatrix.com


From hpk at trillke.net  Thu Aug 19 11:57:33 2004
From: hpk at trillke.net (holger krekel)
Date: Thu, 19 Aug 2004 11:57:33 +0200
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <60ed19d404081902225abc084e@mail.gmail.com>
References: <60ed19d40408181537411f4824@mail.gmail.com>
	<20040819074632.GR27481@solar.trillke>
	<60ed19d4040819011558ca3ea2@mail.gmail.com>
	<20040819084909.GV27481@solar.trillke>
	<60ed19d404081902225abc084e@mail.gmail.com>
Message-ID: <20040819095733.GX27481@solar.trillke>

[Christopher Armstrong Thu, Aug 19, 2004 at 05:22:22AM -0400]
> On Thu, 19 Aug 2004 10:49:09 +0200, holger krekel <hpk at trillke.net> wrote:
> > One basic idea is to be able to allow/disallow the usage of code
> > objects according to in which module they live.  We then
> > wouldn't disallow "import socket" or any other import which
> > leads to importing socket but we would say "when i execute
> > this function f() here i don't want to execute code that comes
> > from the module 'socket'".  (Of course you can also have a positive
> > list of allowed modules etc.pp.).
> >
> > So we would have to keep exact references where code
> > originates/was build (mostly 'co_filename')  and we would need
> > the interpreter to check for these references and some
> > policies at certain points.  This seems easy enough to do in
> > PyPy.
> 
> Hmm, strange. Well, I still don't think it's relevant to my case; I
> doubt I will allow access to *any* stdlib modules in my particular
> use-case (which I will explain further down) for restricted-execution.
>  I will only put very application-specific things in my whitelist.

Maybe you could put those application specific things into a module 
and whitelist it using the above scheme. 

> I guess I might as well explain what I really want to do with this
> restricted execution: as many do, I want to deploy it in the context
> of a multiplayer distributed virtual world where untrusted users
> (i.e., free accounts) can add custom behavior to their avatar or the
> objects they create. [...]

Ah, now it gets interesting! :-) 
 
> In a slightly different application, the users themselves wouldn't be
> able to script their objects, but server administrators (in a
> distributed VW) will. If you want a more focused *game* virtual world,
> it's pretty impractical to let users script their own stuff (well,
> maybe in more limited degrees...). However, it could still be very
> useful for distributed worlds (in a network and social sense). For
> example, you have server A and server B, run by Bob and Jane... They
> want a portal between their servers, and they want to allow users to
> bring objects developed by Bob and Jane to each other's server. But
> Bob doesn't want users on Jane's machine bringing an Ultimate Sword of
> Immense Kitten Slaying onto his server (or an Ultimate Bomb of L33t
> /etc/shadow reading, for that matter). So he'll run her code in his
> restricted execution environment so they can't read his /etc/shadow or
> do more than 5000 points of damage.

I begin to see.  Wouldn't you need quite a lot of access to the
gaming application code if you want to give users freedom to script
their objects? 

Btw, have you looked at zope3's security model?  Kapil Thangavelu has 
written a nice readme file (with an example of distributed agents moving 
between sandboxes where they have different access capabilities): 

    http://cvs.zope.org/Zope3/src/zope/security/readme.txt?rev=1.5.12.3

I have to admit that if i would go for the "scripting within
games" goal i probably wouldn't go for neither of the above.
I'd probably think more about kernel-level sandboxes and using
unix permissions or posix-acls for file accesses (if any) and a
pipe to a VW server to communicate actions and events (with
its own security/sanity checks). 

Hum, maybe there would be a nice application for a (user-mode)
linux-kernel and a lean PyPy on top of it with no c-libraries
involved whatsoever as this could reduce memory/resource usage
to a degree where granting "free accounts" is cheap enough. 

cheers, 

    holger


From radeex at gmail.com  Thu Aug 19 12:30:59 2004
From: radeex at gmail.com (Christopher Armstrong)
Date: Thu, 19 Aug 2004 06:30:59 -0400
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <20040819095733.GX27481@solar.trillke>
References: <60ed19d40408181537411f4824@mail.gmail.com>
	<20040819074632.GR27481@solar.trillke>
	<60ed19d4040819011558ca3ea2@mail.gmail.com>
	<20040819084909.GV27481@solar.trillke>
	<60ed19d404081902225abc084e@mail.gmail.com>
	<20040819095733.GX27481@solar.trillke>
Message-ID: <60ed19d40408190330151cd0f3@mail.gmail.com>

'Sup dude,

On Thu, 19 Aug 2004 11:57:33 +0200, holger krekel <hpk at trillke.net> wrote:
> I begin to see.  Wouldn't you need quite a lot of access to the
> gaming application code if you want to give users freedom to script
> their objects?

Yeah, there would be a sizeable API. But the access would be very
carefully moderated and hand-picked. This is really the hard project;
I think implementing the r-exec system at the core is much easier than
doing the actual API design and auditing.

> Btw, have you looked at zope3's security model?  Kapil Thangavelu has
> written a nice readme file (with an example of distributed agents moving
> between sandboxes where they have different access capabilities):
> 
>     http://cvs.zope.org/Zope3/src/zope/security/readme.txt?rev=1.5.12.3

Hmm, no I haven't. *just skimmed* Did he actually write a sandbox that
allows untrusted python code to run securely? I was under the
impression that running Python code with CPython's runtime securely is
an absolutely futile idea. I'll try to grab kapil on IRC.

> I have to admit that if i would go for the "scripting within
> games" goal i probably wouldn't go for neither of the above.
> I'd probably think more about kernel-level sandboxes and using
> unix permissions or posix-acls for file accesses (if any) and a
> pipe to a VW server to communicate actions and events (with
> its own security/sanity checks).
>
> Hum, maybe there would be a nice application for a (user-mode)
> linux-kernel and a lean PyPy on top of it with no c-libraries
> involved whatsoever as this could reduce memory/resource usage
> to a degree where granting "free accounts" is cheap enough.

That would be cool, but it seems pretty hard (well, maybe in a few
more years of computing power advancement ;), and I think it's
acceptable to only use a limited pool of UMLs that run multiple users'
code. Here's why.

An important distinction to make is that between host and simulation
security; r-exec is still absolutely necessary even if you have a UML
for each script. r-exec *can* protect both the host and the
simulation. Of course, it's probably a good idea to keep a UML or
whatever as an extra layer of security, for the same reason that you
run various system daemons under chroots. For simulation-security,
though, a UML won't help; you still need to carefully restrict every
capability you give to the user (so that he can't use a python object
system trick to gain a reference to someone three rooms over and hit
them with his sword).

-- 
 Twisted | Christopher Armstrong: International Man of Twistery
  Radix  |          Release Manager,  Twisted Project
---------+            http://radix.twistedmatrix.com


From hpk at trillke.net  Thu Aug 19 13:32:28 2004
From: hpk at trillke.net (holger krekel)
Date: Thu, 19 Aug 2004 13:32:28 +0200
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <60ed19d40408190330151cd0f3@mail.gmail.com>
References: <60ed19d40408181537411f4824@mail.gmail.com>
	<20040819074632.GR27481@solar.trillke>
	<60ed19d4040819011558ca3ea2@mail.gmail.com>
	<20040819084909.GV27481@solar.trillke>
	<60ed19d404081902225abc084e@mail.gmail.com>
	<20040819095733.GX27481@solar.trillke>
	<60ed19d40408190330151cd0f3@mail.gmail.com>
Message-ID: <20040819113228.GY27481@solar.trillke>

[Christopher Armstrong Thu, Aug 19, 2004 at 06:30:59AM -0400]
> On Thu, 19 Aug 2004 11:57:33 +0200, holger krekel <hpk at trillke.net> wrote:
> > I begin to see.  Wouldn't you need quite a lot of access to the
> > gaming application code if you want to give users freedom to script
> > their objects?
> 
> Yeah, there would be a sizeable API. But the access would be very
> carefully moderated and hand-picked. This is really the hard project;

indeed. 

> > I have to admit that if i would go for the "scripting within
> > games" goal i probably wouldn't go for neither of the above.
> > I'd probably think more about kernel-level sandboxes and using
> > unix permissions or posix-acls for file accesses (if any) and a
> > pipe to a VW server to communicate actions and events (with
> > its own security/sanity checks).
> >
> > Hum, maybe there would be a nice application for a (user-mode)
> > linux-kernel and a lean PyPy on top of it with no c-libraries
> > involved whatsoever as this could reduce memory/resource usage
> > to a degree where granting "free accounts" is cheap enough.
> 
> That would be cool, but it seems pretty hard (well, maybe in a few
> more years of computing power advancement ;), and I think it's
> acceptable to only use a limited pool of UMLs that run multiple users'
> code. Here's why.

As PyPy will be faster than C i don't see a speed problem :-)  

Seriously, though, if the code is to interact with a gaming
api and not drive e.g. some graphics hardware i don't see a
big computing power problem with PyPy on top of UserModeLinux 
even if PyPy would be five times slower than CPython. 

> An important distinction to make is that between host and simulation
> security; r-exec is still absolutely necessary even if you have a UML
> for each script. r-exec *can* protect both the host and the
> simulation.

Well, a UML configuration (reused for every script with Copy-On-Write 
or read only FS devices) should provide enough host security, possibly
running on some secure linux capabilities based flavour.  

And, indeed, simulation security has to be taken care off separately. 

Wouldn't it make sense to define a "command protocol" with
integrated simulation security restrictions and provide a 
client side python library for speaking this protocol?  This way the 
user is free to program whatever she pleases but is restricted through 
host security (including CPU/RAM/FS restrictions) and can only produce 
commands which pass simulation security at the server side. 

That being said i wouldn't mind if we can come up with some sensible
easy enough r-exec mechanism within PyPy that would suit your 
Massively Scripted Multiplayer (tm) scenario. Any more ideas anyone? 

cheers, 

    Holger


From radeex at gmail.com  Thu Aug 19 22:30:58 2004
From: radeex at gmail.com (Christopher Armstrong)
Date: Thu, 19 Aug 2004 16:30:58 -0400
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <20040819113228.GY27481@solar.trillke>
References: <60ed19d40408181537411f4824@mail.gmail.com>
	<20040819074632.GR27481@solar.trillke>
	<60ed19d4040819011558ca3ea2@mail.gmail.com>
	<20040819084909.GV27481@solar.trillke>
	<60ed19d404081902225abc084e@mail.gmail.com>
	<20040819095733.GX27481@solar.trillke>
	<60ed19d40408190330151cd0f3@mail.gmail.com>
	<20040819113228.GY27481@solar.trillke>
Message-ID: <60ed19d404081913305c923c7@mail.gmail.com>

On Thu, 19 Aug 2004 13:32:28 +0200, holger krekel <hpk at trillke.net> wrote:
> [Christopher Armstrong Thu, Aug 19, 2004 at 06:30:59AM -0400]
> > That would be cool, but it seems pretty hard (well, maybe in a few
> > more years of computing power advancement ;), and I think it's
> > acceptable to only use a limited pool of UMLs that run multiple users'
> > code. Here's why.
> 
> As PyPy will be faster than C i don't see a speed problem :-)
> 
> Seriously, though, if the code is to interact with a gaming
> api and not drive e.g. some graphics hardware i don't see a
> big computing power problem with PyPy on top of UserModeLinux
> even if PyPy would be five times slower than CPython.

Oh, I'm not worried about PyPy's performance -- the performance I was
referring to was that of UML. Running hundreds of UMLs on a machine
right now is totally impractical, if I'm not mistaken.

> Wouldn't it make sense to define a "command protocol" with
> integrated simulation security restrictions and provide a
> client side python library for speaking this protocol?  This way the
> user is free to program whatever she pleases but is restricted through
> host security (including CPU/RAM/FS restrictions) and can only produce
> commands which pass simulation security at the server side.

mmmh... I'll think about this more :)

-- 
 Twisted | Christopher Armstrong: International Man of Twistery
  Radix  |          Release Manager,  Twisted Project
---------+            http://radix.twistedmatrix.com


From hpk at trillke.net  Fri Aug 20 07:19:57 2004
From: hpk at trillke.net (holger krekel)
Date: Fri, 20 Aug 2004 07:19:57 +0200
Subject: [pypy-dev] PyPy for restricted execution Python
In-Reply-To: <60ed19d404081913305c923c7@mail.gmail.com>
References: <60ed19d40408181537411f4824@mail.gmail.com>
	<20040819074632.GR27481@solar.trillke>
	<60ed19d4040819011558ca3ea2@mail.gmail.com>
	<20040819084909.GV27481@solar.trillke>
	<60ed19d404081902225abc084e@mail.gmail.com>
	<20040819095733.GX27481@solar.trillke>
	<60ed19d40408190330151cd0f3@mail.gmail.com>
	<20040819113228.GY27481@solar.trillke>
	<60ed19d404081913305c923c7@mail.gmail.com>
Message-ID: <20040820051957.GA27481@solar.trillke>

[Christopher Armstrong Thu, Aug 19, 2004 at 04:30:58PM -0400]
> On Thu, 19 Aug 2004 13:32:28 +0200, holger krekel <hpk at trillke.net> wrote:
> > [Christopher Armstrong Thu, Aug 19, 2004 at 06:30:59AM -0400]
> > > That would be cool, but it seems pretty hard (well, maybe in a few
> > > more years of computing power advancement ;), and I think it's
> > > acceptable to only use a limited pool of UMLs that run multiple users'
> > > code. Here's why.
> > 
> > As PyPy will be faster than C i don't see a speed problem :-)
> > 
> > Seriously, though, if the code is to interact with a gaming
> > api and not drive e.g. some graphics hardware i don't see a
> > big computing power problem with PyPy on top of UserModeLinux
> > even if PyPy would be five times slower than CPython.
> 
> Oh, I'm not worried about PyPy's performance -- the performance I was
> referring to was that of UML. Running hundreds of UMLs on a machine
> right now is totally impractical, if I'm not mistaken.

I think you are right that running hundreds of UMLs is
impractical (thought i believe this is mostly because you have
to assign static RAM amounts at boot time due to the linux
kernel's VM architecture).  Maybe running thousands of
user-processes on some form of se-linux or OpenBSD with tight
security settings (no opening of ports etc.pp.) would do for
the host security part.  

anyway, "restricted execution" within PyPy surely remains something
we should think about especially if we have concrete use cases. 

have fun, 

    holger

P.S.: let me know what you got out of Kapil and if the zope3 
      security architecture suits you. i am curious in case i
      want to write my own scriptable MMG :-) 


From arigo at tunes.org  Fri Aug 20 11:31:07 2004
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 20 Aug 2004 10:31:07 +0100
Subject: [pypy-dev] genpyrex -> genc ?
In-Reply-To: <20040818111839.GV27481@solar.trillke>
References: <20040817192145.GA8095@vicky.ecs.soton.ac.uk>
	<20040818111839.GV27481@solar.trillke>
Message-ID: <20040820093107.GA8555@vicky.ecs.soton.ac.uk>

Hello Holger,

On Wed, Aug 18, 2004 at 01:18:39PM +0200, holger krekel wrote:
> > Perhaps generating Pyrex code isn't such a great idea.
> 
> i'd dare to say it was a good idea at the time because
> it worked and allowed us to test/debug/hack things quickly. 

You are right on spot.  My quick hack is simple because it doesn't try to do
typing, exceptions, classes, etc.  I've spent some time trying to add typing
in it, with automatic conversions like C-int -> PyIntObject, and as you guess
it's becoming a mess...  Still trying to figure out if there is a clean way to
put it in, e.g. in a separate phase that would come because code generation
itself.


A bient?t,

Armin.


From lac at strakt.com  Sat Aug 21 13:37:05 2004
From: lac at strakt.com (Laura Creighton)
Date: Sat, 21 Aug 2004 13:37:05 +0200
Subject: [pypy-dev] C functions in CPython/Modules
Message-ID: <200408211137.i7LBb5FW023488@ratthing-b246.strakt.com>

Do these lot have unittests someplace?  I cannot seem to find any.
Laura


From mwh at python.net  Mon Aug 23 12:50:24 2004
From: mwh at python.net (Michael Hudson)
Date: Mon, 23 Aug 2004 11:50:24 +0100
Subject: [pypy-dev] Re: C functions in CPython/Modules
References: <200408211137.i7LBb5FW023488@ratthing-b246.strakt.com>
Message-ID: <2m3c2e86xb.fsf@starship.python.net>

Laura Creighton <lac at strakt.com> writes:

> Do these lot have unittests someplace?  I cannot seem to find any.

I'm unsure what you're asking.  If you are talking about the C modules
that live in the .../src/python/Modules/*.c files in the Python source
distribution then their unitests are in Lib/test/test_*.py like
everything else.  There's also Modules/_testcapimodule.c for the tests
that have to be written in C (these are run by
Lib/test/test_capimodule.py).

If that doesn't answer your question, then I obviously didn't
understand it.

Cheers,
mwh

-- 
42. You can measure a programmer's perspective by noting his
    attitude on the continuing vitality of FORTRAN.
  -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html



From lac at strakt.com  Mon Aug 23 12:55:26 2004
From: lac at strakt.com (Laura Creighton)
Date: Mon, 23 Aug 2004 12:55:26 +0200
Subject: [pypy-dev] Re: C functions in CPython/Modules 
In-Reply-To: Message from Michael Hudson <mwh@python.net> of "Mon,
	23 Aug 2004 11:50:24 BST." <2m3c2e86xb.fsf@starship.python.net> 
References: <200408211137.i7LBb5FW023488@ratthing-b246.strakt.com>
	<2m3c2e86xb.fsf@starship.python.net> 
Message-ID: <200408231055.i7NAtQJ8028527@ratthing-b246.strakt.com>

In a message of Mon, 23 Aug 2004 11:50:24 BST, Michael Hudson writes:
>Laura Creighton <lac at strakt.com> writes:
>
>> Do these lot have unittests someplace?  I cannot seem to find any.
>
>I'm unsure what you're asking.  If you are talking about the C modules
>that live in the .../src/python/Modules/*.c files in the Python source
>distribution then their unitests are in Lib/test/test_*.py like
>everything else.  There's also Modules/_testcapimodule.c for the tests
>that have to be written in C (these are run by
>Lib/test/test_capimodule.py).
>
>If that doesn't answer your question, then I obviously didn't
>understand it.
>
>Cheers,
>mwh

It's Lib/test/test_capimodule.py I was looking for.  Thank you.

Laura


From rxe at ukshells.co.uk  Tue Aug 24 01:12:51 2004
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Tue, 24 Aug 2004 00:12:51 +0100 (BST)
Subject: [pypy-dev] Yet another trace tool!
Message-ID: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>


Hi PyPy-ers,

After seeing Armin's flow viewer I got the motivation to look at 
translation.  What was suppose to be gencpp.py turned into another trace 
tool. ;-)

To run go to translator/tool and run flowtrace.py ! There is a debug flag 
to FlowTracer which will give you some idea what is actully being traced!

Well - back to gencpp.py then. :-)

Cheers,
Richard

PS I am wondering if we should create a top level hacks folder to save 
polluting our code base with lots of little noddy scripts?  Or do I have 
free reign to check in what and where I like? :-)




From arigo at tunes.org  Mon Aug 30 12:59:57 2004
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 30 Aug 2004 11:59:57 +0100
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
Message-ID: <20040830105957.GB2021@vicky.ecs.soton.ac.uk>

Hello PyPy-ers,

Sorry, I forgot to CC to the list when I first replied to Richard, and so we
didn't notice that the whole subsequent thread wasn't mirrored to pypy-dev...
I'm bouncing the messages.


Armin


From arigo at tunes.org  Fri Aug 27 10:34:35 2004
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 27 Aug 2004 09:34:35 +0100
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
Message-ID: <20040827083435.GA3528@vicky.ecs.soton.ac.uk>

Hello Richard,

On Tue, Aug 24, 2004 at 12:12:51AM +0100, Richard Emslie wrote:
> To run go to translator/tool and run flowtrace.py ! There is a debug flag 
> to FlowTracer which will give you some idea what is actully being traced!

It uses the earthenware package, about which I can't seem to find more info
with Google...

> Well - back to gencpp.py then. :-)

I am still struggling to find a reasonably clean way to have genc.py emit
typed C code.  I was wondering about using C++'s overloaded functions:  
gencpp.py could write an operation as a call like z=op_add(x,y), which the C++
compiler would resolve into a call to one of the predefined overloaded
implementations of op_add().  The compiler would also insert automatic
conversions with some C++ class trickery.  I'm a bit hesitant to go into that
direction for two reasons: we could get C++ compile errors in the presence of
ambiguities (i.e. when two overloaded functions could equally well be used for
a call); and gcc generates suboptimal code when we use structs or classes with
just one or two fields, like this one:

class PyObjectRef {
    PyObject* p;
  public:
    explicit PyObjectRef(PyObject* np) { p = np; } // constructor, consumes 
                                                   // reference
    ~PyObjectRef() { Py_DECREF(p); }      // destructor
    PyObjectRef(int n) { p = PyInt_FromLong(n); }   // conversion constructor
};

int op_add(int a, int b) { return a+b; }
PyObjectRef op_add(PyObjectRef &a, PyObjectRef &b) {
    return PyObjectRef(PyNumber_Add(a.p, b.p));
}

The conversion constructor from int to PyObjectRef is applied automatically by
the compiler, so it is quite nice, but you get a program that handles pointers
to PyObjectRefs (which are pointers to PyObject*, i.e. PyObject**) instead of
just PyObject*.  There might be a way to fix that, but that starts to be 
hackish.

Maybe Boost.Python already solves all these problems?

Well, is it a good idea to -- again -- target a high-level language and then
have to work around some of its concepts when they are not fully suited to our
low-level needs...?


Armin


From rxe at ukshells.co.uk  Fri Aug 27 13:27:34 2004
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Fri, 27 Aug 2004 12:27:34 +0100 (BST)
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <20040827083435.GA3528@vicky.ecs.soton.ac.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
Message-ID: <Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>


Hi Armin,

Sorry I was meaning to get back to your last mail about c++ pointers.  Was 
going to flesh out gencpp.py a bit this weekend.


On Fri, 27 Aug 2004, Armin Rigo wrote:

> Hello Richard,
>
> On Tue, Aug 24, 2004 at 12:12:51AM +0100, Richard Emslie wrote:
>> To run go to translator/tool and run flowtrace.py ! There is a debug flag
>> to FlowTracer which will give you some idea what is actully being traced!
>
> It uses the earthenware package, about which I can't seem to find more info
> with Google...
>

Oooops.  They can be commented out.  It is an opensource package from my 
company (I'm still there).  It hasn't got full clearance to be released 
and I didn't mean to include it in the code :-(.  I'll fix it tonight. 
Also importing pygame is not happy since there is a local path in working 
directory called pygame.

Anyways it is a throw away hack bit of code.  I wasn't sure I should check 
it in, but incase the tracing is useful, and it is easy to remove from 
svn, I did. I am just playing, trying to get an understanding for the 
flow/annotation code.  I was hoping to get it to a stage where I could 
point it to an imported module and it could convert it (and then maybe try 
intergrate with traceinteractive - although I've no idea how to convert 
app_xxx() space functions - I started hacking the FlowObjSpace <bad idea>, 
but I hoping you'd share you thoughts on that one???? :-))


>> Well - back to gencpp.py then. :-)
>
> I am still struggling to find a reasonably clean way to have genc.py emit
> typed C code.  I was wondering about using C++'s overloaded functions:
> gencpp.py could write an operation as a call like z=op_add(x,y), which the C++
> compiler would resolve into a call to one of the predefined overloaded
> implementations of op_add().

Yes - that was the plan - although I wasn't too sure how it would pan out 
(and I was going to mention it when I wrote to your last mail - but I 
assumed you'd already thought of it :-)).

The first step was to remove the macros from gencpp.h into python and 
generate a load of overloaded functions.  I don't understand the 
annotation code so I haven't much idea what this involves yet.

Also was thinking we could create a number of overloaded functors and 
create a table of them.  We could then write the equivalent of 
flowtrace.py as a c++ entension and attempt to use c++'s dynamic dispatch 
(I'm not confident that c++ would up to the job, and it is likely to be 
pretty slow - but fun to try nevertheless.)

  The compiler would also insert automatic
> conversions with some C++ class trickery.  I'm a bit hesitant to go into that
> direction for two reasons: we could get C++ compile errors in the presence of
> ambiguities (i.e. when two overloaded functions could equally well be used for
> a call);

Previous experience has shown that if the types are c++ classes/structs 
then it does the job.  But I do agree with your point.


> and gcc generates suboptimal code when we use structs or classes with
> just one or two fields, like this one:
>
> class PyObjectRef {
>    PyObject* p;
>  public:
>    explicit PyObjectRef(PyObject* np) { p = np; } // constructor, consumes
>                                                   // reference
>    ~PyObjectRef() { Py_DECREF(p); }      // destructor
>    PyObjectRef(int n) { p = PyInt_FromLong(n); }   // conversion constructor
> };
>
> int op_add(int a, int b) { return a+b; }
> PyObjectRef op_add(PyObjectRef &a, PyObjectRef &b) {
>    return PyObjectRef(PyNumber_Add(a.p, b.p));
> }
>

> The conversion constructor from int to PyObjectRef is applied automatically by
> the compiler, so it is quite nice, but you get a program that handles pointers
> to PyObjectRefs (which are pointers to PyObject*, i.e. PyObject**) instead of
> just PyObject*.  There might be a way to fix that, but that starts to be
> hackish.

I've attached what I have so far with the reference counted pointer (I am 
happy with phrase hackish here :-)) .  It was hacked in the early hours on 
Sunday night - and I haven't used it in anger yet - so it might be riddled 
with bugs / flaws.

In terms of suboptimal code - well it is already suboptimal because it is 
c/c++ on top of cpython. ;-)

Another thought is that if can do a full translation of standard object 
space and then start removing some of the overloaded functions which have 
reference to CPython.  Then the c++ compiler can tell us if they have been 
annotated out or not.  Incrementally we could remove our reliance on 
cpython. (There is fair chance that doesn't make any sense)

>
> Maybe Boost.Python already solves all these problems?
>

I've no idea!

> Well, is it a good idea to -- again -- target a high-level language and then
> have to work around some of its concepts when they are not fully suited to our
> low-level needs...?
>
>

The really nice thing about the pypy translation is that these are 
complete end point in that a genxxx.py has no dependents at all.  Even 
nicer is that we can do several independent translations with various 
levels of not so smart ideas (like everything I do!) - and then combine 
the better ones at the end.

Well chances are I'll be hacking some more this weekend - and be around on 
irc.  We could try to combine forces if you are interested?


Cheers,
Richard
-------------- next part --------------

/************************************************************/
/***  Generic C header section                            ***/

#include <Python.h>

/************************************************************/
/* Ref counted C pointer */

class CPythonProxyObject {
public:

  CPythonProxyObject(PyObject *instance, bool borrow = false) :
    refCount(1),
    borrowed(borrow),
    instance(instance) {
  }
  
  ~CPythonProxyObject() {
  }
  
public:
  
  bool decrement() {
    refCount--;
    if (refCount == 0) {
      
      if (!borrowed and instance != NULL) {
	Py_DECREF(instance);
      }
      return true;
    }
    
    return false;
  }
  
  void increment() {
    refCount++;
  }
  
  // Logically private
public:
  
  // The actual reference counter
  mutable int refCount;
  bool borrowed;

  // The actual data
  PyObject *instance;
};

/************************************************************/

class Handle {

 public:
  explicit Handle() :
    ptProxyObj (getNullProxyObject()) {
    }
  
  // Main constructor
  explicit Handle (PyObject *instance) {
    if (instance == NULL) {
      ptProxyObj = getNullProxyObject();
    } else {
      ptProxyObj = new CPythonProxyObject(instance);
    }
  }
  
  // Copy constuctor (increments)
  Handle (const Handle &ref) :
    ptProxyObj (ref.ptProxyObj) {
    ptProxyObj->increment();
  }
  
  // Assignment constuctor (increments unless itself)
  Handle & operator= (const Handle &ref) {
    if (this->ptProxyObj != ref.ptProxyObj) {
      decrement();
      
      ptProxyObj = ref.ptProxyObj;
      ptProxyObj->increment();
    }
    
    return *this;
  }
  
  ~Handle() {
    decrement();
  }
  
private:
  
  // This saves headaches 
  Handle (CPythonProxyObject *aPtrProxyObj) :
    ptProxyObj (aPtrProxyObj) {
    ptProxyObj->increment();
  }
  
public:

  // BASIC INTERFACE
  PyObject* get() {
    return ptProxyObj->instance;
  }

  bool isNull() const {
    return ptProxyObj == getNullProxyObject();
  }

private:
  // Will delete memory.  Note this is required if we want to add extra elements to proxy object.
  void decrement() {
    if (ptProxyObj->decrement()) {
      delete ptProxyObj;
    }
  }
    
private:
  static CPythonProxyObject *getNullProxyObject() {
    static CPythonProxyObject *nullObject = NULL;
    
    if (nullObject == NULL) {
      nullObject = new CPythonProxyObject ((PyObject *) NULL);
    }
    
    return nullObject;
  }
  
 private:
    CPythonProxyObject *ptProxyObj;
};

/************************************************************/

class PyException {
};

/************************************************************/
/* Compare Ops */

bool handleCmp(Handle x, Handle y) {
  return PyObject_RichCompareBool(x.get(), y.get() ,Py_EQ);
}

#define CREATE_COMPARE_OP(opNAME,Py_NAME) \
void opNAME(Handle x, Handle y, Handle r) { \
  PyObject *xx; \
  if (!(xx=PyObject_RichCompare(x.get(), y.get(), Py_NAME))) \
    throw PyException(); \
  r = Handle(xx); \
}

CREATE_COMPARE_OP(opLT,Py_LT);
CREATE_COMPARE_OP(opLE,Py_LE);
CREATE_COMPARE_OP(opEQ,Py_EQ);
CREATE_COMPARE_OP(opNE,Py_NE);
CREATE_COMPARE_OP(opGT,Py_GT);
CREATE_COMPARE_OP(opGE,Py_GE);

/* Use c++ overloading  */
void opGE(int x, 
	  int y,
	  int &r) {

  if (x > y) {
    r = 1;
  } else {
    r = 0;
  }
}

/************************************************************/

void opIS_TRUE(Handle x, Handle r) {

  PyObject *xx;
  switch (PyObject_IsTrue(x.get())) {
    
    case  0: 
      xx = Py_False; 
      break;
    case -1: 
      throw PyException();  
  default: 
      xx = Py_True; 
  }
  Py_INCREF(xx);
  r = Handle(xx);
}

/************************************************************/
/* Unary Ops */

#define CREATE_UNARY_OP(NAME,CFUNCTION) \
void NAME(Handle x, Handle r) { \
  PyObject *xx; \
  if (!(xx=CFUNCTION(x.get()))) \
	  throw PyException(); \
  r = Handle(xx); \
}

CREATE_UNARY_OP(opNEG,PyNumber_Negative)


/************************************************************/
/* Binary Ops */

#define CREATE_BINARY_OP(NAME,CFUNCTION) \
void NAME(Handle x, Handle y, Handle r) { \
  PyObject *xx; \
  if (!(xx=CFUNCTION(x.get(), y.get()))) \
	  throw PyException(); \
  r = Handle(xx); \
}

CREATE_BINARY_OP(opADD, PyNumber_Add);
CREATE_BINARY_OP(opMOD, PyNumber_Remainder);
CREATE_BINARY_OP(opINPLACE_ADD, PyNumber_InPlaceAdd);


/************************************************************/
/* Macros left over */

#define CASE(x,y) if (handleCmp(x,y))
#define ELSE(x,y) assert(handleCmp(x,y));


/************************************************************/
 /***  The rest is produced by genc.py                     ***/
-------------- next part --------------
Handle is_perfect_number__0(Handle v134)
{
        Handle v199;
        Handle v200;
        Handle v201;
        Handle v202;
        Handle v203;
        Handle v210;
        Handle v212;
        Handle v213;
        Handle v135;
        Handle v245;
        Handle v246;
        Handle v247;
        Handle v248;
        Handle v249;
        Handle v216;
        Handle v217;
        Handle v218;
        Handle v195;
        Handle v196;
        Handle v197;
        Handle v198;
        Handle v258;
        Handle v259;
        Handle v260;
        Handle v236;

  block0:
        // Input args are v134
        // Doing basic block operations

        // Doing exits
        // Doing input args to block1
        v199 = v134;
        v200 = kint_1;
        v201 = kint_0;
        goto block1;
  block1:
        // Input args are v199, v200, v201
        // Doing basic block operations
        opLT(v200, v199, v202);
        opIS_TRUE(v202, v203);

        // Doing exits
        CASE(v203, kint_0) {
                // Doing input args to block2
                v210 = v199;
                v212 = v201;
                goto block2;
        }
        ELSE(v203, kint_1) {
                // Doing input args to block4
                v245 = v199;
                v246 = v200;
                v247 = v201;
                v248 = v202;
                v249 = v203;
                goto block4;
       }
  block2:
        // Input args are v210, v212
        // Doing basic block operations
        opEQ(v210, v212, v213);

        // Doing exits
        // Doing input args to block3
        v135 = v213;
        goto block3;
  block3:
        // Input args are v135
        // Doing basic block operations

        // Doing exits
        return v135;
  block4:
        // Input args are v245, v246, v247, v248, v249
        // Doing basic block operations
        opMOD(v245, v246, v216);
        opEQ(v216, kint_0, v217);
        opIS_TRUE(v217, v218);

        // Doing exits
        CASE(v218, kint_0) {
                // Doing input args to block5
                v195 = v245;
                v196 = v246;
                v197 = v247;
                goto block5;
        }
        ELSE(v218, kint_1) {
                // Doing input args to block6
                v258 = v245;
                v259 = v246;
                v260 = v247;
                goto block6;
        }
  block5:
        // Input args are v195, v196, v197
        // Doing basic block operations
        opINPLACE_ADD(v196, kint_1, v198);

        // Doing exits
        // Doing input args to block1
        v199 = v195;
        v200 = v198;
        v201 = v197;
        goto block1;
  block6:
        // Input args are v258, v259, v260
        // Doing basic block operations
        opINPLACE_ADD(v260, v259, v236);

        // Doing exits
        // Doing input args to block5
        v195 = v258;
        v196 = v259;
        v197 = v236;
        goto block5;
}

From arigo at tunes.org  Fri Aug 27 14:54:03 2004
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 27 Aug 2004 13:54:03 +0100
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
Message-ID: <20040827125403.GA5011@vicky.ecs.soton.ac.uk>

Hello Richard,

On Fri, Aug 27, 2004 at 12:27:34PM +0100, Richard Emslie wrote:
> ...
> directory called pygame.

Argh.  Renaming time...

> flow/annotation code.  I was hoping to get it to a stage where I could 
> point it to an imported module and it could convert it (and then maybe try 
> intergrate with traceinteractive - although I've no idea how to convert 
> app_xxx() space functions - I started hacking the FlowObjSpace <bad idea>, 
> but I hoping you'd share you thoughts on that one???? :-))

Er, I think something had been done about these app-level helpers in 
FlowObjSpace -- or was it just a plan never implemented?

> >typed C code.  I was wondering about using C++'s overloaded functions:
> 
> Yes - that was the plan - although I wasn't too sure how it would pan out 
> (and I was going to mention it when I wrote to your last mail - but I 
> assumed you'd already thought of it :-)).

Actually not :-)

> Previous experience has shown that if the types are c++ classes/structs 
> then it does the job.  But I do agree with your point.

I see.  If the things include primitive types then it gets more messy.  
Attached is an example where the call to op_add() is deemed ambiguous by the
compiler, and I don't see how to solve that problem -- unless we don't use
ints at all as the type of the variables, but only a custom class Integer
containing an int field, as shown in the 2nd attachement.  Sadly, this is
elegant and fine with good C++ optimizers (e.g. Microsoft's) but g++ really
produces bad code in this kind of situation.

As you said, we should still try to do it this way, because it's not so
involved and quite independent from other efforts like generating C code with
conversions in the source.

In your gencpp.h, what is the purpose of CPythonProxyObject and its own
reference counter?  It seems that class Handle could just use a PyObject*
directly and the Py_INCREF/DECREF() macros to manipulate its refcount.

I'll not be too much on IRC for the next 10 days or so, but I'll try to show
up on Sunday.


A bientot,

Armin.


From arigo at tunes.org  Fri Aug 27 14:55:58 2004
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 27 Aug 2004 13:55:58 +0100
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <20040827125403.GA5011@vicky.ecs.soton.ac.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
	<20040827125403.GA5011@vicky.ecs.soton.ac.uk>
Message-ID: <20040827125558.GA28734@vicky.ecs.soton.ac.uk>

Forgot attachements.  Here is the 1st one.  The other one later...

Armin
-------------- next part --------------
#include <Python.h>


class BorrowedRef {
protected:
  PyObject* o;
public:
  BorrowedRef(PyObject* p) :o(p) { }
  BorrowedRef(const BorrowedRef &ref) :o(ref.o) { }
  operator PyObject*() const { return o; }
};

class Ref: public BorrowedRef {
public:
  Ref(PyObject* p) :BorrowedRef(p) { }
  Ref(const BorrowedRef &ref) :BorrowedRef(ref) { Py_INCREF(o); }
  Ref(int n) :BorrowedRef(PyInt_FromLong(n)) { }
  ~Ref() { Py_DECREF(o); }
};

inline int op_add(int x, int y) { return x+y; }
inline Ref op_add(const BorrowedRef &x, const BorrowedRef &y) {
  return PyNumber_Add(x, y);
}


int main()
{
  int r1 = 1234;
  Ref r2 = PyString_FromString("hello");
  Ref r3 = op_add(r1, r2);
}

From rxe at ukshells.co.uk  Fri Aug 27 17:12:14 2004
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Fri, 27 Aug 2004 16:12:14 +0100 (BST)
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <20040827125403.GA5011@vicky.ecs.soton.ac.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
	<20040827125403.GA5011@vicky.ecs.soton.ac.uk>
Message-ID: <Pine.LNX.4.60.0408271426190.26098@echo.ukshells.co.uk>


Hi Armin,

On Fri, 27 Aug 2004, Armin Rigo wrote:

...

>> Previous experience has shown that if the types are c++ classes/structs
>> then it does the job.  But I do agree with your point.
>
> I see.  If the things include primitive types then it gets more messy.
> Attached is an example where the call to op_add() is deemed ambiguous by the
> compiler, and I don't see how to solve that problem -- unless we don't use
> ints at all as the type of the variables, but only a custom class Integer
> containing an int field, as shown in the 2nd attachement.  Sadly, this is
> elegant and fine with good C++ optimizers (e.g. Microsoft's) but g++ really
> produces bad code in this kind of situation.
>

Ahhh, I misunderstood.  I was thinking along far more tedious lines of 
creating every possible combination of arguments and not doing any 
automatic conversion at all.

inline Ref op_add(const BorrowedRef &x, const BorrowedRef &y) ...
inline Ref op_add(int x, int y) ...
inline Ref op_add(int x, const BorrowedRef &y) ...
inline Ref op_add(const BorrowedRef &x, int y) ...
// Same again but returning ints

but the problem IIRC c++ doesn't support overloaded return results. 
Maybe we create a hierarchical tree of refs (including custom classes for 
int, float, etc like you suggested) and return the base type for each op 
(I'm confused with lack of c++ knowhow, need to hack some!).  I imagine 
this shifts from compile time to run time (which kind of defeats the 
purpose of what we are doing - but not too much).

> As you said, we should still try to do it this way, because it's not so
> involved and quite independent from other efforts like generating C code with
> conversions in the source.
>
> In your gencpp.h, what is the purpose of CPythonProxyObject and its own
> reference counter?  It seems that class Handle could just use a PyObject*
> directly and the Py_INCREF/DECREF() macros to manipulate its refcount.


Probably because I hacked some old templated code - but I think you are 
right since PyObject* is already ref counted. :-)  Assignment to self is 
always interesting (since you decrement, which collects and then increment 
which then is on an invalid object - but still don't need layer of 
indirection of CPythonProxyObject).  Also I thought it was good idea not 
to call Py_INCREF and PyDECREF too often - but that is plain silly.  It 
does make nulls easier though (but not much).

> I'll not be too much on IRC for the next 10 days or so, but I'll try to show
> up on Sunday.
>

Ok.  Can you drop us a copy of genc.py which was doing something with the 
annotations if that is ok (doesn't need to be any nice state)? Thanks.

Cheers,
Richard


From arigo at tunes.org  Sat Aug 28 18:46:40 2004
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 28 Aug 2004 17:46:40 +0100
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <Pine.LNX.4.60.0408271426190.26098@echo.ukshells.co.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
	<20040827125403.GA5011@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271426190.26098@echo.ukshells.co.uk>
Message-ID: <20040828164640.GA30860@vicky.ecs.soton.ac.uk>

Hello Richard,

On Fri, Aug 27, 2004 at 04:12:14PM +0100, Richard Emslie wrote:
> inline Ref op_add(const BorrowedRef &x, const BorrowedRef &y) ...
> inline Ref op_add(int x, int y) ...
> inline Ref op_add(int x, const BorrowedRef &y) ...
> inline Ref op_add(const BorrowedRef &x, int y) ...
> // Same again but returning ints
> 
> but the problem IIRC c++ doesn't support overloaded return results.

Right, hence the idea to use the compile-time overloading and automatic
conversions.  op_add() would be defined only for two Ints returning an Int and
for two Refs returning a Ref.  Then if something else like "two Ints returning
a Ref" is needed, the C++ compiler will use the (Int, Int) version and convert
the result from Int to Ref automatically.  If something unsatisfiable is
needed, like (Ref, Ref -> Int), then it means that there is a problem with the
annotation phase.

> Ok.  Can you drop us a copy of genc.py which was doing something with the 
> annotations if that is ok (doesn't need to be any nice state)? Thanks.

I should have saved some intermediate results, but I wasn't on Internet so I
couldn't just check in my progress and now it's really in the middle of a
rewrite...  Give me some more time...


Armin


From arigo at tunes.org  Sat Aug 28 23:02:26 2004
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 28 Aug 2004 22:02:26 +0100
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <Pine.LNX.4.60.0408271426190.26098@echo.ukshells.co.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
	<20040827125403.GA5011@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271426190.26098@echo.ukshells.co.uk>
Message-ID: <20040828210226.GA16050@vicky.ecs.soton.ac.uk>

Hi Richard,

On Fri, Aug 27, 2004 at 04:12:14PM +0100, Richard Emslie wrote:
> Ok.  Can you drop us a copy of genc.py which was doing something with the 
> annotations if that is ok (doesn't need to be any nice state)? Thanks.

Done.  Attached to this e-mail is the small test script I use...

Oups, I forgot to CC all my previous answers to the mailing list.  Sorry, I am
going to resend them now...


Armin
-------------- next part --------------
import autopath
import sys
from pypy.translator.translator import Translator
from pypy.translator.test import snippet
from pypy.translator.tool.buildpyxmodule import make_module_from_c
from pypy.tool.udir import udir
from genc import GenC


t = Translator(snippet.is_perfect_number)
t.simplify()
t.annotate([int])

name = t.functions[0].func_name

GenC(sys.stdout, t, name)

##cfile = udir.join('%s.c' % name)
##f = cfile.open('w')
##GenC(f, t, name)
##f.close()

##sys.stdout.write(cfile.read())

#mod = make_module_from_c(cfile)

From arigo at tunes.org  Mon Aug 30 12:58:00 2004
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 30 Aug 2004 11:58:00 +0100
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <Pine.LNX.4.60.0408282215390.10497@echo.ukshells.co.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
	<20040827125403.GA5011@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271426190.26098@echo.ukshells.co.uk>
	<20040828210226.GA16050@vicky.ecs.soton.ac.uk>
	<20040828210618.GA23672@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408282215390.10497@echo.ukshells.co.uk>
Message-ID: <20040830105800.GA2021@vicky.ecs.soton.ac.uk>

Hi Richard,

On Sat, Aug 28, 2004 at 10:22:31PM +0100, Richard Emslie wrote:
> Also BTW you can't overload c++ return types - unless you know something I 
> don't.

That's right, but it's not necessary.  The C version (genc.h) has a type code
for return type too (e.g. OP_ADD_iii) but that's only so that typer.py can
know what the return type will be, and convert it if needed.  In general I
don't think we need to overload on the return type.  If we define only:

  Int op_add(const Int&, const Int&)
  Ref op_add(const Ref&, const Ref&)

and then do:

  someref = op_add(someint, someint)

then the first signature will be choosen and the return type will be
automatically converted from Int to Ref, which is fine.


Armin


From rxe at ukshells.co.uk  Mon Aug 30 18:36:50 2004
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Mon, 30 Aug 2004 17:36:50 +0100 (BST)
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <20040830105800.GA2021@vicky.ecs.soton.ac.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
	<20040827125403.GA5011@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271426190.26098@echo.ukshells.co.uk>
	<20040828210226.GA16050@vicky.ecs.soton.ac.uk>
	<20040828210618.GA23672@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408282215390.10497@echo.ukshells.co.uk>
	<20040830105800.GA2021@vicky.ecs.soton.ac.uk>
Message-ID: <Pine.LNX.4.60.0408301228270.27579@echo.ukshells.co.uk>


Hi Armin,

On Mon, 30 Aug 2004, Armin Rigo wrote:

> Hi Richard,
>
> On Sat, Aug 28, 2004 at 10:22:31PM +0100, Richard Emslie wrote:
>> Also BTW you can't overload c++ return types - unless you know something I
>> don't.
>
> That's right, but it's not necessary.  The C version (genc.h) has a type code
> for return type too (e.g. OP_ADD_iii) but that's only so that typer.py can
> know what the return type will be, and convert it if needed.  In general I
> don't think we need to overload on the return type.  If we define only:
>
>  Int op_add(const Int&, const Int&)
>  Ref op_add(const Ref&, const Ref&)
>
> and then do:
>
>  someref = op_add(someint, someint)
>
> then the first signature will be choosen and the return type will be
> automatically converted from Int to Ref, which is fine.
>

I get it (partially). :-).  Either we only support ops which explicitly 
match the signature and convert the return type via a temporary or 
not-so-obviously let c++ create a temporary when passing in a const c++ 
reference (c++ : why???  if it wasn't reference then fair enuf... but 
*sigh*).

"""
   Int i;
   Ref r;
   r = op_add(i, i); // calls Int op_add(const Int&, const Int&)
                     // converts return type :-)

   r = op_add(i, r); // i converted to Ref as temporary  :-(
"""

but I assume the latter is not what you mean.

Actually,  to make things more explicit - I would prefer to only have an 
assignment operator for Ref:

class Ref... {
   ...

   Ref & operator= (const Int &ref) {
     ...
     return *this;
   }

}

Cheers,
Richard


From arigo at tunes.org  Mon Aug 30 19:00:59 2004
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 30 Aug 2004 18:00:59 +0100
Subject: [pypy-dev] Yet another trace tool!
In-Reply-To: <Pine.LNX.4.60.0408301228270.27579@echo.ukshells.co.uk>
References: <Pine.LNX.4.60.0408240003410.4345@echo.ukshells.co.uk>
	<20040827083435.GA3528@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271023540.26098@echo.ukshells.co.uk>
	<20040827125403.GA5011@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408271426190.26098@echo.ukshells.co.uk>
	<20040828210226.GA16050@vicky.ecs.soton.ac.uk>
	<20040828210618.GA23672@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408282215390.10497@echo.ukshells.co.uk>
	<20040830105800.GA2021@vicky.ecs.soton.ac.uk>
	<Pine.LNX.4.60.0408301228270.27579@echo.ukshells.co.uk>
Message-ID: <20040830170059.GA24295@vicky.ecs.soton.ac.uk>

Hello Richard,

On Mon, Aug 30, 2004 at 05:36:50PM +0100, Richard Emslie wrote:
> """
>   Int i;
>   Ref r;
>   r = op_add(i, i); // calls Int op_add(const Int&, const Int&)
>                     // converts return type :-)
> 
>   r = op_add(i, r); // i converted to Ref as temporary  :-(
> """
> 
> but I assume the latter is not what you mean.

No: it *is* exactly what I mean.  There is no other way to do that with the C
API of CPython.  To add an int i and a PyObject* r you have no choice but do:

   tmp = PyInt_FromLong(i);
   res = PyNumber_Add(tmp, r);
   Py_DECREF(tmp);

This is exactly what the C++ compiler will generate with automatic
conversions.


Armin