From Ben.Young at risk.sungard.com  Wed Jun  1 11:45:38 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Wed, 1 Jun 2005 10:45:38 +0100
Subject: [pypy-dev] Fw: Expected failure?
Message-ID: <OF98AA08C2.680D9218-ON80257013.003598AA-80257013.0034921F@risk.sungard.com>

Hi there,

I follow the pypy development, and tend to sync to head every day or so. I 
recently noticed some test failures that don't seem to be going away, and 
was wondering whether there is something wrong with my configuration or if 
there was something wrong with the way I was running the tests. I addition 
I noticed that some tests are skipped due to a floating point error. Is 
this expected too?

I run the tests with python 2.4 on cygwin.

Cheers,
Ben

============================= test process starts 
=============================
testing-mode: inprocess
executable:   c:\Python24\python.exe  (2.4.1-final-0)
using py lib: c:\Documents and Settings\YoungB\dist\py <rev unknown>

annotation\test\test_model.py[10] ..........
annotation\test\test_pairtype.py[3] ...
documentation\test_redirections.py[27] ...........................
documentation\website\contact.txt[2] s.
documentation\website\install.txt[3] s..
documentation\website\news.txt[4] s...
documentation\_ref.txt[2] s.
documentation\architecture.txt[9] s........
documentation\coding-guide.txt[6] s.....
documentation\contributor.txt[2] s.
documentation\extradoc.txt[2] s.
documentation\getting_started.txt[10] s.........
documentation\index.txt[27] s..........................
documentation\misc.txt[2] s.
documentation\objspace.txt[7] s......
documentation\release-0.6.txt[2] s.
documentation\svn-help.txt[3] s..
documentation\theory.txt[6] s.....
documentation\translation.txt[7] s......
interpreter\test\test_appinterp.py[15] ...............
interpreter\test\test_class.py[14] ..............
interpreter\test\test_code.py[2] ..
interpreter\test\test_compiler.py[6] ......
interpreter\test\test_descrtypecheck.py[2] ..
interpreter\test\test_eval.py[2] ..
interpreter\test\test_exceptcomp.py[11] ...........
interpreter\test\test_exec.py[11] ...........
interpreter\test\test_function.py[28] ............................
interpreter\test\test_gateway.py[12] ............
interpreter\test\test_generator.py[7] .......
interpreter\test\test_interpreter.py[16] ................
interpreter\test\test_main.py[3] ...
interpreter\test\test_module.py[5] .....
interpreter\test\test_nestedscope.py[6] ......
interpreter\test\test_objspace.py[14] ..............
interpreter\test\test_py.py[5] ....s
interpreter\test\test_pyframe.py[6] ......
interpreter\test\test_raise.py[13] .............
interpreter\test\test_special.py[2] ..
interpreter\test\test_typedef.py[2] ..
lib\test2\test_codeccallbacks.pylib\test2\test_exception_extra.py[1] .
lib\test2\test_exceptions_extra.py[1] .
lib\test2\test_file_extra.py[8] ........
lib\test2\test_imp_extra.py[2] ..
lib\test2\test_md5_extra.py[5] .....
lib\test2\test_sha_extra.py[2] ..
lib\test2\test_string_extra.py[1] .
lib\test2\test_struct_extra.py[1] .
module\__builtin__\test\test_apply.py[3] ...
module\__builtin__\test\test_builtin.py[43] 
...........................................
module\__builtin__\test\test_complexobject.py[11] ...........
module\__builtin__\test\test_descriptor.py[6] ......
module\__builtin__\test\test_filter.py[11] ...........
module\__builtin__\test\test_functional.py[20] ....................
module\__builtin__\test\test_import.py[21] ....................s
module\__builtin__\test\test_minmax.py[14] ..............
module\__builtin__\test\test_range.py[19] ...................
module\__builtin__\test\test_reduce.py[4] ....
module\__builtin__\test\test_vars.py[3] ...
module\__builtin__\test\test_zip.py[8] ........
module\parser\test\test_parser.py[1] .
module\parser\test\test_simple.py[0] 
module\recparser\test\test_pytokenizer.py[5] .....
module\recparser\test\test_samples.py[21] ..F..................
module\sys\test\test_sysmodule.py[30] ..............................
module\unicodedata\test\test_unicodedata.py[2] .F
objspace\flow\test\test_framestate.py[10] ..........
objspace\flow\test\test_model.py[3] ...
objspace\flow\test\test_objspace.py[35] 
...................................
objspace\std\test\test_boolobject.py[6] ......
objspace\std\test\test_dictobject.py[29] .............................
objspace\std\test\test_dictproxy.py[2] ..
objspace\std\test\test_fake.py[1] .
objspace\std\test\test_floatobject.py[12] ............
objspace\std\test\test_instmethobject.py[5] .....
objspace\std\test\test_intobject.py[40] 
........................................
objspace\std\test\test_iterobject.py[7] .......
objspace\std\test\test_listobject.py[32] ................................
objspace\std\test\test_listsort.py[2] ..
objspace\std\test\test_longobject.py[19] ...................
objspace\std\test\test_multimethod.py[7] .......
objspace\std\test\test_noneobject.py[3] ...
objspace\std\test\test_obj.py[4] ....
objspace\std\test\test_operation.py[3] ...
objspace\std\test\test_sliceobject.py[7] .......
objspace\std\test\test_stdobjspace.py[4] ....
objspace\std\test\test_stringformat.py[25] .........................
objspace\std\test\test_stringobject.py[50] 
..................................................
objspace\std\test\test_strutil.py[6] ......
objspace\std\test\test_tupleobject.py[16] ................
objspace\std\test\test_typeobject.py[25] .........................
objspace\std\test\test_unicodestring.py[11] ...........
objspace\std\test\test_userobject.py[15] ...............
objspace\test\test_descriptor.py[4] ....
objspace\test\test_descroperation.py[7] .......
objspace\test\test_thunkobjspace.py[5] .....
objspace\test\test_traceobjspace.py[5] .....
rpython\test\test_llann.py[8] ........
rpython\test\test_lltype.py[14] ..............
rpython\test\test_rarithmetic.py[33] .................................
rpython\test\test_rbool.py[4] ....
rpython\test\test_rclass.py[1] .
rpython\test\test_rfloat.py[4] ....
rpython\test\test_rint.py[4] ....
rpython\test\test_rlist.py[5] .....
rpython\test\test_rstr.py[3] ...
rpython\test\test_rtyper.py[4] ....
test_all.py[0] 
tool\pytest\test\test_overview.py[2] .F
tool\test\test_cache.py[1] .
tool\test\test_conftest1.py[4] ....
tool\test\test_pytestsupport.py[5] .....
tool\test\test_template.py[1] .
translator\c\test\test_database.py[15] ...............
translator\c\test\test_genc.py[4] ....
translator\genc\test\test_annotated.py[16] ................
translator\genc\test\test_lltyped.py[2] ..
translator\genc\test\test_notype.py[33] .................................
translator\genc\test\test_operation.py[1] .
translator\genc\test\test_typed.py[22] .................ss...
translator\llvm\test\test_class.py[12] ssssssssssss
translator\llvm\test\test_genllvm.py[12] ssssssssssss
translator\llvm\test\test_lazyattribute.py[4] ....
translator\llvm\test\test_seq.py[19] sssssssssssssssssss
translator\llvm\test\test_snippet.py[16] ssssssssssssssss
translator\pyrex\test\test_pyrextrans.py[21] .....................
translator\pyrex\test\test_sourcegen.py[3] ...
translator\test\test_annmm.py[2] ..
translator\test\test_annrpython.py[99] 
...................................................................................................
translator\test\test_annsimplifyrpython.py[99] 
...................................................................................................
translator\test\test_cltrans.py[14] ssssssssssssss
translator\test\test_geninterp.py[25] .........................
translator\test\test_rpystone.py[1] .
translator\test\test_translator.py[1] .


__________________________ reasons for skipped tests 
__________________________
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\lib\test2\test_codeccallbacks.py:2
reason: Skipped: this test module doesn't belong here

Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\interpreter\test\test_py.py:80
reason: Skipped: cannot detect process exit code for now

Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\module\__builtin__\test\test_import.py:158
reason: Skipped: unresolved issues with win32 shell quoting rules

Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_seq.py:17
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_genllvm.py:102
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_genllvm.py:165
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_seq.py:111
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_snippet.py:17
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_class.py:17
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_genllvm.py:128
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_genllvm.py:117
reason: Skipped: nothing works for now

Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\test\test_cltrans.py:43
reason: Skipped: Common Lisp neither configured nor detected.

Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_genllvm.py:55
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\llvm\test\test_genllvm.py:32
reason: Skipped: llvm-as not found on path

Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\genc\test\test_typed.py:40
Skipped in c:\Documents and 
Settings\YoungB\dist\pypy\translator\genc\test\test_typed.py:34
reason: Skipped: right now aborting python with Floating Point Error!

Skipped in c:\Documents and 
Settings\YoungB\dist\py\documentation\conftest.py:17
reason: Skipped: docutils not importable

_______________________________________________________________________________

    def check_parse(filepath):
>       pypy_tuples = pypy_parse(filepath)

[c:\Documents and 
Settings\YoungB\dist\pypy\module\recparser\test\test_samples.py:56]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ 

    def pypy_parse(filename):
        """parse <filename> using PyPy's parser module and return nested 
tuples
        """
        pyf = file(filename)
>       builder = parse_file_input(pyf, pythonutil.python_grammar())

[c:\Documents and 
Settings\YoungB\dist\pypy\module\recparser\pythonparse.py:39]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ 

    def parse_file_input(pyf, gram):
        """Parse a python file"""
>       return parse_python_source( pyf.read(), gram, "file_input" )

[c:\Documents and 
Settings\YoungB\dist\pypy\module\recparser\pythonparse.py:25]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ 

    def parse_python_source( textsrc, gram, goal ):
        """Parse a python source according to goal"""
        target = gram.rules[goal]
        src = PythonSource(textsrc)
        builder = BaseGrammarBuilder(debug=False, rules=gram.rules)
        result = target.match(src, builder)
        # <HACK> XXX find a clean way to process encoding declarations
        if src.encoding:
            builder._source_encoding = src.encoding
        # </HACK>
        if not result:
E           raise SyntaxError("at %s" % src.debug() )
>           SyntaxError: at line 3 : f(a,)

[c:\Documents and 
Settings\YoungB\dist\pypy\module\recparser\pythonparse.py:20]
[testcode  : c:\Documents and 
Settings\YoungB\dist\pypy\module\recparser\test\test_samples.py:55]
[modulepath: test_samples[8]]
_______________________________________________________________________________

    def test_cjk(self):
        import unicodedata
        for first, last in ((0x3400, 0x4DB5),
                            (0x4E00, 0x9FA5), # 9FBB in Unicode 4.1
                            (0x20000, 0x2A6D6)):
            # Test at and inside the boundary
            for i in (first, first + 1, last - 1, last):
                charname = 'CJK UNIFIED IDEOGRAPH-%X'%i
E               assert unicodedata.name(unichr(i)) == charname
>               (application-level) ValueError: character code not in 
range(0x110000)

[c:\Documents and Settings\YoungB\dist\pypy\None<c:\Documents and 
Settings\YoungB\dist\pypy\interpreter\gateway.py:573>:9]
[testcode  : c:\Documents and 
Settings\YoungB\dist\pypy\module\unicodedata\test\test_unicodedata.py:43]
[modulepath: AppTestUnicodeData().test_cjk]
_______________________________________________________________________________

    def test_getlatest_datetime(self): 
>       result = self.rc.getlatest('test_datetime', ok=1) 

[c:\Documents and 
Settings\YoungB\dist\pypy\tool\pytest\test\test_overview.py:22]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ 

    def getlatest(self, name, timeout=0, error=0, ok=0): 
        l = []
E       resultlist = self.name2result[name]
>       KeyError: 'test_datetime'

[c:\Documents and Settings\YoungB\dist\pypy\tool\pytest\overview.py:34]
[testcode  : c:\Documents and 
Settings\YoungB\dist\pypy\tool\pytest\test\test_overview.py:21]
[modulepath: TestResultCache().test_getlatest_datetime]
_______________________________________________________________________________
===== tests finished: 1377 passed, 3 failed, 94 skipped in 171.23 seconds 
=====


From hpk at trillke.net  Wed Jun  1 18:46:10 2005
From: hpk at trillke.net (holger krekel)
Date: Wed, 1 Jun 2005 18:46:10 +0200
Subject: [pypy-dev] Fw: Expected failure?
In-Reply-To: <OF98AA08C2.680D9218-ON80257013.003598AA-80257013.0034921F@risk.sungard.com>
References: <OF98AA08C2.680D9218-ON80257013.003598AA-80257013.0034921F@risk.sungard.com>
Message-ID: <20050601164610.GZ5694@solar.trillke.net>

Hi Ben, 

On Wed, Jun 01, 2005 at 10:45 +0100, Ben.Young at risk.sungard.com wrote:
> Hi there,
> 
> I follow the pypy development, and tend to sync to head every day or so. I 
> recently noticed some test failures that don't seem to be going away, and 
> was wondering whether there is something wrong with my configuration or if 
> there was something wrong with the way I was running the tests. I addition 
> I noticed that some tests are skipped due to a floating point error. Is 
> this expected too?
> 
> I run the tests with python 2.4 on cygwin.

the one problem relating to test_overview.py is strange: do
you have a recent checkout of the 'testresult' directory in
your pypy-dist checkout tree?  it should help to do an 'svn
up' in that particular directory which is not updated via the
'svn up pypy-dist' but only when you explicitely work on
generating compliance test results. 

cheers, 
    
    holger


From Ben.Young at risk.sungard.com  Thu Jun  2 10:42:14 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Thu, 2 Jun 2005 09:42:14 +0100
Subject: [pypy-dev] Fw: Expected failure?
In-Reply-To: <20050601164610.GZ5694@solar.trillke.net>
Message-ID: <OF2D1E3F5A.4378E2BA-ON80257014.002FA9F3-80257014.002EC441@risk.sungard.com>

Thanks for the reply. 

I deleted my testresult dir and it appears to work now. The other failures 
are still happening, but I guess they will be fixed over time!

Cheers,
Ben


pypy-dev-bounces at codespeak.net wrote on 01/06/2005 17:46:10:

> Hi Ben, 
> 
> On Wed, Jun 01, 2005 at 10:45 +0100, Ben.Young at risk.sungard.com wrote:
> > Hi there,
> > 
> > I follow the pypy development, and tend to sync to head every day or 
so. I 
> > recently noticed some test failures that don't seem to be going away, 
and 
> > was wondering whether there is something wrong with my configuration 
or if 
> > there was something wrong with the way I was running the tests. I 
addition 
> > I noticed that some tests are skipped due to a floating point error. 
Is 
> > this expected too?
> > 
> > I run the tests with python 2.4 on cygwin.
> 
> the one problem relating to test_overview.py is strange: do
> you have a recent checkout of the 'testresult' directory in
> your pypy-dist checkout tree?  it should help to do an 'svn
> up' in that particular directory which is not updated via the
> 'svn up pypy-dist' but only when you explicitely work on
> generating compliance test results. 
> 
> cheers, 
> 
>     holger
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 
> 



From arigo at tunes.org  Thu Jun  2 14:57:07 2005
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 2 Jun 2005 14:57:07 +0200
Subject: [pypy-dev] dev meeting 2nd June 3pm (CEST)
In-Reply-To: <20050529192116.GG5694@solar.trillke.net>
References: <20050529192116.GG5694@solar.trillke.net>
Message-ID: <20050602125707.GA14690@code1.codespeak.net>

Hi,

On Sun, May 29, 2005 at 09:21:16PM +0200, holger krekel wrote:
> - giving the translation efforts a stable base: 
> 
>   One problem with ongoing development of PyPy is that
>   the translation process is still somewhat fragile with
>   respect to type inference (annotation) on our source tree.  
>   Therefore, it would be good if translation could for some time 
>   rely on a non-changing PyPy base.  One simple idea 
>   is to fork off translation work in a branch, although
>   i'd like to avoid that if possible.  For starters, 
>   documentation changes in the branch would not be 
>   immediately seen on the website. Any other _simple_ ideas? 

We implemented the following scheme, inspired by the 'testresult' svn
directory:  there is a distinct svn directory somewhere in 'pypy/branch'
that contains a copy of a known and stable revision of a selection of
directories, namely 'interpreter', 'module' and 'objspace' without
'flow'.  This is to be checked out *inside* pypy/translator/goal.  Then
translate_pypy does some minor hacking of 'pypy.__path__' to let the
directories from the branch override the directories from the complete
working copy.  Instructions on what to checkout exactly are given when
you try to run translate_pypy.py.

The scaled-down "branch" isn't meant for active development, and will
never be merged back into the trunk.  Occasionally, further fixes in the
trunk that are needed to make the annotator happy are merged into the
branch.


A bientot,

Armin.


From hpk at trillke.net  Thu Jun  2 17:36:10 2005
From: hpk at trillke.net (holger krekel)
Date: Thu, 2 Jun 2005 17:36:10 +0200
Subject: [pypy-dev] dev meeting 2nd June 3pm (CEST)
In-Reply-To: <20050529192116.GG5694@solar.trillke.net>
References: <20050529192116.GG5694@solar.trillke.net>
Message-ID: <20050602153610.GJ5694@solar.trillke.net>

Hi again, 

so the meeting is just finished and here are some minutes
and conclusions we reached.  Feel free to comment here on 
the list (the minutes are also checked into 
svn/pypy/extradoc/irclog/). 

cheers, 

    holger


pypy-dev meeting on #pypy at irc.freenode.net 

    Thursday, 2nd June, 3pm (CEST == GMT+2) 

attendees: Armin Rigo, Samuele Pedroni, Carl Friedrich Bolz, 
           Holger Krekel, Anders Chrigstroem, Anders Lehmann, 
           Adrien di Mascio, Stelios Xanthakis, Bert Freudenberg

topics and decisions: 

.. contents::
.. sectnum::

------------------------------------------- 
news/infos 
------------------------------------------- 

Bert Freudenberg introduces himself as a Squeak Developer
since '97 who intends to enter the EU project and to come
to the Post-EuroPython PyPy sprint. 

Stelios Xanthakis (mayall) has been watching PyPy from the sideline. 
He is the author of 'pyvm', a python virtual machine experiment. 

------------------------------------------- 
short-term release planning 
------------------------------------------- 

around 23rd of June we'd like to do a 0.6.2 release 
which includes moving our lib-python base to 2.4.1 and
improved documentation (especially for newcomers). 

------------------------------------------- 
sprint announcement/topics
------------------------------------------- 

although times and city are fixed already, the actual sprint
topics for the post-europython sprint from 1st-7th July (both 
including) July in Goetheborg (Sweden) are not yet decided and
announced.  However, almost all active developers will come and
it's already safe to reserve the time and book your flights 
if you want to attend a newcomer-friendly sprint. 

Sprint topics (Armin has already formulated a more detailed topic 
announcement draft in extradoc/sprintinfo/EP2005-announcement.txt): 

- translation : - rtyper (low-level impl of RPython objects)
                - genc/genllvm (might be well advanced by EP)
- integrate parser module (possibly making it RPython
  conformant) 

- various topics (name some examples from the tracker) 
  depending on people's interests 

------------------------------------------- 
Advancing Issue Tracking: 
------------------------------------------- 

Holger will extend/modify issue tracking so that 

- each issue is classsified as requiring hard, medium or easy efforts 
  (maybe name that 'effort' or something) 

- milestones should go in favour of a release-field which should
  initially contain '0.6.2', '1.0' (mainly reflecting Milestone 1 
  stuff) and 'general' (not yet tied to any release in particular) 

  later we add release values like a possible 0.7 etc.pp. 

------------------------------------------- 
stable base for Translation efforts
------------------------------------------- 
giving the translation efforts a stable base: 

One problem with ongoing development of PyPy is that
the translation process is still somewhat fragile with
respect to type inference (annotation) on our source tree.  

-> resolved already before the meeting: see 
Armin's pypy-dev posting. 

[current approach: pypy/translator/goal contains a stable
snapshot of interpreter, module and objspace directories 
so that main-line changes don't disturb translation work, 
run 'python translate_pypy.py targetpypy1.py' to get interactive
advise]


From arigo at tunes.org  Sat Jun  4 01:43:20 2005
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 4 Jun 2005 01:43:20 +0200
Subject: [pypy-dev] Regular expressions on 2.4
Message-ID: <20050603234320.GA31411@code1.codespeak.net>

Hi,

Regular expressions don't work at all on 2.4, it seems.  We hoped at one
point that the 're' bytecodes would be kind of compatible, but they seem
to be deeply incompatible.  I can't get any match, actually.
re.compile('a').match('a') returns None.  That's worse than the previous
situation, where 'import re' would crash in 2.4...


Armin


From Ben.Young at risk.sungard.com  Mon Jun  6 10:53:13 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 6 Jun 2005 09:53:13 +0100
Subject: [pypy-dev] New genc implementation
Message-ID: <OF0A33BBCA.34D0917A-ON80257018.003070A4-80257018.002FC0E7@risk.sungard.com>

Hi All,

I was interested in playing with the new c translator, and was wondering 
if there was a way of using it yet through the general Translator class, 
or whether it has not been plugged in at this point? Is it possible to use 
it in translate_pypy, even if it doesn't compile? 

Cheers,
Ben


From arigo at tunes.org  Wed Jun  8 14:53:46 2005
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 8 Jun 2005 14:53:46 +0200
Subject: [pypy-dev] New genc implementation
In-Reply-To: <OF0A33BBCA.34D0917A-ON80257018.003070A4-80257018.002FC0E7@risk.sungard.com>
References: <OF0A33BBCA.34D0917A-ON80257018.003070A4-80257018.002FC0E7@risk.sungard.com>
Message-ID: <20050608125346.GA18524@code1.codespeak.net>

Hi Ben,

On Mon, Jun 06, 2005 at 09:53:13AM +0100, Ben.Young at risk.sungard.com wrote:
> I was interested in playing with the new c translator, and was wondering 
> if there was a way of using it yet through the general Translator class, 
> or whether it has not been plugged in at this point? Is it possible to use 
> it in translate_pypy, even if it doesn't compile? 

It is not plugged in yet, indeed.  There is one last missing bit before
it can start replacing the old genc: generating function calls in the
absence of low-level types.  It can only generate them if the RTyper has
done its job.  The RTyper is very incomplete and crashes on most real
examples at the moment, so it's a problem.

I guess I should just take a few hours and finish this -- in other words
make the new generator able to translate large un-annotated examples --
plug it into the translator, and good-bye old genc.


A bientot,

Armin.


From Ben.Young at risk.sungard.com  Wed Jun  8 15:12:18 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Wed, 8 Jun 2005 14:12:18 +0100
Subject: [pypy-dev] New genc implementation
In-Reply-To: <20050608125346.GA18524@code1.codespeak.net>
Message-ID: <OF42CA0F26.B48762DF-ON8025701A.004830E9-8025701A.004778EA@risk.sungard.com>

Thanks, that was what I was interested in. I was also interested in the 
new ScopedBlock that was going to be introduce to help with all the 
variable movement. Do you happen to know how that is progressing?

Keep up the good work! I noticed that targetrpython2.py is 20 time faster 
than python now. Only 100 times more to go!

Cheers,
Ben

Armin Rigo <arigo at tunes.org> wrote on 08/06/2005 13:53:46:

> Hi Ben,
> 
> On Mon, Jun 06, 2005 at 09:53:13AM +0100, Ben.Young at risk.sungard.com 
wrote:
> > I was interested in playing with the new c translator, and was 
wondering 
> > if there was a way of using it yet through the general Translator 
class, 
> > or whether it has not been plugged in at this point? Is it possible to 
use 
> > it in translate_pypy, even if it doesn't compile? 
> 
> It is not plugged in yet, indeed.  There is one last missing bit before
> it can start replacing the old genc: generating function calls in the
> absence of low-level types.  It can only generate them if the RTyper has
> done its job.  The RTyper is very incomplete and crashes on most real
> examples at the moment, so it's a problem.
> 
> I guess I should just take a few hours and finish this -- in other words
> make the new generator able to translate large un-annotated examples --
> plug it into the translator, and good-bye old genc.
> 
> 
> A bientot,
> 
> Armin.
> 



From Ben.Young at risk.sungard.com  Thu Jun  9 11:08:46 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Thu, 9 Jun 2005 10:08:46 +0100
Subject: [pypy-dev] Re: [pypy-svn] r13221 - in pypy/dist/pypy/rpython: . test
In-Reply-To: <20050609085332.783A327B6D@code1.codespeak.net>
Message-ID: <OF221A1C6E.363AB5F0-ON8025701B.0031C944-8025701B.00318641@risk.sungard.com>

Hi Arigo,

Looking at these commits, I just have a quick question. Can the ll_ 
functions call each other? If they could then all the ll_delitem functions 
could just delagate to the noneg versions. Also, does the bounds checking 
happen at this level, or at a lower one.

In a week or so I may have a little time to do a bit of coding. Is there 
anything specific I should look at? For instance I noticed that at the 
moment the list rep resizes exactly. The python version (and stuff like 
std::vector in c++) have a overallocation stratagy thay minimises 
allocation calls. Would this be an idea I could try to implement in 
rpython?

Cheers,
Ben

pypy-svn-bounces at codespeak.net wrote on 09/06/2005 09:53:32:

> Author: arigo
> Date: Thu Jun  9 10:53:30 2005
> New Revision: 13221
> 
> Modified:
>    pypy/dist/pypy/rpython/rlist.py
>    pypy/dist/pypy/rpython/test/test_rlist.py
> Log:
> list.delitem; tests.
> 
> 
> Modified: pypy/dist/pypy/rpython/rlist.py
> 
==============================================================================
> --- pypy/dist/pypy/rpython/rlist.py   (original)
> +++ pypy/dist/pypy/rpython/rlist.py   Thu Jun  9 10:53:30 2005
> @@ -88,6 +88,14 @@
>              llfn = ll_setitem
>          return hop.gendirectcall(llfn, v_lst, v_index, v_item)
> 
> +    def rtype_delitem((r_lst, r_int), hop):
> +        v_lst, v_index = hop.inputargs(r_lst, Signed)
> +        if hop.args_s[1].nonneg:
> +            llfn = ll_delitem_nonneg
> +        else:
> +            llfn = ll_delitem
> +        return hop.gendirectcall(llfn, v_lst, v_index)
> +
>  class __extend__(pairtype(ListRepr, SliceRepr)):
> 
>      def rtype_getitem((r_lst, r_slic), hop):
> @@ -143,13 +151,39 @@
>          i += len(l.items)
>      return l.items[i].item
> 
> +def ll_setitem_nonneg(l, i, newitem):
> +    l.items[i].item = newitem
> +
>  def ll_setitem(l, i, newitem):
>      if i<0:
>          i += len(l.items)
>      l.items[i].item = newitem
> 
> -def ll_setitem_nonneg(l, i, newitem):
> -    l.items[i].item = newitem
> +def ll_delitem(l, i):
> +    if i < 0:
> +        i += len(l.items)
> +    newlength = len(l.items) - 1
> +    newitems = malloc(typeOf(l).TO.items.TO, newlength)
> +    j = 0
> +    while j < i:
> +        newitems[j].item = l.items[j].item
> +        j += 1
> +    while j < newlength:
> +        newitems[j].item = l.items[j+1].item
> +        j += 1
> +    l.items = newitems
> +
> +def ll_delitem_nonneg(l, i):
> +    newlength = len(l.items) - 1
> +    newitems = malloc(typeOf(l).TO.items.TO, newlength)
> +    j = 0
> +    while j < i:
> +        newitems[j].item = l.items[j].item
> +        j += 1
> +    while j < newlength:
> +        newitems[j].item = l.items[j+1].item
> +        j += 1
> +    l.items = newitems
> 
>  def ll_concat(l1, l2):
>      len1 = len(l1.items)
> 
> Modified: pypy/dist/pypy/rpython/test/test_rlist.py
> 
==============================================================================
> --- pypy/dist/pypy/rpython/test/test_rlist.py   (original)
> +++ pypy/dist/pypy/rpython/test/test_rlist.py   Thu Jun  9 10:53:30 2005
> @@ -30,6 +30,21 @@
>      assert ll_len(l) == 4
>      check_list(l, [42, 43, 44, 45])
> 
> +def test_rlist_set_del():
> +    l = sample_list()
> +    ll_setitem(l, -1, 99)
> +    check_list(l, [42, 43, 44, 99])
> +    ll_setitem_nonneg(l, 1, 77)
> +    check_list(l, [42, 77, 44, 99])
> +    ll_delitem_nonneg(l, 0)
> +    check_list(l, [77, 44, 99])
> +    ll_delitem(l, -2)
> +    check_list(l, [77, 99])
> +    ll_delitem(l, 1)
> +    check_list(l, [77])
> +    ll_delitem(l, 0)
> +    check_list(l, [])
> +
>  def test_rlist_extend_concat():
>      l = sample_list()
>      ll_extend(l, l)
> @@ -110,4 +125,13 @@
>      def dummyfn():
>          l = [5, 6, 7, 8, 9]
>          return l[:2], l[1:4], l[3:]
> -    rtype(dummyfn).view()
> +    rtype(dummyfn)
> +
> +def test_set_del_item():
> +    def dummyfn():
> +        l = [5, 6, 7]
> +        l[1] = 55
> +        l[-1] = 66
> +        del l[0]
> +        del l[-1]
> +    rtype(dummyfn)
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
> 



From Ben.Young at risk.sungard.com  Thu Jun  9 11:10:44 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Thu, 9 Jun 2005 10:10:44 +0100
Subject: [pypy-dev] Fw: [pypy-svn] r13221 - in pypy/dist/pypy/rpython: . test
Message-ID: <OFEA8F910F.C6538E78-ON8025701B.0032503D-8025701B.0031B46C@risk.sungard.com>

(sorry if this appears twice, used wrong reply-to header)

Hi Arigo,

Looking at these commits, I just have a quick question. Can the ll_ 
functions call each other? If they could then all the ll_delitem functions 
could just delagate to the noneg versions. Also, does the bounds checking 
happen at this level, or at a lower one.

In a week or so I may have a little time to do a bit of coding. Is there 
anything specific I should look at? For instance I noticed that at the 
moment the list rep resizes exactly. The python version (and stuff like 
std::vector in c++) have a overallocation stratagy thay minimises 
allocation calls. Would this be an idea I could try to implement in 
rpython?

Cheers,
Ben

pypy-svn-bounces at codespeak.net wrote on 09/06/2005 09:53:32:

> Author: arigo
> Date: Thu Jun  9 10:53:30 2005
> New Revision: 13221
> 
> Modified:
>    pypy/dist/pypy/rpython/rlist.py
>    pypy/dist/pypy/rpython/test/test_rlist.py
> Log:
> list.delitem; tests.
> 
> 
> Modified: pypy/dist/pypy/rpython/rlist.py
> 
==============================================================================
> --- pypy/dist/pypy/rpython/rlist.py   (original)
> +++ pypy/dist/pypy/rpython/rlist.py   Thu Jun  9 10:53:30 2005
> @@ -88,6 +88,14 @@
>              llfn = ll_setitem
>          return hop.gendirectcall(llfn, v_lst, v_index, v_item)
> 
> +    def rtype_delitem((r_lst, r_int), hop):
> +        v_lst, v_index = hop.inputargs(r_lst, Signed)
> +        if hop.args_s[1].nonneg:
> +            llfn = ll_delitem_nonneg
> +        else:
> +            llfn = ll_delitem
> +        return hop.gendirectcall(llfn, v_lst, v_index)
> +
>  class __extend__(pairtype(ListRepr, SliceRepr)):
> 
>      def rtype_getitem((r_lst, r_slic), hop):
> @@ -143,13 +151,39 @@
>          i += len(l.items)
>      return l.items[i].item
> 
> +def ll_setitem_nonneg(l, i, newitem):
> +    l.items[i].item = newitem
> +
>  def ll_setitem(l, i, newitem):
>      if i<0:
>          i += len(l.items)
>      l.items[i].item = newitem
> 
> -def ll_setitem_nonneg(l, i, newitem):
> -    l.items[i].item = newitem
> +def ll_delitem(l, i):
> +    if i < 0:
> +        i += len(l.items)
> +    newlength = len(l.items) - 1
> +    newitems = malloc(typeOf(l).TO.items.TO, newlength)
> +    j = 0
> +    while j < i:
> +        newitems[j].item = l.items[j].item
> +        j += 1
> +    while j < newlength:
> +        newitems[j].item = l.items[j+1].item
> +        j += 1
> +    l.items = newitems
> +
> +def ll_delitem_nonneg(l, i):
> +    newlength = len(l.items) - 1
> +    newitems = malloc(typeOf(l).TO.items.TO, newlength)
> +    j = 0
> +    while j < i:
> +        newitems[j].item = l.items[j].item
> +        j += 1
> +    while j < newlength:
> +        newitems[j].item = l.items[j+1].item
> +        j += 1
> +    l.items = newitems
> 
>  def ll_concat(l1, l2):
>      len1 = len(l1.items)
> 
> Modified: pypy/dist/pypy/rpython/test/test_rlist.py
> 
==============================================================================
> --- pypy/dist/pypy/rpython/test/test_rlist.py   (original)
> +++ pypy/dist/pypy/rpython/test/test_rlist.py   Thu Jun  9 10:53:30 2005
> @@ -30,6 +30,21 @@
>      assert ll_len(l) == 4
>      check_list(l, [42, 43, 44, 45])
> 
> +def test_rlist_set_del():
> +    l = sample_list()
> +    ll_setitem(l, -1, 99)
> +    check_list(l, [42, 43, 44, 99])
> +    ll_setitem_nonneg(l, 1, 77)
> +    check_list(l, [42, 77, 44, 99])
> +    ll_delitem_nonneg(l, 0)
> +    check_list(l, [77, 44, 99])
> +    ll_delitem(l, -2)
> +    check_list(l, [77, 99])
> +    ll_delitem(l, 1)
> +    check_list(l, [77])
> +    ll_delitem(l, 0)
> +    check_list(l, [])
> +
>  def test_rlist_extend_concat():
>      l = sample_list()
>      ll_extend(l, l)
> @@ -110,4 +125,13 @@
>      def dummyfn():
>          l = [5, 6, 7, 8, 9]
>          return l[:2], l[1:4], l[3:]
> -    rtype(dummyfn).view()
> +    rtype(dummyfn)
> +
> +def test_set_del_item():
> +    def dummyfn():
> +        l = [5, 6, 7]
> +        l[1] = 55
> +        l[-1] = 66
> +        del l[0]
> +        del l[-1]
> +    rtype(dummyfn)
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
> 



From hpk at trillke.net  Thu Jun  9 15:01:46 2005
From: hpk at trillke.net (holger krekel)
Date: Thu, 9 Jun 2005 15:01:46 +0200
Subject: [pypy-dev] planning the 0.6.2 PyPy release
Message-ID: <20050609130146.GU21207@solar.trillke.net>

Hi pypy-devers, 

at our 2nd June IRC developer meeting (minutes were posted on this list) 
we decided that we aim for a new micro release 0.6.2 around 23rd June. 
Yesterday the issue tracker has been reformed to help more with
sorting through the various issues we have to resolve in the next
time.  The current 0.6.2 issues are linked from the issue tracker home page
on the left side (you'll get nicer nicer views if you register/login): 

    https://codespeak.net/issue/pypy-dev/

You'll note that the focus of the 0.6.2 release is solely
improved documentation and then move to utilize CPython's
2.4.1 standard library, regression tests and semantics.  The
idea here is that PyPy development will then rely on CPython
2.4 for at least a year.  It is not clear yet how feasible it
will be to have PyPy run as 2.4.1 compatible but on top of
CPython 2.3.  It's certainly desirable. 

Many of you know that the main effort with PyPy at the moment is
translation to low-level code.  Lead by Armin and Samuele with 
a lot of help from Carl, Eric and Christian the efforts there are 
continously moving forward (it would be nice to get some kind of 
status mail some time, btw). 

However, the 0.6.2 release will not have any translation goals
associated with it although we will need to make sure that the
getting-started examples still work.  Because i am going to
Goetheborg around 22nd i suggest to declare 

    15th till 21st June 
   
the 0.6.2 release time and ask everyone to reserve a bit of time 
for helping out, possibly not waiting until the evening of 21st June :-) 
Assign a task to you if you know already what you can try to work on. 

If you have any suggestions or comments, please post them! 

cheers, 

    holger


From arigo at tunes.org  Thu Jun  9 16:31:27 2005
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 9 Jun 2005 16:31:27 +0200
Subject: [pypy-dev] New genc implementation
In-Reply-To: <OF42CA0F26.B48762DF-ON8025701A.004830E9-8025701A.004778EA@risk.sungard.com>
References: <20050608125346.GA18524@code1.codespeak.net>
	<OF42CA0F26.B48762DF-ON8025701A.004830E9-8025701A.004778EA@risk.sungard.com>
Message-ID: <20050609143127.GA3165@code1.codespeak.net>

Hi Ben,

On Wed, Jun 08, 2005 at 02:12:18PM +0100, Ben.Young at risk.sungard.com wrote:
> Thanks, that was what I was interested in. I was also interested in the 
> new ScopedBlock that was going to be introduce to help with all the 
> variable movement. Do you happen to know how that is progressing?

A few different ideas have been floating around -- nothing concrete so
far.  An idea that was not posted here (which I guess will triggers'
Christian attention) is to try, instead of rebuilding variable scopes,
to try to minimize the number of different variables.  It is actually
not difficult to compute a dictionary that maps some variables to some
other variables, as much as possible, while still avoiding conflicts.
This would definitely bring down the number of distinct variables and
explicit moves, and even quite a lot.  It might be sufficient to get the
same benefits as the ScopedBlock analysis.

> Keep up the good work! I noticed that targetrpython2.py is 20 time faster 
> than python now. Only 100 times more to go!

Good :-)


A bientot,

Armin


From arigo at tunes.org  Thu Jun  9 22:36:47 2005
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 9 Jun 2005 22:36:47 +0200
Subject: [pypy-dev] New genc implementation
In-Reply-To: <20050608125346.GA18524@code1.codespeak.net>
References: <OF0A33BBCA.34D0917A-ON80257018.003070A4-80257018.002FC0E7@risk.sungard.com>
	<20050608125346.GA18524@code1.codespeak.net>
Message-ID: <20050609203647.GA7694@code1.codespeak.net>

Hi all,

On Wed, Jun 08, 2005 at 02:53:46PM +0200, Armin Rigo wrote:
> It is not plugged in yet, indeed.  There is one last missing bit before
> it can start replacing the old genc: generating function calls in the
> absence of low-level types.

We have done the switch now.  The old genc should no longer be
automatically used.  (It's still there because its tests are good tests
for -- confusingly -- the new 'c' translator which they now happen to
run with.)

The only test that still has failures is
translator/genc/test/test_typed.py, which has been renamed to
"inprogress_..." for now.  Feel free to run it manually to see what's
missing.

Also, we might have missed things that breaks horribly (a dist/demo/ maybe?).


Armin


From arigo at tunes.org  Fri Jun 10 00:40:16 2005
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 10 Jun 2005 00:40:16 +0200
Subject: [pypy-dev] Fw: [pypy-svn] r13221 - in pypy/dist/pypy/rpython: .
	test
In-Reply-To: <OFEA8F910F.C6538E78-ON8025701B.0032503D-8025701B.0031B46C@risk.sungard.com>
References: <OFEA8F910F.C6538E78-ON8025701B.0032503D-8025701B.0031B46C@risk.sungard.com>
Message-ID: <20050609224016.GA9978@code1.codespeak.net>

Hi Ben,

On Thu, Jun 09, 2005 at 10:10:44AM +0100, Ben.Young at risk.sungard.com wrote:
> Looking at these commits, I just have a quick question. Can the ll_ 
> functions call each other?

Not right now, but of course it's clearly something that would be
useful.  We have an idea on how to enable it but it's not done yet.

> Also, does the bounds checking happen at this level, or at a lower one.

The bounds checking doesn't happen at all :-)  If we wanted to make them
we would have to write them at this level.  But the plan is to make sure
(by testing) that the source code of PyPy doesn't do out-of-bounds
accesses, and then just don't do the checks any more.  Well, I guess it
would actually be useful to have a "debug" flag around so that we can
enable or disable the bound checks...  (Note that a global variable
"debug=True" or "debug=False" that we test in the ll_ functions would do
the trick -- same as Eric's, in rint.py and rfloat.py, but tested from
inside the ll_ functions.)

> In a week or so I may have a little time to do a bit of coding. Is there 
> anything specific I should look at? For instance I noticed that at the 
> moment the list rep resizes exactly. The python version (and stuff like 
> std::vector in c++) have a overallocation stratagy thay minimises 
> allocation calls. Would this be an idea I could try to implement in 
> rpython?

An excellent question!  It's slightly strange, given that
pypy.objspace.std.listobject already does over-allocation, that we might
need it again at this lower level.  It's possible that it would make
sense for some of the lists that we have around, and not for others --
the hard question will eventually be to decide when to use it or not.
But yes, I guess that before deciding that, we'd need to have both
options written :-)  (There are still some open questions there; for
example, it's not too difficult to detect list comprehensions by looking
at the flow graphs; then instead of heuristic over-allocation for the
list that is being built, it's often possible to determine in advance
the size of the list -- e.g. because it will be the same length as
another list that we iterate over.)

Another direction we'll need soon: dictionaries -- we really need them
to translate PyPy, even in a limited fashion.  We don't use a lot of
dicts at all (there are 7 creation points in total in PyPy!), but well,
we need them still; these 7 uses are quite important ones.  They can be
divided in two categorizes: immutable prebuilt dictionaries that
basically use the key object's identity (i.e. whose keys have no __cmp__
or __eq__), and mutable dictionaries with string keys.  We'll probably
need two implementations, one for each case.  The str-keyed dict raises
all the usual questions: do we follow CPython's strategy and use a hash
table with interned strings?  Or do we use some other data structure?
(Note that app-level dicts don't use rdict; that's mainly for passing
keywords around and for the __dict__ of type objects.)

Finally, there is always the need for more low-level operations as well
as low-level versions of some builtins.  The rtyper is not complete as
long as the annotator supports more operations and built-in functions
than it (as seen in annotation.unaryop, annotation.binaryop and
annotation.builtin).


A bientot,

Armin


From faassen at infrae.com  Fri Jun 10 11:40:35 2005
From: faassen at infrae.com (Martijn Faassen)
Date: Fri, 10 Jun 2005 11:40:35 +0200
Subject: [pypy-dev] planning the 0.6.2 PyPy release
In-Reply-To: <20050609130146.GU21207@solar.trillke.net>
References: <20050609130146.GU21207@solar.trillke.net>
Message-ID: <42A96013.8020503@infrae.com>

holger krekel wrote:
> Many of you know that the main effort with PyPy at the moment is
> translation to low-level code.  Lead by Armin and Samuele with 
> a lot of help from Carl, Eric and Christian the efforts there are 
> continously moving forward (it would be nice to get some kind of 
> status mail some time, btw). 

A status mail on this would be much appreciated by this particular 
lurker. :)

Regards,

Martijn


From arigo at tunes.org  Fri Jun 10 12:54:19 2005
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 10 Jun 2005 12:54:19 +0200
Subject: [pypy-dev] planning the 0.6.2 PyPy release
In-Reply-To: <20050609130146.GU21207@solar.trillke.net>
References: <20050609130146.GU21207@solar.trillke.net>
Message-ID: <20050610105419.GA14494@code1.codespeak.net>

Hi!

Here is our status.  The translation is progressing, a lot remains to be
done.  At this point, given the immediate goals, we can say that:

* The low-level types' model is supposedly stable, but actually still
  undergoes small changes.  For example, we added arrays of primitive
  types, because it makes the ll_ functions saner (e.g. we can implement
  strings as arrays of characters instead of arrays of structures that
  each contain a single character).  We plan another update soon: the
  possibility to attach custom deallocators to GcStructs.  And maybe --
  possibly -- later it would be convenient to have arrays of fixed size
  (e.g. arrays of 4 integers).

* The 'translator/c/' code is mostly finished.  It accepts all the
  low-level types and operations and produces rather inefficient C code
  out of it, but that's fine for the time being.

* The focus is on the rtyper (pypy/rpython/r*.py).  A low-level flow
  graph interpreter would be very useful for debugging at this point,
  and it is the kind of thing that neither Samuele nor myself should do
  because it is a good and easy introduction to lltypes.  So let me push
  this forward again.

  We're working on framework things like being able to call ll_
  functions from other ll_ functions.  For the status of what's there
  and missing, try running the test file
  pypy/translator/genc/test/inprogress_test_typed.py shows some
  limitations of the rtyper that we are working on.  In general, we
  have:

  - int, float, bool (thanks Eric)
  - the basics of lists including slices, ranges, iterators
  - tuples
  - the basics of strings and characters
  - very few built-in functions
  - a good enough class and instance model
  - very preliminary support for function pointers

  We need everything else that the annotator supports, e.g.:

  - (easy) unicode characters
  - (hard) prebuilt constants -- PBCs -- and dicts with PBC keys
  - (medium) dicts with string keys, and iterators over them
  - (easy) iterators over strings
  - (medium) iterators over tuples
  - (varying) all built-in functions listed in annotation/builtin.py
  - (mostly easy) a few more operations: id, issubtype, str(?), is
  - (easy) add_ovf --> int_add_ovf
  - (easy) list methods reverse, pop, insert, index(?)
  - (easy) tuple concatenation
  - (easy) string methods startswith(1-arg), endswith(1-arg),
           join (possibly only on the empty string ''), string slices
  - (easy) equality and non-equality are generally missing
  - (medium) limited form of string formatting: 'constant template' only
             which should only generate a sequence of simple operations
             like concatenation and str() on integers


A bientot,

Armin.


From arigo at tunes.org  Sun Jun 12 23:10:29 2005
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 12 Jun 2005 23:10:29 +0200
Subject: [pypy-dev] Post-EuroPython 2005 PyPy Sprint 1st - 7th July 2005
Message-ID: <20050612211029.GA26015@code1.codespeak.net>

Post-EuroPython 2005 PyPy Sprint 1st - 7th July 2005 
======================================================

The next PyPy sprint is scheduled right after EuroPython 2005
in Gothenborg, Sweden.  Its main focus is translation to 
lower level backends but there are also other possible topics. 
We'll give newcomer-friendly introductions.  To learn more
about the new PyPy Python-in-Python implementation look here: 

    http://codespeak.net/pypy 

On a side note, there are a number of sub projects that may be
interesting for participating in google's summer-of-code event
(deadline June 14th!).  The PyPy group is willing to mentor
projects that have some link with PyPy, so if you are accepted
in such a project, the sprint could also serve as a good
meeting and kick-off point.  Further down you'll find some
examples, but there are certainly more and bigger ones :-)

Goals and topics of the sprint 
------------------------------

The main, though not the only, focus of the sprint will be on
the "translation" aspect of PyPy.   The goal here is to
progress towards a completely translated PyPy.  How much will
already have been done before EuroPython is unknown; as a
guess, we will be left with:
 
- completing the "rtyper", the piece of code that assigns low-level
  C-like types to high-level RPython objects (lists, dicts, instances,
  etc.) and low-level control flow graphs to high-level ones;
   
- polish off the GenC and GenLLVM back-ends, responsible for turning
  the low-level C-like flow graphs into real C or LLVM source code.

See http://codespeak.net/pipermail/pypy-dev/2005q2/002136.html for more
information (10th of June status).
 
Non-translation-related topics are welcome too.  Here are some suggestions
from the issue tracker (https://codespeak.net/issue/pypy-dev/):
   
- integrate the parser module, possibly making it RPython
  conformant;
 
- rewrite in Python a C module you are familiar with
  (partial list of missing/incomplete modules: os, math, array,
  regular expressions, binascii...)

- implement Python 2.3's import hook extensions (zip-imports etc.)

- fix Windows-related issues, '%'-formatting rounding errors,
  add missing docstrings on app-level built-in types and functions,
  etc.

- weakrefs (but this requires discussion and planning on pypy-dev
  before the sprint! feel free to start such a discussion, though.)


Location & Accomodation  
------------------------ 

The sprint will be held in the former Math Center building
near the crossing of Gibraltargatan and Eklandagatan. Entrance
is on the middle of the side facing Gibraltargatan. The doors
to the building are normally locked, so you need the phone number
of somebody inside to get in. Instructions on whom to call will be
posted on the door.

The sprint will be co-located with several other sprints. See the 
`EuroPython Wiki`_, to find out what other sprints will be running.

Nearest, and probably cheapest is to book accomodation at SGS Veckobost??der
through the Europython website. This option will be available until about
20 June.

.. _`EuroPython special accomodation`: http://www.europython.org/sections/accomodation/special_accomodation
.. _`EuroPython Wiki`: http://www.europython.org/sections/sprints_and_wiki

Exact times 
-----------

The public Pypy sprint is held Friday 1st July - Thursday 7 July 2005.
Hours will be from 09:00 until people have had enough. It's a good 
idea to arrive a day before the sprint starts.   

(There is a sprint for people who are familiar with the Pypy codebase
before Europython as well. This will be held at Jacob & Laura's home
on G??tabergsgatan 22.)


Network, Food, currency 
------------------------ 

Sweden is not part of the Euro zone. One SEK (krona in singular, kronor
in plural) is roughly 1/10th of a Euro (9.15 SEK to 1 Euro).

There are some pizzerias, kebab places and the like close to the venue.
Their food is edible and cheap, but not very good. For good food, you need to
go downtown.

You need a wireless network card to access the network. You will be 
issued a login to the Chalmers NOMAD network. This will allow you to 
use access points all over Chalmers.  However, we can likely provide 
a wireless/ethernet bridge.  

Sweden uses the same kind of plugs as Germany. 230V AC.

Registration etc.pp. 
-------------------- 

Please subscribe to the `PyPy sprint mailing list`_, introduce
yourself and post a note that you want to come.  Feel free
to ask any questions there! 

.. _`PyPy sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint


-- 
Armin Rigo & the PyPy team



From sanxiyn at gmail.com  Mon Jun 13 03:28:56 2005
From: sanxiyn at gmail.com (Sanghyeon Seo)
Date: Mon, 13 Jun 2005 10:28:56 +0900
Subject: [pypy-dev] Re: [pypy-svn] r13328 - pypy/dist/pypy/translator/test
In-Reply-To: <20050612235024.A473327B50@code1.codespeak.net>
References: <20050612235024.A473327B50@code1.codespeak.net>
Message-ID: <5b024817050612182816d16524@mail.gmail.com>

On 6/13/05, arigo at codespeak.net <arigo at codespeak.net> wrote:
> +py.test.skip("the Translator and back-ends depend on too many conditions "
> +             "to test the back-ends generically")
> +# e.g. some back-ends require annotation and/or specialization to be done
> +#      while some require it *not* to be done.  GenCL for example has no
> +#      chance to understand a specialized graph.  (That's why the test still
> +#      used to pass: forty_two is basically an empty function, so
> +#      specializing it makes no difference.  Now the RTyper produces a few
> +#      helper functions in all cases.)

I am fine with getting rid of current GenCL if it blocks any of PyPy's
development.
Pity I don't have enough free time for PyPy these days.

Seo Sanghyeon


From michal at sabren.com  Mon Jun 13 05:15:30 2005
From: michal at sabren.com (Michal Wallace)
Date: Sun, 12 Jun 2005 23:15:30 -0400 (EDT)
Subject: [pypy-dev] pirate list
Message-ID: <Pine.LNX.4.62.0506122309390.9583@hydrogen.sabren.com>


Hello, PyPy list :)

I've just started a new list for pirate,
a python compiler for the parrot virtual 
machine.

Pirate is written in python and currently
uses the standard compiler package. However,
as parrot supports native types, I would like
to refactor it so that it can also work with
PyPy's type annotation system (and possibly 
integrate with pypy in other ways)

The pirate website is here:

    http://pirate.tangentcode.com/

Quite a bit of work has been done on pirate 
since I last updated the website back in 2003, 
mostly by Sam Ruby of http://intertwingly.net/

Sam is giving a python-on-parrot presentation at
OSCON on August 4, so I'd like to get a few things
cleaned up before then - for example, updating the
site, creating a development roadmap, an fixing
the code to work with the latest version of
parrot, etc.

If you're interested in helping out with any of
these things, or just want to talk about pirate,
then feel free to hop on board. The signup page
is here:

   http://cornerhost.com/mailman/listinfo/pirate

Thanks!

Sincerely,

Michal J Wallace
Sabren Enterprises, Inc.
-------------------------------------
contact: michal at sabren.com
hosting: http://www.cornerhost.com/
my site: http://www.withoutane.com/
-------------------------------------



From Ben.Young at risk.sungard.com  Mon Jun 13 11:06:15 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 13 Jun 2005 10:06:15 +0100
Subject: [pypy-dev] planning the 0.6.2 PyPy release
Message-ID: <OF869E222A.C839C177-ON8025701F.0031A006-8025701F.00314839@risk.sungard.com>

Hi Armin

Just a quick followup question....

>> In a week or so I may have a little time to do a bit of coding. Is 
there 
>> anything specific I should look at? For instance I noticed that at the 
>> moment the list rep resizes exactly. The python version (and stuff like 

>> std::vector in c++) have a overallocation stratagy thay minimises 
>> allocation calls. Would this be an idea I could try to implement in 
>> rpython?

>An excellent question!  It's slightly strange, given that
>pypy.objspace.std.listobject already does over-allocation, that we might
>need it again at this lower level.  It's possible that it would make
>sense for some of the lists that we have around, and not for others --
>the hard question will eventually be to decide when to use it or not.
>But yes, I guess that before deciding that, we'd need to have both
>options written :-)  (There are still some open questions there; for
>example, it's not too difficult to detect list comprehensions by looking
>at the flow graphs; then instead of heuristic over-allocation for the
>list that is being built, it's often possible to determine in advance
>the size of the list -- e.g. because it will be the same length as
>another list that we iterate over.)

Does this mean that when you eventually come to *dynamically* annotate 
code that lists will be an annotated version of the std.listobject, or do 
you expect it to use the rpython typer as well?

Thanks for answering all these questions, it definitely is making things 
clearer in my mind.

Cheers,
Ben


From arigo at tunes.org  Mon Jun 13 13:54:26 2005
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 13 Jun 2005 13:54:26 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13328 - pypy/dist/pypy/translator/test
In-Reply-To: <5b024817050612182816d16524@mail.gmail.com>
References: <20050612235024.A473327B50@code1.codespeak.net>
	<5b024817050612182816d16524@mail.gmail.com>
Message-ID: <20050613115426.GA2358@code1.codespeak.net>

Hi Seo,

On Mon, Jun 13, 2005 at 10:28:56AM +0900, Sanghyeon Seo wrote:
> I am fine with getting rid of current GenCL if it blocks any of PyPy's
> development.

It doesn't really block us.  I think it's fine to spend a little time
maintaining the old back-ends.  In this case, it just shows that all
back-ends have subtly different requirements and it's getting worse;
GenPyrex or the java back-end for example don't have any clue about
fully-specialized flow graphs either.

Some kind of solution for this problem would involve turning the
Translator into some kind of 'make' engine.  The various "phases" would
be described with dependencies: getting the flow graph, performing
various simplifications, annotating, various more simplifications based
on the annotations, typing, generating code, compiling.  This needs to
be fine-grained enough, because for example the typer creates a few new
graphs which must again be annotated and simplified in a precise way.
Internal phases of the annotator or the typer would also benefit from
being described in this way.  The user-visible goal would be that trying
to call a method on the Translator (e.g.  llvmcompile()) would figure
out which phases must be run, or complain if too many phases have
already been run, instead of crashing in mysterious ways if too little
or too much has been done before.


A bientot,

Armin


From arigo at tunes.org  Mon Jun 13 15:08:40 2005
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 13 Jun 2005 15:08:40 +0200
Subject: [pypy-dev] planning the 0.6.2 PyPy release
In-Reply-To: <OF869E222A.C839C177-ON8025701F.0031A006-8025701F.00314839@risk.sungard.com>
References: <OF869E222A.C839C177-ON8025701F.0031A006-8025701F.00314839@risk.sungard.com>
Message-ID: <20050613130840.GB2358@code1.codespeak.net>

Hi Ben,

On Mon, Jun 13, 2005 at 10:06:15AM +0100, Ben.Young at risk.sungard.com wrote:
> >    (There are still some open questions there; for
> >example, it's not too difficult to detect list comprehensions by looking
> >at the flow graphs; then instead of heuristic over-allocation for the
> >list that is being built, it's often possible to determine in advance
> >the size of the list -- e.g. because it will be the same length as
> >another list that we iterate over.)
> 
> Does this mean that when you eventually come to *dynamically* annotate 
> code that lists will be an annotated version of the std.listobject, or do 
> you expect it to use the rpython typer as well?

The dynamic annotation part is still unclear, but I suppose that it will
generally be done at a lower level than std.listobject.  The goal is to
keep it mostly independent on the language being interpreted/compiled,
and only dependent on RPython.

If we start with an approach based on Psyco, it might even not use the
annotator/rtyper directly: just like we obtain a CPython-like piece of C
code by translating the rtyped flow graphs of PyPy into C, we would
obtain a Psyco-like piece of code by translating the same rtyped flow
graphs of PyPy into something different -- i.e. the JIT would work at
the level of low-level operations (getfield, setfield...) which is more
concrete even than RPython.

Assuming for now, for simplicity, that we don't worry about the time the
JIT takes to compile code, we would obtain the following diagram (we're
working on the rtyper and genc parts now; anything above is basically
done, and the rest some potential draft plan):


  PyPy interpreter ------------> flow graphs

            -----annotation----> annotated flow graphs

            ------rtyper-------> low-level flow graphs

                                  /        \
                           genc  /          \
                                /            \ genpickle-like
                              |/              \
                              `-               \
                        C/LLVM code             \|
     user app----------[interpreter]            -'
     (without JIT)                    file containing compact data
                                      describing the ll flow graph
                                                   |
                                                   | input file
                                                   V
     user app-----------------------[flow graph abstract interpreter]
     (with JIT)                            (manually written)


The key component in the JIT part is the flow graph abstract
interpreter.  Its role is to take a specific, frozen flow graph as input
file (which is the flow graph of PyPy) and interpret it.  Doing so
naively would just interpret PyPy interpreting the user program, so it
would be slow again -- double interpretation, even if the flow graph
interpreter itself can probably be quite fast because the flow graphs at
this point contain very simple low-level operations.  Instead, the flow
graph interpreter is "abstract", which means that it will compute and
propagate some annotations on its own.  These annotations are much
simpler than the ones in the current annotator, because they are about
low-level types; they would look like "the field 'ob_type' of this
GcStruct here is known to contain such-and-such constant value", or
"this GcStruct doesn't really have to be allocated on the heap so far
because we can keep its few non-constant fields in local variables".
This process is dynamic because it follows PyPy following the user
application.  For example, if we say that all the data structures
corresponding in PyPy to the PyCode objects are completely constant
(with values that comes from the user program -- they are the constant
code objects present in the user program) then all the bytecode decoding
and dispatching logic done by PyPy (present in the flow graphs of PyPy)
is constant, computable in advance as annotations.

Using internally a process similar to annotation / rtyper / code
generator, but much simpler thanks to the low-level types, the flow
graph abstract interpreter would be able to generate dynamically new
flow graphs that are simplified versions of the input flow graphs, where
only the operations between things that couldn't be annotated as
constants are kept.

This is essentially how Psyco works.  I am not sure about the details or
how to integrate it better with the existing annotation framework
(which, with more efforts, would give interesting hints about user
applications too, instead of just about PyPy).  I am not sure if any of
this will be done as I describe above, actually. :-)  What I am
reasonably sure is only that it *could* be done like this.


A bientot,

Armin.


From Ben.Young at risk.sungard.com  Mon Jun 13 15:34:47 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 13 Jun 2005 14:34:47 +0100
Subject: [pypy-dev] Re: [pypy-svn] r13328 - pypy/dist/pypy/translator/test
In-Reply-To: <20050613115426.GA2358@code1.codespeak.net>
Message-ID: <OFAEDEF1EC.B342E15D-ON8025701F.0049DDCB-8025701F.0049DDC3@risk.sungard.com>

Hi Armin

pypy-dev-bounces at codespeak.net wrote on 13/06/2005 12:54:26:

> Hi Seo,
> 
> On Mon, Jun 13, 2005 at 10:28:56AM +0900, Sanghyeon Seo wrote:
> > I am fine with getting rid of current GenCL if it blocks any of PyPy's
> > development.
> 
> It doesn't really block us.  I think it's fine to spend a little time
> maintaining the old back-ends.  In this case, it just shows that all
> back-ends have subtly different requirements and it's getting worse;
> GenPyrex or the java back-end for example don't have any clue about
> fully-specialized flow graphs either.
> 
> Some kind of solution for this problem would involve turning the
> Translator into some kind of 'make' engine.  The various "phases" would
> be described with dependencies: getting the flow graph, performing
> various simplifications, annotating, various more simplifications based
> on the annotations, typing, generating code, compiling.  This needs to
> be fine-grained enough, because for example the typer creates a few new
> graphs which must again be annotated and simplified in a precise way.
> Internal phases of the annotator or the typer would also benefit from
> being described in this way.  The user-visible goal would be that trying
> to call a method on the Translator (e.g.  llvmcompile()) would figure
> out which phases must be run, or complain if too many phases have
> already been run, instead of crashing in mysterious ways if too little
> or too much has been done before.
> 

Is it ever the case where the translator doesn't know which annotation 
steps/optimization steps are needed? I.e could this not just be done so 
that the translator entry point does all the work setting up what is 
needed? There could be some common code for processing shared options but 
is a dependancy manager really needed?

If a particular engine needs some code to be run, why doesn't it just run 
the code itself? I don't mean that to sound acrimonious, I'm just trying 
to understand!

(I've just realised I am making a lot of comments without actually doing 
any work, so please tell me to shup up if I am annoying you!)

Cheers,
Ben

> 
> A bientot,
> 
> Armin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 
> 



From Ben.Young at risk.sungard.com  Mon Jun 13 15:39:09 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 13 Jun 2005 14:39:09 +0100
Subject: [pypy-dev] planning the 0.6.2 PyPy release
In-Reply-To: <20050613130840.GB2358@code1.codespeak.net>
Message-ID: <OF9062C0A2.05FDA9F7-ON8025701F.004AB161-8025701F.004A441E@risk.sungard.com>

Hi,

pypy-dev-bounces at codespeak.net wrote on 13/06/2005 14:08:40:

> Hi Ben,
> 
> On Mon, Jun 13, 2005 at 10:06:15AM +0100, Ben.Young at risk.sungard.com 
wrote:
> > >    (There are still some open questions there; for
> > >example, it's not too difficult to detect list comprehensions by 
looking
> > >at the flow graphs; then instead of heuristic over-allocation for the
> > >list that is being built, it's often possible to determine in advance
> > >the size of the list -- e.g. because it will be the same length as
> > >another list that we iterate over.)
> > 
> > Does this mean that when you eventually come to *dynamically* annotate 

> > code that lists will be an annotated version of the std.listobject, or 
do 
> > you expect it to use the rpython typer as well?
> 
> The dynamic annotation part is still unclear, but I suppose that it will
> generally be done at a lower level than std.listobject.  The goal is to
> keep it mostly independent on the language being interpreted/compiled,
> and only dependent on RPython.
>
...snip...
> 
> This is essentially how Psyco works.  I am not sure about the details or
> how to integrate it better with the existing annotation framework
> (which, with more efforts, would give interesting hints about user
> applications too, instead of just about PyPy).  I am not sure if any of
> this will be done as I describe above, actually. :-)  What I am
> reasonably sure is only that it *could* be done like this.
> 


Thank you for the detailed explanation. I think I'll have to read it about 
10 times before I fully understand it, but it's definitely helpful!

Am I to understand there is another processing unit forthcoming that 
doesn't exist yet? (The flowgraph interpreter). I've heard its name 
mentioned on the irc logs a few times, and now here. Is there anywhere I 
can read the plans for it?

Cheers,
Ben

> 
> A bientot,
> 
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 
> 



From hpk at trillke.net  Mon Jun 13 16:01:08 2005
From: hpk at trillke.net (holger krekel)
Date: Mon, 13 Jun 2005 16:01:08 +0200
Subject: [pypy-dev] planning the 0.6.2 PyPy release
In-Reply-To: <OF9062C0A2.05FDA9F7-ON8025701F.004AB161-8025701F.004A441E@risk.sungard.com>
References: <20050613130840.GB2358@code1.codespeak.net>
	<OF9062C0A2.05FDA9F7-ON8025701F.004AB161-8025701F.004A441E@risk.sungard.com>
Message-ID: <20050613140108.GP21207@solar.trillke.net>

Hi Ben, 

On Mon, Jun 13, 2005 at 14:39 +0100, Ben.Young at risk.sungard.com wrote:
> Am I to understand there is another processing unit forthcoming that 
> doesn't exist yet? (The flowgraph interpreter). I've heard its name 
> mentioned on the irc logs a few times, and now here. Is there anywhere I 
> can read the plans for it?

There are no written down plans.  I intend to start working on it 
real soon now (like tomorrow).  It's main use should be to perform 
testing and playing around with the lowlevel flowgraph without 
having to test backends at the same time. 

cheers, 

    holger


From Ben.Young at risk.sungard.com  Mon Jun 13 16:10:05 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 13 Jun 2005 15:10:05 +0100
Subject: [pypy-dev] planning the 0.6.2 PyPy release
In-Reply-To: <20050613140108.GP21207@solar.trillke.net>
Message-ID: <OFB79013D5.6DC397B4-ON8025701F.004DCE8E-8025701F.004D1910@risk.sungard.com>

Thanks!

Ben

holger krekel <hpk at trillke.net> wrote on 13/06/2005 15:01:08:

> Hi Ben, 
> 
> On Mon, Jun 13, 2005 at 14:39 +0100, Ben.Young at risk.sungard.com wrote:
> > Am I to understand there is another processing unit forthcoming that 
> > doesn't exist yet? (The flowgraph interpreter). I've heard its name 
> > mentioned on the irc logs a few times, and now here. Is there anywhere 
I 
> > can read the plans for it?
> 
> There are no written down plans.  I intend to start working on it 
> real soon now (like tomorrow).  It's main use should be to perform 
> testing and playing around with the lowlevel flowgraph without 
> having to test backends at the same time. 
> 
> cheers, 
> 
>     holger
> 



From arigo at tunes.org  Mon Jun 13 17:59:55 2005
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 13 Jun 2005 17:59:55 +0200
Subject: [pypy-dev] Fw: [pypy-svn] r13221 - in pypy/dist/pypy/rpython: .
	test
In-Reply-To: <20050609224016.GA9978@code1.codespeak.net>
References: <OFEA8F910F.C6538E78-ON8025701B.0032503D-8025701B.0031B46C@risk.sungard.com>
	<20050609224016.GA9978@code1.codespeak.net>
Message-ID: <20050613155955.GA5500@code1.codespeak.net>

Hi,

On Fri, Jun 10, 2005 at 12:40:16AM +0200, Armin Rigo wrote:
> > Looking at these commits, I just have a quick question. Can the ll_ 
> > functions call each other?
> 
> Not right now, but of course it's clearly something that would be
> useful.  We have an idea on how to enable it but it's not done yet.

This has been implemented now.  The ll_ functions can freely call each
others.


A bientot,

Armin


From hpk at trillke.net  Mon Jun 13 21:35:27 2005
From: hpk at trillke.net (holger krekel)
Date: Mon, 13 Jun 2005 21:35:27 +0200
Subject: [pypy-dev] postponing 0.6.2/the move to CPython 2.4.1
Message-ID: <20050613193527.GW21207@solar.trillke.net>

Hi folks, 

well, it seems our early 0.6.2 planning this month may have been 
a bit too optimistic.  Armin, Samuele, Christian and me discussed
today that we should not go for the move to CPython 2.4.1  
before EuroPython.   The main reasoning is that it requires an 
unknown amount of effort and might also divert from the 
translation focus because of little issues popping up everywhere.  

Instead, the idea is to experiment with the 2.4.1-move at the
Post-EuroPython sprint.  After that we should have a better
picture of when it makes sense to go for a new release. 
Basically we should stably work with the 2.4.1 baseline for 
some time before doing a release.  However, we think that improving 
the documentation before EuroPython and the sprint still makes 
sense.  But that doesn't require a new release per se.  

So unless there is serious disagreement, the 0.6.2 release 
and the 2.4.1 move will not be tackled in June.  At least
we discovered the problem before being well into the release
process so that could be called progress :-) 

cheers, 

    holger


From arigo at tunes.org  Tue Jun 14 11:34:04 2005
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 14 Jun 2005 11:34:04 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13328 - pypy/dist/pypy/translator/test
In-Reply-To: <OFAEDEF1EC.B342E15D-ON8025701F.0049DDCB-8025701F.0049DDC3@risk.sungard.com>
References: <20050613115426.GA2358@code1.codespeak.net>
	<OFAEDEF1EC.B342E15D-ON8025701F.0049DDCB-8025701F.0049DDC3@risk.sungard.com>
Message-ID: <20050614093403.GA12355@code1.codespeak.net>

Hi Ben,

On Mon, Jun 13, 2005 at 02:34:47PM +0100, Ben.Young at risk.sungard.com wrote:
> Is it ever the case where the translator doesn't know which annotation 
> steps/optimization steps are needed? I.e could this not just be done so 
> that the translator entry point does all the work setting up what is 
> needed? There could be some common code for processing shared options but 
> is a dependancy manager really needed?

I was more referring to the class Translator instead of the command-line
front-end.  This class has a number of methods to invoke and inspect the
various phases.  Various tests want to test various phases, so it's not
always the case that we are just interested in getting compiled code and
letting the corresponding back-end figure out what it needs.  Moreover
intermediate phases like annotation actually require some input
(the argument types), so they can't just be run automatically.  (In this
case the depencendy manager would complain if the phase hasn't been run.)


A bientot,

Armin


From arigo at tunes.org  Tue Jun 14 11:37:51 2005
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 14 Jun 2005 11:37:51 +0200
Subject: [pypy-dev] planning the 0.6.2 PyPy release
In-Reply-To: <OF9062C0A2.05FDA9F7-ON8025701F.004AB161-8025701F.004A441E@risk.sungard.com>
References: <20050613130840.GB2358@code1.codespeak.net>
	<OF9062C0A2.05FDA9F7-ON8025701F.004AB161-8025701F.004A441E@risk.sungard.com>
Message-ID: <20050614093751.GB12355@code1.codespeak.net>

Hi Ben,

On Mon, Jun 13, 2005 at 02:39:09PM +0100, Ben.Young at risk.sungard.com wrote:
> Am I to understand there is another processing unit forthcoming that 
> doesn't exist yet? (The flowgraph interpreter).

The one I mentioned in my pictures is yet something else, an "abstract
flow graph interpreter", which would be some kind of far more advanced
version of the one we are mentioning on IRC these days.  The latter is
meant as a testing tool and isn't meant to be fast or optimizing.


Armin


From Ben.Young at risk.sungard.com  Tue Jun 14 11:47:18 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Tue, 14 Jun 2005 10:47:18 +0100
Subject: [pypy-dev] Re: [pypy-svn] r13328 - pypy/dist/pypy/translator/test
In-Reply-To: <20050614093403.GA12355@code1.codespeak.net>
Message-ID: <OFFB60BB7D.7BDA24B9-ON80257020.0035B229-80257020.00350A3C@risk.sungard.com>

Hi Armin,

Thanks for the explanation.

Cheers,
Ben

pypy-dev-bounces at codespeak.net wrote on 14/06/2005 10:34:04:

> Hi Ben,
> 
> On Mon, Jun 13, 2005 at 02:34:47PM +0100, Ben.Young at risk.sungard.com 
wrote:
> > Is it ever the case where the translator doesn't know which annotation 

> > steps/optimization steps are needed? I.e could this not just be done 
so 
> > that the translator entry point does all the work setting up what is 
> > needed? There could be some common code for processing shared options 
but 
> > is a dependancy manager really needed?
> 
> I was more referring to the class Translator instead of the command-line
> front-end.  This class has a number of methods to invoke and inspect the
> various phases.  Various tests want to test various phases, so it's not
> always the case that we are just interested in getting compiled code and
> letting the corresponding back-end figure out what it needs.  Moreover
> intermediate phases like annotation actually require some input
> (the argument types), so they can't just be run automatically.  (In this
> case the depencendy manager would complain if the phase hasn't been 
run.)
> 
> 
> A bientot,
> 
> Armin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 
> 



From arigo at tunes.org  Tue Jun 14 11:53:56 2005
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 14 Jun 2005 11:53:56 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13371 - pypy/branch/pycompiler/bin
In-Reply-To: <20050614091215.E18C127B83@code1.codespeak.net>
References: <20050614091215.E18C127B83@code1.codespeak.net>
Message-ID: <20050614095356.GA12904@code1.codespeak.net>

Hi Adrien,

On Tue, Jun 14, 2005 at 11:12:15AM +0200, adim at codespeak.net wrote:
> Log:
> added readline / history / tab support for translator shell
> (fed up with always retyping the same things ... )

Do you mind if this is copied to the trunk?  I guess it would be
generally useful.  The history file could be stored locally or in the
_cache directory.


Armin


From Adrien.DiMascio at logilab.fr  Tue Jun 14 12:14:46 2005
From: Adrien.DiMascio at logilab.fr (Adrien Di Mascio)
Date: Tue, 14 Jun 2005 12:14:46 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13371 - pypy/branch/pycompiler/bin
In-Reply-To: <20050614095356.GA12904@code1.codespeak.net>
References: <20050614091215.E18C127B83@code1.codespeak.net>
	<20050614095356.GA12904@code1.codespeak.net>
Message-ID: <20050614101445.GA4370@logilab.fr>

Hi Armin,

On Tue, Jun 14, 2005 at 11:53:56AM +0200, Armin Rigo wrote:
> On Tue, Jun 14, 2005 at 11:12:15AM +0200, adim at codespeak.net wrote:
> > Log:
> > added readline / history / tab support for translator shell
> > (fed up with always retyping the same things ... )
> 
> Do you mind if this is copied to the trunk?  I guess it would be
> generally useful.  The history file could be stored locally or in the
> _cache directory.

Done. I kept the local .pypytrhist file for history. Tell me if
storing the history file in the _cache directory would be better.

Cheers,
Adrien.

-- 
Adrien Di Mascio
LOGILAB, Paris (France).
http://www.logilab.com   http://www.logilab.fr   http://www.logilab.org


From arigo at tunes.org  Tue Jun 14 17:30:06 2005
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 14 Jun 2005 17:30:06 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13371 - pypy/branch/pycompiler/bin
In-Reply-To: <20050614101445.GA4370@logilab.fr>
References: <20050614091215.E18C127B83@code1.codespeak.net>
	<20050614095356.GA12904@code1.codespeak.net>
	<20050614101445.GA4370@logilab.fr>
Message-ID: <20050614153005.GA17066@code1.codespeak.net>

Hi Adrien,

On Tue, Jun 14, 2005 at 12:14:46PM +0200, Adrien Di Mascio wrote:
> Done. I kept the local .pypytrhist file for history. Tell me if
> storing the history file in the _cache directory would be better.

I don't mind.  I'm sure Linux distribution packagers won't be happy with
the idea of modifying files in-place, but well, the _cache directory
isn't really better at the moment, so a local .pypytrhist is fine.


A bientot,

Armin.


From Ben.Young at risk.sungard.com  Wed Jun 15 15:04:35 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Wed, 15 Jun 2005 14:04:35 +0100
Subject: [pypy-dev] A small patch
Message-ID: <OF60B91D18.79E61E65-ON80257021.00479B97-80257021.00471822@risk.sungard.com>

Hi Everyone,

After all my pointless question asking, I thought it would be best if I 
actually contributed something. 

Would someone mind applying this patch what allows more tests to compile 
on cygwin, using VC++ as the compiler. It seems to object to empty array 
initialisers.

Cheers,
Ben

Index: translator/c/node.py
===================================================================
--- translator/c/node.py        (revision 13434)
+++ translator/c/node.py        (working copy)
@@ -279,6 +279,9 @@
         if self.T.OF == Void:
             yield '\t%d' % len(self.obj.items)
             yield '}'
+        elif len(self.obj.items) == 0:
+            yield '\t0, NULL'
+            yield '}'
         else:
             yield '\t%d, {' % len(self.obj.items)
             for j in range(len(self.obj.items)):


From arigo at tunes.org  Wed Jun 15 15:36:15 2005
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 15 Jun 2005 15:36:15 +0200
Subject: [pypy-dev] A small patch
In-Reply-To: <OF60B91D18.79E61E65-ON80257021.00479B97-80257021.00471822@risk.sungard.com>
References: <OF60B91D18.79E61E65-ON80257021.00479B97-80257021.00471822@risk.sungard.com>
Message-ID: <20050615133615.GA2268@code1.codespeak.net>

Hi Ben,

On Wed, Jun 15, 2005 at 02:04:35PM +0100, Ben.Young at risk.sungard.com wrote:
> After all my pointless question asking, I thought it would be best if I 
> actually contributed something. 

Thanks!  I removed the NULL, because the array is actually nested in the
structure; it's not a pointer to an array.  Thanks for the fix, though.

Do you know if VC++ is equally unhappy about empty structure
initializers?  (It could be the case that some of our struct
declarations are empty; I don't know if there is a test showing that
behavior at the moment)


Armin


From Ben.Young at risk.sungard.com  Wed Jun 15 15:52:44 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Wed, 15 Jun 2005 14:52:44 +0100
Subject: [pypy-dev] A small patch
In-Reply-To: <20050615133615.GA2268@code1.codespeak.net>
Message-ID: <OFF8BF2A95.D3736BEE-ON80257021.004BDB3E-80257021.004B80C5@risk.sungard.com>

Hi Armin,

pypy-dev-bounces at codespeak.net wrote on 15/06/2005 14:36:15:

> Hi Ben,
> 
> On Wed, Jun 15, 2005 at 02:04:35PM +0100, Ben.Young at risk.sungard.com 
wrote:
> > After all my pointless question asking, I thought it would be best if 
I 
> > actually contributed something. 
> 
> Thanks!  I removed the NULL, because the array is actually nested in the
> structure; it's not a pointer to an array.  Thanks for the fix, though.
> 

Doh! Thats me thinking in C++ (where I'd allocate the variable bit 
seperately), and not this c-ish allocate an a struct, but with extra mem.

> Do you know if VC++ is equally unhappy about empty structure
> initializers?  (It could be the case that some of our struct
> declarations are empty; I don't know if there is a test showing that
> behavior at the moment)
> 

I not really sure. All the tests seem to pass, but thats not really an 
indicator. I'll investigate and let you know.

Cheers,
Ben

> 
> Armin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 
> 



From arigo at tunes.org  Wed Jun 15 19:01:50 2005
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 15 Jun 2005 19:01:50 +0200
Subject: [pypy-dev] Status update
Message-ID: <20050615170150.GA4946@code1.codespeak.net>

Hi,

An update on the translator status.  Dark obscure stuff has been mostly
sorted out.  Exceptions work all the way from the annotator to the
rtyper to GenC.  There is the beginning of a low-level interpreter now.

Still missing are:

* GenC knows little about float operations.  Need to complete
  translator/c/float_include.h.

* Almost all the topics from the rtyper, from the previous
  status report:
  
  - (hard) prebuilt constants -- PBCs -- and dicts with PBC keys
  - (medium) dicts with string keys, and iterators over them
  - (easy) iterators over strings
  - (medium) iterators over tuples
  - (varying) all built-in functions listed in annotation/builtin.py
  - (mostly easy) a few more operations: id, issubtype, str(?), is
  - (DONE) add_ovf --> int_add_ovf
  - (easy) list methods reverse, pop, insert, index(?)
  - (easy) tuple concatenation
  - (easy) string methods startswith(1-arg), endswith(1-arg),
           join (possibly only on the empty string ''), string slices
  - (easy) equality and non-equality are generally missing
  - (medium) limited form of string formatting: 'constant template' only
             which should only generate a sequence of simple operations
             like concatenation and str() on integers

* The lltype model is, hum, mostly stable but still has changed bits and
  pieces all the time.  One important change already done is that arrays
  can be of primitive types directly.  This means that rlist.py and
  rstr.py can be simplified a bit: for example, in rstr, STR can be
  defined with 'chars' as an Array(Char).  It makes reads and writes to
  characters more natural to write in the ll_() functions.  Another
  simplification is the operation cast_pointer, which replaces
  cast_parent, and which can cast freely pointer types as long as the
  cast is done (in any direction) between a structure and a substructure
  which is the first field of the structure -- or the 1st field of the
  1st field, etc.  This made rclass.py simpler.

* A missing piece of lltype that we'll work on soon is proper support
  for deallocation.  The problem right now is that if you have a pointer
  to a GcStruct 'B' which is actually the 1st field of a larger GcStruct
  'A', and this pointer goes away, GenC generates a decref; if the
  refcount reaches zero, it frees the structure beliving that is really
  a 'B' structure.  This means that the extra part of 'A' is ignored and
  just leaks away.  To solve this, we need to introduce minimal support
  for run-time types -- just some kind of opaque pointer that describes
  what the real, "large" GcStruct is.

* lltype will also need to grow a new primitive type, UnicodeChar.


A bientot,

Armin


From arigo at tunes.org  Sun Jun 19 12:20:10 2005
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 19 Jun 2005 12:20:10 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13583 - in
	pypy/dist/pypy/translator/llvm2: . test
In-Reply-To: <20050618162830.4D67427B54@code1.codespeak.net>
References: <20050618162830.4D67427B54@code1.codespeak.net>
Message-ID: <20050619102010.GA24065@code1.codespeak.net>

Hi,

On Sat, Jun 18, 2005 at 06:28:30PM +0200, cfbolz at codespeak.net wrote:
> - added flow graph transform remove_same_as, because same_as is
>   incompatible with SSA in LLVM.

I'm not sure I see why 'same_as' is incompatible with LLVM?  It's not an
"assignment" operation, it's just an operation that creates a new result
variable just like all the other ones.  The optimization is nice,
though.


A bientot,

Armin


From hpk at trillke.net  Sun Jun 19 14:17:22 2005
From: hpk at trillke.net (holger krekel)
Date: Sun, 19 Jun 2005 14:17:22 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13583 - in
	pypy/dist/pypy/translator/llvm2: . test
In-Reply-To: <20050619102010.GA24065@code1.codespeak.net>
References: <20050618162830.4D67427B54@code1.codespeak.net>
	<20050619102010.GA24065@code1.codespeak.net>
Message-ID: <20050619121722.GO21207@solar.trillke.net>

Hi Armin, 

On Sun, Jun 19, 2005 at 12:20 +0200, Armin Rigo wrote:
> On Sat, Jun 18, 2005 at 06:28:30PM +0200, cfbolz at codespeak.net wrote:
> > - added flow graph transform remove_same_as, because same_as is
> >   incompatible with SSA in LLVM.
> 
> I'm not sure I see why 'same_as' is incompatible with LLVM?  It's not an
> "assignment" operation, it's just an operation that creates a new result
> variable just like all the other ones.  The optimization is nice,
> though.

Basically LLVM does not support a generic 

    x1 = x2 
  
and we couldn't think of a way to tweak some equivalent. 
Previously, Carl used e.g. 'x1 = add int 0, x2' but we 
thought we just want to get rid of the problem. 

    holger


From sabre at nondot.org  Sun Jun 19 15:34:00 2005
From: sabre at nondot.org (Chris Lattner)
Date: Sun, 19 Jun 2005 08:34:00 -0500 (CDT)
Subject: [pypy-dev] Re: [pypy-svn] r13583 - in
	pypy/dist/pypy/translator/llvm2: . test
In-Reply-To: <20050619121722.GO21207@solar.trillke.net>
References: <20050618162830.4D67427B54@code1.codespeak.net>
	<20050619102010.GA24065@code1.codespeak.net>
	<20050619121722.GO21207@solar.trillke.net>
Message-ID: <Pine.LNX.4.61.0506190832450.3447@nondot.org>

On Sun, 19 Jun 2005, holger krekel wrote:
> Hi Armin,
>
> On Sun, Jun 19, 2005 at 12:20 +0200, Armin Rigo wrote:
>> On Sat, Jun 18, 2005 at 06:28:30PM +0200, cfbolz at codespeak.net wrote:
>>> - added flow graph transform remove_same_as, because same_as is
>>>   incompatible with SSA in LLVM.
>>
>> I'm not sure I see why 'same_as' is incompatible with LLVM?  It's not an
>> "assignment" operation, it's just an operation that creates a new result
>> variable just like all the other ones.  The optimization is nice,
>> though.
>
> Basically LLVM does not support a generic
>
>    x1 = x2
>
> and we couldn't think of a way to tweak some equivalent.
> Previously, Carl used e.g. 'x1 = add int 0, x2' but we
> thought we just want to get rid of the problem.

Another way that works with bools and floats, is to use a cast:

   x1 = cast int x2 to int

You're right though that if you can just get rid of the concept, that 
would be better.  If you're dealing with SSA form, there should be no 
reason to need to copy a value like that: just use the original.

-Chris

-- 
http://nondot.org/sabre/
http://llvm.cs.uiuc.edu/


From tismer at stackless.com  Sun Jun 19 20:11:37 2005
From: tismer at stackless.com (Christian Tismer)
Date: Sun, 19 Jun 2005 20:11:37 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13606 - pypy/dist/pypy/translator
In-Reply-To: <20050619170733.D2D9A27B88@code1.codespeak.net>
References: <20050619170733.D2D9A27B88@code1.codespeak.net>
Message-ID: <42B5B559.6050104@stackless.com>

arigo at codespeak.net wrote:

> Author: arigo
> Date: Sun Jun 19 19:07:32 2005
> New Revision: 13606
> 
> Modified:
>    pypy/dist/pypy/translator/geninterplevel.py
> Log:
> Use the SSI_to_SSA optimization in geninterplevel.  This allows the generated
> functions to look nicer, with less copying around all over the place the same
> value.

Nice change!
This is also a step towards scopedgraphs, which will probably
be yet another entry in backendoptimization.py
The main difference would be to identify subgraphs with
a single entry and exit point. These can then be rendered
by nesting them into a block and avoiding to transfer the
implied variables over the entry link at all.

ciao - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/


From arigo at tunes.org  Tue Jun 21 02:09:13 2005
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 21 Jun 2005 02:09:13 +0200
Subject: [pypy-dev] Summer of code mentors
Message-ID: <20050621000913.GA11284@code1.codespeak.net>

Hi all,

For Google's Summer of Code, the Python Software Foundation could do
with more mentors willing to supervise projects.  We are in the process
of choosing proposals, and aim roughly at 20-25 accepted ones.  I know
we are at least 3 mentors on this list, but I thought I'd ask anyway if
someone else wants to jump in, it's not too late yet :-)

Inofficially -- hint without guarantee -- we might accept two
PyPy-related proposals.  Would-be mentors will be about other
Python-related stuff -- there is a wide range of subjects and ideas.


Armin


From Ben.Young at risk.sungard.com  Tue Jun 21 10:33:50 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Tue, 21 Jun 2005 09:33:50 +0100
Subject: [pypy-dev] Re: [pypy-svn] r13639 - pypy/dist/pypy/translator/goal
In-Reply-To: <20050620221426.3B5B827B62@code1.codespeak.net>
Message-ID: <OF13A68CA7.A1421D4F-ON80257027.002EDE18-80257027.002E4AB8@risk.sungard.com>

You may be happy to know, on my machine, running cygwin, I get a factor of 
2000 for the fast mode!

Sounds like you are quite a long  way towards your goal!

....
(pypy.rpython.exceptiondata:109) ll_pyexcclass2exc__PyObjectPtr -> 
SomePtr(ll_ptrtype=<ptr(gc) to GcStruct object { typeptr: ptr() to S
truct object_vtable { parenttypeptr: ptr() to Struct object_vtable { ... 
}, rtti: ptr() to RuntimeTypeInfo (opaque), name: ptr() to Arr
ay of Char  } }>)
Back-end optimizations...
------------------------------------------------------------
Generating and compiling C code...
Running!
translated rpystone.pystones/fast time for 1000000000 passes = 2.16684
This machine benchmarks at 4.61501e+008 translated rpystone/fast 
pystones/second
CPython:
rpystone.pystones/fast time for 50000 passes = 0.215902
This machine benchmarks at 231587 rpystone/fast pystones/second
------------------------------------------------------------
Done.

--Return--
> c:\documents and 
settings\youngb\dist\pypy\translator\goal\translate_pypy.py(474)debug()->None
-> func(*args)
(Pdb) 4.61e8/231587
1990.6125991528022
(Pdb)


pypy-svn-bounces at codespeak.net wrote on 20/06/2005 23:14:26:

> Author: tismer
> Date: Tue Jun 21 00:14:25 2005
> New Revision: 13639
> 
> Modified:
>    pypy/dist/pypy/translator/goal/targetrpystone.py
>    pypy/dist/pypy/translator/goal/targetrpystone2.py
> Log:
> did some testing with the newly generated code.
> At first glance, it looks great, if we can trust
> the timing.
> For "slow" mode, I get a factor of 43.
> For "fast"mode, I get 250 or more, and I bumped
> the iterations to 10**9. But real run-time
> seems to be several seconds, so I think we can't trust
> these results. Something must be wrong with the dry-running.
> Will look abit deeper when on the train.
> 
> Modified: pypy/dist/pypy/translator/goal/targetrpystone.py
> 
==============================================================================
> --- pypy/dist/pypy/translator/goal/targetrpystone.py   (original)
> +++ pypy/dist/pypy/translator/goal/targetrpystone.py   Tue Jun 21 
> 00:14:25 2005
> @@ -1,6 +1,6 @@
>  import targetrpystonex
> 
> -LOOPS = 150000
> +LOOPS = 2000000
> 
>  # targetrpystonex.rpystone.setslow(False)
> 
> 
> Modified: pypy/dist/pypy/translator/goal/targetrpystone2.py
> 
==============================================================================
> --- pypy/dist/pypy/translator/goal/targetrpystone2.py   (original)
> +++ pypy/dist/pypy/translator/goal/targetrpystone2.py   Tue Jun 21 
> 00:14:25 2005
> @@ -1,7 +1,7 @@
>  import targetrpystonex
> 
> 
> -LOOPS = 1000000
> +LOOPS = 1000000000 #10**9
> 
>  targetrpystonex.rpystone.setslow(False)
> 
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
> 



From arigo at tunes.org  Tue Jun 21 11:15:20 2005
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 21 Jun 2005 11:15:20 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13639 - pypy/dist/pypy/translator/goal
In-Reply-To: <OF13A68CA7.A1421D4F-ON80257027.002EDE18-80257027.002E4AB8@risk.sungard.com>
References: <20050620221426.3B5B827B62@code1.codespeak.net>
	<OF13A68CA7.A1421D4F-ON80257027.002EDE18-80257027.002E4AB8@risk.sungard.com>
Message-ID: <20050621091520.GA15575@code1.codespeak.net>

Hi Ben!

On Tue, Jun 21, 2005 at 09:33:50AM +0100, Ben.Young at risk.sungard.com wrote:
> Sounds like you are quite a long  way towards your goal!

That's great to impress the crowds indeed :-)  But the so-called fast
pystone is really not representant of anything at all; it's also
extremely machine-dependent, for unknown reasons.  The other pystone for
example gets a speed-up of anywhere between 13 and 40 on various
hardware.  Which means that there is still some work to be done...


A bientot,

Armin


From Ben.Young at risk.sungard.com  Tue Jun 21 15:52:06 2005
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Tue, 21 Jun 2005 14:52:06 +0100
Subject: [pypy-dev] Re: [pypy-svn] r13653 - pypy/dist/pypy/rpython[POSSIBLE
	SPAM]
In-Reply-To: <20050621114842.F30C227B69@code1.codespeak.net>
Message-ID: <OF37D9D119.F4087B2F-ON80257027.004BFE93-80257027.004B6DFF@risk.sungard.com>

Hi Armin,

Not sure if it was this changelist, but you are new generating c with 
nested comments, which doesn't seem to be allowed by all compilers.

e.g.

        /* /* nothing */ = v106238->z_cls_my_method; */

Cheers,
Ben

pypy-svn-bounces at codespeak.net wrote on 21/06/2005 12:48:42:

> Author: arigo
> Date: Tue Jun 21 13:48:41 2005
> New Revision: 13653
> 
> Modified:
>    pypy/dist/pypy/rpython/rpbc.py
>    pypy/dist/pypy/rpython/rtyper.py
> Log:
> - Method calls sanitization, by falling back to function calls code.
> - Nicer error message for hop.inputarg() argument count mismatch.
> 
> 
> Modified: pypy/dist/pypy/rpython/rpbc.py
> 
==============================================================================
> --- pypy/dist/pypy/rpython/rpbc.py   (original)
> +++ pypy/dist/pypy/rpython/rpbc.py   Tue Jun 21 13:48:41 2005
> @@ -355,30 +355,26 @@
>                               "methods can be found: %r" % (
>                  s_pbc.prebuiltinstances,))
>          # the low-level representation is just the bound 'self' 
argument.
> -        self.r_instance = rclass.getinstancerepr(rtyper, self.classdef)
> -        self.lowleveltype = self.r_instance.lowleveltype
> +        self.s_im_self = annmodel.SomeInstance(self.classdef)
> +        self.r_im_self = rclass.getinstancerepr(rtyper, self.classdef)
> +        self.lowleveltype = self.r_im_self.lowleveltype
> 
>      def rtype_simple_call(self, hop):
> -        # XXX the graph of functions used as methods may need to be 
hacked
> -        # XXX so that its 'self' argument accepts a pointer to an 
instance of
> -        # XXX the common base class.  This is needed to make the 
direct_call
> -        # XXX below well-typed.
> -        r_class = self.r_instance.rclass
> +        r_class = self.r_im_self.rclass
>          mangled_name, r_func = r_class.clsfields[self.methodname]
>          assert isinstance(r_func, FunctionsPBCRepr)
> -        #
> -        # XXX try to unify with FunctionsPBCRepr.rtype_simple_call()
> -        f, rinputs, rresult = 
r_func.function_signatures.itervalues().next()
> -        vlist = hop.inputargs(self, *rinputs[1:])  # ignore the 
> self from r_func
> -        if r_func.lowleveltype == Void:
> -            assert len(r_func.function_signatures) == 1
> -            vfunc = hop.inputconst(typeOf(f), f)
> -        else:
> -            vinst = vlist[0]
> -            vcls = self.r_instance.getfield(vinst, '__class__', 
hop.llops)
> -            vfunc = r_class.getclsfield(vcls, self.methodname, 
hop.llops)
> -        vlist.insert(0, vfunc)
> -        return hop.genop('direct_call', vlist, resulttype = rresult)
> +        s_func = r_func.s_pbc
> +
> +        hop2 = hop.copy()
> +        hop2.args_s[0] = self.s_im_self   # make the 1st arg stand 
> for 'im_self'
> +        hop2.args_r[0] = self.r_im_self   # (same lowleveltype as 
'self')
> +
> +        v_im_self = hop.inputarg(self, arg=0)
> +        v_cls = self.r_im_self.getfield(v_im_self, '__class__', 
hop.llops)
> +        v_func = r_class.getclsfield(v_cls, self.methodname, hop.llops)
> +        hop2.v_s_insertfirstarg(v_func, s_func)   # insert 'function'
> +        # now hop2 looks like simple_call(function, self, args...)
> +        return hop2.dispatch()
> 
> 
>  # ____________________________________________________________
> 
> Modified: pypy/dist/pypy/rpython/rtyper.py
> 
==============================================================================
> --- pypy/dist/pypy/rpython/rtyper.py   (original)
> +++ pypy/dist/pypy/rpython/rtyper.py   Tue Jun 21 13:48:41 2005
> @@ -417,9 +417,10 @@
>      inputconst = staticmethod(inputconst)    # export via the 
> HighLevelOp class
> 
>      def inputargs(self, *converted_to):
> -        assert len(converted_to) == self.nb_args, (
> -            "operation argument count mismatch: '%s' has %d arguments" 
% (
> -            self.spaceop.opname, self.nb_args))
> +        if len(converted_to) != self.nb_args:
> +            raise TyperError("operation argument count mismatch:\n"
> +                             "'%s' has %d arguments, rtyper wants %d" % 
(
> +                self.spaceop.opname, self.nb_args, len(converted_to)))
>          vars = []
>          for i in range(len(converted_to)):
>              vars.append(self.inputarg(converted_to[i], i))
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
> 



From pedronis at strakt.com  Tue Jun 21 16:25:33 2005
From: pedronis at strakt.com (Samuele Pedroni)
Date: Tue, 21 Jun 2005 16:25:33 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13653 - pypy/dist/pypy/rpython[POSSIBLE
	SPAM]
In-Reply-To: <OF37D9D119.F4087B2F-ON80257027.004BFE93-80257027.004B6DFF@risk.sungard.com>
References: <OF37D9D119.F4087B2F-ON80257027.004BFE93-80257027.004B6DFF@risk.sungard.com>
Message-ID: <42B8235D.8040101@strakt.com>

Ben.Young at risk.sungard.com wrote:

>Hi Armin,
>
>Not sure if it was this changelist, but you are new generating c with 
>nested comments, which doesn't seem to be allowed by all compilers.
>
>e.g.
>
>        /* /* nothing */ = v106238->z_cls_my_method; */
>
>  
>
this has just been fixed. Some tests failure were indeed provoked by it.


>Cheers,
>Ben
>
>pypy-svn-bounces at codespeak.net wrote on 21/06/2005 12:48:42:
>
>  
>
>>Author: arigo
>>Date: Tue Jun 21 13:48:41 2005
>>New Revision: 13653
>>
>>Modified:
>>   pypy/dist/pypy/rpython/rpbc.py
>>   pypy/dist/pypy/rpython/rtyper.py
>>Log:
>>- Method calls sanitization, by falling back to function calls code.
>>- Nicer error message for hop.inputarg() argument count mismatch.
>>
>>
>>Modified: pypy/dist/pypy/rpython/rpbc.py
>>
>>    
>>
>==============================================================================
>  
>
>>--- pypy/dist/pypy/rpython/rpbc.py   (original)
>>+++ pypy/dist/pypy/rpython/rpbc.py   Tue Jun 21 13:48:41 2005
>>@@ -355,30 +355,26 @@
>>                              "methods can be found: %r" % (
>>                 s_pbc.prebuiltinstances,))
>>         # the low-level representation is just the bound 'self' 
>>    
>>
>argument.
>  
>
>>-        self.r_instance = rclass.getinstancerepr(rtyper, self.classdef)
>>-        self.lowleveltype = self.r_instance.lowleveltype
>>+        self.s_im_self = annmodel.SomeInstance(self.classdef)
>>+        self.r_im_self = rclass.getinstancerepr(rtyper, self.classdef)
>>+        self.lowleveltype = self.r_im_self.lowleveltype
>>
>>     def rtype_simple_call(self, hop):
>>-        # XXX the graph of functions used as methods may need to be 
>>    
>>
>hacked
>  
>
>>-        # XXX so that its 'self' argument accepts a pointer to an 
>>    
>>
>instance of
>  
>
>>-        # XXX the common base class.  This is needed to make the 
>>    
>>
>direct_call
>  
>
>>-        # XXX below well-typed.
>>-        r_class = self.r_instance.rclass
>>+        r_class = self.r_im_self.rclass
>>         mangled_name, r_func = r_class.clsfields[self.methodname]
>>         assert isinstance(r_func, FunctionsPBCRepr)
>>-        #
>>-        # XXX try to unify with FunctionsPBCRepr.rtype_simple_call()
>>-        f, rinputs, rresult = 
>>    
>>
>r_func.function_signatures.itervalues().next()
>  
>
>>-        vlist = hop.inputargs(self, *rinputs[1:])  # ignore the 
>>self from r_func
>>-        if r_func.lowleveltype == Void:
>>-            assert len(r_func.function_signatures) == 1
>>-            vfunc = hop.inputconst(typeOf(f), f)
>>-        else:
>>-            vinst = vlist[0]
>>-            vcls = self.r_instance.getfield(vinst, '__class__', 
>>    
>>
>hop.llops)
>  
>
>>-            vfunc = r_class.getclsfield(vcls, self.methodname, 
>>    
>>
>hop.llops)
>  
>
>>-        vlist.insert(0, vfunc)
>>-        return hop.genop('direct_call', vlist, resulttype = rresult)
>>+        s_func = r_func.s_pbc
>>+
>>+        hop2 = hop.copy()
>>+        hop2.args_s[0] = self.s_im_self   # make the 1st arg stand 
>>for 'im_self'
>>+        hop2.args_r[0] = self.r_im_self   # (same lowleveltype as 
>>    
>>
>'self')
>  
>
>>+
>>+        v_im_self = hop.inputarg(self, arg=0)
>>+        v_cls = self.r_im_self.getfield(v_im_self, '__class__', 
>>    
>>
>hop.llops)
>  
>
>>+        v_func = r_class.getclsfield(v_cls, self.methodname, hop.llops)
>>+        hop2.v_s_insertfirstarg(v_func, s_func)   # insert 'function'
>>+        # now hop2 looks like simple_call(function, self, args...)
>>+        return hop2.dispatch()
>>
>>
>> # ____________________________________________________________
>>
>>Modified: pypy/dist/pypy/rpython/rtyper.py
>>
>>    
>>
>==============================================================================
>  
>
>>--- pypy/dist/pypy/rpython/rtyper.py   (original)
>>+++ pypy/dist/pypy/rpython/rtyper.py   Tue Jun 21 13:48:41 2005
>>@@ -417,9 +417,10 @@
>>     inputconst = staticmethod(inputconst)    # export via the 
>>HighLevelOp class
>>
>>     def inputargs(self, *converted_to):
>>-        assert len(converted_to) == self.nb_args, (
>>-            "operation argument count mismatch: '%s' has %d arguments" 
>>    
>>
>% (
>  
>
>>-            self.spaceop.opname, self.nb_args))
>>+        if len(converted_to) != self.nb_args:
>>+            raise TyperError("operation argument count mismatch:\n"
>>+                             "'%s' has %d arguments, rtyper wants %d" % 
>>    
>>
>(
>  
>
>>+                self.spaceop.opname, self.nb_args, len(converted_to)))
>>         vars = []
>>         for i in range(len(converted_to)):
>>             vars.append(self.inputarg(converted_to[i], i))
>>_______________________________________________
>>pypy-svn mailing list
>>pypy-svn at codespeak.net
>>http://codespeak.net/mailman/listinfo/pypy-svn
>>
>>    
>>
>
>_______________________________________________
>pypy-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/pypy-dev
>  
>



From tismer at stackless.com  Wed Jun 22 00:14:20 2005
From: tismer at stackless.com (Christian Tismer)
Date: Wed, 22 Jun 2005 00:14:20 +0200
Subject: [pypy-dev] Re: [pypy-svn] r13639 - pypy/dist/pypy/translator/goal
In-Reply-To: <20050621091520.GA15575@code1.codespeak.net>
References: <20050620221426.3B5B827B62@code1.codespeak.net>	<OF13A68CA7.A1421D4F-ON80257027.002EDE18-80257027.002E4AB8@risk.sungard.com>
	<20050621091520.GA15575@code1.codespeak.net>
Message-ID: <42B8913C.6050501@stackless.com>

Armin Rigo wrote:
> Hi Ben!
> 
> On Tue, Jun 21, 2005 at 09:33:50AM +0100, Ben.Young at risk.sungard.com wrote:
> 
>>Sounds like you are quite a long  way towards your goal!
> 
> 
> That's great to impress the crowds indeed :-)  But the so-called fast
> pystone is really not representant of anything at all;

Why so negative?  This is a bit much of neglection.
I've been working with the two rpystone flavors for quite some
time, when the old genc code was active, and I have some
feeling about it. It feels very different, now.

Sure, it is true that there is no real measurement posible at the
moment. But in comparison with the former version, chings have
changed absolutely dramatically, IMHO:

On May 11th, I measured a speed gain of 11-12 on my windows machine
in "fast" mode. This is nothing, compared with the current
performance, which seems to be at least an order of magnitude
better.

Without hard facts, this at least tells us that we are generating
code that enables many optimizations which were not possible
at that time. And this is a very good result.

> it's also
> extremely machine-dependent, for unknown reasons.  The other pystone for
> example gets a speed-up of anywhere between 13 and 40 on various
> hardware.  Which means that there is still some work to be done...

If you meant "platform/compiler dependent", I agree.
The timings are not accurate because the code runs too fast.
If fast mode is enabled, it is quite likely that some compilers
optimize almost everything away.
I think dropping fastmode makes sense; it was only created to
see how far the old genc gets, without all the non-supported
data types.

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/


From mwh at python.net  Fri Jun 24 10:53:28 2005
From: mwh at python.net (Michael Hudson)
Date: Fri, 24 Jun 2005 09:53:28 +0100
Subject: [pypy-dev] sprint report, day 1
Message-ID: <2mu0jo2l13.fsf@starship.python.net>

Nothing official, but as I'm stting here waiting for the others to
arrive, I thought I'd wirte a little bit about what we did yesterday
(all my own opninions and recollections, corrections welcome :-).

The people present and hacking were: Armin, Samuele, Christian,
Holger, Michael, Anders (C) and Anders (L).

We started by writing:

http://codespeak.net/svn/pypy/extradoc/sprintinfo/pre-ep2005-planning.txt

(which actually started as a mail Armin sent to pypy-dev a week or so
ago, and was edited during the day to reflect progress).

Then in a pretty informal way we paired or tripled up and got
cracking.  Generally speaking, it was encouraging how quickly everyone
became productive, it says good things about the current design of
codebase (and also indicates how much work there is left to do...).

Almost all of the work happened at the rtyper level:

http://codespeak.net/pypy/index.cgi?doc/translation.html#the-rpython-typer

that sits between the annotator and the language backends.

Samuele and Michael mostly worked on string operations in rpython,
implementing conversions from ints to strings and limited string
formatting operations, did a couple of easy builtins and worked out
which were going to be hard.

Anders (L) and Christian worked mostly on rpython lists and tuples,
and also did string slicing.

Anders (C), Holger and Armin mostly worked on rpython dictionaries
(the ones that will hold keyword arguments during argument processing,
among a few other things), and nearly finished it.

Right, the others are hear now, so it's time to start hacking again...

Cheers,
mwh

-- 
  While preceding your entrance with a grenade is a good tactic in
  Quake, it can lead to problems if attempted at work.    -- C Hacking
               -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html



From florian.proff.schulze at gmx.net  Sat Jun 25 11:38:10 2005
From: florian.proff.schulze at gmx.net (Florian Schulze)
Date: Sat, 25 Jun 2005 11:38:10 +0200
Subject: [pypy-dev] Re: sprint report, day 1
References: <2mu0jo2l13.fsf@starship.python.net>
Message-ID: <opssw6dwfmttxc4i@news.gmane.org>

On Fri, 24 Jun 2005 09:53:28 +0100, Michael Hudson <mwh at python.net> wrote:

[snipped report]

> Cheers,
> mwh

I just wanted to thank you for this, it's always nice to get short 
summarys of what's going on.

Regards,
Florian Schulze



From mwh at python.net  Sat Jun 25 12:59:51 2005
From: mwh at python.net (Michael Hudson)
Date: Sat, 25 Jun 2005 11:59:51 +0100
Subject: [pypy-dev] sprint day 2
Message-ID: <2mfyv63dnc.fsf@starship.python.net>

Yesterday was less obviously productive than day 1, possibly because
you now have to go quite a lot higher up the tree to find the lowest
fruit by now...

Arre and Holger finished off string-keyed dictionary stuff.

Anders and Samuele did the list builtin.

Armin and Michael did isinstance.

Michael and Armin wrote some grotty float parsing code and removed the
only use of the float builtin on a string (if anyone has some nice and
less numerically naive float parsing code or wants to write some, feel
free -- it's a fairly self contained task).

Christian worked on reducing the use of import * in rpython/ (though
really we should have less 'from module import thing ... thing' and
more 'import module ... module.thing', but this still an improvement).

Holger and Armin removed some of the uses of dictionaries in the
to-be-translated code.

Michael and Samuele implemented a very limited str for instances.

Arre and Anders implemented a 'unicode character' for the rtyper and c
backend.

Michael and Samuele investigated issues preventing the translation of
demo/bpnn.py.

Holger and Armin worked on functions like Cache.getorbuild which need
to be treated specially by the annotator and rtyper (because the
'build' part is not allowed to happen at runtime).

Then Michael, Samuele and the two Anders went to a midsomar party, and
at 1pm the day after, Samuele hasn't been seen since...

Cheers,
mwh

-- 
  python py.py ~/Source/python/dist/src/Lib/test/pystone.py
  Pystone(1.1) time for 5000 passes = 19129.1
  This machine benchmarks at 0.261381 pystones/second



From mwh at python.net  Sun Jun 26 11:48:58 2005
From: mwh at python.net (Michael Hudson)
Date: Sun, 26 Jun 2005 10:48:58 +0100
Subject: [pypy-dev] sprint day 3
Message-ID: <2m8y0x30tx.fsf@starship.python.net>

Yesterday was another good day at the pre-EP sprint.

We mostly started by polishing stuff from the day before -- unicode
characters, pre-built caches.

Holger and Michael removed by writing boring code a use of
specialize:memo we weren't feeling intelligent enought to support in
the rtyper.

Armin and Anders made yet another pass through the list of partially
supported 'builtins'[1] and made some decisions about whether support
should be removed, finished or ignored for now.

The we had bit of a planning session and decided that (Christian and
Arre) and (Anders and Armin) would pair on supporting builtins and
Holger, Samuele and Michael would work on translator issues.

Holger and Michael implemented for the C translator the few remaining
unsupported float operations the rtyper could emit.

Arre and Christian implemented a few more operations on rdicts.

Armin and Anders worked on the list of builtins in the usual fashion:
sometimes removing code now deemed to be not rpython, sometimes by
removing special-casing code that was no longer necessary and
sometimes by adding code to the rtyper.

Holger, Michael and Samuele attempted to translate demo/bpnn.py and
fixed the problems they ran into; this included obscure behaviour when
the rtyper hit code calling a statically known bound method of a
Constant, the same when the Constant was of a class that was only seen
by the RTyper as a class of such a Constant and not elsewhere.

Holger and Armin refactored some of the rtyper code in the area of
equality.

Michael and Samuele fixed some broken code in the c translator in the
area of calling C functions not implemented by us.

Anders with help from Armin implemented is_true for PBC and fixed some
bugs.

Holger and Armin implemented yet another sort of dictionary:
'constant' dicts that are built at initialization time and only
queried thereafter.

Then we called it a day and went for dinner.

Today, we plan to carry on the above and do a little planning for the
post-EP sprint (which I won't be at, as I'll be involved in the
much-more-entertaining task of moving house) and maybe, maybe, writing
a talk for that conference we hear is happening quite soon.

Cheers,
mwh

-- 
  The Oxford Bottled Beer Database heartily disapproves of the 
  excessive consumption of alcohol.  No, really.
                        -- http://www.bottledbeer.co.uk/beergames.html
                           (now sadly gone to the big 404 in the sky)



From mwh at python.net  Thu Jun 30 12:48:30 2005
From: mwh at python.net (Michael Hudson)
Date: Thu, 30 Jun 2005 11:48:30 +0100
Subject: [pypy-dev] Re: [pypy-svn] r14022 - pypy/extradoc/sprintinfo
In-Reply-To: <20050630110534.4A36F27B46@code1.codespeak.net>
	(arigo@codespeak.net's
	message of "Thu, 30 Jun 2005 13:05:34 +0200 (CEST)")
References: <20050630110534.4A36F27B46@code1.codespeak.net>
Message-ID: <2mhdfgxgqp.fsf@starship.python.net>

arigo at codespeak.net writes:

> +* fix floats::
> +    - translator/c/float_include.h should catch floating-point signals
> +      with a technique similar to CPython's.

This is not 100% clear, by the way.  I wouldn't claim that all of what
follows is solid fact (but I think I know more about this sort of
thing than most of the other PyPyers...).

The floating point standards define a bunch of conditions:

Overflow (eg huge number * huge number)
Underflow (tiny number / huge number)
DivisionByZero (finite value / zero)
InvalidOperation (0 / 0)
Inexact (for things like 1.0 / 10.0)
Rounded (when rounding occurs, not entirely sure how this differs from
         Inexact, tbh)

and maybe a few more.

Each condition has an independent setting ('trap') as to whether it's
occurence should halt computation.

There is some standard (not sure which, IEEE 754, SUS, POSIX, ...)
that says programs should start up with all traps disabled, so we
shouldn't really need to catch SIGFPE.  I'm also far from convinced
that the SIGFPE code is ever used today in Python (does anyone pass
--with-fpectl to configure?).

I think that there are some platforms which don't start up with all
traps disabled, but the correct approach for them is to find out the
incantation to whack them into standards mode and use that.



At the PyPy level we should probably decide on which traps are enabled
by default (at least DivisionByZero is traditional) and try to be
consistent about applying it (something CPython does less than well,
as various groans about strutil.string_to_float indicate).  We
probably don't want to get into the business of doing this at the FPU
level, or at least not yet.

My opinion: PyPy should trap DivisionByZero and InvalidOperation.
There's a case for trapping Overflow, but CPython doesn't do this
(most of the time...).  It would be insane IMHO to trap Underflow or
Inexact or Rounded.

Another question is whether we want to support user control of this
stuff.

All these issues affect Lib/decimal.py by the way, and reading the
documentation of that module is probably a good way to get an idea of
the area.



A possible post-sprint task would be making the aforementioned
strutil.string_to_float less horribly naive and finally fixing the
rounding issues in _float_formatting.py (for the last, some guy a
while ago sent me some dylan code that might help... let me upload it:

    http://starship.python.net/crew/mwh/format.dylan
    http://starship.python.net/crew/mwh/float.dylan

).  I can provide references for both these tasks, if anyone looks
interested.

Cheers,
mwh

-- 
  You sound surprised.  We're talking about a government department
  here - they have procedures, not intelligence.
                                            -- Ben Hutchings, cam.misc


From tismer at stackless.com  Thu Jun 30 21:00:06 2005
From: tismer at stackless.com (Christian Tismer)
Date: Thu, 30 Jun 2005 21:00:06 +0200
Subject: [pypy-dev] worth reading: Steve Jobs
Message-ID: <42C44136.2030208@stackless.com>

hi friends,

the following is a so-called "commencement" talk.
For those of you who could just enjoy an A-levels talk.
- this one is one done right! -
for those of you who feel too old to read this: show it to your kids.

http://news-service.stanford.edu/news/2005/june15/jobs-061505.html

ciao - chris
-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/