From arigo at tunes.org  Tue Apr  1 09:34:42 2014
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 1 Apr 2014 09:34:42 +0200
Subject: [pypy-dev] JIT not kicking in for callbacks
In-Reply-To: <CAK5idxTnGE_EfJ-qnfgtUbJPA0COuXoi0VdpFCKeHAs6xxEBNA@mail.gmail.com>
References: <CAPkN8xKeOuaW2kkGN4XiPoq7EQUn7V9gS8UdUD1JgsiRZnY7Hg@mail.gmail.com>
 <20140329125815.GA20269@untibox.unti>
 <CAPkN8xL3idihHfGfpGFBSyE0Yw5-gdDPs5iDh1PQAhYB8b_aKQ@mail.gmail.com>
 <BEF6DE65-3B8C-4545-B122-7C742A437B26@gmail.com>
 <CAPkN8xL=iVsmMMEmMf6ew65iEDuRqOkSLWbjrJe6tdfoNXckjg@mail.gmail.com>
 <CA+HFc84mS8Bw9YPXx2e-0eSNm6tymiseEWDquBybO+5LfSOUkQ@mail.gmail.com>
 <lh9jvi$ug9$1@ger.gmane.org> <5339B3E7.50804@gmail.com>
 <CAK5idxTnGE_EfJ-qnfgtUbJPA0COuXoi0VdpFCKeHAs6xxEBNA@mail.gmail.com>
Message-ID: <CAMSv6X1Om9shKau8xxMx=08cBXHOFmaYo9PKvcG02i6sU+YsOA@mail.gmail.com>

Hi,

On 31 March 2014 20:33, Maciej Fijalkowski <fijall at gmail.com> wrote:
> hi "interpret" includes "in the jit".

"In the jit-generated machine code", precisely.  Not "in the JIT while
building machine code".

/me renames this, as it was originally meant indeed to record only
interpretation outside machine code.


A bient?t,

Armin.

From techtonik at gmail.com  Thu Apr  3 10:33:35 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 3 Apr 2014 11:33:35 +0300
Subject: [pypy-dev] Why CFFI is not useful - need direct ABI access 4
	humans
In-Reply-To: <20140330152905.GA18471@ajhurst.org>
References: <CAPkN8xKeOuaW2kkGN4XiPoq7EQUn7V9gS8UdUD1JgsiRZnY7Hg@mail.gmail.com>
 <20140329125815.GA20269@untibox.unti>
 <CAPkN8xL3idihHfGfpGFBSyE0Yw5-gdDPs5iDh1PQAhYB8b_aKQ@mail.gmail.com>
 <BEF6DE65-3B8C-4545-B122-7C742A437B26@gmail.com>
 <CAPkN8xL=iVsmMMEmMf6ew65iEDuRqOkSLWbjrJe6tdfoNXckjg@mail.gmail.com>
 <20140330152905.GA18471@ajhurst.org>
Message-ID: <CAPkN8xJbp2Yv1OQ-ZkQO+f1Syp47dS0ib-kCUcCjx9nRV+f8uQ@mail.gmail.com>

On Sun, Mar 30, 2014 at 6:29 PM, Nathan Hurst <njh at njhurst.com> wrote:
>
> If you want to purely binary data in python there are already some
> good options: struct and numpy.  The first is optimised for short
> records, the latter for multidimensional arrays of binary data.

numpy is not Python. About struct I wrote in the first post - it is a legacy
interface that was not designed to be human friendly with usability
practices of 2014.

> I've used both on occasion for binary file and cross language communications.
>
>> Nice pythonic wrapper that abstracts completely from technical details
>> is not the goal. The goal is to provide practical defaults for
>> language and hardware independent abstraction. The primary object that
>> abstraction should work with is "platform-independent binary data",
>> the method is "to be readable by humans". On implementation level that
>> means that by default there is no ambiguity in syntax that defines
>> binary data (size or endianness), and if there is a dependency on
>> platform (CPU bitness etc.) it should be explicit, so that the
>> behavior of structure should be clear (self-describing type names +
>> type docs that list relevant platforms and effects on every platform).
>> This approach inverts existing practice of using platform dependent
>> binary structures by default.
>
> So struct and numpy are existing solutions to this problem, but I
> think you are thinking too low level for most problems.  It sounds
> like what you really want is a schema based serialisation protocol.

If you can assemble ELF binary in that protocol, then yes. My goal is
to work with low level binary data in Python with convenient tools. Not
necessary high level tools - just convenient for messing with chunks.

> There are a million of these, all alike, but the two I've used the
> most are msgpack and thrift.  Generally you want to be specifying
> statistical distributions rather that the specific encoding scheme;
> the choice between 4 byte and 8 byte ints is not particularly useful
> at the python programmer level, but knowing whether the number is a
> small int or a large one is.  Thrift is the best implementation of a
> remote call I've used, but it still leaves a lot of room for improvment.

Engineering should start with a task at hand. Statistical distribution
is a too high level task. It goes above the binary manipulating logic,
but the task of choosing a size of data based on other data is a
corner stone in all binary format processing.

> If you are actually talking about building something that will talk to
> any existing code in any existing language, then you will need
> something more like CFFI.  However, I don't think you want to take the
> C out of CFFI.  The reason is that the packing of the data is actually
> one of the least important parts of that interface.  As you've read on
> pypy-dev recently, reference counting vs gc is hard to get right.  But
> there are many other problems which you have to address for a truly
> language independent ABI:
>
> Stack convention: what order are things pushed onto the stack (IIRC C
> and pascal grow their heaps in opposite directions in memory).  Are
> things even pushed onto a stack? (LISP stacks are implemented with
> linked lists, some languages don't even bother with stack frames when
> they perform tail recursion removal, using conditional gotos instead)

Right. You will not be able to operate with stack registers, but if you
have a clean interface to construct the contents of binary stack and
read it back to friendly form - this will make the task of working with
these structures easier, so maybe one day it will be possible to have a
clean code that calls different libs with different calling conventions from
Python.

> Packing conventions: different cpus like pointers to be even, or
> aligned to 8 bytes.  Your code won't be portable if you don't handle
> this.

It is good to know about that, so this should be exposed, of course.
There is also a room for high level abstraction once you've covered the
basics.

> Exception handling: There are as many ways to handle exceptions as
> there are compilers, all of them with subtle rules around lifetimes of
> all the objects that are being excepted over.

I would be interested to see this on binary level, but I doubt such
papers exist. Making a library that can produce diagrams for such
structures will be a good starting point to grok at more serious
performance problem when you switch CPUs.

> Virtual methods and the like: In most languages (C is actually
> somewhat unusual here) methods or functions are actually called via a
> dereference or two rather than a direct memory location.  In C++
> virtual methods and Java this looks like a field lookup to find the
> vtable then a jump to a memory location from that table.  This is
> tedious and error prone to implement correctly.

It is tedious and error prone only if it is not obvious. There are
inevitable complex solutions in this world that are impossible to
avoid, but it is possible to deal with complexity.

If JIT can detect that vtable is not modified, it can shorten the
call chain. Of course you won't be able to define all types of
lookup logic in declarative form, but making this code readable
is a step forward. If not Eve Online - I'd never new about
Stackless and why it is awesome. Eve Online provided a user
friendly interface to introduce the technology by showing its
power through media of game design discipline. I think that the
power of alternative low level manipulations with
"data + processor" paradigm can be even better expressed in
language design discipline which PyPy/RPython is all about
IMHO.

> Generics and types: just working out which function to call is
> difficult once you have C++ templates, performing the correct type
> checking is difficult for Java.  Haskell's internal type specification
> alone is probably larger than all of the CFFI interfaces put together.

These are high level API and I am very interested how do they map
to the low level. That gives me an ability to design a better CPU for
these languages (or LPU FWIW) or see that current limitations
exist.

> There are no doubt whole areas I've missed here, but I hope this gives
> you a taste for why language developers love CFFI - it's easy to
> implement, easy enough to use and hides all of these complexities by
> pushing everything through the bottleneck which is the C ABI.
> Language developers would rather make their own language wonderful,
> compiler writers would rather make their own language efficient.

I never argued that CFFI is not loved. But I really think that it is loved
only by people who know C or had to face it. I do not agree that
language developers should not think about efficiency. As much as I
want to simplify the problem too, in real world they should think about
many things, find compromises and trade between them. On of the
reasons I am so pushy about the process and tools - language design
requires more scaffolding and collaborative tools that usual project
issue tracker allows. Python 3 is a living proof - better greater language,
but the adoption rate is very low. I am not speaking the reasons, I just
state the fact. I think that executable language designers should think
about low level.

http://www.joelonsoftware.com/articles/fog0000000319.html

> Nobody really cares enough to make supporting other languages directly
> a goal, the biggest and best effort in this regard was .net and it
> also cheated by folding everything through a small bottleneck (albeit
> a more expressive one).  This folding scars the languages around it -
> F# is still less pure than Haskell, J is incompatible with Java.
>
> If you want to tackle something in this area I would encourage you to
> look at the various serialisation tools out there and work out why
> they are not as good as they could be.  This is probably not the right
> mailing list for this discussion though.

I'm interested to see a list of user stories and conflicts between them,
like which choice was made in every conflict and (if possible) why. The
problem of modern development is that people face with the same
problems over and over again unable to understand the previous
knowledge. Providing tools that bring greater visibility into the domain
field can significantly help the progress. In context of binary manipulation
I really want to see more interchangeable graphical interfaces and
automated visualization tools built on top of that without overengineering
by various international controlling bodies. That means - they try to
complicate stuff to solve some problem no matter what. I think that the
approach should be more relaxed - there are problems that can't be
solved easily, but getting to the root of the problem and playing with it
should be easy.
-- 
anatoly t.

From techtonik at gmail.com  Thu Apr  3 10:33:44 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 3 Apr 2014 11:33:44 +0300
Subject: [pypy-dev] Why CFFI is not useful - need direct ABI access 4
	humans
In-Reply-To: <CA+HFc84mS8Bw9YPXx2e-0eSNm6tymiseEWDquBybO+5LfSOUkQ@mail.gmail.com>
References: <CAPkN8xKeOuaW2kkGN4XiPoq7EQUn7V9gS8UdUD1JgsiRZnY7Hg@mail.gmail.com>
 <20140329125815.GA20269@untibox.unti>
 <CAPkN8xL3idihHfGfpGFBSyE0Yw5-gdDPs5iDh1PQAhYB8b_aKQ@mail.gmail.com>
 <BEF6DE65-3B8C-4545-B122-7C742A437B26@gmail.com>
 <CAPkN8xL=iVsmMMEmMf6ew65iEDuRqOkSLWbjrJe6tdfoNXckjg@mail.gmail.com>
 <CA+HFc84mS8Bw9YPXx2e-0eSNm6tymiseEWDquBybO+5LfSOUkQ@mail.gmail.com>
Message-ID: <CAPkN8xJ6zggHFxxmjASrP376ZeKOGxK5SCsAXFhvdyCBLZG08A@mail.gmail.com>

On Sun, Mar 30, 2014 at 6:32 PM, Leonardo Santagada <santagada at gmail.com> wrote:
> On Sun, Mar 30, 2014 at 9:36 AM, anatoly techtonik <techtonik at gmail.com>
> wrote:
>>
>> On ABI level tools can be more useful, and there is where idea
>> intersects with CFFI. It doesn't cancel the fact that people need safe
>> feet injury prevention interfaces. Look for my previous answer with
>> the phrase "I mix C with binary manipulation" that covers this.
>
>
> I think you want something like https://pypi.python.org/pypi/construct/2.5.1
> right? It exists and is pretty good... some guys in my company are using it
> to model a binary audio control protocol both to prototype and to test the
> embeded implementation (which is in C).

Right. Construct is a starting point. What's why I included Tomer into CC list.
It's primary use, as I see it - is reversing existing binary formats.
I'm not quite
persuaded that it has the best declarative specification for binary data.

The difference is that binary data is multi-level. You can set types as UBInt8,
but that's not user friendly. It says it is unsigned, it is integer,
with 1 byte in
size. Why should I care? In terms of binary protocol I care only about allowed
rage of values. How do I work with them is the task of the next level. I
decompose the problem into multiple levels:

level 0: binary data location and size
level 1: check/get/set binary values as-is
level 2: processing nested binary values and dependent binary data
level 3: converting single values into user data types and back (symmetry)
level 4: serializing complex user data types into binary structures and back

I believe every linker/assembler does that. If the structure can not be
decomposed into these levels - it can not be statically analyzed. There are
also many interesting tasks about detecting constraints, processing
dependencies and most important - the user experience, interface for
humans to work on each level.

The basic drawback for construct is that it is hard to write tools that work
on top of it. There is no obvious decoupling between different levels, that's
why I can hardly think, for example, how can I compile a binary executable
from a lot of bytecodes with it.

Basically, why I raised the question in CFFI context is an attempt to play
with a different user story - assembling binary code - the part that is
currently done by tools written in C.
-- 
anatoly t.

From bokr at oz.net  Thu Apr  3 16:09:29 2014
From: bokr at oz.net (Bengt Richter)
Date: Thu, 03 Apr 2014 16:09:29 +0200
Subject: [pypy-dev] Why CFFI is not useful - need direct ABI access 4
	humans
In-Reply-To: <CAPkN8xJbp2Yv1OQ-ZkQO+f1Syp47dS0ib-kCUcCjx9nRV+f8uQ@mail.gmail.com>
References: <CAPkN8xKeOuaW2kkGN4XiPoq7EQUn7V9gS8UdUD1JgsiRZnY7Hg@mail.gmail.com>
 <20140329125815.GA20269@untibox.unti>
 <CAPkN8xL3idihHfGfpGFBSyE0Yw5-gdDPs5iDh1PQAhYB8b_aKQ@mail.gmail.com>
 <BEF6DE65-3B8C-4545-B122-7C742A437B26@gmail.com>
 <CAPkN8xL=iVsmMMEmMf6ew65iEDuRqOkSLWbjrJe6tdfoNXckjg@mail.gmail.com>
 <20140330152905.GA18471@ajhurst.org>
 <CAPkN8xJbp2Yv1OQ-ZkQO+f1Syp47dS0ib-kCUcCjx9nRV+f8uQ@mail.gmail.com>
Message-ID: <533D6B99.5010306@oz.net>

Hi Anatoly,

On 04/03/2014 10:33 AM anatoly techtonik wrote:

[...]
>  My goal is
> to work with low level binary data in Python with convenient tools. Not
> necessary high level tools - just convenient for messing with chunks.

[Nathan]
>> Exception handling: There are as many ways to handle exceptions as
>> there are compilers, all of them with subtle rules around lifetimes of
>> all the objects that are being excepted over.
>
> I would be interested to see this on binary level, but I doubt such
> papers exist. Making a library that can produce diagrams for such
> structures will be a good starting point to grok at more serious
> performance problem when you switch CPUs.
>

I think you might enjoy reading about LLVM and their approach to
representing bit-level stuff, and interfacing with various languages'
definitions of exceptions etc:

     http://llvm.org/docs/index.html

particularly

     http://llvm.org/docs/LangRef.html

from the introduction of the latter:

     The LLVM code representation is designed to be used in three
     different forms: as an in-memory compiler IR, as an on-disk
     bitcode representation (suitable for fast loading by a
     Just-In-Time compiler), and as a human readable assembly
     language representation. This allows LLVM to provide a
     powerful intermediate representation for efficient compiler
     transformations and analysis, while providing a natural
     means to debug and visualize the transformations. The three
     different forms of LLVM are all equivalent. This document
     describes the human readable representation and notation.

[...]

I think you may have some interesting ideas. I am also interested
in a high level language with ability to get down to the metal, so I
am trying to design one. The trouble is, scouting around for ideas,
you may wind up doing more reading than actually working. Ltu is
particularly dangerous ;-)

     http://lambda-the-ultimate.org/

I think you should try to design the language you want, according
to your ideas. IME there is no better way to gain respect and appreciation
for other people's work in language design ;-)

When you want to write your compiler, python is there. Or you may find
racket's ability to define alternate syntax languages interesting:

     http://docs.racket-lang.org/guide/languages.html

Their home pages is

     http://racket-lang.org/

Have fun ;-)

Regards,
Bengt Richter


From edward.barrett at kcl.ac.uk  Fri Apr  4 11:20:19 2014
From: edward.barrett at kcl.ac.uk (Edd Barrett)
Date: Fri, 4 Apr 2014 10:20:19 +0100
Subject: [pypy-dev] Dynamic Languages Symposium: Call for Papers
Message-ID: <20140404092019.GB7991@wilfred.lan>

===========================================================================
                      Dynamic Languages Symposium 2014
                              October 21, 2014
                                      
               Co-located with SPLASH 2014, Portland, OR, USA
                                      
             http://www.dynamic-languages-symposium.org/dls-14/
===========================================================================

The 10th Dynamic Languages Symposium (DLS) at SPLASH 2014 is the premier
forum for researchers and practitioners to share knowledge and research on
dynamic languages, their implementation, and applications. The influence of
dynamic languages -- from Lisp to Smalltalk to Python to Javascript -- on
real-world practice, and research, continues to grow.

DLS 2014 invites high quality papers reporting original research, innovative
contributions, or experience related to dynamic languages, their
implementation, and applications. Accepted papers will be published in the
ACM Digital Library, and freely available for 2 weeks before and after the
event itself. Areas of interest include but are not limited to:

 * Innovative language features and implementation techniques
 * Development and platform support, tools
 * Interesting applications
 * Domain-oriented programming
 * Very late binding, dynamic composition, and run-time adaptation
 * Reflection and meta-programming
 * Software evolution
 * Language symbiosis and multi-paradigm languages
 * Dynamic optimization
 * Hardware support
 * Experience reports and case studies
 * Educational approaches and perspectives
 * Semantics of dynamic languages


  Submissions

Submissions should not have been published previously nor be under review at
other events. Research papers should describe work that advances the current
state of the art. Experience papers should be of broad interest and should
describe insights gained from substantive practical applications. The program
committee will evaluate each contributed paper based on its relevance,
significance, clarity, length, and originality.

Papers are to be submitted electronically at
http://www.easychair.org/conferences?conf=dls14 in PDF format. Submissions
must be in the ACM format (see http://www.sigplan.org/authorInformation.htm)
and not exceed 12 pages. Authors are reminded that brevity is a virtue.

DLS 2014 will run a two-phase reviewing process to help authors make their
final papers the best that they can be. After the first round of reviews,
papers will be rejected, conditionally accepted, or unconditionally accepted.
Conditionally accepted papers will be given a list of issues raised by
reviewers. Authors will then submit a revised version of the paper with a
cover letter explaining how they have / why they have not addressed these
issues. The reviewers will then consider the cover letter and revised paper
and recommend final acceptance / rejection.


  Important dates

Submissions: June 8 2014 (FIRM DEADLINE)
First phase notification: July 14 2014
Revisions due: August 4 2014
Final notification: August 11 2014
Camera ready: August 15 2014
DLS: October 21 2014


  Programme chair

Laurence Tratt, King's College London, UK
e-mail: dls14 at easychair.org


  Publicity chair

Edd Barrett, King's College London, UK


  Programme committee

Gilad Bracha, Google, US
Jonathan Edwards, MIT, US
Robert Hirschfeld, Hasso-Plattner-Institut Potsdam, DE
Roberto Ierusalimschy, PUC-Rio, BR
Sergio Maffeis, Imperial College London, UK
Stefan Marr, INRIA, FR
Oscar Nierstrasz, University of Bern, CH
James Noble, Victoria University of Wellington, NZ
Shriram Krishnamurthi, Brown University, US
Chris Seaton, University of Manchester, UK
Nikolai Tillmann, Microsoft Research, US
Sam Tobin-Hochstadt, Indiana University, US
Jan Vitek, Purdue University, US
Christian Wimmer, Oracle Labs, US
Peng Wu, IBM Research, US

From mak at issuu.com  Fri Apr  4 15:05:26 2014
From: mak at issuu.com (Martin Koch)
Date: Fri, 4 Apr 2014 15:05:26 +0200
Subject: [pypy-dev] Disabling garbage collection
Message-ID: <CACyk+CK21sk3vS6MivRKqSYicym-+6to1Z=FmN1MdNdXM-+WoA@mail.gmail.com>

Hi List

Is there a simple way of building a version of pypy where garbage
collection of the old generation is disabled? I have a very large, fairly
static graph in memory, and I would rather not traverse it all on major
collections.

I don't care too much if there are leaks, since my application runs on a
host with lots of ram, can be restarted fairly often, and real-time
performance is preferable to blocking because of garbage collection.

I have tried just removing the body of
incminimark.py/major_collection_step(), but this causes pypy to freeze up
after a while.

Thanks,
/Martin Koch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140404/dfcd1476/attachment.html>

From arigo at tunes.org  Fri Apr  4 15:42:05 2014
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 4 Apr 2014 15:42:05 +0200
Subject: [pypy-dev] Disabling garbage collection
In-Reply-To: <CACyk+CK21sk3vS6MivRKqSYicym-+6to1Z=FmN1MdNdXM-+WoA@mail.gmail.com>
References: <CACyk+CK21sk3vS6MivRKqSYicym-+6to1Z=FmN1MdNdXM-+WoA@mail.gmail.com>
Message-ID: <CAMSv6X0D877XRGB1RSxrrSNOLUi3N89mxuqLDx5KkHmWWuvqJQ@mail.gmail.com>

Hi Martin,

On 4 April 2014 15:05, Martin Koch <mak at issuu.com> wrote:
> Is there a simple way of building a version of pypy where garbage collection
> of the old generation is disabled?

It should be enough to tweak some environment variables.  Try to set

    PYPY_GC_MIN=999999GB


A bient?t,

Armin.

From techtonik at gmail.com  Fri Apr  4 17:04:48 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 4 Apr 2014 18:04:48 +0300
Subject: [pypy-dev] Python Execution Contexts Ch.3 - High Level
Message-ID: <CAPkN8xKivhv_mdLQn0xanMFs=-jM=H=U1h-U6tnVGw0aGthSYA@mail.gmail.com>

Hi,

I am proud to present the picture that shows various
places where Python code can be executed. 51k
picture that worths 1k of words.

https://docs.google.com/drawings/d/1B3ktR-htmql9xYeFeqTU6L1yefI9gqLiQ7uIr7gqRIw/edit

Trying to figure out different entrypoints for executing
Python code and context available for Python code
in these entrypoints. Would be nice to get some
feedback on this diagram. I hope it is still better than
text.


I didn't compare __builtins__ - that's the task better
done with HTML table generator, but there are two
questions about Python/PyPy console context.

pypy-2.2.1 >>> dir()
['__builtins__', '__doc__', '__name__']

python-2.7.6 >>> dir()
['__builtins__', '__doc__', '__name__', '__package__']


1. Why there is __package__ that can not be anything
    but None?

I consider this to be a bug in Python 2.x, because I don't
see how this should be used in console, so I don't mind
that PyPy doesn't implement it. But then comes another
question.

2. If PyPy doesn't implement useless console context
    properties like __package__, why there is __doc__
    entry that can not be anything but None?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140404/0c76157f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Python Execution Contexts (High Level).png
Type: image/png
Size: 52073 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140404/0c76157f/attachment-0001.png>

From arigo at tunes.org  Sat Apr  5 09:29:26 2014
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 5 Apr 2014 09:29:26 +0200
Subject: [pypy-dev] Python Execution Contexts Ch.3 - High Level
In-Reply-To: <CAPkN8xKivhv_mdLQn0xanMFs=-jM=H=U1h-U6tnVGw0aGthSYA@mail.gmail.com>
References: <CAPkN8xKivhv_mdLQn0xanMFs=-jM=H=U1h-U6tnVGw0aGthSYA@mail.gmail.com>
Message-ID: <CAMSv6X0HcYhF+7EHuV6B4H0vZV9Zb6B_XQUaFeMo+ahTcjX8qg@mail.gmail.com>

Hi Anatoly,

On 4 April 2014 17:04, anatoly techtonik <techtonik at gmail.com> wrote:
> pypy-2.2.1 >>> dir()
> ['__builtins__', '__doc__', '__name__']
>
> python-2.7.6 >>> dir()
> ['__builtins__', '__doc__', '__name__', '__package__']

Please report this as a bug on https://bugs.pypy.org if you feel it's
important enough.


Armin.

From dimaqq at gmail.com  Wed Apr  9 20:29:26 2014
From: dimaqq at gmail.com (Dima Tisnek)
Date: Wed, 9 Apr 2014 20:29:26 +0200
Subject: [pypy-dev] should os.fdopen(fd-of-a-directory, "r") succeed?
Message-ID: <CAGGBzXJCnhyM3QBM9HEJ1r0MjUQa2zg4jkum8Qc=o63aRc1Rgw@mail.gmail.com>

it fails on py 2.7 (with a bug), fails correctly in py 3.3 and
succeeds in pypy 2.2:

>>>> import os
>>>> os.open("/", os.O_RDONLY)
3
>>>> os.fdopen(3, "r")
<open file '<fdopen>', mode 'r' at 0x0000000105b71830>
>>>> f = _
>>>> f.read(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 21] Is a directory: '<fdopen>'
>>>>

From alex.gaynor at gmail.com  Wed Apr  9 20:32:55 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 9 Apr 2014 14:32:55 -0400
Subject: [pypy-dev] should os.fdopen(fd-of-a-directory, "r") succeed?
In-Reply-To: <CAGGBzXJCnhyM3QBM9HEJ1r0MjUQa2zg4jkum8Qc=o63aRc1Rgw@mail.gmail.com>
References: <CAGGBzXJCnhyM3QBM9HEJ1r0MjUQa2zg4jkum8Qc=o63aRc1Rgw@mail.gmail.com>
Message-ID: <CAFRnB2XsBaK9qk-a0Q4ZkpOsuPHj+bAqhn2VHLid2J3Bd=TSMA@mail.gmail.com>

It fails under CPython 2.7 as well, I guess we should fix it.

Alex


On Wed, Apr 9, 2014 at 2:29 PM, Dima Tisnek <dimaqq at gmail.com> wrote:

> it fails on py 2.7 (with a bug), fails correctly in py 3.3 and
> succeeds in pypy 2.2:
>
> >>>> import os
> >>>> os.open("/", os.O_RDONLY)
> 3
> >>>> os.fdopen(3, "r")
> <open file '<fdopen>', mode 'r' at 0x0000000105b71830>
> >>>> f = _
> >>>> f.read(1)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> IOError: [Errno 21] Is a directory: '<fdopen>'
> >>>>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140409/c49f6f92/attachment.html>

From bdkearns at gmail.com  Wed Apr  9 20:50:03 2014
From: bdkearns at gmail.com (Brian Kearns)
Date: Wed, 9 Apr 2014 11:50:03 -0700
Subject: [pypy-dev] should os.fdopen(fd-of-a-directory, "r") succeed?
In-Reply-To: <CAFRnB2XsBaK9qk-a0Q4ZkpOsuPHj+bAqhn2VHLid2J3Bd=TSMA@mail.gmail.com>
References: <CAGGBzXJCnhyM3QBM9HEJ1r0MjUQa2zg4jkum8Qc=o63aRc1Rgw@mail.gmail.com>
 <CAFRnB2XsBaK9qk-a0Q4ZkpOsuPHj+bAqhn2VHLid2J3Bd=TSMA@mail.gmail.com>
Message-ID: <CACw2m-Z=Qs2svqHiK7CXgG=xBMXy5xcirEtsOkpp+8PsTKJiJQ@mail.gmail.com>

Fixed in fcb0695ec986


On Wed, Apr 9, 2014 at 11:32 AM, Alex Gaynor <alex.gaynor at gmail.com> wrote:

> It fails under CPython 2.7 as well, I guess we should fix it.
>
> Alex
>
>
> On Wed, Apr 9, 2014 at 2:29 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
>
>> it fails on py 2.7 (with a bug), fails correctly in py 3.3 and
>> succeeds in pypy 2.2:
>>
>> >>>> import os
>> >>>> os.open("/", os.O_RDONLY)
>> 3
>> >>>> os.fdopen(3, "r")
>> <open file '<fdopen>', mode 'r' at 0x0000000105b71830>
>> >>>> f = _
>> >>>> f.read(1)
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> IOError: [Errno 21] Is a directory: '<fdopen>'
>> >>>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right
> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: 125F 5C67 DFE9 4084
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140409/65b5a897/attachment.html>

From alex.gaynor at gmail.com  Wed Apr  9 20:57:13 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 9 Apr 2014 14:57:13 -0400
Subject: [pypy-dev] should os.fdopen(fd-of-a-directory, "r") succeed?
In-Reply-To: <CACw2m-Z=Qs2svqHiK7CXgG=xBMXy5xcirEtsOkpp+8PsTKJiJQ@mail.gmail.com>
References: <CAGGBzXJCnhyM3QBM9HEJ1r0MjUQa2zg4jkum8Qc=o63aRc1Rgw@mail.gmail.com>
 <CAFRnB2XsBaK9qk-a0Q4ZkpOsuPHj+bAqhn2VHLid2J3Bd=TSMA@mail.gmail.com>
 <CACw2m-Z=Qs2svqHiK7CXgG=xBMXy5xcirEtsOkpp+8PsTKJiJQ@mail.gmail.com>
Message-ID: <CAFRnB2Ucog4=H296cuhNpeLjkSHPRLCCnBWiRAebrHMSrT57Dw@mail.gmail.com>

Awesome -- thanks Brian!

Alex


On Wed, Apr 9, 2014 at 2:50 PM, Brian Kearns <bdkearns at gmail.com> wrote:

> Fixed in fcb0695ec986
>
>
> On Wed, Apr 9, 2014 at 11:32 AM, Alex Gaynor <alex.gaynor at gmail.com>wrote:
>
>> It fails under CPython 2.7 as well, I guess we should fix it.
>>
>> Alex
>>
>>
>> On Wed, Apr 9, 2014 at 2:29 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
>>
>>> it fails on py 2.7 (with a bug), fails correctly in py 3.3 and
>>> succeeds in pypy 2.2:
>>>
>>> >>>> import os
>>> >>>> os.open("/", os.O_RDONLY)
>>> 3
>>> >>>> os.fdopen(3, "r")
>>> <open file '<fdopen>', mode 'r' at 0x0000000105b71830>
>>> >>>> f = _
>>> >>>> f.read(1)
>>> Traceback (most recent call last):
>>>   File "<stdin>", line 1, in <module>
>>> IOError: [Errno 21] Is a directory: '<fdopen>'
>>> >>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>
>>
>>
>> --
>> "I disapprove of what you say, but I will defend to the death your right
>> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
>> "The people's good is the highest law." -- Cicero
>> GPG Key fingerprint: 125F 5C67 DFE9 4084
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>


-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140409/cbc46fac/attachment.html>

From cfbolz at gmx.de  Wed Apr  9 18:15:43 2014
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 09 Apr 2014 18:15:43 +0200
Subject: [pypy-dev] "aborts" in unroll
Message-ID: <5345722F.7010905@gmx.de>

Hi all,

I found the reason why unroll often produces uncounted "aborts". It's
the constructor here:


class InvalidLoop(JitException):
    """Raised when the optimize*.py detect that the loop that
    we are trying to build cannot possibly make sense as a
    long-running loop (e.g. it cannot run 2 complete iterations)."""

    def __init__(self, msg='?'):
        debug_start("jit-abort")
        debug_print(msg)
        debug_stop("jit-abort")
        self.msg = msg


unroll uses InvalidLoop internally, which does not propagate up. I am
fixing that on my branch.

Cheers,

Carl Friedrich

From matti.picus at gmail.com  Wed Apr  9 22:10:20 2014
From: matti.picus at gmail.com (matti picus)
Date: Wed, 9 Apr 2014 23:10:20 +0300
Subject: [pypy-dev] msvc and floating point compiler flags
Message-ID: <CAO8bCgjB4MDWLAw1kpESuzMY0atbxByg0LuEkShRuaP6WpdmUQ@mail.gmail.com>

looking at this test failure
http://buildbot.pypy.org/summary/longrepr?testname=test_math&builder=own-win-x86-32&build=86&mod=jit.backend.x86.test.test_zmath
which tests that atan2(0.23/1.23) == 1.3.... via OP_FLOAT_EQ(double value,
double value)
OP_FLOAT_EQ is a compiler define to (a == b), which is tricky with double
values.
Here is a nice discussion of the issue on MSVC
http://www.gamasutra.com/view/news/167402/Indepth_Intermediate_floatingpoint_precision.php
The link even has a flow chart for choosing msvc or intel compiler flags.
The test passes if I add /fp:fast to the msvc compile flag
Would anyone object if I add this to the default MSVC flags?

FWIW, the "fast" is misleading, apparently it can be slower than the
default for some cases.
Matti
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140409/8a164a72/attachment.html>

From kennylevinsen at gmail.com  Wed Apr  9 22:44:02 2014
From: kennylevinsen at gmail.com (Kenny Lasse Hoff Levinsen)
Date: Wed, 9 Apr 2014 22:44:02 +0200
Subject: [pypy-dev] msvc and floating point compiler flags
In-Reply-To: <CAO8bCgjB4MDWLAw1kpESuzMY0atbxByg0LuEkShRuaP6WpdmUQ@mail.gmail.com>
References: <CAO8bCgjB4MDWLAw1kpESuzMY0atbxByg0LuEkShRuaP6WpdmUQ@mail.gmail.com>
Message-ID: <95F730A3-0DEC-40E9-8755-9771B9D607D3@gmail.com>

Sorry to butt in here, but? I just saw the comparison, and all my alarm bells with regards to extended precision and other platform-dependent magic started ringing.
What kind of voodoo and virgin goat blood (Am I doing the occult rituals right?) was needed to make that work on other compilers?

Jokes aside, adding /fp:fast should tell the compiler to ignore all rules, and makes the result much more platform dependent than they already were. Literals are no longer trustable either.
MS docs: http://msdn.microsoft.com/en-us/library/e7s85ffb.aspx
SO note on fast-math: http://stackoverflow.com/a/16069874

Shouldn?t the comparison just account for some lack of precision? I suggest having the macro do (a > b*0.99999 && a < b*0.00001) or the likes, after deciding how many digits of precision is wanted for that specific test. It looks stupid (And introduces even more floating point arithmetics!), but it works.

But maybe you should wait for another reply, I may just be too tired to function.

Kenny

On 09 Apr 2014, at 22:10, matti picus <matti.picus at gmail.com> wrote:

> looking at this test failure
> http://buildbot.pypy.org/summary/longrepr?testname=test_math&builder=own-win-x86-32&build=86&mod=jit.backend.x86.test.test_zmath
> which tests that atan2(0.23/1.23) == 1.3.... via OP_FLOAT_EQ(double value, double value)
> OP_FLOAT_EQ is a compiler define to (a == b), which is tricky with double values.
> Here is a nice discussion of the issue on MSVC
> http://www.gamasutra.com/view/news/167402/Indepth_Intermediate_floatingpoint_precision.php
> The link even has a flow chart for choosing msvc or intel compiler flags.
> The test passes if I add /fp:fast to the msvc compile flag
> Would anyone object if I add this to the default MSVC flags?
> 
> FWIW, the "fast" is misleading, apparently it can be slower than the default for some cases.
> Matti
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140409/cfa60a6d/attachment.html>

From matti.picus at gmail.com  Thu Apr 10 00:00:34 2014
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 10 Apr 2014 01:00:34 +0300
Subject: [pypy-dev] msvc and floating point compiler flags
In-Reply-To: <CAO8bCgjB4MDWLAw1kpESuzMY0atbxByg0LuEkShRuaP6WpdmUQ@mail.gmail.com>
References: <CAO8bCgjB4MDWLAw1kpESuzMY0atbxByg0LuEkShRuaP6WpdmUQ@mail.gmail.com>
Message-ID: <5345C302.5070805@gmail.com>

To answer my own question:
It seems cpython, pypy.exe and jitted code (for both msvc and gcc) all
use a floating point model that correctly can do == for doubles. The
failure is only in the zmath jit backend test. I am now looking for how
the test sets up a floating point precision model that is not only
triggers the failure, but is inconsistent with translation.
Matti

On 9/04/2014 11:10 PM, matti picus wrote:
> looking at this test failure
> http://buildbot.pypy.org/summary/longrepr?testname=test_math&builder=own-win-x86-32&build=86&mod=jit.backend.x86.test.test_zmath
> which tests that atan2(0.23/1.23) == 1.3.... via OP_FLOAT_EQ(double
> value, double value)
> OP_FLOAT_EQ is a compiler define to (a == b), which is tricky with
> double values.
> Here is a nice discussion of the issue on MSVC
> http://www.gamasutra.com/view/news/167402/Indepth_Intermediate_floatingpoint_precision.php
> The link even has a flow chart for choosing msvc or intel compiler flags.
> The test passes if I add /fp:fast to the msvc compile flag
> Would anyone object if I add this to the default MSVC flags?
> 
> FWIW, the "fast" is misleading, apparently it can be slower than the
> default for some cases.
> Matti

From naftaliharris at gmail.com  Thu Apr 10 21:05:33 2014
From: naftaliharris at gmail.com (Naftali Harris)
Date: Thu, 10 Apr 2014 12:05:33 -0700
Subject: [pypy-dev] CPython compatibility in the fractions module
Message-ID: <CADFNaB5UscuYia9CzP8C+1aTok-WGnsmy9j5h+rPVO7EPeKZbg@mail.gmail.com>

Hi everyone,

I'd like to report a compatibility issue between CPython and PyPy in the
fractions module.  In short, CPython coerces Fraction objects into int's in
strings formatted with "%d", but PyPy throws a TypeError.  Here's an
example reproducing this:

~/Downloads/pypy-2.2.1-linux64/bin$ ./pypy
Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 18:48:13)
[PyPy 2.2.1 with GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``<arigato> fijal: I'm sure
there
is tons of unspecified context information that I should ideally also be
aware
of''
>>>> from fractions import Fraction
>>>> print "%d" % Fraction(1, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type for int(): 'Fraction'
>>>>
~/Downloads/pypy-2.2.1-linux64/bin$ python
Python 2.7.3 (default, Feb 27 2014, 19:37:34)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from fractions import Fraction
>>> print "%d" % Fraction(1, 1)
1
>>>

Thanks for the great work you all do!

Naftali
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140410/14ce4679/attachment.html>

From arigo at tunes.org  Thu Apr 10 21:37:40 2014
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 10 Apr 2014 21:37:40 +0200
Subject: [pypy-dev] CPython compatibility in the fractions module
In-Reply-To: <CADFNaB5UscuYia9CzP8C+1aTok-WGnsmy9j5h+rPVO7EPeKZbg@mail.gmail.com>
References: <CADFNaB5UscuYia9CzP8C+1aTok-WGnsmy9j5h+rPVO7EPeKZbg@mail.gmail.com>
Message-ID: <CAMSv6X37Xv4ck3g5ZAY9PWjcZcKzzS69er3rjThrMOZ6i-Rmog@mail.gmail.com>

Hi Naftali,

On 10 April 2014 21:05, Naftali Harris <naftaliharris at gmail.com> wrote:
> I'd like to report a compatibility issue between CPython and PyPy in the
> fractions module.  In short, CPython coerces Fraction objects into int's in
> strings formatted with "%d", but PyPy throws a TypeError.

It seems that  "%d" % x  throws a TypeError on CPython, except if x is
an instance of a class that defines the __int__ method (so far,
reasonable, done in PyPy too)... or else, if both the __float__ and
the __trunc__ methods are defined.  Defining only one of them is not
sufficient.

Here, I say: *what?*

Anybody has got a clue about this PHP-esque worth-a-CPython-bug-report
misbehavior?

(Obviously the class Fraction just happens to define not __int__, but
both __float__ and __trunc__.  It all looks like a
design-by-chance...)


A bient?t,

Armin.

From arigo at tunes.org  Thu Apr 10 21:37:40 2014
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 10 Apr 2014 21:37:40 +0200
Subject: [pypy-dev] CPython compatibility in the fractions module
In-Reply-To: <CADFNaB5UscuYia9CzP8C+1aTok-WGnsmy9j5h+rPVO7EPeKZbg@mail.gmail.com>
References: <CADFNaB5UscuYia9CzP8C+1aTok-WGnsmy9j5h+rPVO7EPeKZbg@mail.gmail.com>
Message-ID: <CAMSv6X37Xv4ck3g5ZAY9PWjcZcKzzS69er3rjThrMOZ6i-Rmog@mail.gmail.com>

Hi Naftali,

On 10 April 2014 21:05, Naftali Harris <naftaliharris at gmail.com> wrote:
> I'd like to report a compatibility issue between CPython and PyPy in the
> fractions module.  In short, CPython coerces Fraction objects into int's in
> strings formatted with "%d", but PyPy throws a TypeError.

It seems that  "%d" % x  throws a TypeError on CPython, except if x is
an instance of a class that defines the __int__ method (so far,
reasonable, done in PyPy too)... or else, if both the __float__ and
the __trunc__ methods are defined.  Defining only one of them is not
sufficient.

Here, I say: *what?*

Anybody has got a clue about this PHP-esque worth-a-CPython-bug-report
misbehavior?

(Obviously the class Fraction just happens to define not __int__, but
both __float__ and __trunc__.  It all looks like a
design-by-chance...)


A bient?t,

Armin.

From n210241048576 at gmail.com  Sun Apr 13 05:10:45 2014
From: n210241048576 at gmail.com (Robert Grosse)
Date: Sat, 12 Apr 2014 20:10:45 -0700
Subject: [pypy-dev] performance problems with Krakatau
In-Reply-To: <CAK5idxR_MRvJqFQq6TTVzeMCJ81VH4Oki9XvNLN_A5sw+SeLXg@mail.gmail.com>
References: <CALJjGiu4MyuQJzc3px7x3K-2CFuPsijpOq9nGVfyMYDE+TxoKA@mail.gmail.com>
 <CAK5idxQ=H93SgqbZ9ZyzT6deQJwQ5ydrCY3YTHEB0hcN-g5bVQ@mail.gmail.com>
 <CALJjGitWaXv=W+k5g-=hZd6qhZ-A3X=neA0EJATyAM15k4L+8g@mail.gmail.com>
 <CAK5idxQ8ozhnVi7bkLMQ9Vvb1R3nbdnNvHK5BjySOvdmHR+5Qw@mail.gmail.com>
 <CALJjGitO8frWBG_NwUm=K7iBEYWQerXjnOC+QZc_bP7++WDLgA@mail.gmail.com>
 <CAK5idxQB7fjmfwXoGFLhpZCCs05Dw3r30rNs2o7n20aeNjrwDw@mail.gmail.com>
 <CALJjGiskJPwXB2yK9L3kkukhpd=V3SCCaY0ok+Sdp-XgQdz6DA@mail.gmail.com>
 <CAK5idxRmN5OM8k7RxYwzYJst8fEVQRaMqRn3OyCj2cj53vGFhg@mail.gmail.com>
 <CAK5idxR_MRvJqFQq6TTVzeMCJ81VH4Oki9XvNLN_A5sw+SeLXg@mail.gmail.com>
Message-ID: <CALJjGitW8E8K8ExHmzz5tNdyEO6sBRZ2NGdjLxn3ZAdWG4BNSQ@mail.gmail.com>

Hi again,

I recently updated Pypy to (pypy-c-jit-70483-2d8eaa5f5079-win32), and
Pypy's performance is much better now. I also addressed the previously
mentioned issues in Krakatau so it is faster on both CPython and Pypy.
However, I have noticed that there are still some cases in which CPython
outperforms Pypy.

I created a benchmark using one class I noticed with the biggest discrepancy

https://github.com/Storyyeller/Krakatau.git commit
88a5a24deb3a8e6d0d92aca2052ea1db6a7274a0

You can run it via
python Krakatau\benchmark.py -path whatever\rt.jar
where you pass the path to your JRE's rt.jar as appropriate

This benchmark is based on decompiling a single class,
sun/text/normalizer/Utility from the JRE. The benchmark decompiles the
class 40 times beforehand to warmup the jit and then measures the time
taken to decompile it 200 times using time.time(). I recorded memory usage
manually via the Windows Task Manager using Peak Working Set. I used the
Java 7u51 JRE, but I expect any version to be the same as I doubt the class
changed much.

CPython: 202.8 seconds, 47.5mb
Pypy: 284.3 seconds, 229.2mb

The memory usage isn't too concerning to me, since I imagine that a JIT has
higher fixed overhead, but I find it strange that CPython also executes
faster for this class, since it is all pure Python CPU bound computation.




On Thu, Jan 16, 2014 at 5:51 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> Hi Robert.
>
> This is going to be a long mail, so bear with me :)
>
> The first take away is that pypy warmup is atrocious (that's
> unimpressive, but you might be delighted to hear I'm working on it
> right now, except I'm writing this mail). It's quite a bit of work, so
> it might or might not make it to the next pypy release. We also don't
> know how well it'll work.
>
> The runs that I have now, when running 3 times in the same process
> look like this (this includes other improvements mentioned later):
>
> 46s 32s 29s (cpython takes always 29s)
>
> Now, this is far from ideal and we're working on making it better (in
> fact it's a very useful benchmark), but I can pinpoint some stuff that
> we will fix and some stuff we won't fix in the near future. One thing
> that I've already fixed today is loops over tuple when doing x in
> tuple (so tuple.__contains__).
>
> One of the problems with this code is that I don't think it's very
> efficient. While that's not a good reason to be slower than cpython,
> it gives you an upper bound on what can be optimized away. Example
> (from java/structuring.py):
>
> new = new if old is None else tuple(x for x in old if x in new)
>
> now note that this has a complexity of O(n^2), because you're
> iterating for all of the one tuple and then for each over all of the
> elements of the other tuple.
>
> Another example:
>
> return [x for x in zip(*map(self._doms.get, nodes)) if
> len(set(x))==1][-1][0]
>
> this creates quite a few lists, while all it wants to do is to grab
> the last one.
>
> Those tiny loops are found a bit everywhere. I think more consistent
> data structures will make it a lot faster on both CPython and PyPy.
>
> From our side, we'll improve generator iterators today and warmup some
> time in the not-so-near future.
>
> Speaking of which - memory consumptions is absolutely atrocious. It's
> a combination of JIT using too much memory, generator iterators not
> being cleaned correctly *and* some bug that prevents JIT loops from
> being freed. we'll deal with all of it, give us some time (that said,
> the memory consumption *will* be bigger than cpython, but hopefully by
> not that much).
>
> I'm sorry I can't help you as much as I wanted
>
> Cheers,
> fijal
>
>
> On Thu, Jan 16, 2014 at 10:50 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> > On Wed, Jan 15, 2014 at 7:20 PM, Robert Grosse <n210241048576 at gmail.com>
> wrote:
> >> Oh sorry, I forgot about that.
> >>
> >> You need to find the rt.jar from your Java installation and pass the
> path on
> >> the command line. For example, if it's located in C:\Program
> >> Files\Java\jre7\lib, you could do
> >> python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar -path
> >> "C:\Program Files\Java\jre7\lib\rt.jar"
> >> Obviously on Linux it will be somewhere else. It shouldn't really matter
> >> which version of Java you have since the standard library is pretty
> stable..
> >
> > Thanks, I'm looking into it. Would you mind if we add Krakatau as a
> > benchmark for our nightlies?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140412/066e5ebc/attachment.html>

From fijall at gmail.com  Sun Apr 13 16:21:56 2014
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 13 Apr 2014 16:21:56 +0200
Subject: [pypy-dev] performance problems with Krakatau
In-Reply-To: <CALJjGitW8E8K8ExHmzz5tNdyEO6sBRZ2NGdjLxn3ZAdWG4BNSQ@mail.gmail.com>
References: <CALJjGiu4MyuQJzc3px7x3K-2CFuPsijpOq9nGVfyMYDE+TxoKA@mail.gmail.com>
 <CAK5idxQ=H93SgqbZ9ZyzT6deQJwQ5ydrCY3YTHEB0hcN-g5bVQ@mail.gmail.com>
 <CALJjGitWaXv=W+k5g-=hZd6qhZ-A3X=neA0EJATyAM15k4L+8g@mail.gmail.com>
 <CAK5idxQ8ozhnVi7bkLMQ9Vvb1R3nbdnNvHK5BjySOvdmHR+5Qw@mail.gmail.com>
 <CALJjGitO8frWBG_NwUm=K7iBEYWQerXjnOC+QZc_bP7++WDLgA@mail.gmail.com>
 <CAK5idxQB7fjmfwXoGFLhpZCCs05Dw3r30rNs2o7n20aeNjrwDw@mail.gmail.com>
 <CALJjGiskJPwXB2yK9L3kkukhpd=V3SCCaY0ok+Sdp-XgQdz6DA@mail.gmail.com>
 <CAK5idxRmN5OM8k7RxYwzYJst8fEVQRaMqRn3OyCj2cj53vGFhg@mail.gmail.com>
 <CAK5idxR_MRvJqFQq6TTVzeMCJ81VH4Oki9XvNLN_A5sw+SeLXg@mail.gmail.com>
 <CALJjGitW8E8K8ExHmzz5tNdyEO6sBRZ2NGdjLxn3ZAdWG4BNSQ@mail.gmail.com>
Message-ID: <CAK5idxRYndyuFpCqpcEA-+gJesiJ_q0Uc1_HE+oTzUumV6CPNg@mail.gmail.com>

it seems to be a major issue with our specialization, where we compile
tons and tons of small bridges that bring no value. I commited little
bit of improvements, but it's clearly not done yet. I also did small
adjustements  to use small instances instead of dicts in cases where
you have a small set of well known keys:
http://paste.pound-python.org/show/Rh7p0uP3Is8C8VA6LyNN/

I'll look into it some more, thanks for a useful benchmark.

On Sun, Apr 13, 2014 at 5:10 AM, Robert Grosse <n210241048576 at gmail.com> wrote:
> Hi again,
>
> I recently updated Pypy to (pypy-c-jit-70483-2d8eaa5f5079-win32), and Pypy's
> performance is much better now. I also addressed the previously mentioned
> issues in Krakatau so it is faster on both CPython and Pypy. However, I have
> noticed that there are still some cases in which CPython outperforms Pypy.
>
> I created a benchmark using one class I noticed with the biggest discrepancy
>
> https://github.com/Storyyeller/Krakatau.git commit
> 88a5a24deb3a8e6d0d92aca2052ea1db6a7274a0
>
> You can run it via
> python Krakatau\benchmark.py -path whatever\rt.jar
> where you pass the path to your JRE's rt.jar as appropriate
>
> This benchmark is based on decompiling a single class,
> sun/text/normalizer/Utility from the JRE. The benchmark decompiles the class
> 40 times beforehand to warmup the jit and then measures the time taken to
> decompile it 200 times using time.time(). I recorded memory usage manually
> via the Windows Task Manager using Peak Working Set. I used the Java 7u51
> JRE, but I expect any version to be the same as I doubt the class changed
> much.
>
> CPython: 202.8 seconds, 47.5mb
> Pypy: 284.3 seconds, 229.2mb
>
> The memory usage isn't too concerning to me, since I imagine that a JIT has
> higher fixed overhead, but I find it strange that CPython also executes
> faster for this class, since it is all pure Python CPU bound computation.
>
>
>
>
> On Thu, Jan 16, 2014 at 5:51 AM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
>>
>> Hi Robert.
>>
>> This is going to be a long mail, so bear with me :)
>>
>> The first take away is that pypy warmup is atrocious (that's
>> unimpressive, but you might be delighted to hear I'm working on it
>> right now, except I'm writing this mail). It's quite a bit of work, so
>> it might or might not make it to the next pypy release. We also don't
>> know how well it'll work.
>>
>> The runs that I have now, when running 3 times in the same process
>> look like this (this includes other improvements mentioned later):
>>
>> 46s 32s 29s (cpython takes always 29s)
>>
>> Now, this is far from ideal and we're working on making it better (in
>> fact it's a very useful benchmark), but I can pinpoint some stuff that
>> we will fix and some stuff we won't fix in the near future. One thing
>> that I've already fixed today is loops over tuple when doing x in
>> tuple (so tuple.__contains__).
>>
>> One of the problems with this code is that I don't think it's very
>> efficient. While that's not a good reason to be slower than cpython,
>> it gives you an upper bound on what can be optimized away. Example
>> (from java/structuring.py):
>>
>> new = new if old is None else tuple(x for x in old if x in new)
>>
>> now note that this has a complexity of O(n^2), because you're
>> iterating for all of the one tuple and then for each over all of the
>> elements of the other tuple.
>>
>> Another example:
>>
>> return [x for x in zip(*map(self._doms.get, nodes)) if
>> len(set(x))==1][-1][0]
>>
>> this creates quite a few lists, while all it wants to do is to grab
>> the last one.
>>
>> Those tiny loops are found a bit everywhere. I think more consistent
>> data structures will make it a lot faster on both CPython and PyPy.
>>
>> From our side, we'll improve generator iterators today and warmup some
>> time in the not-so-near future.
>>
>> Speaking of which - memory consumptions is absolutely atrocious. It's
>> a combination of JIT using too much memory, generator iterators not
>> being cleaned correctly *and* some bug that prevents JIT loops from
>> being freed. we'll deal with all of it, give us some time (that said,
>> the memory consumption *will* be bigger than cpython, but hopefully by
>> not that much).
>>
>> I'm sorry I can't help you as much as I wanted
>>
>> Cheers,
>> fijal
>>
>>
>> On Thu, Jan 16, 2014 at 10:50 AM, Maciej Fijalkowski <fijall at gmail.com>
>> wrote:
>> > On Wed, Jan 15, 2014 at 7:20 PM, Robert Grosse <n210241048576 at gmail.com>
>> > wrote:
>> >> Oh sorry, I forgot about that.
>> >>
>> >> You need to find the rt.jar from your Java installation and pass the
>> >> path on
>> >> the command line. For example, if it's located in C:\Program
>> >> Files\Java\jre7\lib, you could do
>> >> python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar -path
>> >> "C:\Program Files\Java\jre7\lib\rt.jar"
>> >> Obviously on Linux it will be somewhere else. It shouldn't really
>> >> matter
>> >> which version of Java you have since the standard library is pretty
>> >> stable..
>> >
>> > Thanks, I'm looking into it. Would you mind if we add Krakatau as a
>> > benchmark for our nightlies?
>
>

From MMezeul at advaoptical.com  Tue Apr 15 05:59:43 2014
From: MMezeul at advaoptical.com (Mike Mezeul)
Date: Tue, 15 Apr 2014 03:59:43 +0000
Subject: [pypy-dev] PyPy support for PPC processors
Message-ID: <87582991CD435F4B87905E9874C36D654CD76B2C@atl-srv-mail10.atl.advaoptical.com>

Hello;

Does anyone have any insight as to why the support for Power PC processors has stalled?

I'm specifically interested in knowing if it was technical hurdles or just lack of interest. If technical, could someone please elaborate on what the issues are?

Thanks,
Mike Mezeul


Michael Mezeul
Senior Director R&D
ADVA Optical Networking
2301 North Greenville Avenue, Suite 300
Richardson, TX 75082, USA
Phone: +1.972.759.1216
Fax:     +1.972.759.1201
Mobile: +1.214.448.9239
e-mail: mmezeul at advaoptical.com<mailto:mmezeul at advaoptical.com>

www.advaoptical.com<http://www.advaoptical.com/>
Let's ADVANCE

ADVA Optical Networking SE is a European stock corporation ("Societas Europaea") with registered offices at Maerzenquelle 1-3, D-98617 Meiningen, Germany * CEO: Brian L. Protiva, Chief Officers: Dr. Christoph Glingener, Christian Unterberger, Jaswir Singh * Chairman of the Supervisory Board: Anthony Maher * AG Jena HRB 508155 * VAT No. DE 175 446 349


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140415/fab5e072/attachment.html>

From fijall at gmail.com  Tue Apr 15 08:16:18 2014
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 15 Apr 2014 08:16:18 +0200
Subject: [pypy-dev] PyPy support for PPC processors
In-Reply-To: <87582991CD435F4B87905E9874C36D654CD76B2C@atl-srv-mail10.atl.advaoptical.com>
References: <87582991CD435F4B87905E9874C36D654CD76B2C@atl-srv-mail10.atl.advaoptical.com>
Message-ID: <CAK5idxRjZeudBgCb185xP+a5sXQ+hrF_iwUAm0ih+enyLN5sBw@mail.gmail.com>

lack of interest. The PPC stuff is not as widely used as ARM or x86,
so core developers were not that interested.

On Tue, Apr 15, 2014 at 5:59 AM, Mike Mezeul <MMezeul at advaoptical.com> wrote:
> Hello;
>
>
>
> Does anyone have any insight as to why the support for Power PC processors
> has stalled?
>
>
>
> I?m specifically interested in knowing if it was technical hurdles or just
> lack of interest. If technical, could someone please elaborate on what the
> issues are?
>
>
>
> Thanks,
>
> Mike Mezeul
>
>
>
>
>
> Michael Mezeul
>
> Senior Director R&D
>
> ADVA Optical Networking
>
> 2301 North Greenville Avenue, Suite 300
>
> Richardson, TX 75082, USA
>
> Phone: +1.972.759.1216
>
> Fax:     +1.972.759.1201
>
> Mobile: +1.214.448.9239
>
> e-mail: mmezeul at advaoptical.com
>
>
>
> www.advaoptical.com
>
> Let?s ADVANCE
>
>
>
> ADVA Optical Networking SE is a European stock corporation ("Societas
> Europaea") with registered offices at Maerzenquelle 1-3, D-98617 Meiningen,
> Germany * CEO: Brian L. Protiva, Chief Officers: Dr. Christoph Glingener,
> Christian Unterberger, Jaswir Singh * Chairman of the Supervisory Board:
> Anthony Maher * AG Jena HRB 508155 * VAT No. DE 175 446 349
>
>
>
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From arigo at tunes.org  Tue Apr 15 12:55:00 2014
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 15 Apr 2014 12:55:00 +0200
Subject: [pypy-dev] OS/X
Message-ID: <CAMSv6X1DdCxSczFkW1VcWWqF3bUjW8FYz2-jT7hXTh_HH_HD8g@mail.gmail.com>

Hi OS/X'ers,

I see in rpython/translator/platform/darwin.py that we use only
"clang" as the compiler on OS/X.  There was some discussion in #pypy,
as well as a proposal from Andrew Dalke to upgrade the gcc on the OS/X
buildbot, which go along the lines of "clang is not necessarily always
better than gcc on OS/X".  So I would like that the RPython toolchain
supports both choices.

Please don't discuss clang-vs-gcc endlessly here, I'm sure there are
people who think that clang is better than gcc and others that have
the opposite view and both camps have strong opinions -- that's not
the point and I don't honestly care.  This e-mail is *only* about
supporting both choices.

Could someone on OS/X check that we can use gcc to translate pypy's?
This can be checked by trying out a smaller example, like
rpython/translator/goal/targetnopstandalone.py .  What fixes do you
need in darwin.py?  Or simply pass some environment variable? Thanks!


A bient?t,

Armin.

From kennylevinsen at gmail.com  Tue Apr 15 15:22:17 2014
From: kennylevinsen at gmail.com (Kenny Lasse Hoff Levinsen)
Date: Tue, 15 Apr 2014 15:22:17 +0200
Subject: [pypy-dev] OS/X
In-Reply-To: <CAMSv6X1DdCxSczFkW1VcWWqF3bUjW8FYz2-jT7hXTh_HH_HD8g@mail.gmail.com>
References: <CAMSv6X1DdCxSczFkW1VcWWqF3bUjW8FYz2-jT7hXTh_HH_HD8g@mail.gmail.com>
Message-ID: <E8CC0A5E-938E-4450-AE82-A7A3D173C605@gmail.com>

The issue is mainly that the bundled GCC is very old (the last GPLv2, IIRC), but I can install a new GCC and try to translate later tonight, unless someone else beats me to it.

Kenny / joushou

> On 15/04/2014, at 12.55, Armin Rigo <arigo at tunes.org> wrote:
> 
> Hi OS/X'ers,
> 
> I see in rpython/translator/platform/darwin.py that we use only
> "clang" as the compiler on OS/X.  There was some discussion in #pypy,
> as well as a proposal from Andrew Dalke to upgrade the gcc on the OS/X
> buildbot, which go along the lines of "clang is not necessarily always
> better than gcc on OS/X".  So I would like that the RPython toolchain
> supports both choices.
> 
> Please don't discuss clang-vs-gcc endlessly here, I'm sure there are
> people who think that clang is better than gcc and others that have
> the opposite view and both camps have strong opinions -- that's not
> the point and I don't honestly care.  This e-mail is *only* about
> supporting both choices.
> 
> Could someone on OS/X check that we can use gcc to translate pypy's?
> This can be checked by trying out a smaller example, like
> rpython/translator/goal/targetnopstandalone.py .  What fixes do you
> need in darwin.py?  Or simply pass some environment variable? Thanks!
> 
> 
> A bient?t,
> 
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From dje.gcc at gmail.com  Tue Apr 15 15:49:19 2014
From: dje.gcc at gmail.com (David Edelsohn)
Date: Tue, 15 Apr 2014 09:49:19 -0400
Subject: [pypy-dev] PyPy support for PPC processors
In-Reply-To: <CAK5idxRjZeudBgCb185xP+a5sXQ+hrF_iwUAm0ih+enyLN5sBw@mail.gmail.com>
References: <87582991CD435F4B87905E9874C36D654CD76B2C@atl-srv-mail10.atl.advaoptical.com>
 <CAK5idxRjZeudBgCb185xP+a5sXQ+hrF_iwUAm0ih+enyLN5sBw@mail.gmail.com>
Message-ID: <CAGWvnym9NV4RvHxz5iD3pqzETfOGj-=a19Rbkwgu=rLGq466Vg@mail.gmail.com>

Some students from Unicamp and I are working to update the PowerPC branch.

The earlier effort stalled when the port encountered a garbage
collection corruption that neither I nor the PyPy core developers
could solve. Some additional GC debugging aids have since been added,
but by that time, I and the original student had run out of time.

- David

On Tue, Apr 15, 2014 at 2:16 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> lack of interest. The PPC stuff is not as widely used as ARM or x86,
> so core developers were not that interested.
>
> On Tue, Apr 15, 2014 at 5:59 AM, Mike Mezeul <MMezeul at advaoptical.com> wrote:

>> Does anyone have any insight as to why the support for Power PC processors
>> has stalled?

>> I?m specifically interested in knowing if it was technical hurdles or just
>> lack of interest. If technical, could someone please elaborate on what the
>> issues are?

From arigo at tunes.org  Tue Apr 15 16:13:47 2014
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 15 Apr 2014 16:13:47 +0200
Subject: [pypy-dev] OS/X
In-Reply-To: <E8CC0A5E-938E-4450-AE82-A7A3D173C605@gmail.com>
References: <CAMSv6X1DdCxSczFkW1VcWWqF3bUjW8FYz2-jT7hXTh_HH_HD8g@mail.gmail.com>
 <E8CC0A5E-938E-4450-AE82-A7A3D173C605@gmail.com>
Message-ID: <CAMSv6X0b9dYpT6bpV9spNZqeQXyF3XQpJnbT2u+P_7HtWwCm0w@mail.gmail.com>

Hi Kenny,

On 15 April 2014 15:22, Kenny Lasse Hoff Levinsen
<kennylevinsen at gmail.com> wrote:
> The issue is mainly that the bundled GCC is very old (the last GPLv2, IIRC), but I can install a new GCC and try to translate later tonight, unless someone else beats me to it.

Yes, that's what I meant (sorry if I was not clear): a recent GCC.
Andrew installed gcc 4.8 through MacPorts.  Thanks for helping!


A bient?t,

Armin.

From MMezeul at advaoptical.com  Tue Apr 15 16:15:31 2014
From: MMezeul at advaoptical.com (Mike Mezeul)
Date: Tue, 15 Apr 2014 14:15:31 +0000
Subject: [pypy-dev] PyPy support for PPC processors
In-Reply-To: <CAGWvnym9NV4RvHxz5iD3pqzETfOGj-=a19Rbkwgu=rLGq466Vg@mail.gmail.com>
References: <87582991CD435F4B87905E9874C36D654CD76B2C@atl-srv-mail10.atl.advaoptical.com>
 <CAK5idxRjZeudBgCb185xP+a5sXQ+hrF_iwUAm0ih+enyLN5sBw@mail.gmail.com>
 <CAGWvnym9NV4RvHxz5iD3pqzETfOGj-=a19Rbkwgu=rLGq466Vg@mail.gmail.com>
Message-ID: <87582991CD435F4B87905E9874C36D654CD777FC@atl-srv-mail10.atl.advaoptical.com>

Very interesting. We "may" have an interest in it on PPC as well and, if so, perhaps we can share our findings and see if we might get the branch moving again.

- Mike

-----Original Message-----
From: David Edelsohn [mailto:dje.gcc at gmail.com] 
Sent: Tuesday, April 15, 2014 8:49 AM
To: Maciej Fijalkowski
Cc: Mike Mezeul; pypy-dev at python.org; Ivan Sichmann; Andre Borba
Subject: Re: [pypy-dev] PyPy support for PPC processors

Some students from Unicamp and I are working to update the PowerPC branch.

The earlier effort stalled when the port encountered a garbage collection corruption that neither I nor the PyPy core developers could solve. Some additional GC debugging aids have since been added, but by that time, I and the original student had run out of time.

- David

On Tue, Apr 15, 2014 at 2:16 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> lack of interest. The PPC stuff is not as widely used as ARM or x86, 
> so core developers were not that interested.
>
> On Tue, Apr 15, 2014 at 5:59 AM, Mike Mezeul <MMezeul at advaoptical.com> wrote:

>> Does anyone have any insight as to why the support for Power PC 
>> processors has stalled?

>> I?m specifically interested in knowing if it was technical hurdles or 
>> just lack of interest. If technical, could someone please elaborate 
>> on what the issues are?

From kennylevinsen at gmail.com  Tue Apr 15 19:21:02 2014
From: kennylevinsen at gmail.com (Kenny Lasse Hoff Levinsen)
Date: Tue, 15 Apr 2014 19:21:02 +0200
Subject: [pypy-dev] OS/X
In-Reply-To: <CAMSv6X0b9dYpT6bpV9spNZqeQXyF3XQpJnbT2u+P_7HtWwCm0w@mail.gmail.com>
References: <CAMSv6X1DdCxSczFkW1VcWWqF3bUjW8FYz2-jT7hXTh_HH_HD8g@mail.gmail.com>
 <E8CC0A5E-938E-4450-AE82-A7A3D173C605@gmail.com>
 <CAMSv6X0b9dYpT6bpV9spNZqeQXyF3XQpJnbT2u+P_7HtWwCm0w@mail.gmail.com>
Message-ID: <3E04822A-13EF-4A4D-9494-26ACB7F92480@gmail.com>

Hi Armin,

I just built on Mac OS 10.9 using gcc-4.9 from Homebrew. Other than a bit of fooling around (Apparently, setting DEFAULT_CC in darwin.py is a bad idea. Things fall apart.), it translated the nop example just fine. It?s also in the process of translating pypy, just for kicks. I?m translating from a fresh default.

I only set CC=gcc-4.9 - no changes required, which is good!

Kenny

On 15 Apr 2014, at 16:13, Armin Rigo <arigo at tunes.org> wrote:

> Hi Kenny,
> 
> On 15 April 2014 15:22, Kenny Lasse Hoff Levinsen
> <kennylevinsen at gmail.com> wrote:
>> The issue is mainly that the bundled GCC is very old (the last GPLv2, IIRC), but I can install a new GCC and try to translate later tonight, unless someone else beats me to it.
> 
> Yes, that's what I meant (sorry if I was not clear): a recent GCC.
> Andrew installed gcc 4.8 through MacPorts.  Thanks for helping!
> 
> 
> A bient?t,
> 
> Armin.


From mantihor at gmail.com  Wed Apr 16 06:31:47 2014
From: mantihor at gmail.com (Bogdan Opanchuk)
Date: Wed, 16 Apr 2014 14:31:47 +1000
Subject: [pypy-dev] RPython as a separate package
Message-ID: <CAFHN2SPL2wDFAV3i0JDEHAHZKxiv529wyDX0n=fyBTvMYL=gEQ@mail.gmail.com>

Hello,

I would like to use the RPython toolchain in my project. The problem
is, RPython is currently hidden inside PyPy and therefore not readily
available. I found this thread
https://mail.python.org/pipermail/pypy-dev/2012-October/010602.html
which goes as far as to state "Note that the fact of splitting this is
not up to discussion, however, how we go about it is.". So, what is
the status of the splitting? Was this idea eventually abandoned?

From benjamin at python.org  Wed Apr 16 06:47:04 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Tue, 15 Apr 2014 21:47:04 -0700
Subject: [pypy-dev] RPython as a separate package
In-Reply-To: <CAFHN2SPL2wDFAV3i0JDEHAHZKxiv529wyDX0n=fyBTvMYL=gEQ@mail.gmail.com>
References: <CAFHN2SPL2wDFAV3i0JDEHAHZKxiv529wyDX0n=fyBTvMYL=gEQ@mail.gmail.com>
Message-ID: <1397623624.9684.107007473.42414591@webmail.messagingengine.com>

On Tue, Apr 15, 2014, at 21:31, Bogdan Opanchuk wrote:
> Hello,
> 
> I would like to use the RPython toolchain in my project. The problem
> is, RPython is currently hidden inside PyPy and therefore not readily
> available. I found this thread
> https://mail.python.org/pipermail/pypy-dev/2012-October/010602.html
> which goes as far as to state "Note that the fact of splitting this is
> not up to discussion, however, how we go about it is.". So, what is
> the status of the splitting? Was this idea eventually abandoned?

The RPython toolchain has been split. It's still in the same repo,
though. See the toplevel rpython/ directory in the pypy checkout.

From mantihor at gmail.com  Wed Apr 16 06:59:17 2014
From: mantihor at gmail.com (Bogdan Opanchuk)
Date: Wed, 16 Apr 2014 14:59:17 +1000
Subject: [pypy-dev] RPython as a separate package
In-Reply-To: <1397623624.9684.107007473.42414591@webmail.messagingengine.com>
References: <CAFHN2SPL2wDFAV3i0JDEHAHZKxiv529wyDX0n=fyBTvMYL=gEQ@mail.gmail.com>
 <1397623624.9684.107007473.42414591@webmail.messagingengine.com>
Message-ID: <CAFHN2SPM+JfrR8+ok+gnbfC1WjPaCdL9h6-nMopzt5DCV0__Ow@mail.gmail.com>

Hi Benjamin,

Thank you, I've seen it in the repo. But one still cannot install it
as a separate package, say, in CPython, and it's not even available as
a package in the PyPy itself (it's only used at build stage, as far as
I understand). Are there any plans to complete the splitting and make
it a standalone package?

On Wed, Apr 16, 2014 at 2:47 PM, Benjamin Peterson <benjamin at python.org> wrote:
> On Tue, Apr 15, 2014, at 21:31, Bogdan Opanchuk wrote:
>> Hello,
>>
>> I would like to use the RPython toolchain in my project. The problem
>> is, RPython is currently hidden inside PyPy and therefore not readily
>> available. I found this thread
>> https://mail.python.org/pipermail/pypy-dev/2012-October/010602.html
>> which goes as far as to state "Note that the fact of splitting this is
>> not up to discussion, however, how we go about it is.". So, what is
>> the status of the splitting? Was this idea eventually abandoned?
>
> The RPython toolchain has been split. It's still in the same repo,
> though. See the toplevel rpython/ directory in the pypy checkout.

From arigo at tunes.org  Wed Apr 16 10:45:25 2014
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 16 Apr 2014 10:45:25 +0200
Subject: [pypy-dev] RPython as a separate package
In-Reply-To: <CAFHN2SPM+JfrR8+ok+gnbfC1WjPaCdL9h6-nMopzt5DCV0__Ow@mail.gmail.com>
References: <CAFHN2SPL2wDFAV3i0JDEHAHZKxiv529wyDX0n=fyBTvMYL=gEQ@mail.gmail.com>
 <1397623624.9684.107007473.42414591@webmail.messagingengine.com>
 <CAFHN2SPM+JfrR8+ok+gnbfC1WjPaCdL9h6-nMopzt5DCV0__Ow@mail.gmail.com>
Message-ID: <CAMSv6X3bHXguLV-8pktkXGxiMaVf9hrm5G+19yKr_L5JhByE+w@mail.gmail.com>

Hi Bogdan,

On 16 April 2014 06:59, Bogdan Opanchuk <mantihor at gmail.com> wrote:
> it's not even available as a package in the PyPy itself

What do you mean?  "rpython" is the name of the top-level directory
we're talking about, with "__init__.py" and everything.  It is a
regular package.


A bient?t,

Armin.

From mantihor at gmail.com  Wed Apr 16 11:00:54 2014
From: mantihor at gmail.com (Bogdan Opanchuk)
Date: Wed, 16 Apr 2014 19:00:54 +1000
Subject: [pypy-dev] RPython as a separate package
In-Reply-To: <CAMSv6X3bHXguLV-8pktkXGxiMaVf9hrm5G+19yKr_L5JhByE+w@mail.gmail.com>
References: <CAFHN2SPL2wDFAV3i0JDEHAHZKxiv529wyDX0n=fyBTvMYL=gEQ@mail.gmail.com>
 <1397623624.9684.107007473.42414591@webmail.messagingengine.com>
 <CAFHN2SPM+JfrR8+ok+gnbfC1WjPaCdL9h6-nMopzt5DCV0__Ow@mail.gmail.com>
 <CAMSv6X3bHXguLV-8pktkXGxiMaVf9hrm5G+19yKr_L5JhByE+w@mail.gmail.com>
Message-ID: <CAFHN2SMZdckfvrAShqqRm8ujoYCv8dudg-9_FsPAZ_wBZJh2-Q@mail.gmail.com>

Hi Armin,

It is a package, but it is not discoverable. I cannot install it with
pip, and I cannot set it as a dependency for some other package.

On Wed, Apr 16, 2014 at 6:45 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Bogdan,
>
> On 16 April 2014 06:59, Bogdan Opanchuk <mantihor at gmail.com> wrote:
>> it's not even available as a package in the PyPy itself
>
> What do you mean?  "rpython" is the name of the top-level directory
> we're talking about, with "__init__.py" and everything.  It is a
> regular package.
>
>
> A bient?t,
>
> Armin.

From sam.e.giles at gmail.com  Wed Apr 16 12:09:58 2014
From: sam.e.giles at gmail.com (Samuel Giles)
Date: Wed, 16 Apr 2014 11:09:58 +0100
Subject: [pypy-dev] RPython as a separate package
In-Reply-To: <CAFHN2SMZdckfvrAShqqRm8ujoYCv8dudg-9_FsPAZ_wBZJh2-Q@mail.gmail.com>
References: <CAFHN2SPL2wDFAV3i0JDEHAHZKxiv529wyDX0n=fyBTvMYL=gEQ@mail.gmail.com>
 <1397623624.9684.107007473.42414591@webmail.messagingengine.com>
 <CAFHN2SPM+JfrR8+ok+gnbfC1WjPaCdL9h6-nMopzt5DCV0__Ow@mail.gmail.com>
 <CAMSv6X3bHXguLV-8pktkXGxiMaVf9hrm5G+19yKr_L5JhByE+w@mail.gmail.com>
 <CAFHN2SMZdckfvrAShqqRm8ujoYCv8dudg-9_FsPAZ_wBZJh2-Q@mail.gmail.com>
Message-ID: <CAPXHV4rs+vZP-s22PWfNDPCEy8CR-EsXRdntigPVDqhn7pOOyg@mail.gmail.com>

Bogdan: apologies, gmail, doesn't default to reply all and I forget
sometimes..

msg:
Bogdan,

I just explicitly have pypy checked out somewhere and reference it with
environment variables in my Makefile.  It's not particularly nice, but it's
really easy to get started on something and its kind of stuck for the
duration of my project.

https://github.com/samgiles/naulang/blob/master/Makefile

Sam


On Wed, Apr 16, 2014 at 10:00 AM, Bogdan Opanchuk <mantihor at gmail.com>wrote:

> Hi Armin,
>
> It is a package, but it is not discoverable. I cannot install it with
> pip, and I cannot set it as a dependency for some other package.
>
> On Wed, Apr 16, 2014 at 6:45 PM, Armin Rigo <arigo at tunes.org> wrote:
> > Hi Bogdan,
> >
> > On 16 April 2014 06:59, Bogdan Opanchuk <mantihor at gmail.com> wrote:
> >> it's not even available as a package in the PyPy itself
> >
> > What do you mean?  "rpython" is the name of the top-level directory
> > we're talking about, with "__init__.py" and everything.  It is a
> > regular package.
> >
> >
> > A bient?t,
> >
> > Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140416/e150140e/attachment.html>

From arigo at tunes.org  Wed Apr 16 12:59:28 2014
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 16 Apr 2014 12:59:28 +0200
Subject: [pypy-dev] OS/X
In-Reply-To: <3E04822A-13EF-4A4D-9494-26ACB7F92480@gmail.com>
References: <CAMSv6X1DdCxSczFkW1VcWWqF3bUjW8FYz2-jT7hXTh_HH_HD8g@mail.gmail.com>
 <E8CC0A5E-938E-4450-AE82-A7A3D173C605@gmail.com>
 <CAMSv6X0b9dYpT6bpV9spNZqeQXyF3XQpJnbT2u+P_7HtWwCm0w@mail.gmail.com>
 <3E04822A-13EF-4A4D-9494-26ACB7F92480@gmail.com>
Message-ID: <CAMSv6X2Cfr2z44pP7ENsFNX98ugjYg7weptYNRh4jp3XP1ObwA@mail.gmail.com>

Hi,

On 15 April 2014 19:21, Kenny Lasse Hoff Levinsen
<kennylevinsen at gmail.com> wrote:
> I only set CC=gcc-4.9 - no changes required, which is good!

Thank you!


Armin

From ronan.lamy at gmail.com  Wed Apr 16 16:21:30 2014
From: ronan.lamy at gmail.com (Ronan Lamy)
Date: Wed, 16 Apr 2014 15:21:30 +0100
Subject: [pypy-dev] RPython as a separate package
In-Reply-To: <CAFHN2SPM+JfrR8+ok+gnbfC1WjPaCdL9h6-nMopzt5DCV0__Ow@mail.gmail.com>
References: <CAFHN2SPL2wDFAV3i0JDEHAHZKxiv529wyDX0n=fyBTvMYL=gEQ@mail.gmail.com>
 <1397623624.9684.107007473.42414591@webmail.messagingengine.com>
 <CAFHN2SPM+JfrR8+ok+gnbfC1WjPaCdL9h6-nMopzt5DCV0__Ow@mail.gmail.com>
Message-ID: <534E91EA.6070209@gmail.com>

Le 16/04/14 05:59, Bogdan Opanchuk a ?crit :
> Hi Benjamin,
>
> Thank you, I've seen it in the repo. But one still cannot install it
> as a separate package, say, in CPython, and it's not even available as
> a package in the PyPy itself (it's only used at build stage, as far as
> I understand). Are there any plans to complete the splitting and make
> it a standalone package?

There are no firm plans, but it is still a goal. There are a few issues 
to solve first, though:

* Unbundling pytest and py lib
* Cleaning up and splitting the docs between PyPy and RPython
* Deciding what to do with the repository


From arigo at tunes.org  Wed Apr 16 16:29:59 2014
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 16 Apr 2014 16:29:59 +0200
Subject: [pypy-dev] Regression for lxml-cffi install (?)
In-Reply-To: <CAAvCB=TYhKScFp6EbJzkdRicCufe3PnVDDWSxuzwb+2DDfUYsg@mail.gmail.com>
References: <CAAvCB=RBXebhN5Qdi-1x9DJxG-ePT2++bu8wTTijCJFVf1SNXw@mail.gmail.com>
 <CAAvCB=TYhKScFp6EbJzkdRicCufe3PnVDDWSxuzwb+2DDfUYsg@mail.gmail.com>
Message-ID: <CAMSv6X1cV3GLqB08pX-k8hmCCzSMYP5uMc8SNJwkwRjjZa17hw@mail.gmail.com>

Hi,

On 28 February 2014 17:13, ????? ??????? <kostia.lopuhin at gmail.com> wrote:
> Maybe I am doing something entirely wrong here (e.g. the install
> command may be bogus).
> I would appreciate any advice :)

Sorry for the delay.  It seems that lxml issues don't trigger a lot of
enthusiastic responses from our side.  I'd recommend to file a regular
bug on https://bugs.pypy.org/ , otherwise the issue will be lost.


A bient?t,

Armin.

From rymg19 at gmail.com  Fri Apr 18 00:42:28 2014
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 17 Apr 2014 17:42:28 -0500
Subject: [pypy-dev] Alternatives to multiple inheritance
Message-ID: <CAO41-mM-CmTR67t3mG9HpJ6RAduO8vgqU-4vr2enxj53ZNnWQA@mail.gmail.com>

I?m writing a scripting language using RPython. I have a base class for all
my objects. Now, I have an exception object. The exception object needs to
derive from my base class in order for me to use polymorphism inside the
interpreter. However, it also needs to derive from the Exception class to
be throwable. My code looked kind of like this:

class BaseObject(object):
    pass
class MyException(BaseObject, Exception):
    ...

However, upon trying to compile the code, I got this error:

[translation:ERROR] AssertionError: multiple inheritance only
supported with _mixin_: <class 'bolt.objects.BoltException'>
[translation:ERROR] Processing block:
[translation:ERROR]  block at 122 is a <class
'rpython.flowspace.flowcontext.SpamBlock'>
[translation:ERROR]  in (bolt.main:10)parse_and_run
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v0 = simple_call((type interpreter))
[translation:ERROR]  --end--

bolt.objects.BoltException is my exception class. If I remove the Exception
base, I get this:

[translation:ERROR] AssertionError
[translation:ERROR] Processing block:
[translation:ERROR]  block at 9 is a <class
'rpython.flowspace.flowcontext.SpamBlock'>
[translation:ERROR]  in (bolt.main:4)run_file
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v0 = simple_call((function create_file), p_0, ('r'))
[translation:ERROR]        v1 = getattr(v0, ('read'))
[translation:ERROR]        v2 = simple_call(v1)
[translation:ERROR]        v3 = simple_call((function parse_and_run), v2)
[translation:ERROR]        v4 = getattr(v0, ('close'))
[translation:ERROR]        v5 = simple_call(v4)
[translation:ERROR]  --end--

The only thing about multiple inheritance and RPython I found was
this<https://mail.python.org/pipermail//pypy-commit/2011-November/055925.html>
.

My current idea is to have my base object derive from Exception instead of
nothing. I was wondering if there?s a better solution, however.

-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140417/af92bc9d/attachment.html>

From benjamin at python.org  Fri Apr 18 01:13:49 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Thu, 17 Apr 2014 16:13:49 -0700
Subject: [pypy-dev] Alternatives to multiple inheritance
In-Reply-To: <CAO41-mM-CmTR67t3mG9HpJ6RAduO8vgqU-4vr2enxj53ZNnWQA@mail.gmail.com>
References: <CAO41-mM-CmTR67t3mG9HpJ6RAduO8vgqU-4vr2enxj53ZNnWQA@mail.gmail.com>
Message-ID: <1397776429.28129.107745853.67245457@webmail.messagingengine.com>

On Thu, Apr 17, 2014, at 15:42, Ryan Gonzalez wrote:
> I?m writing a scripting language using RPython. I have a base class for
> all
> my objects. Now, I have an exception object. The exception object needs
> to
> derive from my base class in order for me to use polymorphism inside the
> interpreter. However, it also needs to derive from the Exception class to
> be throwable. My code looked kind of like this:
> 
> class BaseObject(object):
>     pass
> class MyException(BaseObject, Exception):
>     ...
> 
> However, upon trying to compile the code, I got this error:
> 
> [translation:ERROR] AssertionError: multiple inheritance only
> supported with _mixin_: <class 'bolt.objects.BoltException'>
> [translation:ERROR] Processing block:
> [translation:ERROR]  block at 122 is a <class
> 'rpython.flowspace.flowcontext.SpamBlock'>
> [translation:ERROR]  in (bolt.main:10)parse_and_run
> [translation:ERROR]  containing the following operations:
> [translation:ERROR]        v0 = simple_call((type interpreter))
> [translation:ERROR]  --end--
> 
> bolt.objects.BoltException is my exception class. If I remove the
> Exception
> base, I get this:
> 
> [translation:ERROR] AssertionError
> [translation:ERROR] Processing block:
> [translation:ERROR]  block at 9 is a <class
> 'rpython.flowspace.flowcontext.SpamBlock'>
> [translation:ERROR]  in (bolt.main:4)run_file
> [translation:ERROR]  containing the following operations:
> [translation:ERROR]        v0 = simple_call((function create_file), p_0,
> ('r'))
> [translation:ERROR]        v1 = getattr(v0, ('read'))
> [translation:ERROR]        v2 = simple_call(v1)
> [translation:ERROR]        v3 = simple_call((function parse_and_run), v2)
> [translation:ERROR]        v4 = getattr(v0, ('close'))
> [translation:ERROR]        v5 = simple_call(v4)
> [translation:ERROR]  --end--
> 
> The only thing about multiple inheritance and RPython I found was
> this<https://mail.python.org/pipermail//pypy-commit/2011-November/055925.html>
> .
> 
> My current idea is to have my base object derive from Exception instead
> of
> nothing. I was wondering if there?s a better solution, however.

In this case, you may want to do what the PyPy Python interpreter does.
There is one interpreter level exception for app-level exceptions called
OperationError. OperationError wraps the app-level exception object
inside of it.

From arigo at tunes.org  Fri Apr 18 08:44:33 2014
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 18 Apr 2014 08:44:33 +0200
Subject: [pypy-dev] Alternatives to multiple inheritance
In-Reply-To: <1397776429.28129.107745853.67245457@webmail.messagingengine.com>
References: <CAO41-mM-CmTR67t3mG9HpJ6RAduO8vgqU-4vr2enxj53ZNnWQA@mail.gmail.com>
 <1397776429.28129.107745853.67245457@webmail.messagingengine.com>
Message-ID: <CAMSv6X3UhrD4M76c6SHV1M96hdf1CTi+hhpiB3q5r=wb0pe1yA@mail.gmail.com>

Hi Ryan,

On 18 April 2014 01:13, Benjamin Peterson <benjamin at python.org> wrote:
> On Thu, Apr 17, 2014, at 15:42, Ryan Gonzalez wrote:
>> The exception object needs to
>> derive from my base class in order for me to use polymorphism inside the
>> interpreter. However, it also needs to derive from the Exception class to
>> be throwable.
>
> In this case, you may want to do what the PyPy Python interpreter does.
> There is one interpreter level exception for app-level exceptions called
> OperationError. OperationError wraps the app-level exception object
> inside of it.

In other words, the easiest is to have a class OpError(Exception) that
wraps your real exception object; then raise and catch
"OpError(your_object)" and don't do anything else with the OpError
class.

RPython is not C++ is that you can't throw and catch random things
(like integers...).  But then it is not C++ in that it has better
malloc-removal support :-)


A bient?t,

Armin.

From rymg19 at gmail.com  Fri Apr 18 21:50:28 2014
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Fri, 18 Apr 2014 14:50:28 -0500
Subject: [pypy-dev] Alternatives to multiple inheritance
In-Reply-To: <CAMSv6X3UhrD4M76c6SHV1M96hdf1CTi+hhpiB3q5r=wb0pe1yA@mail.gmail.com>
References: <CAO41-mM-CmTR67t3mG9HpJ6RAduO8vgqU-4vr2enxj53ZNnWQA@mail.gmail.com>
 <1397776429.28129.107745853.67245457@webmail.messagingengine.com>
 <CAMSv6X3UhrD4M76c6SHV1M96hdf1CTi+hhpiB3q5r=wb0pe1yA@mail.gmail.com>
Message-ID: <CAO41-mMd_8KX4eUqNy1k2S=a2i3pg7Th-N2pA=q-N-WrT0f_+A@mail.gmail.com>

That makes sense! Thanks a lot!


On Fri, Apr 18, 2014 at 1:44 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Ryan,
>
> On 18 April 2014 01:13, Benjamin Peterson <benjamin at python.org> wrote:
> > On Thu, Apr 17, 2014, at 15:42, Ryan Gonzalez wrote:
> >> The exception object needs to
> >> derive from my base class in order for me to use polymorphism inside the
> >> interpreter. However, it also needs to derive from the Exception class
> to
> >> be throwable.
> >
> > In this case, you may want to do what the PyPy Python interpreter does.
> > There is one interpreter level exception for app-level exceptions called
> > OperationError. OperationError wraps the app-level exception object
> > inside of it.
>
> In other words, the easiest is to have a class OpError(Exception) that
> wraps your real exception object; then raise and catch
> "OpError(your_object)" and don't do anything else with the OpError
> class.
>
> RPython is not C++ is that you can't throw and catch random things
> (like integers...).  But then it is not C++ in that it has better
> malloc-removal support :-)
>
>
> A bient?t,
>
> Armin.
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140418/93a723a3/attachment.html>

From rymg19 at gmail.com  Mon Apr 21 20:37:55 2014
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 21 Apr 2014 13:37:55 -0500
Subject: [pypy-dev] Blocked Block error
Message-ID: <CAO41-mPDk-nGJ=BJ9KOOcwR7zxhqydmrT2Y0kEQLTxZ6uwR-fw@mail.gmail.com>

First, a quick apology: I don't want anyone to feel like I'm abusing the
list or something. I just really need help, and StackOverflow isn't
necessarily RPython-friendly.

What I want to know is the meaning of the Blocked Block error. I always get
that error at least once. Usually I can solve it myself, but I'm curious as
to what it actually means.

-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140421/a4fc96e4/attachment.html>

From arigo at tunes.org  Tue Apr 22 10:14:25 2014
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 22 Apr 2014 10:14:25 +0200
Subject: [pypy-dev] Blocked Block error
In-Reply-To: <CAO41-mPDk-nGJ=BJ9KOOcwR7zxhqydmrT2Y0kEQLTxZ6uwR-fw@mail.gmail.com>
References: <CAO41-mPDk-nGJ=BJ9KOOcwR7zxhqydmrT2Y0kEQLTxZ6uwR-fw@mail.gmail.com>
Message-ID: <CAMSv6X2XZonMsk2nqPUcWsOByQZyJhQqa6BHg6Lya8f8N9s0yw@mail.gmail.com>

Hi Ryan,

On 21 April 2014 20:37, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> What I want to know is the meaning of the Blocked Block error. I always get
> that error at least once. Usually I can solve it myself, but I'm curious as
> to what it actually means.

When RPython does type inference, it works incrementally.  There are
often some operations that don't seem to make sense temporarily, but
this problem will be resolved by the type inference done on other
parts of the same program.  The typical example is "x = foo.attr"
where, as far as we know, the instance "foo" has no attribute "attr".
This looks like an error, but often isn't: if we continue somewhere
else, we'll see "foo.attr = 42" and then know that "foo" really has an
"attr".

"Blocked Blocks" errors are raised if at the end of the type inference
process there are still such unresolved positions left.

If several errors are printed, often only one of them is relevant
(e.g. a real typo).

The error can also occur becuse the particular type inference done can
prove that "foo" is always None in this case, so that reading
"foo.attr" doesn't make sense.  That can be annoying when running
tests that only try to translate a part of the program.  It's usually
easy to find a workaround, e.g. by inserting "assert foo is not None".
 Unlike "x = foo.attr", which blocks, an assert that always fails does
not create a blocked block, but will simply always raise
AssertionError in the translated program.  (The distinction is a bit
artificial.  We could also never create blocked blocks and simply
raise AssertionErrors if this point is executed, but that seems like a
bad idea in general...)


A bient?t,

Armin.

From matti.picus at gmail.com  Tue Apr 22 20:35:44 2014
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 22 Apr 2014 21:35:44 +0300
Subject: [pypy-dev] 2.3 release is close, please help triage the open bugs
Message-ID: <5356B680.8000803@gmail.com>

The consensus on IRC is that we are ready to release PyPy 2.3 Please
help triage the bugs on bugs.pypy.org, and mark your favorite
release-critical one with the 2.3 Release tag. We could also use a name
for the release, suggestions are welcome. For an idea as to what
changed, see
the release notice
https://bitbucket.org/pypy/pypy/src/20e51c4389ed4469b66bb9d6289ce0ecfc82c4b9/pypy/doc/release-2.3.0.rst?at=release-2.3.x

Matti

From rymg19 at gmail.com  Tue Apr 22 21:24:46 2014
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 22 Apr 2014 14:24:46 -0500
Subject: [pypy-dev] Blocked Block error
In-Reply-To: <CAMSv6X2XZonMsk2nqPUcWsOByQZyJhQqa6BHg6Lya8f8N9s0yw@mail.gmail.com>
References: <CAO41-mPDk-nGJ=BJ9KOOcwR7zxhqydmrT2Y0kEQLTxZ6uwR-fw@mail.gmail.com>
 <CAMSv6X2XZonMsk2nqPUcWsOByQZyJhQqa6BHg6Lya8f8N9s0yw@mail.gmail.com>
Message-ID: <CAO41-mO_PDhueoV9hs=KG5WE4y-dBEyKvCxxU8HmCMojjeJY1w@mail.gmail.com>

Thank you so much again! That makes sense!


On Tue, Apr 22, 2014 at 3:14 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Ryan,
>
> On 21 April 2014 20:37, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> > What I want to know is the meaning of the Blocked Block error. I always
> get
> > that error at least once. Usually I can solve it myself, but I'm curious
> as
> > to what it actually means.
>
> When RPython does type inference, it works incrementally.  There are
> often some operations that don't seem to make sense temporarily, but
> this problem will be resolved by the type inference done on other
> parts of the same program.  The typical example is "x = foo.attr"
> where, as far as we know, the instance "foo" has no attribute "attr".
> This looks like an error, but often isn't: if we continue somewhere
> else, we'll see "foo.attr = 42" and then know that "foo" really has an
> "attr".
>
> "Blocked Blocks" errors are raised if at the end of the type inference
> process there are still such unresolved positions left.
>
> If several errors are printed, often only one of them is relevant
> (e.g. a real typo).
>
> The error can also occur becuse the particular type inference done can
> prove that "foo" is always None in this case, so that reading
> "foo.attr" doesn't make sense.  That can be annoying when running
> tests that only try to translate a part of the program.  It's usually
> easy to find a workaround, e.g. by inserting "assert foo is not None".
>  Unlike "x = foo.attr", which blocks, an assert that always fails does
> not create a blocked block, but will simply always raise
> AssertionError in the translated program.  (The distinction is a bit
> artificial.  We could also never create blocked blocks and simply
> raise AssertionErrors if this point is executed, but that seems like a
> bad idea in general...)
>
>
> A bient?t,
>
> Armin.
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140422/3748d2ff/attachment.html>

From kmod at dropbox.com  Tue Apr 22 21:29:51 2014
From: kmod at dropbox.com (Kevin Modzelewski)
Date: Tue, 22 Apr 2014 12:29:51 -0700
Subject: [pypy-dev] Benchmark that pypy doesn't do well on
Message-ID: <CAO=oM6s=0=4zLgK3RzVs1b+tX=cu8=ZBxk_=j_Q6Y-um6nAYqA@mail.gmail.com>

Hi all, I've finally gotten around to open-sourcing some old code of mine
(coincidentally, the predecessor to Pyston) which doesn't perform that well
on PyPy.  It's not the best benchmark, since it's not deterministic and
produces different results on PyPy and CPython (due to dict ordering, I
think), but PyPy seems to be consistently 20-30% slower than CPython, so I
think the overall effect is reliable even if the exact numbers aren't
meaningful.  The benchmark is "real code" in that I simply added a
benchmark mode to a real project, where the benchmark runs the exact same
thing as a normal invocation (except for disabling some disk output); I'm
definitely not trying to claim that the benchmark is "representative" in
any sense, though.

To run it:
cd ~ # needs to be in the home directly unfortunately
git clone https://github.com/kmod/icbd
# set up python env
time bash icbd/icbd/type_analyzer/run.sh bench

On my machine, CPython runs it in 60s, but PyPy takes 75s; this is on PyPy
1.8 since I can't get a newer version, but I get similar results with
2.2.1.  I should mention that the program doesn't use any non-stdlib
modules, at least in "bench" mode.


I've also tried to extract a part of the program that seemed to run
significantly slower under PyPy, and got this microbenchmark:
https://github.com/dropbox/pyston/blob/master/microbenchmarks/polymorphism.py

It takes about 13s for PyPy, and about 4s for CPython.  This microbenchmark
is only based on the final phase of the program (the part right before
printing all the colored output), so I don't think it's necessarily that
representative, it's just something I happened to notice was much slower.

Just wanted to send it over in case you guys were interested.

kmod
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140422/c498e1c1/attachment.html>

From arigo at tunes.org  Wed Apr 23 09:06:20 2014
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 23 Apr 2014 09:06:20 +0200
Subject: [pypy-dev] Benchmark that pypy doesn't do well on
In-Reply-To: <CAO=oM6s=0=4zLgK3RzVs1b+tX=cu8=ZBxk_=j_Q6Y-um6nAYqA@mail.gmail.com>
References: <CAO=oM6s=0=4zLgK3RzVs1b+tX=cu8=ZBxk_=j_Q6Y-um6nAYqA@mail.gmail.com>
Message-ID: <CAMSv6X18KVDti5C06xpXTZXYyxybjg8UxHv7Fbd1ON42JKNQfA@mail.gmail.com>

Hi Kevin,

On 22 April 2014 21:29, Kevin Modzelewski <kmod at dropbox.com> wrote:
> I've also tried to extract a part of the program that seemed to run
> significantly slower under PyPy, and got this microbenchmark:
> https://github.com/dropbox/pyston/blob/master/microbenchmarks/polymorphism.py

This suffers from extreme warm-up times.  If I run it as posted, I get
9.1s on PyPy versus 4.4s on CPython.  If I increase the number of
iterations from 1k to 10k, I get 18.9s on PyPy versus 44s on CPython.
The reasons for the large warm-up times are multiple: it's a highly
recursive example; a core function in this recursion contains a loop
that runs only twice; and the overall process is repeated 1000 times
--- just before the JIT triggers compilation from the outermost level.

It's an example where we should ideally detect that the loop typically
runs twice, and unroll it.  If we change the source code to manually
unroll it, the four timings above become respectively 3.9s, 3.9s,
6.1s, 39s.


A bient?t,

Armin.

From arigo at tunes.org  Wed Apr 23 09:21:08 2014
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 23 Apr 2014 09:21:08 +0200
Subject: [pypy-dev] 2.3 release is close,
	please help triage the open bugs
In-Reply-To: <5356B680.8000803@gmail.com>
References: <5356B680.8000803@gmail.com>
Message-ID: <CAMSv6X3muEz0j-z9jNfeFVaxFSB03Xh13=ASa=NMNRaEkEPtDw@mail.gmail.com>

Hi all,

On 22 April 2014 20:35, Matti Picus <matti.picus at gmail.com> wrote:
> For an idea as to what changed, see the release notice
> https://bitbucket.org/pypy/pypy/src/20e51c4389ed4469b66bb9d6289ce0ecfc82c4b9/pypy/doc/release-2.3.0.rst?at=release-2.3.x

This precisely *fails* to give you any concrete idea about what
changed in PyPy 2.3 :-)  This "Highlights" list contains exactly the
same bullet points as the 2.2.0 release announcement.

You should point people to
https://bitbucket.org/pypy/pypy/raw/release-2.3.x/pypy/doc/release-2.3.0.rst
, which contains the most up-to-date version of the file.  Soon, it
should be fixed to include what's really new.


A bient?t,

Armin.

From mak at issuu.com  Wed Apr 23 15:39:36 2014
From: mak at issuu.com (Martin Koch)
Date: Wed, 23 Apr 2014 15:39:36 +0200
Subject: [pypy-dev] pickle/cPickle bug
Message-ID: <CACyk+CKOU8Qazg3ftKrPxWuOVRRK9xAdOD86wHVXkXB8Hen_aA@mail.gmail.com>

Hi list

I have found what appears to be a bug in the pickle and cPickle library in
pypy. I can reproduce it both on linux and mac. Basically, I can't unpickle
a file if it has been pickled with a file object returned by open(), but it
works if I use the with construct. The problem is present both using pickle
and cPickle.

On mac, the pypy version is
Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 20:57:21)
[PyPy 2.2.1 with GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)]

On linux, the pypy version is
Python 2.7.3 (2.2.1+dfsg-1, Jan 24 2014, 10:12:37)
[PyPy 2.2.1 with GCC 4.6.3]

*#import cPickle as pickle*
*import pickle*

*l = range(10)*
*# fails:*
*try:*
*    pickle.dump(l, open('f', 'wb'))*
*    print pickle.load(open('f', 'rb'))*
*except EOFError:*
*    print("Yup, that's an EOFError")*


*#succeeds*
*with open('f', 'wb') as f:*
*   pickle.dump(l, f)*

*with open('f', 'rb') as f:*
*   print pickle.load(f)*


If I instead of using open() inline in the argument to pickle assign it to
a variable and explicitly close the file after calling pickle.dump, then
the problem goes away:

*    f = open('f', 'wb')*
*    pickle.dump(l, f)*
*    f.close()*
*    print pickle.load(open('f', 'rb'))*


Apparently, in the first code snippet, the file isn't closed as it should
be when the object returned by open() goes out of scope after pickle.dump.

Thanks,
/Martin Koch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140423/a2bf972b/attachment.html>

From fijall at gmail.com  Wed Apr 23 15:50:50 2014
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 23 Apr 2014 15:50:50 +0200
Subject: [pypy-dev] pickle/cPickle bug
In-Reply-To: <CACyk+CKOU8Qazg3ftKrPxWuOVRRK9xAdOD86wHVXkXB8Hen_aA@mail.gmail.com>
References: <CACyk+CKOU8Qazg3ftKrPxWuOVRRK9xAdOD86wHVXkXB8Hen_aA@mail.gmail.com>
Message-ID: <CAK5idxTB-p6R7XbKzcvA8rDMN9FWT0s1aaLw7f=Oupv8k50FrQ@mail.gmail.com>

Hi Martin.

This is a documented difference with CPython where files are not
flushed unless either closed explicitly or garbage collected (which
will happen later than you might expect).

On Wed, Apr 23, 2014 at 3:39 PM, Martin Koch <mak at issuu.com> wrote:
> Hi list
>
> I have found what appears to be a bug in the pickle and cPickle library in
> pypy. I can reproduce it both on linux and mac. Basically, I can't unpickle
> a file if it has been pickled with a file object returned by open(), but it
> works if I use the with construct. The problem is present both using pickle
> and cPickle.
>
> On mac, the pypy version is
> Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 20:57:21)
> [PyPy 2.2.1 with GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)]
>
> On linux, the pypy version is
> Python 2.7.3 (2.2.1+dfsg-1, Jan 24 2014, 10:12:37)
> [PyPy 2.2.1 with GCC 4.6.3]
>
> #import cPickle as pickle
> import pickle
>
> l = range(10)
> # fails:
> try:
>     pickle.dump(l, open('f', 'wb'))
>     print pickle.load(open('f', 'rb'))
> except EOFError:
>     print("Yup, that's an EOFError")
>
>
> #succeeds
> with open('f', 'wb') as f:
>    pickle.dump(l, f)
>
> with open('f', 'rb') as f:
>    print pickle.load(f)
>
>
> If I instead of using open() inline in the argument to pickle assign it to a
> variable and explicitly close the file after calling pickle.dump, then the
> problem goes away:
>
>     f = open('f', 'wb')
>     pickle.dump(l, f)
>     f.close()
>     print pickle.load(open('f', 'rb'))
>
>
> Apparently, in the first code snippet, the file isn't closed as it should be
> when the object returned by open() goes out of scope after pickle.dump.
>
> Thanks,
> /Martin Koch
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From anton.gulenko at student.hpi.uni-potsdam.de  Thu Apr 24 12:38:33 2014
From: anton.gulenko at student.hpi.uni-potsdam.de (Anton Gulenko)
Date: Thu, 24 Apr 2014 12:38:33 +0200
Subject: [pypy-dev] Virtualizables in RPython
Message-ID: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>

Dear developers,

I am currently working on my Masters thesis about optimizations for SPy
(Squeak VM written using the RPython toolchain).
I am planning to devote one or two chapters to the issue of virtualizable
objects in a VM like SPy (and also in SPy in particular).
Since there is not much documentaion on virtualizables available, I would
appreciate your input. I want to collect details about the underlying
concept, and also about the specific implementation in the RPython JIT. For
example, was this concept first introduced in Pypy, or is it an older idea?
How exactly does the optimizer decide which objects can be virtualized, and
which can not?
I would also appreciate pointers to relevant parts of the Pypy source code,
which is probably the best documentation as of now.

Thank you and best regards,
Anton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140424/d11d1d89/attachment.html>

From matti.picus at gmail.com  Fri Apr 25 11:01:26 2014
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 25 Apr 2014 12:01:26 +0300
Subject: [pypy-dev] release of pypy2.3 is imminent
Message-ID: <535A2466.2040304@gmail.com>

If you have a branch you would like included in pypy-2.3.0, please be
sure to merge it by April 28. At that point I will merge, for the last
time,  default into the release branch.
Matti

From fijall at gmail.com  Fri Apr 25 11:30:49 2014
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 25 Apr 2014 11:30:49 +0200
Subject: [pypy-dev] release of pypy2.3 is imminent
In-Reply-To: <535A2466.2040304@gmail.com>
References: <535A2466.2040304@gmail.com>
Message-ID: <CAK5idxTfEGfyT+_XDkyV-zTNOOnBeL5yddY0bVAZthC0dnKzOQ@mail.gmail.com>

Hey Matti

We have a potential release blocker with an unrolling bug.

On Fri, Apr 25, 2014 at 11:01 AM, Matti Picus <matti.picus at gmail.com> wrote:
> If you have a branch you would like included in pypy-2.3.0, please be
> sure to merge it by April 28. At that point I will merge, for the last
> time,  default into the release branch.
> Matti
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From lac at openend.se  Fri Apr 25 21:15:03 2014
From: lac at openend.se (Laura Creighton)
Date: Fri, 25 Apr 2014 21:15:03 +0200
Subject: [pypy-dev] Interesting  blog post
Message-ID: <201404251915.s3PJF30c030541@fido.openend.se>

https://stripe.com/blog

I wrote them, and yes non-USA citizens/green card holders are eligible.
So if somebody wants to get paid to live in San Francisco for 3 months and
hack on PyPy, send in an application ....

Laura

From alex.gaynor at gmail.com  Fri Apr 25 21:40:02 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Fri, 25 Apr 2014 12:40:02 -0700
Subject: [pypy-dev] Interesting blog post
In-Reply-To: <201404251915.s3PJF30c030541@fido.openend.se>
References: <201404251915.s3PJF30c030541@fido.openend.se>
Message-ID: <CAFRnB2U4iu8NifcXKgcASJEqXsoNTGyBanUSP7VspaSQrEb9Xg@mail.gmail.com>

The stripe folks are pretty awesome (as is their office), so if anyone came
to live here and work on PyPy it would be awesome :-)

Alex


On Fri, Apr 25, 2014 at 12:15 PM, Laura Creighton <lac at openend.se> wrote:

> https://stripe.com/blog
>
> I wrote them, and yes non-USA citizens/green card holders are eligible.
> So if somebody wants to get paid to live in San Francisco for 3 months and
> hack on PyPy, send in an application ....
>
> Laura
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140425/765bcaeb/attachment.html>

From bdkearns at gmail.com  Sat Apr 26 05:41:00 2014
From: bdkearns at gmail.com (Brian Kearns)
Date: Fri, 25 Apr 2014 20:41:00 -0700
Subject: [pypy-dev] release of pypy2.3 is imminent
In-Reply-To: <CAK5idxTfEGfyT+_XDkyV-zTNOOnBeL5yddY0bVAZthC0dnKzOQ@mail.gmail.com>
References: <535A2466.2040304@gmail.com>
 <CAK5idxTfEGfyT+_XDkyV-zTNOOnBeL5yddY0bVAZthC0dnKzOQ@mail.gmail.com>
Message-ID: <CACw2m-YHRVtKAucHZPuuJymFqWHP_wE25XnKjUgZ2Ep7p8wBMg@mail.gmail.com>

Also, what about the existence of other blockers? For example:
https://bugs.pypy.org/issue1695


On Fri, Apr 25, 2014 at 2:30 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:

> Hey Matti
>
> We have a potential release blocker with an unrolling bug.
>
> On Fri, Apr 25, 2014 at 11:01 AM, Matti Picus <matti.picus at gmail.com>
> wrote:
> > If you have a branch you would like included in pypy-2.3.0, please be
> > sure to merge it by April 28. At that point I will merge, for the last
> > time,  default into the release branch.
> > Matti
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140425/dec19e63/attachment.html>

From arigo at tunes.org  Sat Apr 26 09:10:41 2014
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 26 Apr 2014 09:10:41 +0200
Subject: [pypy-dev] Virtualizables in RPython
In-Reply-To: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>
References: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>
Message-ID: <CAMSv6X0r3a9LcsF8dJceFtRHgTdfozjFz9mwJmLmeYU=9BdwuA@mail.gmail.com>

Hi Anton,

On 24 April 2014 12:38, Anton Gulenko
<anton.gulenko at student.hpi.uni-potsdam.de> wrote:
> appreciate your input. I want to collect details about the underlying
> concept, and also about the specific implementation in the RPython JIT. For
> example, was this concept first introduced in Pypy, or is it an older idea?
> How exactly does the optimizer decide which objects can be virtualized, and
> which can not?

We first need to clarify some details.  Are you talking about
"virtualizables", or "virtuals"?  These are two very different
concepts.  The latter is a very strong form of escape analysis, and
probably the single most efficient concept we have in the RPython JIT.
 The former, on the other hand, is more of a hack that was added at
some point in time, and that we're trying to remove --- unsuccessfully
so far, because it gives slighty better results than virtuals only,
for the specific use case where it works.

I'm unsure which one you're talking about: only the frame object is a
"virtualizable", and it's not the job of the optimizer to decide that;
it's explicitly marked in the jitdriver.  See
https://pypy.readthedocs.org/en/latest/jit/virtualizable.html .  On
the other hand, if you're talking about "virtuals", then indeed we
have code in the optimizer to handle them.  It's some kind of escape
analysis leading to allocation removal.


A bient?t,

Armin.

From alex.gaynor at gmail.com  Sat Apr 26 20:59:31 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 26 Apr 2014 11:59:31 -0700
Subject: [pypy-dev] release of pypy2.3 is imminent
In-Reply-To: <CACw2m-YHRVtKAucHZPuuJymFqWHP_wE25XnKjUgZ2Ep7p8wBMg@mail.gmail.com>
References: <535A2466.2040304@gmail.com>
 <CAK5idxTfEGfyT+_XDkyV-zTNOOnBeL5yddY0bVAZthC0dnKzOQ@mail.gmail.com>
 <CACw2m-YHRVtKAucHZPuuJymFqWHP_wE25XnKjUgZ2Ep7p8wBMg@mail.gmail.com>
Message-ID: <CAFRnB2Vg3CXM3g0amthC2AoFRz77vT=PRO-O9ijMYDvWCzD2_A@mail.gmail.com>

I'd like to make https://bugs.pypy.org/issue1743 a blocker for 2.3

Alex


On Fri, Apr 25, 2014 at 8:41 PM, Brian Kearns <bdkearns at gmail.com> wrote:

> Also, what about the existence of other blockers? For example:
> https://bugs.pypy.org/issue1695
>
>
> On Fri, Apr 25, 2014 at 2:30 AM, Maciej Fijalkowski <fijall at gmail.com>wrote:
>
>> Hey Matti
>>
>> We have a potential release blocker with an unrolling bug.
>>
>> On Fri, Apr 25, 2014 at 11:01 AM, Matti Picus <matti.picus at gmail.com>
>> wrote:
>> > If you have a branch you would like included in pypy-2.3.0, please be
>> > sure to merge it by April 28. At that point I will merge, for the last
>> > time,  default into the release branch.
>> > Matti
>> > _______________________________________________
>> > pypy-dev mailing list
>> > pypy-dev at python.org
>> > https://mail.python.org/mailman/listinfo/pypy-dev
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140426/d6c1c958/attachment.html>

From anton.gulenko at student.hpi.uni-potsdam.de  Mon Apr 28 14:30:13 2014
From: anton.gulenko at student.hpi.uni-potsdam.de (Anton Gulenko)
Date: Mon, 28 Apr 2014 14:30:13 +0200
Subject: [pypy-dev] Virtualizables in RPython
In-Reply-To: <CAMSv6X0r3a9LcsF8dJceFtRHgTdfozjFz9mwJmLmeYU=9BdwuA@mail.gmail.com>
References: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>
 <CAMSv6X0r3a9LcsF8dJceFtRHgTdfozjFz9mwJmLmeYU=9BdwuA@mail.gmail.com>
Message-ID: <CAAhvomB5r00Ntvs_P+mFo1mLR7cwaREwnzjFSuWUZ7B2ORqSig@mail.gmail.com>

Dear Armin,

thank you for your reply!
I was not aware of the distinction between virtuals and virtualizables.
It's a good start to have that covered ;)
I thought marking frame objects as virtualizables was a necessary input for
the optimizer, because it cannot decide everything on its own.
Since you are calling virtualizables a hack, it seems the optimizer SHOULD
be able to handle frame objects without additional hints. Could you explain
why it is not able to do so?

My original question was referring to virtuals, but I thought that frame
objects were part of that. Now that I know of the distinction, I would like
to understand both concepts in detail, and how they interact.
I'm not sure how much of this I can ask you to describe to me - probably I
should just read and debug the according code? Any suggestions?

The motivation for this exercise is that the Spy VM is producing JIT traces
that I (we) did not understand. For example, changes in the VM that seemed
totally unrelated, were breaking the "virtualizablility" of frame objects
(or parts of them), like by setting the program counter on each loop
execution (inside the trace!).
I'm not sure if this was unwanted behavior on behalf of the optimizer, but
it seemed pretty non-deterministic, and I would like to understand the
mechanism well enough to troubleshoot and optimize these traces.

Best,
Anton



2014-04-26 9:10 GMT+02:00 Armin Rigo <arigo at tunes.org>:

> Hi Anton,
>
> On 24 April 2014 12:38, Anton Gulenko
> <anton.gulenko at student.hpi.uni-potsdam.de> wrote:
> > appreciate your input. I want to collect details about the underlying
> > concept, and also about the specific implementation in the RPython JIT.
> For
> > example, was this concept first introduced in Pypy, or is it an older
> idea?
> > How exactly does the optimizer decide which objects can be virtualized,
> and
> > which can not?
>
> We first need to clarify some details.  Are you talking about
> "virtualizables", or "virtuals"?  These are two very different
> concepts.  The latter is a very strong form of escape analysis, and
> probably the single most efficient concept we have in the RPython JIT.
>  The former, on the other hand, is more of a hack that was added at
> some point in time, and that we're trying to remove --- unsuccessfully
> so far, because it gives slighty better results than virtuals only,
> for the specific use case where it works.
>
> I'm unsure which one you're talking about: only the frame object is a
> "virtualizable", and it's not the job of the optimizer to decide that;
> it's explicitly marked in the jitdriver.  See
> https://pypy.readthedocs.org/en/latest/jit/virtualizable.html .  On
> the other hand, if you're talking about "virtuals", then indeed we
> have code in the optimizer to handle them.  It's some kind of escape
> analysis leading to allocation removal.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140428/e6764e28/attachment.html>

From n210241048576 at gmail.com  Tue Apr 29 05:43:49 2014
From: n210241048576 at gmail.com (Robert Grosse)
Date: Mon, 28 Apr 2014 23:43:49 -0400
Subject: [pypy-dev] Another benchmark where Pypy is much slower than CPython
Message-ID: <CALJjGitiAL+DKd0ZcEYm6GggXiQ8rh5c_CTs8JCcXPHNgq8z5w@mail.gmail.com>

Hi, while working on some MDP code, I discovered that it was several times
as slow under Pypy as CPython. Attached is a benchmark demonstrating this.

On my computer, the benchmark takes 70.3 seconds under CPython 2.7.3 64bit,
but it takes 273.6 seconds under pypy-c-jit-71056-c8e3b8cbc843-win32, the
latest version I could find.

The code is not at all optimized, but I still found it strange that Pypy
did so much worse. The code doesn't seem that complex, but perhaps the
usage of nameduples is triggering pathological behavior from the JIT's
point of view.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140428/c079c8ad/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: benchmark.zip
Type: application/zip
Size: 3866 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140428/c079c8ad/attachment.zip>

From alex.gaynor at gmail.com  Tue Apr 29 06:59:32 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 28 Apr 2014 21:59:32 -0700
Subject: [pypy-dev] Another benchmark where Pypy is much slower than
	CPython
In-Reply-To: <CALJjGitiAL+DKd0ZcEYm6GggXiQ8rh5c_CTs8JCcXPHNgq8z5w@mail.gmail.com>
References: <CALJjGitiAL+DKd0ZcEYm6GggXiQ8rh5c_CTs8JCcXPHNgq8z5w@mail.gmail.com>
Message-ID: <CAFRnB2W7AkmJd47ro2Xsqe7W3Dr2yedKLMBRjkiS6mm4te+b2w@mail.gmail.com>

I've reproduced the performance on the latest default (on OS X, FWIW). I'm
starting to profile now.

Alex

PS: Your namedtuple usage is fine.


On Mon, Apr 28, 2014 at 8:43 PM, Robert Grosse <n210241048576 at gmail.com>wrote:

> Hi, while working on some MDP code, I discovered that it was several times
> as slow under Pypy as CPython. Attached is a benchmark demonstrating this.
>
> On my computer, the benchmark takes 70.3 seconds under CPython 2.7.3
> 64bit, but it takes 273.6 seconds under
> pypy-c-jit-71056-c8e3b8cbc843-win32, the latest version I could find.
>
> The code is not at all optimized, but I still found it strange that Pypy
> did so much worse. The code doesn't seem that complex, but perhaps the
> usage of nameduples is triggering pathological behavior from the JIT's
> point of view.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140428/64de8f97/attachment.html>

From alex.gaynor at gmail.com  Tue Apr 29 07:08:47 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 28 Apr 2014 22:08:47 -0700
Subject: [pypy-dev] Another benchmark where Pypy is much slower than
	CPython
In-Reply-To: <CAFRnB2W7AkmJd47ro2Xsqe7W3Dr2yedKLMBRjkiS6mm4te+b2w@mail.gmail.com>
References: <CALJjGitiAL+DKd0ZcEYm6GggXiQ8rh5c_CTs8JCcXPHNgq8z5w@mail.gmail.com>
 <CAFRnB2W7AkmJd47ro2Xsqe7W3Dr2yedKLMBRjkiS6mm4te+b2w@mail.gmail.com>
Message-ID: <CAFRnB2VDMVsWgm3B=FwW1xEcfnaL4qCaWg8UbeAgLxTtu-3nGw@mail.gmail.com>

Initial results of profiling (cProfile): most of the time is in generators,
and in evaluate(). 6 aborts, all trace too long. Both tracing and backend
times are a tiny percentage of the overall time. In other words, none of
the usual suspects. I probably won't get any more done before going to bed
tonight.

Alex


On Mon, Apr 28, 2014 at 9:59 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:

> I've reproduced the performance on the latest default (on OS X, FWIW). I'm
> starting to profile now.
>
> Alex
>
> PS: Your namedtuple usage is fine.
>
>
> On Mon, Apr 28, 2014 at 8:43 PM, Robert Grosse <n210241048576 at gmail.com>wrote:
>
>> Hi, while working on some MDP code, I discovered that it was several
>> times as slow under Pypy as CPython. Attached is a benchmark demonstrating
>> this.
>>
>> On my computer, the benchmark takes 70.3 seconds under CPython 2.7.3
>> 64bit, but it takes 273.6 seconds under
>> pypy-c-jit-71056-c8e3b8cbc843-win32, the latest version I could find.
>>
>> The code is not at all optimized, but I still found it strange that Pypy
>> did so much worse. The code doesn't seem that complex, but perhaps the
>> usage of nameduples is triggering pathological behavior from the JIT's
>> point of view.
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right
> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: 125F 5C67 DFE9 4084
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140428/7afe6be6/attachment-0001.html>

From timfelgentreff at gmail.com  Tue Apr 29 13:22:35 2014
From: timfelgentreff at gmail.com (Tim Felgentreff)
Date: Tue, 29 Apr 2014 13:22:35 +0200
Subject: [pypy-dev] Virtualizables in RPython
In-Reply-To: <CAAhvomB5r00Ntvs_P+mFo1mLR7cwaREwnzjFSuWUZ7B2ORqSig@mail.gmail.com>
References: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>
 <CAMSv6X0r3a9LcsF8dJceFtRHgTdfozjFz9mwJmLmeYU=9BdwuA@mail.gmail.com>
 <CAAhvomB5r00Ntvs_P+mFo1mLR7cwaREwnzjFSuWUZ7B2ORqSig@mail.gmail.com>
Message-ID: <CAOu8aBs-MVO=h==YWpsYc8hBR8sM0JdTt70hhF646Wet3Tccpg@mail.gmail.com>

Maybe it's useful to add that even without any changes to the SPy VM,
frame objects in loops are nicely virtualized in the tests (where they
are executing at a stack depth of 1), but already if we execute the
loop at a stack depth of 4 or 5, some frame attributes (but only some,
like the PC) suddenly appeared in the trace and were written all
over...

Frames failed to virtualize entirely at a stack depth of ~100

Part of these questions is that we're trying to understand why this occurs.

cheers,
-Tim

On 28 April 2014 14:30, Anton Gulenko
<anton.gulenko at student.hpi.uni-potsdam.de> wrote:
> Dear Armin,
>
> thank you for your reply!
> I was not aware of the distinction between virtuals and virtualizables. It's
> a good start to have that covered ;)
> I thought marking frame objects as virtualizables was a necessary input for
> the optimizer, because it cannot decide everything on its own.
> Since you are calling virtualizables a hack, it seems the optimizer SHOULD
> be able to handle frame objects without additional hints. Could you explain
> why it is not able to do so?
>
> My original question was referring to virtuals, but I thought that frame
> objects were part of that. Now that I know of the distinction, I would like
> to understand both concepts in detail, and how they interact.
> I'm not sure how much of this I can ask you to describe to me - probably I
> should just read and debug the according code? Any suggestions?
>
> The motivation for this exercise is that the Spy VM is producing JIT traces
> that I (we) did not understand. For example, changes in the VM that seemed
> totally unrelated, were breaking the "virtualizablility" of frame objects
> (or parts of them), like by setting the program counter on each loop
> execution (inside the trace!).
> I'm not sure if this was unwanted behavior on behalf of the optimizer, but
> it seemed pretty non-deterministic, and I would like to understand the
> mechanism well enough to troubleshoot and optimize these traces.
>
> Best,
> Anton
>
>
>
> 2014-04-26 9:10 GMT+02:00 Armin Rigo <arigo at tunes.org>:
>
>> Hi Anton,
>>
>> On 24 April 2014 12:38, Anton Gulenko
>> <anton.gulenko at student.hpi.uni-potsdam.de> wrote:
>> > appreciate your input. I want to collect details about the underlying
>> > concept, and also about the specific implementation in the RPython JIT.
>> > For
>> > example, was this concept first introduced in Pypy, or is it an older
>> > idea?
>> > How exactly does the optimizer decide which objects can be virtualized,
>> > and
>> > which can not?
>>
>> We first need to clarify some details.  Are you talking about
>> "virtualizables", or "virtuals"?  These are two very different
>> concepts.  The latter is a very strong form of escape analysis, and
>> probably the single most efficient concept we have in the RPython JIT.
>>  The former, on the other hand, is more of a hack that was added at
>> some point in time, and that we're trying to remove --- unsuccessfully
>> so far, because it gives slighty better results than virtuals only,
>> for the specific use case where it works.
>>
>> I'm unsure which one you're talking about: only the frame object is a
>> "virtualizable", and it's not the job of the optimizer to decide that;
>> it's explicitly marked in the jitdriver.  See
>> https://pypy.readthedocs.org/en/latest/jit/virtualizable.html .  On
>> the other hand, if you're talking about "virtuals", then indeed we
>> have code in the optimizer to handle them.  It's some kind of escape
>> analysis leading to allocation removal.
>>
>>
>> A bient?t,
>>
>> Armin.
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From arigo at tunes.org  Wed Apr 30 11:13:18 2014
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 30 Apr 2014 11:13:18 +0200
Subject: [pypy-dev] Virtualizables in RPython
In-Reply-To: <CAOu8aBs-MVO=h==YWpsYc8hBR8sM0JdTt70hhF646Wet3Tccpg@mail.gmail.com>
References: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>
 <CAMSv6X0r3a9LcsF8dJceFtRHgTdfozjFz9mwJmLmeYU=9BdwuA@mail.gmail.com>
 <CAAhvomB5r00Ntvs_P+mFo1mLR7cwaREwnzjFSuWUZ7B2ORqSig@mail.gmail.com>
 <CAOu8aBs-MVO=h==YWpsYc8hBR8sM0JdTt70hhF646Wet3Tccpg@mail.gmail.com>
Message-ID: <CAMSv6X2gZhy4wggD_A9LPCBJ4MvdxUOuOvTZT+LdxNgMRov-nQ@mail.gmail.com>

Hi Tim,

On 29 April 2014 13:22, Tim Felgentreff <timfelgentreff at gmail.com> wrote:
> Maybe it's useful to add that even without any changes to the SPy VM,
> frame objects in loops are nicely virtualized in the tests (where they
> are executing at a stack depth of 1), but already if we execute the
> loop at a stack depth of 4 or 5, some frame attributes (but only some,
> like the PC) suddenly appeared in the trace and were written all
> over...
>
> Frames failed to virtualize entirely at a stack depth of ~100
>
> Part of these questions is that we're trying to understand why this occurs.

We might again talk past each other.  What I believe I understand
behind what you're saying: you're looking at the outermost frame (the
one in which the looping occurs).  This frame is not a virtualizable
(because you don't have any), and is also not a virtual (because it
already exists before the loop).  However, in simple examples, the
optimizer is able to remove all writes of the PC to it.  But in larger
examples, these writes show up again.  Is that correct?

This is expected so far.  The writes to a normal, pre-existing object
(the frame) can be delayed but only partly.  I think that Topaz uses a
hack for now to force the write of the PC to be delayed more: instead
of writing the PC as an integer, it writes it as a small object
containing an integer --- every time a new object.  This is really a
hack but it should solve the problem for now.  You should discuss with
Alex Gaynor if this hack turned out to work in the long run or if he
eventually had to give it up and replace it with something else (like
virtualizables).

If your original question is precisely that you did all this already
and want to learn more about virtualizables, then I'm sorry :-)
Reading your original questions again seem to show some confusion
(which might be on my side too).  In two words, a virtualizable object
is not virtual at all; it's just an object in which some specific
fields are specially marked as "delayed", like the PC --- which can
then be a regular integer, without the need for the small container
object hack described above.


A bient?t,

Armin.

From anton.gulenko at student.hpi.uni-potsdam.de  Wed Apr 30 13:11:40 2014
From: anton.gulenko at student.hpi.uni-potsdam.de (Anton Gulenko)
Date: Wed, 30 Apr 2014 13:11:40 +0200
Subject: [pypy-dev] Virtualizables in RPython
In-Reply-To: <CAMSv6X2gZhy4wggD_A9LPCBJ4MvdxUOuOvTZT+LdxNgMRov-nQ@mail.gmail.com>
References: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>
 <CAMSv6X0r3a9LcsF8dJceFtRHgTdfozjFz9mwJmLmeYU=9BdwuA@mail.gmail.com>
 <CAAhvomB5r00Ntvs_P+mFo1mLR7cwaREwnzjFSuWUZ7B2ORqSig@mail.gmail.com>
 <CAOu8aBs-MVO=h==YWpsYc8hBR8sM0JdTt70hhF646Wet3Tccpg@mail.gmail.com>
 <CAMSv6X2gZhy4wggD_A9LPCBJ4MvdxUOuOvTZT+LdxNgMRov-nQ@mail.gmail.com>
Message-ID: <CAAhvomB0aYZLgy3cT_4qngqfduqp9Li+TvFSGugiuf2HOSe=+A@mail.gmail.com>

Hi Armin,

We might again talk past each other.  What I believe I understand
> behind what you're saying: you're looking at the outermost frame (the
> one in which the looping occurs).  This frame is not a virtualizable
> (because you don't have any), and is also not a virtual (because it
> already exists before the loop).  However, in simple examples, the
> optimizer is able to remove all writes of the PC to it.  But in larger
> examples, these writes show up again.  Is that correct?


I'll try to make the example that Tim mentioned more clear.
Building up the deep stack was done INSIDE the loop. It was also the only
thing that happened inside the loop.
That's why we expected the traces for deep and shallow stacks to be very
similar - shouldn't the optimizer simply eliminate the additional frame
objects?
Also, the relevant fields in the frame objects are indeed marked as
virtualizable in the SPy VM.
The smalltalk code was basically this, with a varying argument to
buildStack:

buildStack: depth
     depth <= 0 ifTrue: [ ^ nil ].
     self buildStack: depth - 1

100000 timesRepeat: [ self buildStack: 100 ]

Do you have any thoughts regarding this example?

In two words, a virtualizable object
> is not virtual at all; it's just an object in which some specific
> fields are specially marked as "delayed", like the PC


Ok, so virtuals and virtualizables are two unrelated mechanisms?
How does the optimizer decide which writes to virtualizable fields it is
able to eliminate? Is that based on a similar kind of analysis like the
escape analysis for virtual objects?

Thanks, best regards,
Anton


2014-04-30 11:13 GMT+02:00 Armin Rigo <arigo at tunes.org>:

> Hi Tim,
>
> On 29 April 2014 13:22, Tim Felgentreff <timfelgentreff at gmail.com> wrote:
> > Maybe it's useful to add that even without any changes to the SPy VM,
> > frame objects in loops are nicely virtualized in the tests (where they
> > are executing at a stack depth of 1), but already if we execute the
> > loop at a stack depth of 4 or 5, some frame attributes (but only some,
> > like the PC) suddenly appeared in the trace and were written all
> > over...
> >
> > Frames failed to virtualize entirely at a stack depth of ~100
> >
> > Part of these questions is that we're trying to understand why this
> occurs.
>
> We might again talk past each other.  What I believe I understand
> behind what you're saying: you're looking at the outermost frame (the
> one in which the looping occurs).  This frame is not a virtualizable
> (because you don't have any), and is also not a virtual (because it
> already exists before the loop).  However, in simple examples, the
> optimizer is able to remove all writes of the PC to it.  But in larger
> examples, these writes show up again.  Is that correct?
>
> This is expected so far.  The writes to a normal, pre-existing object
> (the frame) can be delayed but only partly.  I think that Topaz uses a
> hack for now to force the write of the PC to be delayed more: instead
> of writing the PC as an integer, it writes it as a small object
> containing an integer --- every time a new object.  This is really a
> hack but it should solve the problem for now.  You should discuss with
> Alex Gaynor if this hack turned out to work in the long run or if he
> eventually had to give it up and replace it with something else (like
> virtualizables).
>
> If your original question is precisely that you did all this already
> and want to learn more about virtualizables, then I'm sorry :-)
> Reading your original questions again seem to show some confusion
> (which might be on my side too).  In two words, a virtualizable object
> is not virtual at all; it's just an object in which some specific
> fields are specially marked as "delayed", like the PC --- which can
> then be a regular integer, without the need for the small container
> object hack described above.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140430/da4c49fc/attachment.html>

From anto.cuni at gmail.com  Wed Apr 30 14:29:48 2014
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 30 Apr 2014 14:29:48 +0200
Subject: [pypy-dev] Virtualizables in RPython
In-Reply-To: <CAAhvomB0aYZLgy3cT_4qngqfduqp9Li+TvFSGugiuf2HOSe=+A@mail.gmail.com>
References: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>
 <CAMSv6X0r3a9LcsF8dJceFtRHgTdfozjFz9mwJmLmeYU=9BdwuA@mail.gmail.com>
 <CAAhvomB5r00Ntvs_P+mFo1mLR7cwaREwnzjFSuWUZ7B2ORqSig@mail.gmail.com>
 <CAOu8aBs-MVO=h==YWpsYc8hBR8sM0JdTt70hhF646Wet3Tccpg@mail.gmail.com>
 <CAMSv6X2gZhy4wggD_A9LPCBJ4MvdxUOuOvTZT+LdxNgMRov-nQ@mail.gmail.com>
 <CAAhvomB0aYZLgy3cT_4qngqfduqp9Li+TvFSGugiuf2HOSe=+A@mail.gmail.com>
Message-ID: <CAHPg78z4aoGi-u72Rg04VoZY5zi3bKyMW9gEuSTiRXZbY+pT8g@mail.gmail.com>

On Wed, Apr 30, 2014 at 1:11 PM, Anton Gulenko <
anton.gulenko at student.hpi.uni-potsdam.de> wrote:

> I'll try to make the example that Tim mentioned more clear.
> Building up the deep stack was done INSIDE the loop. It was also the only
> thing that happened inside the loop.
> That's why we expected the traces for deep and shallow stacks to be very
> similar - shouldn't the optimizer simply eliminate the additional frame
> objects?
> Also, the relevant fields in the frame objects are indeed marked as
> virtualizable in the SPy VM.
> The smalltalk code was basically this, with a varying argument to
> buildStack:
>
> buildStack: depth
>      depth <= 0 ifTrue: [ ^ nil ].
>      self buildStack: depth - 1
>
> 100000 timesRepeat: [ self buildStack: 100 ]
>
> Do you have any thoughts regarding this example?
>

?just a wild guess: it might be possible that in this example the trace
becomes too long, so tracing aborts and the function is marked as "trace
from start"? In that case, the function call cannot be inlined and needs to
be turned into a real assembler recursive call, which means that the frame
cannot be a virtual because it needs to be passed as an argument.?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20140430/8c960952/attachment.html>

From arigo at tunes.org  Wed Apr 30 20:21:53 2014
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 30 Apr 2014 20:21:53 +0200
Subject: [pypy-dev] Virtualizables in RPython
In-Reply-To: <CAAhvomB0aYZLgy3cT_4qngqfduqp9Li+TvFSGugiuf2HOSe=+A@mail.gmail.com>
References: <CAAhvomAxhOkMUR03U0fAfge=JMxnYme6Uft3zaF1kA_6UDgtQQ@mail.gmail.com>
 <CAMSv6X0r3a9LcsF8dJceFtRHgTdfozjFz9mwJmLmeYU=9BdwuA@mail.gmail.com>
 <CAAhvomB5r00Ntvs_P+mFo1mLR7cwaREwnzjFSuWUZ7B2ORqSig@mail.gmail.com>
 <CAOu8aBs-MVO=h==YWpsYc8hBR8sM0JdTt70hhF646Wet3Tccpg@mail.gmail.com>
 <CAMSv6X2gZhy4wggD_A9LPCBJ4MvdxUOuOvTZT+LdxNgMRov-nQ@mail.gmail.com>
 <CAAhvomB0aYZLgy3cT_4qngqfduqp9Li+TvFSGugiuf2HOSe=+A@mail.gmail.com>
Message-ID: <CAMSv6X0tvNuDU-11iTWaFMyLTMhzPU++dEijzSNGsNhxuGJJZw@mail.gmail.com>

Hi Anton,

On 30 April 2014 13:11, Anton Gulenko
<anton.gulenko at student.hpi.uni-potsdam.de> wrote:
> Ok, so virtuals and virtualizables are two unrelated mechanisms?
> How does the optimizer decide which writes to virtualizable fields it is
> able to eliminate?

It's not decided by the optimizer.  A virtualizable structure has got
a static list of fields to handle (in _virtualizable_ = [...]).  When
we enter the JITted machine code, we have exactly one virtualizable
(the outermost frame), and we read the current value of these fields
in local variables (i.e. registers or machine stack locations).  When
we leave the machine code again, we write back the local variables
into the fields.  In the meantime, these fields have an outdated
value.  This only concerns the single outermost frame.  If there are
more frame objects created by the frame, their status as
"virtualizable" is completely ignored, and instead we simply rely on
the malloc-removal optimization called "virtuals".

What exactly occurs when one of these other frames escapes (following
the theory of Antonio) depends on yet another detail.  If each frame
has a normal "back" field, then escaping any frame will force all
frames below it to escape as well, which will force them all.  In PyPy
we solve this by using "virtualrefs", which is yet another concept,
unrelated to the two previous ones.  Each frame has a "f_back" field
which is not a normal reference to the previous frame, but
"jit.virtual_ref(previous_frame)".  This works a bit like a weakref,
except that it's not weak --- instead, it allows whatever is
referenced to remain virtual even if the small virtual_ref object
itself escapes, based on the assumption that most of the time we'll
not access the real object.


A bient?t,

Armin.