From jdoran at lexmachina.com  Fri Jul  8 16:17:36 2016
From: jdoran at lexmachina.com (Jeff Doran)
Date: Fri, 8 Jul 2016 13:17:36 -0700
Subject: [pypy-dev] Progress, but new issues with PyPy
Message-ID: <CAFsQrzBnU-wCeg31P6Lx0QUH-C4iUeT0E8=DNQwhmBaxHuo5jw@mail.gmail.com>

Hello,

I want to start with highlighting the improvements I've noticed sine late
last year with  PyPy.  At the time I was having issues with weakref
handling in PyPy when trying to do extensive lxml work which failed in
various ways.   That's all good now with lxml 3.6 and PyPy 5.1.0, but I'm
still not able to complete our test suite for code running on Python 2.7.10
on CPython.  My last major hurdle was monkey patching the Betamax playback
package we use to record http sessions.  My mandate was that in order for
us to consider using PyPy we needed all our existing tests to pass and that
meant using the current recordings. There were several places where strings
were compared (think of queries or POST bodies) that were stored at some
point in a dictionary.

Since I've overcome all the consistently repeatable errors in running our
tests via nosetests,  I'm now left with a frustratingly inconsistent, but
frequent error that leaves me with no clue.  It happens at different places
in running our tests, it cannot be captured by using pdb, and reports
nothing but InvocationError when it occurs.  Sometimes it goes maddeningly
far, but it will not finish.

Any insight or suggestions on how to proceed would be appreciated.

Thanks,
- Jeff

Details
----------.

- running PyPy from pypy-5.1-linux_x86_64-portable on debian 8.5
   It's installed in /home/jeff/pypy
   $python --version
    Python 2.7.10 (3260adbeba4a8b6659d1cc0d0b41f266769b74da, Apr 21 2016,
07:28:42)
    [PyPy 5.1.0 with GCC 4.9.2]

- running tests in a virtual machine via 'source
/home/jeff/pypy/bin/activate'

- running tests under tox:
       (pypy) $ tox -v   -e pypy
        using tox.ini: /home/jeff/lexmachina/deus_lex/tox.ini
        using tox-2.3.1 from /home/jeff/pypy/site-packages/tox/__init__.pyc
        pypy reusing: /home/jeff/lexmachina/deus_lex/.tox/pypy
         $ /home/jeff/lexmachina/deus_lex/.tox/pypy/bin/pip freeze
>/home/jeff/lexmachina/deus_lex/.tox/pypy/log/pypy-7.log
        pypy runtests: commands[0] | nosetests -vv --nocapture --pdb
bias.tests.processors.test_dar:TestDar
        ...
        Lots of succssful test output then:

         ERROR: InvocationError:
'/home/jeff/lexmachina/deus_lex/.tox/pypy/bin/nosetests -vv --nocapture
--pdb bias.tests.processors.test_dar:TestDar'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160708/812eec29/attachment.html>

From arigo at tunes.org  Fri Jul  8 17:05:51 2016
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 8 Jul 2016 23:05:51 +0200
Subject: [pypy-dev] Progress, but new issues with PyPy
In-Reply-To: <CAFsQrzBnU-wCeg31P6Lx0QUH-C4iUeT0E8=DNQwhmBaxHuo5jw@mail.gmail.com>
References: <CAFsQrzBnU-wCeg31P6Lx0QUH-C4iUeT0E8=DNQwhmBaxHuo5jw@mail.gmail.com>
Message-ID: <CAMSv6X2sakR04WPjgF5dsXsu+C8gkTAbpHVJ9+PJXhftDVfpMw@mail.gmail.com>

Hi Jeff,

On 8 July 2016 at 22:17, Jeff Doran <jdoran at lexmachina.com> wrote:
> Since I've overcome all the consistently repeatable errors in running our
> tests via nosetests,  I'm now left with a frustratingly inconsistent, but
> frequent error that leaves me with no clue.

I'd recommend using RevPDB (see blog post from today at
morepypy.blogspot.com), but I'd guess you are using at least cpyext.

> Any insight or suggestions on how to proceed would be appreciated.

Since your mail does not mention precisely which project you're
testing or how to reproduce in any way (unless I missed it), it's hard
for us to have any clue...


A bient?t,

Armin.

From arigo at tunes.org  Sat Jul  9 07:07:41 2016
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 9 Jul 2016 13:07:41 +0200
Subject: [pypy-dev] Twisted's deferToThread is much more slower with
 pypy than on cpython
In-Reply-To: <5751359D.5020901@fsn.hu>
References: <5751359D.5020901@fsn.hu>
Message-ID: <CAMSv6X1EpeEK8=Jz2wPL_tS7ebMBcHFB78_oCqdcnzRHkL5vbQ@mail.gmail.com>

Hi Nagy,

On 3 June 2016 at 09:45, Nagy, Attila <bra at fsn.hu> wrote:
> Consider this example program:
> https://gist.github.com/bra-fsn/1fd481b44590a939e849cb9073ba1a41

I've reduced it to a minimal example and created an issue.

https://bitbucket.org/pypy/pypy/issues/2341/multithreading-locks-leading-to-poor-cpu


Armin

From jdoran at lexmachina.com  Sat Jul  9 19:44:02 2016
From: jdoran at lexmachina.com (Jeff Doran)
Date: Sat, 9 Jul 2016 16:44:02 -0700
Subject: [pypy-dev] Progress, but new issues with PyPy
In-Reply-To: <CAMSv6X2sakR04WPjgF5dsXsu+C8gkTAbpHVJ9+PJXhftDVfpMw@mail.gmail.com>
References: <CAFsQrzBnU-wCeg31P6Lx0QUH-C4iUeT0E8=DNQwhmBaxHuo5jw@mail.gmail.com>
 <CAMSv6X2sakR04WPjgF5dsXsu+C8gkTAbpHVJ9+PJXhftDVfpMw@mail.gmail.com>
Message-ID: <CAFsQrzCNJa0X=HDoAaiHAe5ODbw9oB8Via7=9+ELv2W9cqRqrw@mail.gmail.com>

Armin, thanks for the quick response.  I didn't get to trying RevPDB, but
it looks like a good tool for the tool box.  I did upgrade PyPy to 5.3.1,
rebuild the PyPy virtual environment, recreate tox, and restart my linux VM
(how much abstraction can we handle here?).   All seems much happier now so
moving on.   We just might get full PyPy acceptance on our test suite!

- Jeff

On Fri, Jul 8, 2016 at 2:05 PM, Armin Rigo <arigo at tunes.org> wrote:

> Hi Jeff,
>
> On 8 July 2016 at 22:17, Jeff Doran <jdoran at lexmachina.com> wrote:
> > Since I've overcome all the consistently repeatable errors in running our
> > tests via nosetests,  I'm now left with a frustratingly inconsistent, but
> > frequent error that leaves me with no clue.
>
> I'd recommend using RevPDB (see blog post from today at
> morepypy.blogspot.com), but I'd guess you are using at least cpyext.
>
> > Any insight or suggestions on how to proceed would be appreciated.
>
> Since your mail does not mention precisely which project you're
> testing or how to reproduce in any way (unless I missed it), it's hard
> for us to have any clue...
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160709/7b76c608/attachment.html>

From yury at shurup.com  Sun Jul 10 13:26:07 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Sun, 10 Jul 2016 19:26:07 +0200
Subject: [pypy-dev] Update to the win32 buildbot
In-Reply-To: <5749E1FA.6050801@gmail.com>
References: <5749E1FA.6050801@gmail.com>
Message-ID: <1468171567.3560.69.camel@newpride>

Hi Matti,

I've very sorry for such a delay... Did you manage to update
local_2.4.zip already, or, if not, do you have step-by-step instructions
of what should I do with xz-5.0.5-windows.zip ? I.e. simply unpack over
to the directory where local_2.4.zip is unpacked, or else create a
subdirectory, etc. ? Many thanks!

-- 
Sincerely yours,
Yury V. Zaytsev


On Sat, 2016-05-28 at 21:22 +0300, Matti Picus wrote: 
> Hi! I am cc-ing the pypy-dev list so there is some permanent record of 
> this, the mail is for the win32 buildbot maintainer.
> Could you add the lzma development libraries to the build environment of 
> the win32 buildbot?
> They are needed for python 3, as you can see from any of the failing 
> build runs like this one
> http://buildbot.pypy.org/builders/pypy-c-jit-win-x86-32/builds/2398/steps/shell_1/logs/stdio
> 
> Python 3 uses the xz-5.0.5 libraries, as can bee seen in the 
> cpython/PCbuild/readme.txt,
> following their links leads to http://tukaani.org/xz/ and to a download
> http://tukaani.org/xz/xz-5.0.5-windows.zip (library)
> http://tukaani.org/xz/xz-5.0.5-windows.zip.sig (signature)
> 
> I will try to update the local.zip available on our downloads
> https://bitbucket.org/pypy/pypy/downloads/local_2.4.zip
> and the documentation
> http://doc.pypy.org/en/latest/windows.html
> but in the meantime it would be great if you could get this going
> 
> Thanks!
> Matti
> 




From wickedgrey at gmail.com  Mon Jul 11 12:43:08 2016
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Mon, 11 Jul 2016 09:43:08 -0700
Subject: [pypy-dev] pickle numpy array from pypy to cpython?
In-Reply-To: <CAMSv6X18QeDoP=Yh3uVyti9a=DjvLTxzp_nr-D8KWD5ozS7iFQ@mail.gmail.com>
References: <CADa34LCdDuA3hfhdFwZRD4gsvFrpMriG__H_a+VmmRwB1BnAtQ@mail.gmail.com>
 <CAHgd1hHaWkO68EzN_Bc=q0FqoXCrdxCW31fQ+PFooLNM=Y3FrA@mail.gmail.com>
 <CADa34LAy1aE-6ziGWRHFKkaLqwXVJs3eajYM9W1u4PzbswuiXQ@mail.gmail.com>
 <CAEv-cC5EJGZii9SA_5MCJsXnrFhyjjRxY38pboQs1dDdeV6BmQ@mail.gmail.com>
 <CADa34LC74vO5XREMHbLRfiNn5gDHPWTrwJMZ_UqKjFLvt7hnOw@mail.gmail.com>
 <F6AEA760-B6D6-4789-9856-CD896D8EA5E8@gmail.com>
 <CADa34LBGQOJOi75UhO0NhOgds5NEEGXXm7pm2W=iUQtqhe-Agw@mail.gmail.com>
 <CADa34LAW4v9HrEEeGNDp2-a9=F1PCH=6iFXWrRHRPTKheVd13w@mail.gmail.com>
 <CADa34LDR5=pTP4VFE0YGZpJQsj6HvYLTy-wO0hm1g75YYw-MEQ@mail.gmail.com>
 <CAO8bCginopNO0J0yyQtrafyJpdc+5UA=qrw3fnqE4NhaeEU+QA@mail.gmail.com>
 <CADa34LAwkKueBKMnnCAbzJFjUUAaqjqpniJ074L1SKypzsQv=A@mail.gmail.com>
 <576EBEB9.9070804@gmail.com>
 <CADa34LA8FiF7=Fh6Si7dqx17bPM5Rf56LOv9upnnDrLbff6JSg@mail.gmail.com>
 <CADa34LB2u3f6Lc_1pvE7Cgh0JQmiS0-FUvHUrwMgoLYrruQSjQ@mail.gmail.com>
 <577413F0.90909@gmail.com>
 <CADa34LApHdXMzyZi7dK-5y7oLdp4xUOuN1y+QdDjRkOdY96q1A@mail.gmail.com>
 <CAMSv6X18QeDoP=Yh3uVyti9a=DjvLTxzp_nr-D8KWD5ozS7iFQ@mail.gmail.com>
Message-ID: <CADa34LBwyg=hsA2_daEjPoZ95zAjaE0OtAeqZ0+uL=hfOfpJOw@mail.gmail.com>

FWVLIW, I think that conforming to upstream numpy makes the most sense.

I think that the issue would go away if the `_numpypy` module were
renamed to `numpy` (and appropriate things moved into `numpy.core`).
Is there a technical reason to keep the actual implementation in a
separately named module?

Thinking larger picture, would it be possible and sensible to switch
to using the slow cpyext numpy approach for compatability, then
overlay custom implementation of things on top of that when speed is
needed?  I'm imagining a vague inverse of the cpython approach, where
modules are implemented in C when the python performance isn't
acceptable.

Eli

On Wed, Jun 29, 2016 at 10:58 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Eli, hi Matti,
>
> On 29 June 2016 at 21:37, Eli Stevens (Gmail) <wickedgrey at gmail.com> wrote:
>> To make sure I'm understanding, are you saying that upstream/cpython
>> numpy should pick up an alternate way to import multiarray (via
>> _numpypy.multiarray, instead of numpy.core.multiarray)
>
> Hum, in my opinion we should always pickle/unpickle arrays by
> reproducing and expecting the exact same format as CPython's numpy,
> with no warnings.  Any difference (e.g. with complicated dtypes) is a
> bug that should eventually be fixed.
>
>
> A bient?t,
>
> Armin.

From tirnotaure at gmail.com  Wed Jul 13 15:49:55 2016
From: tirnotaure at gmail.com (Andrey Rubik)
Date: Wed, 13 Jul 2016 22:49:55 +0300
Subject: [pypy-dev] integration with java
Message-ID: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>

Hi guys!

I'am new at pypy development and need help.

I want do integration between pypy and jemeter (this project on java: 
http://jmeter.apache.org/).

As i learned on jemeter dev list, i need some pypy-java bindings. It may 
be some .jar file for example.

Where i can get some pypy-java binding (.jar file)?


Thanks in advance!


-- 
Best Regards,
Andrey Rubik


From fijall at gmail.com  Thu Jul 14 05:08:26 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 14 Jul 2016 11:08:26 +0200
Subject: [pypy-dev] integration with java
In-Reply-To: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
References: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
Message-ID: <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>

Hi Andrey

There are no java bindings for PyPy just yet. You would need to write
your own based on cffi. http://jpype.sourceforge.net/ is one of the
examples of how those things can be done, but you would need to use
cffi as opposed to CPython C API. If you need a quick solution, you
can try compiling jpype against PyPys CPython C API compatibility
layer, but it would be at least very slow.

Best regards,
Maciej Fijalkowski

On Wed, Jul 13, 2016 at 9:49 PM, Andrey Rubik <tirnotaure at gmail.com> wrote:
> Hi guys!
>
> I'am new at pypy development and need help.
>
> I want do integration between pypy and jemeter (this project on java:
> http://jmeter.apache.org/).
>
> As i learned on jemeter dev list, i need some pypy-java bindings. It may be
> some .jar file for example.
>
> Where i can get some pypy-java binding (.jar file)?
>
>
> Thanks in advance!
>
>
> --
> Best Regards,
> Andrey Rubik
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From anto.cuni at gmail.com  Thu Jul 14 06:06:58 2016
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 14 Jul 2016 12:06:58 +0200
Subject: [pypy-dev] integration with java
In-Reply-To: <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>
References: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
 <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>
Message-ID: <CAHPg78yasqJJiK4e9Bcx764ZfpVRgUQXuhZ_x1vyeprycNwY-w@mail.gmail.com>

Hi,
you could also check pyjnius, which seems more mature than jpype and it is
used in real life apps to run e.g. python on android:
http://pyjnius.readthedocs.io/en/latest/

I quickly tried to compile pyjnius with pypy (using cpyext) and it seems to
work (at least, the tests pass). However, since it uses cpyext I dont' know
what are the performance.

ciao,
Anto

On Thu, Jul 14, 2016 at 11:08 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Hi Andrey
>
> There are no java bindings for PyPy just yet. You would need to write
> your own based on cffi. http://jpype.sourceforge.net/ is one of the
> examples of how those things can be done, but you would need to use
> cffi as opposed to CPython C API. If you need a quick solution, you
> can try compiling jpype against PyPys CPython C API compatibility
> layer, but it would be at least very slow.
>
> Best regards,
> Maciej Fijalkowski
>
> On Wed, Jul 13, 2016 at 9:49 PM, Andrey Rubik <tirnotaure at gmail.com>
> wrote:
> > Hi guys!
> >
> > I'am new at pypy development and need help.
> >
> > I want do integration between pypy and jemeter (this project on java:
> > http://jmeter.apache.org/).
> >
> > As i learned on jemeter dev list, i need some pypy-java bindings. It may
> be
> > some .jar file for example.
> >
> > Where i can get some pypy-java binding (.jar file)?
> >
> >
> > Thanks in advance!
> >
> >
> > --
> > Best Regards,
> > Andrey Rubik
> >
> > _______________________________________________
> > pypy-dev mailing list
> > pypy-dev at python.org
> > https://mail.python.org/mailman/listinfo/pypy-dev
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160714/831404e7/attachment.html>

From tirnotaure at gmail.com  Thu Jul 14 11:08:11 2016
From: tirnotaure at gmail.com (Andrey Rubik)
Date: Thu, 14 Jul 2016 18:08:11 +0300
Subject: [pypy-dev] integration with java
In-Reply-To: <CAHPg78yasqJJiK4e9Bcx764ZfpVRgUQXuhZ_x1vyeprycNwY-w@mail.gmail.com>
References: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
 <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>
 <CAHPg78yasqJJiK4e9Bcx764ZfpVRgUQXuhZ_x1vyeprycNwY-w@mail.gmail.com>
Message-ID: <5f253069-b1c8-0dc1-0fcb-fcfdfba08ebe@gmail.com>

Hi guys :)

Jmeter already have some python intergration (jython for example).

Jmeter - tool for performance tests. So when i start high load tests 
written in scripting languages i get so much load on jmeter server. That 
is why I choose PyPy - for much speed from testing server.

Sry, but i'am not a developer :(

And write own integration solution between java and pypy its too 
difficult for me :(

Can someone do good java - pypy binding (some .jar file) ?


-- 
Best Regards,
Andrey Rubik


On 07/14/2016 01:06 PM, Antonio Cuni wrote:
> Hi,
> you could also check pyjnius, which seems more mature than jpype and 
> it is used in real life apps to run e.g. python on android:
> http://pyjnius.readthedocs.io/en/latest/
>
> I quickly tried to compile pyjnius with pypy (using cpyext) and it 
> seems to work (at least, the tests pass). However, since it uses 
> cpyext I dont' know what are the performance.
>
> ciao,
> Anto
>
> On Thu, Jul 14, 2016 at 11:08 AM, Maciej Fijalkowski <fijall at gmail.com 
> <mailto:fijall at gmail.com>> wrote:
>
>     Hi Andrey
>
>     There are no java bindings for PyPy just yet. You would need to write
>     your own based on cffi. http://jpype.sourceforge.net/ is one of the
>     examples of how those things can be done, but you would need to use
>     cffi as opposed to CPython C API. If you need a quick solution, you
>     can try compiling jpype against PyPys CPython C API compatibility
>     layer, but it would be at least very slow.
>
>     Best regards,
>     Maciej Fijalkowski
>
>     On Wed, Jul 13, 2016 at 9:49 PM, Andrey Rubik
>     <tirnotaure at gmail.com <mailto:tirnotaure at gmail.com>> wrote:
>     > Hi guys!
>     >
>     > I'am new at pypy development and need help.
>     >
>     > I want do integration between pypy and jemeter (this project on
>     java:
>     > http://jmeter.apache.org/).
>     >
>     > As i learned on jemeter dev list, i need some pypy-java
>     bindings. It may be
>     > some .jar file for example.
>     >
>     > Where i can get some pypy-java binding (.jar file)?
>     >
>     >
>     > Thanks in advance!
>     >
>     >
>     > --
>     > Best Regards,
>     > Andrey Rubik
>     >
>     > _______________________________________________
>     > pypy-dev mailing list
>     > pypy-dev at python.org <mailto:pypy-dev at python.org>
>     > https://mail.python.org/mailman/listinfo/pypy-dev
>     _______________________________________________
>     pypy-dev mailing list
>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>     https://mail.python.org/mailman/listinfo/pypy-dev
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160714/295333f8/attachment-0001.html>

From arigo at tunes.org  Fri Jul 15 03:51:13 2016
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 15 Jul 2016 09:51:13 +0200
Subject: [pypy-dev] integration with java
In-Reply-To: <5f253069-b1c8-0dc1-0fcb-fcfdfba08ebe@gmail.com>
References: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
 <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>
 <CAHPg78yasqJJiK4e9Bcx764ZfpVRgUQXuhZ_x1vyeprycNwY-w@mail.gmail.com>
 <5f253069-b1c8-0dc1-0fcb-fcfdfba08ebe@gmail.com>
Message-ID: <CAMSv6X3f+LpdU_KPtfQBAYQ27NuW0Ct_KKaqo2eKCAF=S=3aow@mail.gmail.com>

Hi Andrey,

On 14 July 2016 at 17:08, Andrey Rubik <tirnotaure at gmail.com> wrote:
> Jmeter - tool for performance tests. So when i start high load tests written
> in scripting languages i get so much load on jmeter server. That is why I
> choose PyPy - for much speed from testing server.
>
> Sry, but i'am not a developer :(

I think you are missing the point of PyPy: it is not a Java
application at all.  If you want the best performance from a Java
application, you should stick to Jython.  (There are some use cases
where I can think of a reason for tying PyPy to a Java application,
but likely, not yours.)

> Can someone do good java - pypy binding (some .jar file) ?

You are unlikely to find anyone on this list who is interested in Java
enough to help.


A bient?t,

Armin.

From tirnotaure at gmail.com  Fri Jul 15 03:59:09 2016
From: tirnotaure at gmail.com (Andrey Rubik)
Date: Fri, 15 Jul 2016 10:59:09 +0300
Subject: [pypy-dev] integration with java
In-Reply-To: <CAMSv6X3f+LpdU_KPtfQBAYQ27NuW0Ct_KKaqo2eKCAF=S=3aow@mail.gmail.com>
References: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
 <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>
 <CAHPg78yasqJJiK4e9Bcx764ZfpVRgUQXuhZ_x1vyeprycNwY-w@mail.gmail.com>
 <5f253069-b1c8-0dc1-0fcb-fcfdfba08ebe@gmail.com>
 <CAMSv6X3f+LpdU_KPtfQBAYQ27NuW0Ct_KKaqo2eKCAF=S=3aow@mail.gmail.com>
Message-ID: <24e61e1e-a66c-987d-af89-06d40faedf78@gmail.com>

Hi
Ok, thank you for answer. I look at the jython.

-- 
Best Regards,
Andrey Rubik


On 07/15/2016 10:51 AM, Armin Rigo wrote:
> Hi Andrey,
>
> On 14 July 2016 at 17:08, Andrey Rubik <tirnotaure at gmail.com> wrote:
>> Jmeter - tool for performance tests. So when i start high load tests written
>> in scripting languages i get so much load on jmeter server. That is why I
>> choose PyPy - for much speed from testing server.
>>
>> Sry, but i'am not a developer :(
> I think you are missing the point of PyPy: it is not a Java
> application at all.  If you want the best performance from a Java
> application, you should stick to Jython.  (There are some use cases
> where I can think of a reason for tying PyPy to a Java application,
> but likely, not yours.)
>
>> Can someone do good java - pypy binding (some .jar file) ?
> You are unlikely to find anyone on this list who is interested in Java
> enough to help.
>
>
> A bient?t,
>
> Armin.


From phyo.arkarlwin at gmail.com  Fri Jul 15 09:58:27 2016
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Fri, 15 Jul 2016 13:58:27 +0000
Subject: [pypy-dev] integration with java
In-Reply-To: <24e61e1e-a66c-987d-af89-06d40faedf78@gmail.com>
References: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
 <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>
 <CAHPg78yasqJJiK4e9Bcx764ZfpVRgUQXuhZ_x1vyeprycNwY-w@mail.gmail.com>
 <5f253069-b1c8-0dc1-0fcb-fcfdfba08ebe@gmail.com>
 <CAMSv6X3f+LpdU_KPtfQBAYQ27NuW0Ct_KKaqo2eKCAF=S=3aow@mail.gmail.com>
 <24e61e1e-a66c-987d-af89-06d40faedf78@gmail.com>
Message-ID: <CA+HjJrhSyvWBpciYttZG_sjno4V5w_N9omMaS-SNopT7R+5ECQ@mail.gmail.com>

if you want python that works well with Android , check Kivy.org

For most  python programmers we stay away from Java as much as possible. We
are allergic to it.

On Fri, Jul 15, 2016 at 2:30 PM Andrey Rubik <tirnotaure at gmail.com> wrote:

> Hi
> Ok, thank you for answer. I look at the jython.
>
> --
> Best Regards,
> Andrey Rubik
>
>
> On 07/15/2016 10:51 AM, Armin Rigo wrote:
> > Hi Andrey,
> >
> > On 14 July 2016 at 17:08, Andrey Rubik <tirnotaure at gmail.com> wrote:
> >> Jmeter - tool for performance tests. So when i start high load tests
> written
> >> in scripting languages i get so much load on jmeter server. That is why
> I
> >> choose PyPy - for much speed from testing server.
> >>
> >> Sry, but i'am not a developer :(
> > I think you are missing the point of PyPy: it is not a Java
> > application at all.  If you want the best performance from a Java
> > application, you should stick to Jython.  (There are some use cases
> > where I can think of a reason for tying PyPy to a Java application,
> > but likely, not yours.)
> >
> >> Can someone do good java - pypy binding (some .jar file) ?
> > You are unlikely to find anyone on this list who is interested in Java
> > enough to help.
> >
> >
> > A bient?t,
> >
> > Armin.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160715/ca522c68/attachment.html>

From david.brochart at gmail.com  Fri Jul 15 11:47:10 2016
From: david.brochart at gmail.com (David Brochart)
Date: Fri, 15 Jul 2016 17:47:10 +0200
Subject: [pypy-dev] pickle numpy array from pypy to cpython?
In-Reply-To: <CADa34LBwyg=hsA2_daEjPoZ95zAjaE0OtAeqZ0+uL=hfOfpJOw@mail.gmail.com>
References: <CADa34LCdDuA3hfhdFwZRD4gsvFrpMriG__H_a+VmmRwB1BnAtQ@mail.gmail.com>
 <CAHgd1hHaWkO68EzN_Bc=q0FqoXCrdxCW31fQ+PFooLNM=Y3FrA@mail.gmail.com>
 <CADa34LAy1aE-6ziGWRHFKkaLqwXVJs3eajYM9W1u4PzbswuiXQ@mail.gmail.com>
 <CAEv-cC5EJGZii9SA_5MCJsXnrFhyjjRxY38pboQs1dDdeV6BmQ@mail.gmail.com>
 <CADa34LC74vO5XREMHbLRfiNn5gDHPWTrwJMZ_UqKjFLvt7hnOw@mail.gmail.com>
 <F6AEA760-B6D6-4789-9856-CD896D8EA5E8@gmail.com>
 <CADa34LBGQOJOi75UhO0NhOgds5NEEGXXm7pm2W=iUQtqhe-Agw@mail.gmail.com>
 <CADa34LAW4v9HrEEeGNDp2-a9=F1PCH=6iFXWrRHRPTKheVd13w@mail.gmail.com>
 <CADa34LDR5=pTP4VFE0YGZpJQsj6HvYLTy-wO0hm1g75YYw-MEQ@mail.gmail.com>
 <CAO8bCginopNO0J0yyQtrafyJpdc+5UA=qrw3fnqE4NhaeEU+QA@mail.gmail.com>
 <CADa34LAwkKueBKMnnCAbzJFjUUAaqjqpniJ074L1SKypzsQv=A@mail.gmail.com>
 <576EBEB9.9070804@gmail.com>
 <CADa34LA8FiF7=Fh6Si7dqx17bPM5Rf56LOv9upnnDrLbff6JSg@mail.gmail.com>
 <CADa34LB2u3f6Lc_1pvE7Cgh0JQmiS0-FUvHUrwMgoLYrruQSjQ@mail.gmail.com>
 <577413F0.90909@gmail.com>
 <CADa34LApHdXMzyZi7dK-5y7oLdp4xUOuN1y+QdDjRkOdY96q1A@mail.gmail.com>
 <CAMSv6X18QeDoP=Yh3uVyti9a=DjvLTxzp_nr-D8KWD5ozS7iFQ@mail.gmail.com>
 <CADa34LBwyg=hsA2_daEjPoZ95zAjaE0OtAeqZ0+uL=hfOfpJOw@mail.gmail.com>
Message-ID: <CAEv-cC57pN2RP0ZGz7bGguF-XKwBRNp+_seo=0ZR6Bde6kvV+w@mail.gmail.com>

Hi,

I'd like to use the (numerical) performances of PyPy as an equivalent to
Numba's @jit decorator (https://github.com/davidbrochart/piopio). The only
thing preventing that right now is the passing around (pickling) of Numpy
arrays, so it would be great to have that compatibility.

David.

On Mon, Jul 11, 2016 at 6:43 PM, Eli Stevens (Gmail) <wickedgrey at gmail.com>
wrote:

> FWVLIW, I think that conforming to upstream numpy makes the most sense.
>
> I think that the issue would go away if the `_numpypy` module were
> renamed to `numpy` (and appropriate things moved into `numpy.core`).
> Is there a technical reason to keep the actual implementation in a
> separately named module?
>
> Thinking larger picture, would it be possible and sensible to switch
> to using the slow cpyext numpy approach for compatability, then
> overlay custom implementation of things on top of that when speed is
> needed?  I'm imagining a vague inverse of the cpython approach, where
> modules are implemented in C when the python performance isn't
> acceptable.
>
> Eli
>
> On Wed, Jun 29, 2016 at 10:58 PM, Armin Rigo <arigo at tunes.org> wrote:
> > Hi Eli, hi Matti,
> >
> > On 29 June 2016 at 21:37, Eli Stevens (Gmail) <wickedgrey at gmail.com>
> wrote:
> >> To make sure I'm understanding, are you saying that upstream/cpython
> >> numpy should pick up an alternate way to import multiarray (via
> >> _numpypy.multiarray, instead of numpy.core.multiarray)
> >
> > Hum, in my opinion we should always pickle/unpickle arrays by
> > reproducing and expecting the exact same format as CPython's numpy,
> > with no warnings.  Any difference (e.g. with complicated dtypes) is a
> > bug that should eventually be fixed.
> >
> >
> > A bient?t,
> >
> > Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160715/856d7de9/attachment-0001.html>

From matti.picus at gmail.com  Sat Jul 16 06:07:36 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 16 Jul 2016 05:07:36 -0500
Subject: [pypy-dev] pickle numpy array from pypy to cpython?
In-Reply-To: <CAEv-cC57pN2RP0ZGz7bGguF-XKwBRNp+_seo=0ZR6Bde6kvV+w@mail.gmail.com>
References: <CADa34LCdDuA3hfhdFwZRD4gsvFrpMriG__H_a+VmmRwB1BnAtQ@mail.gmail.com>
 <F6AEA760-B6D6-4789-9856-CD896D8EA5E8@gmail.com>
 <CADa34LBGQOJOi75UhO0NhOgds5NEEGXXm7pm2W=iUQtqhe-Agw@mail.gmail.com>
 <CADa34LAW4v9HrEEeGNDp2-a9=F1PCH=6iFXWrRHRPTKheVd13w@mail.gmail.com>
 <CADa34LDR5=pTP4VFE0YGZpJQsj6HvYLTy-wO0hm1g75YYw-MEQ@mail.gmail.com>
 <CAO8bCginopNO0J0yyQtrafyJpdc+5UA=qrw3fnqE4NhaeEU+QA@mail.gmail.com>
 <CADa34LAwkKueBKMnnCAbzJFjUUAaqjqpniJ074L1SKypzsQv=A@mail.gmail.com>
 <576EBEB9.9070804@gmail.com>
 <CADa34LA8FiF7=Fh6Si7dqx17bPM5Rf56LOv9upnnDrLbff6JSg@mail.gmail.com>
 <CADa34LB2u3f6Lc_1pvE7Cgh0JQmiS0-FUvHUrwMgoLYrruQSjQ@mail.gmail.com>
 <577413F0.90909@gmail.com>
 <CADa34LApHdXMzyZi7dK-5y7oLdp4xUOuN1y+QdDjRkOdY96q1A@mail.gmail.com>
 <CAMSv6X18QeDoP=Yh3uVyti9a=DjvLTxzp_nr-D8KWD5ozS7iFQ@mail.gmail.com>
 <CADa34LBwyg=hsA2_daEjPoZ95zAjaE0OtAeqZ0+uL=hfOfpJOw@mail.gmail.com>
 <CAEv-cC57pN2RP0ZGz7bGguF-XKwBRNp+_seo=0ZR6Bde6kvV+w@mail.gmail.com>
Message-ID: <578A0768.9090703@gmail.com>

The issue with '_numpypy.multiarray' in the pickle string rather than 
'numpy.core.multiarray' should be fixed on the numpypy_pickle_compat 
branch (thanks to Eli)
A linux 64 build is available 
http://buildbot.pypy.org/nightly/numpypy_pickle_compat/pypy-c-jit-85727-6d909c810029-linux64.tar.bz2.
Eli or David or anyone who uses numpy pickle, could you check that this 
works as advertised? I am concerned about how compatible our pickling is 
with upstream numpy, but do not really use that feature of numpy so 
another pair of eyes would be nice before merging to default.

Note this requires that http://bitbucket.org/pypy/numpy be installed 
since the Unpickler must be able to import numpy.core.multiarray
Matti

On 15/07/16 10:47, David Brochart wrote:
> Hi,
>
> I'd like to use the (numerical) performances of PyPy as an equivalent 
> to Numba's @jit decorator (https://github.com/davidbrochart/piopio). 
> The only thing preventing that right now is the passing around 
> (pickling) of Numpy arrays, so it would be great to have that 
> compatibility.
>
> David.
>
> On Mon, Jul 11, 2016 at 6:43 PM, Eli Stevens (Gmail) 
> <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>
>     FWVLIW, I think that conforming to upstream numpy makes the most
>     sense.
>
>     I think that the issue would go away if the `_numpypy` module were
>     renamed to `numpy` (and appropriate things moved into `numpy.core`).
>     Is there a technical reason to keep the actual implementation in a
>     separately named module?
>
>     Thinking larger picture, would it be possible and sensible to switch
>     to using the slow cpyext numpy approach for compatability, then
>     overlay custom implementation of things on top of that when speed is
>     needed?  I'm imagining a vague inverse of the cpython approach, where
>     modules are implemented in C when the python performance isn't
>     acceptable.
>
>     Eli
>
>     On Wed, Jun 29, 2016 at 10:58 PM, Armin Rigo <arigo at tunes.org
>     <mailto:arigo at tunes.org>> wrote:
>     > Hi Eli, hi Matti,
>     >
>     > On 29 June 2016 at 21:37, Eli Stevens (Gmail)
>     <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>     >> To make sure I'm understanding, are you saying that
>     upstream/cpython
>     >> numpy should pick up an alternate way to import multiarray (via
>     >> _numpypy.multiarray, instead of numpy.core.multiarray)
>     >
>     > Hum, in my opinion we should always pickle/unpickle arrays by
>     > reproducing and expecting the exact same format as CPython's numpy,
>     > with no warnings.  Any difference (e.g. with complicated dtypes)
>     is a
>     > bug that should eventually be fixed.
>     >
>     >
>     > A bient?t,
>     >
>     > Armin.
>     _______________________________________________
>     pypy-dev mailing list
>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>     https://mail.python.org/mailman/listinfo/pypy-dev
>
>
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev


From arigo at tunes.org  Sat Jul 16 07:46:36 2016
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 16 Jul 2016 13:46:36 +0200
Subject: [pypy-dev] integration with java
In-Reply-To: <CA+HjJrhSyvWBpciYttZG_sjno4V5w_N9omMaS-SNopT7R+5ECQ@mail.gmail.com>
References: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
 <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>
 <CAHPg78yasqJJiK4e9Bcx764ZfpVRgUQXuhZ_x1vyeprycNwY-w@mail.gmail.com>
 <5f253069-b1c8-0dc1-0fcb-fcfdfba08ebe@gmail.com>
 <CAMSv6X3f+LpdU_KPtfQBAYQ27NuW0Ct_KKaqo2eKCAF=S=3aow@mail.gmail.com>
 <24e61e1e-a66c-987d-af89-06d40faedf78@gmail.com>
 <CA+HjJrhSyvWBpciYttZG_sjno4V5w_N9omMaS-SNopT7R+5ECQ@mail.gmail.com>
Message-ID: <CAMSv6X0co+1QX_m5O+QCBWGrXZhYS6h2Bmf46vFC6YVXbw71eA@mail.gmail.com>

Hi Phyo,

On 15 July 2016 at 15:58, Phyo Arkar <phyo.arkarlwin at gmail.com> wrote:
> For most  python programmers we stay away from Java as much as possible. We
> are allergic to it.

That's a broad generalization, maybe not appropriate for this list.


A bient?t,

Armin.

From david.brochart at gmail.com  Sat Jul 16 08:04:42 2016
From: david.brochart at gmail.com (David Brochart)
Date: Sat, 16 Jul 2016 14:04:42 +0200
Subject: [pypy-dev] pickle numpy array from pypy to cpython?
In-Reply-To: <578A0768.9090703@gmail.com>
References: <CADa34LCdDuA3hfhdFwZRD4gsvFrpMriG__H_a+VmmRwB1BnAtQ@mail.gmail.com>
 <F6AEA760-B6D6-4789-9856-CD896D8EA5E8@gmail.com>
 <CADa34LBGQOJOi75UhO0NhOgds5NEEGXXm7pm2W=iUQtqhe-Agw@mail.gmail.com>
 <CADa34LAW4v9HrEEeGNDp2-a9=F1PCH=6iFXWrRHRPTKheVd13w@mail.gmail.com>
 <CADa34LDR5=pTP4VFE0YGZpJQsj6HvYLTy-wO0hm1g75YYw-MEQ@mail.gmail.com>
 <CAO8bCginopNO0J0yyQtrafyJpdc+5UA=qrw3fnqE4NhaeEU+QA@mail.gmail.com>
 <CADa34LAwkKueBKMnnCAbzJFjUUAaqjqpniJ074L1SKypzsQv=A@mail.gmail.com>
 <576EBEB9.9070804@gmail.com>
 <CADa34LA8FiF7=Fh6Si7dqx17bPM5Rf56LOv9upnnDrLbff6JSg@mail.gmail.com>
 <CADa34LB2u3f6Lc_1pvE7Cgh0JQmiS0-FUvHUrwMgoLYrruQSjQ@mail.gmail.com>
 <577413F0.90909@gmail.com>
 <CADa34LApHdXMzyZi7dK-5y7oLdp4xUOuN1y+QdDjRkOdY96q1A@mail.gmail.com>
 <CAMSv6X18QeDoP=Yh3uVyti9a=DjvLTxzp_nr-D8KWD5ozS7iFQ@mail.gmail.com>
 <CADa34LBwyg=hsA2_daEjPoZ95zAjaE0OtAeqZ0+uL=hfOfpJOw@mail.gmail.com>
 <CAEv-cC57pN2RP0ZGz7bGguF-XKwBRNp+_seo=0ZR6Bde6kvV+w@mail.gmail.com>
 <578A0768.9090703@gmail.com>
Message-ID: <CAEv-cC5vcQaT-UfJGtgpFZaysuGSn7VfpLpfFVPhMQJMv5Nb3A@mail.gmail.com>

Hi,

I verified that this version of PyPy can load a Numpy array that was
pickled by CPython (and do stuff with it), but it looks like a Numpy array
pickled by PyPy cannot be loaded by CPython, because PyPy still uses
'_numpypy.multiarray'
in the pickle string for dumping:
ImportError: No module named _numpypy.multiarray

David.

On Sat, Jul 16, 2016 at 12:07 PM, Matti Picus <matti.picus at gmail.com> wrote:

> The issue with '_numpypy.multiarray' in the pickle string rather than
> 'numpy.core.multiarray' should be fixed on the numpypy_pickle_compat branch
> (thanks to Eli)
> A linux 64 build is available
> http://buildbot.pypy.org/nightly/numpypy_pickle_compat/pypy-c-jit-85727-6d909c810029-linux64.tar.bz2
> .
> Eli or David or anyone who uses numpy pickle, could you check that this
> works as advertised? I am concerned about how compatible our pickling is
> with upstream numpy, but do not really use that feature of numpy so another
> pair of eyes would be nice before merging to default.
>
> Note this requires that http://bitbucket.org/pypy/numpy be installed
> since the Unpickler must be able to import numpy.core.multiarray
> Matti
>
> On 15/07/16 10:47, David Brochart wrote:
>
>> Hi,
>>
>> I'd like to use the (numerical) performances of PyPy as an equivalent to
>> Numba's @jit decorator (https://github.com/davidbrochart/piopio). The
>> only thing preventing that right now is the passing around (pickling) of
>> Numpy arrays, so it would be great to have that compatibility.
>>
>> David.
>>
>> On Mon, Jul 11, 2016 at 6:43 PM, Eli Stevens (Gmail) <
>> wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>
>>     FWVLIW, I think that conforming to upstream numpy makes the most
>>     sense.
>>
>>     I think that the issue would go away if the `_numpypy` module were
>>     renamed to `numpy` (and appropriate things moved into `numpy.core`).
>>     Is there a technical reason to keep the actual implementation in a
>>     separately named module?
>>
>>     Thinking larger picture, would it be possible and sensible to switch
>>     to using the slow cpyext numpy approach for compatability, then
>>     overlay custom implementation of things on top of that when speed is
>>     needed?  I'm imagining a vague inverse of the cpython approach, where
>>     modules are implemented in C when the python performance isn't
>>     acceptable.
>>
>>     Eli
>>
>>     On Wed, Jun 29, 2016 at 10:58 PM, Armin Rigo <arigo at tunes.org
>>     <mailto:arigo at tunes.org>> wrote:
>>     > Hi Eli, hi Matti,
>>     >
>>     > On 29 June 2016 at 21:37, Eli Stevens (Gmail)
>>     <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>     >> To make sure I'm understanding, are you saying that
>>     upstream/cpython
>>     >> numpy should pick up an alternate way to import multiarray (via
>>     >> _numpypy.multiarray, instead of numpy.core.multiarray)
>>     >
>>     > Hum, in my opinion we should always pickle/unpickle arrays by
>>     > reproducing and expecting the exact same format as CPython's numpy,
>>     > with no warnings.  Any difference (e.g. with complicated dtypes)
>>     is a
>>     > bug that should eventually be fixed.
>>     >
>>     >
>>     > A bient?t,
>>     >
>>     > Armin.
>>     _______________________________________________
>>     pypy-dev mailing list
>>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>>     https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>>
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160716/da1ef18a/attachment-0001.html>

From david.brochart at gmail.com  Sat Jul 16 08:40:10 2016
From: david.brochart at gmail.com (David Brochart)
Date: Sat, 16 Jul 2016 14:40:10 +0200
Subject: [pypy-dev] pickle numpy array from pypy to cpython?
In-Reply-To: <CAEv-cC5vcQaT-UfJGtgpFZaysuGSn7VfpLpfFVPhMQJMv5Nb3A@mail.gmail.com>
References: <CADa34LCdDuA3hfhdFwZRD4gsvFrpMriG__H_a+VmmRwB1BnAtQ@mail.gmail.com>
 <F6AEA760-B6D6-4789-9856-CD896D8EA5E8@gmail.com>
 <CADa34LBGQOJOi75UhO0NhOgds5NEEGXXm7pm2W=iUQtqhe-Agw@mail.gmail.com>
 <CADa34LAW4v9HrEEeGNDp2-a9=F1PCH=6iFXWrRHRPTKheVd13w@mail.gmail.com>
 <CADa34LDR5=pTP4VFE0YGZpJQsj6HvYLTy-wO0hm1g75YYw-MEQ@mail.gmail.com>
 <CAO8bCginopNO0J0yyQtrafyJpdc+5UA=qrw3fnqE4NhaeEU+QA@mail.gmail.com>
 <CADa34LAwkKueBKMnnCAbzJFjUUAaqjqpniJ074L1SKypzsQv=A@mail.gmail.com>
 <576EBEB9.9070804@gmail.com>
 <CADa34LA8FiF7=Fh6Si7dqx17bPM5Rf56LOv9upnnDrLbff6JSg@mail.gmail.com>
 <CADa34LB2u3f6Lc_1pvE7Cgh0JQmiS0-FUvHUrwMgoLYrruQSjQ@mail.gmail.com>
 <577413F0.90909@gmail.com>
 <CADa34LApHdXMzyZi7dK-5y7oLdp4xUOuN1y+QdDjRkOdY96q1A@mail.gmail.com>
 <CAMSv6X18QeDoP=Yh3uVyti9a=DjvLTxzp_nr-D8KWD5ozS7iFQ@mail.gmail.com>
 <CADa34LBwyg=hsA2_daEjPoZ95zAjaE0OtAeqZ0+uL=hfOfpJOw@mail.gmail.com>
 <CAEv-cC57pN2RP0ZGz7bGguF-XKwBRNp+_seo=0ZR6Bde6kvV+w@mail.gmail.com>
 <578A0768.9090703@gmail.com>
 <CAEv-cC5vcQaT-UfJGtgpFZaysuGSn7VfpLpfFVPhMQJMv5Nb3A@mail.gmail.com>
Message-ID: <CAEv-cC6N1H3W=7DSpuQeX-Kd0uZxXcWc1_zgt4Vdb9tS3Sg93g@mail.gmail.com>

To be more precise, PyPy pickling of Numpy arrays works fine, it is when
PyPy pickles a Numpy scalar that I get the error.
David.

On Sat, Jul 16, 2016 at 2:04 PM, David Brochart <david.brochart at gmail.com>
wrote:

> Hi,
>
> I verified that this version of PyPy can load a Numpy array that was
> pickled by CPython (and do stuff with it), but it looks like a Numpy array
> pickled by PyPy cannot be loaded by CPython, because PyPy still uses  '_numpypy.multiarray'
> in the pickle string for dumping:
> ImportError: No module named _numpypy.multiarray
>
> David.
>
> On Sat, Jul 16, 2016 at 12:07 PM, Matti Picus <matti.picus at gmail.com>
> wrote:
>
>> The issue with '_numpypy.multiarray' in the pickle string rather than
>> 'numpy.core.multiarray' should be fixed on the numpypy_pickle_compat branch
>> (thanks to Eli)
>> A linux 64 build is available
>> http://buildbot.pypy.org/nightly/numpypy_pickle_compat/pypy-c-jit-85727-6d909c810029-linux64.tar.bz2
>> .
>> Eli or David or anyone who uses numpy pickle, could you check that this
>> works as advertised? I am concerned about how compatible our pickling is
>> with upstream numpy, but do not really use that feature of numpy so another
>> pair of eyes would be nice before merging to default.
>>
>> Note this requires that http://bitbucket.org/pypy/numpy be installed
>> since the Unpickler must be able to import numpy.core.multiarray
>> Matti
>>
>> On 15/07/16 10:47, David Brochart wrote:
>>
>>> Hi,
>>>
>>> I'd like to use the (numerical) performances of PyPy as an equivalent to
>>> Numba's @jit decorator (https://github.com/davidbrochart/piopio). The
>>> only thing preventing that right now is the passing around (pickling) of
>>> Numpy arrays, so it would be great to have that compatibility.
>>>
>>> David.
>>>
>>> On Mon, Jul 11, 2016 at 6:43 PM, Eli Stevens (Gmail) <
>>> wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>>
>>>     FWVLIW, I think that conforming to upstream numpy makes the most
>>>     sense.
>>>
>>>     I think that the issue would go away if the `_numpypy` module were
>>>     renamed to `numpy` (and appropriate things moved into `numpy.core`).
>>>     Is there a technical reason to keep the actual implementation in a
>>>     separately named module?
>>>
>>>     Thinking larger picture, would it be possible and sensible to switch
>>>     to using the slow cpyext numpy approach for compatability, then
>>>     overlay custom implementation of things on top of that when speed is
>>>     needed?  I'm imagining a vague inverse of the cpython approach, where
>>>     modules are implemented in C when the python performance isn't
>>>     acceptable.
>>>
>>>     Eli
>>>
>>>     On Wed, Jun 29, 2016 at 10:58 PM, Armin Rigo <arigo at tunes.org
>>>     <mailto:arigo at tunes.org>> wrote:
>>>     > Hi Eli, hi Matti,
>>>     >
>>>     > On 29 June 2016 at 21:37, Eli Stevens (Gmail)
>>>     <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>>     >> To make sure I'm understanding, are you saying that
>>>     upstream/cpython
>>>     >> numpy should pick up an alternate way to import multiarray (via
>>>     >> _numpypy.multiarray, instead of numpy.core.multiarray)
>>>     >
>>>     > Hum, in my opinion we should always pickle/unpickle arrays by
>>>     > reproducing and expecting the exact same format as CPython's numpy,
>>>     > with no warnings.  Any difference (e.g. with complicated dtypes)
>>>     is a
>>>     > bug that should eventually be fixed.
>>>     >
>>>     >
>>>     > A bient?t,
>>>     >
>>>     > Armin.
>>>     _______________________________________________
>>>     pypy-dev mailing list
>>>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>>>     https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160716/640cbbb5/attachment.html>

From matti.picus at gmail.com  Sat Jul 16 12:29:19 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 16 Jul 2016 11:29:19 -0500
Subject: [pypy-dev] pickle numpy array from pypy to cpython?
In-Reply-To: <CAEv-cC6N1H3W=7DSpuQeX-Kd0uZxXcWc1_zgt4Vdb9tS3Sg93g@mail.gmail.com>
References: <CADa34LCdDuA3hfhdFwZRD4gsvFrpMriG__H_a+VmmRwB1BnAtQ@mail.gmail.com>
 <CADa34LAW4v9HrEEeGNDp2-a9=F1PCH=6iFXWrRHRPTKheVd13w@mail.gmail.com>
 <CADa34LDR5=pTP4VFE0YGZpJQsj6HvYLTy-wO0hm1g75YYw-MEQ@mail.gmail.com>
 <CAO8bCginopNO0J0yyQtrafyJpdc+5UA=qrw3fnqE4NhaeEU+QA@mail.gmail.com>
 <CADa34LAwkKueBKMnnCAbzJFjUUAaqjqpniJ074L1SKypzsQv=A@mail.gmail.com>
 <576EBEB9.9070804@gmail.com>
 <CADa34LA8FiF7=Fh6Si7dqx17bPM5Rf56LOv9upnnDrLbff6JSg@mail.gmail.com>
 <CADa34LB2u3f6Lc_1pvE7Cgh0JQmiS0-FUvHUrwMgoLYrruQSjQ@mail.gmail.com>
 <577413F0.90909@gmail.com>
 <CADa34LApHdXMzyZi7dK-5y7oLdp4xUOuN1y+QdDjRkOdY96q1A@mail.gmail.com>
 <CAMSv6X18QeDoP=Yh3uVyti9a=DjvLTxzp_nr-D8KWD5ozS7iFQ@mail.gmail.com>
 <CADa34LBwyg=hsA2_daEjPoZ95zAjaE0OtAeqZ0+uL=hfOfpJOw@mail.gmail.com>
 <CAEv-cC57pN2RP0ZGz7bGguF-XKwBRNp+_seo=0ZR6Bde6kvV+w@mail.gmail.com>
 <578A0768.9090703@gmail.com>
 <CAEv-cC5vcQaT-UfJGtgpFZaysuGSn7VfpLpfFVPhMQJMv5Nb3A@mail.gmail.com>
 <CAEv-cC6N1H3W=7DSpuQeX-Kd0uZxXcWc1_zgt4Vdb9tS3Sg93g@mail.gmail.com>
Message-ID: <578A60DF.9080100@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160716/d6d21289/attachment.html>

From phyo.arkarlwin at gmail.com  Sat Jul 16 12:43:42 2016
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Sat, 16 Jul 2016 16:43:42 +0000
Subject: [pypy-dev] integration with java
In-Reply-To: <CAMSv6X0co+1QX_m5O+QCBWGrXZhYS6h2Bmf46vFC6YVXbw71eA@mail.gmail.com>
References: <05d42292-4f48-3c16-741a-d0e688fdd0e1@gmail.com>
 <CAK5idxQAci=J-wWjq5iXLALAns49Di3DGuwQy-KwZVUgM7a-Ng@mail.gmail.com>
 <CAHPg78yasqJJiK4e9Bcx764ZfpVRgUQXuhZ_x1vyeprycNwY-w@mail.gmail.com>
 <5f253069-b1c8-0dc1-0fcb-fcfdfba08ebe@gmail.com>
 <CAMSv6X3f+LpdU_KPtfQBAYQ27NuW0Ct_KKaqo2eKCAF=S=3aow@mail.gmail.com>
 <24e61e1e-a66c-987d-af89-06d40faedf78@gmail.com>
 <CA+HjJrhSyvWBpciYttZG_sjno4V5w_N9omMaS-SNopT7R+5ECQ@mail.gmail.com>
 <CAMSv6X0co+1QX_m5O+QCBWGrXZhYS6h2Bmf46vFC6YVXbw71eA@mail.gmail.com>
Message-ID: <CA+HjJrjCz1T_r6usjKrm+jjV5e2DGa1AW5JOMYDjzQcOMD_h=w@mail.gmail.com>

sorry i just mean it as a joke.

On Sat, Jul 16, 2016 at 6:17 PM Armin Rigo <arigo at tunes.org> wrote:

> Hi Phyo,
>
> On 15 July 2016 at 15:58, Phyo Arkar <phyo.arkarlwin at gmail.com> wrote:
> > For most  python programmers we stay away from Java as much as possible.
> We
> > are allergic to it.
>
> That's a broad generalization, maybe not appropriate for this list.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160716/d464be21/attachment.html>

From wickedgrey at gmail.com  Sat Jul 16 14:12:49 2016
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Sat, 16 Jul 2016 11:12:49 -0700
Subject: [pypy-dev] pickle numpy array from pypy to cpython?
In-Reply-To: <578A60DF.9080100@gmail.com>
References: <CADa34LCdDuA3hfhdFwZRD4gsvFrpMriG__H_a+VmmRwB1BnAtQ@mail.gmail.com>
 <CADa34LAW4v9HrEEeGNDp2-a9=F1PCH=6iFXWrRHRPTKheVd13w@mail.gmail.com>
 <CADa34LDR5=pTP4VFE0YGZpJQsj6HvYLTy-wO0hm1g75YYw-MEQ@mail.gmail.com>
 <CAO8bCginopNO0J0yyQtrafyJpdc+5UA=qrw3fnqE4NhaeEU+QA@mail.gmail.com>
 <CADa34LAwkKueBKMnnCAbzJFjUUAaqjqpniJ074L1SKypzsQv=A@mail.gmail.com>
 <576EBEB9.9070804@gmail.com>
 <CADa34LA8FiF7=Fh6Si7dqx17bPM5Rf56LOv9upnnDrLbff6JSg@mail.gmail.com>
 <CADa34LB2u3f6Lc_1pvE7Cgh0JQmiS0-FUvHUrwMgoLYrruQSjQ@mail.gmail.com>
 <577413F0.90909@gmail.com>
 <CADa34LApHdXMzyZi7dK-5y7oLdp4xUOuN1y+QdDjRkOdY96q1A@mail.gmail.com>
 <CAMSv6X18QeDoP=Yh3uVyti9a=DjvLTxzp_nr-D8KWD5ozS7iFQ@mail.gmail.com>
 <CADa34LBwyg=hsA2_daEjPoZ95zAjaE0OtAeqZ0+uL=hfOfpJOw@mail.gmail.com>
 <CAEv-cC57pN2RP0ZGz7bGguF-XKwBRNp+_seo=0ZR6Bde6kvV+w@mail.gmail.com>
 <578A0768.9090703@gmail.com>
 <CAEv-cC5vcQaT-UfJGtgpFZaysuGSn7VfpLpfFVPhMQJMv5Nb3A@mail.gmail.com>
 <CAEv-cC6N1H3W=7DSpuQeX-Kd0uZxXcWc1_zgt4Vdb9tS3Sg93g@mail.gmail.com>
 <578A60DF.9080100@gmail.com>
Message-ID: <CADa34LCCqb0rre=-+KjO8EjqBBnOq-+dSdLSyUM0wcpnR0kf9g@mail.gmail.com>

I am not surprised that my current branch doesn't cover all cases; it
was specifically targeted at my exact, singular use case.

I'll work on making something more general, as well as improving test coverage.

On Sat, Jul 16, 2016 at 9:29 AM, Matti Picus <matti.picus at gmail.com> wrote:
> So it seems the tests are lacking. Someone should:
> - go through all the existing calls to dumps in tests and add "assert
> '_numpypy' not in data"
> - add tests for scalars
> - fix so the tests pass
> Matti
>
>
> On 16/07/16 07:40, David Brochart wrote:
>
> To be more precise, PyPy pickling of Numpy arrays works fine, it is when
> PyPy pickles a Numpy scalar that I get the error.
> David.
>
> On Sat, Jul 16, 2016 at 2:04 PM, David Brochart <david.brochart at gmail.com>
> wrote:
>>
>> Hi,
>>
>> I verified that this version of PyPy can load a Numpy array that was
>> pickled by CPython (and do stuff with it), but it looks like a Numpy array
>> pickled by PyPy cannot be loaded by CPython, because PyPy still uses
>> '_numpypy.multiarray' in the pickle string for dumping:
>> ImportError: No module named _numpypy.multiarray
>>
>> David.
>>
>> On Sat, Jul 16, 2016 at 12:07 PM, Matti Picus <matti.picus at gmail.com>
>> wrote:
>>>
>>> The issue with '_numpypy.multiarray' in the pickle string rather than
>>> 'numpy.core.multiarray' should be fixed on the numpypy_pickle_compat branch
>>> (thanks to Eli)
>>> A linux 64 build is available
>>> http://buildbot.pypy.org/nightly/numpypy_pickle_compat/pypy-c-jit-85727-6d909c810029-linux64.tar.bz2.
>>> Eli or David or anyone who uses numpy pickle, could you check that this
>>> works as advertised? I am concerned about how compatible our pickling is
>>> with upstream numpy, but do not really use that feature of numpy so another
>>> pair of eyes would be nice before merging to default.
>>>
>>> Note this requires that http://bitbucket.org/pypy/numpy be installed
>>> since the Unpickler must be able to import numpy.core.multiarray
>>> Matti
>>>
>>> On 15/07/16 10:47, David Brochart wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'd like to use the (numerical) performances of PyPy as an equivalent to
>>>> Numba's @jit decorator (https://github.com/davidbrochart/piopio). The only
>>>> thing preventing that right now is the passing around (pickling) of Numpy
>>>> arrays, so it would be great to have that compatibility.
>>>>
>>>> David.
>>>>
>>>> On Mon, Jul 11, 2016 at 6:43 PM, Eli Stevens (Gmail)
>>>> <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>>>
>>>>     FWVLIW, I think that conforming to upstream numpy makes the most
>>>>     sense.
>>>>
>>>>     I think that the issue would go away if the `_numpypy` module were
>>>>     renamed to `numpy` (and appropriate things moved into `numpy.core`).
>>>>     Is there a technical reason to keep the actual implementation in a
>>>>     separately named module?
>>>>
>>>>     Thinking larger picture, would it be possible and sensible to switch
>>>>     to using the slow cpyext numpy approach for compatability, then
>>>>     overlay custom implementation of things on top of that when speed is
>>>>     needed?  I'm imagining a vague inverse of the cpython approach,
>>>> where
>>>>     modules are implemented in C when the python performance isn't
>>>>     acceptable.
>>>>
>>>>     Eli
>>>>
>>>>     On Wed, Jun 29, 2016 at 10:58 PM, Armin Rigo <arigo at tunes.org
>>>>     <mailto:arigo at tunes.org>> wrote:
>>>>     > Hi Eli, hi Matti,
>>>>     >
>>>>     > On 29 June 2016 at 21:37, Eli Stevens (Gmail)
>>>>     <wickedgrey at gmail.com <mailto:wickedgrey at gmail.com>> wrote:
>>>>     >> To make sure I'm understanding, are you saying that
>>>>     upstream/cpython
>>>>     >> numpy should pick up an alternate way to import multiarray (via
>>>>     >> _numpypy.multiarray, instead of numpy.core.multiarray)
>>>>     >
>>>>     > Hum, in my opinion we should always pickle/unpickle arrays by
>>>>     > reproducing and expecting the exact same format as CPython's
>>>> numpy,
>>>>     > with no warnings.  Any difference (e.g. with complicated dtypes)
>>>>     is a
>>>>     > bug that should eventually be fixed.
>>>>     >
>>>>     >
>>>>     > A bient?t,
>>>>     >
>>>>     > Armin.
>>>>     _______________________________________________
>>>>     pypy-dev mailing list
>>>>     pypy-dev at python.org <mailto:pypy-dev at python.org>
>>>>     https://mail.python.org/mailman/listinfo/pypy-dev
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> pypy-dev mailing list
>>>> pypy-dev at python.org
>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>>
>>
>
>

From arigo at tunes.org  Sun Jul 17 03:48:53 2016
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 17 Jul 2016 09:48:53 +0200
Subject: [pypy-dev] Twisted's deferToThread is much more slower with
 pypy than on cpython
In-Reply-To: <5751359D.5020901@fsn.hu>
References: <5751359D.5020901@fsn.hu>
Message-ID: <CAMSv6X2wgWhY8yZ-UM03ZM_fEq4=eJmkf4Y=nh_wwqJ40ViASQ@mail.gmail.com>

Hi Nagy,

On 3 June 2016 at 09:45, Nagy, Attila <bra at fsn.hu> wrote:
> Consider this example program:
> https://gist.github.com/bra-fsn/1fd481b44590a939e849cb9073ba1a41

I think I fixed this problem in 919e00b3e558 two days ago.  Now it
seems to always use about 100% CPU and gets performance that is a bit
better than CPython, instead of spending all its time sleeping.  Yay
:-)

I did some tests to measure how well PyPy and CPython perform when
running 2 or 3 threads in various microbenchmark-like situations
(running pure Python code; acquiring and releasing the same lock;
ping-pong between two threads; calling a fast or slow C function).
Now in all measured cases PyPy should perform not too badly.  Actually
in most cases it was already better than CPython for fairness: for
example, when one thread runs pure Python code and the other thread
does many calls to a very fast C function, then CPython gives about
0.002% of the time(!) to the second thread and the rest to the first
one.

These tests can be found in
https://bitbucket.org/arigo/arigo/src/default/hack/pypy-hack/gil-benchmark/
.


A bient?t,

Armin.

From bra at fsn.hu  Mon Jul 18 05:24:38 2016
From: bra at fsn.hu (Nagy, Attila)
Date: Mon, 18 Jul 2016 11:24:38 +0200
Subject: [pypy-dev] Twisted's deferToThread is much more slower with
 pypy than on cpython
In-Reply-To: <CAMSv6X2wgWhY8yZ-UM03ZM_fEq4=eJmkf4Y=nh_wwqJ40ViASQ@mail.gmail.com>
References: <5751359D.5020901@fsn.hu>
 <CAMSv6X2wgWhY8yZ-UM03ZM_fEq4=eJmkf4Y=nh_wwqJ40ViASQ@mail.gmail.com>
Message-ID: <578CA056.4090106@fsn.hu>

Hi,

On 07/17/2016 09:48 AM, Armin Rigo wrote:
> Hi Nagy,
>
> On 3 June 2016 at 09:45, Nagy, Attila <bra at fsn.hu> wrote:
>> Consider this example program:
>> https://gist.github.com/bra-fsn/1fd481b44590a939e849cb9073ba1a41
> I think I fixed this problem in 919e00b3e558 two days ago.  Now it
> seems to always use about 100% CPU and gets performance that is a bit
> better than CPython, instead of spending all its time sleeping.  Yay
> :-)
>
> I did some tests to measure how well PyPy and CPython perform when
> running 2 or 3 threads in various microbenchmark-like situations
> (running pure Python code; acquiring and releasing the same lock;
> ping-pong between two threads; calling a fast or slow C function).
> Now in all measured cases PyPy should perform not too badly.  Actually
> in most cases it was already better than CPython for fairness: for
> example, when one thread runs pure Python code and the other thread
> does many calls to a very fast C function, then CPython gives about
> 0.002% of the time(!) to the second thread and the rest to the first
> one.
>
>
Thank you very much for taking care about this.

You rock. :)

From phyo.arkarlwin at gmail.com  Thu Jul 21 18:15:44 2016
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Thu, 21 Jul 2016 22:15:44 +0000
Subject: [pypy-dev] Microsoft is starting its own JIT for Python.
Message-ID: <CA+HjJrip7e_rLzgaUSv8ipOphY+Wkebvtebak08j4tZmyMQPuA@mail.gmail.com>

Hello PyPy Team;

I guess you guys already known.
Microsft is starting its own pypy : https://github.com/Microsoft/Pyjion

The main reason they mention there is due to weak CPyext support but It is
just duplicating work, as you guys are already working on it .

Shouldn't microsoft contribute into CPyext work in pypy instead?

Regards,

Phyo.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160721/5fb997c0/attachment.html>

From wickedgrey at gmail.com  Thu Jul 21 19:08:46 2016
From: wickedgrey at gmail.com (Eli Stevens (Gmail))
Date: Thu, 21 Jul 2016 16:08:46 -0700
Subject: [pypy-dev] Microsoft is starting its own JIT for Python.
In-Reply-To: <CA+HjJrip7e_rLzgaUSv8ipOphY+Wkebvtebak08j4tZmyMQPuA@mail.gmail.com>
References: <CA+HjJrip7e_rLzgaUSv8ipOphY+Wkebvtebak08j4tZmyMQPuA@mail.gmail.com>
Message-ID: <CADa34LA5WzUWAWPW1QC4RbS3M=yWhbeXq8327g5iJ8E4eirLng@mail.gmail.com>

Isn't the blurb about PyPy now incorrect, with the cpyext advancements?

"""PyPy is an implementation of Python with its own JIT. The biggest
difference compared to Pyjion is that PyPy doesn't support all C
extension modules without modification unless they use CFFI or work
with the select subset of CPython's C API that PyPy does support.
Pyjion also aims to support many JIT compilers while PyPy only
supports their custom JIT compiler."""

Eli

On Thu, Jul 21, 2016 at 3:15 PM, Phyo Arkar <phyo.arkarlwin at gmail.com> wrote:
> Hello PyPy Team;
>
> I guess you guys already known.
> Microsft is starting its own pypy : https://github.com/Microsoft/Pyjion
>
> The main reason they mention there is due to weak CPyext support but It is
> just duplicating work, as you guys are already working on it .
>
> Shouldn't microsoft contribute into CPyext work in pypy instead?
>
> Regards,
>
> Phyo.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>

From njs at pobox.com  Thu Jul 21 19:27:22 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 21 Jul 2016 16:27:22 -0700
Subject: [pypy-dev] Microsoft is starting its own JIT for Python.
In-Reply-To: <CAPJVwBnkvYDy31DBwHruEoiTJ6Ma3_fhnfmA02+SH0R2AT+yOg@mail.gmail.com>
References: <CA+HjJrip7e_rLzgaUSv8ipOphY+Wkebvtebak08j4tZmyMQPuA@mail.gmail.com>
 <CAPJVwBnkvYDy31DBwHruEoiTJ6Ma3_fhnfmA02+SH0R2AT+yOg@mail.gmail.com>
Message-ID: <CAPJVwBmXt3=Z12fES3GKZwcQvEs-xVsqG6oVn+bcbVoKqbefBw@mail.gmail.com>

It's not "Microsoft", it's an experimental side project of two people who
happen to work at Microsoft. As I understand it, the core motivation is to
explore how far they can get in dragging upstream CPython in a JITwards
direction -- so it's written as a extension module for CPython, and it
requires a very recent version of CPython (no support for python 2!).

The advantage is that it's *extremely* compatible with CPython -- unlike
pypy, it handles the latest version of the language, it has the same
garbage collection rules, and not only do they get essentially perfect
extension module compatibility, they get it with zero speed penalty. They
can also potentially explore different compilation strategies (e.g.
function at a time) that aren't as easy a fit to pypy's meta-tracing
architecture, and can use it as a lever to push CPython to start adding
features like dict versioning.

The disadvantage versus PyPy obviously is that it's nowhere near as mature,
there are reasons to doubt that their optimization strategy could ever be
competitive with PyPy's given the need for CPython compatibility, and PyPy
could conceivably catch up on the relevant axes (better faster cpyext,
python 3 support, etc.) -- though this isn't a given either. Right now I
think it makes more sense to think of it as an interesting experiment to
watch than as a serious product that one hopes to use.

In any case, one can't really get mad at some individuals for investing
their spare time in the wrong thing. This is definitely not some strategic
initiative from Microsoft-the-company.

-n

On Jul 21, 2016 3:16 PM, "Phyo Arkar" <phyo.arkarlwin at gmail.com> wrote:

Hello PyPy Team;

I guess you guys already known.
Microsft is starting its own pypy : https://github.com/Microsoft/Pyjion

The main reason they mention there is due to weak CPyext support but It is
just duplicating work, as you guys are already working on it .

Shouldn't microsoft contribute into CPyext work in pypy instead?

Regards,

Phyo.

_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160721/8d83fe38/attachment.html>

From naylor.b.david at gmail.com  Sun Jul 24 07:45:26 2016
From: naylor.b.david at gmail.com (David Naylor)
Date: Sun, 24 Jul 2016 13:45:26 +0200
Subject: [pypy-dev] PyPy3: Translation issue: "TyperError: arithmetic not
 supported on <INT>, its size is too small"
Message-ID: <1637527.H5yKrObyDF@dragon.local>

Hi

While translating the latest version of pypy3 on FreeBSD (and after applying 
the fix for https://bitbucket.org/pypy/pypy/issues/2348) I get the following 
error:

File "/usr/local/home/dbn/ports/ports/lang/pypy3/work/pypy3.3-v5.2.0-alpha1-
src/rpython/rtyper/rint.py", line 25, in opprefix
    self.lowleveltype)
[translation:ERROR] TyperError: arithmetic not supported on <INT>, its size is 
too small
.. (pypy.module.time.interp_time:988)clock
.. block at 3 with 2 exits(v1573)
.. v1575 = eq(value_25, v1574)

The error occurs when translating pypy/module/time/interp_time, line 996:

        if value == rffi.cast(clock_t, -1):

Although this can be easily fixed by casting value to r_int instead, it then 
causes another issue on line 969:

        cpu_time = float(tms.c_tms_utime + tms.c_tms_stime)

Again, same issue of TyperError with <INT> (this time, instead of with clock_t 
[1][2] but with TMS.tms_utime [3]).  

I have been unable to figure out how a 
rpython.rtyper.lltypesystem.lltype.Number (which a believe an INT is) gets 
converted into a rpython.rtyper.rint.IntegerRepr, or why it gets converted 
without an opprefix set.  

Any suggestions how to fix this issue, or what the root cause is, will be 
greatly appreciated.  

Regards

David

[1] clock_t is defined, in platcheck_73 as:
-+- clock_t
size: 4
unsigned: 0
---
[2] On FreeBSD clock_t is typedef as unsigned long.
[3] TMS is defined in platcheck_7 as:
-+- TMS
align: 4
size: 16
fldofs tms_utime: 0
fldsize tms_utime: 4
fldunsigned tms_utime: 0
fldofs tms_stime: 4
fldsize tms_stime: 4
fldunsigned tms_stime: 0
fldofs tms_cutime: 8
fldsize tms_cutime: 4
fldunsigned tms_cutime: 0
fldofs tms_cstime: 12
fldsize tms_cstime: 4
fldunsigned tms_cstime: 0
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160724/4d2b49a6/attachment.sig>

From marky1991 at gmail.com  Sun Jul 24 13:13:32 2016
From: marky1991 at gmail.com (marky1991 .)
Date: Sun, 24 Jul 2016 13:13:32 -0400
Subject: [pypy-dev] PyPy3: Translation issue: "TyperError: arithmetic
 not supported on <INT>, its size is too small"
In-Reply-To: <1637527.H5yKrObyDF@dragon.local>
References: <1637527.H5yKrObyDF@dragon.local>
Message-ID: <CAG3cHaZiwXC-LujbfNWPAGnzPA5hYRJvd-uUEyaJ4EVDnzWc7Q@mail.gmail.com>

I will try setting up a freebsd box to look at the translation issues.
Sorry about that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160724/640d4168/attachment.html>

From florin.papa at intel.com  Wed Jul 27 02:59:46 2016
From: florin.papa at intel.com (Papa, Florin)
Date: Wed, 27 Jul 2016 06:59:46 +0000
Subject: [pypy-dev] NumPyPy vs NumPy
Message-ID: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>

Hi,

This is Florin Papa from the Dynamic Scripting Languages Optimizations Team at Intel Corporation.

I have been working with NumPyPy to evaluate its performance and it seems significantly slower compared to CPython NumPy or even PyPy NumPy (installed with pip). The results were gathered after running microbenchmarks inspired from here: http://www.labri.fr/perso/nrougier/teaching/numpy.100/

These benchmarks perform basic tasks, such as matrix multiplication, generate Cauchy matrix, generate Gaussian array, find min and max in matrix, convert float array to integer array in place, sum all elements in array. Please see below the results containing run time, normalized to the CPython NumPy results (baseline).

Benchmark   CPython NumPy   PyPy NumPy      PyPy NumPyPy
cauchy          1           5.838852812     4.866947551
pointbypoint    1           4.922654347     0.981008211
numrand         1           2.478997019     1.082185897
rowmean         1           2.512893263     1.062233015
dsums           1           33.58240465     1.013388981
vectsum         1           1.738446611     0.771660704
cauchy          1           2.168377906     0.887388291
polarcoords     1           1.030962402     0.500905427
vectsort        1           2.214586698     0.973727924
arange          1           2.045342386     0.69941044
vectoradd       1           5.447667037     1.513217941
extractint      1           1.655717606     2.671712185
float2int       1           3.1688          0.905406988
insertzeros     1           2.375043445     1.037504453

Is there an official benchmark suite for NumPy or a more relevant workload to compare against CPython? What is NumPyPy's maturity / adoption rate from your knowledge?

The benchmarks used to collect the results are attached in this mail.

Regards,
Florin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpy_bench.zip
Type: application/x-zip-compressed
Size: 6055 bytes
Desc: numpy_bench.zip
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160727/003120bb/attachment.bin>

From yury at shurup.com  Wed Jul 27 03:32:53 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Wed, 27 Jul 2016 09:32:53 +0200 (CEST)
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
Message-ID: <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>

Hi,

On Wed, 27 Jul 2016, Papa, Florin wrote:

> I have been working with NumPyPy to evaluate its performance and it 
> seems significantly slower compared to CPython NumPy or even PyPy NumPy 
> (installed with pip).

After having a brief look at the your table, I'm very confused by this 
assessment:

To me, it seems that PyPy NumPyPy is equal or significantly faster than 
CPython NumPy on most benchmarks, but substantially slower on just a few 
of them.

PyPy NumPy is slower than CPython NumPy on all benchmarks, with some being 
not that bad, and some pretty bad, but this is absolutely to be expected, 
and in fact nevertheless very impressive, considering that it runs via 
CPyExt...

Am I completely misinterpreting your numbers?!

-- 
Sincerely yours,
Yury V. Zaytsev

From planrichi at gmail.com  Wed Jul 27 03:48:52 2016
From: planrichi at gmail.com (Richard Plangger)
Date: Wed, 27 Jul 2016 09:48:52 +0200
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
Message-ID: <47de2d75-efd6-16ae-530a-77f98ae16fc3@gmail.com>

Hi,

> Is there an official benchmark suite for NumPy or a more relevant workload to compare against CPython? What is NumPyPy's maturity / adoption rate from your knowledge?

I do not think there is. I have been looking for something similar for
over a year. It seems though people tend to make their own benchmarks
for their own jit compiler, stressing their optimization. (Well, I did
too for my thesis)

Having that said, it would be beneficial task to sit down and extract
such a benchmark set not targeting a special jit/aot compiler, but
rather thinking about real world application workloads.

> I have been working with NumPyPy to evaluate its performance and it
seems significantly slower compared to CPython NumPy or even PyPy NumPy
(installed with pip).

I agree with Yury, there are 2-3 benchmarks for NumPyPy where it
performs worse than cpython. All others are not significant.

Have you tried turning on the beta verion of the vectorizer in NumPyPy?
(command is $ pypy --jit vec=1 program.py args)

Cheers,
Richard

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160727/8f4bad5e/attachment.sig>

From florin.papa at intel.com  Wed Jul 27 03:59:26 2016
From: florin.papa at intel.com (Papa, Florin)
Date: Wed, 27 Jul 2016 07:59:26 +0000
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>
Message-ID: <3A375A669FBEFF45B6B60E689636EDCA09C05AC7@IRSMSX101.ger.corp.intel.com>

Hi Yury,

The table contains run time values, normalized to the CPython Numpy results. This means that a value of 1 is equal to the CPython NumPy result, less than 1 means faster than CPython NumPy and more than 1 is slower than CPython NumPy.

Let's consider the following line in the table:
Benchmark   CPython NumPy   PyPy NumPy      PyPy NumPyPy
cauchy          1           5.838852812     4.866947551

Here, PyPy NumPy is 5.83 times slower than CPython NumPy and PyPy NumPyPy is 4.86 times slower than CPython NumPy.

Hope this makes the results table more clear. 

Regards,
Florin

From yury at shurup.com  Wed Jul 27 04:16:47 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Wed, 27 Jul 2016 10:16:47 +0200 (CEST)
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <3A375A669FBEFF45B6B60E689636EDCA09C05AC7@IRSMSX101.ger.corp.intel.com>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C05AC7@IRSMSX101.ger.corp.intel.com>
Message-ID: <alpine.LRH.2.20.1607271010450.13656@vps.zaytsev.net>

Hi Florin,

On Wed, 27 Jul 2016, Papa, Florin wrote:

> The table contains run time values, normalized to the CPython Numpy 
> results. This means that a value of 1 is equal to the CPython NumPy 
> result, less than 1 means faster than CPython NumPy and more than 1 is 
> slower than CPython NumPy.

Thank you for the explanation!

I think this supports my assessment though, as I can't see how your 
conclusion can be justified on the basis of this table:

"NumPyPy performance seems to be significantly slower compared to CPython 
NumPy or even PyPy NumPy"

In fact, NumPyPy performance seems to be significantly *faster* compared 
to CPython NumPy and, in any case, PyPy NumPy (with the exception of a few 
benchmarks, such as "cauchy", which should be investigated).

I'd also be very curious as to whether you've tried the vectorizer 
already, or these results were obtained without it.

-- 
Sincerely yours,
Yury V. Zaytsev

From florin.papa at intel.com  Wed Jul 27 04:35:45 2016
From: florin.papa at intel.com (Papa, Florin)
Date: Wed, 27 Jul 2016 08:35:45 +0000
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <alpine.LRH.2.20.1607271010450.13656@vps.zaytsev.net>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C05AC7@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607271010450.13656@vps.zaytsev.net>
Message-ID: <3A375A669FBEFF45B6B60E689636EDCA09C07B24@IRSMSX101.ger.corp.intel.com>

I am sorry, I mistakenly switched the header of the table, the middle column is actually the result for PyPy NumPyPy. The correct table is this:

Benchmark   CPython NumPy   PyPy NumPyPy     PyPy NumPy
cauchy          1           5.838852812     4.866947551
pointbypoint    1           4.922654347     0.981008211
numrand         1           2.478997019     1.082185897
rowmean         1           2.512893263     1.062233015
dsums           1           33.58240465     1.013388981
vectsum         1           1.738446611     0.771660704
cauchy          1           2.168377906     0.887388291
polarcoords     1           1.030962402     0.500905427
vectsort        1           2.214586698     0.973727924
arange          1           2.045342386     0.69941044
vectoradd       1           5.447667037     1.513217941
extractint      1           1.655717606     2.671712185
float2int       1           3.1688          0.905406988
insertzeros     1           2.375043445     1.037504453

The results were gathered without vectorization, I will provide the results with vectorization as soon as I have them. 

Sorry again for the mistake.

Regards,
Florin

From matti.picus at gmail.com  Wed Jul 27 22:05:42 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 27 Jul 2016 21:05:42 -0500
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <3A375A669FBEFF45B6B60E689636EDCA09C07B24@IRSMSX101.ger.corp.intel.com>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C05AC7@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607271010450.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C07B24@IRSMSX101.ger.corp.intel.com>
Message-ID: <7bd58bd6-ec80-1688-b0da-e836cfb7c31c@gmail.com>

On 27/07/2016 3:35 AM, Papa, Florin wrote:

> I am sorry, I mistakenly switched the header of the table, the middle column is actually the result for PyPy NumPyPy. The correct table is this:
>
> Benchmark   CPython NumPy   PyPy NumPyPy     PyPy NumPy
> cauchy          1           5.838852812     4.866947551
> pointbypoint    1           4.922654347     0.981008211
> numrand         1           2.478997019     1.082185897
> rowmean         1           2.512893263     1.062233015
> dsums           1           33.58240465     1.013388981
> vectsum         1           1.738446611     0.771660704
> cauchy          1           2.168377906     0.887388291
> polarcoords     1           1.030962402     0.500905427
> vectsort        1           2.214586698     0.973727924
> arange          1           2.045342386     0.69941044
> vectoradd       1           5.447667037     1.513217941
> extractint      1           1.655717606     2.671712185
> float2int       1           3.1688          0.905406988
> insertzeros     1           2.375043445     1.037504453
>
> The results were gathered without vectorization, I will provide the results with vectorization as soon as I have them.
>
> Sorry again for the mistake.
>
> Regards,
> Florin
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
Thanks for taking the time to test this. You asked in the first message 
"Is there an official benchmark suite for NumPy or a more relevant 
workload to compare against CPython? What is NumPyPy's maturity / 
adoption rate from your knowledge?"

There is no official numpy benchmark, since there is really no "typical" 
numpy workload. Numpy is used as a common container for data processing, 
and each field has its own cases that interest it, for instance a 
workload done by CAFFE for neural network processing is much different 
that one done by OpenCV for image processing, which is different that 
the natural language processing done in NLTK, even though for the most 
part all three of these use numpy. There are a few numpy benchmarks 
available;

https://github.com/serge-sans-paille/numpy-benchmarks (needs to be 
adapted to pypy's slow warmup time)
http://yarikoptic.github.io/numpy-vbench  (also AFAICT never run on PyPy)
https://bitbucket.org/mikefc/numpy-benchmark.git

I would expect numpypy to shine in cases where there is heavy use of 
python together with numpy. Your benchmarks are at the other extreme; 
they demonstrate that our reimplementation of the numpy looping ufuncs 
is slower than C, but do not test the python-numpy interaction nor how 
well the JIT can optimize python code using numpy. For your tests 
Richard's suggestion of turning on vectorization may show a large 
improvement, as it brings numpypy's optimizations closer to the ones 
done by a good C compiler. But even so, it is impressive that without 
vectorization we are only 2-4 times slower than the heavily vectorized c 
implementation, and that the cpyext emulation layer seems not to matter 
that much in your benchmarks.

In general, timeit does a bad job for pypy benchmarks since it does not 
allow for warmup time and is geared to measure a minimum. Your data 
demonstrates some of the pitfalls of benchmarking - note that you show 
two very different results for your "cauchy" benchmark. You may want to 
check out the perf module http://perf.readthedocs.io for a more 
sophisticated way of running benchmarks or read 
https://arxiv.org/abs/1602.00602, which summarizes the problems 
benchmarking.

In order to continue this discussion, could you create a repository with 
these benchmarks and a set of instructions how to reproduce them? You do 
not say what platform you use, what machine you ran the tests on, 
whether you used MKL/BLAS, what versions of pypy and cpython you used, 
... Once we have a conveniently reproducible way to have this 
conversation we may be able to make progress toward reaching some 
operative conclusions, but I'm not sure a mailing list is the best place 
these days.

Matti

From florin.papa at intel.com  Thu Jul 28 09:05:34 2016
From: florin.papa at intel.com (Papa, Florin)
Date: Thu, 28 Jul 2016 13:05:34 +0000
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <7bd58bd6-ec80-1688-b0da-e836cfb7c31c@gmail.com>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C05AC7@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607271010450.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C07B24@IRSMSX101.ger.corp.intel.com>
 <7bd58bd6-ec80-1688-b0da-e836cfb7c31c@gmail.com>
Message-ID: <3A375A669FBEFF45B6B60E689636EDCA09C0A13D@IRSMSX101.ger.corp.intel.com>

Hi Matti,

Thank you for your reply and for indicating additional numpy benchmarks. 

Please see below the results obtained with vectorization turned on. It seems that vectorization will significantly improve run time for some benchmarks (matrixmul, vectoradd, float2int).

Benchmark   CPython NumPy   PyPy NumPyPy    PyPy NumPy      PyPy NumPyPy vectorial
matrixmul           1       5.838852812     4.866947551     3.332052386
pointbypoint        1       4.922654347     0.981008211     4.917323386
numrand             1       2.478997019     1.082185897     2.486596082
rowmean             1       2.512893263     1.062233015     2.531627012
dsums               1       33.58240465     1.013388981     33.73959105
vectsum             1       1.738446611     0.771660704     1.651790546
cauchy              1       2.168377906     0.887388291     1.789566808
polarcoords         1       1.030962402     0.500905427     1.031192576
vectsort            1       2.214586698     0.973727924     2.205043894
arange              1       2.045342386     0.69941044      2.064583705
vectoradd           1       5.447667037     1.513217941     4.838760016
extractint          1       1.655717606     2.671712185     1.633729987
float2int           1       3.1688          0.905406988     2.764488512
insertzeros         1       2.375043445     1.037504453     2.145735211

>I would expect numpypy to shine in cases where there is heavy use of python together
>with numpy. Your benchmarks are at the other extreme; they demonstrate that our
>reimplementation of the numpy looping ufuncs is slower than C, but do not test the
>python-numpy interaction nor how well the JIT can optimize python code using numpy.
>For your tests Richard's suggestion of turning on vectorization may show a large
>improvement, as it brings numpypy's optimizations closer to the ones done by a good
>C compiler. But even so, it is impressive that without vectorization we are only 2-4
>times slower than the heavily vectorized c implementation, and that the cpyext emulation
>layer seems not to matter that much in your benchmarks.
>
>In general, timeit does a bad job for pypy benchmarks since it does not allow for warmup
>time and is geared to measure a minimum. Your data demonstrates some of the pitfalls of
>benchmarking - note that you show two very different results for your "cauchy" benchmark.
>You may want to check out the perf module http://perf.readthedocs.io for a more
>sophisticated way of running benchmarks or read https://arxiv.org/abs/1602.00602, which
>summarizes the problems benchmarking.

The benchmarks I wrote do seem to stress numpy, leaving out python or the cpyext emulation layer. I realize that this is not a realistic scenario for real life workloads, which is why I am interested in more representative workloads that have a high visibility and can emphasize the advantages of PyPy. I will look at the benchmarking links indicated to find more suitable workloads and benchmarking methodology.

Also, I corrected the "cauchy" issue, the first row was actually matrix multiplication.

>In order to continue this discussion, could you create a repository with these benchmarks
>and a set of instructions how to reproduce them? You do not say what platform you use,
>what machine you ran the tests on, whether you used MKL/BLAS, what versions of pypy and
>cpython you used, ... Once we have a conveniently reproducible way to have this conversation
>we may be able to make progress toward reaching some operative conclusions, but I'm not sure
>a mailing list is the best place these days.

Creating a public repository with the benchmarks can be a time consuming procedure due
to internal methodologies, but please find attached the benchmarks and the Python script
used to run them (num_perf.py, similar to perf.py in CPython's benchmark suite). In order to
obtain a csv file with the benchmark results, please follow these steps:

unzip numpy_benchmark.zip
cd numpy_benchmark
python num_perf.py -b all /path/to/python1 /path/to/python2

I do not seem to have MKL on my system, but I do have lapack/blas runtimes installed, as recommended
here [1]. I am running Ubuntu 16.04 LTS on an 18-core Intel(R) Xeon(R) (Haswell) CPU E5-2699 v3 @ 2.30GHz.

BIOS settings:      Intel Turbo Boost Technology: false
                    Hyper-Threading: false

OS configuration:   CPU freq set at fixed: 2.6GHz by
                        echo 2300000 > /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
                        echo 2300000 > /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
                    Address Space Layout Randomization (ASLR) disabled (to reduce run to run variation) by
                        echo 0 > /proc/sys/kernel/randomize_va_space

CPython version is 2.7.11+, the default that comes with Ubuntu 16.04 LTS. PyPy version is 5.3.1, I downloaded
an already compiled binary (7e8df3df9641, Jun 14 2016, 13:58:02).

We can continue this discussion any place you consider suitable (if the mailing list is not the place for this).

[1] https://bitbucket.org/pypy/numpy/overview


Regards,
Florin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: numpy_benchmark.zip
Type: application/x-zip-compressed
Size: 8132 bytes
Desc: numpy_benchmark.zip
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160728/04339650/attachment-0001.bin>

From matti.picus at gmail.com  Thu Jul 28 23:48:42 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 28 Jul 2016 22:48:42 -0500
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <3A375A669FBEFF45B6B60E689636EDCA09C0A13D@IRSMSX101.ger.corp.intel.com>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C05AC7@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607271010450.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C07B24@IRSMSX101.ger.corp.intel.com>
 <7bd58bd6-ec80-1688-b0da-e836cfb7c31c@gmail.com>
 <3A375A669FBEFF45B6B60E689636EDCA09C0A13D@IRSMSX101.ger.corp.intel.com>
Message-ID: <16a190ba-a51d-7726-bf25-865b555de74a@gmail.com>

On 28/07/2016 8:05 AM, Papa, Florin wrote:
> Hi Matti,
>
> Thank you for your reply and for indicating additional numpy benchmarks.
...
> We can continue this discussion any place you consider suitable (if the mailing list is not the place for this).
>
>
>
> Regards,
> Florin
We usually hang out on IRC, you can find me there most evenings European 
time.

The zip file is not a very iterative-freindly format for improving the 
benchmarks, how can I contribute to your work?
- There should be some kind of shell script that downloads and installs 
the packages from a known source so anyone else can reproduce
- You should add np.__config__.show() to the scripts so the output 
reflects the external libraries used
- Examine other suites to find how they display the basic 
python/computer/environment variables in use when you run the bencmarks

- how many cores are in use? How much memory?

- You should check the result, for instance AFAICT the dsums benchmark 
does not run to completion on numpypy, bincount is not implemented
Matti


From peter.xihong.wang at intel.com  Thu Jul 28 19:24:38 2016
From: peter.xihong.wang at intel.com (Wang, Peter Xihong)
Date: Thu, 28 Jul 2016 23:24:38 +0000
Subject: [pypy-dev] how to extend VTUNE support from pypy for application
 hot spot analysis
Message-ID: <371EBC7881C7844EAAF5556BFF21BCCC42DFEC39@ORSMSX105.amr.corp.intel.com>

Hi,

Armin created a minimum working version incorporating VTUNE to a branch at: https://bitbucket.org/pypy/pypy/branch/vtune
This works in a sense such that statistically meaningful micro-architecture data could be collected.  However, for hot spot analysis, we'd like to trace back to the Python application source code.
We know the VTUNE hook point, but don't know how or where to start from the rpython source code.  I ran gdb against pypy binary and found it really hard to get to the entry point/or knowledge we wanted.

Could anyone give us some hints?

Thanks,

Peter





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20160728/dfc578da/attachment.html>

From arigo at tunes.org  Fri Jul 29 05:12:48 2016
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 29 Jul 2016 11:12:48 +0200
Subject: [pypy-dev] NumPyPy vs NumPy
In-Reply-To: <3A375A669FBEFF45B6B60E689636EDCA09C07B24@IRSMSX101.ger.corp.intel.com>
References: <3A375A669FBEFF45B6B60E689636EDCA09C04622@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607270927400.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C05AC7@IRSMSX101.ger.corp.intel.com>
 <alpine.LRH.2.20.1607271010450.13656@vps.zaytsev.net>
 <3A375A669FBEFF45B6B60E689636EDCA09C07B24@IRSMSX101.ger.corp.intel.com>
Message-ID: <CAMSv6X1oZzFCqS_C0ja3OBT3=wbjX=250HT-g-DjyZUGg_Y73Q@mail.gmail.com>

Hi,

On 27 July 2016 at 10:35, Papa, Florin <florin.papa at intel.com> wrote:
> I am sorry, I mistakenly switched the header of the table, the middle column is actually the result for PyPy NumPyPy.

The resulting table makes sense to me: it shows that PyPy NumPy (with
cpyext) is, in most case, running at the same speed as CPython NumPy;
and the rare exceptions can be guessed to be because these benchmarks
happen to invoke a much larger number of CPython C API calls than all
the others.

The table also shows that PyPy NumPyPy is really slower, even with
vectorization enabled.  It seems that the current focus of our work,
on continuing to improve cpyext instead of numpypy, is a good idea.


A bient?t,

Armin.