From fijall at gmail.com  Mon Jan  1 05:27:42 2018
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 1 Jan 2018 10:27:42 +0000
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CADB0KUJDRc5Pgth0pEXM1b+bjuRTUS-joxMpt3GNiOdUjC+89A@mail.gmail.com>
References: <CADB0KUKHf=oRXUPsh7_T=Y9Sp6ZETajdNVkEX_4L1AVRJmz4qQ@mail.gmail.com>
 <CAK5idxSwdNepiqyFrhZTy08CxU_jd9iDS9bq6ueJ4X+ShYm6vQ@mail.gmail.com>
 <CADB0KUL7G2+JjQZM1D0Ai11WONx=MpVYXFHJpW2AeFVSpG4YcA@mail.gmail.com>
 <CAK5idxTJBE2ETWmef20aHY7u0ZqNXmkLW=Xyho0p+vd4KDLPXQ@mail.gmail.com>
 <CADB0KUJDRc5Pgth0pEXM1b+bjuRTUS-joxMpt3GNiOdUjC+89A@mail.gmail.com>
Message-ID: <CAK5idxR=J8CrdQwMO1jVPmPPfNTRDadZWfkNbpjVXd_L2CbDbQ@mail.gmail.com>

Hey Daniel

Great, I'm usually at least twice a year in Prague :-)

Let's catch up in February.

Cheers,
fijal

On Sat, Dec 30, 2017 at 2:42 PM, Kotrfa <kotrfa at gmail.com> wrote:
> Hi,
>
> I am based in Prague (UTC+2), but during January, I am travelling and hence
> I am not very available. It should get better during February, so could we
> schedule a Hangout/Skype call or something then?
>
> Also, depending on the volume of the work on the site, I may not be able to
> work on it before July. But have no problem if someone else want start on it
> themselves and I can join :-) ... But I believe I could find some time
> before then for simple redesign.
>
> Best,
> Daniel
>
> so 30. 12. 2017 v 11:49 odes?latel Maciej Fijalkowski <fijall at gmail.com>
> napsal:
>>
>> Hi Kotrfa.
>>
>> Great, we would definitely appreciate some help. I have very sketchy
>> internet till next week, maybe we can coordinate something next week
>> on IRC? What timezone are you in?
>>
>> Best regards,
>> Maciej Fijalkowski
>>
>> On Sat, Dec 30, 2017 at 9:41 AM, Kotrfa <kotrfa at gmail.com> wrote:
>> > Hey,
>> >
>> > as I said, I am not a professional, but I would be willing to help with
>> > the
>> > site. I have built couple of websites in Django/Wagtail (or plain HTML),
>> > have some basic knowledge about design (+ frontend CSS/JS/HTML). The
>> > site
>> > seems quite simple, so if we just did a revamp, it shouldn't take a
>> > long.
>> > Also, if we went with Wagtail, adding blog capabilities would be also
>> > trivial.
>> >
>> > Best,
>> > Daniel
>> >
>> > so 30. 12. 2017 v 10:02 odes?latel Maciej Fijalkowski <fijall at gmail.com>
>> > napsal:
>> >>
>> >> Hi Kortrfa
>> >>
>> >> I've even paid for a design of new pypy logo that we can use on the
>> >> new website :-) Maybe it's a good time I spend some effort upgrading
>> >> it. Thanks for the reminder, it's something that's on our heads, but
>> >> it's not like we can just hire someone to do it for us.
>> >>
>> >> Cheers,
>> >> fijal
>> >>
>> >> On Wed, Dec 27, 2017 at 9:35 PM, Kotrfa <kotrfa at gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > thank you for all your hard work, I really appreciate it.
>> >> >
>> >> > Firstly, it seems that the website SSL certificate expired or
>> >> > something.
>> >> > At
>> >> > my latest Chromium reports "Not secure".
>> >> >
>> >> > Secondly, I totally understand that there are more pressing issues
>> >> > than
>> >> > cool
>> >> > website, but the current design and functionality is IMO really
>> >> > putting
>> >> > people off. So I am going to through away some things which seems
>> >> > weird
>> >> > to
>> >> > ME (and I understand it's subjective - please take it as a friendly
>> >> > feedback).
>> >> >
>> >> > The jQuery versions suggests November 2012, the design idea seems
>> >> > even
>> >> > older... There are old information such as "We are soon releasing a
>> >> > beta
>> >> > supporting Python 3.3." or the fact that the donations bars are still
>> >> > present even though they are closed (and hence could be moved into
>> >> > some
>> >> > blog
>> >> > post). Blog is also terrible, this is how it renders on my laptop
>> >> > (the
>> >> > same
>> >> > with the external display): https://imgur.com/a/qkjEa . The logo
>> >> > could
>> >> > be
>> >> > surely polished as well (I like the idea, but it seems a bit
>> >> > childish).
>> >> > Even
>> >> > though I am not professional front end developer, I am quite
>> >> > confident
>> >> > this
>> >> > feedback is not far from what a professional would say. It's also not
>> >> > about
>> >> > having "cool" website for itself, but that a nice projects attracts
>> >> > good
>> >> > people. I myself am questioning how good, from technical point of
>> >> > view,
>> >> > Pypy's project can be if the site looks like like it does and the
>> >> > most
>> >> > convenient way how to reach the devs is through mailing list...
>> >> >
>> >> > Best,
>> >> > Daniel
>> >> >
>> >> > _______________________________________________
>> >> > pypy-dev mailing list
>> >> > pypy-dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/pypy-dev
>> >> >

From matti.picus at gmail.com  Tue Jan  2 01:56:08 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 2 Jan 2018 08:56:08 +0200
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <AB2CE940-58A7-4F9C-B882-73B4AAB4FE38@imag.fr>
References: <AB2CE940-58A7-4F9C-B882-73B4AAB4FE38@imag.fr>
Message-ID: <991763ff-c8c0-a00c-d223-219de9de87e0@gmail.com>

On 31/12/2017 10:59 AM, blanch wrote:

> hello, thanks for the 5.10 release!
>
> just a problem on the mac: it seems that the pypy3 binaries are linked against a libffi provided by a package manager (homebrew?), since dyld is looking for /usr/local/opt/libffi/lib/libffi.6.dylib (see below the error reported and otool output).
> since i do not use homebrew, it fails to execute on my mac.
> i have the problem with the pypy3 pre-sierra version, but according to a comment on the morepypy blog [0], the problem exist for high sierra version of pypy3 and for pypy2 (which looks for /usr/local/opt/openssl/lib/libssl.1.0.0.dylib)
>
> thanks again for your work, and best wishes for 2018!
>
> renaud
>
> 0. https://www.blogger.com/comment.g?blogID=3971202189709462152&postID=3223396318213306071&bpli=1
We need someone who uses that OS and knows how to fix it to help us out. 
Any ideas? SInce the same buildslave creates our pypy2 binaries as well, 
I guess the problem occurs there too.
If anyone can help out, please issue a pull request or at least file an 
issue on  https://bitbucket.org/pypy/pypy/issues (login required) with a 
proposal for a fix.

From matt at vazor.com  Tue Jan  2 17:21:27 2018
From: matt at vazor.com (Matt Billenstein)
Date: Tue, 2 Jan 2018 22:21:27 +0000
Subject: [pypy-dev] pypy 5.10 release
Message-ID: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>

Hi all,

So the general issue seems to be Apple not shipping the header files for these
shared libs-- not linking to the brew libffi works (I'm not sure it's linking
to the system libffi at all now however -- or if it needs to), but then the
problem became openssl headers.  I've hacked around this by manually placing
the LibreSSL 2.2.7 headers in /usr/include -- this is what matches the
'openssl' binary on my High Sierra system.

mattb at mattb-mbp2:/Users/pypy/buildarea $ openssl version
LibreSSL 2.2.7

So now I have a build with no brew linkage dependencies:

mattb at mattb-mbp2:/Users/pypy/buildarea $ find . -name 'libpypy*' -type f | xargs otool -L
./pypy-c-jit-macosx-x86-64/build/pypy/goal/libpypy-c.dylib:
        @rpath/libpypy-c.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
        /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
        /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
        /usr/lib/libssl.35.dylib (compatibility version 36.0.0, current version 36.0.0)
        /usr/lib/libcrypto.35.dylib (compatibility version 36.0.0, current version 36.0.0)
        /usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 8.0.0)
        /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)

Which is closer to an older release:

mattb at mattb-mbp2:/Users/pypy/buildarea $ otool -L ~pypy/pypy2-v5.9.0-osx64/bin/libpypy-c.dylib
/Users/pypy/pypy2-v5.9.0-osx64/bin/libpypy-c.dylib:
        @rpath/libpypy-c.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
        /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
        /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
        /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 7.2.0)
        /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)

The problem now is that probably this build won't work on older OSX releases
with only older openssl versions...  Apple does ship the older dylib:

mattb at mattb-mbp2:/Users/pypy/buildarea $ ls -l /usr/lib/*ssl*
-rwxr-xr-x 1 root wheel 1217072 Oct 25 09:37 /usr/lib/libboringssl.dylib
-rwxr-xr-x 1 root wheel  392912 Oct 25 09:37 /usr/lib/libssl.0.9.7.dylib
-rwxr-xr-x 1 root wheel  630144 Oct 25 09:37 /usr/lib/libssl.0.9.8.dylib
-rw-r--r-- 1 root wheel  947104 Oct 25 09:37 /usr/lib/libssl.35.dylib
-rw-r--r-- 1 root wheel  890800 Oct 25 09:37 /usr/lib/libssl.43.dylib
lrwxr-xr-x 1 root wheel      15 Nov  3 01:59 /usr/lib/libssl.dylib -> libssl.35.dylib

So maybe linking to that and manually supplying the openssl 0.9.8 headers is
desirable?

Let me know if anyone has a better idea of how to go about this.

thx

m

> On 31/12/2017 10:59 AM, blanch wrote:
> 
> > hello, thanks for the 5.10 release!
> >
> > just a problem on the mac: it seems that the pypy3 binaries are linked against a libffi provided by a package manager (homebrew?), since dyld is looking for /usr/local/opt/libffi/lib/libffi.6.dylib (see below the error reported and otool output).
> > since i do not use homebrew, it fails to execute on my mac.
> > i have the problem with the pypy3 pre-sierra version, but according to a comment on the morepypy blog [0], the problem exist for high sierra version of pypy3 and for pypy2 (which looks for /usr/local/opt/openssl/lib/libssl.1.0.0.dylib)
> >
> > thanks again for your work, and best wishes for 2018!
> >
> > renaud
> >
> > 0. https://www.blogger.com/comment.g?blogID=3971202189709462152&postID=3223396318213306071&bpli=1
> We need someone who uses that OS and knows how to fix it to help us out. 
> Any ideas? SInce the same buildslave creates our pypy2 binaries as well, 
> I guess the problem occurs there too.
> If anyone can help out, please issue a pull request or at least file an 
> issue on  https://bitbucket.org/pypy/pypy/issues (login required) with a 
> proposal for a fix.

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From kotrfa at gmail.com  Tue Jan  2 17:30:55 2018
From: kotrfa at gmail.com (Kotrfa)
Date: Tue, 02 Jan 2018 22:30:55 +0000
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CAPkN8xJ2BFQt6AU-1nYXAyjFF6dsVJpBE-yRKkzv+cg+4CuOwQ@mail.gmail.com>
References: <CADB0KUKHf=oRXUPsh7_T=Y9Sp6ZETajdNVkEX_4L1AVRJmz4qQ@mail.gmail.com>
 <CAK5idxSwdNepiqyFrhZTy08CxU_jd9iDS9bq6ueJ4X+ShYm6vQ@mail.gmail.com>
 <CADB0KUL7G2+JjQZM1D0Ai11WONx=MpVYXFHJpW2AeFVSpG4YcA@mail.gmail.com>
 <CAPkN8xJ2BFQt6AU-1nYXAyjFF6dsVJpBE-yRKkzv+cg+4CuOwQ@mail.gmail.com>
Message-ID: <CADB0KUKqD9LNkZqmW43XHK=HXkDTGZLZHiA=kkSeGrMMyuZztA@mail.gmail.com>

Yeah, that's a very good point worth mentioning. I am not against an idea
going just with a static version.

po 1. 1. 2018 v 18:05 odes?latel anatoly techtonik <techtonik at gmail.com>
napsal:

> On Sat, Dec 30, 2017 at 12:41 PM, Kotrfa <kotrfa at gmail.com> wrote:
> > Hey,
> >
> > as I said, I am not a professional, but I would be willing to help with
> the
> > site. I have built couple of websites in Django/Wagtail (or plain HTML),
> > have some basic knowledge about design (+ frontend CSS/JS/HTML). The site
> > seems quite simple, so if we just did a revamp, it shouldn't take a long.
> > Also, if we went with Wagtail, adding blog capabilities would be also
> > trivial.
>
> Just want to warn that Wagtail and Django means database, which adds
> maintenance overhead compared to current static version. Logins, spam
> and all that stuff.. I don't mind against custom blog engine, but
> putting all static content from version control into Wagtail/Django
> database doesn't seem a good solution to me. When site gets hacked,
> for example to change donation credentials, there is no trace who did
> what. So going with simple static site solution is probably a better
> way. It also doesn't require permission or privilege to send a pull
> request, and content previews can be made with Netlify.
>
> Happy New Year. =)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180102/d9a7b3a5/attachment.html>

From barrywhart at yahoo.com  Tue Jan  2 18:21:58 2018
From: barrywhart at yahoo.com (Barry Hart)
Date: Tue, 2 Jan 2018 23:21:58 +0000 (UTC)
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CADB0KUKqD9LNkZqmW43XHK=HXkDTGZLZHiA=kkSeGrMMyuZztA@mail.gmail.com>
References: <CADB0KUKHf=oRXUPsh7_T=Y9Sp6ZETajdNVkEX_4L1AVRJmz4qQ@mail.gmail.com>
 <CAK5idxSwdNepiqyFrhZTy08CxU_jd9iDS9bq6ueJ4X+ShYm6vQ@mail.gmail.com>
 <CADB0KUL7G2+JjQZM1D0Ai11WONx=MpVYXFHJpW2AeFVSpG4YcA@mail.gmail.com>
 <CAPkN8xJ2BFQt6AU-1nYXAyjFF6dsVJpBE-yRKkzv+cg+4CuOwQ@mail.gmail.com>
 <CADB0KUKqD9LNkZqmW43XHK=HXkDTGZLZHiA=kkSeGrMMyuZztA@mail.gmail.com>
Message-ID: <544121945.7050109.1514935318318@mail.yahoo.com>

Question about the web site -- does PyPy currently have anything similar to this page for Python 3?
http://py3readiness.org/

I think a page like this, showing which major libraries are compatible with PyPy, could really help drive adoption of PyPy. I know for our team, the Python 3 page was a strong reason we felt "safe" starting to make the switch to Python 3.
I'm not sure how we'd get this information about PyPy library compatibility. One idea would be to install each library on PyPy, run the automated tests, and compare the results against those for CPython.
Barry 

    On Tuesday, January 2, 2018 5:31 PM, Kotrfa <kotrfa at gmail.com> wrote:
 

 Yeah, that's a very good point worth mentioning. I am not against an idea going just with a static version.
po 1. 1. 2018 v 18:05 odes?latel anatoly techtonik <techtonik at gmail.com> napsal:

On Sat, Dec 30, 2017 at 12:41 PM, Kotrfa <kotrfa at gmail.com> wrote:
> Hey,
>
> as I said, I am not a professional, but I would be willing to help with the
> site. I have built couple of websites in Django/Wagtail (or plain HTML),
> have some basic knowledge about design (+ frontend CSS/JS/HTML). The site
> seems quite simple, so if we just did a revamp, it shouldn't take a long.
> Also, if we went with Wagtail, adding blog capabilities would be also
> trivial.

Just want to warn that Wagtail and Django means database, which adds
maintenance overhead compared to current static version. Logins, spam
and all that stuff.. I don't mind against custom blog engine, but
putting all static content from version control into Wagtail/Django
database doesn't seem a good solution to me. When site gets hacked,
for example to change donation credentials, there is no trace who did
what. So going with simple static site solution is probably a better
way. It also doesn't require permission or privilege to send a pull
request, and content previews can be made with Netlify.

Happy New Year. =)

_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180102/a82f484c/attachment-0001.html>

From njs at pobox.com  Tue Jan  2 18:25:17 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 2 Jan 2018 15:25:17 -0800
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
Message-ID: <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>

I'm pretty sure that the right solution is to ship your own copy of openssl
with the build, so that you're totally independent from Apple's ssl
shenanigans. Maybe look at how CPython handles this.

On Jan 2, 2018 2:28 PM, "Matt Billenstein" <matt at vazor.com> wrote:

> Hi all,
>
> So the general issue seems to be Apple not shipping the header files for
> these
> shared libs-- not linking to the brew libffi works (I'm not sure it's
> linking
> to the system libffi at all now however -- or if it needs to), but then the
> problem became openssl headers.  I've hacked around this by manually
> placing
> the LibreSSL 2.2.7 headers in /usr/include -- this is what matches the
> 'openssl' binary on my High Sierra system.
>
> mattb at mattb-mbp2:/Users/pypy/buildarea $ openssl version
> LibreSSL 2.2.7
>
> So now I have a build with no brew linkage dependencies:
>
> mattb at mattb-mbp2:/Users/pypy/buildarea $ find . -name 'libpypy*' -type f
> | xargs otool -L
> ./pypy-c-jit-macosx-x86-64/build/pypy/goal/libpypy-c.dylib:
>         @rpath/libpypy-c.dylib (compatibility version 0.0.0, current
> version 0.0.0)
>         /usr/lib/libutil.dylib (compatibility version 1.0.0, current
> version 1.0.0)
>         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1252.0.0)
>         /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current
> version 1.0.5)
>         /usr/lib/libz.1.dylib (compatibility version 1.0.0, current
> version 1.2.11)
>         /usr/lib/libssl.35.dylib (compatibility version 36.0.0, current
> version 36.0.0)
>         /usr/lib/libcrypto.35.dylib (compatibility version 36.0.0, current
> version 36.0.0)
>         /usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current
> version 8.0.0)
>         /usr/lib/libffi.dylib (compatibility version 1.0.0, current
> version 1.0.0)
>         /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0,
> current version 5.4.0)
>
> Which is closer to an older release:
>
> mattb at mattb-mbp2:/Users/pypy/buildarea $ otool -L
> ~pypy/pypy2-v5.9.0-osx64/bin/libpypy-c.dylib
> /Users/pypy/pypy2-v5.9.0-osx64/bin/libpypy-c.dylib:
>         @rpath/libpypy-c.dylib (compatibility version 0.0.0, current
> version 0.0.0)
>         /usr/lib/libutil.dylib (compatibility version 1.0.0, current
> version 1.0.0)
>         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1197.1.1)
>         /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current
> version 1.0.5)
>         /usr/lib/libz.1.dylib (compatibility version 1.0.0, current
> version 1.2.5)
>         /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current
> version 0.9.8)
>         /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8,
> current version 0.9.8)
>         /usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current
> version 7.2.0)
>         /usr/lib/libffi.dylib (compatibility version 1.0.0, current
> version 1.0.0)
>         /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0,
> current version 5.4.0)
>
> The problem now is that probably this build won't work on older OSX
> releases
> with only older openssl versions...  Apple does ship the older dylib:
>
> mattb at mattb-mbp2:/Users/pypy/buildarea $ ls -l /usr/lib/*ssl*
> -rwxr-xr-x 1 root wheel 1217072 Oct 25 09:37 /usr/lib/libboringssl.dylib
> -rwxr-xr-x 1 root wheel  392912 Oct 25 09:37 /usr/lib/libssl.0.9.7.dylib
> -rwxr-xr-x 1 root wheel  630144 Oct 25 09:37 /usr/lib/libssl.0.9.8.dylib
> -rw-r--r-- 1 root wheel  947104 Oct 25 09:37 /usr/lib/libssl.35.dylib
> -rw-r--r-- 1 root wheel  890800 Oct 25 09:37 /usr/lib/libssl.43.dylib
> lrwxr-xr-x 1 root wheel      15 Nov  3 01:59 /usr/lib/libssl.dylib ->
> libssl.35.dylib
>
> So maybe linking to that and manually supplying the openssl 0.9.8 headers
> is
> desirable?
>
> Let me know if anyone has a better idea of how to go about this.
>
> thx
>
> m
>
> > On 31/12/2017 10:59 AM, blanch wrote:
> >
> > > hello, thanks for the 5.10 release!
> > >
> > > just a problem on the mac: it seems that the pypy3 binaries are linked
> against a libffi provided by a package manager (homebrew?), since dyld is
> looking for /usr/local/opt/libffi/lib/libffi.6.dylib (see below the error
> reported and otool output).
> > > since i do not use homebrew, it fails to execute on my mac.
> > > i have the problem with the pypy3 pre-sierra version, but according to
> a comment on the morepypy blog [0], the problem exist for high sierra
> version of pypy3 and for pypy2 (which looks for /usr/local/opt/openssl/lib/
> libssl.1.0.0.dylib)
> > >
> > > thanks again for your work, and best wishes for 2018!
> > >
> > > renaud
> > >
> > > 0. https://www.blogger.com/comment.g?blogID=
> 3971202189709462152&postID=3223396318213306071&bpli=1
> > We need someone who uses that OS and knows how to fix it to help us out.
> > Any ideas? SInce the same buildslave creates our pypy2 binaries as well,
> > I guess the problem occurs there too.
> > If anyone can help out, please issue a pull request or at least file an
> > issue on  https://bitbucket.org/pypy/pypy/issues (login required) with a
> > proposal for a fix.
>
> --
> Matt Billenstein
> matt at vazor.com
> http://www.vazor.com/
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180102/a54f90f6/attachment.html>

From barrywhart at yahoo.com  Tue Jan  2 18:25:49 2018
From: barrywhart at yahoo.com (Barry Hart)
Date: Tue, 2 Jan 2018 23:25:49 +0000 (UTC)
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
Message-ID: <1108399520.7050235.1514935549351@mail.yahoo.com>

I think supporting just the more recent OS X releases (Sierra, High Sierra) is fine. I recently had to upgrade my Mac to Sierra because I was running into packages that wouldn't work on the older version. (There seems to be a "get_entropy" function in Sierra, and things were failing because it wasn't present in the older OS.)
If my experience is any guide, OS X is not a production environment, it's just a dev environment for things that'll ultimately be deployed on Linux.?
Barry 

    On Tuesday, January 2, 2018 5:28 PM, Matt Billenstein <matt at vazor.com> wrote:
 

 Hi all,

So the general issue seems to be Apple not shipping the header files for these
shared libs-- not linking to the brew libffi works (I'm not sure it's linking
to the system libffi at all now however -- or if it needs to), but then the
problem became openssl headers.? I've hacked around this by manually placing
the LibreSSL 2.2.7 headers in /usr/include -- this is what matches the
'openssl' binary on my High Sierra system.

mattb at mattb-mbp2:/Users/pypy/buildarea $ openssl version
LibreSSL 2.2.7

So now I have a build with no brew linkage dependencies:

mattb at mattb-mbp2:/Users/pypy/buildarea $ find . -name 'libpypy*' -type f | xargs otool -L
./pypy-c-jit-macosx-x86-64/build/pypy/goal/libpypy-c.dylib:
? ? ? ? @rpath/libpypy-c.dylib (compatibility version 0.0.0, current version 0.0.0)
? ? ? ? /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
? ? ? ? /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
? ? ? ? /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
? ? ? ? /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
? ? ? ? /usr/lib/libssl.35.dylib (compatibility version 36.0.0, current version 36.0.0)
? ? ? ? /usr/lib/libcrypto.35.dylib (compatibility version 36.0.0, current version 36.0.0)
? ? ? ? /usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 8.0.0)
? ? ? ? /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 1.0.0)
? ? ? ? /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)

Which is closer to an older release:

mattb at mattb-mbp2:/Users/pypy/buildarea $ otool -L ~pypy/pypy2-v5.9.0-osx64/bin/libpypy-c.dylib
/Users/pypy/pypy2-v5.9.0-osx64/bin/libpypy-c.dylib:
? ? ? ? @rpath/libpypy-c.dylib (compatibility version 0.0.0, current version 0.0.0)
? ? ? ? /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
? ? ? ? /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
? ? ? ? /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
? ? ? ? /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
? ? ? ? /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
? ? ? ? /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
? ? ? ? /usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 7.2.0)
? ? ? ? /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 1.0.0)
? ? ? ? /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)

The problem now is that probably this build won't work on older OSX releases
with only older openssl versions...? Apple does ship the older dylib:

mattb at mattb-mbp2:/Users/pypy/buildarea $ ls -l /usr/lib/*ssl*
-rwxr-xr-x 1 root wheel 1217072 Oct 25 09:37 /usr/lib/libboringssl.dylib
-rwxr-xr-x 1 root wheel? 392912 Oct 25 09:37 /usr/lib/libssl.0.9.7.dylib
-rwxr-xr-x 1 root wheel? 630144 Oct 25 09:37 /usr/lib/libssl.0.9.8.dylib
-rw-r--r-- 1 root wheel? 947104 Oct 25 09:37 /usr/lib/libssl.35.dylib
-rw-r--r-- 1 root wheel? 890800 Oct 25 09:37 /usr/lib/libssl.43.dylib
lrwxr-xr-x 1 root wheel? ? ? 15 Nov? 3 01:59 /usr/lib/libssl.dylib -> libssl.35.dylib

So maybe linking to that and manually supplying the openssl 0.9.8 headers is
desirable?

Let me know if anyone has a better idea of how to go about this.

thx

m

> On 31/12/2017 10:59 AM, blanch wrote:
> 
> > hello, thanks for the 5.10 release!
> >
> > just a problem on the mac: it seems that the pypy3 binaries are linked against a libffi provided by a package manager (homebrew?), since dyld is looking for /usr/local/opt/libffi/lib/libffi.6.dylib (see below the error reported and otool output).
> > since i do not use homebrew, it fails to execute on my mac.
> > i have the problem with the pypy3 pre-sierra version, but according to a comment on the morepypy blog [0], the problem exist for high sierra version of pypy3 and for pypy2 (which looks for /usr/local/opt/openssl/lib/libssl.1.0.0.dylib)
> >
> > thanks again for your work, and best wishes for 2018!
> >
> > renaud
> >
> > 0. https://www.blogger.com/comment.g?blogID=3971202189709462152&postID=3223396318213306071&bpli=1
> We need someone who uses that OS and knows how to fix it to help us out. 
> Any ideas? SInce the same buildslave creates our pypy2 binaries as well, 
> I guess the problem occurs there too.
> If anyone can help out, please issue a pull request or at least file an 
> issue on? https://bitbucket.org/pypy/pypy/issues (login required) with a 
> proposal for a fix.

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/
_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180102/dd6a83ca/attachment-0001.html>

From matt at vazor.com  Tue Jan  2 19:04:15 2018
From: matt at vazor.com (Matt Billenstein)
Date: Wed, 3 Jan 2018 00:04:15 +0000
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
Message-ID: <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>

On Tue, Jan 02, 2018 at 03:25:17PM -0800, Nathaniel Smith wrote:
>    I'm pretty sure that the right solution is to ship your own copy of
>    openssl with the build, so that you're totally independent from Apple's
>    ssl shenanigans. Maybe look at how CPython handles this.

That does seem more common practice -- CPython doesn't do that however, it
links to the old 0.9.8 openssl shipped with OSX since ~10.6.

So it seems we could try to do the same and say the binary pypy releases are
compatible that far back, or link to the newer LibreSSL which would support
perhaps Sierra (10.12) and newer.

m

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From alex.gaynor at gmail.com  Tue Jan  2 19:15:42 2018
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Tue, 2 Jan 2018 19:15:42 -0500
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
Message-ID: <CAFRnB2X198VmOmw5hyYSvvx61JmgNQcOtffPNNWnaTP9Y67vJw@mail.gmail.com>

I'm pretty sure the LibreSSL that macOS includes is not intended for public
linkage.

If you want to ship binaries on macOS that use OpenSSL, the thing to do is
to ship your own OpenSSL and update whenever OpenSSL performs a security
release.

Alex

On Tue, Jan 2, 2018 at 7:04 PM, Matt Billenstein <matt at vazor.com> wrote:

> On Tue, Jan 02, 2018 at 03:25:17PM -0800, Nathaniel Smith wrote:
> >    I'm pretty sure that the right solution is to ship your own copy of
> >    openssl with the build, so that you're totally independent from
> Apple's
> >    ssl shenanigans. Maybe look at how CPython handles this.
>
> That does seem more common practice -- CPython doesn't do that however, it
> links to the old 0.9.8 openssl shipped with OSX since ~10.6.
>
> So it seems we could try to do the same and say the binary pypy releases
> are
> compatible that far back, or link to the newer LibreSSL which would support
> perhaps Sierra (10.12) and newer.
>
> m
>
> --
> Matt Billenstein
> matt at vazor.com
> http://www.vazor.com/
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180102/9f2cf9a2/attachment.html>

From njs at pobox.com  Tue Jan  2 19:21:55 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 2 Jan 2018 16:21:55 -0800
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
Message-ID: <CAPJVwBkTXp0bYqxy6t0FzVFenwkUzycZD5=jJN4hG2KGK02Lqw@mail.gmail.com>

Which version of CPython are you looking at? Here's the patch that switched
the official MacOS builds to always using a private copy of openssl:
https://hg.python.org/cpython/rev/bfd0a73cf907

(Found here: https://bugs.python.org/issue17128)

On Jan 2, 2018 16:04, "Matt Billenstein" <matt at vazor.com> wrote:

> On Tue, Jan 02, 2018 at 03:25:17PM -0800, Nathaniel Smith wrote:
> >    I'm pretty sure that the right solution is to ship your own copy of
> >    openssl with the build, so that you're totally independent from
> Apple's
> >    ssl shenanigans. Maybe look at how CPython handles this.
>
> That does seem more common practice -- CPython doesn't do that however, it
> links to the old 0.9.8 openssl shipped with OSX since ~10.6.
>
> So it seems we could try to do the same and say the binary pypy releases
> are
> compatible that far back, or link to the newer LibreSSL which would support
> perhaps Sierra (10.12) and newer.
>
> m
>
> --
> Matt Billenstein
> matt at vazor.com
> http://www.vazor.com/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180102/111229a5/attachment.html>

From john.m.camara at gmail.com  Tue Jan  2 21:00:26 2018
From: john.m.camara at gmail.com (John Camara)
Date: Tue, 2 Jan 2018 21:00:26 -0500
Subject: [pypy-dev] Pypy's facelift and certificate issues
Message-ID: <CAL=Xk30RJuvOvzXNMLjWc5_mY1UwcOY+nwHKmscDPhgBmZ4tHQ@mail.gmail.com>

http://packages.pypy.org/

Question about the web site -- does PyPy currently have anything similar to
> this page for Python 3?
> http://py3readiness.org/
>
> I think a page like this, showing which major libraries are compatible
> with PyPy, could really help drive adoption of PyPy. I know for our team,
> the Python 3 page was a strong reason we felt "safe" starting to make the
> switch to Python 3.
> I'm not sure how we'd get this information about PyPy library
> compatibility. One idea would be to install each library on PyPy, run the
> automated tests, and compare the results against those for CPython.
> Barry
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180102/c815d599/attachment.html>

From barrywhart at yahoo.com  Tue Jan  2 21:12:43 2018
From: barrywhart at yahoo.com (Barry Hart)
Date: Wed, 3 Jan 2018 02:12:43 +0000 (UTC)
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CAL=Xk30RJuvOvzXNMLjWc5_mY1UwcOY+nwHKmscDPhgBmZ4tHQ@mail.gmail.com>
References: <CAL=Xk30RJuvOvzXNMLjWc5_mY1UwcOY+nwHKmscDPhgBmZ4tHQ@mail.gmail.com>
Message-ID: <2144005767.7130612.1514945563455@mail.yahoo.com>

That page looks great -- thanks! I had previously overlooked the link. It might be helpful if the?link text on this page?summarized the compatibility, e.g "980 of the top 1,000 Python packages install successfully".
I notice that some of the failures look like they might have simple fixes (e.g. numpy or Cython not installed). Is there a good way to know if these issues have been addressed in master? If so, fixing issues like this could be a good way for a noob like me to help out the PyPy project.
Barry 

    On Tuesday, January 2, 2018 9:01 PM, John Camara <john.m.camara at gmail.com> wrote:
 

 http://packages.pypy.org/

Question about the web site -- does PyPy currently have anything similar to this page for Python 3?
http://py3readiness.org/

I think a page like this, showing which major libraries are compatible with PyPy, could really help drive adoption of PyPy. I know for our team, the Python 3 page was a strong reason we felt "safe" starting to make the switch to Python 3.
I'm not sure how we'd get this information about PyPy library compatibility. One idea would be to install each library on PyPy, run the automated tests, and compare the results against those for CPython.
Barry


_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/f08413cf/attachment-0001.html>

From matti.picus at gmail.com  Wed Jan  3 00:45:07 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Wed, 3 Jan 2018 07:45:07 +0200
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
Message-ID: <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>

On 03/01/18 02:04, Matt Billenstein wrote:
> On Tue, Jan 02, 2018 at 03:25:17PM -0800, Nathaniel Smith wrote:
>>     I'm pretty sure that the right solution is to ship your own copy of
>>     openssl with the build, so that you're totally independent from Apple's
>>     ssl shenanigans. Maybe look at how CPython handles this.
> That does seem more common practice -- CPython doesn't do that however, it
> links to the old 0.9.8 openssl shipped with OSX since ~10.6.
>
> So it seems we could try to do the same and say the binary pypy releases are
> compatible that far back, or link to the newer LibreSSL which would support
> perhaps Sierra (10.12) and newer.
>
> m
>

Perhaps we should break with CPython here and statically link on macosx. 
Is there a downside to statically linking ssl and libffi?

Note that on python3 macosx we already do this, the _ssl module which is 
built after translation/compilation using cffi uses a post-build script 
[0] that downloads, patches and then statically links, version are 
hardcoded in that file [1].

For pypy2 could we also statically link to a local version of libressl?

[0] 
https://bitbucket.org/pypy/pypy/src/6c36fd9c5c97428030a10d9b5c694fe1d778a14c/pypy/tool/build_cffi_imports.py?at=py3.5&fileviewer=file-view-default#build_cffi_imports.py-177
[1] 
https://bitbucket.org/pypy/pypy/src/6c36fd9c5c97428030a10d9b5c694fe1d778a14c/pypy/tool/build_cffi_imports.py?at=py3.5&fileviewer=file-view-default#build_cffi_imports.py-27

Matti


From matt at vazor.com  Wed Jan  3 00:58:03 2018
From: matt at vazor.com (Matt Billenstein)
Date: Wed, 3 Jan 2018 05:58:03 +0000
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
Message-ID: <01010160ba9918aa-116cf9f3-6181-4498-ba44-4009aa3e8963-000000@us-west-2.amazonses.com>

On Wed, Jan 03, 2018 at 07:45:07AM +0200, Matti Picus wrote:
> Perhaps we should break with CPython here and statically link on
> macosx. Is there a downside to statically linking ssl and libffi?

Seems like a great solution given you've already sorted this out on pypy3.  No
telling what Apple might do with these ssl/ffi libs in the future.

m

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From yury at shurup.com  Wed Jan  3 00:53:40 2018
From: yury at shurup.com (Yury V. Zaytsev)
Date: Wed, 3 Jan 2018 06:53:40 +0100 (CET)
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
Message-ID: <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>

On Wed, 3 Jan 2018, Matti Picus wrote:

> Perhaps we should break with CPython here and statically link on macosx. 
> Is there a downside to statically linking ssl and libffi?

Basically only bloat (not an argument) and necessity to track OpenSSL 
security updates & release new versions when they come out...

-- 
Sincerely yours,
Yury V. Zaytsev

From armin.rigo at gmail.com  Wed Jan  3 02:53:10 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Wed, 3 Jan 2018 08:53:10 +0100
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <2144005767.7130612.1514945563455@mail.yahoo.com>
References: <CAL=Xk30RJuvOvzXNMLjWc5_mY1UwcOY+nwHKmscDPhgBmZ4tHQ@mail.gmail.com>
 <2144005767.7130612.1514945563455@mail.yahoo.com>
Message-ID: <CAMSv6X1s6NEAc8KKYko75AeGN9tiNE0P9x3ACk1YEYM728ogaQ@mail.gmail.com>

Hi Barry,

On 3 January 2018 at 03:12, Barry Hart via pypy-dev <pypy-dev at python.org> wrote:
> I notice that some of the failures look like they might have simple fixes
> (e.g. numpy or Cython not installed). Is there a good way to know if these
> issues have been addressed in master?

Downloading PyPy and trying to run ``pypy -m pip install <name>``.  I
think that's exactly what this web page does.


A bient?t,

Armin.

From armin.rigo at gmail.com  Wed Jan  3 03:08:22 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Wed, 3 Jan 2018 09:08:22 +0100
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
Message-ID: <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>

Hi,

On 3 January 2018 at 06:53, Yury V. Zaytsev <yury at shurup.com> wrote:
> Basically only bloat (not an argument) and necessity to track OpenSSL
> security updates & release new versions when they come out...

...which I'm sure will not be done, because there is no-one in the
PyPy core group that is both security-oriented and an OS X user.

Given that situation my own vote would be to find the easiest solution
that seems to work (maybe just link to the outdated openssl) and add a
warning next to the download link on our web page, with a link to the
homebrew version.

A slightly different note: we now have to worry about two different
ssl messes instead of one, because pypy3 uses cffi but not pypy2.
Maybe the way forward is to throw away the pypy2 version and use cffi
there too.


A bient?t,

Armin.

From matt at vazor.com  Wed Jan  3 05:12:16 2018
From: matt at vazor.com (Matt Billenstein)
Date: Wed, 3 Jan 2018 10:12:16 +0000
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
Message-ID: <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>

On Wed, Jan 03, 2018 at 09:08:22AM +0100, Armin Rigo wrote:
> Given that situation my own vote would be to find the easiest solution
> that seems to work (maybe just link to the outdated openssl) and add a
> warning next to the download link on our web page, with a link to the
> homebrew version.

Googling around -- it seems the headers are missing because Apple doesn't want
people linking to that old version of openssl anymore:

https://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html

They're saying to ship your own version of openssl or use their newer native
Secure Transport APIs.

I don't think pypy should link to homebrew libs since that now adds homebrew as
a dependency -- homebrew isn't an official Apple thing and there are competing
things like Fink and MacPorts which people still use.

So, I think updating LibreSSL branches every 6-12 months and using the latest
point release for a new pypy release is probably a good plan.

m

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From sebastian.pawlus at gmail.com  Wed Jan  3 08:45:57 2018
From: sebastian.pawlus at gmail.com (=?UTF-8?Q?Sebastian_Pawlu=C5=9B?=)
Date: Wed, 3 Jan 2018 14:45:57 +0100
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CAMSv6X1s6NEAc8KKYko75AeGN9tiNE0P9x3ACk1YEYM728ogaQ@mail.gmail.com>
References: <CAL=Xk30RJuvOvzXNMLjWc5_mY1UwcOY+nwHKmscDPhgBmZ4tHQ@mail.gmail.com>
 <2144005767.7130612.1514945563455@mail.yahoo.com>
 <CAMSv6X1s6NEAc8KKYko75AeGN9tiNE0P9x3ACk1YEYM728ogaQ@mail.gmail.com>
Message-ID: <CAEvNYX+70kV5zMWv8f0QowH9RDnc-3gpxmGpsYWy7QMQOgfcsg@mail.gmail.com>

Hi Barry, Armin

Downloading PyPy and trying to run ``pypy -m pip install <name>``.  I
> think that's exactly what this web page does.
>

Inside a docker container, I does exactly that. The project is available
here https://github.com/baroquesoftware/pypy.packages (it's an extremely
simple thing).
As for the missing dependencies should be matter of adding the correct
ubuntu package to Dockerfile.

I think Fija? has all the rights to extend this project.

best.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/1921445b/attachment.html>

From deronnax at gmail.com  Wed Jan  3 12:00:09 2018
From: deronnax at gmail.com (Mathieu Dupuy)
Date: Wed, 3 Jan 2018 18:00:09 +0100
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CAEvNYX+70kV5zMWv8f0QowH9RDnc-3gpxmGpsYWy7QMQOgfcsg@mail.gmail.com>
References: <CAL=Xk30RJuvOvzXNMLjWc5_mY1UwcOY+nwHKmscDPhgBmZ4tHQ@mail.gmail.com>
 <2144005767.7130612.1514945563455@mail.yahoo.com>
 <CAMSv6X1s6NEAc8KKYko75AeGN9tiNE0P9x3ACk1YEYM728ogaQ@mail.gmail.com>
 <CAEvNYX+70kV5zMWv8f0QowH9RDnc-3gpxmGpsYWy7QMQOgfcsg@mail.gmail.com>
Message-ID: <CAKJVQ88MnMyqwwOUJP8-2LVTDaLcJfsj=A75SDYXMmEJsF5n8A@mail.gmail.com>

Also, maybe now that Pypy targeting Python 3 is good enough, the speed
comparison chart on the front page should feature a Python 3 vs Pypy 3
chart. It would better match peoples' current expectation I think. What do
you think ?

2018-01-03 14:45 GMT+01:00 Sebastian Pawlu? <sebastian.pawlus at gmail.com>:

> Hi Barry, Armin
>
> Downloading PyPy and trying to run ``pypy -m pip install <name>``.  I
>> think that's exactly what this web page does.
>>
>
> Inside a docker container, I does exactly that. The project is available
> here https://github.com/baroquesoftware/pypy.packages (it's an extremely
> simple thing).
> As for the missing dependencies should be matter of adding the correct
> ubuntu package to Dockerfile.
>
> I think Fija? has all the rights to extend this project.
>
> best.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/de9bcf89/attachment.html>

From njs at pobox.com  Wed Jan  3 12:06:29 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 3 Jan 2018 09:06:29 -0800
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
Message-ID: <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>

On Jan 3, 2018 02:17, "Matt Billenstein" <matt at vazor.com> wrote:

So, I think updating LibreSSL branches every 6-12 months and using the
latest
point release for a new pypy release is probably a good plan.


BTW you should consult your local cryptographic engineer ? I guess that's
probably Alex Gaynor? ? before deciding between LibreSSL and OpenSSL. I
don't have any first hand experience here myself, but my second hand
impression is that LibreSSL does not have a good reputation.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/74357f56/attachment.html>

From armin.rigo at gmail.com  Wed Jan  3 12:35:38 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Wed, 3 Jan 2018 18:35:38 +0100
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CAKJVQ88MnMyqwwOUJP8-2LVTDaLcJfsj=A75SDYXMmEJsF5n8A@mail.gmail.com>
References: <CAL=Xk30RJuvOvzXNMLjWc5_mY1UwcOY+nwHKmscDPhgBmZ4tHQ@mail.gmail.com>
 <2144005767.7130612.1514945563455@mail.yahoo.com>
 <CAMSv6X1s6NEAc8KKYko75AeGN9tiNE0P9x3ACk1YEYM728ogaQ@mail.gmail.com>
 <CAEvNYX+70kV5zMWv8f0QowH9RDnc-3gpxmGpsYWy7QMQOgfcsg@mail.gmail.com>
 <CAKJVQ88MnMyqwwOUJP8-2LVTDaLcJfsj=A75SDYXMmEJsF5n8A@mail.gmail.com>
Message-ID: <CAMSv6X3PZZv4RD3ZUvEnGNr9fBJmJPJL4MY5R6j735Xja-A3Pw@mail.gmail.com>

Hi Mathieu,

On 3 January 2018 at 18:00, Mathieu Dupuy <deronnax at gmail.com> wrote:
> Also, maybe now that Pypy targeting Python 3 is good enough, the speed
> comparison chart on the front page should feature a Python 3 vs Pypy 3
> chart. It would better match peoples' current expectation I think. What do
> you think ?

That's probably a good idea, but that's a majorly involved project.
The current speed is obtained by running a large variety of Python 2
projects, not Python 3, of course.


A bient?t,

Armin.

From barrywhart at yahoo.com  Wed Jan  3 15:51:35 2018
From: barrywhart at yahoo.com (Barry Hart)
Date: Wed, 3 Jan 2018 20:51:35 +0000 (UTC)
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CAKJVQ88MnMyqwwOUJP8-2LVTDaLcJfsj=A75SDYXMmEJsF5n8A@mail.gmail.com>
References: <CAL=Xk30RJuvOvzXNMLjWc5_mY1UwcOY+nwHKmscDPhgBmZ4tHQ@mail.gmail.com>
 <2144005767.7130612.1514945563455@mail.yahoo.com>
 <CAMSv6X1s6NEAc8KKYko75AeGN9tiNE0P9x3ACk1YEYM728ogaQ@mail.gmail.com>
 <CAEvNYX+70kV5zMWv8f0QowH9RDnc-3gpxmGpsYWy7QMQOgfcsg@mail.gmail.com>
 <CAKJVQ88MnMyqwwOUJP8-2LVTDaLcJfsj=A75SDYXMmEJsF5n8A@mail.gmail.com>
Message-ID: <2027736702.7619310.1515012695037@mail.yahoo.com>

Sebastian,
Thank for the info on how that web page is generated!
I created a PR against the pypy.packages repo which:   
   - Updates to use the latest release (PyPy 3 5.10)
   - Fixes some missing system and Python packages which prevented successful installation of
   
   - numpy
   - scikit-learn
   - matplotlib

https://github.com/baroquesoftware/pypy.packages/pull/9

What is Fijal's email address? It would be great to get this PR merged and the?PyPy packages page?updated since the current info is a couple of years old, and numpy is a highly visible package in the data science community.
Barry 

    On Wednesday, January 3, 2018 12:06 PM, Mathieu Dupuy <deronnax at gmail.com> wrote:
 

 Also, maybe now that Pypy targeting Python 3 is good enough, the speed comparison chart on the front page should feature a Python 3 vs Pypy 3 chart. It would better match peoples' current expectation I think. What do you think ?
2018-01-03 14:45 GMT+01:00 Sebastian Pawlu? <sebastian.pawlus at gmail.com>:

Hi Barry, Armin

Downloading PyPy and trying to run ``pypy -m pip install <name>``.? I
think that's exactly what this web page does.


Inside a docker container, I does exactly that. The project is available here?https://github.com/ baroquesoftware/pypy.packages (it's an extremely simple thing).?As for the missing dependencies should be matter of adding the correct ubuntu package to Dockerfile.?
I think Fija? has all the rights to extend this project.?
best.??
______________________________ _________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/ mailman/listinfo/pypy-dev



_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/ccb93ead/attachment-0001.html>

From kotrfa at gmail.com  Wed Jan  3 16:31:56 2018
From: kotrfa at gmail.com (Kotrfa)
Date: Wed, 03 Jan 2018 21:31:56 +0000
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <544121945.7050109.1514935318318@mail.yahoo.com>
References: <CADB0KUKHf=oRXUPsh7_T=Y9Sp6ZETajdNVkEX_4L1AVRJmz4qQ@mail.gmail.com>
 <CAK5idxSwdNepiqyFrhZTy08CxU_jd9iDS9bq6ueJ4X+ShYm6vQ@mail.gmail.com>
 <CADB0KUL7G2+JjQZM1D0Ai11WONx=MpVYXFHJpW2AeFVSpG4YcA@mail.gmail.com>
 <CAPkN8xJ2BFQt6AU-1nYXAyjFF6dsVJpBE-yRKkzv+cg+4CuOwQ@mail.gmail.com>
 <CADB0KUKqD9LNkZqmW43XHK=HXkDTGZLZHiA=kkSeGrMMyuZztA@mail.gmail.com>
 <544121945.7050109.1514935318318@mail.yahoo.com>
Message-ID: <CADB0KUJRtNrhCor6kYremgVOw7ma3w7hzwuO0E69XY2kNCtP5w@mail.gmail.com>

Do you mean http://packages.pypy.org/ ?

st 3. 1. 2018 v 0:22 odes?latel Barry Hart <barrywhart at yahoo.com> napsal:

> Question about the web site -- does PyPy currently have anything similar
> to this page for Python 3?
>
> http://py3readiness.org/
>
> I think a page like this, showing which major libraries are compatible
> with PyPy, could really help drive adoption of PyPy. I know for our team,
> the Python 3 page was a strong reason we felt "safe" starting to make the
> switch to Python 3.
>
> I'm not sure how we'd get this information about PyPy library
> compatibility. One idea would be to install each library on PyPy, run the
> automated tests, and compare the results against those for CPython.
>
> Barry
>
>
> On Tuesday, January 2, 2018 5:31 PM, Kotrfa <kotrfa at gmail.com> wrote:
>
>
> Yeah, that's a very good point worth mentioning. I am not against an idea
> going just with a static version.
>
> po 1. 1. 2018 v 18:05 odes?latel anatoly techtonik <techtonik at gmail.com>
> napsal:
>
> On Sat, Dec 30, 2017 at 12:41 PM, Kotrfa <kotrfa at gmail.com> wrote:
> > Hey,
> >
> > as I said, I am not a professional, but I would be willing to help with
> the
> > site. I have built couple of websites in Django/Wagtail (or plain HTML),
> > have some basic knowledge about design (+ frontend CSS/JS/HTML). The site
> > seems quite simple, so if we just did a revamp, it shouldn't take a long.
> > Also, if we went with Wagtail, adding blog capabilities would be also
> > trivial.
>
> Just want to warn that Wagtail and Django means database, which adds
> maintenance overhead compared to current static version. Logins, spam
> and all that stuff.. I don't mind against custom blog engine, but
> putting all static content from version control into Wagtail/Django
> database doesn't seem a good solution to me. When site gets hacked,
> for example to change donation credentials, there is no trace who did
> what. So going with simple static site solution is probably a better
> way. It also doesn't require permission or privilege to send a pull
> request, and content previews can be made with Netlify.
>
> Happy New Year. =)
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/102117bc/attachment.html>

From barrywhart at yahoo.com  Wed Jan  3 16:47:27 2018
From: barrywhart at yahoo.com (Barry Hart)
Date: Wed, 3 Jan 2018 21:47:27 +0000 (UTC)
Subject: [pypy-dev] Pypy's facelift and certificate issues
In-Reply-To: <CADB0KUJRtNrhCor6kYremgVOw7ma3w7hzwuO0E69XY2kNCtP5w@mail.gmail.com>
References: <CADB0KUKHf=oRXUPsh7_T=Y9Sp6ZETajdNVkEX_4L1AVRJmz4qQ@mail.gmail.com>
 <CAK5idxSwdNepiqyFrhZTy08CxU_jd9iDS9bq6ueJ4X+ShYm6vQ@mail.gmail.com>
 <CADB0KUL7G2+JjQZM1D0Ai11WONx=MpVYXFHJpW2AeFVSpG4YcA@mail.gmail.com>
 <CAPkN8xJ2BFQt6AU-1nYXAyjFF6dsVJpBE-yRKkzv+cg+4CuOwQ@mail.gmail.com>
 <CADB0KUKqD9LNkZqmW43XHK=HXkDTGZLZHiA=kkSeGrMMyuZztA@mail.gmail.com>
 <544121945.7050109.1514935318318@mail.yahoo.com>
 <CADB0KUJRtNrhCor6kYremgVOw7ma3w7hzwuO0E69XY2kNCtP5w@mail.gmail.com>
Message-ID: <605683268.7661382.1515016047573@mail.yahoo.com>

Yes, that's the page the PR relates to.
Barry 

    On Wednesday, January 3, 2018 4:32 PM, Kotrfa <kotrfa at gmail.com> wrote:
 

 Do you mean?http://packages.pypy.org/ ?
st 3. 1. 2018 v 0:22 odes?latel Barry Hart <barrywhart at yahoo.com> napsal:

Question about the web site -- does PyPy currently have anything similar to this page for Python 3?
http://py3readiness.org/

I think a page like this, showing which major libraries are compatible with PyPy, could really help drive adoption of PyPy. I know for our team, the Python 3 page was a strong reason we felt "safe" starting to make the switch to Python 3.
I'm not sure how we'd get this information about PyPy library compatibility. One idea would be to install each library on PyPy, run the automated tests, and compare the results against those for CPython.
Barry 

  On Tuesday, January 2, 2018 5:31 PM, Kotrfa <kotrfa at gmail.com> wrote:
 

 Yeah, that's a very good point worth mentioning. I am not against an idea going just with a static version.
po 1. 1. 2018 v 18:05 odes?latel anatoly techtonik <techtonik at gmail.com> napsal:

On Sat, Dec 30, 2017 at 12:41 PM, Kotrfa <kotrfa at gmail.com> wrote:
> Hey,
>
> as I said, I am not a professional, but I would be willing to help with the
> site. I have built couple of websites in Django/Wagtail (or plain HTML),
> have some basic knowledge about design (+ frontend CSS/JS/HTML). The site
> seems quite simple, so if we just did a revamp, it shouldn't take a long.
> Also, if we went with Wagtail, adding blog capabilities would be also
> trivial.

Just want to warn that Wagtail and Django means database, which adds
maintenance overhead compared to current static version. Logins, spam
and all that stuff.. I don't mind against custom blog engine, but
putting all static content from version control into Wagtail/Django
database doesn't seem a good solution to me. When site gets hacked,
for example to change donation credentials, there is no trace who did
what. So going with simple static site solution is probably a better
way. It also doesn't require permission or privilege to send a pull
request, and content previews can be made with Netlify.

Happy New Year. =)

_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev





   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/50372c9c/attachment-0001.html>

From alex.gaynor at gmail.com  Wed Jan  3 18:51:21 2018
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 3 Jan 2018 18:51:21 -0500
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
 <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
Message-ID: <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>

If PyPy releases include a copy of OpenSSL (or LibreSSL) then we need to be
in the business of issuing new releases whenever upstream has a security
release, we can't be shipping people OpenSSLs with known security issues.

Of LibreSSL and OpenSSL, I'd choose to ship OpenSSL -- I've found LibreSSL
fairly frustrating to work with and OpenSSL upstream is considerably
cleaned up in past years.

Alex

On Wed, Jan 3, 2018 at 12:06 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Jan 3, 2018 02:17, "Matt Billenstein" <matt at vazor.com> wrote:
>
> So, I think updating LibreSSL branches every 6-12 months and using the
> latest
> point release for a new pypy release is probably a good plan.
>
>
> BTW you should consult your local cryptographic engineer ? I guess that's
> probably Alex Gaynor? ? before deciding between LibreSSL and OpenSSL. I
> don't have any first hand experience here myself, but my second hand
> impression is that LibreSSL does not have a good reputation.
>
> -n
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/6385000d/attachment.html>

From matt at vazor.com  Wed Jan  3 19:05:07 2018
From: matt at vazor.com (Matt Billenstein)
Date: Thu, 4 Jan 2018 00:05:07 +0000
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
 <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
 <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
Message-ID: <01010160be7c52ff-3b00f475-4f23-42f6-b649-031389737fa4-000000@us-west-2.amazonses.com>

On Wed, Jan 03, 2018 at 06:51:21PM -0500, Alex Gaynor wrote:
>    If PyPy releases include a copy of OpenSSL (or LibreSSL) then we need to
>    be in the business of issuing new releases whenever upstream has a
>    security release, we can't be shipping people OpenSSLs with known security
>    issues.

To a degree correct?  I don't know if everyone who bundles ships every point
release, but, if it's heartbleed all over again, you need to cut a new release.

m

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From alex.gaynor at gmail.com  Wed Jan  3 19:06:13 2018
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 3 Jan 2018 19:06:13 -0500
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <01010160be7c52ff-3b00f475-4f23-42f6-b649-031389737fa4-000000@us-west-2.amazonses.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
 <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
 <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
 <01010160be7c52ff-3b00f475-4f23-42f6-b649-031389737fa4-000000@us-west-2.amazonses.com>
Message-ID: <CAFRnB2WcrTZ3LB-xYuZUfF-yBw_BVNFp3KjPo96+jn5DzaNmCQ@mail.gmail.com>

pyca/cryptography issues a new release on all platforms for any OpenSSL
security releases.

:-),
Alex

On Wed, Jan 3, 2018 at 7:05 PM, Matt Billenstein <matt at vazor.com> wrote:

> On Wed, Jan 03, 2018 at 06:51:21PM -0500, Alex Gaynor wrote:
> >    If PyPy releases include a copy of OpenSSL (or LibreSSL) then we need
> to
> >    be in the business of issuing new releases whenever upstream has a
> >    security release, we can't be shipping people OpenSSLs with known
> security
> >    issues.
>
> To a degree correct?  I don't know if everyone who bundles ships every
> point
> release, but, if it's heartbleed all over again, you need to cut a new
> release.
>
> m
>
> --
> Matt Billenstein
> matt at vazor.com
> http://www.vazor.com/
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180103/f6b67aeb/attachment.html>

From matt at vazor.com  Wed Jan  3 20:11:48 2018
From: matt at vazor.com (Matt Billenstein)
Date: Thu, 4 Jan 2018 01:11:48 +0000
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <CAFRnB2WcrTZ3LB-xYuZUfF-yBw_BVNFp3KjPo96+jn5DzaNmCQ@mail.gmail.com>
References: <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
 <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
 <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
 <01010160be7c52ff-3b00f475-4f23-42f6-b649-031389737fa4-000000@us-west-2.amazonses.com>
 <CAFRnB2WcrTZ3LB-xYuZUfF-yBw_BVNFp3KjPo96+jn5DzaNmCQ@mail.gmail.com>
Message-ID: <01010160beb960a6-910b2fd3-d34e-4bc3-8710-538a5f9f6151-000000@us-west-2.amazonses.com>

On Wed, Jan 03, 2018 at 07:06:13PM -0500, Alex Gaynor wrote:
>    pyca/cryptography issues a new release on all platforms for any OpenSSL
>    security releases.

Yeah, but that's part of the library ecosystem -- not really an end product?

m

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From njs at pobox.com  Wed Jan  3 20:15:54 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 3 Jan 2018 17:15:54 -0800
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
 <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
 <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
Message-ID: <CAPJVwBnbdUR0ZbnYc3dthw=v86Sxmn_aDwDm5HTdZ6ZEGVohoA@mail.gmail.com>

On Wed, Jan 3, 2018 at 3:51 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> If PyPy releases include a copy of OpenSSL (or LibreSSL) then we need to be
> in the business of issuing new releases whenever upstream has a security
> release, we can't be shipping people OpenSSLs with known security issues.
>
> Of LibreSSL and OpenSSL, I'd choose to ship OpenSSL -- I've found LibreSSL
> fairly frustrating to work with and OpenSSL upstream is considerably cleaned
> up in past years.

None of Linux, Windows, or MacOS provide reasonable pre-existing
OpenSSL installs you can use. So it seems to me that if PyPy's going
to ship any binaries at all and take that seriously, then it's going
to have to ship OpenSSL (or LibreSSL), and do whatever security
updates you all decide make sense.

It's also probably not worth spending a lot of time trying to figure
out how to avoid doing security updates for pypy2 on MacOS, if you're
still going to have to do them for other binaries on other platforms.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From matti.picus at gmail.com  Wed Jan  3 23:50:39 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 4 Jan 2018 06:50:39 +0200
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <CAPJVwBnbdUR0ZbnYc3dthw=v86Sxmn_aDwDm5HTdZ6ZEGVohoA@mail.gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
 <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
 <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
 <CAPJVwBnbdUR0ZbnYc3dthw=v86Sxmn_aDwDm5HTdZ6ZEGVohoA@mail.gmail.com>
Message-ID: <4551e8be-5834-c2f7-3807-cb2031048dac@gmail.com>

On 1/4/2018 3:15 AM, Nathaniel Smith wrote:

> On Wed, Jan 3, 2018 at 3:51 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>> If PyPy releases include a copy of OpenSSL (or LibreSSL) then we need to be
>> in the business of issuing new releases whenever upstream has a security
>> release, we can't be shipping people OpenSSLs with known security issues.
>>
>> Of LibreSSL and OpenSSL, I'd choose to ship OpenSSL -- I've found LibreSSL
>> fairly frustrating to work with and OpenSSL upstream is considerably cleaned
>> up in past years.
> None of Linux, Windows, or MacOS provide reasonable pre-existing
> OpenSSL installs you can use. So it seems to me that if PyPy's going
> to ship any binaries at all and take that seriously, then it's going
> to have to ship OpenSSL (or LibreSSL), and do whatever security
> updates you all decide make sense.
>
> It's also probably not worth spending a lot of time trying to figure
> out how to avoid doing security updates for pypy2 on MacOS, if you're
> still going to have to do them for other binaries on other platforms.
>
> -n
>
Let's leave libffi out of the discussion, I assume there is no objection 
to statically linking to it.

As for OpenSSL/LibreSSL the situation is not straight-forward. Here is 
my assessment, please correct me if I am wrong.

In windows, both PyPy and CPython statically link to OpenSSL

In linux, PyPy and CPython use the platform OpenSSL.

On macosx, _ssl cffi (as of the first release v5.10) uses a 
statically-linked LibreSSL with a patch for python3, and on python2 
AFAICT both CPython and PyPy use a platform library, not clear to me 
which one.

What does CPython do for macosx python3?

Matti

From matt at vazor.com  Thu Jan  4 00:11:59 2018
From: matt at vazor.com (Matt Billenstein)
Date: Thu, 4 Jan 2018 05:11:59 +0000
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <4551e8be-5834-c2f7-3807-cb2031048dac@gmail.com>
References: <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
 <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
 <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
 <CAPJVwBnbdUR0ZbnYc3dthw=v86Sxmn_aDwDm5HTdZ6ZEGVohoA@mail.gmail.com>
 <4551e8be-5834-c2f7-3807-cb2031048dac@gmail.com>
Message-ID: <01010160bf954779-26cffdc9-9fb0-4710-aa55-57b55caf6874-000000@us-west-2.amazonses.com>

Looks like they ship a shared lib on osx which is different from how they
handle 2.7:

mattb at mattb-mbp2:/Library/Frameworks/Python.framework/Versions $ find . -name '*ssl*.so' | xargs otool -L
./2.7/lib/python2.7/lib-dynload/_ssl.so:
        /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)

./3.6/lib/python3.6/lib-dynload/_ssl.cpython-36m-darwin.so:
        /Library/Frameworks/Python.framework/Versions/3.6/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        /Library/Frameworks/Python.framework/Versions/3.6/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)

I thought I saw some talk of enhancements on python-dev in ssl in python that
were only being done in python3 which is maybe why they do it this way.

m

On Thu, Jan 04, 2018 at 06:50:39AM +0200, Matti Picus wrote:
> On 1/4/2018 3:15 AM, Nathaniel Smith wrote:
> 
> >On Wed, Jan 3, 2018 at 3:51 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> >>If PyPy releases include a copy of OpenSSL (or LibreSSL) then we need to be
> >>in the business of issuing new releases whenever upstream has a security
> >>release, we can't be shipping people OpenSSLs with known security issues.
> >>
> >>Of LibreSSL and OpenSSL, I'd choose to ship OpenSSL -- I've found LibreSSL
> >>fairly frustrating to work with and OpenSSL upstream is considerably cleaned
> >>up in past years.
> >None of Linux, Windows, or MacOS provide reasonable pre-existing
> >OpenSSL installs you can use. So it seems to me that if PyPy's going
> >to ship any binaries at all and take that seriously, then it's going
> >to have to ship OpenSSL (or LibreSSL), and do whatever security
> >updates you all decide make sense.
> >
> >It's also probably not worth spending a lot of time trying to figure
> >out how to avoid doing security updates for pypy2 on MacOS, if you're
> >still going to have to do them for other binaries on other platforms.
> >
> >-n
> >
> Let's leave libffi out of the discussion, I assume there is no
> objection to statically linking to it.
> 
> As for OpenSSL/LibreSSL the situation is not straight-forward. Here
> is my assessment, please correct me if I am wrong.
> 
> In windows, both PyPy and CPython statically link to OpenSSL
> 
> In linux, PyPy and CPython use the platform OpenSSL.
> 
> On macosx, _ssl cffi (as of the first release v5.10) uses a
> statically-linked LibreSSL with a patch for python3, and on python2
> AFAICT both CPython and PyPy use a platform library, not clear to me
> which one.
> 
> What does CPython do for macosx python3?
> 
> Matti

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From njs at pobox.com  Thu Jan  4 00:57:29 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 3 Jan 2018 21:57:29 -0800
Subject: [pypy-dev] pypy 5.10 release
In-Reply-To: <4551e8be-5834-c2f7-3807-cb2031048dac@gmail.com>
References: <01010160b8f71190-8dfb3390-0cc8-4012-8aad-566d4d691950-000000@us-west-2.amazonses.com>
 <CAPJVwBng92R4eehsMmNpjpNuDBz7ymUVNu79zoepPOKUvr0_xw@mail.gmail.com>
 <01010160b9552c68-d0df4e25-5b3c-496f-a833-1f4d141eb07d-000000@us-west-2.amazonses.com>
 <d3249300-83f3-9871-9386-e8e77969f907@gmail.com>
 <alpine.LRH.2.21.1801030652280.1773@vps.zaytsev.net>
 <CAMSv6X0h4TcC57fOpao2pyBGku5zd47qnTkdMC9mVGe+yH7Fog@mail.gmail.com>
 <01010160bb81d4d5-e96e28bc-1dba-47fa-910d-abefa37766f3-000000@us-west-2.amazonses.com>
 <CAPJVwB=wGDyp7a9939bJ9haf0O-6kg3GX377_zqi3KFpkPgPCw@mail.gmail.com>
 <CAFRnB2Uhgk8P+HXmOSGxJYgyX=dzVyNta3xUZL1t5zMNZDZj-w@mail.gmail.com>
 <CAPJVwBnbdUR0ZbnYc3dthw=v86Sxmn_aDwDm5HTdZ6ZEGVohoA@mail.gmail.com>
 <4551e8be-5834-c2f7-3807-cb2031048dac@gmail.com>
Message-ID: <CAPJVwBnn1xJ4vZ-W2OFrF0B732A-Eh2kO=T21rUFX_TVDpNcQg@mail.gmail.com>

On Wed, Jan 3, 2018 at 8:50 PM, Matti Picus <matti.picus at gmail.com> wrote:
> On 1/4/2018 3:15 AM, Nathaniel Smith wrote:
>> None of Linux, Windows, or MacOS provide reasonable pre-existing
>> OpenSSL installs you can use. So it seems to me that if PyPy's going
>> to ship any binaries at all and take that seriously, then it's going
>> to have to ship OpenSSL (or LibreSSL), and do whatever security
>> updates you all decide make sense.
>>
>> It's also probably not worth spending a lot of time trying to figure
>> out how to avoid doing security updates for pypy2 on MacOS, if you're
>> still going to have to do them for other binaries on other platforms.
>>
>> -n
>>
> Let's leave libffi out of the discussion, I assume there is no objection to
> statically linking to it.
>
> As for OpenSSL/LibreSSL the situation is not straight-forward. Here is my
> assessment, please correct me if I am wrong.
>
> In windows, both PyPy and CPython statically link to OpenSSL
>
> In linux, PyPy and CPython use the platform OpenSSL.

This depends on how PyPy/CPython is distributed. If you're getting
them from your distro, then they use the distro OpenSSL. CPython
doesn't have any official binary releases on Linux, so for them that's
the end of the story. If you're happy with telling Linux users that
they need to get their PyPy from their distro, then maybe that's the
end of the story for you too.

But, I think there are a lot of advantages to PyPy providing pre-built
binaries for Linux that work across distros, and if you want to do
that, then you have to ship your own OpenSSL. This is how Squeaky's
portable builds work.

> On macosx, _ssl cffi (as of the first release v5.10) uses a
> statically-linked LibreSSL with a patch for python3, and on python2 AFAICT
> both CPython and PyPy use a platform library, not clear to me which one.
>
> What does CPython do for macosx python3?

It ships a recent OpenSSL as part of the release.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From hubo at jiedaibao.com  Fri Jan  5 03:19:36 2018
From: hubo at jiedaibao.com (hubo)
Date: Fri, 05 Jan 2018 16:19:36 +0800
Subject: [pypy-dev] Mysterious IndexError in service running with PyPy
In-Reply-To: <5A3C662E.9070506@jiedaibao.com>
References: <5a37757a.4a98500a.ba397.d553SMTPIN_ADDED_BROKEN@mx.google.com>
 <CAHgd1hFAL8M4Ccv8nrFrzrpXWgsWkhSBgZkmTJrMk_2yGipj=A@mail.gmail.com>
 <5A3C662E.9070506@jiedaibao.com>
Message-ID: <5A4F3516.20006@jiedaibao.com>+3E4C406F313C7D2C

Updating:

This issue is reproduced on PyPy 5.9. I will collect running information (local variables, etc.) for hints.

2018-01-05 

hubo 



????"hubo" <hubo at jiedaibao.com>
?????2017-12-22 09:55
???Re: [pypy-dev] Mysterious IndexError in service running with PyPy
????"William ML Leslie"<william.leslie.ttg at gmail.com>
???"PyPy Developer Mailing List"<pypy-dev at python.org>

It is a regular guess, but this method is guareenteed to be called only in the main thread. Thread pool usage is very limited in this program because it is coroutine-based.

2017-12-22 

hubo 



????William ML Leslie <william.leslie.ttg at gmail.com>
?????2017-12-19 20:14
???Re: [pypy-dev] Mysterious IndexError in service running with PyPy
????"hubo"<hubo at jiedaibao.com>
???"PyPy Developer Mailing List"<pypy-dev at python.org>

Where is the code that changes the size of self.heap? How do we know that size(self.heap) is constant? My guess is that some thread changes this; but l is not recomputed.


On 18 Dec 2017 6:59 PM, "hubo" <hubo at jiedaibao.com> wrote:

I'm reporting this issue in this mail group, though I don't know if it is related with PyPy, because it is really strange, and is not able to reproduce stably. But I hope someone may know the reason or have some points.

I'm running some SDN services written in Python with PyPy 5.3.0.  The related code is here:
https://github.com/hubo1016/vlcp/blob/8022e3a3c67cf4305af503d507640a730ca3949d/vlcp/utils/indexedheap.py#L100
The full code is also in the repo, but may be too complex to describe. But this related piece is quite simple:

    def _siftdown(self, pos):
        temp = self.heap[pos]
        l = len(self.heap)
        while pos * 2 + 1 < l:
            cindex = pos * 2 + 1
            pt = self.heap[cindex]
            if cindex + 1 < l and self.heap[cindex+1][0] < pt[0]:
                cindex = cindex + 1
                pt = self.heap[cindex]
            if pt[0] < temp[0]:
                self.heap[pos] = pt
                self.index[pt[1]] = pos
            else:
                break
            pos = cindex
        self.heap[pos] = temp
        self.index[temp[1]] = pos

It is a simple heap operation. The service uses a heap to process timers. When the service is not busy, it usually runs this piece of code several times per minute.

I have 32 servers running this service. They are quite stable in about three months, but one day one of the services crashes on line 100 reporting IndexError:
     pt = self.heap[cindex]

As you can see, cindex = pos * 2 + 1, which is tested by the while pre-conditon just two lines before. And there is not any multi-threading issues here because this piece of code always runs in the same thread. So it is not possible in theory for this to happen.

Only the following facts are known about this issue:
It reproduces  - through quite rarely. I've met this kind of crashes 4 times, each with different machine, so it should not be related to hardware issuess. Since I've got 32 servers, it might take more than one year to reproduce with a single server. 
It seems not to be related to pressures. All of the crashes happens at night, when there are little requests. Only some cleanup tasks are running in fixed interval.
The services are running with PyPy 5.3.0. I've upgraded a few of them to 5.9, but it will take a long time to validate whether this still happens. And It is not validated on CPython too. I'm also trying to collect more debugging information for this issue, but it is very hard since it rarely reproduces.

It is not a serious issue. It could be workarounded with a auto-restart, but I'm searching the cause.


2017-12-18


hubo 

_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180105/703dcbca/attachment.html>

From armin.rigo at gmail.com  Fri Jan  5 03:58:23 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Fri, 5 Jan 2018 09:58:23 +0100
Subject: [pypy-dev] Mysterious IndexError in service running with PyPy
In-Reply-To: <5a4f355f.1a9b500a.94880.b8c3SMTPIN_ADDED_BROKEN@mx.google.com>
References: <5a37757a.4a98500a.ba397.d553SMTPIN_ADDED_BROKEN@mx.google.com>
 <CAHgd1hFAL8M4Ccv8nrFrzrpXWgsWkhSBgZkmTJrMk_2yGipj=A@mail.gmail.com>
 <5A3C662E.9070506@jiedaibao.com>
 <5a4f355f.1a9b500a.94880.b8c3SMTPIN_ADDED_BROKEN@mx.google.com>
Message-ID: <CAMSv6X2sTbsd5A_rRm_gZgfjgOGLU8=P0y7WSwh=TWxkHcgqKA@mail.gmail.com>

Hi Hubo,

On 5 January 2018 at 09:19, hubo <hubo at jiedaibao.com> wrote:
> This issue is reproduced on PyPy 5.9. I will collect running information
> (local variables, etc.) for hints.

As I said, this is unlikely to help (unless of course there is a bug
in your program logic and you find it by looking at the value of local
variables).  We need a way to reproduce the problem ourselves, and
then we can attack it with lower-level tools like gdb, rr, and so on.
Without it, we can't do anything.


A bient?t,

Armin.

From fijall at gmail.com  Sun Jan  7 04:26:48 2018
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 7 Jan 2018 10:26:48 +0100
Subject: [pypy-dev] PyPy sprint in Poland
Message-ID: <CAK5idxThBiakN7nu+Jka3mpBdJh8eJXXbf7wuDcZnmXvRaD8uQ@mail.gmail.com>

Hi Everyone.

It looks like I would be in Europe through April and maybe May. Anyone
fancy a sprint somewhere in Poland? Potential venues:

* we can have a sprint at my climbing spot - it's quite a problem to
get to (~2-3h by train from either Prague or Wroclaw), but it's
incredibly lovely at this time of the year. There is a venue and
internet that we can use. Limited restaurant options are a con.
Endless hiking options are a plus.

* we can try to organize a venue in Warsaw. We don't have a place just
yet, but it's easy to get to, relatively cheap, abundant in places to
eat out. We don't have a place to sprint at just yet, but maybe we can
organize something at the Uni.

* Krakow. Might be easier to organize venue. Slightly harder to get to
than Warsaw, quite a bit nicer.

* Wroclaw. We had a tiny sprint there with Armin when we merged the
JIT to pypy :-) A bit harder to get to than Warsaw, a very nice city,
we don't have a venue organized yet (but it's possible I think),
easier to do a few days excursion from there.

Thoughts?
fijal

From hubo at jiedaibao.com  Mon Jan  8 04:28:08 2018
From: hubo at jiedaibao.com (hubo)
Date: Mon, 08 Jan 2018 17:28:08 +0800
Subject: [pypy-dev] Mysterious IndexError in service running with PyPy
In-Reply-To: <CAMSv6X2sTbsd5A_rRm_gZgfjgOGLU8=P0y7WSwh=TWxkHcgqKA@mail.gmail.com>
References: <5a37757a.4a98500a.ba397.d553SMTPIN_ADDED_BROKEN@mx.google.com>
 <CAHgd1hFAL8M4Ccv8nrFrzrpXWgsWkhSBgZkmTJrMk_2yGipj=A@mail.gmail.com>
 <5A3C662E.9070506@jiedaibao.com>
 <5a4f355f.1a9b500a.94880.b8c3SMTPIN_ADDED_BROKEN@mx.google.com>
 <CAMSv6X2sTbsd5A_rRm_gZgfjgOGLU8=P0y7WSwh=TWxkHcgqKA@mail.gmail.com>
Message-ID: <5A5339A6.9060303@jiedaibao.com>+220B01240AA56492

I understand. I'm just trying to figure out the triggering condition so I can reproduce it easier, and might finally submit a plan to reproduce.

2018-01-08 

hubo 



????Armin Rigo <armin.rigo at gmail.com>
?????2018-01-05 16:58
???Re: [pypy-dev] Mysterious IndexError in service running with PyPy
????"hubo"<hubo at jiedaibao.com>
???"PyPy Developer Mailing List"<pypy-dev at python.org>

Hi Hubo, 

On 5 January 2018 at 09:19, hubo <hubo at jiedaibao.com> wrote: 
> This issue is reproduced on PyPy 5.9. I will collect running information 
> (local variables, etc.) for hints. 

As I said, this is unlikely to help (unless of course there is a bug 
in your program logic and you find it by looking at the value of local 
variables).  We need a way to reproduce the problem ourselves, and 
then we can attack it with lower-level tools like gdb, rr, and so on. 
Without it, we can't do anything. 


A bient?t, 

Armin. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180108/73f1cca9/attachment.html>

From anto.cuni at gmail.com  Mon Jan  8 05:05:43 2018
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 8 Jan 2018 11:05:43 +0100
Subject: [pypy-dev] PyPy sprint in Poland
In-Reply-To: <CAK5idxThBiakN7nu+Jka3mpBdJh8eJXXbf7wuDcZnmXvRaD8uQ@mail.gmail.com>
References: <CAK5idxThBiakN7nu+Jka3mpBdJh8eJXXbf7wuDcZnmXvRaD8uQ@mail.gmail.com>
Message-ID: <CAHPg78yyyQ65uxPnnFGige474Sf_aAqFc8y+jyOEwFPo+xeYPQ@mail.gmail.com>

Hi,
I'd be happy to come. Since I have already been to Warsaw, I vote for
Krakow or Wroclaw.

The only thing is that April is quite busy for me; ATM, the only reasonable
dates are somewhere between 3rd and 13th. May it's surely easier :)

On Sun, Jan 7, 2018 at 10:26 AM, Maciej Fijalkowski <fijall at gmail.com>
wrote:

> Hi Everyone.
>
> It looks like I would be in Europe through April and maybe May. Anyone
> fancy a sprint somewhere in Poland? Potential venues:
>
> * we can have a sprint at my climbing spot - it's quite a problem to
> get to (~2-3h by train from either Prague or Wroclaw), but it's
> incredibly lovely at this time of the year. There is a venue and
> internet that we can use. Limited restaurant options are a con.
> Endless hiking options are a plus.
>
> * we can try to organize a venue in Warsaw. We don't have a place just
> yet, but it's easy to get to, relatively cheap, abundant in places to
> eat out. We don't have a place to sprint at just yet, but maybe we can
> organize something at the Uni.
>
> * Krakow. Might be easier to organize venue. Slightly harder to get to
> than Warsaw, quite a bit nicer.
>
> * Wroclaw. We had a tiny sprint there with Armin when we merged the
> JIT to pypy :-) A bit harder to get to than Warsaw, a very nice city,
> we don't have a venue organized yet (but it's possible I think),
> easier to do a few days excursion from there.
>
> Thoughts?
> fijal
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180108/c502ef4b/attachment.html>

From armin.rigo at gmail.com  Mon Jan  8 05:35:36 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 8 Jan 2018 11:35:36 +0100
Subject: [pypy-dev] Leysin Winter sprint: 17-24 March 2018
Message-ID: <CAMSv6X3FSLHphJw_NoEyrh=0rvrwQCaeB3ySE3R_j9tX_zy9uA@mail.gmail.com>

=====================================================================
      PyPy Leysin Winter Sprint (17th - 24th March 2018)
=====================================================================

The next PyPy sprint will be in Leysin, Switzerland, for the thirteenth
time.  This is a fully public sprint: newcomers and topics other than
those proposed below are welcome.

(Note: this sprint is independent from the suggested April-May sprint in
Poland.)


------------------------------
Goals and topics of the sprint
------------------------------

The list of topics is open, but here is our current list:

- cffi tutorial/overview rewrite
- py3 test runners are too complicated
- make win32 builds green
- make packaging more like cpython/portable builds
- get CI builders for PyPy into mainstream projects (Numpy, Scipy, lxml, uwsgi)
- get more of scientific stack working (tensorflow?)
- cpyext performance improvements
- General 3.5 and 3.6 improvements
- JIT topics: guard-compatible, and the subsequent research project to
save and reuse traces across processes
- finish unicode-utf8
- update www.pypy.org, speed.pypy.org (web devs needed)

As usual, the main side goal is to have fun in winter sports :-)
We can take a day off (for ski or anything else).

-----------
Exact times
-----------

Work days: starting March 18th (~noon), ending March 24th (~noon).

I have pre-booked the week from Saturday March 17th to Saturday March
24th.  If it is possible for you to arrive Sunday before mid-afternoon,
then you can get a booking from Sunday only.  The break day should be
around Wednesday.

It is fine to stay a few more days on either side, or conversely to book
for a part of that time only.

-----------------------
Location & Accomodation
-----------------------

Leysin, Switzerland, "same place as before".  Let me refresh your
memory: both the sprint venue and the lodging will be in a pair of
chalets built specifically for bed & breakfast: http://www.ermina.ch/.
You can also arrange your own lodging elsewhere (as long as you are in
Leysin, you cannot be more than a 15 minutes walk away from the sprint
venue).

Please *confirm* that you are coming so that we can adjust the
reservations as appropriate.

The options of rooms are a bit more limited than on previous years
because the place for bed-and-breakfast is shrinking; but we should
still have enough room for us.  The price is around 60 CHF, breakfast
included, in shared rooms (3 or 4 people).  If there are people that
would prefer a double or single room, please contact me and we'll see
what choices you have.  There is a choice of hotels in Leysin, starting
at 65 CHF the room.

Please register by Mercurial::

  https://bitbucket.org/pypy/extradoc/
  https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2018/

or on the pypy-dev mailing list if you do not yet have check-in rights:

  http://mail.python.org/mailman/listinfo/pypy-dev

You need a Swiss-to-(insert country here) power adapter.  There will be
some Swiss-to-EU adapters around, and at least one EU-format power strip.

From armin.rigo at gmail.com  Mon Jan  8 05:37:59 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 8 Jan 2018 11:37:59 +0100
Subject: [pypy-dev] PyPy sprint in Poland
In-Reply-To: <CAK5idxThBiakN7nu+Jka3mpBdJh8eJXXbf7wuDcZnmXvRaD8uQ@mail.gmail.com>
References: <CAK5idxThBiakN7nu+Jka3mpBdJh8eJXXbf7wuDcZnmXvRaD8uQ@mail.gmail.com>
Message-ID: <CAMSv6X0ufgDRA=j3kCV0L8gGESUTXYdrvVQ96FNOdEab-wFriQ@mail.gmail.com>

Hi,

On 7 January 2018 at 10:26, Maciej Fijalkowski <fijall at gmail.com> wrote:
> * we can have a sprint at my climbing spot - it's quite a problem to
> get to (~2-3h by train from either Prague or Wroclaw), but it's
> incredibly lovely at this time of the year. There is a venue and
> internet that we can use. Limited restaurant options are a con.
> Endless hiking options are a plus.

I'd be +1 on that one.  Two or three hours of train from Prague is not
a big issue, at least for me.

I guess another +1 would be if it's not immediately after the Leysin sprint :-)


A bient?t,

Armin.

From me at manueljacob.de  Tue Jan  9 13:50:50 2018
From: me at manueljacob.de (Manuel Jacob)
Date: Tue, 09 Jan 2018 19:50:50 +0100
Subject: [pypy-dev] PyPy sprint in Poland
In-Reply-To: <CAK5idxThBiakN7nu+Jka3mpBdJh8eJXXbf7wuDcZnmXvRaD8uQ@mail.gmail.com>
References: <CAK5idxThBiakN7nu+Jka3mpBdJh8eJXXbf7wuDcZnmXvRaD8uQ@mail.gmail.com>
Message-ID: <2debe6fff7bc6364f1f7bc0d3a87d7f6@manueljacob.de>

I enjoyed the Warsaw sprint a lot, especially the food options, but I 
guess that the other options would be nice as well.

Shortly after the Leysin sprint isn't optimal, but okay for me.  
However, I don't have time around May 1.

On 2018-01-07 10:26, Maciej Fijalkowski wrote:
> Hi Everyone.
> 
> It looks like I would be in Europe through April and maybe May. Anyone
> fancy a sprint somewhere in Poland? Potential venues:
> 
> * we can have a sprint at my climbing spot - it's quite a problem to
> get to (~2-3h by train from either Prague or Wroclaw), but it's
> incredibly lovely at this time of the year. There is a venue and
> internet that we can use. Limited restaurant options are a con.
> Endless hiking options are a plus.
> 
> * we can try to organize a venue in Warsaw. We don't have a place just
> yet, but it's easy to get to, relatively cheap, abundant in places to
> eat out. We don't have a place to sprint at just yet, but maybe we can
> organize something at the Uni.
> 
> * Krakow. Might be easier to organize venue. Slightly harder to get to
> than Warsaw, quite a bit nicer.
> 
> * Wroclaw. We had a tiny sprint there with Armin when we merged the
> JIT to pypy :-) A bit harder to get to than Warsaw, a very nice city,
> we don't have a venue organized yet (but it's possible I think),
> easier to do a few days excursion from there.
> 
> Thoughts?
> fijal
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From matti at picus.org.il  Thu Jan 11 15:57:11 2018
From: matti at picus.org.il (Matti Picus)
Date: Thu, 11 Jan 2018 22:57:11 +0200
Subject: [pypy-dev] pypy 5.10.1 release candidates are up,
 please try them out
Message-ID: <5846e230-c226-752a-38c2-891e20afc4ae@picus.org.il>

I have uploaded pypy3.5 v5.10.1 bug-fix release candidates to 
https://bitbucket.org/pypy/pypy/downloads please test them out.

The sha256 hashes are available at the end of the source for the website 
https://bitbucket.org/pypy/pypy.org/src/extradoc/source/download.txt?at=extradoc

Any comments/corrections to the release notice would also be appreciated
http://doc.pypy.org/en/latest/release-v5.10.1.html

Matti

From ndbecker2 at gmail.com  Fri Jan 12 07:33:06 2018
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 12 Jan 2018 07:33:06 -0500
Subject: [pypy-dev] pypy 5.10.1 release candidates are up,
 please try them out
References: <5846e230-c226-752a-38c2-891e20afc4ae@picus.org.il>
Message-ID: <p3a9q7$opi$1@blaine.gmane.org>

Matti Picus wrote:

> I have uploaded pypy3.5 v5.10.1 bug-fix release candidates to
> https://bitbucket.org/pypy/pypy/downloads please test them out.
> 
> The sha256 hashes are available at the end of the source for the website
> 
https://bitbucket.org/pypy/pypy.org/src/extradoc/source/download.txt?at=extradoc
> 
> Any comments/corrections to the release notice would also be appreciated
> http://doc.pypy.org/en/latest/release-v5.10.1.html
> 
> Matti

Haven't tried pypy for some time, but just tried it on fedora 27.

pypy
pypy: error while loading shared libraries: libbz2.so.1.0: cannot open 
shared object file: No such file or directory

I d/l pypy3-v5.10.1-linux64.tar.bz2 and installed locally.


From matti.picus at gmail.com  Fri Jan 12 08:44:12 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Fri, 12 Jan 2018 15:44:12 +0200
Subject: [pypy-dev] pypy 5.10.1 release candidates are up,
 please try them out
In-Reply-To: <p3a9q7$opi$1@blaine.gmane.org>
References: <5846e230-c226-752a-38c2-891e20afc4ae@picus.org.il>
 <p3a9q7$opi$1@blaine.gmane.org>
Message-ID: <47d5009a-0fff-981a-513d-b7117876da56@gmail.com>


On 12/01/18 14:33, Neal Becker wrote:
> Haven't tried pypy for some time, but just tried it on fedora 27.
> pypy
> pypy: error while loading shared libraries: libbz2.so.1.0: cannot open
> shared object file: No such file or directory
>
> I d/l pypy3-v5.10.1-linux64.tar.bz2 and installed locally.
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
Thanks for the report. There is a libbz2 package available on rpm.

On python3, the _bz2 module depends on this library. What happens when 
you do

python3 -c"import _bz2; print(_bz2)"
and then check the dependencies of the `_bz2.**.so` with ldd ?
On Ubuntu it also depends on libbz2.so
Matti

From ndbecker2 at gmail.com  Fri Jan 12 14:11:51 2018
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 12 Jan 2018 14:11:51 -0500
Subject: [pypy-dev] pypy 5.10.1 release candidates are up,
 please try them out
References: <5846e230-c226-752a-38c2-891e20afc4ae@picus.org.il>
 <p3a9q7$opi$1@blaine.gmane.org>
 <47d5009a-0fff-981a-513d-b7117876da56@gmail.com>
Message-ID: <p3b15q$khv$1@blaine.gmane.org>

Matti Picus wrote:

> 
> 
> On 12/01/18 14:33, Neal Becker wrote:
>> Haven't tried pypy for some time, but just tried it on fedora 27.
>> pypy
>> pypy: error while loading shared libraries: libbz2.so.1.0: cannot open
>> shared object file: No such file or directory
>>
>> I d/l pypy3-v5.10.1-linux64.tar.bz2 and installed locally.
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
> Thanks for the report. There is a libbz2 package available on rpm.
> 
> On python3, the _bz2 module depends on this library. What happens when
> you do
> 
> python3 -c"import _bz2; print(_bz2)"
> and then check the dependencies of the `_bz2.**.so` with ldd ?
> On Ubuntu it also depends on libbz2.so
> Matti
 python3 -c"import _bz2; print(_bz2)"
<module '_bz2' from '/usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-
x86_64-linux-gnu.so'>

$ ldd /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007ffdfbfaf000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f3ed6099000)


From ndbecker2 at gmail.com  Fri Jan 12 14:35:44 2018
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 12 Jan 2018 14:35:44 -0500
Subject: [pypy-dev] pypy 5.10.1 release candidates are up,
 please try them out
References: <5846e230-c226-752a-38c2-891e20afc4ae@picus.org.il>
 <p3a9q7$opi$1@blaine.gmane.org>
 <47d5009a-0fff-981a-513d-b7117876da56@gmail.com>
 <p3b15q$khv$1@blaine.gmane.org>
Message-ID: <p3b2ik$drm$1@blaine.gmane.org>

Neal Becker wrote:

> Matti Picus wrote:
> 
>> 
>> 
>> On 12/01/18 14:33, Neal Becker wrote:
>>> Haven't tried pypy for some time, but just tried it on fedora 27.
>>> pypy
>>> pypy: error while loading shared libraries: libbz2.so.1.0: cannot open
>>> shared object file: No such file or directory
>>>
>>> I d/l pypy3-v5.10.1-linux64.tar.bz2 and installed locally.
>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>> Thanks for the report. There is a libbz2 package available on rpm.
>> 
>> On python3, the _bz2 module depends on this library. What happens when
>> you do
>> 
>> python3 -c"import _bz2; print(_bz2)"
>> and then check the dependencies of the `_bz2.**.so` with ldd ?
>> On Ubuntu it also depends on libbz2.so
>> Matti
>  python3 -c"import _bz2; print(_bz2)"
> <module '_bz2' from '/usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-
> x86_64-linux-gnu.so'>
> 
> $ ldd
> /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so
> linux-vdso.so.1 (0x00007ffdfbfaf000) libbz2.so.1 => /lib64/libbz2.so.1
> (0x00007f3ed6099000)
 locate libbz2.so
/usr/lib/libbz2.so.1
/usr/lib/libbz2.so.1.0.6
/usr/lib64/libbz2.so
/usr/lib64/libbz2.so.1
/usr/lib64/libbz2.so.1.0.6

So Fedora has libbz2.so.1 and libbz2.so.1.0.6, but no libbz2.so.1.0.  In 
fact, isn't depending on libbz2.so.1.0 an error?


From armin.rigo at gmail.com  Fri Jan 12 17:03:18 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Fri, 12 Jan 2018 23:03:18 +0100
Subject: [pypy-dev] pypy 5.10.1 release candidates are up,
 please try them out
In-Reply-To: <p3b2ik$drm$1@blaine.gmane.org>
References: <5846e230-c226-752a-38c2-891e20afc4ae@picus.org.il>
 <p3a9q7$opi$1@blaine.gmane.org>
 <47d5009a-0fff-981a-513d-b7117876da56@gmail.com>
 <p3b15q$khv$1@blaine.gmane.org> <p3b2ik$drm$1@blaine.gmane.org>
Message-ID: <CAMSv6X09qVmfGTKSkq1X0X-Rkf8wZMt5_dr6jhpK8qBMM_ykRw@mail.gmail.com>

Hi Neal,

On 12 January 2018 at 20:35, Neal Becker <ndbecker2 at gmail.com> wrote:
> So Fedora has libbz2.so.1 and libbz2.so.1.0.6, but no libbz2.so.1.0.  In
> fact, isn't depending on libbz2.so.1.0 an error?

Unless I'm mistaken, it's what we get from ``gcc -lbz2`` on Ubuntu.
If you think doing so gives a dependency that is erroneous, then the
problem is either inside gcc or Ubuntu, not PyPy.

In general, taking a binary compiled for Linux distribution X and
trying to use it in Linux distribution Y only works if the stars are
correctly aligned.  This is known and documented in
http://pypy.org/download.html#linux-binaries-and-common-distributions


A bient?t,

Armin.

From matti.picus at gmail.com  Sat Jan 13 12:54:28 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 13 Jan 2018 19:54:28 +0200
Subject: [pypy-dev] pypy 5.10.1 release candidates are up,
 please try them out
In-Reply-To: <CAMSv6X09qVmfGTKSkq1X0X-Rkf8wZMt5_dr6jhpK8qBMM_ykRw@mail.gmail.com>
References: <5846e230-c226-752a-38c2-891e20afc4ae@picus.org.il>
 <p3a9q7$opi$1@blaine.gmane.org>
 <47d5009a-0fff-981a-513d-b7117876da56@gmail.com>
 <p3b15q$khv$1@blaine.gmane.org> <p3b2ik$drm$1@blaine.gmane.org>
 <CAMSv6X09qVmfGTKSkq1X0X-Rkf8wZMt5_dr6jhpK8qBMM_ykRw@mail.gmail.com>
Message-ID: <7a16d43b-3d85-9945-87d5-a4c0eb97d21a@gmail.com>

On 13/01/18 00:03, Armin Rigo wrote:
> Hi Neal,
>
> On 12 January 2018 at 20:35, Neal Becker <ndbecker2 at gmail.com> wrote:
>> So Fedora has libbz2.so.1 and libbz2.so.1.0.6, but no libbz2.so.1.0.  In
>> fact, isn't depending on libbz2.so.1.0 an error?
> Unless I'm mistaken, it's what we get from ``gcc -lbz2`` on Ubuntu.
> If you think doing so gives a dependency that is erroneous, then the
> problem is either inside gcc or Ubuntu, not PyPy.
>
> In general, taking a binary compiled for Linux distribution X and
> trying to use it in Linux distribution Y only works if the stars are
> correctly aligned.  This is known and documented in
> http://pypy.org/download.html#linux-binaries-and-common-distributions
>
>
> A bient?t,
>
> Armin.
>
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=561175,
confirming what Armin said - this is a Debian/Ubuntu issue, and AFAICT 
only should affect libbz2.
The "solution" for the RPM-based users would be to manually create the 
symbolic link, or to use the portable builds.
Matti


From pengyu.ut at gmail.com  Thu Jan 18 12:59:09 2018
From: pengyu.ut at gmail.com (Peng Yu)
Date: Thu, 18 Jan 2018 11:59:09 -0600
Subject: [pypy-dev] How to install pip for pypy along with the pip for
 python?
Message-ID: <CABrM6wmfedspWr55C=C6SpEtf5gdv8jCLQ6fHFLCHYwsMekt-Q@mail.gmail.com>

Hi,

I installed pip with python without using virtual environment.

https://pip.pypa.io/en/stable/installing/

I also need to install pip with pypy without using virtual
environment. Is it possible? Could anybody show me how to do it?
Thanks.

-- 
Regards,
Peng

From matti.picus at gmail.com  Thu Jan 18 13:02:40 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Thu, 18 Jan 2018 20:02:40 +0200
Subject: [pypy-dev] How to install pip for pypy along with the pip for
 python?
In-Reply-To: <CABrM6wmfedspWr55C=C6SpEtf5gdv8jCLQ6fHFLCHYwsMekt-Q@mail.gmail.com>
References: <CABrM6wmfedspWr55C=C6SpEtf5gdv8jCLQ6fHFLCHYwsMekt-Q@mail.gmail.com>
Message-ID: <01d06ff4-6681-542d-708c-068de1f09d8b@gmail.com>

On 18/01/18 19:59, Peng Yu wrote:
> Hi,
>
> I installed pip with python without using virtual environment.
>
> https://pip.pypa.io/en/stable/installing/
>
> I also need to install pip with pypy without using virtual
> environment. Is it possible? Could anybody show me how to do it?
> Thanks.
>
pypy -m ensurepip

Matti

From pengyu.ut at gmail.com  Thu Jan 18 13:28:03 2018
From: pengyu.ut at gmail.com (Peng Yu)
Date: Thu, 18 Jan 2018 12:28:03 -0600
Subject: [pypy-dev] How to install pip for pypy along with the pip for
 python?
In-Reply-To: <01d06ff4-6681-542d-708c-068de1f09d8b@gmail.com>
References: <CABrM6wmfedspWr55C=C6SpEtf5gdv8jCLQ6fHFLCHYwsMekt-Q@mail.gmail.com>
 <01d06ff4-6681-542d-708c-068de1f09d8b@gmail.com>
Message-ID: <CABrM6w=zpNzp3MK+TOAa77mq4GB1thsp2tzPX3pN8WYX2dCpaw@mail.gmail.com>

Hi, I got this. Is there anything wrong? Also, how to install a python
module with pypy pip? Thanks.

$ pypy -m ensurepip
/var/folders/r7/bvmh1vvx41d63snvgbdz7bl40000gr/T/tmpelZs6W/pip-6.1.1-py2.py3-none-any.whl/pip/_vendor/__init__.py:72:
UserWarning: Module pip was already imported from
/var/folders/r7/bvmh1vvx41d63snvgbdz7bl40000gr/T/tmpelZs6W/pip-6.1.1-py2.py3-none-any.whl/pip/__init__.py,
but /usr/local/lib/pypy/site-packages/pip-8.1.2-py2.7.egg is being
added to sys.path
Ignoring indexes: https://pypi.python.org/simple
Requirement already satisfied (use --upgrade to upgrade): setuptools
in /usr/local/lib/pypy/site-packages/setuptools-26.1.1-py2.7.egg
Requirement already satisfied (use --upgrade to upgrade): pip in
/usr/local/lib/pypy/site-packages/pip-8.1.2-py2.7.egg

On Thu, Jan 18, 2018 at 12:02 PM, Matti Picus <matti.picus at gmail.com> wrote:
> On 18/01/18 19:59, Peng Yu wrote:
>>
>> Hi,
>>
>> I installed pip with python without using virtual environment.
>>
>> https://pip.pypa.io/en/stable/installing/
>>
>> I also need to install pip with pypy without using virtual
>> environment. Is it possible? Could anybody show me how to do it?
>> Thanks.
>>
> pypy -m ensurepip
>
> Matti



-- 
Regards,
Peng

From pengyu.ut at gmail.com  Fri Jan 19 10:49:49 2018
From: pengyu.ut at gmail.com (Peng Yu)
Date: Fri, 19 Jan 2018 09:49:49 -0600
Subject: [pypy-dev] How to install pip for pypy along with the pip for
 python?
In-Reply-To: <c4ca770e-9df8-dee1-0075-4a8897b02769@gmail.com>
References: <CABrM6wmfedspWr55C=C6SpEtf5gdv8jCLQ6fHFLCHYwsMekt-Q@mail.gmail.com>
 <01d06ff4-6681-542d-708c-068de1f09d8b@gmail.com>
 <CABrM6w=zpNzp3MK+TOAa77mq4GB1thsp2tzPX3pN8WYX2dCpaw@mail.gmail.com>
 <8e0bf24a-c4b6-8226-3ab7-eefc5fdf7f1d@gmail.com>
 <CABrM6wnovPtn8V+eve30zKEsnac_oqz8MU4ei+tww5D6g6cv1g@mail.gmail.com>
 <c4ca770e-9df8-dee1-0075-4a8897b02769@gmail.com>
Message-ID: <CABrM6w=OkbK=ez6jkSEyQ3z_JuK7rToBm-Gvv8Szv04dMP3AiA@mail.gmail.com>

Yes. It works. Thanks.

On Fri, Jan 19, 2018 at 2:54 AM, Matti Picus <matti.picus at gmail.com> wrote:
> Does "pypy -mpip install ... " work?
> Matti
>
>
> On 19/01/18 00:55, Peng Yu wrote:
>
> Both pypy and python are installed with homebrew on Mac OS X.
>
> On Thu, Jan 18, 2018 at 3:16 PM, Matti Picus <matti.picus at gmail.com> wrote:
>
> How did you install pypy?
> Matti
>
> On 18/01/18 20:28, Peng Yu wrote:
>
> Hi, I got this. Is there anything wrong? Also, how to install a python
> module with pypy pip? Thanks.
>
> $ pypy -m ensurepip
> /var/folders/r7/bvmh1vvx41d63snvgbdz7bl40000gr/T/tmpelZs6W/pip-6.1.1-py2.py3-none-any.whl/pip/_vendor/__init__.py:72:
> UserWarning: Module pip was already imported from
> /var/folders/r7/bvmh1vvx41d63snvgbdz7bl40000gr/T/tmpelZs6W/pip-6.1.1-py2.py3-none-any.whl/pip/__init__.py,
> but /usr/local/lib/pypy/site-packages/pip-8.1.2-py2.7.egg is being
> added to sys.path
> Ignoring indexes: https://pypi.python.org/simple
> Requirement already satisfied (use --upgrade to upgrade): setuptools
> in /usr/local/lib/pypy/site-packages/setuptools-26.1.1-py2.7.egg
> Requirement already satisfied (use --upgrade to upgrade): pip in
> /usr/local/lib/pypy/site-packages/pip-8.1.2-py2.7.egg
>
> On Thu, Jan 18, 2018 at 12:02 PM, Matti Picus <matti.picus at gmail.com> wrote:
>
> On 18/01/18 19:59, Peng Yu wrote:
>
> Hi,
>
> I installed pip with python without using virtual environment.
>
> https://pip.pypa.io/en/stable/installing/
>
> I also need to install pip with pypy without using virtual
> environment. Is it possible? Could anybody show me how to do it?
> Thanks.
>
> pypy -m ensurepip
>
> Matti
>
>
>
>
>
>



-- 
Regards,
Peng

From tkadm30 at yandex.com  Sun Jan 28 04:22:37 2018
From: tkadm30 at yandex.com (Etienne Robillard)
Date: Sun, 28 Jan 2018 04:22:37 -0500
Subject: [pypy-dev] How to implement a JIT compiler for Django ?
Message-ID: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>

Hi,

Is it possible to use the PyPy JIT compiler embedded into a CPython 
(Django) application?

I would like to translate a Django app into C code then compile the 
binary with clang to optimize the code with a JIT engine.

What do you think?

Etienne

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/


From yury at shurup.com  Sun Jan 28 05:11:05 2018
From: yury at shurup.com (Yury V. Zaytsev)
Date: Sun, 28 Jan 2018 11:11:05 +0100 (CET)
Subject: [pypy-dev] How to implement a JIT compiler for Django ?
In-Reply-To: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
References: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
Message-ID: <alpine.LRH.2.21.1801281044060.1711@vps.zaytsev.net>

On Sun, 28 Jan 2018, Etienne Robillard wrote:

> Is it possible to use the PyPy JIT compiler embedded into a CPython 
> (Django) application?
>
> I would like to translate a Django app into C code then compile the 
> binary with clang to optimize the code with a JIT engine.
>
> What do you think?

I think that you might be confused about the fundamentals of the 
technologies involved here.

Once you translate a Django app into C code (let's assume this is actually 
possible for the sake of the argument) and then compile it into machine 
code using clang there is nothing more left for a JIT to operate upon, 
because machine code is interpreted directly by the CPU.

Tracing JIT engines like PyPy translate bytecode into machine code on the 
fly taking into account invariants discovered during runtime, which 
theoretically enables them to outperform machine code generated without 
knowing the data it processes, and, in any case, run a lot faster than the 
interpreted byte code.

A possible source of confusion is that people often speak of speeding 
things up with LLVM (or nowadays even GCC) JIT; in most cases this amounts 
to so called method level JITs where specific isolated functions are 
compiled on the fly into machine code by the corresponding JIT backend and 
then called from the bytecode interpreter instead of actually interpreting 
the original bytecode for the method.

Anyways, having that said, I can't even infer what your original line of 
thinking was to embed what into what to speed up what exactly...

-- 
Sincerely yours,
Yury V. Zaytsev

From tkadm30 at yandex.com  Sun Jan 28 06:52:50 2018
From: tkadm30 at yandex.com (Etienne Robillard)
Date: Sun, 28 Jan 2018 06:52:50 -0500
Subject: [pypy-dev] How to implement a JIT compiler for Django ?
In-Reply-To: <alpine.LRH.2.21.1801281044060.1711@vps.zaytsev.net>
References: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
 <alpine.LRH.2.21.1801281044060.1711@vps.zaytsev.net>
Message-ID: <ae3dfc98-b45a-d81f-7428-25ba6c93ba3d@yandex.com>



Le 2018-01-28 ? 05:11, Yury V. Zaytsev a ?crit?:
>
> I think that you might be confused about the fundamentals of the 
> technologies involved here.
>

Yes, I admit, i'm really just starting to understand PyPy fundamentals 
and LLVM.

> Once you translate a Django app into C code (let's assume this is 
> actually possible for the sake of the argument) and then compile it 
> into machine code using clang there is nothing more left for a JIT to 
> operate upon, because machine code is interpreted directly by the CPU.
>

I'm really sure its possible to generate a C or C++ file from 
human-generated Python code. So far, I want to use the LLVM backend 
(PyPy) to translate CPython classes into a tracing JIT compiler...

> Tracing JIT engines like PyPy translate bytecode into machine code on 
> the fly taking into account invariants discovered during runtime, 
> which theoretically enables them to outperform machine code generated 
> without knowing the data it processes, and, in any case, run a lot 
> faster than the interpreted byte code.

I'm positive you can use PyPy in embedded C/C++ applications to enable 
trace compilation of Python objects.
>
> A possible source of confusion is that people often speak of speeding 
> things up with LLVM (or nowadays even GCC) JIT; in most cases this 
> amounts to so called method level JITs where specific isolated 
> functions are compiled on the fly into machine code by the 
> corresponding JIT backend and then called from the bytecode 
> interpreter instead of actually interpreting the original bytecode for 
> the method.
>
JIT is cool because it can theoretically makes Django and Python web 
apps outperform C applications.

> Anyways, having that said, I can't even infer what your original line 
> of thinking was to embed what into what to speed up what exactly...
>
I'm looking to use JIT as a replacement for Cython in a upcoming 
Django-hotsauce release. :)

Cheers,

Etienne

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/


From yury at shurup.com  Sun Jan 28 07:35:02 2018
From: yury at shurup.com (Yury V. Zaytsev)
Date: Sun, 28 Jan 2018 13:35:02 +0100 (CET)
Subject: [pypy-dev] How to implement a JIT compiler for Django ?
In-Reply-To: <ae3dfc98-b45a-d81f-7428-25ba6c93ba3d@yandex.com>
References: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
 <alpine.LRH.2.21.1801281044060.1711@vps.zaytsev.net>
 <ae3dfc98-b45a-d81f-7428-25ba6c93ba3d@yandex.com>
Message-ID: <alpine.LRH.2.21.1801281319570.1711@vps.zaytsev.net>

On Sun, 28 Jan 2018, Etienne Robillard wrote:

>> Anyways, having that said, I can't even infer what your original line 
>> of thinking was to embed what into what to speed up what exactly...
>> 
> I'm looking to use JIT as a replacement for Cython in a upcoming 
> Django-hotsauce release. :)

If this is indeed your goal, then I do have some good news for you: you 
don't have to do anything at all other than scraping Cython in order to 
achieve it. Just run your code on PyPy instead of CPython and you will 
benefit directly from PyPy's tracing JIT.

To answer the question of why do you have to scrape Cython before it comes 
up: you *can* run Cython compiled modules on top of PyPy thanks to its 
Python C API emulation layer (CPyExt). However, this will be inefficient 
because (a) crossing PyPy / C boundary is slow, although not as slow as it 
used to be in the past and (b) PyPy's JIT can't see into machine code 
created from Cython-generated C/C++ source code. Therefore, to benefit 
most from PyPy you'd better feed it pure Python code.

>> Once you translate a Django app into C code (let's assume this is 
>> actually possible for the sake of the argument) and then compile it 
>> into machine code using clang there is nothing more left for a JIT to 
>> operate upon, because machine code is interpreted directly by the CPU.
>
> I'm really sure its possible to generate a C or C++ file from 
> human-generated Python code.

In that case, you have to educate yourself, and specifically try to 
understand how Cython really works. Hint: generating C/C++ file from 
Python code != translate Python code into C/C++ code that doesn't require 
CPython runtime.

> So far, I want to use the LLVM backend (PyPy)

PyPy doesn't have anything to do with LLVM at this point. There have been 
multiple attempts in the past to use LLVM as backend for PyPy, but so far 
none of them have really been succesful, where success is defined as 
making it a default backend.

> to translate CPython classes into a tracing JIT compiler...

This statement is devoid of meaning.

> I'm positive you can use PyPy in embedded C/C++ applications to enable 
> trace compilation of Python objects.
>
> JIT is cool because it can theoretically makes Django and Python web 
> apps outperform C applications.

These statements are both correct, but see the beginning of this email.

-- 
Sincerely yours,
Yury V. Zaytsev

From lists at sonnenglanz.net  Sun Jan 28 07:58:07 2018
From: lists at sonnenglanz.net (Pim van der Eijk (Lists))
Date: Sun, 28 Jan 2018 13:58:07 +0100
Subject: [pypy-dev] How to implement a JIT compiler for Django ?
In-Reply-To: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
References: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
Message-ID: <fda9967d-1963-a937-7eaf-2c959961a76e@sonnenglanz.net>



On 28-01-18 10:22, Etienne Robillard wrote:

> Is it possible to use the PyPy JIT compiler embedded into a CPython 
> (Django) application?
>

You should be aware that, while PyPy speeds up many Python applications 
substantially, at least in the past,? Django was not one of them.



From tkadm30 at yandex.com  Sun Jan 28 08:05:07 2018
From: tkadm30 at yandex.com (Etienne Robillard)
Date: Sun, 28 Jan 2018 08:05:07 -0500
Subject: [pypy-dev] How to implement a JIT compiler for Django ?
In-Reply-To: <alpine.LRH.2.21.1801281319570.1711@vps.zaytsev.net>
References: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
 <alpine.LRH.2.21.1801281044060.1711@vps.zaytsev.net>
 <ae3dfc98-b45a-d81f-7428-25ba6c93ba3d@yandex.com>
 <alpine.LRH.2.21.1801281319570.1711@vps.zaytsev.net>
Message-ID: <2066b470-6e19-64ac-b07e-40bc829c117d@yandex.com>

Great! Thanks very much for this post, Yury.

I'll do just what you suggested. :)

Cheers,

Etienne


Le 2018-01-28 ? 07:35, Yury V. Zaytsev a ?crit?:
> On Sun, 28 Jan 2018, Etienne Robillard wrote:
>
>>> Anyways, having that said, I can't even infer what your original 
>>> line of thinking was to embed what into what to speed up what 
>>> exactly...
>>>
>> I'm looking to use JIT as a replacement for Cython in a upcoming 
>> Django-hotsauce release. :)
>
> If this is indeed your goal, then I do have some good news for you: 
> you don't have to do anything at all other than scraping Cython in 
> order to achieve it. Just run your code on PyPy instead of CPython and 
> you will benefit directly from PyPy's tracing JIT.
>
> To answer the question of why do you have to scrape Cython before it 
> comes up: you *can* run Cython compiled modules on top of PyPy thanks 
> to its Python C API emulation layer (CPyExt). However, this will be 
> inefficient because (a) crossing PyPy / C boundary is slow, although 
> not as slow as it used to be in the past and (b) PyPy's JIT can't see 
> into machine code created from Cython-generated C/C++ source code. 
> Therefore, to benefit most from PyPy you'd better feed it pure Python 
> code.
>
>>> Once you translate a Django app into C code (let's assume this is 
>>> actually possible for the sake of the argument) and then compile it 
>>> into machine code using clang there is nothing more left for a JIT 
>>> to operate upon, because machine code is interpreted directly by the 
>>> CPU.
>>
>> I'm really sure its possible to generate a C or C++ file from 
>> human-generated Python code.
>
> In that case, you have to educate yourself, and specifically try to 
> understand how Cython really works. Hint: generating C/C++ file from 
> Python code != translate Python code into C/C++ code that doesn't 
> require CPython runtime.
>
>> So far, I want to use the LLVM backend (PyPy)
>
> PyPy doesn't have anything to do with LLVM at this point. There have 
> been multiple attempts in the past to use LLVM as backend for PyPy, 
> but so far none of them have really been succesful, where success is 
> defined as making it a default backend.
>
>> to translate CPython classes into a tracing JIT compiler...
>
> This statement is devoid of meaning.
>
>> I'm positive you can use PyPy in embedded C/C++ applications to 
>> enable trace compilation of Python objects.
>>
>> JIT is cool because it can theoretically makes Django and Python web 
>> apps outperform C applications.
>
> These statements are both correct, but see the beginning of this email.
>

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/


From tkadm30 at yandex.com  Sun Jan 28 08:24:20 2018
From: tkadm30 at yandex.com (Etienne Robillard)
Date: Sun, 28 Jan 2018 08:24:20 -0500
Subject: [pypy-dev] How to implement a JIT compiler for Django ?
In-Reply-To: <fda9967d-1963-a937-7eaf-2c959961a76e@sonnenglanz.net>
References: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
 <fda9967d-1963-a937-7eaf-2c959961a76e@sonnenglanz.net>
Message-ID: <e53bc000-27af-7b3d-dfec-08138832a393@yandex.com>

Hi Pim,


Le 2018-01-28 ? 07:58, Pim van der Eijk (Lists) a ?crit?:
>
>
> On 28-01-18 10:22, Etienne Robillard wrote:
>
>> Is it possible to use the PyPy JIT compiler embedded into a CPython 
>> (Django) application?
>>
>
> You should be aware that, while PyPy speeds up many Python 
> applications substantially, at least in the past,? Django was not one 
> of them.
>
I'm planning to keep Cython support in the stable branch and experiment 
with PyPy and clang in the development branch. I believe using a tracing 
JIT compiler should be a radical improvement over Cython, thanks to PyPy.

Cheers,

Etienne

>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/


From yury at shurup.com  Sun Jan 28 08:36:05 2018
From: yury at shurup.com (Yury V. Zaytsev)
Date: Sun, 28 Jan 2018 14:36:05 +0100 (CET)
Subject: [pypy-dev] How to implement a JIT compiler for Django ?
In-Reply-To: <fda9967d-1963-a937-7eaf-2c959961a76e@sonnenglanz.net>
References: <5c3af24c-5db5-602c-da2d-7278d36786eb@yandex.com>
 <fda9967d-1963-a937-7eaf-2c959961a76e@sonnenglanz.net>
Message-ID: <alpine.LRH.2.21.1801281432130.1711@vps.zaytsev.net>

On Sun, 28 Jan 2018, Pim van der Eijk (Lists) wrote:

> On 28-01-18 10:22, Etienne Robillard wrote:
>
>> Is it possible to use the PyPy JIT compiler embedded into a CPython 
>> (Django) application?
>
> You should be aware that, while PyPy speeds up many Python applications 
> substantially, at least in the past,? Django was not one of them.

It really depends on the workload, from what I remember the stumbling 
block for getting substantial speedups for typical Django applications was 
the sad ORM story, also in part due to database drivers.

My understanding is that Etienne might not be using ORM at all, so it is 
possible that he'd directly get some decent performance improvements...

-- 
Sincerely yours,
Yury V. Zaytsev

From tinchester at gmail.com  Mon Jan 29 05:22:56 2018
From: tinchester at gmail.com (=?UTF-8?Q?Tin_Tvrtkovi=C4=87?=)
Date: Mon, 29 Jan 2018 10:22:56 +0000
Subject: [pypy-dev] Safety of replacing the instance dict
Message-ID: <CAEJ-mUq0ajdhv-yEWtnGmZJ+6D48GAJGnnJ2gdzS6hjvygsEEA@mail.gmail.com>

Hello,

this question is not directly related to PyPy but to general Python
semantics. I'm posting it here since I know there are very knowledgeable
people here and honestly this is the friendliest list for questions like
this.

I'm working on optimizing __init__s that the attrs library generates. My
question is, is there anything fundamentally wrong with having an __init__
like this:

def __init__(self, a, b, c):
    self.__dict__ = {'a': b, 'b': b, 'c': c}

(basically throwing away the existing instance dict and replacing it with a
fresh one). As far as I can see, this is actually faster on CPython if
there is more than one attribute, and about the same speed on PyPy
(benchmarking on PyPy is kinda hard, the difference is 0.2 ns vs 0.22 ns,
not sure 0.02 ns is a significant difference). And this approach works for
frozen classes (an attrs feature) too.

It's just that doing it this way is unconventional and a little scary.
Would we be violating a Python rule somewhere and making stuff blow up
later if we went this way?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180129/41361a5e/attachment.html>

From armin.rigo at gmail.com  Mon Jan 29 08:41:46 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 29 Jan 2018 14:41:46 +0100
Subject: [pypy-dev] Safety of replacing the instance dict
In-Reply-To: <CAEJ-mUq0ajdhv-yEWtnGmZJ+6D48GAJGnnJ2gdzS6hjvygsEEA@mail.gmail.com>
References: <CAEJ-mUq0ajdhv-yEWtnGmZJ+6D48GAJGnnJ2gdzS6hjvygsEEA@mail.gmail.com>
Message-ID: <CAMSv6X1Q_SgSE=t=LTw0Eoe7ROqf8Vr45grm=mZrpC-=Eu8JKw@mail.gmail.com>

Hi,

On 29 January 2018 at 11:22, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
> It's just that doing it this way is unconventional and a little scary. Would
> we be violating a Python rule somewhere and making stuff blow up later if we
> went this way?

No, it's semantically fine.  But it comes with a heavy penalty on
PyPy.  I guess you don't see it because you measured something tiny,
like creating the instance and then throwing it away---the JIT
optimizes that to nothing at all in both cases.  Not only is the
creation time larger, but attribute access is slower, and the memory
usage is larger.


A bient?t,

Armin.

From tinchester at gmail.com  Mon Jan 29 10:27:51 2018
From: tinchester at gmail.com (=?UTF-8?Q?Tin_Tvrtkovi=C4=87?=)
Date: Mon, 29 Jan 2018 15:27:51 +0000
Subject: [pypy-dev] Safety of replacing the instance dict
In-Reply-To: <CAMSv6X1Q_SgSE=t=LTw0Eoe7ROqf8Vr45grm=mZrpC-=Eu8JKw@mail.gmail.com>
References: <CAEJ-mUq0ajdhv-yEWtnGmZJ+6D48GAJGnnJ2gdzS6hjvygsEEA@mail.gmail.com>
 <CAMSv6X1Q_SgSE=t=LTw0Eoe7ROqf8Vr45grm=mZrpC-=Eu8JKw@mail.gmail.com>
Message-ID: <CAEJ-mUo7oKemM05rSp_uF01bDxK17qCbh8aaesEV2mSfPbppaQ@mail.gmail.com>

Thanks a lot, Armin. If we go this way, it's easy to special case PyPy.

On Mon, Jan 29, 2018 at 2:42 PM Armin Rigo <armin.rigo at gmail.com> wrote:

> Hi,
>
> On 29 January 2018 at 11:22, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
> > It's just that doing it this way is unconventional and a little scary.
> Would
> > we be violating a Python rule somewhere and making stuff blow up later
> if we
> > went this way?
>
> No, it's semantically fine.  But it comes with a heavy penalty on
> PyPy.  I guess you don't see it because you measured something tiny,
> like creating the instance and then throwing it away---the JIT
> optimizes that to nothing at all in both cases.  Not only is the
> creation time larger, but attribute access is slower, and the memory
> usage is larger.
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180129/f398e41b/attachment.html>

From flub at devork.be  Mon Jan 29 20:21:44 2018
From: flub at devork.be (Floris Bruynooghe)
Date: Mon, 29 Jan 2018 17:21:44 -0800
Subject: [pypy-dev] Joining the Leysin sprint
Message-ID: <CAAWk_DzT-phVM0Na8JcMrxdFnFOBdxMKN8JULB4QS+ZYeSt8Ng@mail.gmail.com>

Hi,

I would be interested in joining the Leysin sprint and possibly
contribute to either the cffi work and/or some of the testing work.  I
won''t be able to attend the entire sprint however am tentatively
thinking of the 1st half, so arriving on Sat 17 leaving on Wed 21
March.  Is this something which is possible?  I could probably be
flexible as well and attend the 2nd half of the week instead, starting
from Wed, if this is easier logistically.

Kind Regards,
Floris

From anto.cuni at gmail.com  Tue Jan 30 07:13:19 2018
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 30 Jan 2018 12:13:19 +0000
Subject: [pypy-dev] Safety of replacing the instance dict
In-Reply-To: <CAMSv6X1Q_SgSE=t=LTw0Eoe7ROqf8Vr45grm=mZrpC-=Eu8JKw@mail.gmail.com>
References: <CAEJ-mUq0ajdhv-yEWtnGmZJ+6D48GAJGnnJ2gdzS6hjvygsEEA@mail.gmail.com>
 <CAMSv6X1Q_SgSE=t=LTw0Eoe7ROqf8Vr45grm=mZrpC-=Eu8JKw@mail.gmail.com>
Message-ID: <CAHPg78wcT0OJNR4T-hhhmVRtyKgev7B2jgKw1Ga8ve33TWrRBA@mail.gmail.com>

sidenote: if you do the following, you can replace the __dict__ without
incurring into performance penalties (Armin, please correct me if I'm
wrong):

import __pypy__
def __init__(self):
    self.__dict__ = __pypy__.newdict('instance')

this is not directly useful for your use case (because newdict() always
return an empty dict), but it might be useful to know in general

On Mon, Jan 29, 2018 at 1:41 PM, Armin Rigo <armin.rigo at gmail.com> wrote:

> Hi,
>
> On 29 January 2018 at 11:22, Tin Tvrtkovi? <tinchester at gmail.com> wrote:
> > It's just that doing it this way is unconventional and a little scary.
> Would
> > we be violating a Python rule somewhere and making stuff blow up later
> if we
> > went this way?
>
> No, it's semantically fine.  But it comes with a heavy penalty on
> PyPy.  I guess you don't see it because you measured something tiny,
> like creating the instance and then throwing it away---the JIT
> optimizes that to nothing at all in both cases.  Not only is the
> creation time larger, but attribute access is slower, and the memory
> usage is larger.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180130/455c1cf9/attachment.html>

From matti.picus at gmail.com  Tue Jan 30 15:37:52 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 30 Jan 2018 22:37:52 +0200
Subject: [pypy-dev] adding a pypy runner for speed.python.org
Message-ID: <3296f52d-dd6b-1e69-2c4a-b7d4e3216bf2@gmail.com>

I am cross-posting to both speed and pypy-dev to ask what needs to be 
done to get pypy2 and pypy3 benchmark runners onto speed.python.org
I am willing to be the contact person from the PyPy side, who do we need 
to talk to on the speed maintainers' side?
Instead of spamming the lists again, we could discuss this off-line, and 
report back with a plan, required resources, and a timeline.

Thanks,
Matti Picus

From armin.rigo at gmail.com  Tue Jan 30 15:37:34 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Tue, 30 Jan 2018 21:37:34 +0100
Subject: [pypy-dev] Safety of replacing the instance dict
In-Reply-To: <CAHPg78wcT0OJNR4T-hhhmVRtyKgev7B2jgKw1Ga8ve33TWrRBA@mail.gmail.com>
References: <CAEJ-mUq0ajdhv-yEWtnGmZJ+6D48GAJGnnJ2gdzS6hjvygsEEA@mail.gmail.com>
 <CAMSv6X1Q_SgSE=t=LTw0Eoe7ROqf8Vr45grm=mZrpC-=Eu8JKw@mail.gmail.com>
 <CAHPg78wcT0OJNR4T-hhhmVRtyKgev7B2jgKw1Ga8ve33TWrRBA@mail.gmail.com>
Message-ID: <CAMSv6X1GzHAfbjYqkAGTbuEMrxBrPkKLEkjjbL776r31s1ZRPA@mail.gmail.com>

Hi,

On 30 January 2018 at 13:13, Antonio Cuni <anto.cuni at gmail.com> wrote:
> sidenote: if you do the following, you can replace the __dict__ without
> incurring into performance penalties (Armin, please correct me if I'm
> wrong):
>
> import __pypy__
> def __init__(self):
>     self.__dict__ = __pypy__.newdict('instance')

This is a no-op, which takes time to execute but indeed doesn't seem
to disable the optimization.  However I don't see the point.  You may
as well write it like that:

   def __init__(self, x, y, z):
        d = self.__dict__
        d['x'] = x
        d['y'] = y
        d['z'] = z

It is a bit slower than "self.x = x; self.y = y; self.z = z" in PyPy
but doesn't throw away the optimization.  On CPython it is probably
faster.


Armin

From armin.rigo at gmail.com  Tue Jan 30 17:07:46 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Tue, 30 Jan 2018 23:07:46 +0100
Subject: [pypy-dev] Joining the Leysin sprint
In-Reply-To: <CAAWk_DzT-phVM0Na8JcMrxdFnFOBdxMKN8JULB4QS+ZYeSt8Ng@mail.gmail.com>
References: <CAAWk_DzT-phVM0Na8JcMrxdFnFOBdxMKN8JULB4QS+ZYeSt8Ng@mail.gmail.com>
Message-ID: <CAMSv6X18V=3-fS78Hese6BJQt869F0gtC9hU5qu4iEyAZE_zqg@mail.gmail.com>

Hi Floris,

On 30 January 2018 at 02:21, Floris Bruynooghe <flub at devork.be> wrote:
> I would be interested in joining the Leysin sprint and possibly
> contribute to either the cffi work and/or some of the testing work.  I
> won''t be able to attend the entire sprint however am tentatively
> thinking of the 1st half, so arriving on Sat 17 leaving on Wed 21
> March.  Is this something which is possible?  I could probably be
> flexible as well and attend the 2nd half of the week instead, starting
> from Wed, if this is easier logistically.

Welcome !  Either half of the week would work, logistically.  There
will be two more people during the first half, which makes it probably
more interesting to be in :-)

I've written down your name for the 17-21 of March.


A bient?t,

Armin.

From anto.cuni at gmail.com  Wed Jan 31 07:16:15 2018
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 31 Jan 2018 12:16:15 +0000
Subject: [pypy-dev] [pypy-commit] pypy cpyext-faster-arg-passing:
 document branch
In-Reply-To: <5a7073b3.11a4df0a.80a9d.41b8@mx.google.com>
References: <5a7073b3.11a4df0a.80a9d.41b8@mx.google.com>
Message-ID: <CAHPg78xvpvZDp7JzVRzJcTNnwcn1Yg-Exn=LYepxmJ1zjOqqzw@mail.gmail.com>

Hi Carl,
wow, this looks awesome. Did you run benchmarks to measure the speedup? If
yes, should we add them to my repo?
https://github.com/antocuni/cpyext-benchmarks

ciao,
Anto

On Tue, Jan 30, 2018 at 1:31 PM, cfbolz <pypy.commits at gmail.com> wrote:

> Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
> Branch: cpyext-faster-arg-passing
> Changeset: r93724:627a1425607c
> Date: 2018-01-30 14:30 +0100
> http://bitbucket.org/pypy/pypy/changeset/627a1425607c/
>
> Log:    document branch
>
> diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
> --- a/pypy/doc/whatsnew-head.rst
> +++ b/pypy/doc/whatsnew-head.rst
> @@ -23,3 +23,11 @@
>  added, then the performance using mapdict is linear in the number of
>  attributes. This is now fixed (by switching to a regular dict after 80
>  attributes).
> +
> +
> +.. branch: cpyext-faster-arg-passing
> +
> +When using cpyext, improve the speed of passing certain objects from PyPy
> to C
> +code, most notably None, True, False, types, all instances of C-defined
> types.
> +Before, a dict lookup was needed every time such an object crossed over,
> now it
> +is just a field read.
> _______________________________________________
> pypy-commit mailing list
> pypy-commit at python.org
> https://mail.python.org/mailman/listinfo/pypy-commit
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180131/d7f20d31/attachment.html>

From cfbolz at gmx.de  Wed Jan 31 07:34:40 2018
From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick)
Date: Wed, 31 Jan 2018 13:34:40 +0100
Subject: [pypy-dev] [pypy-commit] pypy cpyext-faster-arg-passing:
 document branch
In-Reply-To: <CAHPg78xvpvZDp7JzVRzJcTNnwcn1Yg-Exn=LYepxmJ1zjOqqzw@mail.gmail.com>
References: <5a7073b3.11a4df0a.80a9d.41b8@mx.google.com>
 <CAHPg78xvpvZDp7JzVRzJcTNnwcn1Yg-Exn=LYepxmJ1zjOqqzw@mail.gmail.com>
Message-ID: <DAC3F2E2-5467-42E2-ACBF-13FA3400CFE0@gmx.de>

Hi Anto,

Yes, I ran your benchmarks and they are improved, particularly the ones that pass arguments. I need to rerun them now that I merged default. However, I would prefer it if we could find a real life cpyext benchmark (maybe using numpy?).

Cheers,

Carl Friedrich 

Carl Friedrich

On January 31, 2018 1:16:15 PM GMT+01:00, Antonio Cuni <anto.cuni at gmail.com> wrote:
>Hi Carl,
>wow, this looks awesome. Did you run benchmarks to measure the speedup?
>If
>yes, should we add them to my repo?
>https://github.com/antocuni/cpyext-benchmarks
>
>ciao,
>Anto
>
>On Tue, Jan 30, 2018 at 1:31 PM, cfbolz <pypy.commits at gmail.com> wrote:
>
>> Author: Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>
>> Branch: cpyext-faster-arg-passing
>> Changeset: r93724:627a1425607c
>> Date: 2018-01-30 14:30 +0100
>> http://bitbucket.org/pypy/pypy/changeset/627a1425607c/
>>
>> Log:    document branch
>>
>> diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
>> --- a/pypy/doc/whatsnew-head.rst
>> +++ b/pypy/doc/whatsnew-head.rst
>> @@ -23,3 +23,11 @@
>>  added, then the performance using mapdict is linear in the number of
>>  attributes. This is now fixed (by switching to a regular dict after
>80
>>  attributes).
>> +
>> +
>> +.. branch: cpyext-faster-arg-passing
>> +
>> +When using cpyext, improve the speed of passing certain objects from
>PyPy
>> to C
>> +code, most notably None, True, False, types, all instances of
>C-defined
>> types.
>> +Before, a dict lookup was needed every time such an object crossed
>over,
>> now it
>> +is just a field read.
>> _______________________________________________
>> pypy-commit mailing list
>> pypy-commit at python.org
>> https://mail.python.org/mailman/listinfo/pypy-commit
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180131/61f8b360/attachment-0001.html>

From anto.cuni at gmail.com  Wed Jan 31 08:49:18 2018
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 31 Jan 2018 13:49:18 +0000
Subject: [pypy-dev] [pypy-commit] pypy cpyext-faster-arg-passing:
 document branch
In-Reply-To: <DAC3F2E2-5467-42E2-ACBF-13FA3400CFE0@gmx.de>
References: <5a7073b3.11a4df0a.80a9d.41b8@mx.google.com>
 <CAHPg78xvpvZDp7JzVRzJcTNnwcn1Yg-Exn=LYepxmJ1zjOqqzw@mail.gmail.com>
 <DAC3F2E2-5467-42E2-ACBF-13FA3400CFE0@gmx.de>
Message-ID: <CAHPg78xXcxZ54pUm4zzQmdLTG9bLa7Zoh3hYHHLR=U25arq9XA@mail.gmail.com>

On Wed, Jan 31, 2018 at 12:34 PM, Carl Friedrich Bolz-Tereick <cfbolz at gmx.de
> wrote:

> Hi Anto,
>
> Yes, I ran your benchmarks and they are improved, particularly the ones
> that pass arguments.
>

?cool :)?



> I need to rerun them now that I merged default. However, I would prefer it
> if we could find a real life cpyext benchmark (maybe using numpy?).
>

?yes sure, mine are microbenchmarks, although they are still useful for
guiding optimizations.
?
About a real life-ish benchmark, what about this (the first version of
course, the one using cpyext)?
https://morepypy.blogspot.co.uk/2017/10/how-to-make-your-code-80-times-faster.html

IIRC, at some point I tried to run it before and after the branches we did
in Cape Town, and I measured a good speedup (although I don't remember how
much). It'd be interesting to check how much we win with your work. I
cannot do it now easily because I'm not at home until the end of the week,
though.

ciao,
Anto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180131/5cd0c0ed/attachment.html>

From victor.stinner at gmail.com  Wed Jan 31 08:47:01 2018
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 Jan 2018 14:47:01 +0100
Subject: [pypy-dev] [Speed] adding a pypy runner for speed.python.org
In-Reply-To: <3296f52d-dd6b-1e69-2c4a-b7d4e3216bf2@gmail.com>
References: <3296f52d-dd6b-1e69-2c4a-b7d4e3216bf2@gmail.com>
Message-ID: <CAMpsgwZbCByt5g4jKLRquSnRn1-AR2xD38+j10ONFTRszP_4MQ@mail.gmail.com>

Hi,

I tried but failed to find someone in PyPy to adjust performance
benchmarks for PyPy. Currently, the JIT is not properly warmed up, and
the results can be dishonnest or not reliable.

My latest attempt to support is PyPy is:
http://vstinner.readthedocs.io/pypy_warmups.html

IMHO we need to develop a statistical methodology in perf to compute
when values become stable. That's hard to define and it was proven
that "performance stability" doesn't exist (see " Virtual Machine
Warmup Blows Hot and Cold" paper).

Fijal from PyPy would like to use hardcoded configuration for the
number of warmup values. I like the idea, but nobody implemented it
yet.

Victor

2018-01-30 21:37 GMT+01:00 Matti Picus <matti.picus at gmail.com>:
> I am cross-posting to both speed and pypy-dev to ask what needs to be done
> to get pypy2 and pypy3 benchmark runners onto speed.python.org
> I am willing to be the contact person from the PyPy side, who do we need to
> talk to on the speed maintainers' side?
> Instead of spamming the lists again, we could discuss this off-line, and
> report back with a plan, required resources, and a timeline.
>
> Thanks,
> Matti Picus
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed

From tobami at gmail.com  Wed Jan 31 09:14:13 2018
From: tobami at gmail.com (Miquel Torres)
Date: Wed, 31 Jan 2018 14:14:13 +0000
Subject: [pypy-dev] [Speed] adding a pypy runner for speed.python.org
In-Reply-To: <CAMpsgwZbCByt5g4jKLRquSnRn1-AR2xD38+j10ONFTRszP_4MQ@mail.gmail.com>
References: <3296f52d-dd6b-1e69-2c4a-b7d4e3216bf2@gmail.com>
 <CAMpsgwZbCByt5g4jKLRquSnRn1-AR2xD38+j10ONFTRszP_4MQ@mail.gmail.com>
Message-ID: <CAGf+9VzmS9zt7O12VnzKOWto_F==RN8fuWGjaUYRLBd1A=Ag0A@mail.gmail.com>

Hi Matti,

I'll gladly help setting up the speed site

El El mi?, 31 ene 2018 a las 13:47, Victor Stinner <victor.stinner at gmail.com>
escribi?:

> Hi,
>
> I tried but failed to find someone in PyPy to adjust performance
> benchmarks for PyPy. Currently, the JIT is not properly warmed up, and
> the results can be dishonnest or not reliable.
>
> My latest attempt to support is PyPy is:
> http://vstinner.readthedocs.io/pypy_warmups.html
>
> IMHO we need to develop a statistical methodology in perf to compute
> when values become stable. That's hard to define and it was proven
> that "performance stability" doesn't exist (see " Virtual Machine
> Warmup Blows Hot and Cold" paper).
>
> Fijal from PyPy would like to use hardcoded configuration for the
> number of warmup values. I like the idea, but nobody implemented it
> yet.
>
> Victor
>
> 2018-01-30 21:37 GMT+01:00 Matti Picus <matti.picus at gmail.com>:
> > I am cross-posting to both speed and pypy-dev to ask what needs to be
> done
> > to get pypy2 and pypy3 benchmark runners onto speed.python.org
> > I am willing to be the contact person from the PyPy side, who do we need
> to
> > talk to on the speed maintainers' side?
> > Instead of spamming the lists again, we could discuss this off-line, and
> > report back with a plan, required resources, and a timeline.
> >
> > Thanks,
> > Matti Picus
> > _______________________________________________
> > Speed mailing list
> > Speed at python.org
> > https://mail.python.org/mailman/listinfo/speed
> _______________________________________________
> Speed mailing list
> Speed at python.org
> https://mail.python.org/mailman/listinfo/speed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180131/9fccd359/attachment.html>