From renesd at gmail.com  Thu Mar  1 11:36:02 2018
From: renesd at gmail.com (=?UTF-8?Q?Ren=C3=A9_Dudfield?=)
Date: Thu, 1 Mar 2018 17:36:02 +0100
Subject: [pypy-dev] Joining the Leysin sprint?
Message-ID: <CAEiAHjK-h_RDmCOh1s9LEUeHCVv-8UhHTsUj0SA-fwf7hMAK_A@mail.gmail.com>

Hi,

I would be interested in joining the pypy sprint.
Is there room left still? 18.3 - 24.3

It would be most interesting for me to finish off pygame-on-pypy bugs,
and to help out on other topics (windows, web, packaging, scikit ... dunno).

PR:
https://bitbucket.org/pypy/extradoc/pull-requests/11/added-myself-to-the-sprint/diff

Hope I'm not too late!


cheers,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180301/4da281d9/attachment.html>

From armin.rigo at gmail.com  Thu Mar  1 16:45:13 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Thu, 1 Mar 2018 22:45:13 +0100
Subject: [pypy-dev] Joining the Leysin sprint?
In-Reply-To: <CAEiAHjK-h_RDmCOh1s9LEUeHCVv-8UhHTsUj0SA-fwf7hMAK_A@mail.gmail.com>
References: <CAEiAHjK-h_RDmCOh1s9LEUeHCVv-8UhHTsUj0SA-fwf7hMAK_A@mail.gmail.com>
Message-ID: <CAMSv6X3Fb=hdnHiQi_hEFgS3bF_jDq1x9zZpj9gUz-j1viUnVA@mail.gmail.com>

Hi Ren?,

On 1 March 2018 at 17:36, Ren? Dudfield <renesd at gmail.com> wrote:
> I would be interested in joining the pypy sprint.
> Is there room left still? 18.3 - 24.3
>
> It would be most interesting for me to finish off pygame-on-pypy bugs,
> and to help out on other topics (windows, web, packaging, scikit ... dunno).

Yes, welcome !  We still have room, assuming you're ready to stay in
rooms of 3 or 4 people.  Is that OK?  (Otherwise, I need to ask, but
you might have to book a place somewhere else in town; there are
plenty of hotels.)


A bient?t,

Armin

From renesd at gmail.com  Thu Mar  1 16:49:58 2018
From: renesd at gmail.com (=?UTF-8?Q?Ren=C3=A9_Dudfield?=)
Date: Thu, 1 Mar 2018 22:49:58 +0100
Subject: [pypy-dev] Joining the Leysin sprint?
In-Reply-To: <CAMSv6X3Fb=hdnHiQi_hEFgS3bF_jDq1x9zZpj9gUz-j1viUnVA@mail.gmail.com>
References: <CAEiAHjK-h_RDmCOh1s9LEUeHCVv-8UhHTsUj0SA-fwf7hMAK_A@mail.gmail.com>
 <CAMSv6X3Fb=hdnHiQi_hEFgS3bF_jDq1x9zZpj9gUz-j1viUnVA@mail.gmail.com>
Message-ID: <CAEiAHjJfRNWDL5-eyJvttR8j4oWvhCo-Zfr03pLfVzz5x85Phg@mail.gmail.com>

Hello,

oh great! Yes, that would be fine.

Thanks, and see you there.

On Thursday, March 1, 2018, Armin Rigo <armin.rigo at gmail.com> wrote:

> Hi Ren?,
>
> On 1 March 2018 at 17:36, Ren? Dudfield <renesd at gmail.com> wrote:
> > I would be interested in joining the pypy sprint.
> > Is there room left still? 18.3 - 24.3
> >
> > It would be most interesting for me to finish off pygame-on-pypy bugs,
> > and to help out on other topics (windows, web, packaging, scikit ...
> dunno).
>
> Yes, welcome !  We still have room, assuming you're ready to stay in
> rooms of 3 or 4 people.  Is that OK?  (Otherwise, I need to ask, but
> you might have to book a place somewhere else in town; there are
> plenty of hotels.)
>
>
> A bient?t,
>
> Armin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180301/793d2867/attachment.html>

From tbaldridge at gmail.com  Thu Mar  1 21:22:16 2018
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Thu, 1 Mar 2018 19:22:16 -0700
Subject: [pypy-dev] Status of "remove the gil"
Message-ID: <CAL36E+uKfEY3yJ0GknjiyGOH=r8r=odkjPeLRtQtk_XZb_953g@mail.gmail.com>

There was some talk awhile back of using fine-grained locks in PyPy to
remove the gil. From the bitbucket repo it looks like the experiments never
made it into master. Is there any work that was done here that could be
used by other RPython interpreters, especially for interpreters that
default to immutable or share-nothing semantics?

Thanks,

Timothy Baldridge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180301/46fec222/attachment.html>

From armin.rigo at gmail.com  Sun Mar  4 06:49:41 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 4 Mar 2018 12:49:41 +0100
Subject: [pypy-dev] Status of "remove the gil"
In-Reply-To: <CAL36E+uKfEY3yJ0GknjiyGOH=r8r=odkjPeLRtQtk_XZb_953g@mail.gmail.com>
References: <CAL36E+uKfEY3yJ0GknjiyGOH=r8r=odkjPeLRtQtk_XZb_953g@mail.gmail.com>
Message-ID: <CAMSv6X2jr1=77gztadeHWbcU=kSAP7oPN-0On4ddK-q79zTFww@mail.gmail.com>

Hi Timothy,

On 2 March 2018 at 03:22, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> There was some talk awhile back of using fine-grained locks in PyPy to
> remove the gil. From the bitbucket repo it looks like the experiments never
> made it into master. Is there any work that was done here that could be used
> by other RPython interpreters, especially for interpreters that default to
> immutable or share-nothing semantics?

It was done in the nogil-unsafe-2 branch, but abandoned for now.
You'd have to dig in that branch and see if it can be used as a
starting point for your case.  Notably, there's the GC which was made
resistant to multithreaded usage.  As far as I remember, though,
nothing was done yet on the JIT front.


A bient?t,

Armin.

From armin.rigo at gmail.com  Sun Mar  4 12:26:49 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 4 Mar 2018 18:26:49 +0100
Subject: [pypy-dev] Sprint Leysin: rooms full, but we can try to get one more
Message-ID: <CAMSv6X02R24Y_rpmOgOdkPVPoh7v5Jv4NtAx9aOXi2YC+4UpGQ@mail.gmail.com>

Hi all,

Just a quick note: the Leysin sprint rooms are fully booked so far.
That's a room with 4 people and a room with 5 people, in addition to a
separate booking for a single room and a room for two.  That will be
quite a busy sprint!

There is an option to get another room for 3 people.  We can book this
extra room either now or only if there is yet another person joining.
The advantage is more confort, as the 5 people room is pretty basic
and will be full.  If there are two or three persons that would prefer
the nicer 3 people room, I can book it now.  (The prices should be
around 50-55 CHF in the big room and maybe 65-75 CHF in the smaller
room; I can check the details if needed.)


A bient?t,

Armin.

From matti.picus at gmail.com  Sun Mar  4 13:02:57 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Sun, 4 Mar 2018 10:02:57 -0800
Subject: [pypy-dev] Sprint Leysin: rooms full,
 but we can try to get one more
In-Reply-To: <CAMSv6X02R24Y_rpmOgOdkPVPoh7v5Jv4NtAx9aOXi2YC+4UpGQ@mail.gmail.com>
References: <CAMSv6X02R24Y_rpmOgOdkPVPoh7v5Jv4NtAx9aOXi2YC+4UpGQ@mail.gmail.com>
Message-ID: <a31195e5-4f14-400b-0c82-6dffcb1bd10b@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180304/e5d4f1e3/attachment.html>

From armin.rigo at gmail.com  Sun Mar  4 13:11:06 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 4 Mar 2018 19:11:06 +0100
Subject: [pypy-dev] Sprint Leysin: rooms full,
 but we can try to get one more
In-Reply-To: <a31195e5-4f14-400b-0c82-6dffcb1bd10b@gmail.com>
References: <CAMSv6X02R24Y_rpmOgOdkPVPoh7v5Jv4NtAx9aOXi2YC+4UpGQ@mail.gmail.com>
 <a31195e5-4f14-400b-0c82-6dffcb1bd10b@gmail.com>
Message-ID: <CAMSv6X152HL6kYN6V+jNbF4zLEOkXrp1mzvZZesN6MhYQ8eXTA@mail.gmail.com>

Hi all,

On 4 March 2018 at 19:02, Matti Picus <matti.picus at gmail.com> wrote:
> That's great news, hopefully we can make some good progress.
> I would not mind being in the three-person room, if needed.
>
> If we are 12-15 people, will we have enough space and outlets to work?

Good point.  I have two multi-socket power strips of the Swiss and
Europe kind, and a few Swiss-to-Europe converters; but if a couple of
people have got another power strip to bring with them, it would be
welcome.

Physical space should be sufficient.  After all, we'll be squatting
the breakfast area, which is big enough for all regular guests at
Ermina.  We will likely have to move laptops around in the evenings to
make some space, but that shouldn't be a problem.


A bient?t,

Armin.

From flub at devork.be  Mon Mar  5 14:38:55 2018
From: flub at devork.be (Floris Bruynooghe)
Date: Mon, 05 Mar 2018 20:38:55 +0100
Subject: [pypy-dev] Sprint Leysin: rooms full,
 but we can try to get one more
In-Reply-To: <CAMSv6X152HL6kYN6V+jNbF4zLEOkXrp1mzvZZesN6MhYQ8eXTA@mail.gmail.com>
References: <CAMSv6X02R24Y_rpmOgOdkPVPoh7v5Jv4NtAx9aOXi2YC+4UpGQ@mail.gmail.com>
 <a31195e5-4f14-400b-0c82-6dffcb1bd10b@gmail.com>
 <CAMSv6X152HL6kYN6V+jNbF4zLEOkXrp1mzvZZesN6MhYQ8eXTA@mail.gmail.com>
Message-ID: <871sgynwhc.fsf@powell.devork.be>

On Sun 04 Mar 2018 at 19:11 +0100, Armin Rigo wrote:
> On 4 March 2018 at 19:02, Matti Picus <matti.picus at gmail.com> wrote:
>> That's great news, hopefully we can make some good progress.
>> I would not mind being in the three-person room, if needed.

Likewise

>> If we are 12-15 people, will we have enough space and outlets to work?
>
> Good point.  I have two multi-socket power strips of the Swiss and
> Europe kind, and a few Swiss-to-Europe converters; but if a couple of
> people have got another power strip to bring with them, it would be
> welcome.

I can bring UK and ~EU extension leads.  I also have ~2 adaptors swiss
adaptors to plug them in.


Cheers,
Floris

From tbaldridge at gmail.com  Fri Mar  9 19:26:19 2018
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Fri, 9 Mar 2018 17:26:19 -0700
Subject: [pypy-dev] Not sure what lltype to use in this case
Message-ID: <CAL36E+uHjC89SA-oiReD16Mx=gEPwpa3V+S-2cfCfbVsUg3G0A@mail.gmail.com>

I have a rather strange use-case for a data type in a rpython interpreter.
What I need is a raw malloc, but with a custom GC hook. I understand how to
setup the GC hook, but what I can't figure out is what lltype to use with
malloc. This type will inherit directly from object so I don't need any
polymorphism. However I do need the struct to be of a variable size. So
something like the C struct hack:

struct MyObject {
   Type* header;
   char data[0];
}

Thanks,

Timothy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180309/2421f9d6/attachment.html>

From armin.rigo at gmail.com  Sat Mar 10 02:26:29 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sat, 10 Mar 2018 08:26:29 +0100
Subject: [pypy-dev] Not sure what lltype to use in this case
In-Reply-To: <CAL36E+uHjC89SA-oiReD16Mx=gEPwpa3V+S-2cfCfbVsUg3G0A@mail.gmail.com>
References: <CAL36E+uHjC89SA-oiReD16Mx=gEPwpa3V+S-2cfCfbVsUg3G0A@mail.gmail.com>
Message-ID: <CAMSv6X0vYbHsr2kLKYPcuBFbNBzv2cU9zv1Hn9y4E1OS7Gr3qw@mail.gmail.com>

Hi Timothy,

On 10 March 2018 at 01:26, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> I have a rather strange use-case for a data type in a rpython interpreter.
> What I need is a raw malloc, but with a custom GC hook.

GC hooks make no sense with non-GC structures.  Try to describe more
what you're trying to achieve.  For example, maybe you have some GC
object that itself contains a pointer to the raw variable-sized
structure, and you want the GC to follow references from this GC
object to other GC objects via the raw variable-sized structure.  Then
you'd put the GC hook on this GC object.


A bient?t,

Armin.

From tbaldridge at gmail.com  Sat Mar 10 08:37:06 2018
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Sat, 10 Mar 2018 06:37:06 -0700
Subject: [pypy-dev] Not sure what lltype to use in this case
In-Reply-To: <CAMSv6X0vYbHsr2kLKYPcuBFbNBzv2cU9zv1Hn9y4E1OS7Gr3qw@mail.gmail.com>
References: <CAL36E+uHjC89SA-oiReD16Mx=gEPwpa3V+S-2cfCfbVsUg3G0A@mail.gmail.com>
 <CAMSv6X0vYbHsr2kLKYPcuBFbNBzv2cU9zv1Hn9y4E1OS7Gr3qw@mail.gmail.com>
Message-ID: <CAL36E+sA=+YYa+5qKFdprSOsKokmCCWc+okr9g+9zEQX+RruiA@mail.gmail.com>

Sorry about that, let me explain it at a higher level. I'm looking to make
a runtime-defined GC'd struct. Some members will be GC'd others will be
primitive. So I have a definition that says:

Foo {member1=int, member2=Object, member3=float}

>From what I can tell I can't create lltypes on-the-fly at runtime, so I'm
left laying out the memory for this by hand, and making a custom GC hook.
But I'm still figuring out how to do that.

Timothy

On Sat, Mar 10, 2018 at 12:26 AM, Armin Rigo <armin.rigo at gmail.com> wrote:

> Hi Timothy,
>
> On 10 March 2018 at 01:26, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> > I have a rather strange use-case for a data type in a rpython
> interpreter.
> > What I need is a raw malloc, but with a custom GC hook.
>
> GC hooks make no sense with non-GC structures.  Try to describe more
> what you're trying to achieve.  For example, maybe you have some GC
> object that itself contains a pointer to the raw variable-sized
> structure, and you want the GC to follow references from this GC
> object to other GC objects via the raw variable-sized structure.  Then
> you'd put the GC hook on this GC object.
>
>
> A bient?t,
>
> Armin.
>



-- 
?One of the main causes of the fall of the Roman Empire was that?lacking
zero?they had no way to indicate successful termination of their C
programs.?
(Robert Firth)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180310/56be0572/attachment.html>

From tbaldridge at gmail.com  Sat Mar 10 10:02:55 2018
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Sat, 10 Mar 2018 08:02:55 -0700
Subject: [pypy-dev] Not sure what lltype to use in this case
In-Reply-To: <CAL36E+sA=+YYa+5qKFdprSOsKokmCCWc+okr9g+9zEQX+RruiA@mail.gmail.com>
References: <CAL36E+uHjC89SA-oiReD16Mx=gEPwpa3V+S-2cfCfbVsUg3G0A@mail.gmail.com>
 <CAMSv6X0vYbHsr2kLKYPcuBFbNBzv2cU9zv1Hn9y4E1OS7Gr3qw@mail.gmail.com>
 <CAL36E+sA=+YYa+5qKFdprSOsKokmCCWc+okr9g+9zEQX+RruiA@mail.gmail.com>
Message-ID: <CAL36E+tW1O-ko0v3CCUKngXKmSxGmtN0pDAA-Qygs-7bmvhvhQ@mail.gmail.com>

One thing I forgot to mention that I'm also trying to keep this as
low-overhead as possible. So I'd like a struct of type MyStruct
{member1=int} to be two words wide. One for the pointer to the
StructDefinition, and the other for the (unwrapped) int.

On Sat, Mar 10, 2018 at 6:37 AM, Timothy Baldridge <tbaldridge at gmail.com>
wrote:

> Sorry about that, let me explain it at a higher level. I'm looking to make
> a runtime-defined GC'd struct. Some members will be GC'd others will be
> primitive. So I have a definition that says:
>
> Foo {member1=int, member2=Object, member3=float}
>
> From what I can tell I can't create lltypes on-the-fly at runtime, so I'm
> left laying out the memory for this by hand, and making a custom GC hook.
> But I'm still figuring out how to do that.
>
> Timothy
>
> On Sat, Mar 10, 2018 at 12:26 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
>
>> Hi Timothy,
>>
>> On 10 March 2018 at 01:26, Timothy Baldridge <tbaldridge at gmail.com>
>> wrote:
>> > I have a rather strange use-case for a data type in a rpython
>> interpreter.
>> > What I need is a raw malloc, but with a custom GC hook.
>>
>> GC hooks make no sense with non-GC structures.  Try to describe more
>> what you're trying to achieve.  For example, maybe you have some GC
>> object that itself contains a pointer to the raw variable-sized
>> structure, and you want the GC to follow references from this GC
>> object to other GC objects via the raw variable-sized structure.  Then
>> you'd put the GC hook on this GC object.
>>
>>
>> A bient?t,
>>
>> Armin.
>>
>
>
>
> --
> ?One of the main causes of the fall of the Roman Empire was that?lacking
> zero?they had no way to indicate successful termination of their C
> programs.?
> (Robert Firth)
>



-- 
?One of the main causes of the fall of the Roman Empire was that?lacking
zero?they had no way to indicate successful termination of their C
programs.?
(Robert Firth)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180310/f7f64bf9/attachment.html>

From armin.rigo at gmail.com  Sat Mar 10 10:13:38 2018
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sat, 10 Mar 2018 16:13:38 +0100
Subject: [pypy-dev] Not sure what lltype to use in this case
In-Reply-To: <CAL36E+tW1O-ko0v3CCUKngXKmSxGmtN0pDAA-Qygs-7bmvhvhQ@mail.gmail.com>
References: <CAL36E+uHjC89SA-oiReD16Mx=gEPwpa3V+S-2cfCfbVsUg3G0A@mail.gmail.com>
 <CAMSv6X0vYbHsr2kLKYPcuBFbNBzv2cU9zv1Hn9y4E1OS7Gr3qw@mail.gmail.com>
 <CAL36E+sA=+YYa+5qKFdprSOsKokmCCWc+okr9g+9zEQX+RruiA@mail.gmail.com>
 <CAL36E+tW1O-ko0v3CCUKngXKmSxGmtN0pDAA-Qygs-7bmvhvhQ@mail.gmail.com>
Message-ID: <CAMSv6X0TALewMZjN3j6euvnwg4kPqtaMudo86qd_OTihzSO6gA@mail.gmail.com>

Hi Timothy,

On 10 March 2018 at 16:02, Timothy Baldridge <tbaldridge at gmail.com> wrote:
> One thing I forgot to mention that I'm also trying to keep this as
> low-overhead as possible. So I'd like a struct of type MyStruct
> {member1=int} to be two words wide. One for the pointer to the
> StructDefinition, and the other for the (unwrapped) int.

Ok, that makes sense.  But you have to consider also where such a raw
MyStruct would live; that is, who has got the raw pointer to the
MyStruct data?  I would imagine that you want this pointer to be
inside another GC object of a specific class.  It's this GC object
there that needs to have a custom GC hook.  If you really want no such
thing, then you still need to make an artificial GC object in order to
have a place where you can stick the custom GC hook.  Also, remember
that the GC assumes 'x.custom_gc_hook()' always returns the same list
by default; you need to manually call 'rgc.write_barrier(x)' whenever
the custom GC hook on object 'x' would give a different answer.


A bient?t,

Armin.

From arek.bulski at gmail.com  Mon Mar 12 04:50:53 2018
From: arek.bulski at gmail.com (Arkadiusz Bulski)
Date: Mon, 12 Mar 2018 08:50:53 +0000
Subject: [pypy-dev] Ordered **kw officially suported?
Message-ID: <CAKnUdEZ30-BH-Qp1FfAU5xVpeKUROFcfgDbdY-Dspnk0AnNfpw@mail.gmail.com>

Hi,

For the record, I am the developer of Construct library.
https://construct.readthedocs.io/en/latest/#

I noticed that test cases using ordered **kw arguments, which is a feature
of 3.6, also pass on pypy (both 2.7 and 3.5). I need to ask, is this
feature officially supported by pypy, or just passes by chance?

-- 
~ Arkadiusz Bulski
~ tel 503 357 111 lub WhatsApp/Telegram
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180312/fd18b6ec/attachment.html>

From cfbolz at gmx.de  Mon Mar 12 06:55:52 2018
From: cfbolz at gmx.de (Carl Friedrich Bolz-Tereick)
Date: Mon, 12 Mar 2018 11:55:52 +0100
Subject: [pypy-dev] Ordered **kw officially suported?
In-Reply-To: <CAKnUdEZ30-BH-Qp1FfAU5xVpeKUROFcfgDbdY-Dspnk0AnNfpw@mail.gmail.com>
References: <CAKnUdEZ30-BH-Qp1FfAU5xVpeKUROFcfgDbdY-Dspnk0AnNfpw@mail.gmail.com>
Message-ID: <217F7B54-CC67-4201-93DE-E1370827557B@gmx.de>

Hi Arkadiusz,

Yes, normal dictionaries in pypy are ordered, in all versions. There are some exceptions around instance dicts, but for **args and dictionaries you create explicitly, this is definitely the case. 

Cheers,
Carl Friedrich

On March 12, 2018 9:50:53 AM GMT+01:00, Arkadiusz Bulski <arek.bulski at gmail.com> wrote:
>Hi,
>
>For the record, I am the developer of Construct library.
>https://construct.readthedocs.io/en/latest/#
>
>I noticed that test cases using ordered **kw arguments, which is a
>feature
>of 3.6, also pass on pypy (both 2.7 and 3.5). I need to ask, is this
>feature officially supported by pypy, or just passes by chance?
>
>-- 
>~ Arkadiusz Bulski
>~ tel 503 357 111 lub WhatsApp/Telegram
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180312/3f09acb9/attachment.html>

From bra at fsn.hu  Sun Mar 18 06:59:21 2018
From: bra at fsn.hu (Nagy, Attila)
Date: Sun, 18 Mar 2018 11:59:21 +0100
Subject: [pypy-dev] Preheating pypy
Message-ID: <d37ce23a-c81c-75b4-7627-1897e6ca03a9@fsn.hu>

Hi,

I use pypy to run an application in gunicorn.
Gunicorn (as well) has a "preload" capability in multiprocess workers, 
which means it loads the application in one interpreter/process and 
instead of starting new interpreters/processes for additional workers, 
it just forks from the first one.
This means the OS can share memory pages between the processes, which 
makes the app use less memory and start faster.

This works nicely with pypy too and the memory savings are significant 
(taking into account that pypy uses much more memory than cpython, this 
is even more true).

The problem is that each pypy process before the fork is "cold" and the 
JIT starts to compile code in the forked processes, independently from 
the original process, which makes a good deal of additonal memory (and 
CPU) go wasted.

It would be nice to make this happen in the original process, so the 
operating system's COW semantics could work for the JIT-ed code too.

Is there a way to achieve this?

Thanks,


From matt at vazor.com  Sun Mar 18 18:14:24 2018
From: matt at vazor.com (Matt Billenstein)
Date: Sun, 18 Mar 2018 22:14:24 +0000
Subject: [pypy-dev] Preheating pypy
In-Reply-To: <d37ce23a-c81c-75b4-7627-1897e6ca03a9@fsn.hu>
References: <d37ce23a-c81c-75b4-7627-1897e6ca03a9@fsn.hu>
Message-ID: <010101623b2d8fd6-5ac3a660-f749-496b-939c-61043af94d4e-000000@us-west-2.amazonses.com>

Seems you need to just trigger whatever heuristic causes the JIT to run on the
interesting codepaths during application startup.

Would having a small for loop in a module global namespace that called down
through your stack do the trick?

===== somemodule.py

def foo(...):
    # maybe this function calls through several layers of code in other
    # modules...
    ...

for i in range(20):
    foo(...)  # prewarm foo...

=====

m

On Sun, Mar 18, 2018 at 11:59:21AM +0100, Nagy, Attila wrote:
> Hi,
> 
> I use pypy to run an application in gunicorn.
> Gunicorn (as well) has a "preload" capability in multiprocess workers, which
> means it loads the application in one interpreter/process and instead of
> starting new interpreters/processes for additional workers, it just forks
> from the first one.
> This means the OS can share memory pages between the processes, which makes
> the app use less memory and start faster.
> 
> This works nicely with pypy too and the memory savings are significant
> (taking into account that pypy uses much more memory than cpython, this is
> even more true).
> 
> The problem is that each pypy process before the fork is "cold" and the JIT
> starts to compile code in the forked processes, independently from the
> original process, which makes a good deal of additonal memory (and CPU) go
> wasted.
> 
> It would be nice to make this happen in the original process, so the
> operating system's COW semantics could work for the JIT-ed code too.
> 
> Is there a way to achieve this?
> 
> Thanks,
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From grant.shepherd7 at gmail.com  Mon Mar 19 17:33:09 2018
From: grant.shepherd7 at gmail.com (Grant Shepherd)
Date: Mon, 19 Mar 2018 15:33:09 -0600
Subject: [pypy-dev] Pip fails to work
Message-ID: <CAFEyuECY4010KKhHHRe=yR+dfombPMcP1TiBVUbOgH1Ohx5rJQ@mail.gmail.com>

PyPy seems like a great python version

Except the extremely painful effort trying to get pip to work

Can't connect to HTTPS URL because the SSL module is not available. -
skipping
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180319/cb7cffb1/attachment.html>

From rymg19 at gmail.com  Tue Mar 20 19:27:13 2018
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 20 Mar 2018 18:27:13 -0500
Subject: [pypy-dev] Pip fails to work
In-Reply-To: <CAFEyuECY4010KKhHHRe=yR+dfombPMcP1TiBVUbOgH1Ohx5rJQ@mail.gmail.com>
References: <CAFEyuECY4010KKhHHRe=yR+dfombPMcP1TiBVUbOgH1Ohx5rJQ@mail.gmail.com>
Message-ID: <1521588433.13051.0@smtp.gmail.com>

Did you build PyPy yourself? If so, you didn't compile the ssl module. 
Not really PyPy's fault...

OTOH if this is a distro package/binary, where did you get it from? You 
can try the portable PyPy binaries: 
https://github.com/squeaky-pl/portable-pypy
--
Ryan (????)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
https://refi64.com/

On Mon, Mar 19, 2018 at 4:33 PM, Grant Shepherd 
<grant.shepherd7 at gmail.com> wrote:
> PyPy seems like a great python version
> 
> Except the extremely painful effort trying to get pip to work
> 
> Can't connect to HTTPS URL because the SSL module is not available. - 
> skipping
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180320/8cad9d5d/attachment.html>

From bra at fsn.hu  Wed Mar 21 04:06:20 2018
From: bra at fsn.hu (Nagy, Attila)
Date: Wed, 21 Mar 2018 09:06:20 +0100
Subject: [pypy-dev] Preheating pypy
In-Reply-To: <010101623b2d8fd6-5ac3a660-f749-496b-939c-61043af94d4e-000000@us-west-2.amazonses.com>
References: <d37ce23a-c81c-75b4-7627-1897e6ca03a9@fsn.hu>
 <010101623b2d8fd6-5ac3a660-f749-496b-939c-61043af94d4e-000000@us-west-2.amazonses.com>
Message-ID: <ef283601-d0a7-059b-4e77-dad29a8fae47@fsn.hu>

On 03/18/2018 11:14 PM, Matt Billenstein wrote:
> Seems you need to just trigger whatever heuristic causes the JIT to run on the
> interesting codepaths during application startup.
>
> Would having a small for loop in a module global namespace that called down
> through your stack do the trick?
>
> ===== somemodule.py
>
> def foo(...):
>      # maybe this function calls through several layers of code in other
>      # modules...
>      ...
>
> for i in range(20):
>      foo(...)  # prewarm foo...
>
> =====
>
>
That would be hard, I guess. For example I couldn't reach the processing 
layer of a connection object this way, only with real queries.


From matt at vazor.com  Wed Mar 21 18:55:38 2018
From: matt at vazor.com (Matt Billenstein)
Date: Wed, 21 Mar 2018 22:55:38 +0000
Subject: [pypy-dev] Preheating pypy
In-Reply-To: <ef283601-d0a7-059b-4e77-dad29a8fae47@fsn.hu>
References: <d37ce23a-c81c-75b4-7627-1897e6ca03a9@fsn.hu>
 <010101623b2d8fd6-5ac3a660-f749-496b-939c-61043af94d4e-000000@us-west-2.amazonses.com>
 <ef283601-d0a7-059b-4e77-dad29a8fae47@fsn.hu>
Message-ID: <010101624ac6626c-9fa9f3a9-71cc-4570-8181-d44c8f3e99b4-000000@us-west-2.amazonses.com>

Yeah, not very clean, but you could run dummy queries there that just exercised
the hot code paths.

It would be interesting to see if it worked though -- have a function that runs
much faster after jit, warm it like this, then see if behaves the same after
gunicorn forks the worker process.

m

On Wed, Mar 21, 2018 at 09:06:20AM +0100, Nagy, Attila wrote:
> On 03/18/2018 11:14 PM, Matt Billenstein wrote:
> > Seems you need to just trigger whatever heuristic causes the JIT to run on the
> > interesting codepaths during application startup.
> > 
> > Would having a small for loop in a module global namespace that called down
> > through your stack do the trick?
> > 
> > ===== somemodule.py
> > 
> > def foo(...):
> >      # maybe this function calls through several layers of code in other
> >      # modules...
> >      ...
> > 
> > for i in range(20):
> >      foo(...)  # prewarm foo...
> > 
> > =====
> > 
> > 
> That would be hard, I guess. For example I couldn't reach the processing
> layer of a connection object this way, only with real queries.

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From grant.shepherd7 at gmail.com  Sat Mar 24 03:27:26 2018
From: grant.shepherd7 at gmail.com (Grant Shepherd)
Date: Sat, 24 Mar 2018 01:27:26 -0600
Subject: [pypy-dev] Pip fails to work
In-Reply-To: <1521588433.13051.0@smtp.gmail.com>
References: <CAFEyuECY4010KKhHHRe=yR+dfombPMcP1TiBVUbOgH1Ohx5rJQ@mail.gmail.com>
 <1521588433.13051.0@smtp.gmail.com>
Message-ID: <012101d3c341$9004f190$b00ed4b0$@gmail.com>

 

PyPy needs more effort cleaning up how it relates info on its website providing better directions on overcoming its reliance on a specific versions of the OpenSSL library.

I have been working in I.T. for 30 years and I know when I see a something that needs better explanation.

 

I tried a few different tactics with pre-compiled binaries, symlink tricks and straight up compiling and still could not overcome the issue of getting PyPy 3 to work for Pip and SSL on CentOS 7.

And Googling the problem did not help either which is why I emailed the mailing list which I never do ever..

In the end I used Squeaky's portable Linux binaries which performed 27% slower then CPython in my Website Stack testing.

 

So I had a bad experience with PyPy, I don?t see how that helps a project get followers and grow its reputation?

 

 

From: Ryan Gonzalez [mailto:rymg19 at gmail.com] 
Sent: Tuesday, 20 March 2018 5:27 PM
To: Grant Shepherd
Cc: pypy-dev at python.org
Subject: Re: [pypy-dev] Pip fails to work

 

Did you build PyPy yourself? If so, you didn't compile the ssl module. Not really PyPy's fault...

 

OTOH if this is a distro package/binary, where did you get it from? You can try the portable PyPy binaries: https://github.com/squeaky-pl/portable-pypy

-- Ryan (????) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/


On Mon, Mar 19, 2018 at 4:33 PM, Grant Shepherd <grant.shepherd7 at gmail.com <mailto:grant.shepherd7 at gmail.com> > wrote:



PyPy seems like a great python version

 

Except the extremely painful effort trying to get pip to work

 

Can't connect to HTTPS URL because the SSL module is not available. - skipping

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180324/b3d1de3c/attachment.html>

From matti.picus at gmail.com  Sat Mar 24 15:27:16 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 24 Mar 2018 22:27:16 +0300
Subject: [pypy-dev] Pip fails to work
In-Reply-To: <012101d3c341$9004f190$b00ed4b0$@gmail.com>
References: <CAFEyuECY4010KKhHHRe=yR+dfombPMcP1TiBVUbOgH1Ohx5rJQ@mail.gmail.com>
 <1521588433.13051.0@smtp.gmail.com>
 <012101d3c341$9004f190$b00ed4b0$@gmail.com>
Message-ID: <38a3b7e8-5b23-7c35-653b-842b38017118@gmail.com>

On 24/03/18 10:27, Grant Shepherd wrote:
>
> PyPy needs more effort cleaning up how it relates info on its website 
> providing better directions on overcoming its reliance on a specific 
> versions of the OpenSSL library.
>
> I have been working in I.T. for 30 years and I know when I see a 
> something that needs better explanation.
>
> I tried a few different tactics with pre-compiled binaries, symlink 
> tricks and straight up compiling and still could not overcome the 
> issue of getting PyPy 3 to work for Pip and SSL on CentOS 7.
>
> And Googling the problem did not help either which is why I emailed 
> the mailing list which I never do ever..
>
> In the end I used Squeaky's portable Linux binaries which performed 
> 27% slower then CPython in my Website Stack testing.
>
> So I had a bad experience with PyPy, I don?t see how that helps a 
> project get followers and grow its reputation?
>
>

Sorry to disappoint. Please let us know which part of the website you 
find unclear so that we can improve.


If you have an easily repeatable benchmark we can replicate, it might 
help us make PyPy faster.

Matti


From tkadm30 at yandex.com  Mon Mar 26 09:55:31 2018
From: tkadm30 at yandex.com (Etienne Robillard)
Date: Mon, 26 Mar 2018 09:55:31 -0400
Subject: [pypy-dev] Problem with cgi module and pypy handling POST request
Message-ID: <7a856e47-8dc0-60a8-55c5-4115161c039b@yandex.com>

Hi,

I got the following error playing with pypy 5.9.0 under my linux32 
development server:

Traceback (most recent call last):
   File "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/controllers/base.py", line 195, in get_response
     response = callback(request, *args, **kwargs)
   File "/home/erob/src/django-hotsauce-stable/extras/libauthkit/authkit/authorize/decorators.py", line 67, in _wrapper
     return view_func(request, **kwargs)
   File "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/dbapi/orm/decorators.py", line 35, in _wrapper
     return view_func(req, **kwargs)
   File "/home/erob/src/blogengine2-devel/lib/blogengine2/views/category.py", line 57, in add
     new_data = request.POST.copy()
   File "/usr/local/pypy/pypy2-v5.9.0-linux32/site-packages/werkzeug/local.py", line 347, in __getattr__
     return getattr(self._get_current_object(), name)
   File "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/utils/wsgilib/request.py", line 96, in POST
     return self.get_POST()
   File "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/utils/wsgilib/request.py", line 69, in get_POST
     keep_blank_values=True)
   File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 507, in __init__
     self.read_multi(environ, keep_blank_values, strict_parsing)
   File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 631, in read_multi
     environ, keep_blank_values, strict_parsing)
   File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 509, in __init__
     self.read_single()
   File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 646, in read_single
     self.read_lines()
   File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 668, in read_lines
     self.read_lines_to_outerboundary()
   File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 696, in read_lines_to_outerboundary
     line = self.fp.readline(1<<16)
TypeError: readline() takes exactly 1 argument (2 given)


Any ideas how to fix this issue?

Thank you in advance,


Etienne

https://www.isotopesoftware.ca


From matti.picus at gmail.com  Mon Mar 26 11:04:36 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 26 Mar 2018 18:04:36 +0300
Subject: [pypy-dev] Problem with cgi module and pypy handling POST
 request
In-Reply-To: <7a856e47-8dc0-60a8-55c5-4115161c039b@yandex.com>
References: <7a856e47-8dc0-60a8-55c5-4115161c039b@yandex.com>
Message-ID: <2888718b-f95f-499f-f3c5-c146cf78a9db@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180326/1c771918/attachment.html>

From tkadm30 at yandex.com  Mon Mar 26 11:33:35 2018
From: tkadm30 at yandex.com (Etienne Robillard)
Date: Mon, 26 Mar 2018 11:33:35 -0400
Subject: [pypy-dev] Problem with cgi module and pypy handling POST
 request
In-Reply-To: <2888718b-f95f-499f-f3c5-c146cf78a9db@gmail.com>
References: <7a856e47-8dc0-60a8-55c5-4115161c039b@yandex.com>
 <2888718b-f95f-499f-f3c5-c146cf78a9db@gmail.com>
Message-ID: <d65e258f-ced1-5d28-3143-59ee2fbf78f9@yandex.com>

The output is: <wsgiref.validate.InputWrapper instance at 0xb3e1c784>


I also managed to fix this issue by removing the "1<<16" parameter in 
self.fp.readline().


Regards,


Etienne



Le 2018-03-26 ? 11:04, Matti Picus a ?crit?:
> On 26/03/18 16:55, Etienne Robillard wrote:
>> Hi,
>>
>> I got the following error playing with pypy 5.9.0 under my linux32 
>> development server:
>>
>> Traceback (most recent call last):
>> ? File 
>> "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/controllers/base.py", 
>> line 195, in get_response
>> ??? response = callback(request, *args, **kwargs)
>> ? File 
>> "/home/erob/src/django-hotsauce-stable/extras/libauthkit/authkit/authorize/decorators.py", 
>> line 67, in _wrapper
>> ??? return view_func(request, **kwargs)
>> ? File 
>> "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/dbapi/orm/decorators.py", 
>> line 35, in _wrapper
>> ??? return view_func(req, **kwargs)
>> ? File 
>> "/home/erob/src/blogengine2-devel/lib/blogengine2/views/category.py", 
>> line 57, in add
>> ??? new_data = request.POST.copy()
>> ? File 
>> "/usr/local/pypy/pypy2-v5.9.0-linux32/site-packages/werkzeug/local.py", 
>> line 347, in __getattr__
>> ??? return getattr(self._get_current_object(), name)
>> ? File 
>> "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/utils/wsgilib/request.py", 
>> line 96, in POST
>> ??? return self.get_POST()
>> ? File 
>> "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/utils/wsgilib/request.py", 
>> line 69, in get_POST
>> ??? keep_blank_values=True)
>> ? File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", 
>> line 507, in __init__
>> ??? self.read_multi(environ, keep_blank_values, strict_parsing)
>> ? File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", 
>> line 631, in read_multi
>> ??? environ, keep_blank_values, strict_parsing)
>> ? File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", 
>> line 509, in __init__
>> ??? self.read_single()
>> ? File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", 
>> line 646, in read_single
>> ??? self.read_lines()
>> ? File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", 
>> line 668, in read_lines
>> ??? self.read_lines_to_outerboundary()
>> ? File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", 
>> line 696, in read_lines_to_outerboundary
>> ??? line = self.fp.readline(1<<16)
>> TypeError: readline() takes exactly 1 argument (2 given)
>>
>>
>> Any ideas how to fix this issue?
>>
>> Thank you in advance,
>>
>>
>> Etienne
>>
>> https://www.isotopesoftware.ca
>
> Could you put a print just before line 696 and find out what self.fp is?
> Matti

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180326/3dafb46a/attachment.html>

From matti.picus at gmail.com  Mon Mar 26 13:55:17 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 26 Mar 2018 20:55:17 +0300
Subject: [pypy-dev] Problem with cgi module and pypy handling POST
 request
In-Reply-To: <d65e258f-ced1-5d28-3143-59ee2fbf78f9@yandex.com>
References: <7a856e47-8dc0-60a8-55c5-4115161c039b@yandex.com>
 <2888718b-f95f-499f-f3c5-c146cf78a9db@gmail.com>
 <d65e258f-ced1-5d28-3143-59ee2fbf78f9@yandex.com>
Message-ID: <6260d911-47ea-49a1-1ab6-ca7ec9195412@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180326/97d56858/attachment.html>

From tkadm30 at yandex.com  Mon Mar 26 17:00:42 2018
From: tkadm30 at yandex.com (Etienne Robillard)
Date: Mon, 26 Mar 2018 17:00:42 -0400
Subject: [pypy-dev] Problem with cgi module and pypy handling POST
 request
In-Reply-To: <6260d911-47ea-49a1-1ab6-ca7ec9195412@gmail.com>
References: <7a856e47-8dc0-60a8-55c5-4115161c039b@yandex.com>
 <2888718b-f95f-499f-f3c5-c146cf78a9db@gmail.com>
 <d65e258f-ced1-5d28-3143-59ee2fbf78f9@yandex.com>
 <6260d911-47ea-49a1-1ab6-ca7ec9195412@gmail.com>
Message-ID: <72f626f1-164c-5618-1c74-86a1260a3c80@yandex.com>



Le 2018-03-26 ? 13:55, Matti Picus a ?crit?:
> On 26/03/18 18:33, Etienne Robillard wrote:
>>
>> The output is: <wsgiref.validate.InputWrapper instance at 0xb3e1c784>
>>
>>
>> I also managed to fix this issue by removing the "1<<16" parameter in 
>> self.fp.readline().
>>
>>
>> Regards,
>>
>>
>> Etienne
>>
>>
>>
> This would fail in the same way on Cpython 2, it is a "feature" of 
> that method.
> In both PyPy and CPython, the interface was changed for python3 to 
> accept a size parameter.
>
> Matti

Sorry, but without my patch I cannot make PyPy 5.9.0 to handle a POST 
request.

Could you please suggest any other fix?

Etienne

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180326/ac1a8ff3/attachment.html>

From matti.picus at gmail.com  Mon Mar 26 17:06:36 2018
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 27 Mar 2018 00:06:36 +0300
Subject: [pypy-dev] Problem with cgi module and pypy handling POST
 request
In-Reply-To: <72f626f1-164c-5618-1c74-86a1260a3c80@yandex.com>
References: <7a856e47-8dc0-60a8-55c5-4115161c039b@yandex.com>
 <2888718b-f95f-499f-f3c5-c146cf78a9db@gmail.com>
 <d65e258f-ced1-5d28-3143-59ee2fbf78f9@yandex.com>
 <6260d911-47ea-49a1-1ab6-ca7ec9195412@gmail.com>
 <72f626f1-164c-5618-1c74-86a1260a3c80@yandex.com>
Message-ID: <10ca5c20-5aec-1bd2-a02c-3263ca9530c8@gmail.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20180327/e977df73/attachment.html>