From mhroncok at redhat.com  Thu Dec  2 13:45:32 2021
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=)
Date: Thu, 2 Dec 2021 19:45:32 +0100
Subject: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
Message-ID: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>

Hello PyPy folks,

I co-maintain the PyPy and CPython interpreter packages in Fedora and I am 
excited but a bit puzzled by a certain change that happened in PyPy 3.8.
The installation layout of PyPy 3.8 was changed to be more like the CPython layout:

https://doc.pypy.org/en/latest/release-v7.3.6.html

 > The 3.8 package now uses the same layout as CPython, and many of the
 > PyPy-specific changes to sysconfig, distutils.sysconfig,
 > and distutils.commands.install.py have been removed. The stdlib now is
 > located in <base>/lib/pypy3.8 on posix systems, and in <base>/Lib on Windows.
 > The include files on windows remain the same, on posix they are in
 > <base>/include/pypy3.8 . Note we still use the pypy prefix to prevent mixing
 > the files with CPython (which uses python).

Up until now (e.g. with PyPy 3.7) the installation layout of the Fedora package 
looked like this:

/usr/lib64/pypy3.7/
??? bin
??? ??? ...
??? include
??? ??? *.h
??? lib_pypy
??? ??? ...
??? lib-python
??? ??? 3
???     ??? ...
??? site-packages
     ??? ...
/usr/bin/pypy3.7 -> /usr/lib64/pypy3.7/bin/pypy3.7
(sometimes) /usr/bin/pypy3 -> /usr/bin/pypy3.7

We basically took the advice from https://doc.pypy.org/en/latest/build.html

 > To install PyPy system wide on unix-like systems, it is recommended to put
 > the whole hierarchy alone (e.g. in /opt/pypy) and put a symlink to the pypy
 > executable into /usr/bin or /usr/local/bin.

Where we used "/usr/lib64/pypy3.7" instead of "/opt/pypy".
Several different versions of pypy (e.g. 3.6 and 3.7) could be installed 
alongside each other, as long as only one of them "owned" /usr/bin/pypy3.

Now with PyPy 3.8, if we are to keep this scheme, the entire tree gets a bit 
more nested:

/usr/lib64/pypy3.8/
??? bin
?   ??? ...
??? include
?   ??? PyPy.h
?   ??? pypy3.8
?       ??? *.h
??? lib
     ??? pypy3.8
         ??? ...
         ??? site-packages
             ??? ...
(/usr/bin setup remains the same)

This e.g. means that the standard library is installed in:

  /usr/lib64/pypy3.8/lib/pypy3.8/

Or the header files are in:

  /usr/lib64/pypy3.8/include/pypy3.8/

Double namespacing like this seems a bit overengineered, and since the idea was 
to resemble CPython, I was aiming at this setup:

/usr/
??? bin
?   ??? ...
??? include
?   ??? pypy3.8
?       ??? *.h
??? lib(64)
     ??? pypy3.8
         ??? ...
         ??? site-packages
             ??? ...

Which would be the exact copy of our CPython setup | sed s/python/pypy/

However, I found 4 problems:

1) The PyPy install scheme only has "lib", not "lib64"

I think we could sort this by a tiny patch that we used to carry in CPython 3.8 
before configurable platlibdir was introduced in 3.9.


2) The /usr/include/PyPy.h file is *not* namespaced with PyPy Python version

I don't know if we even need to ship this file. It seems deprecated. Maybe it 
can be in /usr/include/pypy3.8/PyPy.h?


3) /usr/bin/pypy3.8 is a symbolic link to /usr/bin/pypy3

I would expect it to be the other way around, so both /usr/bin/pypy3.X and 
/usr/bin/pypy3.Y can co-exist and /usr/bin/pypy3 is an (optional) symbolic link 
to one of them (as is the situtaion with CPython). I guess I can somehow 
reverse the order manually, but I have not tested if it breaks anything.


4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced

This is the one thing I cannot wrap my head around. I do not wish to install 
this file "globally" to /usr/bin for two reasons:

  a) it conflicts between different PyPy Python versions
  b) it ends up on $PATH but segfaults when executed


So, I am asking on this list: How should we package PyPy 3.8 for Fedora? Do we 
need to install it to /usr/lib64/pypy3.8/ prefix even thou most (but not all) 
of the directories within would contain a "pypy3.8" subdirectory? Or is there a 
better option I do not yet see?

Thanks for your help.

-- 
Miro Hron?ok
-- 
Phone: +420777974800
IRC: mhroncok


From mgorny at gentoo.org  Thu Dec  2 15:02:45 2021
From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=)
Date: Thu, 02 Dec 2021 21:02:45 +0100
Subject: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
In-Reply-To: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>
References: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>
Message-ID: <9ea07e2404fc3dbc313b7b85660c19464973dbf7.camel@gentoo.org>

On Thu, 2021-12-02 at 19:45 +0100, Miro Hron?ok wrote:
> However, I found 4 problems:
> 
> 1) The PyPy install scheme only has "lib", not "lib64"
> 
> I think we could sort this by a tiny patch that we used to carry in CPython 3.8 
> before configurable platlibdir was introduced in 3.9.

Are you trying to make PyPy multilib?  Plain "lib" seems correct for
libexecdir-style packages.

> 2) The /usr/include/PyPy.h file is *not* namespaced with PyPy Python version
> 
> I don't know if we even need to ship this file. It seems deprecated. Maybe it 
> can be in /usr/include/pypy3.8/PyPy.h?

FWICS we're installing it there on Gentoo and we haven't had any
problems because of it yet.

> 3) /usr/bin/pypy3.8 is a symbolic link to /usr/bin/pypy3
> 
> I would expect it to be the other way around, so both /usr/bin/pypy3.X and 
> /usr/bin/pypy3.Y can co-exist and /usr/bin/pypy3 is an (optional) symbolic link 
> to one of them (as is the situtaion with CPython). I guess I can somehow 
> reverse the order manually, but I have not tested if it breaks anything.

We're symlinking pypy3 -> pypy3-c-7.3.7 and it doesn't break anything,
in case that helps.

> 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced
> 
> This is the one thing I cannot wrap my head around. I do not wish to install 
> this file "globally" to /usr/bin for two reasons:
> 
>   a) it conflicts between different PyPy Python versions
>   b) it ends up on $PATH but segfaults when executed

TBH I've never really understood the purpose of this file.
We've stopped using it at some point and nothing really changed for us.

> So, I am asking on this list: How should we package PyPy 3.8 for Fedora?

No, please don't, that would mean Gentoo would use its edge over Fedora
;-).

-- 
Best regards,
Micha? G?rny


From armin.rigo at gmail.com  Thu Dec  2 17:48:33 2021
From: armin.rigo at gmail.com (Armin Rigo)
Date: Thu, 2 Dec 2021 23:48:33 +0100
Subject: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
In-Reply-To: <9ea07e2404fc3dbc313b7b85660c19464973dbf7.camel@gentoo.org>
References: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>
 <9ea07e2404fc3dbc313b7b85660c19464973dbf7.camel@gentoo.org>
Message-ID: <CAMSv6X14w9W5XPX-WFuy9AKJXUQyWv+YT+XDHmNDJy4dike3AA@mail.gmail.com>

Hi,

On Thu, 2 Dec 2021 at 21:10, Micha? G?rny <mgorny at gentoo.org> wrote:

> > 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced
>
> TBH I've never really understood the purpose of this file.
> We've stopped using it at some point and nothing really changed for us.

It is needed for "embedding" situations, which nowadays means
https://cffi.readthedocs.io/en/latest/embedding.html.  If we want to
go the full CPython way, we need to rename and move
"$localdir/libpypy3-c.so" to something like "/usr/lib/libpypy38.so"
and have /usr/bin/pypy3.8 be a program that is linked to
"libpypy38.so" instead of "$localdir/libpypy3-c.so".  This would have
consequences for my own habits---e.g. the executable would no longer
find its .so if it lives simply in the same directory.  Arguably not a
big issue :-)  A freshly translated pypy would not work without
LD_LIBRARY_PATH tricks, just like a freshly compiled CPython does not.
The CPython solution to that annoyance is to compile statically by
default, if no option is passed to ./configure; you need to give the
--enable-shared option explicitly, and all distributions do.  So if we
want to copy all of that for PyPy, we could.

(Note that the CPython explanation is based on my 2.7 knowledge and
may be outdated.)


A bient?t,

Armin.

From anto.cuni at gmail.com  Sat Dec  4 05:21:32 2021
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sat, 4 Dec 2021 11:21:32 +0100
Subject: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
In-Reply-To: <CAMSv6X14w9W5XPX-WFuy9AKJXUQyWv+YT+XDHmNDJy4dike3AA@mail.gmail.com>
References: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>
 <9ea07e2404fc3dbc313b7b85660c19464973dbf7.camel@gentoo.org>
 <CAMSv6X14w9W5XPX-WFuy9AKJXUQyWv+YT+XDHmNDJy4dike3AA@mail.gmail.com>
Message-ID: <CAHPg78whtABvju7P2XgcROE_WDmh6r08ri554ge5yWv-zu9i7Q@mail.gmail.com>

On Thu, Dec 2, 2021 at 11:49 PM Armin Rigo <armin.rigo at gmail.com> wrote:

>
> https://cffi.readthedocs.io/en/latest/embedding.html.  If we want to
> go the full CPython way, we need to rename and move
> "$localdir/libpypy3-c.so" to something like "/usr/lib/libpypy38.so"
> and have /usr/bin/pypy3.8 be a program that is linked to
> "libpypy38.so" instead of "$localdir/libpypy3-c.so".  This would have
> consequences for my own habits---e.g. the executable would no longer
> find its .so if it lives simply in the same directory.  Arguably not a
> big issue :-)  A freshly translated pypy would not work without
> LD_LIBRARY_PATH tricks, just like a freshly compiled CPython does not.
>

I fear this would break much more than your own habits. To start with, it
would break mine :)
Jokes apart, it would break three big uses cases:
1. Download&run: if you want to try pypy, currently you can download the
tarball, unpack and run, and it just works
2. non-privileged installation: currently you can install PyPy for your own
user even if you don't have root power
3. install pypy in docker: this is basically the same as (1) but it's worth
mentioning because a pattern which I saw a lot in real life production
codie: it's very easy to install pypy in a Dockerfile, you just
wget&unpack the tarball. If we require libpypy3.so so be in a system-wide
directory, people will have to figure out themselves how to "install" pypy
and/or we would need to provide an "install" script which will be very
complex because of the zillions of slightly different details of linux
distros

So, I'm -1 to remove the possibility of $localdir/libpypy3-c.so.
Maybe a solution which solves both problems is a compilation flag:
1. by default, we link to $localdir/libpypy3-c.so as we do now (and maybe
we could even rename the file nowadats. The '-c' suffix is a relic of the
past and it's probably no longer needed);
2. if you translate pypy with a special option, we link to a system-wide
libpypy38.so, so that distros can compile a pypy which suits their needs

Also, note that an poor's man (2) is already possible nowadays: we
link to $localdir/libpypy3-c.so because
pypy3-c it has an rpath set to $ORIGIN, but I think you can just patch the
binary to remove it:
https://stackoverflow.com/questions/13769141/can-i-change-rpath-in-an-already-compiled-binary

ciao,
Anto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/pypy-dev/attachments/20211204/939c8a07/attachment.html>

From mgorny at gentoo.org  Sat Dec  4 05:31:24 2021
From: mgorny at gentoo.org (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=)
Date: Sat, 04 Dec 2021 11:31:24 +0100
Subject: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
In-Reply-To: <CAMSv6X14w9W5XPX-WFuy9AKJXUQyWv+YT+XDHmNDJy4dike3AA@mail.gmail.com>
References: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>
 <9ea07e2404fc3dbc313b7b85660c19464973dbf7.camel@gentoo.org>
 <CAMSv6X14w9W5XPX-WFuy9AKJXUQyWv+YT+XDHmNDJy4dike3AA@mail.gmail.com>
Message-ID: <59ff2f2d2aed35a09f2c0276442dcce18caf954d.camel@gentoo.org>

On Thu, 2021-12-02 at 23:48 +0100, Armin Rigo wrote:
> Hi,
> 
> On Thu, 2 Dec 2021 at 21:10, Micha? G?rny <mgorny at gentoo.org> wrote:
> 
> > > 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced
> > 
> > TBH I've never really understood the purpose of this file.
> > We've stopped using it at some point and nothing really changed for us.
> 
> It is needed for "embedding" situations, which nowadays means
> https://cffi.readthedocs.io/en/latest/embedding.html.  If we want to
> go the full CPython way, we need to rename and move
> "$localdir/libpypy3-c.so" to something like "/usr/lib/libpypy38.so"
> and have /usr/bin/pypy3.8 be a program that is linked to
> "libpypy38.so" instead of "$localdir/libpypy3-c.so".  This would have
> consequences for my own habits---e.g. the executable would no longer
> find its .so if it lives simply in the same directory.  Arguably not a
> big issue :-)  A freshly translated pypy would not work without
> LD_LIBRARY_PATH tricks, just like a freshly compiled CPython does not.
> The CPython solution to that annoyance is to compile statically by
> default, if no option is passed to ./configure; you need to give the
> --enable-shared option explicitly, and all distributions do.  So if we
> want to copy all of that for PyPy, we could.

Well, I'm not sure how portable it is (but quite likely portable to all
major platforms) but you can use something like

  -Wl,-rpath,$ORIGIN/../lib64

to make PyPy find the library in the correct directory relative to its
executable.  I think this shouldn't cause any major issues neither for
running from extracted tarball, nor for system-wide installs.

-- 
Best regards,
Micha? G?rny


From matti.picus at gmail.com  Sat Dec  4 13:18:03 2021
From: matti.picus at gmail.com (Matti Picus)
Date: Sat, 4 Dec 2021 20:18:03 +0200
Subject: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
In-Reply-To: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>
References: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>
Message-ID: <0af2df4f-08d2-82e1-f6a1-c12f4ca477ef@gmail.com>

On 2/12/21 8:45 pm, Miro Hron?ok wrote:

> Hello PyPy folks,
>
> I co-maintain the PyPy and CPython interpreter packages in Fedora


Thanks for supporting PyPy


> We basically took the advice from 
> https://doc.pypy.org/en/latest/build.html
>

That documentation may now need some updating.

> Several different versions of pypy (e.g. 3.6 and 3.7) could be 
> installed alongside each other, as long as only one of them "owned" 
> /usr/bin/pypy3.
>
>
> This e.g. means that the standard library is installed in:
>
> ?/usr/lib64/pypy3.8/lib/pypy3.8/
>
> Or the header files are in:
>
> ?/usr/lib64/pypy3.8/include/pypy3.8/
>
> Double namespacing like this seems a bit overengineered, and since the 
> idea was to resemble CPython, I was aiming at this setup:
> ...


It seems reasonable that PyPy should follow CPython's lead in layout.

> Which would be the exact copy of our CPython setup | sed s/python/pypy/
>
Indeed, that is the goal.


> However, I found 4 problems:
>
> 1) The PyPy install scheme only has "lib", not "lib64"
>
> I think we could sort this by a tiny patch that we used to carry in 
> CPython 3.8 before configurable platlibdir was introduced in 3.9.
>
If it is part of CPython 3.9, it should be part of PyPy 3.9 as well. Can 
you check that the latest nightly at 
https://buildbot.pypy.org/nightly/py3.9/ indeed no longer needs that change?

>
> 2) The /usr/include/PyPy.h file is *not* namespaced with PyPy Python 
> version
>
> I don't know if we even need to ship this file. It seems deprecated. 
> Maybe it can be in /usr/include/pypy3.8/PyPy.h?
>
I was confused by this as well. As Armin pointed out, it was part of the 
older cffi embedding code, and I think it can be dropped. See 
https://foss.heptapod.net/pypy/pypy/-/issues/3608
>
> 3) /usr/bin/pypy3.8 is a symbolic link to /usr/bin/pypy3
>
> I would expect it to be the other way around


This makes sense. For testing, we need pypy3, but can create it as a 
symlink already as part of the build.

>
> 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems 
> misplaced
>
> This is the one thing I cannot wrap my head around. I do not wish to 
> install this file "globally" to /usr/bin


This also makes sense and some ideas were floated in other responses to 
this thread. It would be best if someone with a plan could open an issue 
for discussion, and even better a PR.

>
> So, I am asking on this list: How should we package PyPy 3.8 for 
> Fedora? Do we need to install it to /usr/lib64/pypy3.8/ prefix even 
> thou most (but not all) of the directories within would contain a 
> "pypy3.8" subdirectory? Or is there a better option I do not yet see?
>
While it is too late for pypy3.8, now is the time to tweak pypy3.9. I am 
not sure what support you expect from PyPy to recreate the same layout 
as the one used in CPython. Are there patches we can upstream?


There are other repackagers: debian, BSD, and conda-forge, so any 
changes would need to be done with an eye toward not breaking their 
workflow as well.


Matti


From mhroncok at redhat.com  Mon Dec  6 06:30:18 2021
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=)
Date: Mon, 6 Dec 2021 12:30:18 +0100
Subject: [pypy-dev] Installation layout of the PyPy 3.8 Fedora package
In-Reply-To: <0af2df4f-08d2-82e1-f6a1-c12f4ca477ef@gmail.com>
References: <bb52d1bb-3dd2-86a0-9f93-3409a343e058@redhat.com>
 <0af2df4f-08d2-82e1-f6a1-c12f4ca477ef@gmail.com>
Message-ID: <816e9a88-9945-7c0d-0f26-f7b94f37c063@redhat.com>

On 04. 12. 21 19:18, Matti Picus wrote:
> On 2/12/21 8:45 pm, Miro Hron?ok wrote:
> 
>> Hello PyPy folks,
>>
>> I co-maintain the PyPy and CPython interpreter packages in Fedora
> 
> 
> Thanks for supporting PyPy
> 
> 
>> We basically took the advice from https://doc.pypy.org/en/latest/build.html
>>
> 
> That documentation may now need some updating.
> 
>> Several different versions of pypy (e.g. 3.6 and 3.7) could be installed 
>> alongside each other, as long as only one of them "owned" /usr/bin/pypy3.
>>
>>
>> This e.g. means that the standard library is installed in:
>>
>> ?/usr/lib64/pypy3.8/lib/pypy3.8/
>>
>> Or the header files are in:
>>
>> ?/usr/lib64/pypy3.8/include/pypy3.8/
>>
>> Double namespacing like this seems a bit overengineered, and since the idea 
>> was to resemble CPython, I was aiming at this setup:
>> ...
> 
> 
> It seems reasonable that PyPy should follow CPython's lead in layout.
> 
>> Which would be the exact copy of our CPython setup | sed s/python/pypy/
>>
> Indeed, that is the goal.

OK, let's focus on this goal for PyPy 3.9.

>> However, I found 4 problems:
>>
>> 1) The PyPy install scheme only has "lib", not "lib64"
>>
>> I think we could sort this by a tiny patch that we used to carry in CPython 
>> 3.8 before configurable platlibdir was introduced in 3.9.
>>
> If it is part of CPython 3.9, it should be part of PyPy 3.9 as well. Can you 
> check that the latest nightly at https://buildbot.pypy.org/nightly/py3.9/ 
> indeed no longer needs that change?

The standard library is prepared for this, but when building CPython 3.9+, we 
pass --with-platlibdir=lib64 to configure. I wouldn't know where to pas this to 
PyPy build.

>> 2) The /usr/include/PyPy.h file is *not* namespaced with PyPy Python version
>>
>> I don't know if we even need to ship this file. It seems deprecated. Maybe it 
>> can be in /usr/include/pypy3.8/PyPy.h?
>>
> I was confused by this as well. As Armin pointed out, it was part of the older 
> cffi embedding code, and I think it can be dropped. See 
> https://foss.heptapod.net/pypy/pypy/-/issues/3608

Ack.

>> 3) /usr/bin/pypy3.8 is a symbolic link to /usr/bin/pypy3
>>
>> I would expect it to be the other way around
> 
> 
> This makes sense. For testing, we need pypy3, but can create it as a symlink 
> already as part of the build.

I can easily change the symlink order downstream. But if done upstream, it 
would make more sense.

>> 4) The /usr/bin/libpypy3-c.so file is *not* namespaced and seems misplaced
>>
>> This is the one thing I cannot wrap my head around. I do not wish to install 
>> this file "globally" to /usr/bin
> 
> 
> This also makes sense and some ideas were floated in other responses to this 
> thread. It would be best if someone with a plan could open an issue for 
> discussion, and even better a PR.

Well, I have a plan that would work for Fedora (have 
/usr/lib64/libpypy3.9.so(.1.0)), but I am not sure how to implement it.

>> So, I am asking on this list: How should we package PyPy 3.8 for Fedora? Do 
>> we need to install it to /usr/lib64/pypy3.8/ prefix even thou most (but not 
>> all) of the directories within would contain a "pypy3.8" subdirectory? Or is 
>> there a better option I do not yet see?
>>
> While it is too late for pypy3.8, now is the time to tweak pypy3.9. I am not 
> sure what support you expect from PyPy to recreate the same layout as the one 
> used in CPython.

OK, so for me, there are now 2 problems :)

1) How to package PyPy 3.8.
2) Help make PyPy 3.9 more like CPython 3.9 in this regard.


I suppose for (1) we can live with "weird" paths for a while.

> Are there patches we can upstream?

Not yet.

-- 
Miro Hron?ok
-- 
Phone: +420777974800
IRC: mhroncok


From teammember0x01 at gmail.com  Sat Dec 18 00:17:55 2021
From: teammember0x01 at gmail.com (M A)
Date: Sat, 18 Dec 2021 00:17:55 -0500
Subject: [pypy-dev] Feature request: support for class variables and class
 functions
Message-ID: <A942FE13-1C78-4855-8C59-F7C698FC6A21@gmail.com>

Hi I ask that support for class variables and class functions be added to RPython. I was recently trying to work on some code for PyPy and found out this feature wasn't supported. This feature is a nice feature to have and would benefit the PyPy community.

Thank you.