From encukou at gmail.com  Fri Apr 15 09:15:07 2016
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 15 Apr 2016 15:15:07 +0200
Subject: [Import-SIG] [Python-ideas] Module lifecycle: simple
 alternative to PEP 3121/PEP 489
In-Reply-To: <CANawmycgasmDKwqtgutkk4ZkVkPVHBCjZg5GQwDDbNqYPj62XA@mail.gmail.com>
References: <CANawmycWLWcRbL+a=c4e2b1-bM=-xUYbSxsZRpqLdEaUyWJSdg@mail.gmail.com>
 <CADiSq7eK3brB8YhpvaCPeVs12mNn6TD+WQOEaXEpE5bgKjau=Q@mail.gmail.com>
 <CANawmydKh8B3MCoybo54g1GifxMo9FWKSXM8P5kav89OuMj2pQ@mail.gmail.com>
 <5710B36A.2060709@gmail.com>
 <CANawmycgasmDKwqtgutkk4ZkVkPVHBCjZg5GQwDDbNqYPj62XA@mail.gmail.com>
Message-ID: <5710E95B.3070900@gmail.com>

Let's move the discussion to import-sig, as Nick explained in the other
subthread. Please drop python-ideas from CC when you reply.

On 04/15/2016 12:58 PM, Nikita Nemkin wrote:
> On Fri, Apr 15, 2016 at 2:24 PM, Petr Viktorin <encukou at gmail.com> wrote:
>>>
>>> 1) define a new "calling convention" flag like METH_GLOBALS.
>>> 2) store module ref in PyCFunctionObject.m_module
>>>     (currently it stores only the module name)
>>
>> Wouldn't that break backwards compatibility, though?
> 
> It will, and I consider this level of breakage acceptable. Alternatively,
> another field can be added to this struct.
> 
>> My planned approach is a bit more flexible:
>> - Add a reference to the module (ht_module) to heap types
>> - Create a calling convention METH_METHOD, where methods are passed the
>> class that defines the method (which might PyTYPE(self) or a superclass
>> of it)
>>
>> This way methods can get both module state and the class they are
>> defined on, and the replacement for PyModule_State is two indirections.
> 
> I've read the linked import-sig thread and realized the depth of issues
> involved...
> Those poor heap type methods don't even have access their own type
> pointer! In the light of that, your variant is more useful than mine.
> 
> Still, without a good slot support option, new METH_X conventions
> don't look attractive at all. Such fundamental change, but only solves
> half of the problem.

Well, it solves the problem for methods that have calling conventions,
and I'm pretty sure by now that a full solution will need this *plus*
another solution for slots. So I'm looking at the problems as two
separate parts, and I also think that when it comes to writing PEPs,
having two separate PEPs would make this more understandable.

> Also, MRO walking is actually not as slow as it seems.
> Checking Py_TYPE(self) and Py_TYPE(self)->tp_base *inline*
> will minimize performance overhead in the (very) common case.

I think so as well. This would mean that module state access in named
methods is fast; in slot methods it's possible (and usually fast
*enough*), and the full solution with __typeslots__ would still be possible.

> If non-slot methods had a suitable static anchor (the equivalent
> of slot function address for slots), they could use MRO walking too.

I think a new METH_* calling style and explicit pointers is a better
alternative here.



From ncoghlan at gmail.com  Mon Apr 25 22:19:04 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 26 Apr 2016 12:19:04 +1000
Subject: [Import-SIG] File based configuration for the import system
Message-ID: <CADiSq7e7_Fq7akWi10hqXwibA17757RwuNNjAXHV08WCW7iWAA@mail.gmail.com>

This isn't a fully thought out idea, but I figured it was worth posting
here as a potentially improved motivation for the interpreter bootstrapping
improvements proposed in PEP 432.

Currently, the proposed motivation there is for a system Python executable
that has different defaults from the normal CPython runtime:
https://www.python.org/dev/peps/pep-0432/#a-system-python-executable

It turns out that isn't a particularly compelling motivation for the degree
of change proposed, since Christian Heimes was able to implement the -I
switch for isolated mode without too much hassle, and you can fairly easily
provide a wrapper script that implicitly sets -I for all invocations.

For me though, one of the other motivations behind the proposed changes in
PEP 432 is getting to replace the current sys.path calculation code in
getpath.c and getpathp.c with Python code in importlib._bootstrap or
(depending on boot sequence details) importlib._bootstrap_external.

Once we did that, there would be a range of ideas (like allowing additional
path handling customisations in venv configuration files) that become
significantly more feasible to implement and maintain.

So, does that sound reasonable to folks as a revised primary motivation for
that PEP: moving the default sys.path calculation code to Python for ease
of reuse across implementations, and for ease of maintenance and
enhancement within CPython itself?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20160426/cdb83984/attachment.html>

From solipsis at pitrou.net  Tue Apr 26 11:08:29 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Apr 2016 17:08:29 +0200
Subject: [Import-SIG] File based configuration for the import system
References: <CADiSq7e7_Fq7akWi10hqXwibA17757RwuNNjAXHV08WCW7iWAA@mail.gmail.com>
Message-ID: <20160426170829.0966d543@fsol>

On Tue, 26 Apr 2016 12:19:04 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> For me though, one of the other motivations behind the proposed changes in
> PEP 432 is getting to replace the current sys.path calculation code in
> getpath.c and getpathp.c with Python code in importlib._bootstrap or
> (depending on boot sequence details) importlib._bootstrap_external.
> 
> Once we did that, there would be a range of ideas (like allowing additional
> path handling customisations in venv configuration files) that become
> significantly more feasible to implement and maintain.
> 
> So, does that sound reasonable to folks as a revised primary motivation for
> that PEP: moving the default sys.path calculation code to Python for ease
> of reuse across implementations, and for ease of maintenance and
> enhancement within CPython itself?

Platonically, that sounds like a good idea, but obviously I'm not the
one doing the work (and good luck with the bootstrapping issues ;-)).

Regards

Antoine.



From guido at python.org  Tue Apr 26 11:26:21 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 26 Apr 2016 08:26:21 -0700
Subject: [Import-SIG] File based configuration for the import system
In-Reply-To: <CADiSq7e7_Fq7akWi10hqXwibA17757RwuNNjAXHV08WCW7iWAA@mail.gmail.com>
References: <CADiSq7e7_Fq7akWi10hqXwibA17757RwuNNjAXHV08WCW7iWAA@mail.gmail.com>
Message-ID: <CAP7+vJKvZBUMW55vWF44LEy1xaQih_8407DMto0aqeLh+jVUjg@mail.gmail.com>

On Mon, Apr 25, 2016 at 7:19 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> This isn't a fully thought out idea, but I figured it was worth posting
> here as a potentially improved motivation for the interpreter bootstrapping
> improvements proposed in PEP 432.
>
> Currently, the proposed motivation there is for a system Python executable
> that has different defaults from the normal CPython runtime:
> https://www.python.org/dev/peps/pep-0432/#a-system-python-executable
>
> It turns out that isn't a particularly compelling motivation for the
> degree of change proposed, since Christian Heimes was able to implement the
> -I switch for isolated mode without too much hassle, and you can fairly
> easily provide a wrapper script that implicitly sets -I for all invocations.
>
> For me though, one of the other motivations behind the proposed changes in
> PEP 432 is getting to replace the current sys.path calculation code in
> getpath.c and getpathp.c with Python code in importlib._bootstrap or
> (depending on boot sequence details) importlib._bootstrap_external.
>

Isn't code that lives in importlib._bootstrap just about as hard to
maintain as code written in C?


> Once we did that, there would be a range of ideas (like allowing
> additional path handling customisations in venv configuration files) that
> become significantly more feasible to implement and maintain.
>
> So, does that sound reasonable to folks as a revised primary motivation
> for that PEP: moving the default sys.path calculation code to Python for
> ease of reuse across implementations, and for ease of maintenance and
> enhancement within CPython itself?
>

Much of the sys.path calculation (everything related to packages) is
already in site.py. And you can skip it with -S.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20160426/2c7d529c/attachment.html>

From ncoghlan at gmail.com  Tue Apr 26 22:36:31 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Apr 2016 12:36:31 +1000
Subject: [Import-SIG] File based configuration for the import system
In-Reply-To: <CAP7+vJKvZBUMW55vWF44LEy1xaQih_8407DMto0aqeLh+jVUjg@mail.gmail.com>
References: <CADiSq7e7_Fq7akWi10hqXwibA17757RwuNNjAXHV08WCW7iWAA@mail.gmail.com>
 <CAP7+vJKvZBUMW55vWF44LEy1xaQih_8407DMto0aqeLh+jVUjg@mail.gmail.com>
Message-ID: <CADiSq7c6ZesaWBUJeeGACspOyB9qi3saiTP6G7tHC2PjhD0LtQ@mail.gmail.com>

On 27 April 2016 at 01:26, Guido van Rossum <guido at python.org> wrote:
> On Mon, Apr 25, 2016 at 7:19 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>
>> This isn't a fully thought out idea, but I figured it was worth posting
>> here as a potentially improved motivation for the interpreter bootstrapping
>> improvements proposed in PEP 432.
>>
>> Currently, the proposed motivation there is for a system Python executable
>> that has different defaults from the normal CPython runtime:
>> https://www.python.org/dev/peps/pep-0432/#a-system-python-executable
>>
>> It turns out that isn't a particularly compelling motivation for the
>> degree of change proposed, since Christian Heimes was able to implement the
>> -I switch for isolated mode without too much hassle, and you can fairly
>> easily provide a wrapper script that implicitly sets -I for all invocations.
>>
>> For me though, one of the other motivations behind the proposed changes in
>> PEP 432 is getting to replace the current sys.path calculation code in
>> getpath.c and getpathp.c with Python code in importlib._bootstrap or
>> (depending on boot sequence details) importlib._bootstrap_external.
>
> Isn't code that lives in importlib._bootstrap just about as hard to maintain
> as code written in C?

It's somewhere in the middle - for _bootstrap, you're limited to the
builtins and the injected os and sys modules, while
_bootstrap_external is free to import any builtin and frozen modules
it needs.

The main pain with getpath.c in its current form is that when it runs
we don't even have the builtin *types* available, so it's all written
in terms of wchar_t* and char*. Even being able to replace those with
PyUnicode and PyList would be a respectable step forward, but still
not quite as nice as stepping up to the Python level and also
benefiting from the implicit memory management.

>> Once we did that, there would be a range of ideas (like allowing
>> additional path handling customisations in venv configuration files) that
>> become significantly more feasible to implement and maintain.
>>
>> So, does that sound reasonable to folks as a revised primary motivation
>> for that PEP: moving the default sys.path calculation code to Python for
>> ease of reuse across implementations, and for ease of maintenance and
>> enhancement within CPython itself?
>
> Much of the sys.path calculation (everything related to packages) is already
> in site.py. And you can skip it with -S.

Aye, the C level initialisation only covers finding the standard
library and the initial setting of sys.path[0].

That's one of the big reasons this refactoring proposal has lingered
for so long - the status quo for bootstrapping the interpreter isn't
particularly elegant, but it *works*, and most of the things folks
want to do can already be achieved in other ways.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia