[Python-ideas] Defining an easily installable "Recommended baseline package set"

Tue Oct 31 12:26:27 EDT 2017

OK, go ahead and write the PEP! I'm actually happy with the responses you
gave, so your last email will make a good start for some of the contents of
the PEP.

On Tue, Oct 31, 2017 at 9:19 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 1 November 2017 at 00:53, Guido van Rossum <guido at python.org> wrote:
>
>> On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>> On 31 October 2017 at 02:29, Guido van Rossum <guido at python.org> wrote:
>>>
>>>> What's your proposed process to arrive at the list of recommended
>>>> packages?
>>>>
>>>
>>> I'm thinking it makes the most sense to treat inclusion in the
>>> recommended packages list as a possible outcome of proposals for standard
>>> library inclusion, rather than being something we'd provide a way to
>>> propose specifically.
>>>
>>
>> I don't think that gets you off the hook for a process proposal. We need
>> some criteria to explain why a module should be on the recommended list --
>> not just a ruling as to why it shouldn't be in the stdlib.
>>
>
> The developer guide already has couple of sections on this aspect:
>
> * https://devguide.python.org/stdlibchanges/#acceptable-types-of-modules
> * https://devguide.python.org/stdlibchanges/#requirements
>
> I don't think either of those sections is actually quite right (since
> we've approved new modules that wouldn't meet them), but they're not
> terrible as a starting point in general, and they're accurate for the
> recommended packages use case.
>
>
>> We'd only use it in cases where a proposal would otherwise meet the
>>> criteria for stdlib inclusion, but the logistics of actually doing so don't
>>> work for some reason.
>>>
>>
>> But that would exclude most of the modules you mention below, since one
>> of the criteria is that their development speed be matched with Python's
>> release cycle. I think there must be some form of "popularity" combined
>> with "best of breed". In particular I'd like to have a rule that explains
>> why flask and Django would never make the list. (I don't know what that
>> rule is, or I would tell you -- my gut tells me it's something to do with
>> having their own community *and* competing for the same spot.)
>>
>
> The developer guide words this as "The module needs to be considered
> best-of-breed.". In some categories (like WSGI frameworks), there are
> inherent trade-offs that mean there will *never* be a single best-of-breed
> solution, since projects like Django, Flask, and Pyramid occupy
> deliberately different points in the space of available design decisions.
>
> Running the initial 5 proposals through that filter:
>>>
>>> * six: a cross-version compatibility layer clearly needs to be outside
>>> the standard library
>>>
>>
>> Hm... Does six still change regularly? If not I think it *would* be a
>> candidate for actual stdlib inclusion. Just like we added u"..." literals
>> to Python 3.4.
>>
>
> It still changes as folks porting new projects discover additional
> discrepancies between the 2.x and 3.x standard library layouts and
> behaviour (e.g. we found recently that the 3.x subprocess module's
> emulation of the old commands module APIs actually bit shifts the status
> codes relative to the 2.7 versions). The rate of change has definitely
> slowed down a lot since the early days, but it isn't zero.
>
> In addition, the only folks that need it are those that already care about
> older versions of Python - if you can start with whatever the latest
> version of Python is, and don't have any reason to support users still
> running older version, you can happily pretend those older versions don't
> exist, and hence don't need a compatibility library like six. As a separate
> library, it can just gracefully fade away as folks stop depending on it as
> they drop Python 2.7 support. By contrast, if we were to bring it into the
> 3.x standard library, then we'd eventually have to figure out when we could
> deprecate and remove it again.
>
>
>> * setuptools: we want to update this in line with the PyPA interop specs,
>>> not the Python language version
>>>
>>
>> But does that exclude stdlib inclusion? Why would those specs change, and
>> why couldn't they wait for a new Python release?
>>
>
> The specs mainly change when we want to offer publishers new capabilities
> while still maintaining compatibility with older installation clients (and
> vice-versa: we want folks still running Python 2.7 to be able to publish
> wheel files and use recently added metadata fields like
> Description-Content-Type).
>
> The reason we can't wait for new Python releases is because when we add
> such things, we need them to work on *all* supported Python releases
> (including 2.7 and security-release-only 3.x versions).
>
> There are also other drivers for setuptools updates, which include:
>
> - operating system build toolchain changes (e.g. finding new versions of
> Visual Studio or XCode)
> - changes to PyPI's operations (e.g. the legacy upload API getting turned
> off due to persistent service stability problems, switching to HTTPS only
> access)
>
> With setuptools as a separate project, a whole lot of package publication
> problems can be solved via "pip install --upgrade setuptools wheel" in a
> virtual environment, which is a luxury we don't have with plain distutils.
>
>
>>  * cffi: updates may be needed for PyPA interop specs, Python
>> implementation updates or C language definition updates
>>
>> Hm, again, I don't recall that this was debated -- I think it's a failure
>> that it's not in the stdlib.
>>
>
> A couple of years ago I would have agreed with you, but I've spent enough
> time on packaging problems now to realise that cffi actually qualifies as a
> build tool due to the way it generates extension module wrappers when used
> in "out-of-line" mode.
>
> Being outside the standard library means that cffi still has significant
> flexibility to evolve how it separates its buildtime functionality from its
> runtime functionality, and may eventually be adjusted so that only
> "_cffi_backend" needs to be installed at runtime for the out-of-line
> compilation mode, without the full C header parsing and inline extension
> module compilation capabilities of CFFI itself (see
> https://cffi.readthedocs.io/en/latest/cdef.html#preparing-
> and-distributing-modules for the details). Being separate also means that
> cffi can be updated to generate more efficient code even for existing
> Python versions.
>
>
>> * requests: updates are more likely to be driven by changes in network
>>> protocols and client platform APIs than Python language changes
>>>
>>
>> Here I agree. There's no alternative (except aiohttp, but that's
>> asyncio-based) and it can't be in the stdlib because it's actively being
>> developed.
>>
>>
>>> * regex: we don't want two regex engines in the stdlib, transparently
>>> replacing _sre would be difficult, and _sre is still good enough for most
>>> purposes
>>>
>>
>> I think this needn't be recommended at all. For 99.9% of regular
>> expression uses, re is just fine. Let's just work on a strategy for
>> introducing regex into the stdlib.
>>
>
> Given your informational PEP suggestion below, I'd probably still include
> it, but in a separate section from the others (e.g. the others might be
> listed as "Misaligned Feature Release Cycles", which is an inherent
> logistical problem that no amount of coding can fix, while regex would
> instead be categorised as "Technical Challenges").
>
>
>> Of the 5, I'd suggest that regex is the only one that could potentially
>>> still make its way into the standard library some day - it would just
>>> require someone with both the time and inclination to create a CPython
>>> variant that used _regex instead of _sre as the default regex engine, and
>>> then gathered evidence to show that it was "compatible enough" with _sre to
>>> serve as the default engine for CPython.
>>>
>>> For the first four, there are compelling arguments that their drivers
>>> for new feature additions are such that their release cycles shouldn't ever
>>> be tied to the rate at which we update the Python language definition.
>>>
>>
>> As you can tell from my arguing, the reasons need to be written up in
>> more detail.
>>
>>
>>> And is it really just going to be a list of names, or is there going to
>>>> be some documentation (about the vetting, not about the contents of the
>>>> packages) for each name?
>>>>
>>>
>>> I'm thinking a new subsection in https://docs.python.org/devgui
>>> de/stdlibchanges.html for "Recommended Third Party Packages" would make
>>> sense, covering what I wrote above.
>>>
>>
>> That's too well hidden for my taste.
>>
>>
>>> It also occurred to me that since the recommendations are independent of
>>> the Python version, they don't really belong in the version specific
>>> documentation.
>>>
>>
>> But that doesn't mean they can't (also) be listed there. (And each
>> probably has its version dependencies.)
>>
>>
>>> While the Developer's Guide isn't really the right place for the list
>>> either (except as an easier way to answer "Why isn't <X> in the standard
>>> library?" questions), it could be a good interim option until I get around
>>> to actually writing a first draft of https://github.com/python/redi
>>> stributor-guide/ (which I was talking to Barry about at the dev sprint,
>>> but didn't end up actually creating any content for since I went down a
>>> signal handling rabbit hole instead).
>>>
>>
>> Hm, let's not put more arbitrary check boxes in the way of progress.
>> Maybe it can be an informational PEP that's occasionally updated?
>>
>
> If I'm correctly reading that as "Could the list of Recommended Third
> Party Packages be an informational PEP?", then I agree that's probably a
> good way to tackle it, since it will cover both the developer-centric "Why
> isn't this in the standard library yet?" aspect *and* the "Redistributors
> should probably provide this" aspect.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20171031/90c52a74/attachment-0001.html>