[Distutils] Q about best practices now (or near future)

[Daniel Holth]

On Sat, Jul 20, 2013 at 2:10 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 20 July 2013 01:47, PJ Eby <pje at telecommunity.com> wrote:
>> On Fri, Jul 19, 2013 at 9:10 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Right, I think the reasonable near term solutions are for pip to either:
>>>
>>> 1. generate zc.buildout style wrappers with absolute paths to avoid
>>> the implied runtime dependency
>>> 2. interpret use of script entry points as an implied dependency on
>>> setuptools and install it even if not otherwise requested
>>>
>>> Either way, pip would need to do something about its *own* command
>>> line script, which heavily favours option 1
>>
>> Option 1 also would address some or all of the startup performance complaint.
>>
>> It occurs to me that it might actually be a good idea *not* to put the
>> script wrappers in the standard entry points file, even if that's what
>> setuptools does right now: if lots of packages use that approach,
>> it'll slow down the effective indexing for code that's scanning
>> multiple packages for something like a sqlalchemy adapter.
>>
>> (Alternately, we could use something like
>> 'exports-some.group.name.json' so that each export group is a separate
>> file; this would keep scripts separate from everything else, and
>> optimize plugin searches falling in a particular group.  In fact, the
>> files needn't have any contents; it'd be okay to just parse the main
>> .json for any distribution that has exports in the group you're
>> looking for.  i.e., the real purpose of the separation of entry points
>> was always just to avoid loading metadata for distributions that don't
>> have the kind of exports you're looking for.  In the old world, few
>> distributions exported anything, so just identifying whether a
>> distribution had exports was sufficient.  In the new world, more and
>> more distributions over time will have some kind of export, so knowing
>> *which* exports they have will become more important.)
>
> A not-so-quick sketch of my current thinking:
>
> Two new fields in PEP 426: commands and exports
>
> Like the core dependency metadata, both get generated files:
> pydist-commands.json and pydist-exports.json
>
> (As far as the performance concern goes, I think longer term we'll
> probably move to a richer installation database format that includes
> an SQLite cache file managed by the installers. But near term, I like
> the idea of being able to check "has commands or not" and "has exports
> or not" with a single stat call for the appropriate file)
>
> Rather than using the "module.name:qualified.name" format (as the PEP
> currently does for the install_hooks), "export specifiers" would be
> defined as a mapping with the following subfields:
>
>     * module
>     * qualname (as per PEP 3155)
>     * extra
>
> Both qualname and extra would be optional. "extra" indicates that the
> export is only present if that extra is installed.
>
> The top level commands field would have three subfields:
> "wrap_console", "wrap_gui" and "prebuilt". The wrap_console and
> wrap_gui subfields would both be maps of command names to export
> specifiers (i.e. requests for an installer to generate the appropriate
> wrappers), while prebuilt would be a mapping of command names to paths
> relative to the scripts directory (as strings).
>
> Note that given that Python 2.7+ and 3.2+ can execute packages with a
> __main__ submodule, the export specifier for a command entry *may*
> just be the module component and it should still work.
>
> The exports field is just a rebranded and slightly rearranged
> entry_points structure: the top level keys in the hash map are "export
> groups" (defined in the same way as metadata extensions are defined)
> and the individual entries in each export group are arbitrary keys
> (meaning determined by the export group) mapping to export specifiers.
>
> With this change, I may even move the current top level
> "install_hooks" field inside the "exports" field. Even if it stay at
> the top level, the values will become export specifiers rather than
> using the entry points string format.
>
> Not sure when I'll get that tidied up and incorporated into a new
> draft of PEP 426, but I think it covers everything.
>
> For those wondering about my dividing line between "custom string
> format" and "structured data": the custom string formats in PEP 426
> should be limited to things that are likely to be passed as command
> line arguments (like requirement specifiers and their assorted
> components), or those where using structured data would be
> extraordinarily verbose (like environment markers). If I have any
> custom string formats still in there that don't fit either of those
> categories, then let me know and I'll see if I can replace them with
> structured data.
>
> Cheers,
> Nick.

It may be worth mentioning that I am not aware of any package that
uses the "entry point requires extra" feature.

IIUC pkg_resources doesn't just check whether something's installed
but attempts to add the requirements of the entry point's distribution
and any requested extras to sys.path as part of resolution.