[Distutils] setup_requires for dev environments

[Donald Stufft]

> On Mar 16, 2015, at 3:32 PM, Daniel Holth <dholth at gmail.com> wrote:
> 
> On Mon, Mar 16, 2015 at 3:19 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>>> On Mar 16, 2015, at 2:53 PM, Daniel Holth <dholth at gmail.com> wrote:
>>> 
>>> No one should be asked to learn how to extend distutils, and in
>>> practice no one knows how.
>> 
>> Some people know how, pytest figured it out, pbr figured it out. There’s
>> some documentation at https://pythonhosted.org/setuptools/setuptools.html#extending-and-reusing-setuptools
>> 
>> It is true though that it’s not particularly great documentation and
>> actually doing it is somewhat of an annoyance.
>> 
>>> 
>>> People have been begging for years for working setup_requires, far
>>> longer than I've been interested in it, and all they want to do is
>>> 
>>> import fetch_version
>>> setup(version=fetch_version(), ...)
>>> 
>>> Then they will eventually notice setup_requires has never worked the
>>> way most people expect. As a result there are too few setup.py
>>> abstractions.
>>> 
>>> The other proposal is a little bit interesting. Parse setup.py without
>>> running it, extract setup_requires, and pip install locally before
>>> running the file? It would be easy as long as the setup_requires were
>>> defined as a literal list in the setup() call, but you would have to
>>> tell people they were not allowed to write normal clever Python code.
>>> I think the gotchas would be severe…
>> 
>> I wasn’t going to try and parse the setup.py, I was going to execute it.
>> 
>> Here’s a proof of concept I started on awhile back to try and validate
>> the overall idea: https://github.com/pypa/pip/compare/develop...dstufft:eldritch-horror
>> 
>> Since then I’ve thought about it more and decided it’d probably be better
>> to instead of trying to shuffle arguments into the subprocess, have the
>> subprocess write out into a file or stdout or something what all of the
>> setup_requires are. This would require executing the setup.py 3x instead
>> of 2x like pip is currently doing.
>> 
>> This would also enable people do something like:
>> 
>> try:
>>    import fetch_version
>> except ImportError:
>>    def fetch_version():
>>        return “UNKNOWN”
>> 
>> setup(version=fetch_version(), …)
>> 
>> If they are happy with mandating that their thing can only be installed from
>> sdist with pip newer than X, because given a three pass installation (once
>> to discover setup_requires, once to write egg_info, once to actually install)
>> as long as the setup_requires list doesn’t rely on anything installed then
>> the first pass can have no real information except the setup_requires.
>> 
>> It actually wouldn’t even really be completely broken, it’d just have a nonsense
>> version number (or whatever piece of metadata can’t be located).
> 
> But it would still work in older pip if the setup requirements were
> installed already? Users would have to try/catch every import? Explain
> why this is better than reading out of a different file in the sdist,
> no matter the format? Would it let you change your setup_requires with
> Python code before the initial .egg-info is written out?

It’s better because it solves the second problem you mentioned “people
don’t want easy_install to install things” for everything that uses
setup_requires with no effort required on package authors.

And yes it would still work in older pip if the setup requirements were
installed already. It would also continue to work in setuptools/easy_install
if they were installed already.

It makes the existing mechanism somewhat better instead of trying to
replace it with a whole new mechanism. It’s also a more general solution,
for people who aren’t willing to break older things, they continue to use
setup_requires with delayed imports and they get pip installing things for
them. For people who don’t want to delay imports but are happy breaking
older things, it lets them do what they want with only a minor amount of
discomfortable (needing to catch an import error).

> 
> We've already talked too many times about my 34-line setup.py prefix
> that does exactly what I want by parsing and installing requirements
> from a config file, but people tend to complain about it not being a
> pip feature. If it would help, I could have it accept a different
> format, then it would be possible to publish
> backwards-compatible-with-pip sdists that parsed some preferable
> requirements format. If they were already installed the bw compat code
> would do nothing.
> https://bitbucket.org/dholth/setup-requires/src/03eda33c7681bc4102164c976e5a8cec3c4ffa9c/setup.py

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150316/76fa39d9/attachment-0001.sig>