[Distutils] setup_requires for dev environments

[Donald Stufft]

> On Mar 16, 2015, at 12:32 PM, Daniel Holth <dholth at gmail.com> wrote:
> 
> Problem: Users would like to be able to import stuff in setup.py. This
> could be anything from a version fetcher to a replacement for
> distutils itself. However, if setup.py is the only place to specify
> these requirements there's a bit of a chicken and egg problem, unless
> they have unusually good setuptools knowledge, especially if you want
> to replace the  entire setup() implementation.

So you *can* import things inside of a setup.py today, you just have to delay
the imports by subclassing a command. You can see an example of doing this with
the example command given for pytest in the documentation for pytest [1]. So
this problem essentially boils down to people wanting to import at the module
scope of their setup.py instead of needing to delay the import for it.

This particular problem I believe the solution is worse than the problem. There
is a supported solution *today* they can use and it works and importantly it
works in all versions of pip and setuptools that I'm aware of. It'a also going
to continue to work for years and years.

[1] http://pytest.org/latest/goodpractises.html#integration-with-setuptools-test-commands

> 
> Problem: Having easy_install do it is not what people want and misses
> some important use cases.

This problem I'm aware of, and as I said in my previous email I believe a
better interim solution to this problem is to have pip forcibly take control
over the setup_requires inside of a setup.py. This has the advantage of
requiring nobody to make any changes to their packages so it'll work on all
new and existing projects that rely on setup_requires and it's completely
self contained within pip.

> 
> Problem: Based on empirical evidence PEP 426 will never be done. Its
> current purpose is to shut down discussion of pragmatic solutions.

This is just FUD and I would appreciate it if you'd stop repeating it. It's
only been ~3 months since PEP 440 was completed and released inside of pip,
setuptools, and PyPI. I've since switched back over to focusing primarily on
getting Warehouse ready to replace PyPI so the bulk of my time is being spent
focusing on that. After that's done my plan is to likely switch back to working
on putting PEP 426 through the same hard look that I put PEP 440 though to try
and iron out as many problems as I can find before implementing it and pushing
it out to people.

The bulk of the effort of pushing the standards, pip, and PyPI through is done
by a handful of people, and of those handful I believe that the largest share
is done by myself. That's not to toot my own horn or any such nonsense but to
simply state the fact that the available bandwidth of people able and willing
to work on problems is low. However the things we bless here as official are
things which need to be able to last for a decade or more, which means that
they do need careful consideration before we bless them.

> 
> Solution: Add requirements to setup.cfg, installed by pip before
> setup.py is touched.
> 
> Old pip: requirements will not be installed. This is what happens now
> if anyone tries to use a non-stdlib module in setup.py, and plenty of
> packages do. User will have to install the extra requirements manually
> before running setup.py.
> 
> Proposed pip: requirements will be installed. Hooray!
> 
> Result: Users will begin writing packages that only work with new pip.
> 
> If we implement this, users will do the same thing they are already
> doing (import non-stdlib packages inside setup.py), only more often.
> 
> 
> On Mon, Mar 16, 2015 at 12:03 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>>> On Mar 16, 2015, at 11:24 AM, Daniel Holth <dholth at gmail.com> wrote:
>>> 
>>> We could support this syntax right now. It's so simple. Don't deride
>>> it as a pseudo standard, turn it into an actual standard and praise it
>>> as something practical that will not take years to implement. Then
>>> after those years have passed and the new PEP actually works and has a
>>> distutils replacement to drive it, deprecate the old standard.
>>> 
>>> If you can come up with something better that can ship before 2016, by
>>> all means.
>>> 
>>> [metadata]
>>> setup-requires = cffi
>>>   pip
>>>   pycparser >= 2.10
>>> 
>>> 
>>> 
>>> https://bitbucket.org/dholth/setup-requires
>> 
>> It is a psuedo standard unless it is backed by a defined standard. That's not
>> a derision, it's just a fact.
>> 
>> The first step is to determine *what* the problem is that it's actually
>> attempting to solve. That's not clear to me currently other than some vague
>> statements about pain, but what pain? What's actually occuring and how does
>> this address those problems?
>> 
>> After figuring out what the actual problem is, we can look at the proposed
>> solution and see how well it actually solves that problem, if there is maybe
>> a better solution to the problem, and if the benefits outweigh the costs or
>> not.
>> 
>> The ease of implementation is not the only factor in deciding if something is
>> a good idea or not. We have to take into account forwards and backwards
>> compatiblity. If we implement it and people start to depend on it then it's
>> something that's going to have to exist forever, and any new installer is going
>> to have to replicate that behavior. If people don't depend on it then
>> implementing it was a waste of time and effort.
>> 
>> For instance, if the problem is "when setuptools does the install, then things
>> get installed differently, with different options, SSL certs, proxies, etc"
>> then I think a better solution is that pip does terrible hacks in order to
>> forcibly take control of setup_requires from setuptools and installs them into
>> a temporary directory (or something like that). That is something that would
>> require no changes on the part of authors or people installing software, and
>> is backwards compatible with everything that's already been published using
>> setup_requires. That's the primary problem that I'm aware of.
>> 
>> If I try and guess at other problems people might be solving, one might be
>> that in order to use setup_requires you have to delay your imports until after
>> the setup_requires get processed. This typically means you do things like
>> imports inside of functions that get called as part of the setup.py
>> build/install process. This isn't the most fun way to write software, however
>> it works. Specifying the setup_requires in a static location outside would
>> enable pip to then install those things into a temporary directory prior to
>> executing the setup.py which then lets you do imports and other related work
>> at the module scope of the setup.py. This particular problem I'm not sure it's
>> worth fixing with a stop gap solution. It would require breaking the entire
>> existing install base of installation tools if anyone actually took advantage
>> of this fact, which I don't think is generally worth it to have slightly nicer
>> use of things in your setup.py (essentially allowing you to import at the top
>> level and not require subclassing command classes).
>> 
>> So yea, what's the actual problem that this is attempting to solve?
>> 
>> ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150316/eff2287c/attachment.sig>