From thomas at kluyver.me.uk Tue Aug 1 12:31:53 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 01 Aug 2017 17:31:53 +0100 Subject: [Distutils] status check on PEP 517 In-Reply-To: References: <1501271616.3751500.1055976240.2E1277F3@webmail.messagingengine.com> <1501275533.3769236.1056041016.23F3C467@webmail.messagingengine.com> <1501320665.888343.1056400072.59A66F97@webmail.messagingengine.com> Message-ID: <1501605113.2646382.1059777640.1267FB6C@webmail.messagingengine.com> Are we content to say that sys.path includes the source directory where the hook is run? Shall I prepare a PR against the PEP for that? On Sun, Jul 30, 2017, at 02:12 PM, Nick Coghlan wrote: > On 30 July 2017 at 02:46, Nathaniel Smith wrote: > > Or am I worrying about a non-issue and it's fine if flit imports click from > > the source tree? > > Don't worry about it too much, as the problem here isn't really any > worse than it is for normal runtime dependencies of any other project > that relies on having the current directory first in sys.path. It just > so happens that the project in question in this case is a Python > project's build system. > > Due to the preference for a flat module namespace as the default, > there are plenty of ways to hit name shadowing problems in Python, and > as Donald notes, build systems have other motives to vendor their > dependencies rather than installing them normally. > > Just switching the path order as has been suggested also doesn't solve > the problem, as it merely inverts the issue: having "some_name" > installed in site-packages would break source installations of > packages that expected to be able to import "some_name" from their own > root directory. > > If the problem does come up in practice, then there are a number of > ways for affected projects to work around it in their project > directory structure: > > 1. Use a top-level "src" directory (we may want to reserve "src" on PyPI) > 2. Use a top-level "tools" directory (we may want to reserve "tools" on > PyPI) > 3. Add a leading or trailing underscore to the local directory name > (as while that's legal for Python imports, it's prohibited for PyPI > project names, and hence will often sidestep naming conflicts with > published packages) > > Beyond that, the only approaches I'm aware of that systematically > avoid this kind of problem at the language design level are to either > use URL-based imports (ala Java or Go), or else to have separate > syntax for "system-only" and "local resolution permitted" imports (ala > C and C++), and Guido opted not to pursue either of those strategies > for Python. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Tue Aug 1 23:04:06 2017 From: dholth at gmail.com (Daniel Holth) Date: Wed, 02 Aug 2017 03:04:06 +0000 Subject: [Distutils] status check on PEP 517 In-Reply-To: <1501605113.2646382.1059777640.1267FB6C@webmail.messagingengine.com> References: <1501271616.3751500.1055976240.2E1277F3@webmail.messagingengine.com> <1501275533.3769236.1056041016.23F3C467@webmail.messagingengine.com> <1501320665.888343.1056400072.59A66F97@webmail.messagingengine.com> <1501605113.2646382.1059777640.1267FB6C@webmail.messagingengine.com> Message-ID: Works for me El mar., 1 de ago. de 2017 11:32, Thomas Kluyver escribi?: > Are we content to say that sys.path includes the source directory where > the hook is run? Shall I prepare a PR against the PEP for that? > > On Sun, Jul 30, 2017, at 02:12 PM, Nick Coghlan wrote: > > On 30 July 2017 at 02:46, Nathaniel Smith wrote: > > > Or am I worrying about a non-issue and it's fine if flit imports click > from > > > the source tree? > > > > Don't worry about it too much, as the problem here isn't really any > > worse than it is for normal runtime dependencies of any other project > > that relies on having the current directory first in sys.path. It just > > so happens that the project in question in this case is a Python > > project's build system. > > > > Due to the preference for a flat module namespace as the default, > > there are plenty of ways to hit name shadowing problems in Python, and > > as Donald notes, build systems have other motives to vendor their > > dependencies rather than installing them normally. > > > > Just switching the path order as has been suggested also doesn't solve > > the problem, as it merely inverts the issue: having "some_name" > > installed in site-packages would break source installations of > > packages that expected to be able to import "some_name" from their own > > root directory. > > > > If the problem does come up in practice, then there are a number of > > ways for affected projects to work around it in their project > > directory structure: > > > > 1. Use a top-level "src" directory (we may want to reserve "src" on PyPI) > > 2. Use a top-level "tools" directory (we may want to reserve "tools" on > > PyPI) > > 3. Add a leading or trailing underscore to the local directory name > > (as while that's legal for Python imports, it's prohibited for PyPI > > project names, and hence will often sidestep naming conflicts with > > published packages) > > > > Beyond that, the only approaches I'm aware of that systematically > > avoid this kind of problem at the language design level are to either > > use URL-based imports (ala Java or Go), or else to have separate > > syntax for "system-only" and "local resolution permitted" imports (ala > > C and C++), and Guido opted not to pursue either of those strategies > > for Python. > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Aug 2 01:11:40 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 Aug 2017 22:11:40 -0700 Subject: [Distutils] status check on PEP 517 In-Reply-To: <1501605113.2646382.1059777640.1267FB6C@webmail.messagingengine.com> References: <1501271616.3751500.1055976240.2E1277F3@webmail.messagingengine.com> <1501275533.3769236.1056041016.23F3C467@webmail.messagingengine.com> <1501320665.888343.1056400072.59A66F97@webmail.messagingengine.com> <1501605113.2646382.1059777640.1267FB6C@webmail.messagingengine.com> Message-ID: On Tue, Aug 1, 2017 at 9:31 AM, Thomas Kluyver wrote: > Are we content to say that sys.path includes the source directory where > the hook is run? Shall I prepare a PR against the PEP for that? It doesn't matter whether sys.path includes the source directory when the hook is run. At that point the hook can manipulate sys.path however it likes, just like setup.py can. What matters is whether sys.path includes the source directory before the backend is imported. And I think that even if we keep the source directory in sys.path, then there's at least a further piece of refinement needed. The obvious mechanism for adding the source directory to sys.path is by setting PYTHONPATH before invoking the sub-python that runs the build backend. That's how we put other directories on sys.path for injecting requirements, and there's a bit of text in the PEP talking about how the execution environment needs to be inherited over calls like subprocess.run([sys.executable, ...]); if this requirement also applies to the source tree's presence in sys.path, then we *have* to put the source tree in PYTHONPATH. But... this creates some special problems. Putting the source directory in PYTHONPATH is different from the familiar 'python setup.py' behavior. When you run 'python setup.py', the sequence of events is: 1) The interpreter bootstraps itself 2) The interpreter adds the script directory to the front of sys.path 3) The code in setup.py starts running, and can choose to either leave sys.path alone or modify it before it imports anything with the new sys.path But if we use PYTHONPATH to add the source tree to sys.path, then what happens is: 1) The source tree gets added to the front of sys.path 2) The interpreter bootstraps itself 3) The frontend imports the backend 4) The frontend invokes the backend hook 5) The backend starts running, and can set up whatever environment it likes before doing anything else So the difference is that with setup.py, there aren't any imports between the sys.path manipulation and handing control the script, so if the script doesn't like their path they can easily change it. And if there is an import problem, then it's because of a clash between something that the package author's setup.py did, and something they put in their package, both of which are under their control. (For example, it's quite common to see setup.py's and other scripts do a 'sys.path.pop(0)' at the top to avoid any chance of mistakes.) The sys.path manipulation is a convenience, but it doesn't actually affect anything by itself. With PYTHONPATH, the sys.path manipulation happens earlier, so you can get collisions between the source tree and the Python bootstrap logic. For example: ~$ mkdir temp ~$ cd temp ~/temp$ touch io.py setup.py # No problem running a script in this dir ~/temp$ python3 setup.py # But with PYTHONPATH... ~/temp$ PYTHONPATH=. python3 setup.py Fatal Python error: Py_Initialize: can't initialize sys standard streams AttributeError: module 'io' has no attribute 'OpenWrapper' Current thread 0x00007fd862d83700 (most recent call first): zsh: abort PYTHONPATH=. python3 setup.py I bet if someone filed a bug saying "I ran 'pip install foo' and got that error mesage" then it would take some time to figure out what went wrong :-). Of course, it's unlikely that people will have io.py files sitting around. But there's a whole list of modules that get imported automatically during interpreter startup, and which are shadowed by PYTHONPATH. For example, I just checked Github's bigquery data and found dozens of repos that have a file called 'site.py' in their root -- probably because they're a web site or something. But of course if we use PYTHONPATH then this shadows Python's internal site.py that's imported for bootstrapping... We could avoid the interpreter bootstrap problems by adding language to the spec saying that yes, the source tree goes at the beginning of sys.path, BUT this MUST NOT be done using PYTHONPATH, and update the environment inheritance language to specify that this is a special case that is *not* inherited by child pythons. Kind of annoying when the only argument for automatically putting the source tree on sys.path is to simplify the spec, but ok... And then of course you can also get collisions between the source tree and the import of the build backend, as discussed earlier in the thread. Obviously this can't be fixed, since the whole point is to support cases where the build backend is in the source tree. If this does happen then... I dunno, maybe the backend shouldn't have imported that stuff, or maybe the source tree shouldn't have had a directory called that? But too late, it's out there on PyPI and pip is getting bug reports and there isn't even clear whose fault it is or who should fix it. If it were setup.py the obvious solution would be to add a 'sys.path.pop(0)', but here we've eliminated that option. I get the impression that a lot of people are like "well, that's a low probability event, and the rest of the time it's really handy, so whatever, it's worth it". And I agree it's a low probability event, but what I'm not sure people are fully realizing is... it's actually not very handy either :-). The *only* people who benefit from putting the source tree in sys.path are package authors who want to ship some custom one-off build backend inside their package. If you're writing a build backend that ships via PyPI, or you're writing a package that uses an existing build system pulled from PyPI, then having build frontends put the source directory on sys.path provides *zero* value to balance this small-but-non-zero risk. I think there's a real chance that if we ship this enabled by default in the name of "simplicity", then not only will we first have to add the special case alluded to above, but we'll eventually decide that we need to provide an opt-out flag, and then we'll have to communicate to projects that almost all of them actually want this opt-out flag, because the default is tuned for the minority of projects who are are doing something special. I don't know about you, but I am *so tired* of trying to explain to folks about "best practices" where the default is wrong. OTOH in the alternative approach where this opt-in, then the only folks who have to worry about this are the ones writing one-off build backends, and seriously once you've made the life choice to implement a one-off build backend then adding a line to pyproject.toml is really the least of your worries. People with special needs should be the ones paying the special tax. -n > On Sun, Jul 30, 2017, at 02:12 PM, Nick Coghlan wrote: >> On 30 July 2017 at 02:46, Nathaniel Smith wrote: >> > Or am I worrying about a non-issue and it's fine if flit imports click from >> > the source tree? >> >> Don't worry about it too much, as the problem here isn't really any >> worse than it is for normal runtime dependencies of any other project >> that relies on having the current directory first in sys.path. It just >> so happens that the project in question in this case is a Python >> project's build system. >> >> Due to the preference for a flat module namespace as the default, >> there are plenty of ways to hit name shadowing problems in Python, and >> as Donald notes, build systems have other motives to vendor their >> dependencies rather than installing them normally. >> >> Just switching the path order as has been suggested also doesn't solve >> the problem, as it merely inverts the issue: having "some_name" >> installed in site-packages would break source installations of >> packages that expected to be able to import "some_name" from their own >> root directory. >> >> If the problem does come up in practice, then there are a number of >> ways for affected projects to work around it in their project >> directory structure: >> >> 1. Use a top-level "src" directory (we may want to reserve "src" on PyPI) >> 2. Use a top-level "tools" directory (we may want to reserve "tools" on >> PyPI) >> 3. Add a leading or trailing underscore to the local directory name >> (as while that's legal for Python imports, it's prohibited for PyPI >> project names, and hence will often sidestep naming conflicts with >> published packages) >> >> Beyond that, the only approaches I'm aware of that systematically >> avoid this kind of problem at the language design level are to either >> use URL-based imports (ala Java or Go), or else to have separate >> syntax for "system-only" and "local resolution permitted" imports (ala >> C and C++), and Guido opted not to pursue either of those strategies >> for Python. >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Wed Aug 2 08:05:51 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 2 Aug 2017 22:05:51 +1000 Subject: [Distutils] status check on PEP 517 In-Reply-To: References: <1501271616.3751500.1055976240.2E1277F3@webmail.messagingengine.com> <1501275533.3769236.1056041016.23F3C467@webmail.messagingengine.com> <1501320665.888343.1056400072.59A66F97@webmail.messagingengine.com> <1501605113.2646382.1059777640.1267FB6C@webmail.messagingengine.com> Message-ID: On 2 August 2017 at 15:11, Nathaniel Smith wrote: > On Tue, Aug 1, 2017 at 9:31 AM, Thomas Kluyver wrote: >> Are we content to say that sys.path includes the source directory where >> the hook is run? Shall I prepare a PR against the PEP for that? > > It doesn't matter whether sys.path includes the source directory when > the hook is run. At that point the hook can manipulate sys.path > however it likes, just like setup.py can. What matters is whether > sys.path includes the source directory before the backend is imported. > > And I think that even if we keep the source directory in sys.path, > then there's at least a further piece of refinement needed. The > obvious mechanism for adding the source directory to sys.path is by > setting PYTHONPATH before invoking the sub-python that runs the build > backend. I wouldn't say that, as the two most clearly side effect free ways of putting the source directory on sys.path are: 1. make it the current working directory for a "python -m" or "python -c" invocation, so that the current directory gets injected as sys.path[0] by the interpreter. After all, it's already a requirement in the PEP that the current working directory be the directory containing pyproject.toml, so this approach handles that as well. 2. run a backend script directly (so it's the script directory that gets implicitly added to sys.path by the interpreter), and then inject the current working directory as sys.path[0] before importing the nominated backend If a frontend decides to go messing about with PYTHONPATH instead of doing either of those, then that's on them (and after it breaks horribly, they'll hopefully realise the error of their ways and switch to one of the other more reliable approaches). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From lucasboppre at gmail.com Fri Aug 4 16:43:49 2017 From: lucasboppre at gmail.com (Lucas Boppre Niehues) Date: Fri, 4 Aug 2017 17:43:49 -0300 Subject: [Distutils] new PyPI: a rant from a package maintainer Message-ID: I just had a dreadful day dealing with PyPI and the migrations that are happening. I'm posting this rant here in hopes of reaching the people who can fix it (apologies if I'm in the wrong list), or at least offer an explanation of the events that happened. Disclaimers: - Any and all feedback is welcome, but please be charitable. - I'm angry *at* things, I hope not to offend anybody. - I'm posting this rant here and not reddit or hn to ensure the discussion stays between the interested parties and not the whole internet. For formatting reasons the text is here: https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21 Thank you for your attention -------------- next part -------------- An HTML attachment was scrubbed... URL: From ubernostrum at gmail.com Fri Aug 4 17:29:38 2017 From: ubernostrum at gmail.com (James Bennett) Date: Fri, 4 Aug 2017 14:29:38 -0700 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: Since I have less direct personal involvement, I'll say this: Yes, there are problems. A lot of them are around documentation, and timing of when things switch over (the whole Python packaging ecosystem has been in a state of flux/rewrite for a while now; this is unfortunate, but also necessary). Nobody *likes* this. Nobody *wants* this. Nobody *defends* this. I get that this is really frustrating to someone who's brand new to trying to use the tools and services. At times it's frustrating to me, and I've been building and distributing Python packages through PyPI for something like a decade now. But I think what you posted is more about lashing out in anger than anything else, and while it feels good to yell and curse and rant, it also doesn't really help anything to do so. The problems are known, and they're being worked on, and the real root problem is a chronic lack of people with time to keep plugging away at it and fixing problems and implementing the improved solutions. The only way it gets better is by supporting the people who are doing that, or becoming one of the people who are doing that. I'm sorry that you're frustrated. You're not the only person who's frustrated, and in fact I'd bet a lot of the people who are trying to get this stuff to work seamlessly and perfectly for you are at least as frustrated as you are. They don't need more people yelling at them. They know there's pain. They've had to face it every day for a long time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Aug 4 17:51:51 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 4 Aug 2017 17:51:51 -0400 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: > On Aug 4, 2017, at 4:43 PM, Lucas Boppre Niehues wrote: > > I just had a dreadful day dealing with PyPI and the migrations that are happening. I'm posting this rant here in hopes of reaching the people who can fix it (apologies if I'm in the wrong list), or at least offer an explanation of the events that happened. > > Disclaimers: > - Any and all feedback is welcome, but please be charitable. > - I'm angry *at* things, I hope not to offend anybody. > - I'm posting this rant here and not reddit or hn to ensure the discussion stays between the interested parties and not the whole internet. > > For formatting reasons the text is here: > > https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21 > > > Thank you for your attention > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Going through your text, there?s a mixture of things that sort of combined here, I?m kind of just going to stream of conscious reply to them here, because I can?t reply inline. > error: No dist file created in earlier command Yes, ``setup.py upload`` *requires* you to create the files it is uploading within the same command during the upload. This is one of the reasons that twine as a tool exists and is generally recommended. > It's the exact same file. You could add a check for it and ignore, but no, it had to be an error. This seems like a reasonable feature request, please file an issue with Warehouse asking for it so I don?t forget it. > It seems that long_description in setup.py was missed entirely. I?m not sure what happened here. If you share the package I can probably figure it out though. I?m not currently aware of any bugs with how long_description is handled. > By the way, why was this feature included in the register command? This was not an explicit feature of the register command, and more of an accident in the way things got implemented in PyPI?s decade+ history. > [Stuff about editing the description] You located the issue on Warehouse. The fundamental issue here is trying to find the right balance between constraining authors so that end users can have a consistent behavior between packages and giving authors power to best manage their own projects. Discussion about editing the description is probably best handled on the warehouse issue though (Which I forgot to update with my thoughts!). > My setup.py has been communicating with https://upload.pypi.org/legacy/ by default. Note the legacy part. This is confusing, and it is essentially due to the fact there are two things here which are considered ?legacy?. One is the legacy codebase/deployment that currently powers pypi.python.org which are are slowly replacing and migrating things over to the new, modern code base that powers pypi.org . The other thing is the actual upload API itself, which has stayed the same currently, but which I plan to replace at some point in the future. This API is also considered ?legacy? (it just doesn?t have a replacement yet). So the legacy in https://upload.pypi.org/legacy/ has to do with the API rather than the location/deployment. I probably should have named it something other than /legacy/, my goal was mostly that changing URLs is hard (it requires a bunch of documentation/updates in different packages and N years for that to percolate out) so since we were forcing people to change the URLs with the move from pypi.python.org to pypi.org , might as well get it all done at once. It might be reasonable to name it something else now, and just keep the /legacy/ around as an alias. I?m not sure if that adds or subtracts from the confusion, but if you think that would have helped you, please open a new issue on Warehouse. > The "new APIs" seem to be missing at last one important feature Some of this is purposeful as we attempt to rein in some of the more ?random? features that PyPI has grown over time. Some of it is accidental as there?s not really any documentation about what PyPI provides, and people have, over the decade of it?s existence, depended on all sorts of random bits and pieces of PyPI which makes it hard to tell what people are even using. > I must have missed the deprecation period that preceded the complete removal of the old APIs. A friendly email would have been nice if you expect me to migrate to a new solution. This is a hard balance to strike. When I have generated emails in the past I tend to get a slew of people berating me for sending them emails about things. Particularly for changes like this where there belief is that for most people, the impact should be minimal, if there is any at all. I do try to at least tweet and post to distutils-sig about possibly disruptive changes like this. For this specific change, that thread was at https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html . > The new website is "pre-production?. This is mostly just a warning at this point that the UI might return funky results, some things might not be great yet, and to not start throwing the full force of people?s production traffic at it. The current hosting arrangement can?t handle the full amount of traffic we get from ``pip install ?`` (but uploads are fine!) and we?re working on building on the new infrastructure with our future hosting arrangement that can handle the full production load. > HTTPError: 400 Client Error: Invalid file extension. for url: https://upload.pypi.org/legacy/ The ability to upload anything besides sdists, wheels, and eggs was deprecated and removed. You can read the PEP that removed them at https://www.python.org/dev/peps/pep-0527/ . > Maybe the tutorial is outdated, and TestPyPI supports auto-registration now? I don?t think anyone has kept the tutorial on wiki.python.org up to date. To be frank, I don?t even know how to update wiki.python.org . Generally we?ve been trying to consolidate packaging documentation onto packaging.python.org ? although I don?t think we have one for TestPyPI yet. TestPyPI itself is also severely under maintained. I don?t use it so I rarely even think about it and my time is stretched thin to an extreme degree. I plan on getting rid of TestPyPI at some point and replacing it with something better, You can see discussion about that at https://github.com/pypa/warehouse/issues/726 . > I try python setup.py bdist upload As I mentioned above, we deprecated/removed everything but sdist, wheels, and eggs as per PEP 527. We could probably do better error messages here though. > I can't even specify the description of a package, not even during registration or upload. Tell me your package name privately or publicly and I?ll figure out what went wrong. > I cannot even report the issues. [ ... ] because there's no clear location to report them. This is a problem, and we don?t really have a good solution for them. I think generally what happens is people just open them on whichever tool they think is the best fit, and the authors of these tools all know each other, and if it ends up getting filed in the incorrect place, we just redirect people tot he correct place. > I am trying to donate my precious free time to the Python community by offering free software, receiving absolutely no compensation whatsoever, and every step of the experience has been shitty. I/we understand that, and we?re generally trying to do the best we can with what limited resources we have. The flip side of this is that we?re *also* (for the most part) donating our precious free time to the Python community to run PyPI (and the other PSF services too!). For the record, in July PyPI used something just under 1 Petabyte of bandwidth and served 38 billion HTTP requests. People working on that include myself with 16h/week of paid time (split between *all* of the packaging related tools I work on) plus something like 20-30h/week of volunteer time (on top of the additional 24h/week of time I spent working on my employers projects). On the ops side there is another individual (who I won?t name here to prevent him from getting drug into this [1]) who donates his spare time as well (though I don?t know offhand how much that is). As far as keeping PyPI running as well as getting the new code base developed and deployed? that?s about it [2]. This is a service used by ~everyone in the Python community without even a single full time person on it. We?re trying to slowly migrate to a new code base which is (A) easier to maintain/scale taking up fewer hours (B) easier to actually work on to develop new features, which will hopefully help attract and retain new contributors and (C) has removed decades of cruft that has built up and either explicitly made something supported or explicit decided it is no longer supported, instead of a quasi supported ?well it?s what PyPI does? scenario we have now. This probably won?t be the last less than perfect phase of the migration to the new code base. My best recommendation if you don?t want to discover these things JIT (even though we *do* try to minimize them) is to subscribe to and follow distutils-sig. Alternatively if you want to get more involved, one thing we?re severely lacking is a better mechanism for communicating these things to users, and if you wanted to propose/get sign off/implement/etc something for that, then that is another possibility. Finally if you wanted you could also get involved in actually shaping the direction of these tools, either by discussing on distutils-sig and in their respective issue trackers, or by becoming an active contributor. That?s the best way to ensure that your relevant use cases are considered during the design or execution of things. [1] Folks can yell at me if they want, don?t go yelling at other contributors. Generally the buck stops with me anyways. [2] There is a designer working on the new code base, and there have been some other contributors, but since this is about the APIs and maintenance I left them out. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Fri Aug 4 17:51:53 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 4 Aug 2017 14:51:53 -0700 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: On Fri, Aug 4, 2017 at 2:29 PM, James Bennett wrote: > I'm sorry that you're frustrated. You're not the only person who's > frustrated, and in fact I'd bet a lot of the people who are trying to get > this stuff to work seamlessly and perfectly for you are at least as > frustrated as you are. They don't need more people yelling at them. They > know there's pain. They've had to face it every day for a long time. I for one am actually very encouraged by the amount of activity around improving packaging. There are lots of people that care deeply about this stuff and are trying to get it right and improve things. We should be supportive of them and help where we can. If something isn't working like you think it should, I would always try asking first. In my experience, people on this list are very helpful in that regard, perhaps in part because they know certain things are in flux. --Chris > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From njs at pobox.com Fri Aug 4 18:15:21 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 Aug 2017 15:15:21 -0700 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: On Fri, Aug 4, 2017 at 2:51 PM, Donald Stufft wrote: > This API is also considered ?legacy? (it just doesn?t have a > replacement yet). So the legacy in https://upload.pypi.org/legacy/ has to do > with the API rather than the location/deployment. > > I probably should have named it something other than /legacy/, my goal was > mostly that changing URLs is hard (it requires a bunch of > documentation/updates in different packages and N years for that to > percolate out) so since we were forcing people to change the URLs with the > move from pypi.python.org to pypi.org, might as well get it all done at > once. It might be reasonable to name it something else now, and just keep > the /legacy/ around as an alias. I?m not sure if that adds or subtracts from > the confusion, but if you think that would have helped you, please open a > new issue on Warehouse. I guess calling it /v1 would avoid these issues and also make life easier when v3 comes around... -n -- Nathaniel J. Smith -- https://vorpus.org From brunson at brunson.com Fri Aug 4 18:22:54 2017 From: brunson at brunson.com (Eric Brunson) Date: Fri, 4 Aug 2017 22:22:54 +0000 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: <0100015daf58214c-0b76af6a-ed31-4016-abf9-7b4617ff7f7b-000000@email.amazonses.com> Donald, thanks for the thoughtful and complete reply to the original poster. On Fri, Aug 4, 2017 at 3:52 PM Donald Stufft wrote: > Going through your text, there?s a mixture of things that sort of combined > here, I?m kind of just going to stream of conscious reply to them here, > because I can?t reply inline. > > > [...] > > I must have missed the deprecation period that preceded the complete > removal of the old APIs. A friendly email would have been nice if you > expect me to migrate to a new solution. > > This is a hard balance to strike. When I have generated emails in the past > I tend to get a slew of people berating me for sending them emails about > things. Particularly for changes like this where there belief is that for > most people, the impact should be minimal, if there is any at all. I do try > to at least tweet and post to distutils-sig about possibly disruptive > changes like this. For this specific change, that thread was at > https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html. > > > [...] > > HTTPError: 400 Client Error: Invalid file extension. for url: > https://upload.pypi.org/legacy/ > > The ability to upload anything besides sdists, wheels, and eggs was > deprecated and removed. You can read the PEP that removed them at > https://www.python.org/dev/peps/pep-0527/. > > [...] > > > I try python setup.py bdist upload > > As I mentioned above, we deprecated/removed everything but sdist, wheels, > and eggs as per PEP 527. We could probably do better error messages here > though. > > Has the team considered putting deprecation messages in the tools to warn of these deprecations before they're removed? A warning for the API deprecations in pip or the underlying libraries, though not the perfect solution, might go a long way towards informing users of impending changes. Thanks, as always, for your efforts. Sincerely, e. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucasboppre at gmail.com Fri Aug 4 20:42:44 2017 From: lucasboppre at gmail.com (Lucas Boppre Niehues) Date: Fri, 4 Aug 2017 21:42:44 -0300 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: Thank you for the kind responses. I hope I haven't offended anybody, and please know that I'm incredibly grateful for all the sweat and blood that has been poured into these projects. I'm complaining about the current situation to show the largest pain points from my perspective. That's also why I'm posting here and not somewhere more visible. James, please see this is not just me lashing out to vent. This is the first time I ever post a rant. I'm pointing out the problems I experienced in the hopes that the people responsible notice the patterns, and derive solutions from them. I'm sorry for the flowery language, but be sure that the original events contained a lot more expletives, and I think people's reactions are important to prioritization. I've changed development focus before because someone emailed me complaining about a specific part of a project. Donald, thank you very much for the reasoned, helpful and throughout response. Here's my reply to the most important points: > The fundamental issue here is trying to find the right balance between constraining authors so that end users can have a consistent behavior between packages and giving authors power to best manage their own projects. > [...] > Some of this is purposeful as we attempt to rein in some of the more ?random? features that PyPI has grown over time I'm very glad to hear that. Speaking for myself, I wouldn't mind if PyPI became an API-only service that hosts immutable packages. Even if it makes my life a bit harder and I have to re-invent my release process, I would rather the platform be more solid. > The ability to upload anything besides sdists, wheels, and eggs was deprecated and removed. You can read the PEP that removed them at https://www.python.org/dev/peps/pep-0527/. That's... interesting. Thank you for the link, this is the first time I'm seeing it. I really liked the Windows installers, but I understand the need to slim down the infrastructure. Also, I now see some of my confusion was because of the invisible "allow legacy file types to be uploaded" flag in some of the projects but not others. >> I can't even specify the description of a package, not even during registration or upload. > Tell me your package name privately or publicly and I?ll figure out what went wrong. https://pypi.python.org/pypi/mouse/0.6.0 https://test.pypi.org/project/mouse/0.6.0 The long description was originally Markdown, and converted to RST by pandoc. I would 100% understand if this conversion triggered some bug. My gripe was that one by one my debugging tools failed in confusing ways. >> I cannot even report the issues. [ ... ] because there's no clear location to report them. > This is a problem, and we don?t really have a good solution for them. I think generally what happens is people just open them on whichever tool they think is the best fit, and the authors of these tools all know each other, and if it ends > up getting filed in the incorrect place, we just redirect people tot he correct place. Just my 2 cents, but I often not post a bug report at all for fear of wasting the wrong person's time. I'll keep this case in mind, but you may be missing some important reports with this configuration. The "packaging-problems" repo was almost perfect, but I avoided it due to the feeling of being abandoned or exclusive for PyPI developers. An explicit "It's ok to post X, Y and Z bugs here if you are unsure" would have changed my mind. > This is a service used by ~everyone in the Python community without even a single full time person on it. I'm deeply grateful for the work you and the other contributors have put here. I see the project is in good hands, if an insufficient number of them. But this is also an incredibly worrying statement. I work in security, and I know I'll have nightmares after reading this :( > When I have generated emails in the past I tend to get a slew of people berating me for sending them emails about things. > Particularly for changes like this where there belief is that for most people, the impact should be minimal, if there is any at all. That's awful, I'm sorry. Maybe an opt-in system during upload, to appease both crowds? I know I would set "email_me_important_announcements=True" in a heartbeat in my setup.py. Also note that people who don't like impact won't like the changes regardless if you email them or not, though I understand it's not nice to receive hate-mail. To be fair this migration is indeed mostly smooth. PyPI.org looks good, pypi.python.org is still working well (minus one or two deprecated endpoints leading to 410 gone), my setuptools automatically migrated to the legacy API, and I have nothing but praise for twine. And I found the use of HTTP error responses including detailed migration information to be a clever last-resort attempt at reaching the user. I faced several problems in my rant, but fixing any one of them would have solved my situation. I'm sure you guys can get there. Thank you for your patience and hard work, and I hope PyPI continues being one of the pillars of the community. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Aug 4 22:12:12 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 4 Aug 2017 22:12:12 -0400 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: <10E1F97C-DE19-4B3D-8FD3-231B237982BB@stufft.io> > On Aug 4, 2017, at 8:42 PM, Lucas Boppre Niehues wrote: > > > >> I can't even specify the description of a package, not even during registration or upload. > > Tell me your package name privately or publicly and I?ll figure out what went wrong. > > https://pypi.python.org/pypi/mouse/0.6.0 > https://test.pypi.org/project/mouse/0.6.0 > > The long description was originally Markdown, and converted to RST by pandoc. I would 100% understand if this conversion triggered some bug. My gripe was that one by one my debugging tools failed in confusing ways. So you uploaded the wheel first, which means that?s where PyPI would have gotten the long_description from. When I download that wheel and open it up and look at mouse-0.6.0.dist-info/METADATA I see a file that looks like: --- Metadata-Version: 2.0 Name: mouse Version: 0.6.0 Summary: Hook and simulate global mouse events in pure Python Home-page: https://github.com/boppreh/mouse Author: BoppreH Author-email: boppreh at gmail.com License: MIT mouse --- So it appears that the wheel isn?t being properly generated. I would expect the full contents of your long description to show up, and not just the word ?mouse?. I didn?t dig into it further to see if this is setuptools, wheel, or your pandoc version at fault, but my guess is it?s one of those. Given lots of people are successfully uploading, my first guess would be something in the pandoc build pipeline you have? but it could also be an edge case triggering a latent bug in setuptools or wheel. > > > >> I cannot even report the issues. [ ... ] because there's no clear location to report them. > > This is a problem, and we don?t really have a good solution for them. I think generally what happens is people just open them on whichever tool they think is the best fit, and the authors of these tools all know each other, and if it ends > > up getting filed in the incorrect place, we just redirect people tot he correct place. > > Just my 2 cents, but I often not post a bug report at all for fear of wasting the wrong person's time. I'll keep this case in mind, but you may be missing some important reports with this configuration. The "packaging-problems" repo was almost perfect, but I avoided it due to the feeling of being abandoned or exclusive for PyPI developers. An explicit "It's ok to post X, Y and Z bugs here if you are unsure" would have changed my mind. I?ll see what I can do here. It might just make sense for these related set of tools to just ditch the Github issue trackers and share an issue tracker that can divide things up by project somehow, but where we can readily move issues between them. Maybe we just need something (or someones) able to triage incoming issues on a meta tracker and redirect people. Not sure, I?ll give it some thought. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucasboppre at gmail.com Fri Aug 4 22:55:25 2017 From: lucasboppre at gmail.com (Lucas Boppre Niehues) Date: Fri, 4 Aug 2017 23:55:25 -0300 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: <10E1F97C-DE19-4B3D-8FD3-231B237982BB@stufft.io> References: <10E1F97C-DE19-4B3D-8FD3-231B237982BB@stufft.io> Message-ID: > So it appears that the wheel isn?t being properly generated. Ah, this helps a lot. I didn't realize it's the first upload that defines the description for the package. Testing different configurations I found that the wheel generation chokes and ignores everything after the first Windows-style newline '\r\n'. Meanwhile sdist just copies the string exactly as-is, but inside a Unix-style file, resulting in a PKG-INFO that doesn't break anything but looks weird. And you're also right that this is a bit of pypandoc's fault, because it returns a Windows-style string even if the input was Unix-style, but this is no reason to mix styles or choke. I can open a bug report later if you prefer. Thank you for the help, that solves the only unexplained issue I had. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucasboppre at gmail.com Sat Aug 5 00:15:25 2017 From: lucasboppre at gmail.com (Lucas Boppre Niehues) Date: Sat, 5 Aug 2017 01:15:25 -0300 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: <10E1F97C-DE19-4B3D-8FD3-231B237982BB@stufft.io> Message-ID: I created an issue on setuptools tracker about Windows newlines ( https://github.com/pypa/setuptools/issues/1126), and another on Warehouse about the "Invalid file extension" message and PEP 517 ( https://github.com/pypa/warehouse/issues/2281). I'm not creating one for ignoring duplicated files in `upload` because twine already has this feature (--skip-existing), and other people may be relying on this behavior. Also, I agree with Nathaniel Smith that numbered versioning of the API (/v0/, /v1/, /v/2) would be preferable to /legacy/. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Aug 5 04:50:22 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 5 Aug 2017 09:50:22 +0100 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: <10E1F97C-DE19-4B3D-8FD3-231B237982BB@stufft.io> References: <10E1F97C-DE19-4B3D-8FD3-231B237982BB@stufft.io> Message-ID: On 5 August 2017 at 03:12, Donald Stufft wrote: > I?ll see what I can do here. It might just make sense for these related set > of tools to just ditch the Github issue trackers and share an issue tracker > that can divide things up by project somehow, but where we can readily move > issues between them. Maybe we just need something (or someones) able to > triage incoming issues on a meta tracker and redirect people. Not sure, I?ll > give it some thought. I'd be willing to subscribe to a generalised tracker and triage issues there to direct people to the correct projects. I'm less sure that a single "Python packaging tools" tracker would be worthwhile (migration of existing issues seems like it would be a big exercise). Paul From thomas at kluyver.me.uk Sat Aug 5 13:03:13 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sat, 05 Aug 2017 18:03:13 +0100 Subject: [Distutils] status check on PEP 517 In-Reply-To: References: <1501271616.3751500.1055976240.2E1277F3@webmail.messagingengine.com> <1501275533.3769236.1056041016.23F3C467@webmail.messagingengine.com> <1501320665.888343.1056400072.59A66F97@webmail.messagingengine.com> <1501605113.2646382.1059777640.1267FB6C@webmail.messagingengine.com> Message-ID: <1501952593.1778142.1064163288.1846F481@webmail.messagingengine.com> On Wed, Aug 2, 2017, at 01:05 PM, Nick Coghlan wrote: > I wouldn't say that, as the two most clearly side effect free ways of > putting the source directory on sys.path are: > > 1. make it the current working directory for a "python -m" or "python > -c" invocation, so that the current directory gets injected as > sys.path[0] by the interpreter. After all, it's already a requirement > in the PEP that the current working directory be the directory > containing pyproject.toml, so this approach handles that as well. > 2. run a backend script directly (so it's the script directory that > gets implicitly added to sys.path by the interpreter), and then inject > the current working directory as sys.path[0] before importing the > nominated backend > > If a frontend decides to go messing about with PYTHONPATH instead of > doing either of those, then that's on them (and after it breaks > horribly, they'll hopefully realise the error of their ways and switch > to one of the other more reliable approaches). Nathaniel, can you live with the source directory being on sys.path to load the hooks? Or would you like to have another go at convincing people that it shouldn't be? Thomas From chris.barker at noaa.gov Sat Aug 5 16:09:32 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 5 Aug 2017 13:09:32 -0700 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: one point: I probably should have named it something other than /legacy/, > yes, it should have -- names matter! and having a "legacy" in teh name when there is not "modern" or "current", or, indeed, anything else is very confusing. pypi.org, might as well get it all done at once. It might be reasonable to > name it something else now, and just keep the /legacy/ around as an alias. > I?m not sure if that adds or subtracts from the confusion, but if you think > that would have helped you, please open a new issue on Warehouse. > I think that would be better -- something like "version1" something would be more clear. However, we have a major jumble of semi-out of date documentation scattered all over teh internet -- so ANOTHER name change, might just make things worse -- it's also confusing to have two names for the exact same thing! tough call. > HTTPError: 400 Client Error: Invalid file extension. for url: https://upload.pypi.org/legacy/ The ability to upload anything besides sdists, wheels, and eggs was deprecated and removed. A nice message to that effect would be great. Also -- I know that the various tools involved are maintained by different people, but it sure would be nice if, for instance, the latest versions of setuptools and twine would not try to upload non longer supported file types! I've thought for a couple years that we need a setuptool-lite that would ONLY do what we think is really should do -- but no one ele seemed to think that would be useful. > Maybe the tutorial is outdated, and TestPyPI supports auto-registration now? I don?t think anyone has kept the tutorial on wiki.python.org up to date. To be frank, I don?t even know how to update wiki.python.org. Someone that does really needs to add a "this isn't updated, please see packaging.python.org note on there !! As far as keeping PyPI running as well as getting the new code base > developed and deployed? that?s about it [2]. This is a service used by > ~everyone in the Python community without even a single full time person on > it. This is a core service -- we really need to figure out a better way to keep it maintained. I suspect the issue is project management, rather than volunteers -- there are a heck of a lot of talented folks that depend on these services. Of course, I have no good ideas about how to move forward -- is there anyone at the PSF that could take leadership on this? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Aug 5 16:15:13 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 5 Aug 2017 13:15:13 -0700 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: On Fri, Aug 4, 2017 at 5:42 PM, Lucas Boppre Niehues wrote: > > The long description was originally Markdown, and converted to RST by > pandoc. I would 100% understand if this conversion triggered some bug. > It's a good idea to run docutils on it yourself -- but in any case, broken RST has always resulted in plain text rendering on PyPi in the past -- as it should. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Sat Aug 5 16:42:17 2017 From: tritium-list at sdamon.com (Alex Walters) Date: Sat, 5 Aug 2017 16:42:17 -0400 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: > Also -- I know that the various tools involved are maintained by different people, but it sure would be nice if, for instance, the latest versions of setuptools and twine would not try to upload non longer supported file types! The tools should have better reporting on the errors pypi gives, but hard coding into the tools what pypi will and won?t accept makes policy decisions on pypi more painful to change. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 5 00:15:50 2017 From: xoviat at gmail.com (xoviat at gmail.com) Date: Fri, 4 Aug 2017 23:15:50 -0500 Subject: [Distutils] status check on PEP 517 Message-ID: <5985467a.5501ca0a.f1f27.6906@mx.google.com> Folks, Can we get these issues sorted before pip 10 so that the pip committers will accept my PR addressing the horrible easy_install behavior? Regards, xoviat -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 5 19:18:49 2017 From: xoviat at gmail.com (12345 67890) Date: Sat, 5 Aug 2017 18:18:49 -0500 Subject: [Distutils] PEP 517 Status Message-ID: <5986525e.0cafca0a.43835.d8ea@mx.google.com> Hello all: I have been attempting to resolve the horrible easy_install behavior where compiled wheels published on pypi are not used to fulfill setup requirements. To address this issue, I have submitted pull requests to both setuptools and pip that would allow a proprietary communication method that can later be easily adjusted to be fully PEP 517 compliant, saving work while resolving a specific issue. However, everyone appears to be ignoring these pull requests, in my estimation, because PEP 517 is not fully completed. That?s fine, but partial PEP 517 compliance in one area is better than nothing at all, which is currently the default scenario. Regards, xoviat -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Aug 6 14:58:30 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 6 Aug 2017 19:58:30 +0100 Subject: [Distutils] PEP 517 Status In-Reply-To: <5986525e.0cafca0a.43835.d8ea@mx.google.com> References: <5986525e.0cafca0a.43835.d8ea@mx.google.com> Message-ID: On 6 August 2017 at 00:18, 12345 67890 wrote: > I have been attempting to resolve the horrible easy_install behavior where > compiled wheels published on pypi are not used to fulfill setup > requirements. To address this issue, I have submitted pull requests to both > setuptools and pip that would allow a proprietary communication method that > can later be easily adjusted to be fully PEP 517 compliant, saving work > while resolving a specific issue. > > However, everyone appears to be ignoring these pull requests, in my > estimation, because PEP 517 is not fully completed. That?s fine, but partial > PEP 517 compliance in one area is better than nothing at all, which is > currently the default scenario. That's not correct (at least in my case). I'm "ignoring", as you put it, your PR, because I work on pip in my spare time, and I have extremely limited time to spend on pip, and I'd rather invest my time in other areas. That's my choice, and you complaining about my priorities won't affect it. *All* of the pip developers have very limited time, and complaining that you are being ignored is frankly not the best way of getting attention. To be explicit, I'm currently spending my time on: 1. Mailing list discussions (because I can work on these when not at a PC with a development setup). 2. Triage of issues (explaining misunderstandings, redirecting issues that are actually for other projects, confirming issues are real and requesting PRs, etc) 3. Reviews of PRs where other core pip maintainers or key developers need them. That pretty much uses up all of the time I have available for pip at the moment. I'd also note that I've never myself encountered the "horrible easy_install behavior" you mention, and your description doesn't remind me of any specific issues that have been raised. That's not to say that it's not a significant issue - maybe it is - but if you want to persuade me that it's urgent enough to merit me spending time on it, I'd appreciate links to your PR, and to one or more issues reported against pip explaining how the problem is affecting users. Also, it's important to remember that *no* change will improve things for users until it's in a released version of pip. The next planned version is pip 10, and I for one hope that PEP 517 will be in that version - so why rush in a fix that will only need modifying once PEP 517 is implemented? Even if your proposed change is highly urgent, it would still be sensible to resolve the question of whether PEP 517 will be implemented in pip 10 before rushing to implement a partial solution, that by your own admission uses a "proprietary" protocol rather than following PEP 517. Sorry if this isn't the response you were hoping for - but hopefully it explains the situation a little for you. Paul From xoviat at gmail.com Sun Aug 6 19:42:46 2017 From: xoviat at gmail.com (12345 67890) Date: Sun, 6 Aug 2017 18:42:46 -0500 Subject: [Distutils] PEP 517 Status In-Reply-To: References: <5986525e.0cafca0a.43835.d8ea@mx.google.com> Message-ID: <5987a977.d357ca0a.bbe60.b716@mx.google.com> I'd also note that I've never myself encountered the "horrible easy_install behavior" you mention, and your description doesn't remind me of any specific issues that have been raised. See setuptools issue #917. Setuptools does not use published wheels to fulfill setup_requires, which creates a host of issues when the installed distribution is different to what would have been installed by pip. *All* of the pip developers have very limited time, and complaining that you are being ignored is frankly not the best way of getting attention. I understand that but when I am attempting to use my own time to resolve a serious issue and then no one responded to it, I became suspicious that people were waiting on PEP 517 to resolve the issue. Also, it's important to remember that *no* change will improve things for users until it's in a released version of pip. The next planned version is pip 10, and I for one hope that PEP 517 will be in that version - so why rush in a fix that will only need modifying once PEP 517 is implemented? Precisely because I am highly concerned that PEP 517 does not appear on track to be resolved on distutils-sig before the next release of pip 10. I am more than happy to provide a complete implementation but I cannot proceed until the PEP is accepted. Regards, xoviat From: Paul Moore Sent: Sunday, August 6, 2017 1:58 PM To: 12345 67890 Cc: distutils-sig at python.org Subject: Re: [Distutils] PEP 517 Status On 6 August 2017 at 00:18, 12345 67890 wrote: > I have been attempting to resolve the horrible easy_install behavior where > compiled wheels published on pypi are not used to fulfill setup > requirements. To address this issue, I have submitted pull requests to both > setuptools and pip that would allow a proprietary communication method that > can later be easily adjusted to be fully PEP 517 compliant, saving work > while resolving a specific issue. > > However, everyone appears to be ignoring these pull requests, in my > estimation, because PEP 517 is not fully completed. That?s fine, but partial > PEP 517 compliance in one area is better than nothing at all, which is > currently the default scenario. That's not correct (at least in my case). I'm "ignoring", as you put it, your PR, because I work on pip in my spare time, and I have extremely limited time to spend on pip, and I'd rather invest my time in other areas. That's my choice, and you complaining about my priorities won't affect it. *All* of the pip developers have very limited time, and complaining that you are being ignored is frankly not the best way of getting attention. To be explicit, I'm currently spending my time on: 1. Mailing list discussions (because I can work on these when not at a PC with a development setup). 2. Triage of issues (explaining misunderstandings, redirecting issues that are actually for other projects, confirming issues are real and requesting PRs, etc) 3. Reviews of PRs where other core pip maintainers or key developers need them. That pretty much uses up all of the time I have available for pip at the moment. I'd also note that I've never myself encountered the "horrible easy_install behavior" you mention, and your description doesn't remind me of any specific issues that have been raised. That's not to say that it's not a significant issue - maybe it is - but if you want to persuade me that it's urgent enough to merit me spending time on it, I'd appreciate links to your PR, and to one or more issues reported against pip explaining how the problem is affecting users. Also, it's important to remember that *no* change will improve things for users until it's in a released version of pip. The next planned version is pip 10, and I for one hope that PEP 517 will be in that version - so why rush in a fix that will only need modifying once PEP 517 is implemented? Even if your proposed change is highly urgent, it would still be sensible to resolve the question of whether PEP 517 will be implemented in pip 10 before rushing to implement a partial solution, that by your own admission uses a "proprietary" protocol rather than following PEP 517. Sorry if this isn't the response you were hoping for - but hopefully it explains the situation a little for you. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Aug 6 22:17:54 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 Aug 2017 12:17:54 +1000 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: <10E1F97C-DE19-4B3D-8FD3-231B237982BB@stufft.io> Message-ID: On 5 August 2017 at 14:15, Lucas Boppre Niehues wrote: > Also, I agree with Nathaniel Smith that numbered versioning of the API > (/v0/, /v1/, /v/2) would be preferable to /legacy/. I don't have a lot to add to Donald's other responses (Thanks Donald!), but wanted to register a +1 on the idea of replacing the current "/legacy/" upload URL with a "/v0/" URL (with "/legacy/" preserved as a compatibility alias). The "v0" version number still indicates that it isn't really a properly defined service API, without the potential for confusion between "legacy PyPI" and "legacy upload API". I'll also note that the lack of good service announcement channels is in some respects a PSF level problem, rather than just a PyPI level one, so there's a meta-tracker issue for that here: https://github.com/python/psf-infra-meta/issues/1 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From randy at thesyrings.us Sun Aug 6 22:19:06 2017 From: randy at thesyrings.us (Randy Syring) Date: Sun, 6 Aug 2017 22:19:06 -0400 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: References: Message-ID: <48d57b06-edf9-99fc-02ab-eb67f335ad81@thesyrings.us> Issues created: * Ignore dup uploads instead of error: https://github.com/pypa/warehouse/issues/2284 * Rename "/legacy" API URL: https://github.com/pypa/warehouse/issues/2285 * Propose "packaging announce" mailing list: https://github.com/pypa/packaging-problems/issues/98 o Previous issue noting challenges in communicating on packaging, many of the same concerns still seem relevant: https://github.com/pypa/packaging-problems/issues/21 * Explain what "pre-production" means on Warehouse: https://github.com/pypa/warehouse/issues/2287 * Deprecation notice added to: https://wiki.python.org/moin/TestPyPI o Issue created to enhance docs for TestPyPI: https://github.com/pypa/python-packaging-user-guide/issues/362 * Better error message for PEP 527 accepted file types: https://github.com/pypa/packaging-problems/issues/99 I hope I got all the right places for those issues. Thanks Donald for the clear reply from which it was easy to create these separate issues. I didn't realize you had so little paid time to work on packaging issues. I'm surprised with all the large companies that use Python that the PSF can't secure more donations to invest in fixing these issues. Or that someone like Redhat can't get a couple engineers to devote time for a couple years like they do to other OSS projects. (No criticism intended, just musing). *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/ On 08/04/2017 05:51 PM, Donald Stufft wrote: > >> On Aug 4, 2017, at 4:43 PM, Lucas Boppre Niehues >> > wrote: >> >> I just had a dreadful day dealing with PyPI and the migrations that >> are happening. I'm posting this rant here in hopes of reaching the >> people who can fix it (apologies if I'm in the wrong list), or at >> least offer an explanation of the events that happened. >> >> Disclaimers: >> - Any and all feedback is welcome, but please be charitable. >> - I'm angry *at* things, I hope not to offend anybody. >> - I'm posting this rant here and not reddit or hn to ensure the >> discussion stays between the interested parties and not the whole >> internet. >> >> For formatting reasons the text is here: >> >> https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21 >> >> >> Thank you for your attention >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> >> https://mail.python.org/mailman/listinfo/distutils-sig > > > Going through your text, there?s a mixture of things that sort of > combined here, I?m kind of just going to stream of conscious reply to > them here, because I can?t reply inline. > > > > error: No dist file created in earlier command > > Yes, ``setup.py upload`` *requires* you to create the files it is > uploading within the same command during the upload. This is one of > the reasons that twine as a tool exists and is generally recommended. > > > > It's the exact same file. You could add a check for it and ignore, > but no, it had to be an error. > > This seems like a reasonable feature request, please file an issue > with Warehouse asking for it so I don?t forget it. > > > > It seems that long_description in setup.py was missed entirely. > > I?m not sure what happened here. If you share the package I can > probably figure it out though. I?m not currently aware of any bugs > with how long_description is handled. > > > > By the way, why was this feature included in the register command? > > This was not an explicit feature of the register command, and more of > an accident in the way things got implemented in PyPI?s decade+ history. > > > > [Stuff about editing the description] > > You located the issue on Warehouse. The fundamental issue here is > trying to find the right balance between constraining authors so that > end users can have a consistent behavior between packages and giving > authors power to best manage their own projects. Discussion about > editing the description is probably best handled on the warehouse > issue though (Which I forgot to update with my thoughts!). > > > > My setup.py has been communicating with > https://upload.pypi.org/legacy/ by default. Note the legacy part. > > This is confusing, and it is essentially due to the fact there are two > things here which are considered ?legacy?. One is the legacy > codebase/deployment that currently powers pypi.python.org > which are are slowly replacing and migrating > things over to the new, modern code base that powers pypi.org > . The other thing is the actual upload API itself, > which has stayed the same currently, but which I plan to replace at > some point in the future. This API is also considered ?legacy? (it > just doesn?t have a replacement yet). So the legacy in > https://upload.pypi.org/legacy/ has to do with the API rather than the > location/deployment. > > I probably should have named it something other than /legacy/, my goal > was mostly that changing URLs is hard (it requires a bunch of > documentation/updates in different packages and N years for that to > percolate out) so since we were forcing people to change the URLs with > the move from pypi.python.org to pypi.org > , might as well get it all done at once. It might be > reasonable to name it something else now, and just keep the /legacy/ > around as an alias. I?m not sure if that adds or subtracts from the > confusion, but if you think that would have helped you, please open a > new issue on Warehouse. > > > > The "new APIs" seem to be missing at last one important feature > > Some of this is purposeful as we attempt to rein in some of the more > ?random? features that PyPI has grown over time. Some of it is > accidental as there?s not really any documentation about what PyPI > provides, and people have, over the decade of it?s existence, depended > on all sorts of random bits and pieces of PyPI which makes it hard to > tell what people are even using. > > > > I must have missed the deprecation period that preceded the complete > removal of the old APIs. A friendly email would have been nice if you > expect me to migrate to a new solution. > > This is a hard balance to strike. When I have generated emails in the > past I tend to get a slew of people berating me for sending them > emails about things. Particularly for changes like this where there > belief is that for most people, the impact should be minimal, if there > is any at all. I do try to at least tweet and post to distutils-sig > about possibly disruptive changes like this. For this specific change, > that thread was at > https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html. > > > > The new website is "pre-production?. > > This is mostly just a warning at this point that the UI might return > funky results, some things might not be great yet, and to not start > throwing the full force of people?s production traffic at it. The > current hosting arrangement can?t handle the full amount of traffic we > get from ``pip install ?`` (but uploads are fine!) and we?re working > on building on the new infrastructure with our future hosting > arrangement that can handle the full production load. > > > > HTTPError: 400 Client Error: Invalid file extension. for url: > https://upload.pypi.org/legacy/ > > The ability to upload anything besides sdists, wheels, and eggs was > deprecated and removed. You can read the PEP that removed them at > https://www.python.org/dev/peps/pep-0527/. > > > > Maybe the tutorial is outdated, and TestPyPI supports > auto-registration now? > > I don?t think anyone has kept the tutorial on wiki.python.org > up to date. To be frank, I don?t even know > how to update wiki.python.org . Generally > we?ve been trying to consolidate packaging documentation onto > packaging.python.org ? although I don?t > think we have one for TestPyPI yet. > > TestPyPI itself is also severely under maintained. I don?t use it so I > rarely even think about it and my time is stretched thin to an extreme > degree. I plan on getting rid of TestPyPI at some point and replacing > it with something better, You can see discussion about that at > https://github.com/pypa/warehouse/issues/726. > > > > I try python setup.py bdist upload > > As I mentioned above, we deprecated/removed everything but sdist, > wheels, and eggs as per PEP 527. We could probably do better error > messages here though. > > > > I can't even specify the description of a package, not even during > registration or upload. > > Tell me your package name privately or publicly and I?ll figure out > what went wrong. > > > > I cannot even report the issues. [ ... ] because there's no clear > location to report them. > > This is a problem, and we don?t really have a good solution for them. > I think generally what happens is people just open them on whichever > tool they think is the best fit, and the authors of these tools all > know each other, and if it ends up getting filed in the incorrect > place, we just redirect people tot he correct place. > > > > I am trying to donate my precious free time to the Python community > by offering free software, receiving absolutely no compensation > whatsoever, and every step of the experience has been shitty. > > I/we understand that, and we?re generally trying to do the best we can > with what limited resources we have. The flip side of this is that > we?re *also* (for the most part) donating our precious free time to > the Python community to run PyPI (and the other PSF services too!). > For the record, in July PyPI used something just under 1 Petabyte of > bandwidth and served 38 billion HTTP requests. > > People working on that include myself with 16h/week of paid time > (split between *all* of the packaging related tools I work on) plus > something like 20-30h/week of volunteer time (on top of the additional > 24h/week of time I spent working on my employers projects). On the ops > side there is another individual (who I won?t name here to prevent him > from getting drug into this [1]) who donates his spare time as well > (though I don?t know offhand how much that is). > > As far as keeping PyPI running as well as getting the new code base > developed and deployed? that?s about it [2]. This is a service used by > ~everyone in the Python community without even a single full time > person on it. > > We?re trying to slowly migrate to a new code base which is (A) easier > to maintain/scale taking up fewer hours (B) easier to actually work on > to develop new features, which will hopefully help attract and retain > new contributors and (C) has removed decades of cruft that has built > up and either explicitly made something supported or explicit decided > it is no longer supported, instead of a quasi supported ?well it?s > what PyPI does? scenario we have now. > > This probably won?t be the last less than perfect phase of the > migration to the new code base. My best recommendation if you don?t > want to discover these things JIT (even though we *do* try to minimize > them) is to subscribe to and follow distutils-sig. Alternatively if > you want to get more involved, one thing we?re severely lacking is a > better mechanism for communicating these things to users, and if you > wanted to propose/get sign off/implement/etc something for that, then > that is another possibility. Finally if you wanted you could also get > involved in actually shaping the direction of these tools, either by > discussing on distutils-sig and in their respective issue trackers, or > by becoming an active contributor. That?s the best way to ensure that > your relevant use cases are considered during the design or execution > of things. > > > [1] Folks can yell at me if they want, don?t go yelling at other > contributors. Generally the buck stops with me anyways. > [2] There is a designer working on the new code base, and there have > been some other contributors, but since this is about the APIs and > maintenance I left them out. > > ? > Donald Stufft > > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Aug 7 01:02:29 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 Aug 2017 15:02:29 +1000 Subject: [Distutils] new PyPI: a rant from a package maintainer In-Reply-To: <48d57b06-edf9-99fc-02ab-eb67f335ad81@thesyrings.us> References: <48d57b06-edf9-99fc-02ab-eb67f335ad81@thesyrings.us> Message-ID: On 7 August 2017 at 12:19, Randy Syring wrote: > Thanks Donald for the clear reply from which it was easy to create these > separate issues. I didn't realize you had so little paid time to work on > packaging issues. I'm surprised with all the large companies that use > Python that the PSF can't secure more donations to invest in fixing these > issues. Or that someone like Redhat can't get a couple engineers to devote > time for a couple years like they do to other OSS projects. (No criticism > intended, just musing). Please feel free to criticise Python's commercial redistributors loudly and publicly on this point, as our collective failure to deal with the situation appropriately is something our respective customers *should* be questioning and challenging us over, especially when support contracts are up for renegotiation :) As far as I've been able to work out, the main problem appears to be that the professional Python user base currently self-selects into two primary categories: 1. "We're happy to rely on self-support and community support, so we don't need a Python vendor" 2. "Our ops team pays a commercial Python runtime vendor, but our dev teams have no ability to file support requests with that vendor" In combination, these behaviours mean vendors' commercial demand signals are thoroughly broken, creating an environment where it becomes difficult to make the business case that increased upstream investment will lead to proportional increases in revenue. (To help illustrate the scale of the problem, while omitting names to protect the guilty: "Do people in large organisations actually use Python for web development?" is something I have had a redistributor ask as a genuine question in the same month where Python topped IEEE Spectrum's annual survey of language popularity for the first time. When we have that level of disconnect in the perception of ecosystem adoption, something significant is clearly amiss) At the Python Software Foundation level, one of the things we've realised we need to do is to provide more streamlined ways for organisations using Python to direct funding specifically towards infrastructure support for PyPI (with the Packaging Working Group then being responsible for managing those funds), since "We should fund the platform that hosts our Python dependencies" is a potentially easier pitch for engineering teams to make than advocating for their employers to become full PSF Sponsor members. Towards that end, we've recently applied to Mozilla for a development grant to help the PSF fund the design, development, and project management efforts needed to get those improved infrastructure sustainability measures in place, as well as getting to the point where the legacy PyPI service can be shut down entirely, eliminating one of the current major barriers to improving the shared infrastructure (credit goes to Eric Holscher for getting that request through to the grant submission stage - the Packaging WG had started drafting it some time ago, but we'd stalled before reaching the point of actually submitting it). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From philip.deegan at gmail.com Thu Aug 10 12:30:43 2017 From: philip.deegan at gmail.com (P) Date: Thu, 10 Aug 2017 17:30:43 +0100 Subject: [Distutils] Distuils MSVC link failure between libraries Message-ID: Hi there, I'm trying to create multiple C++ extensions that have dependencies between them. This works fine on Linux with gcc, but I get link failures on Windows with MSVC due to this "*IMPLIB*" is an optional argument, when not specified, the generation of exp/lib files default to the same filename and directory as the dll. If precompiled extensions are to be published with MSVC and expected to be linked against (maybe this is not recommended?) the lib files are required so it seems a bit peculiar to me to not have them with the dll as is the default behavior. Proof of concept Thanks Philip -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Thu Aug 10 22:16:45 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 10 Aug 2017 19:16:45 -0700 Subject: [Distutils] pip-installing from a VCS url with a commit hash Message-ID: I was interested in working on the following issue but thought it would be worth asking this list about first: https://github.com/pypa/pip/issues/4507 If pip-installing a dependency from a VCS url of the following form: git+https://myvcs.com/some_dependency at sometag#egg=SomeDependency and "sometag" is the hash of a commit, should pip interpret it as a commit hash, or should it first check for a branch or tag of that name? One argument in favor of interpreting it as a commit hash is greater determinism. If someone later adds a branch or tag with that name, the dependency will continue to resolve to the original commit. (However, someone could still remove the original commit before adding a branch or tag with that name.) Another argument is that, without this behavior, once a branch or tag with that name is added, there will no longer be a way to refer to the original commit. Either way, I think the log message can be improved. Every time I see that log message, I think to myself, "of course it's a commit, it's a 40-character hexadecimal string!" --Chris From lists at cooperlees.com Sun Aug 13 02:53:35 2017 From: lists at cooperlees.com (Cooper Ry Lees) Date: Sun, 13 Aug 2017 14:53:35 +0800 Subject: [Distutils] Mirroring PyPI JSON Locally Message-ID: Hi all, First time emailer, so please be kind. Also, if this is not the right mailing list for PyPA talk, I apologize. Please point me in the right direction if so (Brett Canon pointed me here). The main reason I have emailed here is I believe it may be PEP time to standardize the JSON metadata that PyPI makes available, like what was done for the `'simple API` described in PEP503. I've been doing a bit of work on `bandersnatch` (I didn't name it), which is a PEP 381 mirroring package and wanted to enhance it to also mirror the handy JSON metadata PyPI generates and makes available @ https://pypi.python.org/pypi/PKG_NAME/json. I've done a PR on bandersnatch as a POC that mirrors both the PyPI directory structure (URL/pypi/PKG_NAME/json) and created a standardizable URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some other proxy). I'm also contemplating naming the directory 'metadata' rather than JSON so if some new hotness / we want to change the format down the line we're not stuck with json as the dirname. This PR can be found here: https://bitbucket.org/pypa/bandersnatch/pull-requests/33/save-json- metadata-to-mirror My main use case is to write a very simple async 'verifier' tool that will crawl all the JSON files and then ensure the packages directory on each of my internal mirrors (I have a mirror per region / datacenter) have all the files they should. I sync centrally (to save resource on the PyPI infrastructure) and then rsync out all the diffs to each region / datacenter, and under some failure scenarios I could miss a file or many. So I feel using JSON pulled down from the authoritative source will allow an async job to verify the MD5 of all the package files on each mirror. What are peoples thoughts here? Is it worth a PEP similar to PEP503 going forward? Can people enhance / share some thoughts on this idea. Thanks, Cooper Ry Lees me at cooperlees.com https://cooperlees.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Tue Aug 15 01:11:08 2017 From: xoviat at gmail.com (12345 67890) Date: Tue, 15 Aug 2017 00:11:08 -0500 Subject: [Distutils] status check on PEP 517 Message-ID: <5992826b.023fca0a.101dc.c12e@mx.google.com> Nick: Do you have any update on when the PEP will be completed? Regards, xoviat -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Aug 15 06:14:17 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Aug 2017 20:14:17 +1000 Subject: [Distutils] status check on PEP 517 In-Reply-To: <5992826b.023fca0a.101dc.c12e@mx.google.com> References: <5992826b.023fca0a.101dc.c12e@mx.google.com> Message-ID: On 15 August 2017 at 15:11, 12345 67890 wrote: > Do you have any update on when the PEP will be completed? >From my perspective, the main item from the last round of discussions that has yet to be explicitly added to the PEP is the ability for build_sdist and build_wheel to raise NotImplementedError to indicate that an operation has failed for aniticipated reasons and to provide an error message explaining the problem. There were a couple of other items where the PEP authors weren't necessarily sure that what was in the PEP was exactly what they wanted to propose implementing, but my own view on those aspects is that what's currently in the PEP is fine, as the last round of discussions cleaned up some key problems where the way the explicitly out-of-tree build support was designed in earlier versions of the API didn't align with the way most build systems actually work (i.e. they expect to be given the target directory as a build setting, allowing the build system itself to decide exactly how to handle that). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Tue Aug 15 11:42:06 2017 From: brett at python.org (Brett Cannon) Date: Tue, 15 Aug 2017 15:42:06 +0000 Subject: [Distutils] status check on PEP 517 In-Reply-To: References: <5992826b.023fca0a.101dc.c12e@mx.google.com> Message-ID: I'm also willing to bet that Thomas is busy prepping for JupyterCon. On Tue, 15 Aug 2017 at 03:14 Nick Coghlan wrote: > On 15 August 2017 at 15:11, 12345 67890 wrote: > > Do you have any update on when the PEP will be completed? > > From my perspective, the main item from the last round of discussions > that has yet to be explicitly added to the PEP is the ability for > build_sdist and build_wheel to raise NotImplementedError to indicate > that an operation has failed for aniticipated reasons and to provide > an error message explaining the problem. > > There were a couple of other items where the PEP authors weren't > necessarily sure that what was in the PEP was exactly what they wanted > to propose implementing, but my own view on those aspects is that > what's currently in the PEP is fine, as the last round of discussions > cleaned up some key problems where the way the explicitly out-of-tree > build support was designed in earlier versions of the API didn't align > with the way most build systems actually work (i.e. they expect to be > given the target directory as a build setting, allowing the build > system itself to decide exactly how to handle that). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Aug 15 11:42:53 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 15 Aug 2017 08:42:53 -0700 Subject: [Distutils] Distuils MSVC link failure between libraries In-Reply-To: References: Message-ID: <5227309882669953123@unknownmsgid> I'm trying to create multiple C++ extensions that have dependencies between them. This works fine on Linux with gcc, but I get link failures on Windows with MSVC due to this Is that the only issue? MSVC and FCC linking is pretty different-- I know I tried to do something like this (a few years ago) and it didn't work -- I think because the symbols I needed weren't exported, or something like that. If precompiled extensions are to be published with MSVC and expected to be linked against (maybe this is not recommended?) Indeed, it is not recommended-- onhe passive sense -- I.e. the system isn't designed to support that. IIUC, while extension modules are dlls, they are specifically designed to be called from cpython, not as general purpose libraries to be called from other C code. The solution is to build a dill with all the C/C++ code you need, and then have all your modules link against that dll. Proof of concept If that works, and doesn't cause other problems, (I am not qualified to assess that), it could be handy. There are times I would have liked distutils to be able to build a dll for me that could be used by other modules. -CHB -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Tue Aug 15 11:47:43 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 15 Aug 2017 16:47:43 +0100 Subject: [Distutils] status check on PEP 517 In-Reply-To: References: <5992826b.023fca0a.101dc.c12e@mx.google.com> Message-ID: <1502812063.1140071.1074199392.56B83525@webmail.messagingengine.com> On Tue, Aug 15, 2017, at 04:42 PM, Brett Cannon wrote: > I'm also willing to bet that Thomas is busy prepping for JupyterCon. Sadly I won't be at JupyterCon. I'm avoiding all travel to the US in the current political situation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 17 12:05:01 2017 From: xoviat at gmail.com (12345 67890) Date: Thu, 17 Aug 2017 11:05:01 -0500 Subject: [Distutils] PEP 517: Build frontend responsibilities Message-ID: <5995beae.86f0ca0a.da31f.d175@mx.google.com> All: After reviewing the PEP, I personally feel that it relies too much on the backend Doing The Right Thing. As it currently stands, it is my understanding that the build backend is called in the source directory and is then responsible for handling the entire process to build a compliant wheel. In a PEP 517 world, the build backends may be poorly written initially and may not fully comply with the PEP?s goal to produce the same wheel that would be produced from an sdist. Current process: - Frontend calls backend in source directory build-wheel Proposed process: - Frontend copies source tree to temporary directory - Frontend invokes build-sdist to build an sdist - Frontend extracts sdist to new temporary directory - Frontend reloads backend from sdist directory and invokes build-wheel The proposed process is more computationally intensive, but moves compliance logic out of the build backend. In addition, we can make some modifications based on this proposed process: - Remove parameter for build_dir in build_wheel since it is always the current directory - Optimization: add new parameter to build_sdist called copy_extra, that when set to false, only includes files in sdist needed to build a wheel. - Specify that build_wheel should fail if PKG-INFO is not present so that it is always invoked in an sdist I realized that I may have pushed to have this completed earlier, but I agree with Donald that we need to enforce a process rather than hoping for the best. Regards, xoviat -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Aug 17 22:50:23 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 18 Aug 2017 02:50:23 +0000 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: <5995beae.86f0ca0a.da31f.d175@mx.google.com> References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> Message-ID: I agree that the way you want to do packaging is fundamentally incompatible with build systems that do not resemble distutils. Perhaps since this is distutils sig some here are too used to distutils as the only model of how packaging might work. Beware that, when fighting monsters, you yourself do not become a monster... for when you gaze long into the abyss. The abyss gazes also into you. I suggest letting the pep go provisional. If open source publishers, who are highly motivated to produce useful free software, are suddenly also motivated to publish broken build systems, and if you can't vote with your feet by picking an alternative package, then demote the pep. On Thu, Aug 17, 2017, 21:52 12345 67890 wrote: > All: > > > > After reviewing the PEP, I personally feel that it relies too much on the > backend Doing The Right Thing. As it currently stands, it is my > understanding that the build backend is called in the source directory and > is then responsible for handling the entire process to build a compliant > wheel. In a PEP 517 world, the build backends may be poorly written > initially and may not fully comply with the PEP?s goal to produce the same > wheel that would be produced from an sdist. > > > > Current process: > > > > - Frontend calls backend in source directory build-wheel > > > > Proposed process: > > - Frontend copies source tree to temporary directory > - Frontend invokes build-sdist to build an sdist > - Frontend extracts sdist to new temporary directory > - Frontend reloads backend from sdist directory and invokes build-wheel > > > > The proposed process is more computationally intensive, but moves > compliance logic out of the build backend. In addition, we can make some > modifications based on this proposed process: > > > > - Remove parameter for build_dir in build_wheel since it is always the > current directory > - Optimization: add new parameter to build_sdist called copy_extra, > that when set to false, only includes files in sdist needed to build a > wheel. > - Specify that build_wheel should fail if PKG-INFO is not present so > that it is always invoked in an sdist > > > > I realized that I may have pushed to have this completed earlier, but I > agree with Donald that we need to enforce a process rather than hoping for > the best. > > > > > > Regards, > > > > > > xoviat > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Aug 17 22:51:25 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Aug 2017 12:51:25 +1000 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: <5995beae.86f0ca0a.da31f.d175@mx.google.com> References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> Message-ID: On 18 August 2017 at 02:05, 12345 67890 wrote: > After reviewing the PEP, I personally feel that it relies too much on the > backend Doing The Right Thing. As it currently stands, it is my > understanding that the build backend is called in the source directory and > is then responsible for handling the entire process to build a compliant > wheel. In a PEP 517 world, the build backends may be poorly written > initially and may not fully comply with the PEP?s goal to produce the same > wheel that would be produced from an sdist. > > Current process: > > Frontend calls backend in source directory build-wheel The PEP *allows* this, but it doesn't *require* it. > Proposed process: > > Frontend copies source tree to temporary directory > Frontend invokes build-sdist to build an sdist > Frontend extracts sdist to new temporary directory > Frontend reloads backend from sdist directory and invokes build-wheel A particularly paranoid front-end remains free to work this way if it wants to. >From the PEP's point of view, it's just four distinct operations: Step 0: make a copy of the original source tree Step 1: build an sdist from the copied source tree Step 2: unpack the sdist Step 3: build a wheel from the unpacked sdist > The proposed process is more computationally intensive, but moves compliance > logic out of the build backend. In addition, we can make some modifications > based on this proposed process: > > * Remove parameter for build_dir in build_wheel since it is always the current > directory We've had that discussion - build_dir models something different from what you propose (specifically, it's designed to invoke the backend's native out-of-tree build support, when that's what the frontend actually wants to invoke. If a frontend doesn't want that, don't use the setting) > Optimization: add new parameter to build_sdist called copy_extra, that when > set to false, only includes files in sdist needed to build a wheel. No, this is the operation we decided makes no sense (and hence removed from the latest iteration of the PEP), because no build system we could find actually provides it natively. That means it would necessarily be a Python-specific capability anyway, and that further means that any backend which could be trusted to implement this feature correctly can instead be trusted to implement building an sdist or wheel directly from a source tree correctly. It's also not a major concern if early backend iterations *don't* do things correctly - those errors will be reported as bugs, fixed versions of the backends will be released, and future builds will start doing the right thing. > Specify that build_wheel should fail if PKG-INFO is not present so that it > is always invoked in an sdist No, we don't want to make that a mandatory requirement, since frontends are already free to enforce that rule if they want to (by checking for PKG-INFO before they call build_wheel), and particular backends are also free to enforce it for the packages they manage (by raising NotImplemented when they don't find PKG-INFO). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From xoviat at gmail.com Fri Aug 18 00:16:29 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 17 Aug 2017 23:16:29 -0500 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> Message-ID: I'm still not convinced, but this PEP has been WIP for too long. I think it should be marked Accepted. On Aug 17, 2017 9:51 PM, "Nick Coghlan" wrote: > On 18 August 2017 at 02:05, 12345 67890 wrote: > > After reviewing the PEP, I personally feel that it relies too much on the > > backend Doing The Right Thing. As it currently stands, it is my > > understanding that the build backend is called in the source directory > and > > is then responsible for handling the entire process to build a compliant > > wheel. In a PEP 517 world, the build backends may be poorly written > > initially and may not fully comply with the PEP?s goal to produce the > same > > wheel that would be produced from an sdist. > > > > Current process: > > > > Frontend calls backend in source directory build-wheel > > The PEP *allows* this, but it doesn't *require* it. > > > Proposed process: > > > > Frontend copies source tree to temporary directory > > Frontend invokes build-sdist to build an sdist > > Frontend extracts sdist to new temporary directory > > Frontend reloads backend from sdist directory and invokes build-wheel > > A particularly paranoid front-end remains free to work this way if it > wants to. > > From the PEP's point of view, it's just four distinct operations: > > Step 0: make a copy of the original source tree > Step 1: build an sdist from the copied source tree > Step 2: unpack the sdist > Step 3: build a wheel from the unpacked sdist > > > The proposed process is more computationally intensive, but moves > compliance > > logic out of the build backend. In addition, we can make some > modifications > > based on this proposed process: > > > > * Remove parameter for build_dir in build_wheel since it is always the > current > > directory > > We've had that discussion - build_dir models something different from > what you propose (specifically, it's designed to invoke the backend's > native out-of-tree build support, when that's what the frontend > actually wants to invoke. If a frontend doesn't want that, don't use > the setting) > > > Optimization: add new parameter to build_sdist called copy_extra, that > when > > set to false, only includes files in sdist needed to build a wheel. > > No, this is the operation we decided makes no sense (and hence removed > from the latest iteration of the PEP), because no build system we > could find actually provides it natively. > > That means it would necessarily be a Python-specific capability > anyway, and that further means that any backend which could be trusted > to implement this feature correctly can instead be trusted to > implement building an sdist or wheel directly from a source tree > correctly. > > It's also not a major concern if early backend iterations *don't* do > things correctly - those errors will be reported as bugs, fixed > versions of the backends will be released, and future builds will > start doing the right thing. > > > Specify that build_wheel should fail if PKG-INFO is not present so that > it > > is always invoked in an sdist > > No, we don't want to make that a mandatory requirement, since > frontends are already free to enforce that rule if they want to (by > checking for PKG-INFO before they call build_wheel), and particular > backends are also free to enforce it for the packages they manage (by > raising NotImplemented when they don't find PKG-INFO). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Aug 18 02:09:25 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Aug 2017 16:09:25 +1000 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> Message-ID: On 18 August 2017 at 14:16, xoviat wrote: > I'm still not convinced, but this PEP has been WIP for too long. I think it > should be marked Accepted. As previously noted, I'm mostly ready to mark it as Accepted (we're just missing some specific wording around the use of NotImplementedError to signal anticipated backend failures from build_sdist and build_wheel that may trigger a fallback option in the frontend). The other aspect I'm not clear on is whether or not both PEP authors are otherwise happy for the current version to be *considered* for acceptance - that requires an explicit request for pronouncement, and we haven't had a fresh one of those since the last round of updates :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From iahartle at ucsd.edu Thu Aug 17 22:17:25 2017 From: iahartle at ucsd.edu (Ian Hartley) Date: Thu, 17 Aug 2017 19:17:25 -0700 Subject: [Distutils] Transfering modules Message-ID: Hello, I have a question that I haven't been able to find online or in the python documentation. While trying to install a module I have encountered significant trouble, stalling my current experiment. I recently installed python 3.6.2, and I want to know how to transfer the numerous modules I have on 3.6.1 to the newest version, usable by an interpreter. Since I was not able to transfer packages to the newer version, I tried to install my module to the working python 3.6.1 version, but similarly, could not find any documentation about how to do this. I also tried to completely uninstall the python 3.6.2 build, but after running sudo-remove commands, and then testing by booting python3, the build running is still 3.6.2, even after removing the framework etc. I am sincerely confused by this and it has unfortunately brought my experiment to a standstill. If you could offer any illumination on this, I would be extremely grateful. I truly thank you in advance for any help you can provide. Best regards, Ian Hartley -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Aug 18 04:03:37 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Aug 2017 18:03:37 +1000 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: On 18 August 2017 at 12:17, Ian Hartley wrote: > Hello, > > I have a question that I haven't been able to find online or in the python > documentation. > While trying to install a module I have encountered significant trouble, > stalling my current experiment. > > I recently installed python 3.6.2, and I want to know how to transfer the > numerous modules I have on 3.6.1 to the newest version, usable by an > interpreter. Without knowing more about your system (operating system, where you installed Python from, how you're installing Python dependencies), it's hard to say what might be going wrong, since most mechanisms for upgrading Python will treat 3.6.2 as a direct replacement for 3.6.1, and hence you shouldn't have to do anything at all to start using Python 3.6.2 instead (since they're part of the same feature release stream, they'll both use `3.6` to qualify the directories they access, and hence should be able to see the same set of importable modules). However, from your comments, it sounds like you managed to do a parallel install instead, in which case the answers would be: 1. Don't do that, as you'll make life more difficult for yourself by defeating the main point of offering in-place maintenance updates 2. If you really need to do it, *and* you're using a virtual environment to manage your dependencies, then you can technically change which Python an existing virtual environment is using by fiddling with the symlinks (but I wouldn't recommend it) 3. If you're *not* using a virtual environment to manage your dependencies, then running `python3 -m site` with each version will tell you where their respective site-packages are, so you can copy all the files from your 3.6.1 installation to your new 3.6.2 installation (although again, I wouldn't really recommend it) The ideal case would be having all your dependencies listed in a requirements.txt file, so setting up a new virtual environment would just be a matter of activating it and then running "pip install -r requirements.txt". (Or, equivalently, define a conda environment file and set up your experimental system with "conda env create": https://conda.io/docs/using/envs.html#create-environment-file-by-hand) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Fri Aug 18 04:09:57 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 18 Aug 2017 09:09:57 +0100 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> Message-ID: <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: > The other aspect I'm not clear on is whether or not both PEP authors > are otherwise happy for the current version to be *considered* for > acceptance Nathaniel has mentioned to me a list of issues he sees that we still need to work through. So I think we're going to be working on this for a while yet. Thomas From waynejwerner at gmail.com Fri Aug 18 07:57:38 2017 From: waynejwerner at gmail.com (Wayne Werner) Date: Fri, 18 Aug 2017 11:57:38 +0000 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: You could potentially also run `/path/to/python3.6.2/bin/python3 -m pip install $(/path/to/python3.6.1/bin/python3 -m pip freeze)`, naturally replacing path with the correct path. pip freeze lists the versions of things you have installed. HTH, Wayne On Fri, Aug 18, 2017, 3:03 AM Nick Coghlan wrote: > On 18 August 2017 at 12:17, Ian Hartley wrote: > > Hello, > > > > I have a question that I haven't been able to find online or in the > python > > documentation. > > While trying to install a module I have encountered significant trouble, > > stalling my current experiment. > > > > I recently installed python 3.6.2, and I want to know how to transfer the > > numerous modules I have on 3.6.1 to the newest version, usable by an > > interpreter. > > Without knowing more about your system (operating system, where you > installed Python from, how you're installing Python dependencies), > it's hard to say what might be going wrong, since most mechanisms for > upgrading Python will treat 3.6.2 as a direct replacement for 3.6.1, > and hence you shouldn't have to do anything at all to start using > Python 3.6.2 instead (since they're part of the same feature release > stream, they'll both use `3.6` to qualify the directories they access, > and hence should be able to see the same set of importable modules). > > However, from your comments, it sounds like you managed to do a > parallel install instead, in which case the answers would be: > > 1. Don't do that, as you'll make life more difficult for yourself by > defeating the main point of offering in-place maintenance updates > 2. If you really need to do it, *and* you're using a virtual > environment to manage your dependencies, then you can technically > change which Python an existing virtual environment is using by > fiddling with the symlinks (but I wouldn't recommend it) > 3. If you're *not* using a virtual environment to manage your > dependencies, then running `python3 -m site` with each version will > tell you where their respective site-packages are, so you can copy all > the files from your 3.6.1 installation to your new 3.6.2 installation > (although again, I wouldn't really recommend it) > > The ideal case would be having all your dependencies listed in a > requirements.txt file, so setting up a new virtual environment would > just be a matter of activating it and then running "pip install -r > requirements.txt". (Or, equivalently, define a conda environment file > and set up your experimental system with "conda env create": > https://conda.io/docs/using/envs.html#create-environment-file-by-hand) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 18 13:23:50 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 18 Aug 2017 12:23:50 -0500 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> Message-ID: Thomas: What are the specific issues that need to be worked out? Regards, xoviat 2017-08-18 3:09 GMT-05:00 Thomas Kluyver : > On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: > > The other aspect I'm not clear on is whether or not both PEP authors > > are otherwise happy for the current version to be *considered* for > > acceptance > > Nathaniel has mentioned to me a list of issues he sees that we still > need to work through. So I think we're going to be working on this for a > while yet. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Aug 18 13:25:41 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 18 Aug 2017 18:25:41 +0100 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> Message-ID: <1503077141.944012.1077810968.5A4478BA@webmail.messagingengine.com> I'll let Nathaniel relay them. On Fri, Aug 18, 2017, at 06:23 PM, xoviat wrote: > Thomas: > > > What are the specific issues that need to be worked out? > > > Regards, > > > xoviat > > 2017-08-18 3:09 GMT-05:00 Thomas Kluyver : >> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: >> > The other aspect I'm not clear on is whether or not both PEP >> > authors are otherwise happy for the current version to be >> > *considered* for acceptance >> >> Nathaniel has mentioned to me a list of issues he sees that we still>> need to work through. So I think we're going to be working on >> this for a>> while yet. >> >> Thomas >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Aug 18 13:27:06 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 18 Aug 2017 17:27:06 +0000 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> Message-ID: Apart from the issues, can we get some prototype implementations? On Fri, Aug 18, 2017, 13:24 xoviat wrote: > Thomas: > > > What are the specific issues that need to be worked out? > > > Regards, > > > xoviat > > 2017-08-18 3:09 GMT-05:00 Thomas Kluyver : > >> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: >> > The other aspect I'm not clear on is whether or not both PEP authors >> > are otherwise happy for the current version to be *considered* for >> > acceptance >> >> Nathaniel has mentioned to me a list of issues he sees that we still >> need to work through. So I think we're going to be working on this for a >> while yet. >> >> Thomas >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Aug 18 13:37:46 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 18 Aug 2017 18:37:46 +0100 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> Message-ID: <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> There's prototype machinery to call the hooks here, though it's outdated because the spec keeps changing:https://github.com/takluyver/pep517 There's a prototype backend in a branch of flit, but it's even more outdated:https://github.com/takluyver/flit/tree/toml-config On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote: > Apart from the issues, can we get some prototype implementations? > > On Fri, Aug 18, 2017, 13:24 xoviat wrote: >> Thomas: >> >> >> What are the specific issues that need to be worked out? >> >> >> Regards, >> >> >> xoviat >> >> 2017-08-18 3:09 GMT-05:00 Thomas Kluyver : >>> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: >>> > The other aspect I'm not clear on is whether or not both PEP >>> > authors are otherwise happy for the current version to be >>> > *considered* for acceptance >>> >>> Nathaniel has mentioned to me a list of issues he sees that we still>>> need to work through. So I think we're going to be working on this >>> for a>>> while yet. >>> >>> Thomas >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 18 15:08:16 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 18 Aug 2017 14:08:16 -0500 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> Message-ID: I have attempted to implement prototype machinery here and here , but it's been widely ignored IHMO because as Thomas said, the spec keeps changing. 2017-08-18 12:37 GMT-05:00 Thomas Kluyver : > There's prototype machinery to call the hooks here, though it's outdated > because the spec keeps changing: > https://github.com/takluyver/pep517 > > There's a prototype backend in a branch of flit, but it's even more > outdated: > https://github.com/takluyver/flit/tree/toml-config > > > On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote: > > Apart from the issues, can we get some prototype implementations? > > On Fri, Aug 18, 2017, 13:24 xoviat wrote: > > Thomas: > > > What are the specific issues that need to be worked out? > > > Regards, > > > xoviat > > 2017-08-18 3:09 GMT-05:00 Thomas Kluyver : > > On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: > > The other aspect I'm not clear on is whether or not both PEP authors > > are otherwise happy for the current version to be *considered* for > > acceptance > > Nathaniel has mentioned to me a list of issues he sees that we still > need to work through. So I think we're going to be working on this for a > while yet. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 18 15:17:21 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 18 Aug 2017 14:17:21 -0500 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> Message-ID: At this point, I can only offer implementations as I have been attempting to do (some of which are outdated because the PEP has changed), but I cannot make anyone pay attention to or accept my pull requests and it seems I cannot accelerate the discussion here. 2017-08-18 14:08 GMT-05:00 xoviat : > I have attempted to implement prototype machinery here > and here > , but it's been widely > ignored IHMO because as Thomas said, the spec keeps changing. > > 2017-08-18 12:37 GMT-05:00 Thomas Kluyver : > >> There's prototype machinery to call the hooks here, though it's outdated >> because the spec keeps changing: >> https://github.com/takluyver/pep517 >> >> There's a prototype backend in a branch of flit, but it's even more >> outdated: >> https://github.com/takluyver/flit/tree/toml-config >> >> >> On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote: >> >> Apart from the issues, can we get some prototype implementations? >> >> On Fri, Aug 18, 2017, 13:24 xoviat wrote: >> >> Thomas: >> >> >> What are the specific issues that need to be worked out? >> >> >> Regards, >> >> >> xoviat >> >> 2017-08-18 3:09 GMT-05:00 Thomas Kluyver : >> >> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: >> > The other aspect I'm not clear on is whether or not both PEP authors >> > are otherwise happy for the current version to be *considered* for >> > acceptance >> >> Nathaniel has mentioned to me a list of issues he sees that we still >> need to work through. So I think we're going to be working on this for a >> while yet. >> >> Thomas >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Aug 18 15:28:41 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 18 Aug 2017 20:28:41 +0100 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> Message-ID: <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> I wouldn't expect established tools like pip & setuptools to accept pull requests implementing a PEP which is still under discussion and subject to change. They will naturally want a stable, accepted spec before they consider merging code for it. We've probably all wished that the discussion could be brought to a swift conclusion. But there are real questions to work out, and people have many other things to pay attention to. I'm frustrated by how long it's taking as well, but there's no magic button anyone can press to make it go quickly. Thomas On Fri, Aug 18, 2017, at 08:17 PM, xoviat wrote: > At this point, I can only offer implementations as I have been > attempting to do (some of which are outdated because the PEP has > changed), but I cannot make anyone pay attention to or accept my pull > requests and it seems I cannot accelerate the discussion here.> > 2017-08-18 14:08 GMT-05:00 xoviat : >> I have attempted to implement prototype machinery here[1] and >> here[2], but it's been widely ignored IHMO because as Thomas said, >> the spec keeps changing.>> >> 2017-08-18 12:37 GMT-05:00 Thomas Kluyver : >>> __ >>> There's prototype machinery to call the hooks here, though it's >>> outdated because the spec keeps changing:>>> https://github.com/takluyver/pep517 >>> >>> There's a prototype backend in a branch of flit, but it's even more >>> outdated:>>> https://github.com/takluyver/flit/tree/toml-config >>> >>> >>> On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote: >>>> Apart from the issues, can we get some prototype implementations?>>>> >>>> On Fri, Aug 18, 2017, 13:24 xoviat wrote: >>>>> Thomas: >>>>> >>>>> >>>>> What are the specific issues that need to be worked out? >>>>> >>>>> >>>>> Regards, >>>>> >>>>> >>>>> xoviat >>>>> >>>>> 2017-08-18 3:09 GMT-05:00 Thomas Kluyver : >>>>>> On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: >>>>>> > The other aspect I'm not clear on is whether or not both PEP >>>>>> > authors are otherwise happy for the current version to be >>>>>> > *considered* for acceptance >>>>>> >>>>>> Nathaniel has mentioned to me a list of issues he sees that we >>>>>> still>>>>>> need to work through. So I think we're going to be working on >>>>>> this for a>>>>>> while yet. >>>>>> >>>>>> Thomas >>>>>> _______________________________________________ >>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>>> >>>>> _______________________________________________ >>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>> https://mail.python.org/mailman/listinfo/distutils-sig >>> Links: 1. https://github.com/pypa/pip/pull/4589 2. https://github.com/pypa/setuptools/pull/1039 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Aug 18 23:41:45 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Aug 2017 13:41:45 +1000 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: On 19 August 2017 at 05:28, Thomas Kluyver wrote: > We've probably all wished that the discussion could be brought to a swift > conclusion. But there are real questions to work out, and people have many > other things to pay attention to. I'm frustrated by how long it's taking as > well, but there's no magic button anyone can press to make it go quickly. Technically, commercial redistributors do have a magic button we could press called "ongoing funding for sustaining engineering in the upstream Python ecosystem" (since that kind of funding can also cover needs-driven UX improvements), but alas, whether or not we ever actually press that is conditional on potential customers pressing the "customer demand" button hard enough and often enough to light up the "viable business opportunity" sign :P I'm actually genuine curious to see how those commercial dynamics start changing as the end date for community support of the Python 2.x series gets closer :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Aug 18 23:46:25 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Aug 2017 13:46:25 +1000 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: On 19 August 2017 at 04:39, Ian Hartley wrote: > Thanks all, > > I was able to copy the files from the site-packages, and transfer to 3.6.2. > For some reason, even though I've transfered all the modules, only some are > appearing. Very strange. Either way I think it should work for now... > > Let me know if you know why only some of the packages would be showing up. I'm still unclear on how you even managed to get both 3.6.1 and 3.6.2 installed on the same system (since 3.6.2 would overwrite 3.6.1 entirely in most circumstances), and without knowing that it's hard to guess what might be going wrong in transferring packages between the two :) Would you mind posting the output of "python3 -m site" for both installations? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.jerdonek at gmail.com Sat Aug 19 00:03:18 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 18 Aug 2017 21:03:18 -0700 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: On Fri, Aug 18, 2017 at 1:03 AM, Nick Coghlan wrote: > On 18 August 2017 at 12:17, Ian Hartley wrote: >> I recently installed python 3.6.2, and I want to know how to transfer the >> numerous modules I have on 3.6.1 to the newest version, usable by an >> interpreter. > > Without knowing more about your system (operating system, where you > installed Python from, how you're installing Python dependencies), > it's hard to say what might be going wrong, since most mechanisms for > upgrading Python will treat 3.6.2 as a direct replacement for 3.6.1, > and hence you shouldn't have to do anything at all to start using > Python 3.6.2 instead (since they're part of the same feature release > stream, they'll both use `3.6` to qualify the directories they access, > and hence should be able to see the same set of importable modules). I would just like to point out that the normal workflow for pyenv [1], which seems very popular (and which I use), is to do parallel installs rather than upgrades. The OP didn't actually say "upgrade," so it's possible they're using a tool like pyenv. I believe the OP's original questions would also apply to users of pyenv -- i.e. having a simple way to "copy" installed dependencies from one pyenv install to a new one. In practice, I myself don't have this issue since I usually install from requirements.txt, as you suggested, Nick. --Chris [1] https://github.com/pyenv/pyenv > > However, from your comments, it sounds like you managed to do a > parallel install instead, in which case the answers would be: > > 1. Don't do that, as you'll make life more difficult for yourself by > defeating the main point of offering in-place maintenance updates > 2. If you really need to do it, *and* you're using a virtual > environment to manage your dependencies, then you can technically > change which Python an existing virtual environment is using by > fiddling with the symlinks (but I wouldn't recommend it) > 3. If you're *not* using a virtual environment to manage your > dependencies, then running `python3 -m site` with each version will > tell you where their respective site-packages are, so you can copy all > the files from your 3.6.1 installation to your new 3.6.2 installation > (although again, I wouldn't really recommend it) > > The ideal case would be having all your dependencies listed in a > requirements.txt file, so setting up a new virtual environment would > just be a matter of activating it and then running "pip install -r > requirements.txt". (Or, equivalently, define a conda environment file > and set up your experimental system with "conda env create": > https://conda.io/docs/using/envs.html#create-environment-file-by-hand) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Sat Aug 19 01:57:23 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Aug 2017 15:57:23 +1000 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: On 19 August 2017 at 14:03, Chris Jerdonek wrote: > I would just like to point out that the normal workflow for pyenv [1], > which seems very popular (and which I use), is to do parallel installs > rather than upgrades. The OP didn't actually say "upgrade," so it's > possible they're using a tool like pyenv. Right, it was the references to sudo-remove, rebooting, and python3 that indicated to me we were talking system wide operations, rather than any of the environment management tools like pew, pyenv, conda, etc. That's why I'm somewhat puzzled, since even a raw "make altinstall" will overwrite an existing installation of that version. The most likely candidate for a system level kinda-sorta parallel installation would be via the distinction between /usr/local/* and /usr/*, and if it's a Debian or Ubuntu system, that would result in the actual system version being able to see components in dist-packages that would be invisible to an unpatched upstream build, each version having its own site-packages directory, but both of them still sharing a common user site-packages directory. Hence my interest in the output of " -m site" for the affected versions: seeing the default contents of sys.path will likely answer the question of what's going on definitively, rather than having to guess based on incomplete information. I believe the OP's original > questions would also apply to users of pyenv -- i.e. having a simple > way to "copy" installed dependencies from one pyenv install to a new > one. As Wayne noted, generating a requirements.txt file with `pip freeze` and then reinstalling the dependencies essentially *is* that mechanism. If you still have the original wheelhouse cache around, it shouldn't be substantially slower than a plain filesystem copy (since it will just be a matter of unpacking the archived wheel files again, rather than doing any binary builds). For a more literal copy operation, there's "pew cp", but the documentation for that has "this may result in a subtly broken virtual environment" caveats on it for good reasons :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From iahartle at ucsd.edu Fri Aug 18 14:39:24 2017 From: iahartle at ucsd.edu (Ian Hartley) Date: Fri, 18 Aug 2017 11:39:24 -0700 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: Thanks all, I was able to copy the files from the site-packages, and transfer to 3.6.2. For some reason, even though I've transfered all the modules, only some are appearing. Very strange. Either way I think it should work for now... Let me know if you know why only some of the packages would be showing up. Thanks for all your help. Best, Ian On Fri, Aug 18, 2017 at 4:57 AM, Wayne Werner wrote: > You could potentially also run `/path/to/python3.6.2/bin/python3 -m pip > install $(/path/to/python3.6.1/bin/python3 -m pip freeze)`, naturally > replacing path with the correct path. pip freeze lists the versions of > things you have installed. > > HTH, > Wayne > > On Fri, Aug 18, 2017, 3:03 AM Nick Coghlan wrote: > >> On 18 August 2017 at 12:17, Ian Hartley wrote: >> > Hello, >> > >> > I have a question that I haven't been able to find online or in the >> python >> > documentation. >> > While trying to install a module I have encountered significant trouble, >> > stalling my current experiment. >> > >> > I recently installed python 3.6.2, and I want to know how to transfer >> the >> > numerous modules I have on 3.6.1 to the newest version, usable by an >> > interpreter. >> >> Without knowing more about your system (operating system, where you >> installed Python from, how you're installing Python dependencies), >> it's hard to say what might be going wrong, since most mechanisms for >> upgrading Python will treat 3.6.2 as a direct replacement for 3.6.1, >> and hence you shouldn't have to do anything at all to start using >> Python 3.6.2 instead (since they're part of the same feature release >> stream, they'll both use `3.6` to qualify the directories they access, >> and hence should be able to see the same set of importable modules). >> >> However, from your comments, it sounds like you managed to do a >> parallel install instead, in which case the answers would be: >> >> 1. Don't do that, as you'll make life more difficult for yourself by >> defeating the main point of offering in-place maintenance updates >> 2. If you really need to do it, *and* you're using a virtual >> environment to manage your dependencies, then you can technically >> change which Python an existing virtual environment is using by >> fiddling with the symlinks (but I wouldn't recommend it) >> 3. If you're *not* using a virtual environment to manage your >> dependencies, then running `python3 -m site` with each version will >> tell you where their respective site-packages are, so you can copy all >> the files from your 3.6.1 installation to your new 3.6.2 installation >> (although again, I wouldn't really recommend it) >> >> The ideal case would be having all your dependencies listed in a >> requirements.txt file, so setting up a new virtual environment would >> just be a matter of activating it and then running "pip install -r >> requirements.txt". (Or, equivalently, define a conda environment file >> and set up your experimental system with "conda env create": >> https://conda.io/docs/using/envs.html#create-environment-file-by-hand) >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 19 14:23:46 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 19 Aug 2017 14:23:46 -0400 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: I assume that the outstanding issues mentioned are related to sys.path in the build tree. My view on that is the following: the build frontend should be responsible using any mechanism that it chooses for invoking the build backend, which must be imported with '' at the front of sys.path. This obviously means that if the build frontend experiences faults before it manages to import and invoke the backend, then it's the build frontend's fault. The only potential issue remaining that I can think of then is: what about modules that are already imported when the build backend is called? WRT standard library modules, we could simply say that build backends must be prepared to function in any environment where standard library modules are already imported, which I think is a reasonable requirement. But what about imports that aren't standard library modules but are fairly common, like pip imports? The simplest way to simplify the interface is probably to say that all non-standard library modules must be removed from sys.modules (unloaded) before the build backend is called. What do others think? 2017-08-18 23:41 GMT-04:00 Nick Coghlan : > On 19 August 2017 at 05:28, Thomas Kluyver wrote: > > We've probably all wished that the discussion could be brought to a swift > > conclusion. But there are real questions to work out, and people have > many > > other things to pay attention to. I'm frustrated by how long it's taking > as > > well, but there's no magic button anyone can press to make it go quickly. > > Technically, commercial redistributors do have a magic button we could > press called "ongoing funding for sustaining engineering in the > upstream Python ecosystem" (since that kind of funding can also cover > needs-driven UX improvements), but alas, whether or not we ever > actually press that is conditional on potential customers pressing the > "customer demand" button hard enough and often enough to light up the > "viable business opportunity" sign :P > > I'm actually genuine curious to see how those commercial dynamics > start changing as the end date for community support of the Python 2.x > series gets closer :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 19 14:30:40 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 19 Aug 2017 14:30:40 -0400 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: Ah the joy of Python 2.7; it seems I've forgotten its perils: unloading is not possible which means that we would need to come up with another solution to this problem. Perhaps setting aside a namespace for the build frontend in the subprocess? 2017-08-19 14:23 GMT-04:00 xoviat : > I assume that the outstanding issues mentioned are related to sys.path in > the build tree. My view on that is the following: the build frontend should > be responsible using any mechanism that it chooses for invoking the build > backend, which must be imported with '' at the front of sys.path. This > obviously means that if the build frontend experiences faults before it > manages to import and invoke the backend, then it's the build frontend's > fault. > > The only potential issue remaining that I can think of then is: what about > modules that are already imported when the build backend is called? WRT > standard library modules, we could simply say that build backends must be > prepared to function in any environment where standard library modules are > already imported, which I think is a reasonable requirement. But what about > imports that aren't standard library modules but are fairly common, like > pip imports? The simplest way to simplify the interface is probably to say > that all non-standard library modules must be removed from sys.modules > (unloaded) before the build backend is called. > > What do others think? > > 2017-08-18 23:41 GMT-04:00 Nick Coghlan : > >> On 19 August 2017 at 05:28, Thomas Kluyver wrote: >> > We've probably all wished that the discussion could be brought to a >> swift >> > conclusion. But there are real questions to work out, and people have >> many >> > other things to pay attention to. I'm frustrated by how long it's >> taking as >> > well, but there's no magic button anyone can press to make it go >> quickly. >> >> Technically, commercial redistributors do have a magic button we could >> press called "ongoing funding for sustaining engineering in the >> upstream Python ecosystem" (since that kind of funding can also cover >> needs-driven UX improvements), but alas, whether or not we ever >> actually press that is conditional on potential customers pressing the >> "customer demand" button hard enough and often enough to light up the >> "viable business opportunity" sign :P >> >> I'm actually genuine curious to see how those commercial dynamics >> start changing as the end date for community support of the Python 2.x >> series gets closer :) >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Aug 19 14:38:10 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 19 Aug 2017 19:38:10 +0100 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: If this were to be done, then IMO yes, a PEP would be the right way to standardise the JSON API. But tools like pip don't use the JSON API much, and tools like devpi that expose the index API don't bother with the JSON API (so making it less likely that consumers that want to work with indexes other than PyPI will use it). So you may not get much interest. On the other hand, a PEP that simply documents the API and says "Index providers that choose to support the JSON API must do so with this interface" would probably be useful, and unlikely to get a lot of pushback (assuming you document what Warehouse and PyPI provide, and allow other providers to simply not provide anything). Paul On 13 August 2017 at 07:53, Cooper Ry Lees wrote: > Hi all, > > First time emailer, so please be kind. Also, if this is not the right > mailing list for PyPA talk, I apologize. Please point me in the right > direction if so (Brett Canon pointed me here). The main reason I have > emailed here is I believe it may be PEP time to standardize the JSON > metadata that PyPI makes available, like what was done for the `'simple API` > described in PEP503. > > I've been doing a bit of work on `bandersnatch` (I didn't name it), which is > a PEP 381 mirroring package and wanted to enhance it to also mirror the > handy JSON metadata PyPI generates and makes available @ > https://pypi.python.org/pypi/PKG_NAME/json. > > I've done a PR on bandersnatch as a POC that mirrors both the PyPI directory > structure (URL/pypi/PKG_NAME/json) and created a standardizable > URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some > other proxy). I'm also contemplating naming the directory 'metadata' rather > than JSON so if some new hotness / we want to change the format down the > line we're not stuck with json as the dirname. This PR can be found here: > https://bitbucket.org/pypa/bandersnatch/pull-requests/33/save-json-metadata-to-mirror > > My main use case is to write a very simple async 'verifier' tool that will > crawl all the JSON files and then ensure the packages directory on each of > my internal mirrors (I have a mirror per region / datacenter) have all the > files they should. I sync centrally (to save resource on the PyPI > infrastructure) and then rsync out all the diffs to each region / > datacenter, and under some failure scenarios I could miss a file or many. So > I feel using JSON pulled down from the authoritative source will allow an > async job to verify the MD5 of all the package files on each mirror. > > What are peoples thoughts here? Is it worth a PEP similar to PEP503 going > forward? Can people enhance / share some thoughts on this idea. > > Thanks, > Cooper Ry Lees > me at cooperlees.com > https://cooperlees.com/ > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From dholth at gmail.com Sat Aug 19 16:10:53 2017 From: dholth at gmail.com (Daniel Holth) Date: Sat, 19 Aug 2017 20:10:53 +0000 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: It's probably a tiny wrapper running the hook in its own subprocess. Probably few modules loaded. On Sat, Aug 19, 2017, 14:31 xoviat wrote: > Ah the joy of Python 2.7; it seems I've forgotten its perils: unloading is > not possible which means that we would need to come up with another > solution to this problem. Perhaps setting aside a namespace for the build > frontend in the subprocess? > > 2017-08-19 14:23 GMT-04:00 xoviat : > >> I assume that the outstanding issues mentioned are related to sys.path in >> the build tree. My view on that is the following: the build frontend should >> be responsible using any mechanism that it chooses for invoking the build >> backend, which must be imported with '' at the front of sys.path. This >> obviously means that if the build frontend experiences faults before it >> manages to import and invoke the backend, then it's the build frontend's >> fault. >> >> The only potential issue remaining that I can think of then is: what >> about modules that are already imported when the build backend is called? >> WRT standard library modules, we could simply say that build backends must >> be prepared to function in any environment where standard library modules >> are already imported, which I think is a reasonable requirement. But what >> about imports that aren't standard library modules but are fairly common, >> like pip imports? The simplest way to simplify the interface is probably to >> say that all non-standard library modules must be removed from sys.modules >> (unloaded) before the build backend is called. >> >> What do others think? >> >> 2017-08-18 23:41 GMT-04:00 Nick Coghlan : >> >>> On 19 August 2017 at 05:28, Thomas Kluyver wrote: >>> > We've probably all wished that the discussion could be brought to a >>> swift >>> > conclusion. But there are real questions to work out, and people have >>> many >>> > other things to pay attention to. I'm frustrated by how long it's >>> taking as >>> > well, but there's no magic button anyone can press to make it go >>> quickly. >>> >>> Technically, commercial redistributors do have a magic button we could >>> press called "ongoing funding for sustaining engineering in the >>> upstream Python ecosystem" (since that kind of funding can also cover >>> needs-driven UX improvements), but alas, whether or not we ever >>> actually press that is conditional on potential customers pressing the >>> "customer demand" button hard enough and often enough to light up the >>> "viable business opportunity" sign :P >>> >>> I'm actually genuine curious to see how those commercial dynamics >>> start changing as the end date for community support of the Python 2.x >>> series gets closer :) >>> >>> Cheers, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> >> >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 19 16:22:06 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 19 Aug 2017 16:22:06 -0400 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: Yes, it probably is. But then again, I assumed that it was obvious that PYTHONPATH would not be modified before invoking the hook (at least obvious to me) until the 'PEP 517 Status' discussion implied that it was not so obvious and needed to be specified in the PEP. 2017-08-19 16:10 GMT-04:00 Daniel Holth : > It's probably a tiny wrapper running the hook in its own subprocess. > Probably few modules loaded. > > On Sat, Aug 19, 2017, 14:31 xoviat wrote: > >> Ah the joy of Python 2.7; it seems I've forgotten its perils: unloading >> is not possible which means that we would need to come up with another >> solution to this problem. Perhaps setting aside a namespace for the build >> frontend in the subprocess? >> >> 2017-08-19 14:23 GMT-04:00 xoviat : >> >>> I assume that the outstanding issues mentioned are related to sys.path >>> in the build tree. My view on that is the following: the build frontend >>> should be responsible using any mechanism that it chooses for invoking the >>> build backend, which must be imported with '' at the front of sys.path. >>> This obviously means that if the build frontend experiences faults before >>> it manages to import and invoke the backend, then it's the build frontend's >>> fault. >>> >>> The only potential issue remaining that I can think of then is: what >>> about modules that are already imported when the build backend is called? >>> WRT standard library modules, we could simply say that build backends must >>> be prepared to function in any environment where standard library modules >>> are already imported, which I think is a reasonable requirement. But what >>> about imports that aren't standard library modules but are fairly common, >>> like pip imports? The simplest way to simplify the interface is probably to >>> say that all non-standard library modules must be removed from sys.modules >>> (unloaded) before the build backend is called. >>> >>> What do others think? >>> >>> 2017-08-18 23:41 GMT-04:00 Nick Coghlan : >>> >>>> On 19 August 2017 at 05:28, Thomas Kluyver >>>> wrote: >>>> > We've probably all wished that the discussion could be brought to a >>>> swift >>>> > conclusion. But there are real questions to work out, and people have >>>> many >>>> > other things to pay attention to. I'm frustrated by how long it's >>>> taking as >>>> > well, but there's no magic button anyone can press to make it go >>>> quickly. >>>> >>>> Technically, commercial redistributors do have a magic button we could >>>> press called "ongoing funding for sustaining engineering in the >>>> upstream Python ecosystem" (since that kind of funding can also cover >>>> needs-driven UX improvements), but alas, whether or not we ever >>>> actually press that is conditional on potential customers pressing the >>>> "customer demand" button hard enough and often enough to light up the >>>> "viable business opportunity" sign :P >>>> >>>> I'm actually genuine curious to see how those commercial dynamics >>>> start changing as the end date for community support of the Python 2.x >>>> series gets closer :) >>>> >>>> Cheers, >>>> Nick. >>>> >>>> -- >>>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>>> >>> >>> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 19 16:23:17 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 19 Aug 2017 16:23:17 -0400 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: Excuse me, but what I meant to say is that sys.path would be adjusted after the subprocess was loaded (in my implementation I adjust sys.path and os.environ['PYTHONPATH']). 2017-08-19 16:22 GMT-04:00 xoviat : > Yes, it probably is. But then again, I assumed that it was obvious that > PYTHONPATH would not be modified before invoking the hook (at least obvious > to me) until the 'PEP 517 Status' discussion implied that it was not so > obvious and needed to be specified in the PEP. > > 2017-08-19 16:10 GMT-04:00 Daniel Holth : > >> It's probably a tiny wrapper running the hook in its own subprocess. >> Probably few modules loaded. >> >> On Sat, Aug 19, 2017, 14:31 xoviat wrote: >> >>> Ah the joy of Python 2.7; it seems I've forgotten its perils: unloading >>> is not possible which means that we would need to come up with another >>> solution to this problem. Perhaps setting aside a namespace for the build >>> frontend in the subprocess? >>> >>> 2017-08-19 14:23 GMT-04:00 xoviat : >>> >>>> I assume that the outstanding issues mentioned are related to sys.path >>>> in the build tree. My view on that is the following: the build frontend >>>> should be responsible using any mechanism that it chooses for invoking the >>>> build backend, which must be imported with '' at the front of sys.path. >>>> This obviously means that if the build frontend experiences faults before >>>> it manages to import and invoke the backend, then it's the build frontend's >>>> fault. >>>> >>>> The only potential issue remaining that I can think of then is: what >>>> about modules that are already imported when the build backend is called? >>>> WRT standard library modules, we could simply say that build backends must >>>> be prepared to function in any environment where standard library modules >>>> are already imported, which I think is a reasonable requirement. But what >>>> about imports that aren't standard library modules but are fairly common, >>>> like pip imports? The simplest way to simplify the interface is probably to >>>> say that all non-standard library modules must be removed from sys.modules >>>> (unloaded) before the build backend is called. >>>> >>>> What do others think? >>>> >>>> 2017-08-18 23:41 GMT-04:00 Nick Coghlan : >>>> >>>>> On 19 August 2017 at 05:28, Thomas Kluyver >>>>> wrote: >>>>> > We've probably all wished that the discussion could be brought to a >>>>> swift >>>>> > conclusion. But there are real questions to work out, and people >>>>> have many >>>>> > other things to pay attention to. I'm frustrated by how long it's >>>>> taking as >>>>> > well, but there's no magic button anyone can press to make it go >>>>> quickly. >>>>> >>>>> Technically, commercial redistributors do have a magic button we could >>>>> press called "ongoing funding for sustaining engineering in the >>>>> upstream Python ecosystem" (since that kind of funding can also cover >>>>> needs-driven UX improvements), but alas, whether or not we ever >>>>> actually press that is conditional on potential customers pressing the >>>>> "customer demand" button hard enough and often enough to light up the >>>>> "viable business opportunity" sign :P >>>>> >>>>> I'm actually genuine curious to see how those commercial dynamics >>>>> start changing as the end date for community support of the Python 2.x >>>>> series gets closer :) >>>>> >>>>> Cheers, >>>>> Nick. >>>>> >>>>> -- >>>>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>>>> >>>> >>>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 19 16:30:15 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 19 Aug 2017 16:30:15 -0400 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: Also, I disagree with this point. I think it's still possible (and in fact preferable) to have setuptools and pip 'test this out' with appropriate fallbacks before opening this up to the wider community. Most people wouldn't even notice because they wouldn't be using HEAD. 2017-08-18 15:28 GMT-04:00 Thomas Kluyver : > I wouldn't expect established tools like pip & setuptools to accept pull > requests implementing a PEP which is still under discussion and subject to > change. They will naturally want a stable, accepted spec before they > consider merging code for it. > > We've probably all wished that the discussion could be brought to a swift > conclusion. But there are real questions to work out, and people have many > other things to pay attention to. I'm frustrated by how long it's taking as > well, but there's no magic button anyone can press to make it go quickly. > > Thomas > > > > > On Fri, Aug 18, 2017, at 08:17 PM, xoviat wrote: > > At this point, I can only offer implementations as I have been attempting > to do (some of which are outdated because the PEP has changed), but I > cannot make anyone pay attention to or accept my pull requests and it seems > I cannot accelerate the discussion here. > > 2017-08-18 14:08 GMT-05:00 xoviat : > > I have attempted to implement prototype machinery here > and here > , but it's been widely > ignored IHMO because as Thomas said, the spec keeps changing. > > 2017-08-18 12:37 GMT-05:00 Thomas Kluyver : > > > There's prototype machinery to call the hooks here, though it's outdated > because the spec keeps changing: > https://github.com/takluyver/pep517 > > There's a prototype backend in a branch of flit, but it's even more > outdated: > https://github.com/takluyver/flit/tree/toml-config > > > On Fri, Aug 18, 2017, at 06:27 PM, Daniel Holth wrote: > > Apart from the issues, can we get some prototype implementations? > > On Fri, Aug 18, 2017, 13:24 xoviat wrote: > > Thomas: > > > What are the specific issues that need to be worked out? > > > Regards, > > > xoviat > > 2017-08-18 3:09 GMT-05:00 Thomas Kluyver : > > On Fri, Aug 18, 2017, at 07:09 AM, Nick Coghlan wrote: > > The other aspect I'm not clear on is whether or not both PEP authors > > are otherwise happy for the current version to be *considered* for > > acceptance > > Nathaniel has mentioned to me a list of issues he sees that we still > need to work through. So I think we're going to be working on this for a > while yet. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Aug 19 16:48:34 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 19 Aug 2017 21:48:34 +0100 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: On 19 August 2017 at 21:30, xoviat wrote: > Also, I disagree with this point. I think it's still possible (and in fact > preferable) to have setuptools and pip 'test this out' with appropriate > fallbacks before opening this up to the wider community. Most people > wouldn't even notice because they wouldn't be using HEAD. The same people who would notice it are probably just as able to apply a patch/PR or grab a fork. So your PRs are available for people to test if they want to, they don't need to be merged for that to happen. The setuptools and pip master branches are for things which *are* expected to go into the next version, not for experimental changes (for pip, we should be able to release from master whenever we want to, I suspect setuptools has a similar policy). Paul From xoviat at gmail.com Sat Aug 19 17:00:07 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 19 Aug 2017 17:00:07 -0400 Subject: [Distutils] PEP 517: Build frontend responsibilities In-Reply-To: References: <5995beae.86f0ca0a.da31f.d175@mx.google.com> <1503043797.3894720.1077320960.46CA9C4E@webmail.messagingengine.com> <1503077866.946662.1077820024.3CABD170@webmail.messagingengine.com> <1503084521.979774.1077915816.6527E300@webmail.messagingengine.com> Message-ID: > The setuptools and pip master branches are for things which *are* expected to go into the next version, not for experimental changes I agree with that statement (the PRs as they currently are need work because they have become stale) in the sense that code in the master branch should work at all times. But that's different than the PEP being fully fleshed out for all build systems rather than just pip and setuptools. And also, a big reason for this is that to be frank, the first implementation will probably not work. That's not just some special rule that I have applied to myself, but it's also the reality with PEP 518. The reality is that PEP 518 does not currently work and it is in the master branch, but people did not discover that fact until it was in the master branch. You can't know what you can't know. If an implementation that does not comply with the PEP yet but works with appropriate fallbacks to verify setuptools support and compliance, then it's not the end of the world if the packages are still built and installed correctly. 2017-08-19 16:48 GMT-04:00 Paul Moore : > On 19 August 2017 at 21:30, xoviat wrote: > > Also, I disagree with this point. I think it's still possible (and in > fact > > preferable) to have setuptools and pip 'test this out' with appropriate > > fallbacks before opening this up to the wider community. Most people > > wouldn't even notice because they wouldn't be using HEAD. > > The same people who would notice it are probably just as able to apply > a patch/PR or grab a fork. So your PRs are available for people to > test if they want to, they don't need to be merged for that to happen. > > The setuptools and pip master branches are for things which *are* > expected to go into the next version, not for experimental changes > (for pip, we should be able to release from master whenever we want > to, I suspect setuptools has a similar policy). > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sat Aug 19 18:03:39 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 19 Aug 2017 17:03:39 -0500 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: (cc'ing here from python-ideas) Here are some notes re: changing metadata: https://github.com/pypa/interoperability-peps/issues/31 (closed; but still very relevant) https://www.google.com/search?q=pep426jsonld Towards JSONLD is the best approach, I think. So, that means it would be best to, if you need to add additional metadata (?) and must key things, also copy the key into an object: {"thing1": {"@id": "thing1", "url": "..."}} Instead of just: {"thing1": {"url": "..."}} https://github.com/pypa/interoperability-peps/issues/31#issuecomment-233195564 On Sunday, August 13, 2017, Cooper Ry Lees wrote: > Hi all, > > First time emailer, so please be kind. Also, if this is not the right > mailing list for PyPA talk, I apologize. Please point me in the right > direction if so (Brett Canon pointed me here). The main reason I have > emailed here is I believe it may be PEP time to standardize the JSON > metadata that PyPI makes available, like what was done for the `'simple > API` described in PEP503. > > I've been doing a bit of work on `bandersnatch` (I didn't name it), which > is a PEP 381 mirroring package and wanted to enhance it to also mirror the > handy JSON metadata PyPI generates and makes available @ > https://pypi.python.org/pypi/PKG_NAME/json. > > I've done a PR on bandersnatch as a POC that mirrors both the PyPI > directory structure (URL/pypi/PKG_NAME/json) and created a standardizable > URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some > other proxy). I'm also contemplating naming the directory 'metadata' rather > than JSON so if some new hotness / we want to change the format down the > line we're not stuck with json as the dirname. This PR can be found here: > https://bitbucket.org/pypa/bandersnatch/pull-requests/ > 33/save-json-metadata-to-mirror > > My main use case is to write a very simple async 'verifier' tool that will > crawl all the JSON files and then ensure the packages directory on each of > my internal mirrors (I have a mirror per region / datacenter) have all the > files they should. I sync centrally (to save resource on the PyPI > infrastructure) and then rsync out all the diffs to each region / > datacenter, and under some failure scenarios I could miss a file or many. > So I feel using JSON pulled down from the authoritative source will allow > an async job to verify the MD5 of all the package files on each mirror. > > What are peoples thoughts here? Is it worth a PEP similar to PEP503 going > forward? Can people enhance / share some thoughts on this idea. > > Thanks, > Cooper Ry Lees > me at cooperlees.com > https://cooperlees.com/ > Here are some notes on making this more efficient: "Add API endpoint to get latest version of all projects" https://github.com/pypa/warehouse/issues/347 ... http://markmail.org/search/?q=list:org.python.distutils-sig + { metadata , pep426jsonld } -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Aug 19 23:56:05 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Aug 2017 13:56:05 +1000 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: On 20 August 2017 at 05:27, Ian Hartley wrote: > Hi all, > > I'm using PyCharm CE as my development environment, on a macintosh running > mac OS 10.12.3. > Ah, that would explain why my intuitions were all wrong - I know how the Windows and *nix installation models work (and I believe the latter also covers homebrew on macOS), but not how the macOS framework installers work :) While pip+wheels should cover most of your needs for financial analytics in Python these days, it may still be worth your while to spend a bit of time poking around at conda if you think you may ever need components written in other languages like R: https://conda.io/docs/ Cheers, Nick. P.S. As far as your research topic goes, while I'm not aware of anyone specifically researching tensorflow in that context, looking through Dr Yves Hilpisch's publication list at http://hilpisch.com/#/11 may point you towards events where that kind of material may have been presented. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Aug 20 07:09:55 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 20 Aug 2017 12:09:55 +0100 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: On 19 August 2017 at 23:03, Wes Turner wrote: > Here are some notes re: changing metadata: I thought the proposal was to document the current state of affairs. Proposing a *change* to the JSON API would be a much bigger and more controversial proposal (and one I see little need for, personally). Paul From wes.turner at gmail.com Sun Aug 20 10:51:41 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 20 Aug 2017 09:51:41 -0500 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: On Sunday, August 20, 2017, Paul Moore wrote: > On 19 August 2017 at 23:03, Wes Turner > wrote: > > Here are some notes re: changing metadata: > > I thought the proposal was to document the current state of affairs. There are links to the current PEPs and source codes in there, too. > Proposing a *change* to the JSON API would be a much bigger and more > controversial proposal (and one I see little need for, personally). We'd probably want to review how the dependency resolver work is going. https://github.com/pypa/pip/issues/988#issuecomment-322255801 https://pradyunsg.github.io/gsoc-2017/08/14/final-lap/ IIUC, the task is still to: Download transitive portions of a [linked data] graph as JSON[LD] (optimally without iteratively downloading and decompressing package archives in order to retrieve their platform-dependent dependency edge metadata from a setup.py that is executed with filesystem privileges). https://en.wikipedia.org/wiki/Pip_(Great_Expectations) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Aug 20 11:55:26 2017 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 20 Aug 2017 17:55:26 +0200 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: <1E085E4B-60DF-46E8-B19A-BFEA551FA13A@mac.com> > On 20 Aug 2017, at 05:56, Nick Coghlan wrote: > > On 20 August 2017 at 05:27, Ian Hartley > wrote: > Hi all, > > I'm using PyCharm CE as my development environment, on a macintosh running mac OS 10.12.3. > > Ah, that would explain why my intuitions were all wrong - I know how the Windows and *nix installation models work (and I believe the latter also covers homebrew on macOS), but not how the macOS framework installers work :) FWIW: The framework installers on python.org work just like the other installers on upgrades: installing 3.6.2 on a system already running 3.6.1 replaces the 3.6.1 installation and doesn?t perform a side-by-side installation. The major differences w.r.t. a *nix installation are having an ?odd? sys.prefix, and having different sys.prefix values for different feature releases. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Aug 21 04:54:30 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 21 Aug 2017 18:54:30 +1000 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: On 21 August 2017 at 00:51, Wes Turner wrote: > IIUC, the task is still to: > Download transitive portions of a [linked data] graph as JSON[LD] (optimally > without iteratively downloading and decompressing package archives in order > to retrieve their platform-dependent dependency edge metadata from a > setup.py that is executed with filesystem privileges). While I'm still generally negative on the idea of native reliance on JSON-LD, I'll note one thing that has changed since I last looked at it: I now see some potential concrete practical benefits to adopting it, rather than purely theoretical ones. In particular, https://github.com/scienceai/jsonld-vis now exists, and there wasn't anything like that around at the time of previous discussions. However, that's still only of potential interest for PEP 426, which in turn still isn't needed for any of our practical near term objectives (not even the "dependencies without downloads" one - if we were to prioritise that, we'd likely go for something closer to the way client side dependency resolution already works, such as extracting the METADATA file from uploaded wheels and making it available for download in addition to the full wheel archives). So for this thread, Paul's right: a PEP-503-style document describing the current PyPI JSON API would likely be a reasonable thing to write, as it would allow for more complete emulations of the current production PyPI service, including checking that Warehouse replicates that part of the API correctly. Anything more than that is still on the "No, not until some point after Legacy PyPI has been shut down" list. Cheers, Nick. P.S. Some of the tools mentioned at http://www.seoskeptic.com/structured-data-markup-validation-testing-tools/ may also prove useful if we go down the JSON-LD path. However, that is the only reason I'd ever support us going down that path: useful functionality that we get for free by virtue of adopting an established convention. I've wrangled volunteer contributors to open source projects for long enough now to know that "because it's the right thing to do" simply doesn't cut it as a motivational tool - there's need to be some kind of actual benefit to the folks doing the work :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Aug 21 04:59:18 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 21 Aug 2017 18:59:18 +1000 Subject: [Distutils] Transfering modules In-Reply-To: References: <1E085E4B-60DF-46E8-B19A-BFEA551FA13A@mac.com> Message-ID: On 21 August 2017 at 02:16, Ian Hartley wrote: > Ronald, > > I was sure that was the case re: python.org installation. It's strange that > it installed in two different locations, I really have no idea how that > might have come about. FYI I've included a pic of the file pathways when > you look at the interpreter photo. I'd first installed python using > terminal btw. Investigating a but further, I think that's actually the source of the discrepancy: /usr/local/Cellar is where homebrew puts things, while the python.org installers put things into /Library/Frameworks/. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Aug 21 05:38:16 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 21 Aug 2017 10:38:16 +0100 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: On 21 August 2017 at 09:54, Nick Coghlan wrote: > While I'm still generally negative on the idea of native reliance on > JSON-LD, I'll note one thing that has changed since I last looked at > it: I now see some potential concrete practical benefits to adopting > it, rather than purely theoretical ones. In particular, > https://github.com/scienceai/jsonld-vis now exists, and there wasn't > anything like that around at the time of previous discussions. Personally, I fairly often write adhoc scripts that use the JSON API, and as it stands it's very convenient for that. From what I can see of JSON-LD (which basically equates to "it adds some extra metadata keys that don't change the data content but do change the list of keys and maybe the nesting levels") it would be somewhat inconvenient for my scripts, and add no extra capability that I would ever use. Before we consider anything like JSON-LD, I think we need a much clearer picture of who uses the JSON API. If it's production-type applications, then maybe it would be useful, but if it's mostly advoc scripts (as I suspect) it's additional complexity for little or no benefit. But this remains off-topic for now, so that's all I'll say. Paul From setuptools at bugs.python.org Mon Aug 21 06:29:56 2017 From: setuptools at bugs.python.org (Salvo Tomaselli) Date: Mon, 21 Aug 2017 10:29:56 +0000 Subject: [Distutils] [issue166] Install path changes when data_files is present Message-ID: <1503311396.09.0.739850775737.issue166@psf.upfronthosting.co.za> New submission from Salvo Tomaselli: I have a python thing that I'm installing in a custom path, using setup.py install --root=/tmp/turi13 --install-purelib=custom/path However, if the setup.py file contains a data_files field, then, unless i also specify --install-data, the --install-purelib parameter gets ignored. I think an error should be generated and the entire thing should fail instead. Same for all of those --install-* parameters. ---------- messages: 791 nosy: ltworf priority: bug status: unread title: Install path changes when data_files is present _______________________________________________ Setuptools tracker _______________________________________________ From ncoghlan at gmail.com Mon Aug 21 06:53:17 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 21 Aug 2017 20:53:17 +1000 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: On 21 August 2017 at 19:38, Paul Moore wrote: > On 21 August 2017 at 09:54, Nick Coghlan wrote: >> While I'm still generally negative on the idea of native reliance on >> JSON-LD, I'll note one thing that has changed since I last looked at >> it: I now see some potential concrete practical benefits to adopting >> it, rather than purely theoretical ones. In particular, >> https://github.com/scienceai/jsonld-vis now exists, and there wasn't >> anything like that around at the time of previous discussions. > > Personally, I fairly often write adhoc scripts that use the JSON API, > and as it stands it's very convenient for that. From what I can see of > JSON-LD (which basically equates to "it adds some extra metadata keys > that don't change the data content but do change the list of keys and > maybe the nesting levels") it would be somewhat inconvenient for my > scripts, and add no extra capability that I would ever use. Right, and this is still my main concern with the idea as well: I'd never be OK with a JSON-LD-only API, because it adds too much irrelevant cognitive overhead for the vast majority of Python packaging specific use cases. (I would see it as being akin to Python itself deciding to require type annotations, rather than merely allowing them). However, where I'm starting to see a potential niche for it is as an opt-in capability, whereby we explicitly define how our metadata can be translated *to* JSON-LD, for folks that want to apply general purpose tools that know how to manipulate arbitrary JSON-LD data (like the graph visualiser I linked earlier). That way, everybody wins - folks that have never heard of schema.org or linked data in general won't need to learn any concepts that are completely irrelevant to them, while folks that are aware of those things and the related tools will be free to use them without first having to figure out their own mapping from the Python specific metadata formats to a JSON-LD compatible format. That approach then doesn't even need to wait for PEP 426: it could be done using the wheel METADATA file as a basis instead. It will probably still be up to Wes to actually define that transformation though - I don't think anybody else is anywhere near keen enough to make use of the available JSON-LD tooling to spend any time working on enabling it :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Mon Aug 21 11:46:52 2017 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 21 Aug 2017 10:46:52 -0500 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: On Monday, August 21, 2017, Nick Coghlan wrote: > On 21 August 2017 at 19:38, Paul Moore > > wrote: > > On 21 August 2017 at 09:54, Nick Coghlan > wrote: > >> While I'm still generally negative on the idea of native reliance on > >> JSON-LD, I'll note one thing that has changed since I last looked at > >> it: I now see some potential concrete practical benefits to adopting > >> it, rather than purely theoretical ones. In particular, > >> https://github.com/scienceai/jsonld-vis now exists, and there wasn't > >> anything like that around at the time of previous discussions. > > > > Personally, I fairly often write adhoc scripts that use the JSON API, > > and as it stands it's very convenient for that. From what I can see of > > JSON-LD (which basically equates to "it adds some extra metadata keys > > that don't change the data content but do change the list of keys and > > maybe the nesting levels") it would be somewhat inconvenient for my > > scripts, and add no extra capability that I would ever use. > > Right, and this is still my main concern with the idea as well: I'd > never be OK with a JSON-LD-only API, because it adds too much > irrelevant cognitive overhead for the vast majority of Python > packaging specific use cases. (I would see it as being akin to Python > itself deciding to require type annotations, rather than merely > allowing them). > > However, where I'm starting to see a potential niche for it is as an > opt-in capability, whereby we explicitly define how our metadata can > be translated *to* JSON-LD, for folks that want to apply general > purpose tools that know how to manipulate arbitrary JSON-LD data (like > the graph visualiser I linked earlier). > > That way, everybody wins - folks that have never heard of schema.org > or linked data in general won't need to learn any concepts that are > completely irrelevant to them, while folks that are aware of those > things and the related tools will be free to use them without first > having to figure out their own mapping from the Python specific > metadata formats to a JSON-LD compatible format. > > That approach then doesn't even need to wait for PEP 426: it could be > done using the wheel METADATA file as a basis instead. > > It will probably still be up to Wes to actually define that > transformation though - I don't think anybody else is anywhere near > keen enough to make use of the available JSON-LD tooling to spend any > time working on enabling it :) So, ## Justify JSONLD - This is a graph. If we use an existing spec for graphs as JSON (ie JSONLD), we win: - all of the tools that already exist for working with said graphs in that format - easy indexability (as RDF quads) - compatibility with compatible specs like ld-signatures ## Implement JSONLD - [ ] decide which URI(s) a project on {pypi,} is identified by - some projects should not have an implicit pypi.org URI prefix - [ ] write a new JSONLD view for pypi and warehouse - [ ] write a JSONLD metadata spec for eggs and wheels ## Support metadata retrieval without exec'ing setup.py - develop a declarative format for expressing {sys.platform[...],}-dependent dependency edges Signed, Wes T. P.P.S. This is just a hard week for me, -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Aug 21 17:05:38 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 21 Aug 2017 16:05:38 -0500 Subject: [Distutils] PEP 517: Bootstrapping setuptools Message-ID: Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip. The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Aug 21 17:17:02 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 21 Aug 2017 16:17:02 -0500 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: Of course, to be frank, the principle failure with this plan is that third-party tools (buildout, os packagers) will not be compliant with PEP 517 even after it is adopted, and will then complain about having to update their build systems. 2017-08-21 16:05 GMT-05:00 xoviat : > Previously, the attempt to move setuptools off of vendored dependencies > failed because it was not done correctly: install_requires was set to the > vendored dependencies but not setup_requires, which would have been > required to correctly specify dependencies. However, setup_requires would > have introduced circular dependencies that are impossible to correctly > resolve, so that would have simply shifted an impossible problem onto pip. > > The principle issue then, is that setuptools requires setuptools to build > itself. And although, this problem was previously not worth solving because > of 'setup.py install', it is now not a difficult problem to solve if we > rethink slightly what should be included in the setuptools respository: > - Rather than re-generating egg_info on each setup.py invocation, we can > cache dist-info in the setuptools respository. > - We can implement a simple entry_points script that updates dist-info > based on the contents of setup.py, which is only runnable of course after > setuptools is installed > - We can implement the reference build backend for setuptools: simply copy > the files and dist-info into a new compliant zip archive > > Viola! Setuptools is no longer required to build setuptools, and the > installation process is fully compliant with Python PEPs. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at jimfulton.info Mon Aug 21 17:27:42 2017 From: jim at jimfulton.info (Jim Fulton) Date: Mon, 21 Aug 2017 17:27:42 -0400 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: On Mon, Aug 21, 2017 at 5:17 PM, xoviat wrote: > Of course, to be frank, the principle failure with this plan is that > third-party tools (buildout, os packagers) will not be compliant with PEP > 517 even after it is adopted, and will then complain about having to update > their build systems. > I don't understand this statement. If buildout needs to be compliant, it will be (eventually). I can imagine a future where everything is distributed with wheels and buildout wouldn't need to build anything. Jim > > 2017-08-21 16:05 GMT-05:00 xoviat : > >> Previously, the attempt to move setuptools off of vendored dependencies >> failed because it was not done correctly: install_requires was set to the >> vendored dependencies but not setup_requires, which would have been >> required to correctly specify dependencies. However, setup_requires would >> have introduced circular dependencies that are impossible to correctly >> resolve, so that would have simply shifted an impossible problem onto pip. >> >> The principle issue then, is that setuptools requires setuptools to build >> itself. And although, this problem was previously not worth solving because >> of 'setup.py install', it is now not a difficult problem to solve if we >> rethink slightly what should be included in the setuptools respository: >> - Rather than re-generating egg_info on each setup.py invocation, we can >> cache dist-info in the setuptools respository. >> - We can implement a simple entry_points script that updates dist-info >> based on the contents of setup.py, which is only runnable of course after >> setuptools is installed >> - We can implement the reference build backend for setuptools: simply >> copy the files and dist-info into a new compliant zip archive >> >> Viola! Setuptools is no longer required to build setuptools, and the >> installation process is fully compliant with Python PEPs. >> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- Jim Fulton http://jimfulton.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Aug 21 17:39:20 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 21 Aug 2017 16:39:20 -0500 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: This statement comes from something that Donald said: > The unvendoring means that setuptools and the project code are now competing over who gets to define what an acceptable version is for these libraries to be installed with. As if this isn't going to be true for any other build system, and setuptools won't at all be in any way special in a PEP 517 world. So you are always going to have complaints about "well build system X required a newer version but my project requires an older version" when the real problem is that people are requiring an older version/ projects do not have a stable API. 2017-08-21 16:27 GMT-05:00 Jim Fulton : > > > On Mon, Aug 21, 2017 at 5:17 PM, xoviat wrote: > >> Of course, to be frank, the principle failure with this plan is that >> third-party tools (buildout, os packagers) will not be compliant with PEP >> 517 even after it is adopted, and will then complain about having to update >> their build systems. >> > > I don't understand this statement. If buildout needs to be compliant, it > will be (eventually). I can imagine a future where everything is > distributed with wheels and buildout wouldn't need to build anything. > > Jim > > >> >> 2017-08-21 16:05 GMT-05:00 xoviat : >> >>> Previously, the attempt to move setuptools off of vendored dependencies >>> failed because it was not done correctly: install_requires was set to the >>> vendored dependencies but not setup_requires, which would have been >>> required to correctly specify dependencies. However, setup_requires would >>> have introduced circular dependencies that are impossible to correctly >>> resolve, so that would have simply shifted an impossible problem onto pip. >>> >>> The principle issue then, is that setuptools requires setuptools to >>> build itself. And although, this problem was previously not worth solving >>> because of 'setup.py install', it is now not a difficult problem to solve >>> if we rethink slightly what should be included in the setuptools >>> respository: >>> - Rather than re-generating egg_info on each setup.py invocation, we can >>> cache dist-info in the setuptools respository. >>> - We can implement a simple entry_points script that updates dist-info >>> based on the contents of setup.py, which is only runnable of course after >>> setuptools is installed >>> - We can implement the reference build backend for setuptools: simply >>> copy the files and dist-info into a new compliant zip archive >>> >>> Viola! Setuptools is no longer required to build setuptools, and the >>> installation process is fully compliant with Python PEPs. >>> >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > > > -- > Jim Fulton > http://jimfulton.info > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Aug 22 00:44:14 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 22 Aug 2017 14:44:14 +1000 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: On 22 August 2017 at 01:46, Wes Turner wrote: > ## Justify JSONLD > - This is a graph. If we use an existing spec for graphs as JSON (ie > JSONLD), we win: > - all of the tools that already exist for working with said graphs in that > format > - easy indexability (as RDF quads) > - compatibility with compatible specs like ld-signatures No, that's the argument I've already said doesn't work, since it doesn't address the readability problem Paul mentioned: yes, JSON-LD *can* represent aribtrary graphs, but to *read* a JSON-LD data structure as a human, you need to know how *JSON-LD* represents graphs. I don't consider that an acceptable limitation: the raw metadata needs to be readable by someone that's *only* familiar with the specifics of dependency management in Python, and couldn't care less about the representation of graphs as a general concept. > ## Implement JSONLD > - [ ] decide which URI(s) a project on {pypi,} is identified by > - some projects should not have an implicit pypi.org URI prefix > - [ ] write a new JSONLD view for pypi and warehouse > - [ ] write a JSONLD metadata spec for eggs and wheels None of which are dependent on JSON-LD being the raw format for the metadata - this can instead be done as a postprocessing step that accepts any of the existing metadata formats as input. Defining such a transformation is going to be critical for your goals of making the JSON-LD representation useful anyway, as even if we defined a new metadata format tomorrow, that would still mean there were more than 700 thousand existing releases on pypi.org that didn't natively provide their metadata in that format. The added bonus of doing things that way is that it means that you *don't* need anyone else's agreement or consensus to start design work - you can do an initial proof of concept using a domain you control, similar to the way Donald started out by building the PyPI that he wished existed as an independent service before we thanked him for his efforts by lumbering him with the spectacularly difficult task of figuring out how to upgrade or replace pypi.python.org itself :) > ## Support metadata retrieval without exec'ing setup.py > > - develop a declarative format for expressing {sys.platform[...],}-dependent > dependency edges This is already part of PEP 508: https://www.python.org/dev/peps/pep-0508/#environment-markers This is why given a wheel file, you can *already* extract declarative dependency metadata, using the METADATA file + PEP 508. Given just an sdist, you can also do something similar by looking at PKG-INFO, but that's less reliable (since that file may not even be present, and even if it is, the sdist -> wheel build step may still inject additional dependencies). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jim at jimfulton.info Tue Aug 22 09:13:29 2017 From: jim at jimfulton.info (Jim Fulton) Date: Tue, 22 Aug 2017 09:13:29 -0400 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: On Mon, Aug 21, 2017 at 5:39 PM, xoviat wrote: > This statement comes from something that Donald said: > > > The unvendoring means that setuptools and the project code are now > competing over who gets to define what an acceptable version is for these > libraries to be installed with. > > As if this isn't going to be true for any other build system, and > setuptools won't at all be in any way special in a PEP 517 world. So you > are always going to have complaints about "well build system X required a > newer version but my project requires an older version" when the real > problem is that people are requiring an older version/ projects do not have > a stable API. > This boils down to implementation details. pip dealt with this issue and buildout stumbled because of differences in the way they managed their paths. Buildout could have coped (eventually). Jim > 2017-08-21 16:27 GMT-05:00 Jim Fulton : > >> >> >> On Mon, Aug 21, 2017 at 5:17 PM, xoviat wrote: >> >>> Of course, to be frank, the principle failure with this plan is that >>> third-party tools (buildout, os packagers) will not be compliant with PEP >>> 517 even after it is adopted, and will then complain about having to update >>> their build systems. >>> >> >> I don't understand this statement. If buildout needs to be compliant, it >> will be (eventually). I can imagine a future where everything is >> distributed with wheels and buildout wouldn't need to build anything. >> >> Jim >> >> >>> >>> 2017-08-21 16:05 GMT-05:00 xoviat : >>> >>>> Previously, the attempt to move setuptools off of vendored dependencies >>>> failed because it was not done correctly: install_requires was set to the >>>> vendored dependencies but not setup_requires, which would have been >>>> required to correctly specify dependencies. However, setup_requires would >>>> have introduced circular dependencies that are impossible to correctly >>>> resolve, so that would have simply shifted an impossible problem onto pip. >>>> >>>> The principle issue then, is that setuptools requires setuptools to >>>> build itself. And although, this problem was previously not worth solving >>>> because of 'setup.py install', it is now not a difficult problem to solve >>>> if we rethink slightly what should be included in the setuptools >>>> respository: >>>> - Rather than re-generating egg_info on each setup.py invocation, we >>>> can cache dist-info in the setuptools respository. >>>> - We can implement a simple entry_points script that updates dist-info >>>> based on the contents of setup.py, which is only runnable of course after >>>> setuptools is installed >>>> - We can implement the reference build backend for setuptools: simply >>>> copy the files and dist-info into a new compliant zip archive >>>> >>>> Viola! Setuptools is no longer required to build setuptools, and the >>>> installation process is fully compliant with Python PEPs. >>>> >>> >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >> >> >> -- >> Jim Fulton >> http://jimfulton.info >> > > -- Jim Fulton http://jimfulton.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Aug 22 09:23:42 2017 From: dholth at gmail.com (Daniel Holth) Date: Tue, 22 Aug 2017 13:23:42 +0000 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: Isn't this a special case of needing . on sys.path when building a package? Then the same version of setuptools that is being packaged builds itself. I've experimented with a different kind of bootstrapping where pip installs setuptools on demand when the first setuptools requiring package is built. You kindof need pkg-resources as a separate package for this. It is a cool parlor trick but it's simpler and harmless to just always install setuptools like we do. On Tue, Aug 22, 2017, 09:14 Jim Fulton wrote: > On Mon, Aug 21, 2017 at 5:39 PM, xoviat wrote: > >> This statement comes from something that Donald said: >> >> > The unvendoring means that setuptools and the project code are now >> competing over who gets to define what an acceptable version is for these >> libraries to be installed with. >> >> As if this isn't going to be true for any other build system, and >> setuptools won't at all be in any way special in a PEP 517 world. So you >> are always going to have complaints about "well build system X required a >> newer version but my project requires an older version" when the real >> problem is that people are requiring an older version/ projects do not have >> a stable API. >> > > This boils down to implementation details. pip dealt with this > issue and buildout stumbled because of differences in the way they managed > their paths. Buildout could have coped (eventually). > > Jim > > >> 2017-08-21 16:27 GMT-05:00 Jim Fulton : >> >>> >>> >>> On Mon, Aug 21, 2017 at 5:17 PM, xoviat wrote: >>> >>>> Of course, to be frank, the principle failure with this plan is that >>>> third-party tools (buildout, os packagers) will not be compliant with PEP >>>> 517 even after it is adopted, and will then complain about having to update >>>> their build systems. >>>> >>> >>> I don't understand this statement. If buildout needs to be compliant, it >>> will be (eventually). I can imagine a future where everything is >>> distributed with wheels and buildout wouldn't need to build anything. >>> >>> Jim >>> >>> >>>> >>>> 2017-08-21 16:05 GMT-05:00 xoviat : >>>> >>>>> Previously, the attempt to move setuptools off of vendored >>>>> dependencies failed because it was not done correctly: install_requires was >>>>> set to the vendored dependencies but not setup_requires, which would have >>>>> been required to correctly specify dependencies. However, setup_requires >>>>> would have introduced circular dependencies that are impossible to >>>>> correctly resolve, so that would have simply shifted an impossible problem >>>>> onto pip. >>>>> >>>>> The principle issue then, is that setuptools requires setuptools to >>>>> build itself. And although, this problem was previously not worth solving >>>>> because of 'setup.py install', it is now not a difficult problem to solve >>>>> if we rethink slightly what should be included in the setuptools >>>>> respository: >>>>> - Rather than re-generating egg_info on each setup.py invocation, we >>>>> can cache dist-info in the setuptools respository. >>>>> - We can implement a simple entry_points script that updates dist-info >>>>> based on the contents of setup.py, which is only runnable of course after >>>>> setuptools is installed >>>>> - We can implement the reference build backend for setuptools: simply >>>>> copy the files and dist-info into a new compliant zip archive >>>>> >>>>> Viola! Setuptools is no longer required to build setuptools, and the >>>>> installation process is fully compliant with Python PEPs. >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>> >>>> >>> >>> >>> -- >>> Jim Fulton >>> http://jimfulton.info >>> >> >> > > > -- > Jim Fulton > http://jimfulton.info > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Tue Aug 22 10:57:02 2017 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Tue, 22 Aug 2017 11:57:02 -0300 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: Hi, On Mon, Aug 21, 2017 at 5:39 PM, xoviat wrote: > This statement comes from something that Donald said: >> >> > The unvendoring means that setuptools and the project code are now >> competing over who gets to define what an acceptable version is for these >> libraries to be installed with. >> >> As if this isn't going to be true for any other build system, and >> setuptools won't at all be in any way special in a PEP 517 world. So you >> are always going to have complaints about "well build system X required a >> newer version but my project requires an older version" when the real >> problem is that people are requiring an older version/ projects do not have >> a stable API. >> > It seems to me that this is one of those things where the solution is staring us in the face. If PEP 517 simply clarifies that, since the build steps should be run with either the original checkout or a cleaned-up sdist as the current directory, this directory will be naturally the first on `sys.path` (and shouldn't be removed), so build backends can rely on its presence there. In `setuptools` case, if it wants to depend at build time (i.e. `setup_requires`) on other packages that depend on `setuptools', it can't specify them as `[build-system]requires`, but it's free to specify them as `get_requires_for_build*()`, since `setuptools` itself will be on `sys.path`. `get_requires_for_build*()` just needs to arrange for the source distribution directory to contain enough of whatever metadata format is needed to be recognized as an installed dist to be depended upon while being on sys.path. As mentioned, the real trouble begins when `setuptools` wants to depend at runtime (`install_requires`) on other packages since incompatible versions of these packages could spell trouble to packages that depend on setuptools AND on these incompatible versions. If setuptools doesn't want to "hard vendor" these dependencies (i.e. keep a copy of them in its own repository), one possible solution is for `setuptools` to vendor these dependencies at build time: - make them build dependencies as above - locate where they were installed by their `.dist-info/RECORD`, or failing that, their `.__file__` module attribute. - copy them over to it's own `setuptools._vendor` package, patching them if necessary. An enviroment (e.g. upstream distributor) wanting to bootstrap everything from source would then need to make sure setuptools and all its dependencies are installed in the same python environment in "development mode" (/me handwaves the fact that this operation is not strictly defined). A frontend could be written to do just that. Regards, Leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at jimfulton.info Tue Aug 22 11:02:45 2017 From: jim at jimfulton.info (Jim Fulton) Date: Tue, 22 Aug 2017 11:02:45 -0400 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth wrote: > Isn't this a special case of needing . on sys.path when building a > package? Then the same version of setuptools that is being packaged builds > itself. > No. The issue for us it wasn't setuptools itself, but it's dependencies and those dependencies conflicted with dependencies of of packages we were installing *and* those packages importing these dependences (indirectly) in their setup scripts. Setup scripts that import the thing they're about to install, generally to get the version :(, is something I'd love to see go away. Jim -- Jim Fulton http://jimfulton.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at nycresistor.com Tue Aug 22 11:15:04 2017 From: matt at nycresistor.com (Matt Joyce) Date: Tue, 22 Aug 2017 11:15:04 -0400 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: venvs within venvs... terrifying concept. On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton wrote: > > > On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth wrote: > >> Isn't this a special case of needing . on sys.path when building a >> package? Then the same version of setuptools that is being packaged builds >> itself. >> > No. The issue for us it wasn't setuptools itself, but it's dependencies > and those dependencies conflicted with dependencies of of packages we were > installing *and* those packages importing these dependences (indirectly) in > their setup scripts. Setup scripts that import the thing they're about to > install, generally to get the version :(, is something I'd love to see go > away. > > Jim > > -- > Jim Fulton > http://jimfulton.info > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at jimfulton.info Tue Aug 22 11:23:15 2017 From: jim at jimfulton.info (Jim Fulton) Date: Tue, 22 Aug 2017 11:23:15 -0400 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: I didn't mention (nor do I recall anyone mentioning) venvs. Jim On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce wrote: > venvs within venvs... terrifying concept. > > On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton wrote: > >> >> >> On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth wrote: >> >>> Isn't this a special case of needing . on sys.path when building a >>> package? Then the same version of setuptools that is being packaged builds >>> itself. >>> >> No. The issue for us it wasn't setuptools itself, but it's dependencies >> and those dependencies conflicted with dependencies of of packages we were >> installing *and* those packages importing these dependences (indirectly) in >> their setup scripts. Setup scripts that import the thing they're about to >> install, generally to get the version :(, is something I'd love to see go >> away. >> >> Jim >> >> -- >> Jim Fulton >> http://jimfulton.info >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -- Jim Fulton http://jimfulton.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Tue Aug 22 14:06:27 2017 From: xoviat at gmail.com (xoviat) Date: Tue, 22 Aug 2017 13:06:27 -0500 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: So are we going for a "primarily isolation" approach then where builds are only isolated on the first level, but subsequent levels share the same build environment? 2017-08-22 10:23 GMT-05:00 Jim Fulton : > I didn't mention (nor do I recall anyone mentioning) venvs. > > Jim > > > On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce wrote: > >> venvs within venvs... terrifying concept. >> >> On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton wrote: >> >>> >>> >>> On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth wrote: >>> >>>> Isn't this a special case of needing . on sys.path when building a >>>> package? Then the same version of setuptools that is being packaged builds >>>> itself. >>>> >>> No. The issue for us it wasn't setuptools itself, but it's dependencies >>> and those dependencies conflicted with dependencies of of packages we were >>> installing *and* those packages importing these dependences (indirectly) in >>> their setup scripts. Setup scripts that import the thing they're about to >>> install, generally to get the version :(, is something I'd love to see go >>> away. >>> >>> Jim >>> >>> -- >>> Jim Fulton >>> http://jimfulton.info >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >> > > > -- > Jim Fulton > http://jimfulton.info > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at jimfulton.info Tue Aug 22 14:50:56 2017 From: jim at jimfulton.info (Jim Fulton) Date: Tue, 22 Aug 2017 14:50:56 -0400 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: Oh, gawd. From now on, someone will have to say buildout 3 times before I appear. On Tue, Aug 22, 2017 at 2:06 PM, xoviat wrote: > So are we going for a "primarily isolation" approach then where builds are > only isolated on the first level, but subsequent levels share the same > build environment? > I'm a huge fan of isolation, but isolation seems to me to be orthogonal to this discussion. I'm done with this thread. :) Jim > > 2017-08-22 10:23 GMT-05:00 Jim Fulton : > >> I didn't mention (nor do I recall anyone mentioning) venvs. >> >> Jim >> >> >> On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce >> wrote: >> >>> venvs within venvs... terrifying concept. >>> >>> On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton wrote: >>> >>>> >>>> >>>> On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth wrote: >>>> >>>>> Isn't this a special case of needing . on sys.path when building a >>>>> package? Then the same version of setuptools that is being packaged builds >>>>> itself. >>>>> >>>> No. The issue for us it wasn't setuptools itself, but it's dependencies >>>> and those dependencies conflicted with dependencies of of packages we were >>>> installing *and* those packages importing these dependences (indirectly) in >>>> their setup scripts. Setup scripts that import the thing they're about to >>>> install, generally to get the version :(, is something I'd love to see go >>>> away. >>>> >>>> Jim >>>> >>>> -- >>>> Jim Fulton >>>> http://jimfulton.info >>>> >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>> >>>> >>> >> >> >> -- >> Jim Fulton >> http://jimfulton.info >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -- Jim Fulton http://jimfulton.info -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Tue Aug 22 15:24:34 2017 From: xoviat at gmail.com (xoviat) Date: Tue, 22 Aug 2017 14:24:34 -0500 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: The OP was about "bootstrapping setuptools." If, as Leonardo said, we only isolate on the first level (requirements from install_requires or user will be isolated but requirements from build_requires will share the same environment as the invoker of build_requires if they are not wheels), then that opens up some interesting possibilities. However, I don't necessarily think that this is a good idea because it opens up situations that could be nasty to debug. I still think the simpler method would be as I have said previously, to have a very minimal backend that doesn't depend on anything and simply archives the files and info. We can even put this backend into a separate package that requires and depends on nothing, so that future build systems don't have to re-invent the wheel here. The question that I have is: how does flit address this issue? Does flit require external dependencies to bootstrap itself? 2017-08-22 13:50 GMT-05:00 Jim Fulton : > Oh, gawd. From now on, someone will have to say buildout 3 times before I > appear. > > On Tue, Aug 22, 2017 at 2:06 PM, xoviat wrote: > >> So are we going for a "primarily isolation" approach then where builds >> are only isolated on the first level, but subsequent levels share the same >> build environment? >> > > I'm a huge fan of isolation, but isolation seems to me to be orthogonal to > this discussion. > > I'm done with this thread. :) > > Jim > > >> >> 2017-08-22 10:23 GMT-05:00 Jim Fulton : >> >>> I didn't mention (nor do I recall anyone mentioning) venvs. >>> >>> Jim >>> >>> >>> On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce >>> wrote: >>> >>>> venvs within venvs... terrifying concept. >>>> >>>> On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton >>>> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth >>>>> wrote: >>>>> >>>>>> Isn't this a special case of needing . on sys.path when building a >>>>>> package? Then the same version of setuptools that is being packaged builds >>>>>> itself. >>>>>> >>>>> No. The issue for us it wasn't setuptools itself, but it's >>>>> dependencies and those dependencies conflicted with dependencies of of >>>>> packages we were installing *and* those packages importing these >>>>> dependences (indirectly) in their setup scripts. Setup scripts that import >>>>> the thing they're about to install, generally to get the version :(, is >>>>> something I'd love to see go away. >>>>> >>>>> Jim >>>>> >>>>> -- >>>>> Jim Fulton >>>>> http://jimfulton.info >>>>> >>>>> _______________________________________________ >>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>>> >>>>> >>>> >>> >>> >>> -- >>> Jim Fulton >>> http://jimfulton.info >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >> > > > -- > Jim Fulton > http://jimfulton.info > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Aug 23 00:37:22 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 Aug 2017 14:37:22 +1000 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: On 23 August 2017 at 05:24, xoviat wrote: > The question that I have is: how does flit address this issue? Does flit > require external dependencies to bootstrap itself? It does, but those dependencies currently use setuptools as their packaging system rather than flit. Bootstrapping self-hosted packaging tools is a genuinely hard problem - it's why pip vendors most of its dependencies, and relies on preinstalled SSL/TLS support in the standard library. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Wed Aug 23 05:54:39 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Wed, 23 Aug 2017 10:54:39 +0100 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: <1503482079.757332.1082262952.6FF34943@webmail.messagingengine.com> On Wed, Aug 23, 2017, at 05:37 AM, Nick Coghlan wrote: > On 23 August 2017 at 05:24, xoviat wrote: > > The question that I have is: how does flit address this issue? Does flit > > require external dependencies to bootstrap itself? > > It does, but those dependencies currently use setuptools as their > packaging system rather than flit. Yup. I actually moved a package which I maintain (zipfile36) back to setuptools packaging specifically so that flit could use it without a dependency loop. The other dependencies for bootstrapping are requests and docutils. There's one other dependency listed, but that can be ignored for bootstrapping. Thomas From leorochael at gmail.com Wed Aug 23 11:31:07 2017 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Wed, 23 Aug 2017 12:31:07 -0300 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: On 23 August 2017 at 01:37, Nick Coghlan wrote: > [...] > > Bootstrapping self-hosted packaging tools is a genuinely hard problem > True, but on the other hand, it's the self-hosted packaging tools' own problem. It's up to the tools themselves to solve it. if PEP 517 simply establishes that the current directory of the source distribution will be naturally at the head of `sys.path` (whether an original checkout or a cleaned up sdist), then self-hosted packaging tools can be their own backend, and do whatever magick they need to self-bootstrap. It makes sense to think a bit about whether we're not making these cases harder, but we shouldn't block progress of PEP 517 on account of these cases. If we specifically want to make these cases easier in the future, we might think of another PEP to expose to backends certain frontend operations like "provide me the sdist (or wheel) for requirement foo >= 2.3.0 but don't install it and don't fetch dependencies" so we can make sure certain configurations like "what is the index server?" or "how do I use a web proxy?" are centralized and consistent. Regards, Leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Aug 23 14:59:59 2017 From: donald at stufft.io (Donald Stufft) Date: Wed, 23 Aug 2017 14:59:59 -0400 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: Message-ID: <8C07FE46-459A-49CF-B7F5-515041E5C4B3@stufft.io> > On Aug 20, 2017, at 7:09 AM, Paul Moore wrote: > > On 19 August 2017 at 23:03, Wes Turner wrote: >> Here are some notes re: changing metadata: > > I thought the proposal was to document the current state of affairs. > Proposing a *change* to the JSON API would be a much bigger and more > controversial proposal (and one I see little need for, personally). > I can answer this a bit more forcefully? At this point in time Warehouse is not interested in any drastic changes to it?s API until after legacy PyPI is gone. Documenting the existing APIs and small tweaks that may make it easier/better to use are fine, but any wholesale new API or large scale refactors are not going to be happening in the short term. I don?t have a major opinion on a PEP for the JSON api or not. It depends I guess on whether tools like bandersnatch/devpi/etc want to offer it. Given that this is all brought on by a PR to bandersnatch it appears that there is a reasonable argument that it is something that those tools want, and standardizing it is a good idea. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From doko at debian.org Tue Aug 22 10:10:32 2017 From: doko at debian.org (Matthias Klose) Date: Tue, 22 Aug 2017 16:10:32 +0200 Subject: [Distutils] PEP 517: Bootstrapping setuptools In-Reply-To: References: Message-ID: On 22.08.2017 15:23, Daniel Holth wrote: > You kindof need pkg-resources as a separate package for this. It is a cool > parlor trick but it's simpler and harmless to just always install > setuptools like we do. when will we have a split out pkg-resources? From iahartle at ucsd.edu Sat Aug 19 15:27:42 2017 From: iahartle at ucsd.edu (Ian Hartley) Date: Sat, 19 Aug 2017 12:27:42 -0700 Subject: [Distutils] Transfering modules In-Reply-To: References: Message-ID: Hi all, I'm using PyCharm CE as my development environment, on a macintosh running mac OS 10.12.3. Here's a screenshot of what happens when I run python3 -m site. [image: Inline image 1] As an explanation - here's the exact processes that I'd run to achieve the trouble (and how I fixed it). After installing the current version with a pip3 install in terminal, I went to the python website, and downloaded the 3.6.2 package, and ran the installer. After realizing that my 3.6.1 modules weren't importable by the 3.6.2 version, I attempted to uninstall 3.6.2 using this method: https://stackoverflow.com/questions/3819449/how-to-uninstall-python-2-7-on-a-mac-os-x-10-6-4/3819829#3819829 This removed 3.6.2 files but somehow was still running 3.6.2 when I booted 'python3' from terminal. It was at this point when I reached out to you guys. I then heeded your advice. I reinstalled 3.6.2, and migrated the modules from the 'site-packages' of 3.6.1 to 'site-packages' in 3.6.2. Through this process I found that 3.6.1 had somehow been installed in a different disc location than the entire 3.6 build, which contained 3.6.2. After migrating the packages to the real 3.6 build containing 3.6.2, I left the remaining 3.6.1 location on my disc. Here's what I still see when trying to configure the PyCharm CE interpreter. [image: Inline image 2] Everything seems to be working now though. Thanks again for your help guys - let me know if you have any additional questions and I'll do my best to furnish an answer. I'm studying applications of tensorflow in financial prediction models at the moment. If any of you have read any academic papers on the topic that stand out, I'd be delighted if you might send me a link, or a way I could find it! Additionally, if you happen to know another Python community member who's researching the same topic, it would be great to get in touch. Best regards, Ian On Fri, Aug 18, 2017 at 10:57 PM, Nick Coghlan wrote: > On 19 August 2017 at 14:03, Chris Jerdonek > wrote: > > I would just like to point out that the normal workflow for pyenv [1], > > which seems very popular (and which I use), is to do parallel installs > > rather than upgrades. The OP didn't actually say "upgrade," so it's > > possible they're using a tool like pyenv. > > Right, it was the references to sudo-remove, rebooting, and python3 > that indicated to me we were talking system wide operations, rather > than any of the environment management tools like pew, pyenv, conda, > etc. That's why I'm somewhat puzzled, since even a raw "make > altinstall" will overwrite an existing installation of that version. > > The most likely candidate for a system level kinda-sorta parallel > installation would be via the distinction between /usr/local/* and > /usr/*, and if it's a Debian or Ubuntu system, that would result in > the actual system version being able to see components in > dist-packages that would be invisible to an unpatched upstream build, > each version having its own site-packages directory, but both of them > still sharing a common user site-packages directory. > > Hence my interest in the output of " -m site" for the > affected versions: seeing the default contents of sys.path will likely > answer the question of what's going on definitively, rather than > having to guess based on incomplete information. > > I believe the OP's original > > questions would also apply to users of pyenv -- i.e. having a simple > > way to "copy" installed dependencies from one pyenv install to a new > > one. > > As Wayne noted, generating a requirements.txt file with `pip freeze` > and then reinstalling the dependencies essentially *is* that > mechanism. If you still have the original wheelhouse cache around, it > shouldn't be substantially slower than a plain filesystem copy (since > it will just be a matter of unpacking the archived wheel files again, > rather than doing any binary builds). > > For a more literal copy operation, there's "pew cp", but the > documentation for that has "this may result in a subtly broken virtual > environment" caveats on it for good reasons :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2017-08-19 at 12.12.14 PM.png Type: image/png Size: 145996 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2017-08-19 at 12.21.51 PM.png Type: image/png Size: 132912 bytes Desc: not available URL: From iahartle at ucsd.edu Sun Aug 20 12:16:30 2017 From: iahartle at ucsd.edu (Ian Hartley) Date: Sun, 20 Aug 2017 09:16:30 -0700 Subject: [Distutils] Transfering modules In-Reply-To: <1E085E4B-60DF-46E8-B19A-BFEA551FA13A@mac.com> References: <1E085E4B-60DF-46E8-B19A-BFEA551FA13A@mac.com> Message-ID: Ronald, I was sure that was the case re: python.org installation. It's strange that it installed in two different locations, I really have no idea how that might have come about. FYI I've included a pic of the file pathways when you look at the interpreter photo. I'd first installed python using terminal btw. Ian > On Aug 20, 2017, at 8:55 AM, Ronald Oussoren wrote: > > >> On 20 Aug 2017, at 05:56, Nick Coghlan wrote: >> >>> On 20 August 2017 at 05:27, Ian Hartley wrote: >>> Hi all, >>> >>> I'm using PyCharm CE as my development environment, on a macintosh running mac OS 10.12.3. >> >> Ah, that would explain why my intuitions were all wrong - I know how the Windows and *nix installation models work (and I believe the latter also covers homebrew on macOS), but not how the macOS framework installers work :) > > FWIW: The framework installers on python.org work just like the other installers on upgrades: installing 3.6.2 on a system already running 3.6.1 replaces the 3.6.1 installation and doesn?t perform a side-by-side installation. The major differences w.r.t. a *nix installation are having an ?odd? sys.prefix, and having different sys.prefix values for different feature releases. > > Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From samuel.carlsson at gmail.com Mon Aug 21 04:11:27 2017 From: samuel.carlsson at gmail.com (Samuel Carlsson) Date: Mon, 21 Aug 2017 10:11:27 +0200 Subject: [Distutils] Claiming packages as per PEP 541? Message-ID: I've seem people claiming packages on the Support Requests page on source forge, I've also done so myself. However, most of these request seems to go unnoticed. The PEP 541 does not seem to specify any way actually performing claims to transfer ownership of packages. In short: What is the procedure to claim a project on PyPI? /samuel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Aug 23 22:04:51 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 24 Aug 2017 12:04:51 +1000 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: <8C07FE46-459A-49CF-B7F5-515041E5C4B3@stufft.io> References: <8C07FE46-459A-49CF-B7F5-515041E5C4B3@stufft.io> Message-ID: On 24 August 2017 at 04:59, Donald Stufft wrote: > I don?t have a major opinion on a PEP for the JSON api or not. It depends I > guess on whether tools like bandersnatch/devpi/etc want to offer it. Given > that this is all brought on by a PR to bandersnatch it appears that there is > a reasonable argument that it is something that those tools want, and > standardizing it is a good idea. +1, especially as it will help clarify the required test cases for Warehouse as well (I'm not sure how much of the JSON API has been implemented at this point). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Aug 24 01:58:15 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 24 Aug 2017 15:58:15 +1000 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: <9569C737-9C63-4F1C-ABFF-185C85CCD5E5@cooperlees.com> References: <8C07FE46-459A-49CF-B7F5-515041E5C4B3@stufft.io> <9569C737-9C63-4F1C-ABFF-185C85CCD5E5@cooperlees.com> Message-ID: On 24 August 2017 at 13:59, Cooper Ry Lees wrote: > Thanks all for your points. > > Is it fine if I get a Informational PEP going to discuss 'Metadata Repository API'? I will structure it similar to PEP503, but also talk about: > > - The data exposed today (Specification) +1 > -- And possibly call on some PyPI people to correct me where I guess wrong Yep, that's part of the PEP review process - to indicate this, use the same BDFL-Delegate and Discussions-To values as are in PEP 503. > - How mirrors should mirror it In particular, it would be ideal if they could generate it themselves from the already mirrored PEP 503 data, rather than having to make additional API calls to the main PyPI server. > - Possible furure enhancements (JSONLD etc.) Discussion of future enhancements doesn't really belong in an Informational PEP. Instead, I've filed https://github.com/pypa/packaging-problems/issues/102 as a common place for folks to put notes about what they may want to see in a possible future API. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Thu Aug 24 09:11:01 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 24 Aug 2017 14:11:01 +0100 Subject: [Distutils] PEP 517 again Message-ID: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Nathaniel seems to be busy with other things at the moment, so I hope he won't mind me passing on this list of things he'd like to resolve with the draft PEP. I'll quote his comments and put my responses inline. > - needs some sort of NotImplemented(Error) handling specified We've gone round return-value vs exception a few times, but I don't think that debate is particularly heated, so it probably just needs someone to say "this is what we're doing". I can be that someone if needs be. But does anyone feel strongly about it? > - The out-of-tree build example (that makes an sdist and then builds > it) seems like exactly the kind of implementation that Donald says he > doesn't want to see? OTOH I think Nick said he wants to see a more > elaborated specification of out-of-tree build strategies with this > specifically as one of them. > - Nick has suggested that the to-be-defined NotImplemented(Error) > handling can be used by build_wheel to indicate that it can't do > out-of-tree builds, so maybe the frontend should try an in-tree build > instead. I don't know how to convert this into a real spec, though -- > like in general, if I'm a frontend and I call `hook(*args, **kwargs)` > and it says "can't do that", then how do I know what the problem is > and what I should try instead? > - What happens if prepare_build_metadata returns NotImplemented / > raises NotImplementedError? > - I don't understand how out-of-tree builds and prepare_build_metadata > are supposed to interact. > - Also, AFAICT the out-of-tree build feature has no motivation > anymore. The only mentioned use case is to support incremental build > features in automatic build pipelines like Travis, but there are a > number of these build pipelines, and lots of existing build systems > that support out-of-tree builds, and AFAICT no-one actually uses them > together like this. (I think it's because most build systems' > out-of-tree build features are designed on the assumption that there's > a human running the show who will deal with any edge cases.) I believe the out-of-tree build option Nathaniel is referring to is the build_directory parameter to build_wheel(). His proposed APIs remove that parameter, and elsewhere in his email he describes that no-one seems to want it: everyone thinks someone else wants it. By my understanding, the reasons for including the build_directory parameter are: 1. Without an explicit build directory, the developers of pip are concerned that build backends will modify the source tree and cause issues which users report as bugs to pip. 2. A frontend-controlled build directory could be used for caching intermediate build artifacts. This was a secondary argument after we had the idea, and we've never really fleshed out how we expect it to work. There's also a concern that if the first round of frontends always use an empty tempdir, backends will end up relying on that. I think this second argument is a weak one unless we spend the time to figure out the details. Do other people see the situation in a similar way? How might we move forwards on this? > - needs some sort of conclusion to the srcdir-on-sys.path issue. While Nathaniel is in the minority regarding srcdir-on-sys.path, he argues that most of us are assuming a default position without really thinking through it, which is certainly true of me. I don't feel strongly about this topic, but I do want to come to a conclusion. As Nathaniel does feel strongly about it, does anyone object to either: A. Leaving the source dir off sys.path when loading the hooks, and making a special backend which exposes hooks from the CWD. B. Adding another key in the TOML file to control whether hooks can be loaded from the source dir. I can live with either, but I prefer A, because it means fewer options to describe in the spec. Thomas From xoviat at gmail.com Thu Aug 24 09:21:28 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 08:21:28 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Message-ID: I mean is this golang or Python? In Python, you raise notimplementederror. On Aug 24, 2017 8:17 AM, "Thomas Kluyver" wrote: > Nathaniel seems to be busy with other things at the moment, so I hope he > won't mind me passing on this list of things he'd like to resolve with > the draft PEP. I'll quote his comments and put my responses inline. > > > - needs some sort of NotImplemented(Error) handling specified > > We've gone round return-value vs exception a few times, but I don't > think that debate is particularly heated, so it probably just needs > someone to say "this is what we're doing". I can be that someone if > needs be. But does anyone feel strongly about it? > > > - The out-of-tree build example (that makes an sdist and then builds > > it) seems like exactly the kind of implementation that Donald says he > > doesn't want to see? OTOH I think Nick said he wants to see a more > > elaborated specification of out-of-tree build strategies with this > > specifically as one of them. > > - Nick has suggested that the to-be-defined NotImplemented(Error) > > handling can be used by build_wheel to indicate that it can't do > > out-of-tree builds, so maybe the frontend should try an in-tree build > > instead. I don't know how to convert this into a real spec, though -- > > like in general, if I'm a frontend and I call `hook(*args, **kwargs)` > > and it says "can't do that", then how do I know what the problem is > > and what I should try instead? > > - What happens if prepare_build_metadata returns NotImplemented / > > raises NotImplementedError? > > - I don't understand how out-of-tree builds and prepare_build_metadata > > are supposed to interact. > > - Also, AFAICT the out-of-tree build feature has no motivation > > anymore. The only mentioned use case is to support incremental build > > features in automatic build pipelines like Travis, but there are a > > number of these build pipelines, and lots of existing build systems > > that support out-of-tree builds, and AFAICT no-one actually uses them > > together like this. (I think it's because most build systems' > > out-of-tree build features are designed on the assumption that there's > > a human running the show who will deal with any edge cases.) > > I believe the out-of-tree build option Nathaniel is referring to is the > build_directory parameter to build_wheel(). His proposed APIs remove > that parameter, and elsewhere in his email he describes that no-one > seems to want it: everyone thinks someone else wants it. > > By my understanding, the reasons for including the build_directory > parameter are: > > 1. Without an explicit build directory, the developers of pip are > concerned that build backends will modify the source tree and cause > issues which users report as bugs to pip. > 2. A frontend-controlled build directory could be used for caching > intermediate build artifacts. This was a secondary argument after we had > the idea, and we've never really fleshed out how we expect it to work. > There's also a concern that if the first round of frontends always use > an empty tempdir, backends will end up relying on that. I think this > second argument is a weak one unless we spend the time to figure out the > details. > > Do other people see the situation in a similar way? How might we move > forwards on this? > > > - needs some sort of conclusion to the srcdir-on-sys.path issue. > > While Nathaniel is in the minority regarding srcdir-on-sys.path, he > argues that most of us are assuming a default position without really > thinking through it, which is certainly true of me. I don't feel > strongly about this topic, but I do want to come to a conclusion. As > Nathaniel does feel strongly about it, does anyone object to either: > > A. Leaving the source dir off sys.path when loading the hooks, and > making a special backend which exposes hooks from the CWD. > B. Adding another key in the TOML file to control whether hooks can be > loaded from the source dir. > > I can live with either, but I prefer A, because it means fewer options > to describe in the spec. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at cooperlees.com Wed Aug 23 23:59:40 2017 From: me at cooperlees.com (Cooper Ry Lees) Date: Wed, 23 Aug 2017 20:59:40 -0700 Subject: [Distutils] Mirroring PyPI JSON Locally In-Reply-To: References: <8C07FE46-459A-49CF-B7F5-515041E5C4B3@stufft.io> Message-ID: <9569C737-9C63-4F1C-ABFF-185C85CCD5E5@cooperlees.com> Thanks all for your points. Is it fine if I get a Informational PEP going to discuss 'Metadata Repository API'? I will structure it similar to PEP503, but also talk about: - The data exposed today (Specification) -- And possibly call on some PyPI people to correct me where I guess wrong - How mirrors should mirror it - Possible furure enhancements (JSONLD etc.) What else should we have in this PEP? Cooper > On Aug 23, 2017, at 7:04 PM, Nick Coghlan wrote: > > On 24 August 2017 at 04:59, Donald Stufft wrote: >> I don?t have a major opinion on a PEP for the JSON api or not. It depends I >> guess on whether tools like bandersnatch/devpi/etc want to offer it. Given >> that this is all brought on by a PR to bandersnatch it appears that there is >> a reasonable argument that it is something that those tools want, and >> standardizing it is a good idea. > > +1, especially as it will help clarify the required test cases for > Warehouse as well (I'm not sure how much of the JSON API has been > implemented at this point). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From thomas at kluyver.me.uk Thu Aug 24 09:28:01 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 24 Aug 2017 14:28:01 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Message-ID: <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote: > I mean is this golang or Python? In Python, you raise > notimplementederror. But there's a NotImplemented singleton in Python as well. The argument for using a return value is that the hook code has to deliberately return that, whereas a NotImplementedError could bubble up from some internal call, in which case it should really be registered as an error. -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Thu Aug 24 09:36:26 2017 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Thu, 24 Aug 2017 10:36:26 -0300 Subject: [Distutils] PEP 517 again In-Reply-To: <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 10:28, Thomas Kluyver wrote: > On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote: > > I mean is this golang or Python? In Python, you raise notimplementederror. > > > But there's a NotImplemented singleton in Python as well. The argument for > using a return value is that the hook code has to deliberately return that, > whereas a NotImplementedError could bubble up from some internal call, in > which case it should really be registered as an error. > This! And a backend that wanted a bubbled up NotImplementedError (or some other exception) to mean it actually doesn't support an operation, can catch the error and return. The ideal Pythonic way would be to raise a specific exception (something like PEP517UnsupportedOperation), but since it's not a builtin we would be left with the problem of defining it somewhere, which would become a dependency of all backends, and no, we don't want that. All this has been hashed and rehashed before. I think we can move on from this argument and define `return NotImplemented` as the way to define a backend operation as unsupported. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 09:41:08 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 08:41:08 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> Message-ID: ---------- Forwarded message ---------- From: "xoviat" Date: Aug 24, 2017 8:39 AM Subject: Re: [Distutils] PEP 517 again To: "Thomas Kluyver" Cc: That's actually the general argument against exceptions and why golang doesn't have them. I have not seen notimplemented used in the wild ever though. Also, regarding the build_dir argument, it won't be used in pip. The pip devs may have said that but it's too much work to support now and most of them seem to have a shortage of time. Pip will be copying to a temporary directory for a long time. On Aug 24, 2017 8:28 AM, "Thomas Kluyver" wrote: > On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote: > > I mean is this golang or Python? In Python, you raise notimplementederror. > > > But there's a NotImplemented singleton in Python as well. The argument for > using a return value is that the hook code has to deliberately return that, > whereas a NotImplementedError could bubble up from some internal call, in > which case it should really be registered as an error. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 10:05:47 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 09:05:47 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> Message-ID: WRT the return value, I don't really care. But I am strongly in favor of removing the build_dir option as I think that the entire premise is wrong. During development, you want to use an editable install, which does modify the source tree. And if you're just installing, then you don't care about cache. And there is also nothing preventing pip from caching the temporary directory, which may contain build artifacts. On Aug 24, 2017 8:41 AM, "xoviat" wrote: > ---------- Forwarded message ---------- > From: "xoviat" > Date: Aug 24, 2017 8:39 AM > Subject: Re: [Distutils] PEP 517 again > To: "Thomas Kluyver" > Cc: > > That's actually the general argument against exceptions and why golang > doesn't have them. I have not seen notimplemented used in the wild ever > though. > > Also, regarding the build_dir argument, it won't be used in pip. The pip > devs may have said that but it's too much work to support now and most of > them seem to have a shortage of time. Pip will be copying to a temporary > directory for a long time. > > On Aug 24, 2017 8:28 AM, "Thomas Kluyver" wrote: > >> On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote: >> >> I mean is this golang or Python? In Python, you raise notimplementederror. >> >> >> But there's a NotImplemented singleton in Python as well. The argument >> for using a return value is that the hook code has to deliberately return >> that, whereas a NotImplementedError could bubble up from some internal >> call, in which case it should really be registered as an error. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Thu Aug 24 10:07:30 2017 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Thu, 24 Aug 2017 11:07:30 -0300 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 10:41, xoviat wrote: > On Aug 24, 2017 8:28 AM, "Thomas Kluyver" wrote: > >> On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote: >> >> I mean is this golang or Python? In Python, you raise notimplementederror. >> >> >> But there's a NotImplemented singleton in Python as well. The argument >> for using a return value is that the hook code has to deliberately return >> that, whereas a NotImplementedError could bubble up from some internal >> call, in which case it should really be registered as an error. >> > > > That's actually the general argument against exceptions and why golang > doesn't have them. I have not seen notimplemented used in the wild ever > though. > https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Lib/fractions.py#L382 So, yes, it's used in the wild, by stdlib itself no less. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Thu Aug 24 10:26:47 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 24 Aug 2017 15:26:47 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> Message-ID: <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> On Thu, Aug 24, 2017, at 03:05 PM, xoviat wrote: > During development, you want to use an editable install, which does > modify the source tree. I think it was before you joined the discussion, but editable installs came up before and have deliberately been left out of PEP 517 because of the complexities. I am very keen that we do not go back into that discussion. I particularly want to hear from Nick, Daniel, Donald and Paul on their understanding of the build_directory parameter. There were reasons why it got added, and even if those reasons no longer seem so convincing to me, it's only going to be removed again if we can reach a consensus on working without it. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 10:29:48 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 09:29:48 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: I understand that, but what I disagree with is modifying build_wheel to make up for a lack of consensus on editable installs. On Aug 24, 2017 9:27 AM, "Thomas Kluyver" wrote: > On Thu, Aug 24, 2017, at 03:05 PM, xoviat wrote: > > During development, you want to use an editable install, which does modify > the source tree. > > > I think it was before you joined the discussion, but editable installs > came up before and have deliberately been left out of PEP 517 because of > the complexities. I am very keen that we do not go back into that > discussion. > > I particularly want to hear from Nick, Daniel, Donald and Paul on their > understanding of the build_directory parameter. There were reasons why it > got added, and even if those reasons no longer seem so convincing to me, > it's only going to be removed again if we can reach a consensus on working > without it. > > Thomas > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Thu Aug 24 10:31:55 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 24 Aug 2017 15:31:55 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: <1503585115.2865226.1083736744.7BDD445E@webmail.messagingengine.com> On Thu, Aug 24, 2017, at 03:29 PM, xoviat wrote: > I understand that, but what I disagree with is modifying build_wheel > to make up for a lack of consensus on editable installs. The reasoning for the build_directory parameter was not based around editable installs. This is the confusion I was referring to about the multiple meanings of 'out-of-tree builds'. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 10:37:12 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 09:37:12 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503585115.2865226.1083736744.7BDD445E@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503585115.2865226.1083736744.7BDD445E@webmail.messagingengine.com> Message-ID: Thomas, I did not receive the last message. On Aug 24, 2017 9:31 AM, "Thomas Kluyver" wrote: > On Thu, Aug 24, 2017, at 03:29 PM, xoviat wrote: > > I understand that, but what I disagree with is modifying build_wheel to > make up for a lack of consensus on editable installs. > > > The reasoning for the build_directory parameter was not based around > editable installs. This is the confusion I was referring to about the > multiple meanings of 'out-of-tree builds'. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Aug 24 10:51:48 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 24 Aug 2017 15:51:48 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 15:26, Thomas Kluyver wrote: > I particularly want to hear from Nick, Daniel, Donald and Paul on their > understanding of the build_directory parameter. There were reasons why it > got added, and even if those reasons no longer seem so convincing to me, > it's only going to be removed again if we can reach a consensus on working > without it. OK. I've not had much time recently (RL issues) and so my apologies if I've forgotten context here. But my feelings are: 1. I've been pretty much persuaded that including features in the interface to defend against badly written backends is not worthwhile. (I still worry that it may be hard to provide good error reporting for backend errors, but that's a lesser issue). 2. I'm not completely clear on how pip's implementation will work - I think the intention is to always build a sdist and build a wheel from that, unless the backend reports it can't build a sdist, in which case we ask it to build a wheel directly. 3. I'm not particularly comfortable with backends saying that they can only produce a wheel and not a sdist, but accept that might have to be a reality in some cases. I would be *really* bothered by a backend that wasn't even willing to accept that sdist->wheel and direct wheel builds should be equivalent - but I don't know if that's something that is likely to happen (we can of course state that expectation in the PEP). The PEP seems to think this is possible, though[1] (see my next point). 4. The point of all this is that the definition of build_directory says "If build_directory is None, the backend may do an 'in place' build which modifies the source directory and may produce different results from those that would be obtained by exporting an sdist first". That's the bit that bothers me a lot - if we lose build_directory, we lose the option for pip to request that a backend produce a wheel "equivalent to producing an sdist first". And that capability is something I think pip should require[2]. I don't particularly care who specifies the actual directory (as per your comment, I thought someone else wanted this, for caching or something) but I *do* care about telling backends we don't want "different results from those that would be obtained by exporting an sdist first". Paul [1] The only example I can think of is flit where the user doesn't have git installed. But then I'd expect that a wheel generated directly in that case would be the same as if the user installed git then did sdist->wheel, so I'd still consider that to be maintaining the invariant I care about. [2] I'm not likely to be the one implementing pip's PEP 517 support, so I could be overridden on this, of course. From ncoghlan at gmail.com Thu Aug 24 10:52:38 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Aug 2017 00:52:38 +1000 Subject: [Distutils] PEP 517 again In-Reply-To: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 23:11, Thomas Kluyver wrote: > Nathaniel seems to be busy with other things at the moment, so I hope he > won't mind me passing on this list of things he'd like to resolve with > the draft PEP. I'll quote his comments and put my responses inline. > >> - needs some sort of NotImplemented(Error) handling specified > > We've gone round return-value vs exception a few times, but I don't > think that debate is particularly heated, so it probably just needs > someone to say "this is what we're doing". I can be that someone if > needs be. But does anyone feel strongly about it? Aye, I do, and it should be "raise NotImplementedError('Explanation of the failed check')" Rationale: - Python isn't C or Go, so we indicate failures with exceptions, not error codes (NotImplemented is an necessary performance hack for operand coercion in tight loops, not an example to be emulated in other APIs) - allows the backend to provide information on what went wrong - means an unhandled backend error results in a traceback pointing to where the build failed, not some later point in the frontend code - if a backend developer is sufficiently worried about accidentally propagating NotImplementedError that they want to pretend they're not writing Python any more, they can use this idiom: def public_hook_api(*args, **kwds): try: result, error_msg = _internal_hook_implementation(*args, **kwds) except NotImplementedError as exc: raise RuntimeError("Unexpected NotImplementedError") from exc if result is NotImplemented: raise NotImplementedError(error_msg) return result That provides the backend with all the same assurances against accidentally letting NotImplementedError escape that a return code based public API would, without frontends even needing to be aware of the backend developer's aversion to reporting errors as exceptions. >> - The out-of-tree build example (that makes an sdist and then builds >> it) seems like exactly the kind of implementation that Donald says he >> doesn't want to see? OTOH I think Nick said he wants to see a more >> elaborated specification of out-of-tree build strategies with this >> specifically as one of them. Not really - I raised that prospect because Nathaniel was insisting that out-of-tree builds would be too hard for backend developers to implement, and I pointed out that they really weren't that complicated: - if you're wrapping a backend that supports them, pass down the directory setting - if you're not, implement them as semantically equivalent to build sdist -> unpack sdist -> build wheel - since build_sdist can fail with NotImplementedError, out-of-tree builds are also permitted to fail that way >> - Nick has suggested that the to-be-defined NotImplemented(Error) >> handling can be used by build_wheel to indicate that it can't do >> out-of-tree builds, so maybe the frontend should try an in-tree build >> instead. I don't know how to convert this into a real spec, though -- >> like in general, if I'm a frontend and I call `hook(*args, **kwargs)` >> and it says "can't do that", then how do I know what the problem is >> and what I should try instead? The fallback chains are defined by frontends, not the spec, and they depend on: - what the frontend is trying to do - what the fronted is trying to prevent For the case of "build a wheel from a source tree" for example, a reasonable fallback chain might be: - try build_sdist - if that raises NotImplementedError, try an explicitly out-of-tree build_wheel call - if that also raises NotImplementedError, try an unqualified build_wheel call It would also be equally reasonable to skip either or both of the first two options. >> - What happens if prepare_build_metadata returns NotImplemented / >> raises NotImplementedError? Up to the frontend, but failing outright seems reasonable (since there isn't any real reason to expect generating the entire wheel to succeed if generating the metadata failed) >> - I don't understand how out-of-tree builds and prepare_build_metadata >> are supposed to interact. They don't, since the backend should only implement prepare_build_metadata if it can generate the metadata without actually triggering a full build of all the binary artifacts. >> - Also, AFAICT the out-of-tree build feature has no motivation >> anymore. The only mentioned use case is to support incremental build >> features in automatic build pipelines like Travis, but there are a >> number of these build pipelines, and lots of existing build systems >> that support out-of-tree builds, and AFAICT no-one actually uses them >> together like this. (I think it's because most build systems' >> out-of-tree build features are designed on the assumption that there's >> a human running the show who will deal with any edge cases.) > > I believe the out-of-tree build option Nathaniel is referring to is the > build_directory parameter to build_wheel(). His proposed APIs remove > that parameter, and elsewhere in his email he describes that no-one > seems to want it: everyone thinks someone else wants it. > > By my understanding, the reasons for including the build_directory > parameter are: > > 1. Without an explicit build directory, the developers of pip are > concerned that build backends will modify the source tree and cause > issues which users report as bugs to pip. This is the motivation that Donald and Paul have indicated isn't necessarily valid any more, since they'd be OK with a setup where pip uses the following build model by default: 1. First try the explicit build_sdist -> unpack sdist -> build_wheel path 2. If build_sdist raises NotImplementedError, fall back to trying build_wheel directly > 2. A frontend-controlled build directory could be used for caching > intermediate build artifacts. This was a secondary argument after we had > the idea, and we've never really fleshed out how we expect it to work. > There's also a concern that if the first round of frontends always use > an empty tempdir, backends will end up relying on that. I think this > second argument is a weak one unless we spend the time to figure out the > details. > > Do other people see the situation in a similar way? How might we move > forwards on this? As long as Donald & Paul are OK with it for pip, I'm OK with omitting the build_directory parameter for now - since we're going to make supporting it optional regardless, that means it doesn't matter as much whether its in the initial iteration of the API or not. >> - needs some sort of conclusion to the srcdir-on-sys.path issue. > > While Nathaniel is in the minority regarding srcdir-on-sys.path, he > argues that most of us are assuming a default position without really > thinking through it, which is certainly true of me. I don't feel > strongly about this topic, but I do want to come to a conclusion. As > Nathaniel does feel strongly about it, does anyone object to either: > > A. Leaving the source dir off sys.path when loading the hooks, and > making a special backend which exposes hooks from the CWD. > B. Adding another key in the TOML file to control whether hooks can be > loaded from the source dir. I'm mainly interested in the way things fail, and who has the power to fix them when they break. - when the default is "source tree is set as sys.path[0]": - that's essentially the same as the status quo with setup.py - if a project's build process has a name shadowing problem, the publisher is going to hit that locally and fix it prior to release - when the default is "the source tree is not set as sys.path[0]", we know that: - any setuptools backend is going to have to ensure that the old default is in place when running setup.py - any backend that runs a Python script is going to end up with the CWD as sys.path[0] in that script anyway due to Python's default behaviour - if a project self-hosts its own build backend, we either have to say "that's not supported", or else provide a way to change the default So I think this is a case where we have an established precedent (i.e. the way setup.py currently works), and the potential for accidental name shadowing doesn't offer a sufficiently compelling rationale for changing that default. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Aug 24 11:07:54 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Aug 2017 01:07:54 +1000 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: On 25 August 2017 at 00:51, Paul Moore wrote: > 4. The point of all this is that the definition of build_directory > says "If build_directory is None, the backend may do an 'in place' > build which modifies the source directory and may produce different > results from those that would be obtained by exporting an sdist > first". That's the bit that bothers me a lot - if we lose > build_directory, we lose the option for pip to request that a backend > produce a wheel "equivalent to producing an sdist first". And that > capability is something I think pip should require[2]. I don't > particularly care who specifies the actual directory (as per your > comment, I thought someone else wanted this, for caching or something) > but I *do* care about telling backends we don't want "different > results from those that would be obtained by exporting an sdist > first". That's a phrasing error in the PEP - ideally, backends should always produce consistent results regardless of whether or not the frontend builds an sdist first. What that phrasing was attempting to acknowledge is that actually ensuring sdist/wheel consistency may have some extra requirements (like the git CLI) that may not apply to the "just give me a usable wheel, I don't care whether it's publication quality or not" case. The key thing that changed my mind on whether or not we needed to include the build_directory parameter from the start was actually the NotImplementedError discussion: as soon as we permitted backends to raise NotImplementedError for any non-None build_directory setting, we'd effectively made supporting the feature optional. And if supporting the feature's going to be optional anyway, then the *whole point* of defining the backend API in Python rather than as a CLI was to make it easy for us to add new optional methods later, which means we can start without build_directory support, and if we find we miss it for some specific reason, then we can add it back as a suitably named optional method like "build_wheel_variant" (if the use-case is cross-compiling for multiple ABIs without clobbering intermediate artifacts), or "build_wheel_with_artifact_cache" (if the use case is to allow the frontend to ask that the backend cache intermediate values in a particular place). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From xoviat at gmail.com Thu Aug 24 11:15:14 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 10:15:14 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: > 2. I'm not completely clear on how pip's implementation will work - I think the intention is to always build a sdist and build a wheel from that, unless the backend reports it can't build a sdist, in which case we ask it to build a wheel directly. This was the exact process that I proposed, but was told that: > I agree that the way you want to do packaging is fundamentally incompatible with build systems that do not resemble distutils. Perhaps since this is distutils sig some here are too used to distutils as the only model of how packaging might work. Beware that, when fighting monsters, you yourself do not become a monster... for when you gaze long into the abyss. The abyss gazes also into you. There seems to be some miscommunication about the actual process being proposed. 2017-08-24 10:07 GMT-05:00 Nick Coghlan : > On 25 August 2017 at 00:51, Paul Moore wrote: > > 4. The point of all this is that the definition of build_directory > > says "If build_directory is None, the backend may do an 'in place' > > build which modifies the source directory and may produce different > > results from those that would be obtained by exporting an sdist > > first". That's the bit that bothers me a lot - if we lose > > build_directory, we lose the option for pip to request that a backend > > produce a wheel "equivalent to producing an sdist first". And that > > capability is something I think pip should require[2]. I don't > > particularly care who specifies the actual directory (as per your > > comment, I thought someone else wanted this, for caching or something) > > but I *do* care about telling backends we don't want "different > > results from those that would be obtained by exporting an sdist > > first". > > That's a phrasing error in the PEP - ideally, backends should always > produce consistent results regardless of whether or not the frontend > builds an sdist first. What that phrasing was attempting to > acknowledge is that actually ensuring sdist/wheel consistency may have > some extra requirements (like the git CLI) that may not apply to the > "just give me a usable wheel, I don't care whether it's publication > quality or not" case. > > The key thing that changed my mind on whether or not we needed to > include the build_directory parameter from the start was actually the > NotImplementedError discussion: as soon as we permitted backends to > raise NotImplementedError for any non-None build_directory setting, > we'd effectively made supporting the feature optional. > > And if supporting the feature's going to be optional anyway, then the > *whole point* of defining the backend API in Python rather than as a > CLI was to make it easy for us to add new optional methods later, > which means we can start without build_directory support, and if we > find we miss it for some specific reason, then we can add it back as a > suitably named optional method like "build_wheel_variant" (if the > use-case is cross-compiling for multiple ABIs without clobbering > intermediate artifacts), or "build_wheel_with_artifact_cache" (if the > use case is to allow the frontend to ask that the backend cache > intermediate values in a particular place). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 11:20:09 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 10:20:09 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: > I *do* care about telling backends we don't want "different results from those that would be obtained by exporting an sdist first". I completely agree with this statement, but I don't believe that it can or should be accomplished with this parameter. Let me just quote the process that I proposed: > Proposed process: - Frontend copies source tree to temporary directory - Frontend invokes build-sdist to build an sdist - Frontend extracts sdist to new temporary directory - Frontend reloads backend from sdist directory and invokes build-wheel This process makes the most sense to me because there will be a limited number of frontends AND the frontends will enforce correctness. I don't think we should be too cautious about burdening the frontends will more responsibility, because it will only be implemented once or twice. And to clarify, the build_sdist is required. 2017-08-24 9:51 GMT-05:00 Paul Moore : > On 24 August 2017 at 15:26, Thomas Kluyver wrote: > > I particularly want to hear from Nick, Daniel, Donald and Paul on their > > understanding of the build_directory parameter. There were reasons why it > > got added, and even if those reasons no longer seem so convincing to me, > > it's only going to be removed again if we can reach a consensus on > working > > without it. > > OK. I've not had much time recently (RL issues) and so my apologies if > I've forgotten context here. But my feelings are: > > 1. I've been pretty much persuaded that including features in the > interface to defend against badly written backends is not worthwhile. > (I still worry that it may be hard to provide good error reporting for > backend errors, but that's a lesser issue). > 2. I'm not completely clear on how pip's implementation will work - I > think the intention is to always build a sdist and build a wheel from > that, unless the backend reports it can't build a sdist, in which case > we ask it to build a wheel directly. > 3. I'm not particularly comfortable with backends saying that they can > only produce a wheel and not a sdist, but accept that might have to be > a reality in some cases. I would be *really* bothered by a backend > that wasn't even willing to accept that sdist->wheel and direct wheel > builds should be equivalent - but I don't know if that's something > that is likely to happen (we can of course state that expectation in > the PEP). The PEP seems to think this is possible, though[1] (see my > next point). > 4. The point of all this is that the definition of build_directory > says "If build_directory is None, the backend may do an 'in place' > build which modifies the source directory and may produce different > results from those that would be obtained by exporting an sdist > first". That's the bit that bothers me a lot - if we lose > build_directory, we lose the option for pip to request that a backend > produce a wheel "equivalent to producing an sdist first". And that > capability is something I think pip should require[2]. I don't > particularly care who specifies the actual directory (as per your > comment, I thought someone else wanted this, for caching or something) > but I *do* care about telling backends we don't want "different > results from those that would be obtained by exporting an sdist > first". > > Paul > > [1] The only example I can think of is flit where the user doesn't > have git installed. But then I'd expect that a wheel generated > directly in that case would be the same as if the user installed git > then did sdist->wheel, so I'd still consider that to be maintaining > the invariant I care about. > [2] I'm not likely to be the one implementing pip's PEP 517 support, > so I could be overridden on this, of course. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Aug 24 11:21:55 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 24 Aug 2017 16:21:55 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 16:15, xoviat wrote: >> 2. I'm not completely clear on how pip's implementation will work - I > think the intention is to always build a sdist and build a wheel from > that, unless the backend reports it can't build a sdist, in which case > we ask it to build a wheel directly. > > This was the exact process that I proposed, but was told that: > > >> I agree that the way you want to do packaging is fundamentally >> incompatible > with build systems that do not resemble distutils. Perhaps since this is > distutils sig some here are too used to distutils as the only model of how > packaging might work. Beware that, when fighting monsters, you yourself do > not become a monster... for when you gaze long into the abyss. The abyss > gazes also into you. That wasn't me that said that. > There seems to be some miscommunication about the actual process being > proposed. Possibly - or there's some confusion about what you're proposing. I certainly have found your suggestions difficult to follow, and I don't recall you having said "we do sdist->wheel then fall back to requesting wheels directly". I also don't see how that's incompatible with systems that don't resemble distutils, so I suspect that whoever you quoted (which I think might have been Daniel) understood whatever you did say to mean something different. I know Daniel has been involved in the discussion and I don't think he's raised any such objection about the PEP (and he developed enscons, so he has direct experience of writing backends). Paul From xoviat at gmail.com Thu Aug 24 11:23:45 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 10:23:45 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: > I don't recall you having said "we do sdist->wheel then fall back to requesting wheels directly" You're correct. I did not say that because build_sdist is required. 2017-08-24 10:21 GMT-05:00 Paul Moore : > On 24 August 2017 at 16:15, xoviat wrote: > >> 2. I'm not completely clear on how pip's implementation will work - I > > think the intention is to always build a sdist and build a wheel from > > that, unless the backend reports it can't build a sdist, in which case > > we ask it to build a wheel directly. > > > > This was the exact process that I proposed, but was told that: > > > > > >> I agree that the way you want to do packaging is fundamentally > >> incompatible > > with build systems that do not resemble distutils. Perhaps since this is > > distutils sig some here are too used to distutils as the only model of > how > > packaging might work. Beware that, when fighting monsters, you yourself > do > > not become a monster... for when you gaze long into the abyss. The abyss > > gazes also into you. > > That wasn't me that said that. > > > There seems to be some miscommunication about the actual process being > > proposed. > > Possibly - or there's some confusion about what you're proposing. I > certainly have found your suggestions difficult to follow, and I don't > recall you having said "we do sdist->wheel then fall back to > requesting wheels directly". I also don't see how that's incompatible > with systems that don't resemble distutils, so I suspect that whoever > you quoted (which I think might have been Daniel) understood whatever > you did say to mean something different. > > I know Daniel has been involved in the discussion and I don't think > he's raised any such objection about the PEP (and he developed > enscons, so he has direct experience of writing backends). > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Aug 24 11:24:24 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 24 Aug 2017 16:24:24 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 16:20, xoviat wrote: >> I *do* care about telling backends we don't want "different > results from those that would be obtained by exporting an sdist > first". > > I completely agree with this statement, but I don't believe that it can or > should be accomplished with this parameter. Let me just quote the process > that I proposed: > > >> Proposed process: > - Frontend copies source tree to temporary directory That step's the problem. If the frontend does that it can potentially be copying a lot of unneeded stuff (VCS history, for example). We tried that with pip and it was a major issue. That problem is the *whole point* of all the discussions about the various proposals that ended up with build_tree. > - Frontend invokes build-sdist to build an sdist > - Frontend extracts sdist to new temporary directory > - Frontend reloads backend from sdist directory and invokes build-wheel Paul From xoviat at gmail.com Thu Aug 24 11:24:24 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 10:24:24 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: > I know Daniel has been involved in the discussion and I don't think he's raised any such objection about the PEP (and he developed enscons, so he has direct experience of writing backends). Daniel raised that specific objection. 2017-08-24 10:21 GMT-05:00 Paul Moore : > On 24 August 2017 at 16:15, xoviat wrote: > >> 2. I'm not completely clear on how pip's implementation will work - I > > think the intention is to always build a sdist and build a wheel from > > that, unless the backend reports it can't build a sdist, in which case > > we ask it to build a wheel directly. > > > > This was the exact process that I proposed, but was told that: > > > > > >> I agree that the way you want to do packaging is fundamentally > >> incompatible > > with build systems that do not resemble distutils. Perhaps since this is > > distutils sig some here are too used to distutils as the only model of > how > > packaging might work. Beware that, when fighting monsters, you yourself > do > > not become a monster... for when you gaze long into the abyss. The abyss > > gazes also into you. > > That wasn't me that said that. > > > There seems to be some miscommunication about the actual process being > > proposed. > > Possibly - or there's some confusion about what you're proposing. I > certainly have found your suggestions difficult to follow, and I don't > recall you having said "we do sdist->wheel then fall back to > requesting wheels directly". I also don't see how that's incompatible > with systems that don't resemble distutils, so I suspect that whoever > you quoted (which I think might have been Daniel) understood whatever > you did say to mean something different. > > I know Daniel has been involved in the discussion and I don't think > he's raised any such objection about the PEP (and he developed > enscons, so he has direct experience of writing backends). > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Aug 24 11:27:04 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 24 Aug 2017 16:27:04 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 16:23, xoviat wrote: >> I don't recall you having said "we do sdist->wheel then fall back to > requesting wheels directly" > > You're correct. I did not say that because build_sdist is required. Wait, what? Flit specifically needs to be allowed to refuse to build sdists (as I understand it), for example if the user doesn't have git installed. I thought that's what the NotImplemented exception was for? You're right that the PEP doesn't seem to mention this. Paul From xoviat at gmail.com Thu Aug 24 11:27:38 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 10:27:38 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: That is actually the first time that build_dir makes sense to me now. Thank you. 2017-08-24 10:24 GMT-05:00 Paul Moore : > On 24 August 2017 at 16:20, xoviat wrote: > >> I *do* care about telling backends we don't want "different > > results from those that would be obtained by exporting an sdist > > first". > > > > I completely agree with this statement, but I don't believe that it can > or > > should be accomplished with this parameter. Let me just quote the process > > that I proposed: > > > > > >> Proposed process: > > - Frontend copies source tree to temporary directory > > That step's the problem. If the frontend does that it can potentially > be copying a lot of unneeded stuff (VCS history, for example). We > tried that with pip and it was a major issue. That problem is the > *whole point* of all the discussions about the various proposals that > ended up with build_tree. > > > - Frontend invokes build-sdist to build an sdist > > - Frontend extracts sdist to new temporary directory > > - Frontend reloads backend from sdist directory and invokes build-wheel > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 11:32:31 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 10:32:31 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: May I ask what is wrong *in principle* with the setuptools "build" folder (other than the fact that it does not contain all tree changes)? 2017-08-24 10:27 GMT-05:00 xoviat : > That is actually the first time that build_dir makes sense to me now. > Thank you. > > 2017-08-24 10:24 GMT-05:00 Paul Moore : > >> On 24 August 2017 at 16:20, xoviat wrote: >> >> I *do* care about telling backends we don't want "different >> > results from those that would be obtained by exporting an sdist >> > first". >> > >> > I completely agree with this statement, but I don't believe that it can >> or >> > should be accomplished with this parameter. Let me just quote the >> process >> > that I proposed: >> > >> > >> >> Proposed process: >> > - Frontend copies source tree to temporary directory >> >> That step's the problem. If the frontend does that it can potentially >> be copying a lot of unneeded stuff (VCS history, for example). We >> tried that with pip and it was a major issue. That problem is the >> *whole point* of all the discussions about the various proposals that >> ended up with build_tree. >> >> > - Frontend invokes build-sdist to build an sdist >> > - Frontend extracts sdist to new temporary directory >> > - Frontend reloads backend from sdist directory and invokes build-wheel >> >> Paul >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 11:48:51 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 10:48:51 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: If the entire idea of copying out-of-tree is to work around setuptools deficiencies, then perhaps it would be a better idea to push this onto the setuptools build backend rather than bring these problems into PEP 517? 2017-08-24 10:32 GMT-05:00 xoviat : > May I ask what is wrong *in principle* with the setuptools "build" folder > (other than the fact that it does not contain all tree changes)? > > 2017-08-24 10:27 GMT-05:00 xoviat : > >> That is actually the first time that build_dir makes sense to me now. >> Thank you. >> >> 2017-08-24 10:24 GMT-05:00 Paul Moore : >> >>> On 24 August 2017 at 16:20, xoviat wrote: >>> >> I *do* care about telling backends we don't want "different >>> > results from those that would be obtained by exporting an sdist >>> > first". >>> > >>> > I completely agree with this statement, but I don't believe that it >>> can or >>> > should be accomplished with this parameter. Let me just quote the >>> process >>> > that I proposed: >>> > >>> > >>> >> Proposed process: >>> > - Frontend copies source tree to temporary directory >>> >>> That step's the problem. If the frontend does that it can potentially >>> be copying a lot of unneeded stuff (VCS history, for example). We >>> tried that with pip and it was a major issue. That problem is the >>> *whole point* of all the discussions about the various proposals that >>> ended up with build_tree. >>> >>> > - Frontend invokes build-sdist to build an sdist >>> > - Frontend extracts sdist to new temporary directory >>> > - Frontend reloads backend from sdist directory and invokes build-wheel >>> >>> Paul >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Aug 24 12:26:43 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 24 Aug 2017 16:26:43 +0000 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: build_dir: bdist_wheel sometimes has a problem with unclean builds. admittedly this could be fixed. pip copies entire source tree, including unclean build/ directory, to tmpdir bdist_wheel runs in tmpdir, including copied leftover build/ data in wheel by including the build_dir parameter, a nearly universal build system concept, pip gets an elegant way to ask for either a clean or unclean build. sdist optional: enscons can use .hg manifest instead of MANIFEST.in. what is this git nonsense? user has decided to build from an unpacked sdist impossible to create new sdist from unpacked sdist On Thu, Aug 24, 2017 at 11:49 AM xoviat wrote: > If the entire idea of copying out-of-tree is to work around setuptools > deficiencies, then perhaps it would be a better idea to push this onto the > setuptools build backend rather than bring these problems into PEP 517? > > 2017-08-24 10:32 GMT-05:00 xoviat : > >> May I ask what is wrong *in principle* with the setuptools "build" folder >> (other than the fact that it does not contain all tree changes)? >> >> 2017-08-24 10:27 GMT-05:00 xoviat : >> >>> That is actually the first time that build_dir makes sense to me now. >>> Thank you. >>> >>> 2017-08-24 10:24 GMT-05:00 Paul Moore : >>> >>>> On 24 August 2017 at 16:20, xoviat wrote: >>>> >> I *do* care about telling backends we don't want "different >>>> > results from those that would be obtained by exporting an sdist >>>> > first". >>>> > >>>> > I completely agree with this statement, but I don't believe that it >>>> can or >>>> > should be accomplished with this parameter. Let me just quote the >>>> process >>>> > that I proposed: >>>> > >>>> > >>>> >> Proposed process: >>>> > - Frontend copies source tree to temporary directory >>>> >>>> That step's the problem. If the frontend does that it can potentially >>>> be copying a lot of unneeded stuff (VCS history, for example). We >>>> tried that with pip and it was a major issue. That problem is the >>>> *whole point* of all the discussions about the various proposals that >>>> ended up with build_tree. >>>> >>>> > - Frontend invokes build-sdist to build an sdist >>>> > - Frontend extracts sdist to new temporary directory >>>> > - Frontend reloads backend from sdist directory and invokes >>>> build-wheel >>>> >>>> Paul >>>> >>> >>> >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Thu Aug 24 12:34:09 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 24 Aug 2017 17:34:09 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> On Thu, Aug 24, 2017, at 05:26 PM, Daniel Holth wrote: > by including the build_dir parameter, a nearly universal build > system concept, pip gets an elegant way to ask for either a clean or > unclean build. Is there a reason to ask for an 'unclean' build, though? There may be a performance optimisation from reusing cached data, but building in a separate directory doesn't preclude caching intermediates somewhere else. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Aug 24 13:20:13 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 24 Aug 2017 17:20:13 +0000 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> Message-ID: On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver wrote: > On Thu, Aug 24, 2017, at 05:26 PM, Daniel Holth wrote: > > by including the build_dir parameter, a nearly universal build system > concept, pip gets an elegant way to ask for either a clean or unclean build. > > > Is there a reason to ask for an 'unclean' build, though? There may be a > performance optimisation from reusing cached data, > For the same reason you would ever ask for incremental builds, to more quickly iterate while hacking, imagining that you are using the PEP 517 interface to develop, perhaps to have a uniform interface to patch something when you are not familiar with exactly the build system it uses. > but building in a separate directory doesn't preclude caching > intermediates somewhere else. > The argument is necessarily a hint to the build system? If it works it can do whatever. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 13:40:12 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 12:40:12 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> Message-ID: > For the same reason you would ever ask for incremental builds, to more quickly iterate while hacking, imagining that you are using the PEP 517 interface to develop, perhaps to have a uniform interface to patch something when you are not familiar with exactly the build system it uses. And so I reiterate: > I understand that, but what I disagree with is modifying build_wheel to make up for a lack of consensus on editable installs. I don't think the build_wheel hook should be used for iteration or development, because the intended purpose is for production. And I don't think we should even consider *any* specific deficiencies of setuptools or wheel when designing a specification. 2017-08-24 12:20 GMT-05:00 Daniel Holth : > On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver > wrote: > >> On Thu, Aug 24, 2017, at 05:26 PM, Daniel Holth wrote: >> >> by including the build_dir parameter, a nearly universal build system >> concept, pip gets an elegant way to ask for either a clean or unclean build. >> >> >> Is there a reason to ask for an 'unclean' build, though? There may be a >> performance optimisation from reusing cached data, >> > > For the same reason you would ever ask for incremental builds, to more > quickly iterate while hacking, imagining that you are using the PEP 517 > interface to develop, perhaps to have a uniform interface to patch > something when you are not familiar with exactly the build system it uses. > > >> but building in a separate directory doesn't preclude caching >> intermediates somewhere else. >> > > The argument is necessarily a hint to the build system? If it works it can > do whatever. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Thu Aug 24 14:13:22 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 24 Aug 2017 19:13:22 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> Message-ID: <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote: > On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver > wrote:>> Is there a reason to ask for an 'unclean' build, though? There may be >> a performance optimisation from reusing cached data,> > For the same reason you would ever ask for incremental builds, to > more quickly iterate while hacking, imagining that you are using the > PEP 517 interface to develop, perhaps to have a uniform interface to > patch something when you are not familiar with exactly the build > system it uses. Yup, this is what I meant about using cached data for performance. But I don't think this requires builds to modify the source directory. The build system can cache data elsewhere on the filesystem, whether or not it's given an explicit build_directory. I think we can specify one kind of build that is both 'clean' (doesn't modify the source directory) and incremental (can use cached data to speed up the build). If that's workable, it seems like it would save a lot of headaches rather than trying to specify them as two different options. We have explicitly excluded editable installs (i.e. inplace builds) from this PEP, though we can add a hook for that in a later PEP. I agree with you that this PEP should allow for fast incremental (but non-inplace) builds if possible, though. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 14:20:58 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 13:20:58 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> Message-ID: To be clear, is everyone okay with the following? build_wheel(wheel_directory, config_settings=None, metadata_directory=None): [REQUIRED] build_wheel_incremental(wheel_directory, config_settings=None, metadata_directory=None): [OPTIONAL] I still think that the frontend should not provide a build_directory, but if someone *really* wants the build_directory in the incremental function, then I will "disagree and commit." 2017-08-24 13:13 GMT-05:00 Thomas Kluyver : > On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote: > > On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver > wrote: > > Is there a reason to ask for an 'unclean' build, though? There may be a > performance optimisation from reusing cached data, > > > For the same reason you would ever ask for incremental builds, to more > quickly iterate while hacking, imagining that you are using the PEP 517 > interface to develop, perhaps to have a uniform interface to patch > something when you are not familiar with exactly the build system it uses. > > > Yup, this is what I meant about using cached data for performance. But I > don't think this requires builds to modify the source directory. The build > system can cache data elsewhere on the filesystem, whether or not it's > given an explicit build_directory. > > I think we can specify one kind of build that is both 'clean' (doesn't > modify the source directory) and incremental (can use cached data to speed > up the build). If that's workable, it seems like it would save a lot of > headaches rather than trying to specify them as two different options. > > We have explicitly excluded editable installs (i.e. inplace builds) from > this PEP, though we can add a hook for that in a later PEP. I agree with > you that this PEP should allow for fast incremental (but non-inplace) > builds if possible, though. > > Thomas > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Thu Aug 24 14:30:14 2017 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Thu, 24 Aug 2017 15:30:14 -0300 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 15:13, Thomas Kluyver wrote: > On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote: > > On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver > wrote: > > Is there a reason to ask for an 'unclean' build, though? There may be a > performance optimisation from reusing cached data, > > > For the same reason you would ever ask for incremental builds, to more > quickly iterate while hacking, imagining that you are using the PEP 517 > interface to develop, perhaps to have a uniform interface to patch > something when you are not familiar with exactly the build system it uses. > > It looks like a lot of trouble for a feature that is designed to solve a problem for a very thin intersection of the Venn diagram of people who: a. wants to quickly iterate while hacking a package b. doesn't want to bother learning how to trigger the incremental build system of that package or how to install it in editable mode. For better or worse, editable installs exist and don't need to be defined by PEP 517 just to handle this case. I'm not convinced it's worth the additional complexity of the spec just for that. Yup, this is what I meant about using cached data for performance. But I > don't think this requires builds to modify the source directory. The build > system can cache data elsewhere on the filesystem, whether or not it's > given an explicit build_directory. > Is there any backend doing this "caching elsewhere" today? Is any backend planning to? > I think we can specify one kind of build that is both 'clean' (doesn't > modify the source directory) and incremental (can use cached data to speed > up the build). If that's workable, it seems like it would save a lot of > headaches rather than trying to specify them as two different options. > > We have explicitly excluded editable installs (i.e. inplace builds) from > this PEP, though we can add a hook for that in a later PEP. I agree with > you that this PEP should allow for fast incremental (but non-inplace) > builds if possible, though. > Can we arrive at some more concrete examples of how that would work? Like how would pip be invoked, how would the backend react. And how (and for whom) this would be better than either doing an editable install or straight clean install? I fear we're complicating the spec for something that will be hard to happen in practice. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 14:39:37 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 13:39:37 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> Message-ID: > It looks like a lot of trouble for a feature that is designed to solve a problem for a very thin intersection of the Venn diagram of people who: For the record, I don't agree at all that build_wheel_incremental should be specified here. The suggestion is simply a compromise that I can tolerate because it's optional. > For better or worse, editable installs exist and don't need to be defined by PEP 517 just to handle this case. I'm not convinced it's worth the additional complexity of the spec just for that. The editable installs in my opinion should have been specified in this PEP because people are clearly trying to work around a lack of consensus on that issue so that they don't have to wait for the next PEP and waste their time waiting for uncached builds. I can't say I disagree with this but it's trying to fit a square peg into a round hole. > Is there any backend doing this "caching elsewhere" today? Is any backend planning to? Yes. Distutils caches in the build folder. It's very useful when working on complex projects with Extensions. > Can we arrive at some more concrete examples of how that would work? Like how would pip be invoked, how would the backend react. The reason that I'm willing to live with a separate optional function is that I don't think it will ever be implemented. It will be a note in Python history until install_editable is specified. 2017-08-24 13:30 GMT-05:00 Leonardo Rochael Almeida : > > On 24 August 2017 at 15:13, Thomas Kluyver wrote: > >> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote: >> >> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver >> wrote: >> >> Is there a reason to ask for an 'unclean' build, though? There may be a >> performance optimisation from reusing cached data, >> >> >> For the same reason you would ever ask for incremental builds, to more >> quickly iterate while hacking, imagining that you are using the PEP 517 >> interface to develop, perhaps to have a uniform interface to patch >> something when you are not familiar with exactly the build system it uses. >> >> It looks like a lot of trouble for a feature that is designed to solve a > problem for a very thin intersection of the Venn diagram of people who: > > a. wants to quickly iterate while hacking a package > > b. doesn't want to bother learning how to trigger the incremental build > system of that package or how to install it in editable mode. > > For better or worse, editable installs exist and don't need to be defined > by PEP 517 just to handle this case. I'm not convinced it's worth the > additional complexity of the spec just for that. > > Yup, this is what I meant about using cached data for performance. But I >> don't think this requires builds to modify the source directory. The build >> system can cache data elsewhere on the filesystem, whether or not it's >> given an explicit build_directory. >> > > Is there any backend doing this "caching elsewhere" today? Is any backend > planning to? > > >> I think we can specify one kind of build that is both 'clean' (doesn't >> modify the source directory) and incremental (can use cached data to speed >> up the build). If that's workable, it seems like it would save a lot of >> headaches rather than trying to specify them as two different options. >> >> We have explicitly excluded editable installs (i.e. inplace builds) from >> this PEP, though we can add a hook for that in a later PEP. I agree with >> you that this PEP should allow for fast incremental (but non-inplace) >> builds if possible, though. >> > > Can we arrive at some more concrete examples of how that would work? Like > how would pip be invoked, how would the backend react. And how (and for > whom) this would be better than either doing an editable install or > straight clean install? > > I fear we're complicating the spec for something that will be hard to > happen in practice. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Aug 24 14:58:52 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 24 Aug 2017 18:58:52 +0000 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> Message-ID: The distutils build/ folder is what moves if you pass build_dir to the proposed function. On Thu, Aug 24, 2017, 14:39 xoviat wrote: > > It looks like a lot of trouble for a feature that is designed to solve a > problem for a very thin intersection of the Venn diagram of people who: > > For the record, I don't agree at all that build_wheel_incremental should > be specified here. The suggestion is simply a compromise that I can > tolerate because it's optional. > > > For better or worse, editable installs exist and don't need to be > defined by PEP 517 just to handle this case. I'm not convinced it's worth > the additional complexity of the spec just for that. > > The editable installs in my opinion should have been specified in this PEP > because people are clearly trying to work around a lack of consensus on > that issue so that they don't have to wait for the next PEP and waste their > time waiting for uncached builds. I can't say I disagree with this but it's > trying to fit a square peg into a round hole. > > > Is there any backend doing this "caching elsewhere" today? Is any > backend planning to? > > Yes. Distutils caches in the build folder. It's very useful when working > on complex projects with Extensions. > > > Can we arrive at some more concrete examples of how that would work? > Like how would pip be invoked, how would the backend react. > > The reason that I'm willing to live with a separate optional function is > that I don't think it will ever be implemented. It will be a note in Python > history until install_editable is specified. > > 2017-08-24 13:30 GMT-05:00 Leonardo Rochael Almeida > : > >> >> On 24 August 2017 at 15:13, Thomas Kluyver wrote: >> >>> On Thu, Aug 24, 2017, at 06:20 PM, Daniel Holth wrote: >>> >>> On Thu, Aug 24, 2017 at 12:34 PM Thomas Kluyver >>> wrote: >>> >>> Is there a reason to ask for an 'unclean' build, though? There may be a >>> performance optimisation from reusing cached data, >>> >>> >>> For the same reason you would ever ask for incremental builds, to more >>> quickly iterate while hacking, imagining that you are using the PEP 517 >>> interface to develop, perhaps to have a uniform interface to patch >>> something when you are not familiar with exactly the build system it uses. >>> >>> It looks like a lot of trouble for a feature that is designed to solve a >> problem for a very thin intersection of the Venn diagram of people who: >> >> a. wants to quickly iterate while hacking a package >> >> b. doesn't want to bother learning how to trigger the incremental build >> system of that package or how to install it in editable mode. >> >> For better or worse, editable installs exist and don't need to be defined >> by PEP 517 just to handle this case. I'm not convinced it's worth the >> additional complexity of the spec just for that. >> >> Yup, this is what I meant about using cached data for performance. But I >>> don't think this requires builds to modify the source directory. The build >>> system can cache data elsewhere on the filesystem, whether or not it's >>> given an explicit build_directory. >>> >> >> Is there any backend doing this "caching elsewhere" today? Is any backend >> planning to? >> >> >>> I think we can specify one kind of build that is both 'clean' (doesn't >>> modify the source directory) and incremental (can use cached data to speed >>> up the build). If that's workable, it seems like it would save a lot of >>> headaches rather than trying to specify them as two different options. >>> >>> We have explicitly excluded editable installs (i.e. inplace builds) from >>> this PEP, though we can add a hook for that in a later PEP. I agree with >>> you that this PEP should allow for fast incremental (but non-inplace) >>> builds if possible, though. >>> >> >> Can we arrive at some more concrete examples of how that would work? Like >> how would pip be invoked, how would the backend react. And how (and for >> whom) this would be better than either doing an editable install or >> straight clean install? >> >> I fear we're complicating the spec for something that will be hard to >> happen in practice. >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Thu Aug 24 15:00:53 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 24 Aug 2017 20:00:53 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> Message-ID: <1503601253.3215112.1084035856.7F9F1FA5@webmail.messagingengine.com> On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote: > It looks like a lot of trouble for a feature that is designed to > solve a problem for a very thin intersection of the Venn diagram of > people who:> > a. wants to quickly iterate while hacking a package > > b. doesn't want to bother learning how to trigger the incremental > build system of that package or how to install it in editable > mode.> > For better or worse, editable installs exist and don't need to be > defined by PEP 517 just to handle this case. I'm not convinced it's > worth the additional complexity of the spec just for that. This is a good point. Given that developers can use the build tool directly for development purposes, rather than going through the PEP 517 interface, we don't need to prioritise that use case for the spec. So for the time being, I'd prefer to say that the build_wheel() hook should not create or modify files in the source directory. Backends may keep their own caches elsewhere to speed up repeated builds, but any more provisions for developer use cases should be left for a future PEP. And developers can invoke enscons/flit/... directly in the meantime. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 15:24:37 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 14:24:37 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503601253.3215112.1084035856.7F9F1FA5@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> <1503601253.3215112.1084035856.7F9F1FA5@webmail.messagingengine.com> Message-ID: So in summary: - NotImplementedError - Remove build_dir and specify that no modifications can be made to the source directory On the third issue: I understand where Nathaniel is coming from in regards to sys.path. But there should at least be an option for this because of for example, numpy: numpy builds itself with it's own build system that is defined in the source directory. If you remove '' from sys.path, then numpy cannot build itself because there is no other numpy.distutils package. 2017-08-24 14:00 GMT-05:00 Thomas Kluyver : > On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote: > > It looks like a lot of trouble for a feature that is designed to solve a > problem for a very thin intersection of the Venn diagram of people who: > > a. wants to quickly iterate while hacking a package > > b. doesn't want to bother learning how to trigger the incremental build > system of that package or how to install it in editable mode. > > For better or worse, editable installs exist and don't need to be defined > by PEP 517 just to handle this case. I'm not convinced it's worth the > additional complexity of the spec just for that. > > > This is a good point. Given that developers can use the build tool > directly for development purposes, rather than going through the PEP 517 > interface, we don't need to prioritise that use case for the spec. > > So for the time being, I'd prefer to say that the build_wheel() hook > should not create or modify files in the source directory. Backends may > keep their own caches elsewhere to speed up repeated builds, but any more > provisions for developer use cases should be left for a future PEP. And > developers can invoke enscons/flit/... directly in the meantime. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 15:26:10 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 14:26:10 -0500 Subject: [Distutils] Fwd: Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> <1503601253.3215112.1084035856.7F9F1FA5@webmail.messagingengine.com> Message-ID: ---------- Forwarded message ---------- From: xoviat Date: 2017-08-24 14:26 GMT-05:00 Subject: Re: [Distutils] Fwd: Re: PEP 517 again To: Thomas Kluyver And also, are Nick and Paul okay with this? 2017-08-24 14:24 GMT-05:00 xoviat : > So in summary: > - NotImplementedError > - Remove build_dir and specify that no modifications can be made to the > source directory > > On the third issue: > > I understand where Nathaniel is coming from in regards to sys.path. But > there should at least be an option for this because of for example, numpy: > numpy builds itself with it's own build system that is defined in the > source directory. If you remove '' from sys.path, then numpy cannot build > itself because there is no other numpy.distutils package. > > 2017-08-24 14:00 GMT-05:00 Thomas Kluyver : > >> On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote: >> >> It looks like a lot of trouble for a feature that is designed to solve a >> problem for a very thin intersection of the Venn diagram of people who: >> >> a. wants to quickly iterate while hacking a package >> >> b. doesn't want to bother learning how to trigger the incremental build >> system of that package or how to install it in editable mode. >> >> For better or worse, editable installs exist and don't need to be defined >> by PEP 517 just to handle this case. I'm not convinced it's worth the >> additional complexity of the spec just for that. >> >> >> This is a good point. Given that developers can use the build tool >> directly for development purposes, rather than going through the PEP 517 >> interface, we don't need to prioritise that use case for the spec. >> >> So for the time being, I'd prefer to say that the build_wheel() hook >> should not create or modify files in the source directory. Backends may >> keep their own caches elsewhere to speed up repeated builds, but any more >> provisions for developer use cases should be left for a future PEP. And >> developers can invoke enscons/flit/... directly in the meantime. >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Aug 24 18:30:29 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 24 Aug 2017 23:30:29 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503601253.3215112.1084035856.7F9F1FA5@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> <1503601253.3215112.1084035856.7F9F1FA5@webmail.messagingengine.com> Message-ID: On 24 August 2017 at 20:00, Thomas Kluyver wrote: > So for the time being, I'd prefer to say that the build_wheel() hook should > not create or modify files in the source directory. Backends may keep their > own caches elsewhere to speed up repeated builds, but any more provisions > for developer use cases should be left for a future PEP. And developers can > invoke enscons/flit/... directly in the meantime. That means a PEP 517 compatible interface to setuptools is more complex, as it would *have* to specify a temporary build directory. And I'm not sure whether there are any *other* files it dumps in the source directory (egg info comes to mind, and PKG-INFO) and if so whether it's easy to get setuptools to place them elsewhere. That's fine, as far as I'm concerned, but it should be made explicit, IMO. I'm not yet sure who (if anyone) is proposing to write a PEP 517 wrapper for setuptools (or if we'll need to special-case setuptools in pip for "legacy" support) but we should make it as easy as possible for anyone who does to get the details right, as it's likely to be a very widely used backend. It would be a lot easier to handle the edge cases here if there were someone writing a setuptools wrapper. Any takers? (I'm not sure I can make the time to do so, and I *am* sure that if I did, I don't have the time to support and maintain it going forward...) Paul From xoviat at gmail.com Thu Aug 24 18:33:53 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 17:33:53 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> <1503601253.3215112.1084035856.7F9F1FA5@webmail.messagingengine.com> Message-ID: I'm handling the setuptools backend. I will make sure that it complies with the clean directory. On Aug 24, 2017 5:30 PM, "Paul Moore" wrote: > On 24 August 2017 at 20:00, Thomas Kluyver wrote: > > So for the time being, I'd prefer to say that the build_wheel() hook > should > > not create or modify files in the source directory. Backends may keep > their > > own caches elsewhere to speed up repeated builds, but any more provisions > > for developer use cases should be left for a future PEP. And developers > can > > invoke enscons/flit/... directly in the meantime. > > That means a PEP 517 compatible interface to setuptools is more > complex, as it would *have* to specify a temporary build directory. > And I'm not sure whether there are any *other* files it dumps in the > source directory (egg info comes to mind, and PKG-INFO) and if so > whether it's easy to get setuptools to place them elsewhere. That's > fine, as far as I'm concerned, but it should be made explicit, IMO. > I'm not yet sure who (if anyone) is proposing to write a PEP 517 > wrapper for setuptools (or if we'll need to special-case setuptools in > pip for "legacy" support) but we should make it as easy as possible > for anyone who does to get the details right, as it's likely to be a > very widely used backend. > > It would be a lot easier to handle the edge cases here if there were > someone writing a setuptools wrapper. Any takers? (I'm not sure I can > make the time to do so, and I *am* sure that if I did, I don't have > the time to support and maintain it going forward...) > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 18:50:35 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 17:50:35 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <1503592449.2895275.1083884408.2E52DF4A@webmail.messagingengine.com> <1503598402.2187660.1083986024.44F4254F@webmail.messagingengine.com> <1503601253.3215112.1084035856.7F9F1FA5@webmail.messagingengine.com> Message-ID: Actually I submitted a compliant PR 3 months ago when the PEP was first marked as accepted. Then it was marked as draft and I thought that I would just wait for distutils-sig to handle it. And then I came over here to find what was going on. On Aug 24, 2017 5:33 PM, "xoviat" wrote: > I'm handling the setuptools backend. I will make sure that it complies > with the clean directory. > > On Aug 24, 2017 5:30 PM, "Paul Moore" wrote: > >> On 24 August 2017 at 20:00, Thomas Kluyver wrote: >> > So for the time being, I'd prefer to say that the build_wheel() hook >> should >> > not create or modify files in the source directory. Backends may keep >> their >> > own caches elsewhere to speed up repeated builds, but any more >> provisions >> > for developer use cases should be left for a future PEP. And developers >> can >> > invoke enscons/flit/... directly in the meantime. >> >> That means a PEP 517 compatible interface to setuptools is more >> complex, as it would *have* to specify a temporary build directory. >> And I'm not sure whether there are any *other* files it dumps in the >> source directory (egg info comes to mind, and PKG-INFO) and if so >> whether it's easy to get setuptools to place them elsewhere. That's >> fine, as far as I'm concerned, but it should be made explicit, IMO. >> I'm not yet sure who (if anyone) is proposing to write a PEP 517 >> wrapper for setuptools (or if we'll need to special-case setuptools in >> pip for "legacy" support) but we should make it as easy as possible >> for anyone who does to get the details right, as it's likely to be a >> very widely used backend. >> >> It would be a lot easier to handle the edge cases here if there were >> someone writing a setuptools wrapper. Any takers? (I'm not sure I can >> make the time to do so, and I *am* sure that if I did, I don't have >> the time to support and maintain it going forward...) >> >> Paul >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Aug 24 20:06:44 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 25 Aug 2017 12:06:44 +1200 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> Message-ID: <599F6A14.9010903@canterbury.ac.nz> Paul Moore wrote: > 4. The point of all this is that the definition of build_directory > says "If build_directory is None, the backend may do an 'in place' > build which modifies the source directory and may produce different > results from those that would be obtained by exporting an sdist > first". Is it really necessary for pip to specify exactly *where* to do an out-of-place build? Why should it care, as long as the end result is a clean wheel? -- Greg From dholth at gmail.com Thu Aug 24 20:08:58 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 25 Aug 2017 00:08:58 +0000 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <599F6A14.9010903@canterbury.ac.nz> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: It would be simpler if the only requirement was to produce a correct wheel. On Thu, Aug 24, 2017, 20:07 Greg Ewing wrote: > Paul Moore wrote: > > 4. The point of all this is that the definition of build_directory > > says "If build_directory is None, the backend may do an 'in place' > > build which modifies the source directory and may produce different > > results from those that would be obtained by exporting an sdist > > first". > > Is it really necessary for pip to specify exactly *where* to do > an out-of-place build? Why should it care, as long as the end > result is a clean wheel? > > -- > Greg > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 24 20:11:36 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 19:11:36 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <599F6A14.9010903@canterbury.ac.nz> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: The current proposal is to remove build_dir and leave it up to the backend. On Aug 24, 2017 7:07 PM, "Greg Ewing" wrote: > Paul Moore wrote: > >> 4. The point of all this is that the definition of build_directory >> says "If build_directory is None, the backend may do an 'in place' >> build which modifies the source directory and may produce different >> results from those that would be obtained by exporting an sdist >> first". >> > > Is it really necessary for pip to specify exactly *where* to do > an out-of-place build? Why should it care, as long as the end > result is a clean wheel? > > -- > Greg > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Aug 24 20:29:06 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Aug 2017 17:29:06 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Message-ID: On Thu, Aug 24, 2017 at 7:52 AM, Nick Coghlan wrote: >>> - I don't understand how out-of-tree builds and prepare_build_metadata >>> are supposed to interact. > > They don't, since the backend should only implement > prepare_build_metadata if it can generate the metadata without > actually triggering a full build of all the binary artifacts. The reason I had this on the list is that in many build systems, these two steps are intimately linked. E.g. in autoconf, "figure out which dependencies I'm going to use" (aka prepare_build_metadata) and "set up an out-of-tree build directory" are the same operation. Alternatively "sniff the system and config files etc. to figure out what what dependencies we're going to use" could be just another build target -- but then it still requires that the build directory be set up first. -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Thu Aug 24 20:32:43 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 19:32:43 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Message-ID: Nathaniel: Are you okay with what has been proposed? On Aug 24, 2017 7:29 PM, "Nathaniel Smith" wrote: > On Thu, Aug 24, 2017 at 7:52 AM, Nick Coghlan wrote: > >>> - I don't understand how out-of-tree builds and prepare_build_metadata > >>> are supposed to interact. > > > > They don't, since the backend should only implement > > prepare_build_metadata if it can generate the metadata without > > actually triggering a full build of all the binary artifacts. > > The reason I had this on the list is that in many build systems, these > two steps are intimately linked. E.g. in autoconf, "figure out which > dependencies I'm going to use" (aka prepare_build_metadata) and "set > up an out-of-tree build directory" are the same operation. > Alternatively "sniff the system and config files etc. to figure out > what what dependencies we're going to use" could be just another build > target -- but then it still requires that the build directory be set > up first. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Aug 24 20:35:57 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Aug 2017 17:35:57 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Message-ID: On Thu, Aug 24, 2017 at 6:11 AM, Thomas Kluyver wrote: > Nathaniel seems to be busy with other things at the moment, so I hope he > won't mind me passing on this list of things he'd like to resolve with > the draft PEP. I'll quote his comments and put my responses inline. More like taking a break for mental health reasons, really, but I've been meaning to get back to it -- thanks for the nudge and I don't mind your posting it at all. -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Thu Aug 24 22:13:21 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 21:13:21 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Message-ID: In case anyone is interested, here is a mostly correct implementation of PEP 517 based on the behavior discussed here: https://github.com/pypa/setuptools/pull/1039/files#diff-522bd9826e33902f7f58bd003c6a370c As I said, there are a few items to be worked on still, but I really don't think that it's too complicated. 2017-08-24 19:35 GMT-05:00 Nathaniel Smith : > On Thu, Aug 24, 2017 at 6:11 AM, Thomas Kluyver > wrote: > > Nathaniel seems to be busy with other things at the moment, so I hope he > > won't mind me passing on this list of things he'd like to resolve with > > the draft PEP. I'll quote his comments and put my responses inline. > > More like taking a break for mental health reasons, really, but I've > been meaning to get back to it -- thanks for the nudge and I don't > mind your posting it at all. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Aug 24 23:13:26 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Aug 2017 13:13:26 +1000 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: On 25 August 2017 at 10:08, Daniel Holth wrote: > It would be simpler if the only requirement was to produce a correct wheel. +1 To be clear on my current position: * due to the introduction of NotImplementedError, I'm now OK with explicitly out-of-tree builds being deferred to a later iteration of the API * I'm *not* OK with banning in-tree builds in the spec, since that's both unnecessary and unenforceable If a frontend really wants to ensure sdist/wheel consistency, then the way for it to do so is to build the sdist from the source tree, and then build the wheel from the sdist. Building a wheel directly from the source tree should ideally give the same results as building the sdist first, but it's explicitly OK for there to be cases where they end up being different (such as when build_sdist would raise NotImplementedError). As far as build_directory goes, the spec should just leave the question of artifact caching (or the lack thereof) up to the backend implementations for now - if/when we decide to add explicit artifact caching support to the API specification we can make a more informed decision at a future date based on what the initial generation of backends are actually doing. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From xoviat at gmail.com Fri Aug 25 00:17:53 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 23:17:53 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: I'm *not* OK with banning in-tree builds in the spec, since that's both unnecessary and unenforceable Well then either we can trust the backend or we cannot. If we can, then this is both necessary and enforceable. If not, then we're back to pip copying files. You can't make and argument that it's okay to trust build_sdist but not build_wheel. On Aug 24, 2017 10:14 PM, "Nick Coghlan" wrote: > On 25 August 2017 at 10:08, Daniel Holth wrote: > > It would be simpler if the only requirement was to produce a correct > wheel. > > +1 > > To be clear on my current position: > > * due to the introduction of NotImplementedError, I'm now OK with > explicitly out-of-tree builds being deferred to a later iteration of > the API > * I'm *not* OK with banning in-tree builds in the spec, since that's > both unnecessary and unenforceable > > If a frontend really wants to ensure sdist/wheel consistency, then the > way for it to do so is to build the sdist from the source tree, and > then build the wheel from the sdist. Building a wheel directly from > the source tree should ideally give the same results as building the > sdist first, but it's explicitly OK for there to be cases where they > end up being different (such as when build_sdist would raise > NotImplementedError). > > As far as build_directory goes, the spec should just leave the > question of artifact caching (or the lack thereof) up to the backend > implementations for now - if/when we decide to add explicit artifact > caching support to the API specification we can make a more informed > decision at a future date based on what the initial generation of > backends are actually doing. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Aug 25 00:24:14 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Aug 2017 21:24:14 -0700 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: On Thu, Aug 24, 2017 at 9:17 PM, xoviat wrote: > > I'm *not* OK with banning in-tree builds in the spec, since that's > > both unnecessary and unenforceable > > Well then either we can trust the backend or we cannot. If we can, then this > is both necessary and enforceable. If not, then we're back to pip copying > files. You can't make and argument that it's okay to trust build_sdist but > not build_wheel. I think at this point everyone has made their peace with the pip developers' decision that they want to keep copying files -- at least for now -- and that's just how it's going to be. This email has a more detailed discussion of the options, their "threat model", and the tradeoffs: https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html I can see an argument for adding language saying that build_sdist SHOULD avoid modifying the source tree if possible, and MAY write scratch files to the sdist_directory. -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Fri Aug 25 00:33:50 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 23:33:50 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: Just this morning, Paul said the following: That step's the problem. If the frontend does that it can potentially be copying a lot of unneeded stuff (VCS history, for example). We tried that with pip and it was a major issue. That problem is the *whole point* of all the discussions about the various proposals that ended up with build_tree. I took that to mean that we were trusting the backend to do the right thing. And most people agreed with that. I don't personally care but there does seem to be some miscommunication here. On Aug 24, 2017 11:24 PM, "Nathaniel Smith" wrote: > On Thu, Aug 24, 2017 at 9:17 PM, xoviat wrote: > > > I'm *not* OK with banning in-tree builds in the spec, since that's > > > both unnecessary and unenforceable > > > > Well then either we can trust the backend or we cannot. If we can, then > this > > is both necessary and enforceable. If not, then we're back to pip copying > > files. You can't make and argument that it's okay to trust build_sdist > but > > not build_wheel. > > I think at this point everyone has made their peace with the pip > developers' decision that they want to keep copying files -- at least > for now -- and that's just how it's going to be. This email has a more > detailed discussion of the options, their "threat model", and the > tradeoffs: > > https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html > > I can see an argument for adding language saying that build_sdist > SHOULD avoid modifying the source tree if possible, and MAY write > scratch files to the sdist_directory. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 00:52:38 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 24 Aug 2017 23:52:38 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: Actually after reading the email that Nathaniel referenced, I disagree with Nick's position on in-tree builds. It's absolutely necessary because the tree may be read only and it's enforceable by pip through a simple recursive list. On Aug 24, 2017 11:33 PM, "xoviat" wrote: > Just this morning, Paul said the following: > > That step's the problem. If the frontend does that it can potentially > be copying a lot of unneeded stuff (VCS history, for example). We > tried that with pip and it was a major issue. That problem is the > *whole point* of all the discussions about the various proposals that > ended up with build_tree. > > I took that to mean that we were trusting the backend to do the right > thing. And most people agreed with that. I don't personally care but there > does seem to be some miscommunication here. > > On Aug 24, 2017 11:24 PM, "Nathaniel Smith" wrote: > >> On Thu, Aug 24, 2017 at 9:17 PM, xoviat wrote: >> > > I'm *not* OK with banning in-tree builds in the spec, since that's >> > > both unnecessary and unenforceable >> > >> > Well then either we can trust the backend or we cannot. If we can, then >> this >> > is both necessary and enforceable. If not, then we're back to pip >> copying >> > files. You can't make and argument that it's okay to trust build_sdist >> but >> > not build_wheel. >> >> I think at this point everyone has made their peace with the pip >> developers' decision that they want to keep copying files -- at least >> for now -- and that's just how it's going to be. This email has a more >> detailed discussion of the options, their "threat model", and the >> tradeoffs: >> >> https://mail.python.org/pipermail/distutils-sig/2017-July/031020.html >> >> I can see an argument for adding language saying that build_sdist >> SHOULD avoid modifying the source tree if possible, and MAY write >> scratch files to the sdist_directory. >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 01:18:16 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 00:18:16 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: And one more thing: it would probably be prescient to require a get_backend_info hook that returns the name and url of the backend so that pip can direct people where to file bugs when there are problems. On Aug 24, 2017 11:52 PM, "xoviat" wrote: > Actually after reading the email that Nathaniel referenced, I disagree > with Nick's position on in-tree builds. It's absolutely necessary because > the tree may be read only and it's enforceable by pip through a simple > recursive list. > > On Aug 24, 2017 11:33 PM, "xoviat" wrote: > >> Just this morning, Paul said the following: >> >> That step's the problem. If the frontend does that it can potentially >> be copying a lot of unneeded stuff (VCS history, for example). We >> tried that with pip and it was a major issue. That problem is the >> *whole point* of all the discussions about the various proposals that >> ended up with build_tree. >> >> I took that to mean that we were trusting the backend to do the right >> thing. And most people agreed with that. I don't personally care but there >> does seem to be some miscommunication here. >> >> On Aug 24, 2017 11:24 PM, "Nathaniel Smith" wrote: >> >>> On Thu, Aug 24, 2017 at 9:17 PM, xoviat wrote: >>> > > I'm *not* OK with banning in-tree builds in the spec, since that's >>> > > both unnecessary and unenforceable >>> > >>> > Well then either we can trust the backend or we cannot. If we can, >>> then this >>> > is both necessary and enforceable. If not, then we're back to pip >>> copying >>> > files. You can't make and argument that it's okay to trust build_sdist >>> but >>> > not build_wheel. >>> >>> I think at this point everyone has made their peace with the pip >>> developers' decision that they want to keep copying files -- at least >>> for now -- and that's just how it's going to be. This email has a more >>> detailed discussion of the options, their "threat model", and the >>> tradeoffs: >>> >>> https://mail.python.org/pipermail/distutils-sig/2017-July/03 >>> 1020.html >>> >>> I can see an argument for adding language saying that build_sdist >>> SHOULD avoid modifying the source tree if possible, and MAY write >>> scratch files to the sdist_directory. >>> >>> -n >>> >>> -- >>> Nathaniel J. Smith -- https://vorpus.org >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Aug 25 01:54:09 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Aug 2017 15:54:09 +1000 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: On 25 August 2017 at 14:33, xoviat wrote: > Just this morning, Paul said the following: > > That step's the problem. If the frontend does that it can potentially > be copying a lot of unneeded stuff (VCS history, for example). We > tried that with pip and it was a major issue. That problem is the > *whole point* of all the discussions about the various proposals that > ended up with build_tree. > > I took that to mean that we were trusting the backend to do the right thing. > And most people agreed with that. I don't personally care but there does > seem to be some miscommunication here. Folks are wanting the "right thing" to be defined in the spec, but that's the wrong place for it, since "the right thing" will be use case dependent. So I'm now saying "stop trying to define what the right thing is (beyond producing standards compliant artifacts), and instead just define a usable thing, run with it, and let consensus on 'the right thing' emerge over time". Donald and Paul originally didn't like that notion, but then Paul realised that pip had never actually tried implementing the "always build an sdist first" approach. That means pip's notion of "the right thing" can be adequately captured as "source tree -> sdist -> wheel" producing the same results as "source tree -> wheel", and pip can strongly encourage that behaviour in backend implementations by always trying build_sdist first, and only falling back to calling build_wheel directly if build_sdist raises NotImplementedError. Particular *frontends* (including pip) may further refine their definition of the "right thing" to categorically disallow source tree changes, but that's something they can do over time based on the bug reports they actually receive, rather than needing to be mandated a priori in the interface specification. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From xoviat at gmail.com Fri Aug 25 01:59:06 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 00:59:06 -0500 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: Okay, but I would like to hear Paul's response to that first. On Aug 25, 2017 12:54 AM, "Nick Coghlan" wrote: > On 25 August 2017 at 14:33, xoviat wrote: > > Just this morning, Paul said the following: > > > > That step's the problem. If the frontend does that it can potentially > > be copying a lot of unneeded stuff (VCS history, for example). We > > tried that with pip and it was a major issue. That problem is the > > *whole point* of all the discussions about the various proposals that > > ended up with build_tree. > > > > I took that to mean that we were trusting the backend to do the right > thing. > > And most people agreed with that. I don't personally care but there does > > seem to be some miscommunication here. > > Folks are wanting the "right thing" to be defined in the spec, but > that's the wrong place for it, since "the right thing" will be use > case dependent. So I'm now saying "stop trying to define what the > right thing is (beyond producing standards compliant artifacts), and > instead just define a usable thing, run with it, and let consensus on > 'the right thing' emerge over time". > > Donald and Paul originally didn't like that notion, but then Paul > realised that pip had never actually tried implementing the "always > build an sdist first" approach. That means pip's notion of "the right > thing" can be adequately captured as "source tree -> sdist -> wheel" > producing the same results as "source tree -> wheel", and pip can > strongly encourage that behaviour in backend implementations by always > trying build_sdist first, and only falling back to calling build_wheel > directly if build_sdist raises NotImplementedError. > > Particular *frontends* (including pip) may further refine their > definition of the "right thing" to categorically disallow source tree > changes, but that's something they can do over time based on the bug > reports they actually receive, rather than needing to be mandated a > priori in the interface specification. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Aug 25 03:47:04 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Aug 2017 08:47:04 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: On 25 August 2017 at 06:54, Nick Coghlan wrote: > On 25 August 2017 at 14:33, xoviat wrote: >> Just this morning, Paul said the following: >> >> That step's the problem. If the frontend does that it can potentially >> be copying a lot of unneeded stuff (VCS history, for example). We >> tried that with pip and it was a major issue. That problem is the >> *whole point* of all the discussions about the various proposals that >> ended up with build_tree. >> >> I took that to mean that we were trusting the backend to do the right thing. >> And most people agreed with that. I don't personally care but there does >> seem to be some miscommunication here. > > Folks are wanting the "right thing" to be defined in the spec, but > that's the wrong place for it, since "the right thing" will be use > case dependent. So I'm now saying "stop trying to define what the > right thing is (beyond producing standards compliant artifacts), and > instead just define a usable thing, run with it, and let consensus on > 'the right thing' emerge over time". I do believe that the PEP is the right place to make the point that *users* (not just front ends) have the right to expect that the direct wheel and sdist->wheel routes will produce the same results. Originally, I would have expected this to be self-evident, but with all the debates around incremental builds and similar, I now feel that it's necessary to clarify that this is the expected behaviour. Too many tools (not just pip, tox is another one) depend on the equivalence for us to leave it open to interpretation. If anyone feels that it's *not* reasonable for users to expect "build a wheel" and "build a sdist and build a wheel from that" to produce the same results for a given source tree (assuming the backend supports both building wheels and sdists from the given source tree) then they need to speak up with a *specific* example of a use case, so we can include it in the PEP as a motivating example, and use it in future debates. At the moment too much discussion is getting bogged down with comments like "I think someone said that XXX needs this..." I'm *not* interested in trying to tie down "the correct behaviour" any more tightly than this. > Donald and Paul originally didn't like that notion, but then Paul > realised that pip had never actually tried implementing the "always > build an sdist first" approach. That means pip's notion of "the right > thing" can be adequately captured as "source tree -> sdist -> wheel" > producing the same results as "source tree -> wheel", and pip can > strongly encourage that behaviour in backend implementations by always > trying build_sdist first, and only falling back to calling build_wheel > directly if build_sdist raises NotImplementedError. Precisely. Does any tool or participant in this discussion *not* see this as the minimal definition of correct behaviour, though? > Particular *frontends* (including pip) may further refine their > definition of the "right thing" to categorically disallow source tree > changes, but that's something they can do over time based on the bug > reports they actually receive, rather than needing to be mandated a > priori in the interface specification. I've never cared about whether the backend modifies the source tree, only that the direct-wheel / sdist->wheel equivalence holds. Paul From thomas at kluyver.me.uk Fri Aug 25 04:38:54 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 09:38:54 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: <1503650334.1367352.1084595560.165CEB69@webmail.messagingengine.com> On Fri, Aug 25, 2017, at 08:47 AM, Paul Moore wrote: > I've never cared about whether the backend modifies the source tree, > only that the direct-wheel / sdist->wheel equivalence holds. If a user types 'pip install .' in a directory where they don't have write permissions, what should happen? 1. The backend is capable of building without writing in that directory. 2. The frontend (pip) notices and pre-emptively copies to a writable tmp directory. 3. The user accepts that the operation may fail, depending on the backend. I'd go for option 1 or 3. ---- Here's a proposed update to the build_wheel section of the PEP: build_wheel(wheel_directory, config_settings=None, metadata_directory=None): ... Must build a .whl file, and place it in the specified wheel_directory. It must return the basename (not the full path) of the .whl file it creates, as a unicode string. If the build frontend has previously called prepare_metadata_for_build_wheel and depends on the wheel resulting from this call to have metadata matching this earlier call, then it should provide the path to the created .dist-info directory as the metadata_directory argument. If this argument is provided, then build_wheel MUST produce a wheel with identical metadata. The directory passed in by the build frontend MUST be identical to the directory created by prepare_metadata_for_build_wheel, including any unrecognized files it created. Backends which do not provide the prepare_metadata_for_build_wheel hook may either silently ignore the metadata_directory parameter to build_wheel, or else raise an exception when it is set to anything other than None. Wheels built by this hook should, as far as possible, be equivalent to first building an sdist and then building a wheel from that sdist. Frontends which wish to ensure this consistency may call build_sdist (see below) first, and then call build_wheel in the unpacked sdist. But if build_sdist fails with NotImplementedError, they will fall back to calling build_wheel in the source directory, and rely on the backend to produce an equivalent wheel. The source directory may be read-only. Backends should therefore be prepared to build without creating or modifying any files in the source directory, but they may opt not to handle this case, in which case failures will be visible to the user. Frontends are not responsible for any special handling of read-only source directories. The backend may store intermediate artifacts in cache locations or temporary directories. The presence or absence of any caches should not make a material difference to the final result of the build. From guettliml at thomas-guettler.de Fri Aug 25 04:36:21 2017 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 25 Aug 2017 10:36:21 +0200 Subject: [Distutils] Conditionless setup.py Message-ID: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> The setup.py of numpy looks big and complicated: https://github.com/numpy/numpy/blob/master/setup.py I guess there are even more complicated setup.py files. Do you think it will be possible in the future to make setup.py conditionless (no "if", no "else", no function calls)? I mean that the setuptools/distutils/whatever libraries will handle all needed conditions? This would mean setup.py would be a pure data structure which could be encoded in json or yaml. What do you think? Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From pradyunsg at gmail.com Fri Aug 25 04:45:00 2017 From: pradyunsg at gmail.com (Pradyun Gedam) Date: Fri, 25 Aug 2017 08:45:00 +0000 Subject: [Distutils] Conditionless setup.py In-Reply-To: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> Message-ID: Hi Thomas! Have you seen Pipfile? It's something along those lines - a declarative specification of project data. :) github.com/pypa/pipfile Regards, Pradyun Gedam On Fri, Aug 25, 2017, 14:12 Thomas G?ttler wrote: > The setup.py of numpy looks big and complicated: > https://github.com/numpy/numpy/blob/master/setup.py > > I guess there are even more complicated setup.py files. > > Do you think it will be possible in the future to make setup.py > conditionless (no "if", no "else", no function calls)? > > I mean that the setuptools/distutils/whatever libraries will handle all > needed conditions? > > This would mean setup.py would be a pure data structure which could be > encoded in json or yaml. > > What do you think? > > Regards, > Thomas G?ttler > > > -- > Thomas Guettler http://www.thomas-guettler.de/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Aug 25 05:45:17 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Aug 2017 10:45:17 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503650334.1367352.1084595560.165CEB69@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> <1503650334.1367352.1084595560.165CEB69@webmail.messagingengine.com> Message-ID: On 25 August 2017 at 09:38, Thomas Kluyver wrote: > On Fri, Aug 25, 2017, at 08:47 AM, Paul Moore wrote: >> I've never cared about whether the backend modifies the source tree, >> only that the direct-wheel / sdist->wheel equivalence holds. > > If a user types 'pip install .' in a directory where they don't have > write permissions, what should happen? > > 1. The backend is capable of building without writing in that directory. > 2. The frontend (pip) notices and pre-emptively copies to a writable tmp > directory. > 3. The user accepts that the operation may fail, depending on the > backend. > > I'd go for option 1 or 3. I'd go for option (1). I don't think we want to get into (2), with the frontend trying to spot possible issues. I think (3) is always a possibility, but users have a right to expect that backends will do whatever they can to avoid it. I'd *also* expect that the wheel built (and hence what gets installed) would be the same as what you'd get if the user added write permissions and then ran the command. That's part of what I mean by direct wheel and sdist->wheel being equivalent. I'd completely support any user who complained that getting different results in those 2 cases is a bug in the backend. But as Nick pointed out, this is probably more detail than we need in the PEP. > Here's a proposed update to the build_wheel section of the PEP: (Apologies if I miss a change here, I don't have the original copy right now to compare against). > build_wheel(wheel_directory, config_settings=None, > metadata_directory=None): > ... > > Must build a .whl file, and place it in the specified wheel_directory. > It must return the basename (not the full path) of the .whl file it > creates, as a unicode string. > > If the build frontend has previously called > prepare_metadata_for_build_wheel and depends on the wheel resulting from > this call to have metadata matching this earlier call, then it should > provide the path to the created .dist-info directory as the > metadata_directory argument. If this argument is provided, then > build_wheel MUST produce a wheel with identical metadata. The directory > passed in by the build frontend MUST be identical to the directory > created by prepare_metadata_for_build_wheel, including any unrecognized > files it created. > > Backends which do not provide the prepare_metadata_for_build_wheel hook > may either silently ignore the metadata_directory parameter to > build_wheel, or else raise an exception when it is set to anything other > than None. > > Wheels built by this hook should, as far as possible, be equivalent to > first building an sdist and then building a wheel from that sdist. This is the point that I care about. > Frontends which wish to ensure this consistency may call build_sdist > (see below) first, and then call build_wheel in the unpacked sdist. But > if build_sdist fails with NotImplementedError, they will fall back to > calling build_wheel in the source directory, and rely on the backend to > produce an equivalent wheel. This seems to contradict the previous point - I'd rephrase it as "Frontends which want to protect against backends violating this expectation may call build_sdist..." and at that point it starts to seem like excessive paranoia on the part of frontends. I repeat my earlier comment here - do we have any actual cases where we expect in-place wheel builds to *not* enforce the equivalence? This was originally a huge debate issue because of all the comments about development builds and incremental compilation, and caching. Are the people who insisted that backends be allowed to carry build artifacts between compiles now accepting that they have to ensure build consistency? Or have they just dropped out of the discussion? I'm pretty sure Nathaniel was one of those people, but I'm equally sure he's fed up of debating theoretical possibilities. I suspect he still considers my assumptions fairly naive, though, as I have no experience of the sort of complexities involved in building the scientific stack... Either way, pip *can* go sdist->wheel, but as things are worded here, I don't see a real reason to bother. I'd be more inclined to push any bug reports about getting different results to the backend, quoting the PEP's requirement for build equivalence, and only implement sdist->wheel in pip if the backends pushed back (at which point, we'd need to reopen the debate on what the statement in the PEP about equivalence actually meant... ) > The source directory may be read-only. Backends should therefore be > prepared to build without creating or modifying any files in the source > directory, but they may opt not to handle this case, in which case > failures will be visible to the user. Frontends are not responsible for > any special handling of read-only source directories. This is a useful addition, particularly as it's a general point, not tied to any specific backend implementation. Personally, I'd expect a backend to do its best to avoid failing - for example by copying the files to a temporary location and doing a build there - but that's reasonably considered a backend quality of implementation issue. > The backend may store intermediate artifacts in cache locations or > temporary directories. The presence or absence of any caches should not > make a material difference to the final result of the build. I'm fine with this point. I'd also be fine with noting that such cache locations can be stored in the source tree if the backend wishes to do so (in which case, the sdist->wheel route is effectively a way of deleting any such caches). But that's probably unnecessary. Overall, this looks good to me. Paul From p.f.moore at gmail.com Fri Aug 25 05:50:11 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Aug 2017 10:50:11 +0100 Subject: [Distutils] Conditionless setup.py In-Reply-To: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> Message-ID: On 25 August 2017 at 09:36, Thomas G?ttler wrote: > The setup.py of numpy looks big and complicated: > https://github.com/numpy/numpy/blob/master/setup.py > > I guess there are even more complicated setup.py files. > > Do you think it will be possible in the future to make setup.py > conditionless (no "if", no "else", no function calls)? > > I mean that the setuptools/distutils/whatever libraries will handle all > needed conditions? > > This would mean setup.py would be a pure data structure which could be > encoded in json or yaml. > > What do you think? The goal (under PEP 517, setuptools is merely one possible "backend") is that some backends will allow purely declarative descriptions of build requirements. Flit already does this. Whether setuptools will ever switch to a purely declarative form I don't know (but I doubt it - it's more likely that someone will write an alternative backend to replace some or all of setuptools, as flit did). And whether projects with requirements as complex as numpy and similar will be able to manage with a purely declarative build description is even more questionable, I suspect. But certainly, once PEP 517 is implemented and as flit gains popularity, I fully expect more and more projects to use a static data structure for their metadata (flit.ini, specifically). Paul From thomas at kluyver.me.uk Fri Aug 25 06:16:06 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 11:16:06 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> <1503650334.1367352.1084595560.165CEB69@webmail.messagingengine.com> Message-ID: <1503656166.1646533.1084673824.6387ABBC@webmail.messagingengine.com> Thanks Paul, On Fri, Aug 25, 2017, at 10:45 AM, Paul Moore wrote: > I'd *also* expect that the wheel built (and hence what gets installed) > would be the same as what you'd get if the user added write > permissions and then ran the command. That's part of what I mean by > direct wheel and sdist->wheel being equivalent. I'd completely support > any user who complained that getting different results in those 2 > cases is a bug in the backend. But as Nick pointed out, this is > probably more detail than we need in the PEP. Agreed. > > Frontends which wish to ensure this consistency may call build_sdist > > (see below) first, and then call build_wheel in the unpacked sdist. But > > if build_sdist fails with NotImplementedError, they will fall back to > > calling build_wheel in the source directory, and rely on the backend to > > produce an equivalent wheel. > > This seems to contradict the previous point - I'd rephrase it as > "Frontends which want to protect against backends violating this > expectation may call build_sdist..." and at that point it starts to > seem like excessive paranoia on the part of frontends. Your wording is fine by me - I was struggling to find the right phrasing there. > I repeat my earlier comment here - do we have any actual cases where > we expect in-place wheel builds to *not* enforce the equivalence? Flit cannot strictly enforce it if it can't build an sdist: with the way I've chosen to do sdists, it's not possible to know what would be in an sdist in the case where we can't build one. The 'on-label' use case for flit is packages with no built or generated parts. All of the modules and data files inside the package should be checked in to version control, which means they would be included in an sdist. I don't think I can technically stop people from running a separate step to generate files before building a wheel, but I'd consider that creative misuse of flit. (There is a discussion issue open - #119 - about adding support for build steps in flit. If we decide to do that, the situation will change, but that's some way off even if we do do it) Thomas From p.f.moore at gmail.com Fri Aug 25 06:45:20 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Aug 2017 11:45:20 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503656166.1646533.1084673824.6387ABBC@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> <1503650334.1367352.1084595560.165CEB69@webmail.messagingengine.com> <1503656166.1646533.1084673824.6387ABBC@webmail.messagingengine.com> Message-ID: On 25 August 2017 at 11:16, Thomas Kluyver wrote: >> I repeat my earlier comment here - do we have any actual cases where >> we expect in-place wheel builds to *not* enforce the equivalence? > > Flit cannot strictly enforce it if it can't build an sdist: with the way > I've chosen to do sdists, it's not possible to know what would be in an > sdist in the case where we can't build one. If it can't build a sdist then there's no opportunity for frontends to fall back to sdist->wheel anyway, so that's not an issue[1]. It's only if sdist->wheel and direct wheel are both possible that frontends have a choice, and I feel that we should mandate that backends ensure that it doesn't matter which route the frontend chooses. Paul [1] Excepting the case where the user modifies things to allow sdists (e.g. by installing git) and expects to get the same result, which is a quality of implementation point and I'm sure flit does its best to get that right, so I'm not worried about that case. From thomas at kluyver.me.uk Fri Aug 25 06:56:58 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 11:56:58 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> <1503650334.1367352.1084595560.165CEB69@webmail.messagingengine.com> <1503656166.1646533.1084673824.6387ABBC@webmail.messagingengine.com> Message-ID: <1503658618.1655512.1084718912.51936D7B@webmail.messagingengine.com> On Fri, Aug 25, 2017, at 11:45 AM, Paul Moore wrote: > If it can't build a sdist then there's no opportunity for frontends to > fall back to sdist->wheel anyway, so that's not an issue[1]. > > It's only if sdist->wheel and direct wheel are both possible that > frontends have a choice, and I feel that we should mandate that > backends ensure that it doesn't matter which route the frontend > chooses. Great, I think we're on the same page. I'm going to make a PR against the PEP while I still remember what page it is. ;-) From thomas at kluyver.me.uk Fri Aug 25 07:08:17 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 12:08:17 +0100 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: <1503658618.1655512.1084718912.51936D7B@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> <1503650334.1367352.1084595560.165CEB69@webmail.messagingengine.com> <1503656166.1646533.1084673824.6387ABBC@webmail.messagingengine.com> <1503658618.1655512.1084718912.51936D7B@webmail.messagingengine.com> Message-ID: <1503659297.1657385.1084727944.40028F56@webmail.messagingengine.com> On Fri, Aug 25, 2017, at 11:56 AM, Thomas Kluyver wrote: > Great, I think we're on the same page. I'm going to make a PR against > the PEP while I still remember what page it is. ;-) https://github.com/python/peps/pull/364 From fungi at yuggoth.org Fri Aug 25 07:14:42 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 25 Aug 2017 11:14:42 +0000 Subject: [Distutils] Conditionless setup.py In-Reply-To: References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> Message-ID: <20170825111442.GZ2409@yuggoth.org> On 2017-08-25 10:50:11 +0100 (+0100), Paul Moore wrote: [...] > once PEP 517 is implemented and as flit gains popularity, I fully > expect more and more projects to use a static data structure for > their metadata (flit.ini, specifically). This has also been possible for years already using either PBR or distutils2. For example, hundreds of Python packages produced by the OpenStack community use a branchless boilerplate setup.py which declares a setup_requires on the pbr package, and then everything else goes into an INI-formatted setup.cfg file (except for install_requires which are drawn from requirements.txt instead). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From ncoghlan at gmail.com Fri Aug 25 08:29:51 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Aug 2017 22:29:51 +1000 Subject: [Distutils] Fwd: Re: PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1503581281.2850341.1083663896.645E11AF@webmail.messagingengine.com> <1503584807.2863649.1083721080.35BA17E8@webmail.messagingengine.com> <599F6A14.9010903@canterbury.ac.nz> Message-ID: On 25 August 2017 at 17:47, Paul Moore wrote: > If anyone feels that it's *not* reasonable for users to expect "build > a wheel" and "build a sdist and build a wheel from that" to produce > the same results for a given source tree (assuming the backend > supports both building wheels and sdists from the given source tree) > then they need to speak up with a *specific* example of a use case, so > we can include it in the PEP as a motivating example, and use it in > future debates. Just the NotImplementedError case: running build_wheel against a source tree is permitted to succeed in cases where build_sdist raises NotImplementedError, and in these cases, the result may *not* match what you would have gotten if the necessary inputs and tools for creating the sdist were available. > At the moment too much discussion is getting bogged > down with comments like "I think someone said that XXX needs this..." > > I'm *not* interested in trying to tie down "the correct behaviour" any > more tightly than this. Cool, we're on the same page then - the specific thing that was bothering me was the request to start articulating "permitted side effects" at the PEP level, and I don't think the PEP is the right place for that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From graffatcolmingov at gmail.com Fri Aug 25 08:37:23 2017 From: graffatcolmingov at gmail.com (Ian Stapleton Cordasco) Date: Fri, 25 Aug 2017 07:37:23 -0500 Subject: [Distutils] Conditionless setup.py In-Reply-To: <20170825111442.GZ2409@yuggoth.org> References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> Message-ID: Except that OpenStack frequently rejects outside use cases as I learned as an OpenStack developer who tried to improve PBR. Sadly it will never be seen as a global solution as long as that continues On Aug 25, 2017 6:32 AM, "Jeremy Stanley" wrote: > On 2017-08-25 10:50:11 +0100 (+0100), Paul Moore wrote: > [...] > > once PEP 517 is implemented and as flit gains popularity, I fully > > expect more and more projects to use a static data structure for > > their metadata (flit.ini, specifically). > > This has also been possible for years already using either PBR or > distutils2. For example, hundreds of Python packages produced by the > OpenStack community use a branchless boilerplate setup.py which > declares a setup_requires on the pbr package, and then everything > else goes into an INI-formatted setup.cfg file (except for > install_requires which are drawn from requirements.txt instead). > -- > Jeremy Stanley > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Aug 25 08:42:00 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 25 Aug 2017 12:42:00 +0000 Subject: [Distutils] Conditionless setup.py In-Reply-To: <20170825111442.GZ2409@yuggoth.org> References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> Message-ID: It's important to realize that build is a Turing-complete activity. We write programs to build programs. Where setup.py gets in trouble is by making both the metadata and the build script the same thing, making it harder to reason about that metadata. That is why we are working towards being able to express the metadata in pyproject.toml, while letting you have any kind of build script you please. We expect that many more projects will be able to deal with this limitation compared to e.g. always having to express the whole build with static configuration of something like distutils2 or PBR. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri Aug 25 08:44:48 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 25 Aug 2017 12:44:48 +0000 Subject: [Distutils] Conditionless setup.py In-Reply-To: References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> Message-ID: <20170825124448.GA2409@yuggoth.org> On 2017-08-25 07:37:23 -0500 (-0500), Ian Stapleton Cordasco wrote: > Except that OpenStack frequently rejects outside use cases as I > learned as an OpenStack developer who tried to improve PBR. Sadly > it will never be seen as a global solution as long as that > continues Many projects frequently reject use cases regardless of their proponents, if they don't fit with the scope as identified by their maintainers. No project can be all things to all users and expect to remain manageable in the long term. I've not seen patches rejected out of hand because of who/where the use case came from, and while I'm curious to know which change(s) this was it's probably off-topic for this ML. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From p.f.moore at gmail.com Fri Aug 25 09:00:25 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Aug 2017 14:00:25 +0100 Subject: [Distutils] Conditionless setup.py In-Reply-To: <20170825111442.GZ2409@yuggoth.org> References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> Message-ID: On 25 August 2017 at 12:14, Jeremy Stanley wrote: > On 2017-08-25 10:50:11 +0100 (+0100), Paul Moore wrote: > [...] >> once PEP 517 is implemented and as flit gains popularity, I fully >> expect more and more projects to use a static data structure for >> their metadata (flit.ini, specifically). > > This has also been possible for years already using either PBR or > distutils2. For example, hundreds of Python packages produced by the > OpenStack community use a branchless boilerplate setup.py which > declares a setup_requires on the pbr package, and then everything > else goes into an INI-formatted setup.cfg file (except for > install_requires which are drawn from requirements.txt instead). Cool. I'm not that familiar with those tools, but if they enable that sort of use then that's great. I did get the impression that they were for more complex/specialised use cases, though, whereas flit (with PEP 517) is much more about simple configuration for the majority of (pure Python) projects that don't need complex behaviour. But that's mostly about target audiences than capabilities. One thought - are the PBR and/or distutils2 teams looking at providing PEP 517 support? Assuming they are, have they had a change to review the PEP to ensure it suits their needs? And if they aren't, what is it about the PEP that makes them unwilling to do so? Paul From donald at stufft.io Fri Aug 25 12:06:28 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 25 Aug 2017 12:06:28 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> Message-ID: <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> > On Aug 24, 2017, at 10:52 AM, Nick Coghlan wrote: > > Aye, I do, and it should be "raise NotImplementedError('Explanation of > the failed check')" > > Rationale: > > - Python isn't C or Go, so we indicate failures with exceptions, not > error codes (NotImplemented is an necessary performance hack for > operand coercion in tight loops, not an example to be emulated in > other APIs) > - allows the backend to provide information on what went wrong > - means an unhandled backend error results in a traceback pointing to > where the build failed, not some later point in the frontend code > - if a backend developer is sufficiently worried about accidentally > propagating NotImplementedError that they want to pretend they're not > writing Python any more, they can use this idiom: > > def public_hook_api(*args, **kwds): > try: > result, error_msg = _internal_hook_implementation(*args, **kwds) > except NotImplementedError as exc: > raise RuntimeError("Unexpected NotImplementedError") from exc > if result is NotImplemented: > raise NotImplementedError(error_msg) > return result > > That provides the backend with all the same assurances against > accidentally letting NotImplementedError escape that a return code > based public API would, without frontends even needing to be aware of > the backend developer's aversion to reporting errors as exceptions. I?m not really a fan of using NotImplementedError instead of NotImplemented. We?re not going to implement it by showing a traceback to where the NotImplementedError happened because it?s not an error case. And really that?s the important bit here, this is not an error case (as far as the API is concerned), this is just one of the possible return values that this function can produce. A front end may *choose* to make this an error case of course, but that is at a different layer than this API is operating. It?s arguably not even the correct usage of NotImplementedError, since that is (and I quote from the Python docs): "In user defined base classes, abstract methods should raise this exception when they require derived classes to override the method, or while the class is being developed to indicate that the real implementation still needs to be added.? This is not a case of some real implementation not having yet been added or some stub code getting called before it?s ready. This use case more closely resembles NotImplemented. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 12:38:33 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 11:38:33 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> Message-ID: According to the documentation, NotImplemented isn't appropriate either, as is for binary operations only. There is no one value that's taylor made for this situation, but an exception may be more appropriate as the underlying cause is probably an error. On Aug 25, 2017 11:11 AM, "Donald Stufft" wrote: > > On Aug 24, 2017, at 10:52 AM, Nick Coghlan wrote: > > Aye, I do, and it should be "raise NotImplementedError('Explanation of > the failed check')" > > Rationale: > > - Python isn't C or Go, so we indicate failures with exceptions, not > error codes (NotImplemented is an necessary performance hack for > operand coercion in tight loops, not an example to be emulated in > other APIs) > - allows the backend to provide information on what went wrong > - means an unhandled backend error results in a traceback pointing to > where the build failed, not some later point in the frontend code > - if a backend developer is sufficiently worried about accidentally > propagating NotImplementedError that they want to pretend they're not > writing Python any more, they can use this idiom: > > def public_hook_api(*args, **kwds): > try: > result, error_msg = _internal_hook_implementation(*args, > **kwds) > except NotImplementedError as exc: > raise RuntimeError("Unexpected NotImplementedError") from exc > if result is NotImplemented: > raise NotImplementedError(error_msg) > return result > > That provides the backend with all the same assurances against > accidentally letting NotImplementedError escape that a return code > based public API would, without frontends even needing to be aware of > the backend developer's aversion to reporting errors as exceptions. > > > > I?m not really a fan of using NotImplementedError instead of > NotImplemented. We?re not going to implement it by showing a traceback to > where the NotImplementedError happened because it?s not an error case. And > really that?s the important bit here, this is not an error case (as far as > the API is concerned), this is just one of the possible return values that > this function can produce. > > A front end may *choose* to make this an error case of course, but that is > at a different layer than this API is operating. > > It?s arguably not even the correct usage of NotImplementedError, since > that is (and I quote from the Python docs): "In user defined base classes, > abstract methods should raise this exception when they require derived > classes to override the method, or while the class is being developed to > indicate that the real implementation still needs to be added.? > > This is not a case of some real implementation not having yet been added > or some stub code getting called before it?s ready. This use case more > closely resembles NotImplemented. > > ? > Donald Stufft > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Aug 25 12:45:36 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 25 Aug 2017 12:45:36 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> Message-ID: <8DDF7095-AB62-4253-9466-ADB7FED18E2C@stufft.io> > On Aug 25, 2017, at 12:38 PM, xoviat wrote: > > According to the documentation, NotImplemented isn't appropriate either, as is for binary operations only. There is no one value that's taylor made for this situation, but an exception may be more appropriate as the underlying cause is probably an error. > The underlying cause is *not* an error just the same as it?s not an error for a __eq__ to not know how to test equality against a specific `other`. The underlying cause is explicitly ?I do not want to or know how to handle this case? not ?I?ve accidentally called some code that wasn?t implemented yet?. The use case is almost exactly the same as the binop use case. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 12:48:54 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 11:48:54 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <8DDF7095-AB62-4253-9466-ADB7FED18E2C@stufft.io> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <8DDF7095-AB62-4253-9466-ADB7FED18E2C@stufft.io> Message-ID: By the way, what is the status on sys.path? On Aug 25, 2017 11:45 AM, "Donald Stufft" wrote: > > On Aug 25, 2017, at 12:38 PM, xoviat wrote: > > According to the documentation, NotImplemented isn't appropriate either, > as is for binary operations only. There is no one value that's taylor made > for this situation, but an exception may be more appropriate as the > underlying cause is probably an error. > > > > The underlying cause is *not* an error just the same as it?s not an error > for a __eq__ to not know how to test equality against a specific `other`. > The underlying cause is explicitly ?I do not want to or know how to handle > this case? not ?I?ve accidentally called some code that wasn?t implemented > yet?. The use case is almost exactly the same as the binop use case. > > > ? > Donald Stufft > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Aug 25 12:49:27 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 17:49:27 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> Message-ID: <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> Can I gently ask everyone involved to consider whether the notimplemented/error discussion is verging into bikeshedding (http://bikeshed.org/)? The technical arguments I have seen so far are: - The exception can include a message - The return value can't 'bubble up' from the internals of a hook like an exception I don't think the discussion of semantics is going to go anywhere: they are both reasonable ways for the backend to reply "sorry, Dave, I can't do that". On Fri, Aug 25, 2017, at 05:38 PM, xoviat wrote: > According to the documentation, NotImplemented isn't appropriate > either, as is for binary operations only. There is no one value that's > taylor made for this situation, but an exception may be more > appropriate as the underlying cause is probably an error.> > On Aug 25, 2017 11:11 AM, "Donald Stufft" wrote: >> >>> On Aug 24, 2017, at 10:52 AM, Nick Coghlan >>> wrote:>>> >>> Aye, I do, and it should be "raise >>> NotImplementedError('Explanation of>>> the failed check')" >>> >>> Rationale: >>> >>> - Python isn't C or Go, so we indicate failures with exceptions, not>>> error codes (NotImplemented is an necessary performance hack for >>> operand coercion in tight loops, not an example to be emulated in >>> other APIs) >>> - allows the backend to provide information on what went wrong >>> - means an unhandled backend error results in a traceback >>> pointing to>>> where the build failed, not some later point in the frontend code >>> - if a backend developer is sufficiently worried about accidentally>>> propagating NotImplementedError that they want to pretend >>> they're not>>> writing Python any more, they can use this idiom: >>> >>> def public_hook_api(*args, **kwds): >>> try: >>> result, error_msg = _internal_hook_implementation(*args, >>> **kwds)>>> except NotImplementedError as exc: >>> raise RuntimeError("Unexpected NotImplementedError") from >>> exc>>> if result is NotImplemented: >>> raise NotImplementedError(error_msg) >>> return result >>> >>> That provides the backend with all the same assurances against >>> accidentally letting NotImplementedError escape that a return code >>> based public API would, without frontends even needing to be >>> aware of>>> the backend developer's aversion to reporting errors as exceptions.>> >> >> I?m not really a fan of using NotImplementedError instead of >> NotImplemented. We?re not going to implement it by showing a >> traceback to where the NotImplementedError happened because it?s not >> an error case. And really that?s the important bit here, this is not >> an error case (as far as the API is concerned), this is just one of >> the possible return values that this function can produce.>> >> A front end may *choose* to make this an error case of course, but >> that is at a different layer than this API is operating.>> >> It?s arguably not even the correct usage of NotImplementedError, >> since that is (and I quote from the Python docs): "In user defined >> base classes, abstract methods should raise this exception when they >> require derived classes to override the method, or while the class is >> being developed to indicate that the real implementation still needs >> to be added.?>> >> This is not a case of some real implementation not having yet been >> added or some stub code getting called before it?s ready. This use >> case more closely resembles NotImplemented.>> >> ? >> Donald Stufft >> >> >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > _________________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 12:53:05 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 11:53:05 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> Message-ID: We could support both options on the frontend side. It's only a tiny bit of duplication in pip. On Aug 25, 2017 11:50 AM, "Thomas Kluyver" wrote: > Can I gently ask everyone involved to consider whether the > notimplemented/error discussion is verging into bikeshedding ( > http://bikeshed.org/)? > > The technical arguments I have seen so far are: > - The exception can include a message > - The return value can't 'bubble up' from the internals of a hook like an > exception > > I don't think the discussion of semantics is going to go anywhere: they > are both reasonable ways for the backend to reply "sorry, Dave, I can't do > that". > > On Fri, Aug 25, 2017, at 05:38 PM, xoviat wrote: > > According to the documentation, NotImplemented isn't appropriate either, > as is for binary operations only. There is no one value that's taylor made > for this situation, but an exception may be more appropriate as the > underlying cause is probably an error. > > On Aug 25, 2017 11:11 AM, "Donald Stufft" wrote: > > > On Aug 24, 2017, at 10:52 AM, Nick Coghlan wrote: > > Aye, I do, and it should be "raise NotImplementedError('Explanation of > the failed check')" > > Rationale: > > - Python isn't C or Go, so we indicate failures with exceptions, not > error codes (NotImplemented is an necessary performance hack for > operand coercion in tight loops, not an example to be emulated in > other APIs) > - allows the backend to provide information on what went wrong > - means an unhandled backend error results in a traceback pointing to > where the build failed, not some later point in the frontend code > - if a backend developer is sufficiently worried about accidentally > propagating NotImplementedError that they want to pretend they're not > writing Python any more, they can use this idiom: > > def public_hook_api(*args, **kwds): > try: > result, error_msg = _internal_hook_implementation(*args, > **kwds) > except NotImplementedError as exc: > raise RuntimeError("Unexpected NotImplementedError") from exc > if result is NotImplemented: > raise NotImplementedError(error_msg) > return result > > That provides the backend with all the same assurances against > accidentally letting NotImplementedError escape that a return code > based public API would, without frontends even needing to be aware of > the backend developer's aversion to reporting errors as exceptions. > > > > I?m not really a fan of using NotImplementedError instead of > NotImplemented. We?re not going to implement it by showing a traceback to > where the NotImplementedError happened because it?s not an error case. And > really that?s the important bit here, this is not an error case (as far as > the API is concerned), this is just one of the possible return values that > this function can produce. > > A front end may *choose* to make this an error case of course, but that is > at a different layer than this API is operating. > > It?s arguably not even the correct usage of NotImplementedError, since > that is (and I quote from the Python docs): "In user defined base classes, > abstract methods should raise this exception when they require derived > classes to override the method, or while the class is being developed to > indicate that the real implementation still needs to be added.? > > This is not a case of some real implementation not having yet been added > or some stub code getting called before it?s ready. This use case more > closely resembles NotImplemented. > > ? > Donald Stufft > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > *_______________________________________________* > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Aug 25 12:57:26 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 25 Aug 2017 12:57:26 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> Message-ID: <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> > On Aug 25, 2017, at 12:49 PM, Thomas Kluyver wrote: > > Can I gently ask everyone involved to consider whether the notimplemented/error discussion is verging into bikeshedding (http://bikeshed.org/ )? > > The technical arguments I have seen so far are: > - The exception can include a message > - The return value can't 'bubble up' from the internals of a hook like an exception > > I don't think the discussion of semantics is going to go anywhere: they are both reasonable ways for the backend to reply "sorry, Dave, I can't do that". I don?t think they are both reasonable ways any more than it?s reasonable to do ``raise Return(value)`` instead of ``return value``. The semantics here are important because using exceptions for non-exceptional, non erroneous cases has *always*, in my experience, lead to weirdness [1]. [1] Like for example, StopIteration which was deemed so bad as to need to break backwards compatibility and break consistency with all other uses of exceptions just to handle the weirdness in a saner way. Unfortunately we can?t modify the Python interpreter to fix our weirdness that is going to happen. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 13:06:08 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 12:06:08 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> Message-ID: Is pip going to fall back to building a wheel directly if any other error is raised? That's what happens with setup.py install. If so, then it may be fine if unexpected exceptions bubble up. On Aug 25, 2017 11:57 AM, "Donald Stufft" wrote: > > On Aug 25, 2017, at 12:49 PM, Thomas Kluyver wrote: > > Can I gently ask everyone involved to consider whether the > notimplemented/error discussion is verging into bikeshedding ( > http://bikeshed.org/)? > > The technical arguments I have seen so far are: > - The exception can include a message > - The return value can't 'bubble up' from the internals of a hook like an > exception > > I don't think the discussion of semantics is going to go anywhere: they > are both reasonable ways for the backend to reply "sorry, Dave, I can't do > that". > > > > I don?t think they are both reasonable ways any more than it?s reasonable > to do ``raise Return(value)`` instead of ``return value``. The semantics > here are important because using exceptions for non-exceptional, non > erroneous cases has *always*, in my experience, lead to weirdness [1]. > > [1] Like for example, StopIteration which was deemed so bad as to need to > break backwards compatibility and break consistency with all other uses of > exceptions just to handle the weirdness in a saner way. Unfortunately we > can?t modify the Python interpreter to fix our weirdness that is going to > happen. > > ? > Donald Stufft > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Aug 25 13:17:45 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 25 Aug 2017 13:17:45 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> Message-ID: <084ED45B-9C2F-4EC0-BBBE-C446392CADB9@stufft.io> > On Aug 25, 2017, at 1:06 PM, xoviat wrote: > > Is pip going to fall back to building a wheel directly if any other error is raised? That's what happens with setup.py install. If so, then it may be fine if unexpected exceptions bubble up. > I doubt it. It will likely just print out the error and fail. The current ?assume any error means fallback? situation is not very good, because most errors are not recoverable and what happens is we just make things slower by trying to repeat the operation which is going to fail instead of bailing as son as we know it?s going to fail. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Aug 25 14:23:28 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Aug 2017 19:23:28 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> Message-ID: On 25 August 2017 at 18:06, xoviat wrote: > Is pip going to fall back to building a wheel directly if any other error is > raised? That's what happens with setup.py install. If so, then it may be > fine if unexpected exceptions bubble up. Given that hooks need to be called in a subprocess (see https://www.python.org/dev/peps/pep-0517/#id15, "Frontends should call each hook in a fresh subprocess, so that backends are free to change process global state") there's no "bubbling up" involved at all. The frontend code would be something along the lines of hook_stub = ''' import backend try: backend.build_sdist(...) except NotImplementedError: sys.exit(1) sys.exit(0) ''' # Or... hook_stub = ''' import backend if backend.build_sdist(...) == NotImplemented: sys.exit(1) sys.exit(0) ''' if subprocess.call([sys.executable, "-c", hook_stub]) != 0: # We didn't build a sdist There's little or no opportunity here for letting exceptions bubble up to the user, or passing complex data back to the frontend. Ultimately, it's pretty much immaterial which form of reporting is used. Paul From xoviat at gmail.com Fri Aug 25 14:33:09 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 13:33:09 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> Message-ID: The issue is about exceptions bubbling up to the interface level, not about crashing pip. For example, if some class raises notimplemented error and pip interprets that to mean it should call build_wheel. On Aug 25, 2017 1:23 PM, "Paul Moore" wrote: > On 25 August 2017 at 18:06, xoviat wrote: > > Is pip going to fall back to building a wheel directly if any other > error is > > raised? That's what happens with setup.py install. If so, then it may be > > fine if unexpected exceptions bubble up. > > Given that hooks need to be called in a subprocess (see > https://www.python.org/dev/peps/pep-0517/#id15, "Frontends should call > each hook in a fresh subprocess, so that backends are free to change > process global state") there's no "bubbling up" involved at all. The > frontend code would be something along the lines of > > hook_stub = ''' > import backend > try: > backend.build_sdist(...) > except NotImplementedError: > sys.exit(1) > sys.exit(0) > ''' > # Or... > hook_stub = ''' > import backend > if backend.build_sdist(...) == NotImplemented: > sys.exit(1) > sys.exit(0) > ''' > > if subprocess.call([sys.executable, "-c", hook_stub]) != 0: > # We didn't build a sdist > > There's little or no opportunity here for letting exceptions bubble up > to the user, or passing complex data back to the frontend. Ultimately, > it's pretty much immaterial which form of reporting is used. > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Aug 25 14:34:39 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 25 Aug 2017 14:34:39 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> Message-ID: <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> The thing being bubbled up is a backend accidentally calling an API that has yet to be implemented (an error that should be reported) being bubbled up and erroneously handled as the backend reporting it doesn't support making a sdist (not an error, son no traceback). Sent from my iPhone > On Aug 25, 2017, at 2:23 PM, Paul Moore wrote: > > There's little or no opportunity here for letting exceptions bubble up > to the user, or passing complex data back to the frontend. Ultimately, > it's pretty much immaterial which form of reporting is used. From xoviat at gmail.com Fri Aug 25 15:04:52 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 14:04:52 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> Message-ID: I agree with Nick that exceptions are the way to do things in Python and with Donald that in general, Python's use of exceptions has caused problems. But I don't think that this forum is the correct place to discuss Python conventions, and so I would ordinarily say that we should just accept that there will be problems and use the NotImplementedError, assuming that "solution" doesn't cause a firestorm that delays acceptance for a significant period of time. The reasoning behind this is that Python has in general adopted this approach (Nick is right that they would have used NotImplementedError for binary operations except for performance issues) even with its problems. However, another solution was dismissed without much thought: using the "UnsupportedOperation" exception. We could for now simply require an extra two lines in all backends: class DistutilsUnsupportedOperation(RuntimeError): pass and then put UnsupportedOperation in the "distutils" library in 3.7, with an eventual move to that. It's not pretty but there have been plenty of hacks for backwards compatibility so far (it's rare to see a project without some sort of compat.py file). 2017-08-25 13:34 GMT-05:00 Donald Stufft : > The thing being bubbled up is a backend accidentally calling an API that > has yet to be implemented (an error that should be reported) being bubbled > up and erroneously handled as the backend reporting it doesn't support > making a sdist (not an error, son no traceback). > > Sent from my iPhone > > > On Aug 25, 2017, at 2:23 PM, Paul Moore wrote: > > > > There's little or no opportunity here for letting exceptions bubble up > > to the user, or passing complex data back to the frontend. Ultimately, > > it's pretty much immaterial which form of reporting is used. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Aug 25 15:25:26 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 25 Aug 2017 15:25:26 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> Message-ID: It's silly to require every backend to write code to guard against possible issues when we can structure the API to prevent those issues. Sent from my iPhone > On Aug 25, 2017, at 3:04 PM, xoviat wrote: > > I agree with Nick that exceptions are the way to do things in Python and with Donald that in general, Python's use of exceptions has caused problems. But I don't think that this forum is the correct place to discuss Python conventions, and so I would ordinarily say that we should just accept that there will be problems and use the NotImplementedError, assuming that "solution" doesn't cause a firestorm that delays acceptance for a significant period of time. The reasoning behind this is that Python has in general adopted this approach (Nick is right that they would have used NotImplementedError for binary operations except for performance issues) even with its problems. > > However, another solution was dismissed without much thought: using the "UnsupportedOperation" exception. We could for now simply require an extra two lines in all backends: > > class DistutilsUnsupportedOperation(RuntimeError): > pass > > and then put UnsupportedOperation in the "distutils" library in 3.7, with an eventual move to that. It's not pretty but there have been plenty of hacks for backwards compatibility so far (it's rare to see a project without some sort of compat.py file). > > 2017-08-25 13:34 GMT-05:00 Donald Stufft : >> The thing being bubbled up is a backend accidentally calling an API that has yet to be implemented (an error that should be reported) being bubbled up and erroneously handled as the backend reporting it doesn't support making a sdist (not an error, son no traceback). >> >> Sent from my iPhone >> >> > On Aug 25, 2017, at 2:23 PM, Paul Moore wrote: >> > >> > There's little or no opportunity here for letting exceptions bubble up >> > to the user, or passing complex data back to the frontend. Ultimately, >> > it's pretty much immaterial which form of reporting is used. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 15:31:18 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 14:31:18 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> Message-ID: Do you mean in addition to the two lines proposed here? I envision the typical use case as attempting to do something that is required to build an sdist deep inside the backend, discovering that it cannot be done, and then raising the proposed exception. That's (other than the two lines for the exception, which can put in Python later) shorter than catching some custom exception in the hook and then returning NotImplemented as a result. If you don't use exceptions, you'll probably need to unwind the stack manually with return values. 2017-08-25 14:25 GMT-05:00 Donald Stufft : > It's silly to require every backend to write code to guard against > possible issues when we can structure the API to prevent those issues. > > Sent from my iPhone > > On Aug 25, 2017, at 3:04 PM, xoviat wrote: > > I agree with Nick that exceptions are the way to do things in Python and > with Donald that in general, Python's use of exceptions has caused > problems. But I don't think that this forum is the correct place to discuss > Python conventions, and so I would ordinarily say that we should just > accept that there will be problems and use the NotImplementedError, > assuming that "solution" doesn't cause a firestorm that delays acceptance > for a significant period of time. The reasoning behind this is that Python > has in general adopted this approach (Nick is right that they would have > used NotImplementedError for binary operations except for performance > issues) even with its problems. > > However, another solution was dismissed without much thought: using the > "UnsupportedOperation" exception. We could for now simply require an extra > two lines in all backends: > > class DistutilsUnsupportedOperation(RuntimeError): > pass > > and then put UnsupportedOperation in the "distutils" library in 3.7, with > an eventual move to that. It's not pretty but there have been plenty of > hacks for backwards compatibility so far (it's rare to see a project > without some sort of compat.py file). > > 2017-08-25 13:34 GMT-05:00 Donald Stufft : > >> The thing being bubbled up is a backend accidentally calling an API that >> has yet to be implemented (an error that should be reported) being bubbled >> up and erroneously handled as the backend reporting it doesn't support >> making a sdist (not an error, son no traceback). >> >> Sent from my iPhone >> >> > On Aug 25, 2017, at 2:23 PM, Paul Moore wrote: >> > >> > There's little or no opportunity here for letting exceptions bubble up >> > to the user, or passing complex data back to the frontend. Ultimately, >> > it's pretty much immaterial which form of reporting is used. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri Aug 25 16:00:43 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 25 Aug 2017 20:00:43 +0000 Subject: [Distutils] Conditionless setup.py In-Reply-To: References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> Message-ID: <20170825200042.GB2409@yuggoth.org> On 2017-08-25 14:00:25 +0100 (+0100), Paul Moore wrote: [...] > I'm not that familiar with those tools, but if they enable that > sort of use then that's great. I did get the impression that they > were for more complex/specialised use cases, though, whereas flit > (with PEP 517) is much more about simple configuration for the > majority of (pure Python) projects that don't need complex > behaviour. [...] Not really, no; pretty much everything in OpenStack is pure Python as well and relies on PBR. I can't personally think of any non-pure package I've seen using PBR, so it may not even be usable outside pure Python packaging for all I know (I've certainly never tried anyway). PBR basically started with the primary goals of reducing copied code by simplifying common setup.py file patterns and integrating features to help avoid additional setup_requires which couldn't easily be versioned (especially back in those days) without causing all manner of breakage when they might end up conflicting with transitive install_requires. With PEP 517 on the way now I consider these earlier attempts at declarative package definitions will be more of a historical footnote soon enough, but the intent was pretty similar. Here's an example setup.cfg from a reasonably simple command-line utility: https://git.openstack.org/cgit/openstack-infra/bindep/tree/setup.cfg Anyway, my point wasn't to advertise these tools specifically, but rather to point out that there's nothing stopping projects who want to extract their package metadata into static configuration and bundle their setuptools logic into a separate reusable module (for example by leveraging a setuptools entrypoint the way PBR does) without having to wait around for PEP 517. Then they can always revisit the design once PEP 517 support is more generally rolled out in standard packaging tools. > One thought - are the PBR and/or distutils2 teams looking at > providing PEP 517 support? Assuming they are, have they had a > change to review the PEP to ensure it suits their needs? And if > they aren't, what is it about the PEP that makes them unwilling to > do so? As far as I know, Distutils2 has essentially been dead upstream for ~5 years, so I wouldn't expect it to receive any updates in that regard. I've had conversations with the primary maintainers of PBR about what would need to be done to coexist with and eventually take advantage of PEP 517 paradigms. On-the-fly generation of pyproject.toml files during or, if necessary, preceding the sdist build phase is probably the way they'll be looking to go there, though the details have yet to be completely hashed out. (The community around it is sensitive to gender diversity issues and wants to avoid acquiring more of a "brogrammer" image, so some of us worry that any conspicuous TOML files checked into revision control repositories could be seen as a tacit endorsement of the author's alleged behavior at GH a few years ago.) -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From thomas at kluyver.me.uk Fri Aug 25 16:14:59 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 21:14:59 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> Message-ID: <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> What if hooks returned instances of the exception class? That way it doesn't bubble up from internal code, it can carry a message, and we can all agree that it's a semantically terrible idea that doesn't fit with any conventions. Even I'm not sure if this is a serious suggestion. ;-) On Fri, Aug 25, 2017, at 08:31 PM, xoviat wrote: > Do you mean in addition to the two lines proposed here? I envision the > typical use case as attempting to do something that is required to > build an sdist deep inside the backend, discovering that it cannot be > done, and then raising the proposed exception. That's (other than the > two lines for the exception, which can put in Python later) shorter > than catching some custom exception in the hook and then returning > NotImplemented as a result. If you don't use exceptions, you'll > probably need to unwind the stack manually with return values.> > 2017-08-25 14:25 GMT-05:00 Donald Stufft : >> It's silly to require every backend to write code to guard against >> possible issues when we can structure the API to prevent those >> issues.>> >> Sent from my iPhone >> >> On Aug 25, 2017, at 3:04 PM, xoviat wrote: >>> I agree with Nick that exceptions are the way to do things in Python >>> and with Donald that in general, Python's use of exceptions has >>> caused problems. But I don't think that this forum is the correct >>> place to discuss Python conventions, and so I would ordinarily say >>> that we should just accept that there will be problems and use the >>> NotImplementedError, assuming that "solution" doesn't cause a >>> firestorm that delays acceptance for a significant period of time. >>> The reasoning behind this is that Python has in general adopted this >>> approach (Nick is right that they would have used >>> NotImplementedError for binary operations except for performance >>> issues) even with its problems.>>> >>> However, another solution was dismissed without much thought: using >>> the "UnsupportedOperation" exception. We could for now simply >>> require an extra two lines in all backends:>>> >>> class DistutilsUnsupportedOperation(RuntimeError): >>> pass >>> >>> and then put UnsupportedOperation in the "distutils" library in 3.7, >>> with an eventual move to that. It's not pretty but there have been >>> plenty of hacks for backwards compatibility so far (it's rare to see >>> a project without some sort of compat.py file).>>> >>> 2017-08-25 13:34 GMT-05:00 Donald Stufft : >>>> The thing being bubbled up is a backend accidentally calling an API >>>> that has yet to be implemented (an error that should be reported) >>>> being bubbled up and erroneously handled as the backend reporting >>>> it doesn't support making a sdist (not an error, son no traceback).>>>> >>>> Sent from my iPhone >>>> >>>> >>>> > On Aug 25, 2017, at 2:23 PM, Paul Moore >>>> > wrote:>>>> > >>>> > There's little or no opportunity here for letting exceptions >>>> > bubble up>>>> > to the user, or passing complex data back to the frontend. >>>> > Ultimately,>>>> > it's pretty much immaterial which form of reporting is used. >>>> > _________________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 16:41:45 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 15:41:45 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> Message-ID: There's no need to return an instance of the exception class. As long as it is defined in the same name space as the hooks the front end will be able to access it. On Aug 25, 2017 3:15 PM, "Thomas Kluyver" wrote: > What if hooks returned instances of the exception class? That way it > doesn't bubble up from internal code, it can carry a message, and we can > all agree that it's a semantically terrible idea that doesn't fit with any > conventions. > > Even I'm not sure if this is a serious suggestion. ;-) > > > On Fri, Aug 25, 2017, at 08:31 PM, xoviat wrote: > > Do you mean in addition to the two lines proposed here? I envision the > typical use case as attempting to do something that is required to build an > sdist deep inside the backend, discovering that it cannot be done, and then > raising the proposed exception. That's (other than the two lines for the > exception, which can put in Python later) shorter than catching some custom > exception in the hook and then returning NotImplemented as a result. If you > don't use exceptions, you'll probably need to unwind the stack manually > with return values. > > 2017-08-25 14:25 GMT-05:00 Donald Stufft : > > It's silly to require every backend to write code to guard against > possible issues when we can structure the API to prevent those issues. > > Sent from my iPhone > > On Aug 25, 2017, at 3:04 PM, xoviat wrote: > > I agree with Nick that exceptions are the way to do things in Python and > with Donald that in general, Python's use of exceptions has caused > problems. But I don't think that this forum is the correct place to discuss > Python conventions, and so I would ordinarily say that we should just > accept that there will be problems and use the NotImplementedError, > assuming that "solution" doesn't cause a firestorm that delays acceptance > for a significant period of time. The reasoning behind this is that Python > has in general adopted this approach (Nick is right that they would have > used NotImplementedError for binary operations except for performance > issues) even with its problems. > > However, another solution was dismissed without much thought: using the > "UnsupportedOperation" exception. We could for now simply require an extra > two lines in all backends: > > class DistutilsUnsupportedOperation(RuntimeError): > pass > > and then put UnsupportedOperation in the "distutils" library in 3.7, with > an eventual move to that. It's not pretty but there have been plenty of > hacks for backwards compatibility so far (it's rare to see a project > without some sort of compat.py file). > > 2017-08-25 13:34 GMT-05:00 Donald Stufft : > > The thing being bubbled up is a backend accidentally calling an API that > has yet to be implemented (an error that should be reported) being bubbled > up and erroneously handled as the backend reporting it doesn't support > making a sdist (not an error, son no traceback). > > Sent from my iPhone > > > > On Aug 25, 2017, at 2:23 PM, Paul Moore wrote: > > > > There's little or no opportunity here for letting exceptions bubble up > > to the user, or passing complex data back to the frontend. Ultimately, > > it's pretty much immaterial which form of reporting is used. > > *_______________________________________________* > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Aug 25 16:47:54 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 21:47:54 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> Message-ID: <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> On Fri, Aug 25, 2017, at 09:41 PM, xoviat wrote: > There's no need to return an instance of the exception class. As long > as it is defined in the same name space as the hooks the front end > will be able to access it. No, I meant NotImplementedError. As in: if i_cant_do_it: return NotImplementedError(msg) Thereby neatly annoying all sides of the debate at once. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 16:50:20 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 15:50:20 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> Message-ID: Genius! 2017-08-25 15:47 GMT-05:00 Thomas Kluyver : > On Fri, Aug 25, 2017, at 09:41 PM, xoviat wrote: > > There's no need to return an instance of the exception class. As long as > it is defined in the same name space as the hooks the front end will be > able to access it. > > > No, I meant NotImplementedError. As in: > > if i_cant_do_it: > return NotImplementedError(msg) > > Thereby neatly annoying all sides of the debate at once. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Aug 25 16:51:55 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 21:51:55 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> Message-ID: <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> On Fri, Aug 25, 2017, at 09:50 PM, xoviat wrote: > Genius! 1% inspiration, 99% frustration :-P -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Aug 25 16:54:51 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Aug 2017 13:54:51 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> Message-ID: On Fri, Aug 25, 2017 at 1:51 PM, Thomas Kluyver wrote: > On Fri, Aug 25, 2017, at 09:50 PM, xoviat wrote: > > > Genius! > > > 1% inspiration, 99% frustration :-P This joke is so clever that I fear we may be forced to implement the solution after all, just to punish Thomas. -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Fri Aug 25 17:02:22 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 16:02:22 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> Message-ID: While we're waiting for others to respond, what are we going to do with respect to the sys.path issue? I don't think there has been discussion on that yet. 2017-08-25 15:54 GMT-05:00 Nathaniel Smith : > On Fri, Aug 25, 2017 at 1:51 PM, Thomas Kluyver > wrote: > > On Fri, Aug 25, 2017, at 09:50 PM, xoviat wrote: > > > > > Genius! > > > > > > 1% inspiration, 99% frustration :-P > > This joke is so clever that I fear we may be forced to implement the > solution after all, just to punish Thomas. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Fri Aug 25 17:12:30 2017 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Fri, 25 Aug 2017 18:12:30 -0300 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> Message-ID: On 25 August 2017 at 18:02, xoviat wrote: > While we're waiting for others to respond, what are we going to do with > respect to the sys.path issue? I don't think there has been discussion on > that yet. > There was plenty of discussion. The summary is: - removing srcdir from sys.path requires explicitly removing "" from the head of sys.path, which is not normally done. - it would break many setuptools based projects, probably forcing the setuptools backend to reinsert it, defeating the purpose of removing it in the first place. - if any backend launches a python based subprocess (on top of the one already launched by the frontend for calling the backend), "" goes right back to the head of sys.path, defeating the purpose of removing it in the first place - it will make it impossible for packages to carry or be their own backends, which is an important usecase for the backend packages themselves and for complex setups conspicuously absent from the discussion was an actual decision on whether to remove or just document the presence of srcdir on sys.path. Cheers, Leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Aug 25 17:13:37 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Aug 2017 14:13:37 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> Message-ID: On Fri, Aug 25, 2017 at 9:49 AM, Thomas Kluyver wrote: > Can I gently ask everyone involved to consider whether the > notimplemented/error discussion is verging into bikeshedding > (http://bikeshed.org/)? > > The technical arguments I have seen so far are: > - The exception can include a message > - The return value can't 'bubble up' from the internals of a hook like an > exception I believe Nick also feels that an important advantage of NotImplementedError is: if a frontend doesn't bother to properly implement the spec and special case NotImplemented, then you'll end up eventually getting some obscure error like "'NotImplementedType' object has no attribute ..." when it tries to treat it as a normal return value. With NotImplementedError, if the frontend doesn't treat it specially, the default is to treat it like other exceptions and show a proper traceback and error message. So lazy frontends give better UX for NotImplementedError than NotImplemented. Personally, I don't find the argument about lazy frontends terribly compelling because if we're assuming that we're hitting some buggy corner case with code not following the spec, then we should also consider the case of accidentally bubbled NotImplementedErrors. Between these two cases, an accidentally bubbled NotImplementedError causes even more confusing outcomes (the build frontend may silently start invoking other things! vs. a suboptimal error message), and it's harder to guard against (both designs require properly written frontends to contain a few lines of code special casing NotImplemented/NotImplementedError; only NotImplementedError also requires all careful *backends* to contain just-in-case try/except guards). Another minor point that's made me less happy with NotImplemented is: originally I thought we could make it a general fact about all the hooks that returning "NotImplemented" should be treated the same as if the hook were undefined. (Which is pretty much how it works for __binop__ methods.) On further consideration though I don't think this is a good idea. (Rationale: it's not really what we want for get_build_requires_for_sdist, & if we define future hooks that normally have no return value then there's a danger of buggy frontends missing it entirely, & it wouldn't have worked for Nick's suggestion that build_wheel(build_directory=foo) triggering a NotImplemented should fall back to build_wheel(build_directory=None), which is gone from the spec now but suggests that this could cause problems in the future.) So the bottom line of all this is that if we do go with NotImplemented, I now think it should only be a defined return value for get_requires_for_build_sdist and build_sdist, and should have special "sorry I can't do that Dave" semantics that are different from e.g. a missing get_requires_for_build_sdist hook. All of which will work fine, it's just less... aesthetically pleasing. Personally, I still have a weak preference for NotImplemented over NotImplementedError, but I don't care enough to have a big fight about it. It sounds like Nick and Donald are the only two folks who really have strong opinions here: can the two of you work something out? Should we flip a coin? -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Fri Aug 25 17:17:27 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 16:17:27 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> Message-ID: Nathaniel: What do you think of the proposal regarding DistutilsUnsupportedOperation? 2017-08-25 16:13 GMT-05:00 Nathaniel Smith : > On Fri, Aug 25, 2017 at 9:49 AM, Thomas Kluyver > wrote: > > Can I gently ask everyone involved to consider whether the > > notimplemented/error discussion is verging into bikeshedding > > (http://bikeshed.org/)? > > > > The technical arguments I have seen so far are: > > - The exception can include a message > > - The return value can't 'bubble up' from the internals of a hook like an > > exception > > I believe Nick also feels that an important advantage of > NotImplementedError is: if a frontend doesn't bother to properly > implement the spec and special case NotImplemented, then you'll end up > eventually getting some obscure error like "'NotImplementedType' > object has no attribute ..." when it tries to treat it as a normal > return value. With NotImplementedError, if the frontend doesn't treat > it specially, the default is to treat it like other exceptions and > show a proper traceback and error message. So lazy frontends give > better UX for NotImplementedError than NotImplemented. > > Personally, I don't find the argument about lazy frontends terribly > compelling because if we're assuming that we're hitting some buggy > corner case with code not following the spec, then we should also > consider the case of accidentally bubbled NotImplementedErrors. > Between these two cases, an accidentally bubbled NotImplementedError > causes even more confusing outcomes (the build frontend may silently > start invoking other things! vs. a suboptimal error message), and it's > harder to guard against (both designs require properly written > frontends to contain a few lines of code special casing > NotImplemented/NotImplementedError; only NotImplementedError also > requires all careful *backends* to contain just-in-case try/except > guards). > > Another minor point that's made me less happy with NotImplemented is: > originally I thought we could make it a general fact about all the > hooks that returning "NotImplemented" should be treated the same as if > the hook were undefined. (Which is pretty much how it works for > __binop__ methods.) On further consideration though I don't think this > is a good idea. (Rationale: it's not really what we want for > get_build_requires_for_sdist, & if we define future hooks that > normally have no return value then there's a danger of buggy frontends > missing it entirely, & it wouldn't have worked for Nick's suggestion > that build_wheel(build_directory=foo) triggering a NotImplemented > should fall back to build_wheel(build_directory=None), which is gone > from the spec now but suggests that this could cause problems in the > future.) So the bottom line of all this is that if we do go with > NotImplemented, I now think it should only be a defined return value > for get_requires_for_build_sdist and build_sdist, and should have > special "sorry I can't do that Dave" semantics that are different from > e.g. a missing get_requires_for_build_sdist hook. All of which will > work fine, it's just less... aesthetically pleasing. > > Personally, I still have a weak preference for NotImplemented over > NotImplementedError, but I don't care enough to have a big fight about > it. > > It sounds like Nick and Donald are the only two folks who really have > strong opinions here: can the two of you work something out? Should we > flip a coin? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Aug 25 17:21:58 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 22:21:58 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> Message-ID: <1503696118.3699286.1085305016.4B10B031@webmail.messagingengine.com> On Fri, Aug 25, 2017, at 10:02 PM, xoviat wrote: > While we're waiting for others to respond, what are we going to do > with respect to the sys.path issue? I don't think there has been > discussion on that yet. I'm more or less persuaded by Nathaniel's argument that the source directory shouldn't be on sys.path, because: - Most projects won't need it to be - There are plausible ways it could go wrong - I think we could make a backend which loads the real hooks from the source dir, so there's a backdoor for projects that do want it. IIRC, however, Nick was still strongly in favour of including the source directory on sys.path, and he can usually persuade me of his position with a few emails. I do think that behaviour is more like the 'obvious thing' that Python programmers would expect. Leo: > - removing srcdir from sys.path requires explicitly removing "" from > the head of sys.path, which is not normally done. Not necessarily. If the hook process is launched with a script at the command line, it is the script directory, not the cwd, that is added to sys.path. My WIP hook-calling module works this way:https://github.com/takluyver/pep517/blob/ee43a9334c377d7c37badcc8527cb7a8500180f7/pep517/wrappers.py#L75 With this code, we would actually need to explicitly *add* the source directory (CWD) to sys.path before importing the hook. > - it would break many setuptools based projects, probably forcing the > setuptools backend to reinsert it, defeating the purpose of removing > it in the first place. I don't think we should decide this based on what setuptools does; I expect it to need quite a bit of special handling in any case. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Aug 25 17:26:36 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 16:26:36 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <1503696118.3699286.1085305016.4B10B031@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> <1503696118.3699286.1085305016.4B10B031@webmail.messagingengine.com> Message-ID: > I'm more or less persuaded by Nathaniel's argument that the source directory shouldn't be on sys.path I do too. There should be an option in pyproject.toml to disable this behavior though so that numpy can build itself. 2017-08-25 16:21 GMT-05:00 Thomas Kluyver : > On Fri, Aug 25, 2017, at 10:02 PM, xoviat wrote: > > While we're waiting for others to respond, what are we going to do with > respect to the sys.path issue? I don't think there has been discussion on > that yet. > > > I'm more or less persuaded by Nathaniel's argument that the source > directory shouldn't be on sys.path, because: > > - Most projects won't need it to be > - There are plausible ways it could go wrong > - I think we could make a backend which loads the real hooks from the > source dir, so there's a backdoor for projects that do want it. > > IIRC, however, Nick was still strongly in favour of including the source > directory on sys.path, and he can usually persuade me of his position with > a few emails. I do think that behaviour is more like the 'obvious thing' > that Python programmers would expect. > > Leo: > > > - removing srcdir from sys.path requires explicitly removing "" from > the head of sys.path, which is not normally done. > > Not necessarily. If the hook process is launched with a script at the > command line, it is the script directory, not the cwd, that is added to > sys.path. My WIP hook-calling module works this way: > https://github.com/takluyver/pep517/blob/ee43a9334c377d7c37badcc8527cb7 > a8500180f7/pep517/wrappers.py#L75 > > With this code, we would actually need to explicitly *add* the source > directory (CWD) to sys.path before importing the hook. > > > - it would break many setuptools based projects, probably forcing the > setuptools backend to reinsert it, defeating the purpose of removing it in > the first place. > > I don't think we should decide this based on what setuptools does; I > expect it to need quite a bit of special handling in any case. > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Aug 25 17:56:33 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 25 Aug 2017 22:56:33 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> <1503696118.3699286.1085305016.4B10B031@webmail.messagingengine.com> Message-ID: <1503698193.3708869.1085334176.79268DAF@webmail.messagingengine.com> On Fri, Aug 25, 2017, at 10:26 PM, xoviat wrote: > > I'm more or less persuaded by Nathaniel's argument that the source > > directory shouldn't be on sys.path> > I do too. There should be an option in pyproject.toml to disable this > behavior though so that numpy can build itself. I'm not sure that's necessary: - As I've described, a special shim backend could load hooks from the source directory. This doesn't need us to specify another option in the toml file.- Nathaniel is one of the core numpy developers, so if he's not asking for that switch, numpy presumably doesn't need it. :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Aug 25 18:25:16 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Aug 2017 15:25:16 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> Message-ID: On Fri, Aug 25, 2017 at 2:17 PM, xoviat wrote: > Nathaniel: > > What do you think of the proposal regarding DistutilsUnsupportedOperation? So yeah, we could potentially say: 1) Every backend must have an attribute SdistBuildNotSupportedError, which is a type object subclassing Exception 2) When invoking the sdist build hooks, the frontend does something like: try: requires = backend.get_requires_for_build_sdist() except backend.SdistBuildNotSupportedError: switch_to_fallback_path() It's not ridiculous. If we did this then the name should be more specific than "DistutilsUnsupportedOperation", plus distutils has nothing to do with this. Also I don't think we'd ever want to move the exception into the stdlib, because the advantages are minor compared to the transition costs. And it does have the advantage that it resolves Nick's concern (IIUC) and also solves the accidental bubbling problem. I'm not sure if it would make Donald happy or not. It also provides a general template for how to handle custom errors in future hooks. The main annoyance would be that every backend has to contain some boilerplate like class SdistBuildNotSupportedError(Exception): pass even though most of them that won't ever raise this error, because otherwise the frontend will get an AttributeError in its except: statement. This is an awkward wart. To mitigate it, I guess we could make it optional for backends to export the exception, and frontends would instead write: try: requires = backend.get_requires_for_build_sdist() except getattr(backend, "SdistBuildNotSupportedError", ()): invoke_fallbacks() That's also kind of awkward, but... could be worse? -n > 2017-08-25 16:13 GMT-05:00 Nathaniel Smith : >> >> On Fri, Aug 25, 2017 at 9:49 AM, Thomas Kluyver >> wrote: >> > Can I gently ask everyone involved to consider whether the >> > notimplemented/error discussion is verging into bikeshedding >> > (http://bikeshed.org/)? >> > >> > The technical arguments I have seen so far are: >> > - The exception can include a message >> > - The return value can't 'bubble up' from the internals of a hook like >> > an >> > exception >> >> I believe Nick also feels that an important advantage of >> NotImplementedError is: if a frontend doesn't bother to properly >> implement the spec and special case NotImplemented, then you'll end up >> eventually getting some obscure error like "'NotImplementedType' >> object has no attribute ..." when it tries to treat it as a normal >> return value. With NotImplementedError, if the frontend doesn't treat >> it specially, the default is to treat it like other exceptions and >> show a proper traceback and error message. So lazy frontends give >> better UX for NotImplementedError than NotImplemented. >> >> Personally, I don't find the argument about lazy frontends terribly >> compelling because if we're assuming that we're hitting some buggy >> corner case with code not following the spec, then we should also >> consider the case of accidentally bubbled NotImplementedErrors. >> Between these two cases, an accidentally bubbled NotImplementedError >> causes even more confusing outcomes (the build frontend may silently >> start invoking other things! vs. a suboptimal error message), and it's >> harder to guard against (both designs require properly written >> frontends to contain a few lines of code special casing >> NotImplemented/NotImplementedError; only NotImplementedError also >> requires all careful *backends* to contain just-in-case try/except >> guards). >> >> Another minor point that's made me less happy with NotImplemented is: >> originally I thought we could make it a general fact about all the >> hooks that returning "NotImplemented" should be treated the same as if >> the hook were undefined. (Which is pretty much how it works for >> __binop__ methods.) On further consideration though I don't think this >> is a good idea. (Rationale: it's not really what we want for >> get_build_requires_for_sdist, & if we define future hooks that >> normally have no return value then there's a danger of buggy frontends >> missing it entirely, & it wouldn't have worked for Nick's suggestion >> that build_wheel(build_directory=foo) triggering a NotImplemented >> should fall back to build_wheel(build_directory=None), which is gone >> from the spec now but suggests that this could cause problems in the >> future.) So the bottom line of all this is that if we do go with >> NotImplemented, I now think it should only be a defined return value >> for get_requires_for_build_sdist and build_sdist, and should have >> special "sorry I can't do that Dave" semantics that are different from >> e.g. a missing get_requires_for_build_sdist hook. All of which will >> work fine, it's just less... aesthetically pleasing. >> >> Personally, I still have a weak preference for NotImplemented over >> NotImplementedError, but I don't care enough to have a big fight about >> it. >> >> It sounds like Nick and Donald are the only two folks who really have >> strong opinions here: can the two of you work something out? Should we >> flip a coin? >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Fri Aug 25 18:36:34 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Aug 2017 15:36:34 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> <1503696118.3699286.1085305016.4B10B031@webmail.messagingengine.com> Message-ID: On Fri, Aug 25, 2017 at 2:26 PM, xoviat wrote: >> I'm more or less persuaded by Nathaniel's argument that the source >> directory shouldn't be on sys.path > > I do too. There should be an option in pyproject.toml to disable this > behavior though so that numpy can build itself. My original proposal was to leave the srcdir off of sys.path, and then have a key in pyproject.toml like: [build-system] backend-python-path = ["."] Frontends would then do something like: os.chdir(srcdir) # This line is new to handle the above proposal: sys.path[:0] = [abspath(p) for p in config["build-system"].get("backend-python-path", [])] backend = resolve_entrypoint(config["build-system"]["build-backend"]) I don't have a strong opinion on whether we put this into PEP 517 (it's pretty light weight and doesn't interact with any other features AFAICT), or make it a followup PEP, or start out by deferring this option to a dedicated build backend, like: [build-system] requires = ["override_backend_path"] build-backend = "override_backend_path" [tool.override_backend_path] python-path = ["."] real-backend = "my_awesome:backend" These are all pretty similar. I think the big question for debate is: should sys.path be configurable, or not configurable? IIUC, the main argument for putting the source directory on the path was that extra configuration options are annoying, so we don't want one of those, but we do want to support in-tree backends (even though we expect that most projects won't use this), so we had better put the srcdir on sys.path. My feeling is that in practice, though, that the "no configuration, srcdir always on sys.path" approach is not going to hold up. So first, obviously, the hack above works just fine to make it configurable even if we don't put it in any PEP, so.... in fact it's configurable no matter what. Plus, sooner or later, someone's going to say "hey distutils-sig, I have a build backend package that I want to be able to bootstrap itself, AND I want to put my package inside src/ for reasons [1], wouldn't it be nice if I could put src/ on sys.path without jumping through hoops?". Or someone will release a new version of their build backend that adds a new dependency, or one of their transitive dependencies will release a new version that adds a new dependency, and it collides with some already-released package that uses that build-backend, and the project suffering the collision gets annoyed at being told they need to rearrange their source tree (retroactively, in already released versions!). And they'll come here and say "hey distutils-sig, can solve this problem once and for all?". And we'll be like... uh, fixing this would take what, <5 lines of new code in pip? Kinda hard to argue with that. So... it'll be configurable, one way or another. And if it's configurable... then the question is about whether the default configuration: is it srcdir on sys.path ("opt-out"), or srcdir not on sys.path ("opt-in"). And it seems to me that in this case, all the standard criteria say it should be opt-in. If it's opt-in, then everyone using build backends distributed on PyPI -- which we expect to be the vast majority of project -- never have to think about it and it does the right thing, with no risk of collisions or anything. In fact the only people who have to think about it are the ones implementing in-tree backends, and if you're already like, writing a whole backend and configuring your pyproject.toml to invoke it, then asking you to add one more line of configuration is really not a big deal. OTOH if it's opt-out, then it becomes Yet Another Bad Packaging Default, where conscientious package authors will fret about the risk of collisions and write blog posts about how every project needs to make sure to opt-out of this as a Best Practice, and I am so, so tired of those. -n [1] https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Fri Aug 25 18:50:10 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 17:50:10 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <1503692099.3680760.1085255856.6298E753@webmail.messagingengine.com> <1503694074.3689497.1085284184.0F3942E6@webmail.messagingengine.com> <1503694315.3691405.1085287952.5B76D92D@webmail.messagingengine.com> <1503696118.3699286.1085305016.4B10B031@webmail.messagingengine.com> Message-ID: For the record, I agree with the proposals made in Nathaniel's last two emails. 2017-08-25 17:36 GMT-05:00 Nathaniel Smith : > On Fri, Aug 25, 2017 at 2:26 PM, xoviat wrote: > >> I'm more or less persuaded by Nathaniel's argument that the source > >> directory shouldn't be on sys.path > > > > I do too. There should be an option in pyproject.toml to disable this > > behavior though so that numpy can build itself. > > My original proposal was to leave the srcdir off of sys.path, and then > have a key in pyproject.toml like: > > [build-system] > backend-python-path = ["."] > > Frontends would then do something like: > > os.chdir(srcdir) > # This line is new to handle the above proposal: > sys.path[:0] = [abspath(p) for p in > config["build-system"].get("backend-python-path", [])] > backend = resolve_entrypoint(config["build-system"]["build-backend"]) > > I don't have a strong opinion on whether we put this into PEP 517 > (it's pretty light weight and doesn't interact with any other features > AFAICT), or make it a followup PEP, or start out by deferring this > option to a dedicated build backend, like: > > [build-system] > requires = ["override_backend_path"] > build-backend = "override_backend_path" > > [tool.override_backend_path] > python-path = ["."] > real-backend = "my_awesome:backend" > > These are all pretty similar. > > I think the big question for debate is: should sys.path be > configurable, or not configurable? IIUC, the main argument for putting > the source directory on the path was that extra configuration options > are annoying, so we don't want one of those, but we do want to support > in-tree backends (even though we expect that most projects won't use > this), so we had better put the srcdir on sys.path. > > My feeling is that in practice, though, that the "no configuration, > srcdir always on sys.path" approach is not going to hold up. So first, > obviously, the hack above works just fine to make it configurable even > if we don't put it in any PEP, so.... in fact it's configurable no > matter what. Plus, sooner or later, someone's going to say "hey > distutils-sig, I have a build backend package that I want to be able > to bootstrap itself, AND I want to put my package inside src/ for > reasons [1], wouldn't it be nice if I could put src/ on sys.path > without jumping through hoops?". Or someone will release a new version > of their build backend that adds a new dependency, or one of their > transitive dependencies will release a new version that adds a new > dependency, and it collides with some already-released package that > uses that build-backend, and the project suffering the collision gets > annoyed at being told they need to rearrange their source tree > (retroactively, in already released versions!). And they'll come here > and say "hey distutils-sig, can solve this problem once and for all?". > And we'll be like... uh, fixing this would take what, <5 lines of new > code in pip? Kinda hard to argue with that. > > So... it'll be configurable, one way or another. > > And if it's configurable... then the question is about whether the > default configuration: is it srcdir on sys.path ("opt-out"), or srcdir > not on sys.path ("opt-in"). And it seems to me that in this case, all > the standard criteria say it should be opt-in. > > If it's opt-in, then everyone using build backends distributed on PyPI > -- which we expect to be the vast majority of project -- never have to > think about it and it does the right thing, with no risk of collisions > or anything. In fact the only people who have to think about it are > the ones implementing in-tree backends, and if you're already like, > writing a whole backend and configuring your pyproject.toml to invoke > it, then asking you to add one more line of configuration is really > not a big deal. > > OTOH if it's opt-out, then it becomes Yet Another Bad Packaging > Default, where conscientious package authors will fret about the risk > of collisions and write blog posts about how every project needs to > make sure to opt-out of this as a Best Practice, and I am so, so tired > of those. > > -n > > [1] https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Aug 25 19:16:39 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Aug 2017 16:16:39 -0700 Subject: [Distutils] Conditionless setup.py In-Reply-To: <20170825200042.GB2409@yuggoth.org> References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> <20170825200042.GB2409@yuggoth.org> Message-ID: On Fri, Aug 25, 2017 at 1:00 PM, Jeremy Stanley wrote: > (The > community around it is sensitive to gender diversity issues and > wants to avoid acquiring more of a "brogrammer" image, so some of us > worry that any conspicuous TOML files checked into revision control > repositories could be seen as a tacit endorsement of the author's > alleged behavior at GH a few years ago.) I was one of the folks championing TOML during the original discussions, and this is an issue that also worried me a lot. In case it's a useful data point: I actually contacted several of the main rust/cargo developers, since they were the major users of TOML and are also well known to be sensitive to these issues, to ask if they've had any issues with this, and they said that they haven't heard any complaints. Obviously there's a difference between "no-one complained" and "no-one was bothered", and I suspect the community's existing reputation may affect how this is interpreted as well, but... maybe useful as a data point. Between this and the way the TOML spec appears to have been abandoned at v0.4 (with the admonition "you should assume that is is unstable and act accordingly") I've wondered if we should fork it, rename it "the obvious minimal language", and release our own 1.0. -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Fri Aug 25 20:46:07 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 19:46:07 -0500 Subject: [Distutils] Conditionless setup.py In-Reply-To: References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> <20170825200042.GB2409@yuggoth.org> Message-ID: I personally do not understand the aversion to YAML. I mean yes, the specification is more complicated, but it's also more popular and the YAML files will not be complex enough for a C library to help that much. And since it's more popular, people might even prefer specifying package metadata in a pyproject.yaml. pip could even cache a wheel of the pyyaml package between builds that could be imported at build time with a zipimporter rather than vendoring the package. And as a plus it's not named after an alleged sexist. Honestly this is not an issue that interests me very much but this rant is because I was surprised that toml was chosen when I first found out about it. 2017-08-25 18:16 GMT-05:00 Nathaniel Smith : > On Fri, Aug 25, 2017 at 1:00 PM, Jeremy Stanley wrote: > > (The > > community around it is sensitive to gender diversity issues and > > wants to avoid acquiring more of a "brogrammer" image, so some of us > > worry that any conspicuous TOML files checked into revision control > > repositories could be seen as a tacit endorsement of the author's > > alleged behavior at GH a few years ago.) > > I was one of the folks championing TOML during the original > discussions, and this is an issue that also worried me a lot. In case > it's a useful data point: I actually contacted several of the main > rust/cargo developers, since they were the major users of TOML and are > also well known to be sensitive to these issues, to ask if they've had > any issues with this, and they said that they haven't heard any > complaints. > > Obviously there's a difference between "no-one complained" and "no-one > was bothered", and I suspect the community's existing reputation may > affect how this is interpreted as well, but... maybe useful as a data > point. > > Between this and the way the TOML spec appears to have been abandoned > at v0.4 (with the admonition "you should assume that is is unstable > and act accordingly") I've wondered if we should fork it, rename it > "the obvious minimal language", and release our own 1.0. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Aug 25 20:52:14 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 26 Aug 2017 12:52:14 +1200 Subject: [Distutils] PEP 517 again In-Reply-To: <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> Message-ID: <59A0C63E.70905@canterbury.ac.nz> Donald Stufft wrote: > The thing being bubbled up is a backend accidentally calling an API that has > yet to be implemented (an error that should be reported) being bubbled up and > erroneously handled as the backend reporting it doesn't support making a > sdist (not an error, son no traceback). To my way of thinking, the only justification for using an exception instead of a return value as part of a protocol between a caller and a callee is if you need some kind of out-of-band return value. For example, __next__ can't use a return value to signal end of iteration, because any value could be a legitimate return value. That doesn't apply here, so there's no reason to use an exception. -- Greg From greg.ewing at canterbury.ac.nz Fri Aug 25 21:00:11 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 26 Aug 2017 13:00:11 +1200 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> Message-ID: <59A0C81B.2010303@canterbury.ac.nz> xoviat wrote: > I agree with Nick that exceptions are the way to do things in Python I don't agree that exceptions are the way to do "things" in general. They're the way to so *some* things. The question is whether the thing we're talking about is one of those things. > The reasoning behind this > is that Python has in general adopted this approach (Nick is right that > they would have used NotImplementedError for binary operations except > for performance issues) But only because NotImplemented is an out-of-band condition for operations in general. We're not talking about a general operation here, but a very specific one that only has a couple of possible results. -- Greg From njs at pobox.com Fri Aug 25 21:10:43 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Aug 2017 18:10:43 -0700 Subject: [Distutils] Conditionless setup.py In-Reply-To: References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> <20170825200042.GB2409@yuggoth.org> Message-ID: On Fri, Aug 25, 2017 at 5:46 PM, xoviat wrote: > I personally do not understand the aversion to YAML. I mean yes, the > specification is more complicated, but it's also more popular and the YAML > files will not be complex enough for a C library to help that much. And > since it's more popular, people might even prefer specifying package > metadata in a pyproject.yaml. pip could even cache a wheel of the pyyaml > package between builds that could be imported at build time with a > zipimporter rather than vendoring the package. And as a plus it's not named > after an alleged sexist. > > Honestly this is not an issue that interests me very much but this rant is > because I was surprised that toml was chosen when I first found out about > it. If you want to know why it was chosen then there's lots of discussion in the list archives. I don't think this is a great place to relitigate it. -n -- Nathaniel J. Smith -- https://vorpus.org From greg.ewing at canterbury.ac.nz Fri Aug 25 21:16:18 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 26 Aug 2017 13:16:18 +1200 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> Message-ID: <59A0CBE2.6040803@canterbury.ac.nz> Nathaniel Smith wrote: > I now think it should only be a defined return value > for get_requires_for_build_sdist and build_sdist, and should have > special "sorry I can't do that Dave" semantics that are different from > e.g. a missing get_requires_for_build_sdist hook. I don't think there's anything wrong with NotImplemented triggering different actions in different circumstances. It already has very context-dependent "fall back to something else" semantics for binary operators. -- Greg From xoviat at gmail.com Fri Aug 25 21:17:20 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 20:17:20 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <59A0C81B.2010303@canterbury.ac.nz> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> Message-ID: > I don't agree that exceptions are the way to do "things" in general. They're the way to so *some* things. The question is whether the thing we're talking about is one of those things. I mean how is opening a file different than attempting to build an sdist? open() could return true or false and not raise any exception. But it doesn't. If you can't open the file, you get an exception. It's the same idea except here we are asking for an sdist. We would greatly prefer to have one. And if we don't have one, then we have to do something else to handle the error. This *is* Python. Sure, it may not be a good idea, though I think it's nice most of the time. It may not even be accepted for this PEP. But, although I don't have a strong opinion about whether it *should* be in the PEP, I do have a strong opinion about whether it would be an exception to the norms of Python, and my opinion is that it would. The fact that Nick, who I think is the only core developer here, immediately jumped on this issue confirms my suspicions here. 2017-08-25 20:00 GMT-05:00 Greg Ewing : > xoviat wrote: > >> I agree with Nick that exceptions are the way to do things in Python >> > > I don't agree that exceptions are the way to do "things" > in general. They're the way to so *some* things. The question > is whether the thing we're talking about is one of those > things. > > The reasoning behind this is that Python has in general adopted this >> approach (Nick is right that they would have used NotImplementedError for >> binary operations except for performance issues) >> > > But only because NotImplemented is an out-of-band condition > for operations in general. We're not talking about a general > operation here, but a very specific one that only has a couple > of possible results. > > -- > Greg > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Aug 25 21:28:14 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 26 Aug 2017 13:28:14 +1200 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> Message-ID: <59A0CEAE.4060800@canterbury.ac.nz> xoviat wrote: > I mean how is opening a file different than attempting to build an > sdist? 1. Opening a file is a very common operation. 2. Most file opens are expected to succeed, and if one doesn't, the appropriate place to deal with that is almost never at the site of the open call, but somewhere further up. In contrast, there's probably only about one place in any given frontend where the backend's build_sdist method gets invoked, and it's unlikely the ability to let a not-implemented exception bubble up from that point would be of great use. -- Greg From xoviat at gmail.com Fri Aug 25 21:53:06 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 20:53:06 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: Oh, BDFL, will you settle whether we should raise an exception or return Notimplemented? On Aug 25, 2017 8:28 PM, "Greg Ewing" wrote: xoviat wrote: > I mean how is opening a file different than attempting to build an sdist? > 1. Opening a file is a very common operation. 2. Most file opens are expected to succeed, and if one doesn't, the appropriate place to deal with that is almost never at the site of the open call, but somewhere further up. In contrast, there's probably only about one place in any given frontend where the backend's build_sdist method gets invoked, and it's unlikely the ability to let a not-implemented exception bubble up from that point would be of great use. -- Greg _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri Aug 25 21:54:45 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Sat, 26 Aug 2017 01:54:45 +0000 Subject: [Distutils] Conditionless setup.py In-Reply-To: References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> <20170825200042.GB2409@yuggoth.org> Message-ID: <20170826015444.GD2409@yuggoth.org> On 2017-08-25 16:16:39 -0700 (-0700), Nathaniel Smith wrote: [...] > I've wondered if we should fork it, rename it "the obvious minimal > language", and release our own 1.0. And make it a toggleable option to configparser? ;) The basic format is pretty similar, granted TOML has a lot more flexibility over configparser's classic loader. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From xoviat at gmail.com Fri Aug 25 21:57:17 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 25 Aug 2017 20:57:17 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: Looks like Nick is on the losing side here, but we shall see. On Aug 25, 2017 8:28 PM, "Greg Ewing" wrote: > Oh, BDFL, will you settle whether we should raise an exception or return > Notimplemented? > > On Aug 25, 2017 8:28 PM, "Greg Ewing" wrote: > > xoviat wrote: > >> I mean how is opening a file different than attempting to build an sdist? >> > > 1. Opening a file is a very common operation. > > 2. Most file opens are expected to succeed, and if one doesn't, > the appropriate place to deal with that is almost never at the > site of the open call, but somewhere further up. > > In contrast, there's probably only about one place in any > given frontend where the backend's build_sdist method gets > invoked, and it's unlikely the ability to let a not-implemented > exception bubble up from that point would be of great use. > > > -- > Greg > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Aug 25 22:17:06 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 25 Aug 2017 19:17:06 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: I don't have any context here, but "return NotImplemented" is a very narrow idiom intended only for binary operators (e.g. __add__) when the interpreter should give the other operand a chance (e.g. __radd__) or use a default implementation. In pretty much any other context, if you have an operation that returns an regular value or an error value, the error value should be None. (Exceptions include e.g. returning a non-negative int or -1 for errors, or True for success and False for errors.) On Fri, Aug 25, 2017 at 6:53 PM, xoviat wrote: > Oh, BDFL, will you settle whether we should raise an exception or return > Notimplemented? > > On Aug 25, 2017 8:28 PM, "Greg Ewing" wrote: > > xoviat wrote: > >> I mean how is opening a file different than attempting to build an sdist? >> > > 1. Opening a file is a very common operation. > > 2. Most file opens are expected to succeed, and if one doesn't, > the appropriate place to deal with that is almost never at the > site of the open call, but somewhere further up. > > In contrast, there's probably only about one place in any > given frontend where the backend's build_sdist method gets > invoked, and it's unlikely the ability to let a not-implemented > exception bubble up from that point would be of great use. > > > -- > Greg > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Aug 26 05:21:34 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 26 Aug 2017 10:21:34 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> Message-ID: On 26 August 2017 at 02:17, xoviat wrote: > The fact that Nick, who I think is > the only core developer here, immediately jumped on this issue confirms my > suspicions here. *ahem* Donald and myself are both core devs too. And possibly others, I honestly don't know. (Not that I think it makes much difference - arguments should be taken on their merit, not on who made them). Paul From p.f.moore at gmail.com Sat Aug 26 05:29:41 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 26 Aug 2017 10:29:41 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On 26 August 2017 at 03:17, Guido van Rossum wrote: > In pretty much any other context, if you have an operation that returns an > regular value or an error value, the error value should be None. (Exceptions > include e.g. returning a non-negative int or -1 for errors, or True for > success and False for errors.) So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None. Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do. Paul From dmertz at anaconda.com Sat Aug 26 11:29:09 2017 From: dmertz at anaconda.com (David Mertz) Date: Sat, 26 Aug 2017 08:29:09 -0700 Subject: [Distutils] Conditionless setup.py In-Reply-To: References: <6d717aef-0fb9-c0cb-3d8b-a43c2f6baf03@thomas-guettler.de> <20170825111442.GZ2409@yuggoth.org> <20170825200042.GB2409@yuggoth.org> Message-ID: Me too. YAML is *so much* more widely used, and the complicated edge cases can simply be ignored for this requirement. Maybe it's just because I've never heard of whatever improper behavior the author engaged in, but I don't think a data format needs to answer to the actions of its creator(s). If G. Klyne or C. Newman were to do something dreadful I wouldn't want dates to stop following ISO-8601.[*] [*] I don't even know the 8601 authors' first names, and assume they are good and honorable people. Just making an analogy. On Aug 25, 2017 5:46 PM, "xoviat" wrote: > I personally do not understand the aversion to YAML. I mean yes, the > specification is more complicated, but it's also more popular and the YAML > files will not be complex enough for a C library to help that much. And > since it's more popular, people might even prefer specifying package > metadata in a pyproject.yaml. pip could even cache a wheel of the pyyaml > package between builds that could be imported at build time with a > zipimporter rather than vendoring the package. And as a plus it's not named > after an alleged sexist. > > Honestly this is not an issue that interests me very much but this rant is > because I was surprised that toml was chosen when I first found out about > it. > > 2017-08-25 18:16 GMT-05:00 Nathaniel Smith : > >> On Fri, Aug 25, 2017 at 1:00 PM, Jeremy Stanley >> wrote: >> > (The >> > community around it is sensitive to gender diversity issues and >> > wants to avoid acquiring more of a "brogrammer" image, so some of us >> > worry that any conspicuous TOML files checked into revision control >> > repositories could be seen as a tacit endorsement of the author's >> > alleged behavior at GH a few years ago.) >> >> I was one of the folks championing TOML during the original >> discussions, and this is an issue that also worried me a lot. In case >> it's a useful data point: I actually contacted several of the main >> rust/cargo developers, since they were the major users of TOML and are >> also well known to be sensitive to these issues, to ask if they've had >> any issues with this, and they said that they haven't heard any >> complaints. >> >> Obviously there's a difference between "no-one complained" and "no-one >> was bothered", and I suspect the community's existing reputation may >> affect how this is interpreted as well, but... maybe useful as a data >> point. >> >> Between this and the way the TOML spec appears to have been abandoned >> at v0.4 (with the admonition "you should assume that is is unstable >> and act accordingly") I've wondered if we should fork it, rename it >> "the obvious minimal language", and release our own 1.0. >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 26 11:59:15 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 10:59:15 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: Is everyone on board with that? On Aug 26, 2017 4:29 AM, "Paul Moore" wrote: > On 26 August 2017 at 03:17, Guido van Rossum wrote: > > In pretty much any other context, if you have an operation that returns > an > > regular value or an error value, the error value should be None. > (Exceptions > > include e.g. returning a non-negative int or -1 for errors, or True for > > success and False for errors.) > > So, given that build_sdist returns the path of the newly built sdist, > the correct way to signal "I didn't manage to build a sdist" would be > to return None. > > Now that it's put this way, it seems glaringly obvious to me that this > is the correct thing to do. > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sat Aug 26 12:00:14 2017 From: dholth at gmail.com (Daniel Holth) Date: Sat, 26 Aug 2017 16:00:14 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: I'm bored with that On Sat, Aug 26, 2017, 11:59 xoviat wrote: > Is everyone on board with that? > > On Aug 26, 2017 4:29 AM, "Paul Moore" wrote: > >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> an >> > regular value or an error value, the error value should be None. >> (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> Paul >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 26 12:05:07 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 11:05:07 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: Should probably take Guido out of this email chain now. And does everyone agree with what Nathaniel said on sys.path? On Aug 26, 2017 11:00 AM, "Daniel Holth" wrote: I'm bored with that On Sat, Aug 26, 2017, 11:59 xoviat wrote: > Is everyone on board with that? > > On Aug 26, 2017 4:29 AM, "Paul Moore" wrote: > >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> an >> > regular value or an error value, the error value should be None. >> (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> Paul >> > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Aug 26 15:17:30 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 26 Aug 2017 12:17:30 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: [removed Guido from CC] On Aug 26, 2017 02:29, "Paul Moore" wrote: On 26 August 2017 at 03:17, Guido van Rossum wrote: > In pretty much any other context, if you have an operation that returns an > regular value or an error value, the error value should be None. (Exceptions > include e.g. returning a non-negative int or -1 for errors, or True for > success and False for errors.) So, given that build_sdist returns the path of the newly built sdist, the correct way to signal "I didn't manage to build a sdist" would be to return None. Now that it's put this way, it seems glaringly obvious to me that this is the correct thing to do. Eh... I would really prefer something that's (a) more explicit about what specifically went wrong, and (b) harder to return by accident. It's not at all obvious that if the list of requirements is 'None' that means 'this build supports making sdists in general but cannot make them from this source tree but might still be able to make a wheel'. And if you forget to put in a return statement, then python returns None for you, which seems like it could lead to some super confusing error modes. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 26 15:25:14 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 14:25:14 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: The current PEP requires that build_sdist return the basename of the sdist. So forgetting a return statement is not an option unless people really don't read the PEP. On Aug 26, 2017 2:18 PM, "Nathaniel Smith" wrote: > [removed Guido from CC] > > On Aug 26, 2017 02:29, "Paul Moore" wrote: > > On 26 August 2017 at 03:17, Guido van Rossum wrote: > > In pretty much any other context, if you have an operation that returns > an > > regular value or an error value, the error value should be None. > (Exceptions > > include e.g. returning a non-negative int or -1 for errors, or True for > > success and False for errors.) > > So, given that build_sdist returns the path of the newly built sdist, > the correct way to signal "I didn't manage to build a sdist" would be > to return None. > > Now that it's put this way, it seems glaringly obvious to me that this > is the correct thing to do. > > > Eh... I would really prefer something that's (a) more explicit about what > specifically went wrong, and (b) harder to return by accident. It's not at > all obvious that if the list of requirements is 'None' that means 'this > build supports making sdists in general but cannot make them from this > source tree but might still be able to make a wheel'. And if you forget to > put in a return statement, then python returns None for you, which seems > like it could lead to some super confusing error modes. > > -n > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Aug 26 15:54:49 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 26 Aug 2017 20:54:49 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On 26 August 2017 at 20:17, Nathaniel Smith wrote: > Eh... I would really prefer something that's (a) more explicit about what > specifically went wrong, and (b) harder to return by accident. It's not at > all obvious that if the list of requirements is 'None' that means 'this > build supports making sdists in general but cannot make them from this > source tree but might still be able to make a wheel'. And if you forget to > put in a return statement, then python returns None for you, which seems > like it could lead to some super confusing error modes. Well, we've had an extensive discussion about how frontends need to trust backends to get things right. I don't really see it as reasonable to now argue that backends might "forget" to return the right value - they might just as well "forget" to properly isolate builds... As regards an explicit description of what went wrong, why can't we just use the same reporting methods that we will for any other build issue (backends simply report the problem on stdout/stderr)? I don't see why the backend has to package up its error information and send it to the frontend to report, when we already have a perfectly effective way for backends to report errors and/or warnings to the user. If you're worried that the frontend might suppress the information (maybe because it's planning on falling back to a direct wheel build) then isn't that just the converse - backends need to trust frontends to do the right thing? It's hard to come up with convincing counterarguments to hypothetical concerns. Do you have a specific scenario that worries you? I see "return the path, or None indicating you couldn't build the sdist" as a nice, simple solution. If there's a case we know of right now where it's *too* simple, then let's understand the specifics - otherwise, why not defer worrying about it for now? We can revise the PEP later if experience shows we missed something. Paul From njs at pobox.com Sat Aug 26 16:42:02 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 26 Aug 2017 13:42:02 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore wrote: > On 26 August 2017 at 20:17, Nathaniel Smith wrote: >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. > > Well, we've had an extensive discussion about how frontends need to > trust backends to get things right. I don't really see it as > reasonable to now argue that backends might "forget" to return the > right value - they might just as well "forget" to properly isolate > builds... It's not about division of responsibilities, it's about handling errors gracefully when they happen. There are three bins: - creating an sdist succeeded - creating an sdist failed for expected reasons, and a clever frontend might be able to handle the problem automatically if it understands what the problem is (sdist creation isn't supported in this case) and understands its goals (just trying to build a wheel really, so the sdist isn't crucial) - creating an sdist failed for unexpected reasons, that need a human to sort out (due to a broken system, or bugs ? hey, they happen ? or ...) The whole discussion has been about how we can most reliably distinguish between the second and third categories, and give good error messages for the third category. The argument for NotImplemented is that it avoids cases where some internal call raises NotImplementedError and it "leaks out" accidentally, causing a unexpected error to be incorrectly treated as expected error -- we don't want pip to be hiding real bugs in backend code. The argument for NotImplementedError is that it produces better error messages on buggy frontends. 'return None' is kind of the worst of both worlds, in that it's an easy thing to return accidentally, and it gives confusing error messages if the frontend fails to handle it properly. (Even more confusing, actually, because 'NoneType object has no attribute ...' is even harder to track down than 'NotImplementedType object has no attribute ...'.) > As regards an explicit description of what went wrong, why can't we > just use the same reporting methods that we will for any other build > issue (backends simply report the problem on stdout/stderr)? I don't > see why the backend has to package up its error information and send > it to the frontend to report, when we already have a perfectly > effective way for backends to report errors and/or warnings to the > user. If you're worried that the frontend might suppress the > information (maybe because it's planning on falling back to a direct > wheel build) then isn't that just the converse - backends need to > trust frontends to do the right thing? What I mean is more, if you're some random user and you see this in a build backend, what do you guess it means? def get_requires_for_build_sdist(config_settings=None): return None Now how about these? def get_requires_for_build_sdist(config_settings=None): return NotImplemented def get_requires_for_build_sdist(config_settings=None): raise NotImplementedError def get_requires_for_build_sdist(config_settings=None): raise SdistBuildNotSupported I mean, obviously return None will work. Basically anything that's different from "return a list or string" will work :-). That's what makes this a bikeshed topic, and I still think we're mostly just spinning our wheels here until Nick and Donald have a chance to hash something out that they both can agree on. But I really don't see any advantages to 'return None' compared to the other options that have been discussed -n -- Nathaniel J. Smith -- https://vorpus.org From c at anthonyrisinger.com Sat Aug 26 16:47:05 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Sat, 26 Aug 2017 15:47:05 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: > [removed Guido from CC] > > On Aug 26, 2017 02:29, "Paul Moore" wrote: > > On 26 August 2017 at 03:17, Guido van Rossum wrote: > > In pretty much any other context, if you have an operation that returns > an > > regular value or an error value, the error value should be None. > (Exceptions > > include e.g. returning a non-negative int or -1 for errors, or True for > > success and False for errors.) > > So, given that build_sdist returns the path of the newly built sdist, > the correct way to signal "I didn't manage to build a sdist" would be > to return None. > > Now that it's put this way, it seems glaringly obvious to me that this > is the correct thing to do. > > > Eh... I would really prefer something that's (a) more explicit about what > specifically went wrong, and (b) harder to return by accident. It's not at > all obvious that if the list of requirements is 'None' that means 'this > build supports making sdists in general but cannot make them from this > source tree but might still be able to make a wheel'. And if you forget to > put in a return statement, then python returns None for you, which seems > like it could lead to some super confusing error modes. > Why does the frontend need to know why an sdist was not created? Frontend is asking the backend, given the current state of the world, to either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I would expect build_sdist to make some sanity checks about the world, then make a binary choice about whether sdist creation is a valid goal. If not possible, return None or NotImplemented or False or dict-of-reasons or whatever. Only if creation was *attempted*, and in the exceptional event it then failed, would I expect an Exception. We don't have structured exceptions sadly so they can't really carry much useful information from a protocol perspective above and beyond a simple None or the like anyway. I'd personally like to see some parity between build_sdist and build_wheel in this regard. Maybe the disconnect here is we have a way to specify hard reqs for building a wheel, statically or dynamically, and build_wheel is expected to never fail, but no way to specify hard reqs needed for build_sdist, necessitating this optional signaling path? If we had some definitive way for the frontend to know ahead of time if build_sdist is even expected to work, it could be called with more confidence. This could be a new sdist-related key in [build-system], a new table like [sdist-system].requires, or making the get_requires_for_* less optional, and defaulting to None instead of [ ]. Frontend is responsible for prepping the world, so if it can't get a list of reqs, somehow, for build_sdist, it knows it can't work. Same for build_wheel, because you have to specify the backend itself, so there is at least one requirement! Thus if you are a backend that can produce an sdist without additional requirements beyond build reqs, you should explicitly return empty list from get_requires_for_build_sdist. -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 26 17:04:56 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 16:04:56 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: Why does the frontend need to know why an sdist was not created? I was of the opinion that such a distinction is not necessary because building a source distribution doesn't take that much time. However Donald thought that there needed to be a distinction because of the wasted time in attempting to build a wheel that was going to fail anyway. One of the things to consider is that site cythonizing takes time and maybe called for building source distribution. However since I think we're of the agreement that a source distribution should be as close to a checkout as possible, that may not be an issue because cythonizing may not be required to build the sdist. On Aug 26, 2017 3:47 PM, "C Anthony Risinger" wrote: > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: > >> [removed Guido from CC] >> >> On Aug 26, 2017 02:29, "Paul Moore" wrote: >> >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> an >> > regular value or an error value, the error value should be None. >> (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> >> >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. >> > > Why does the frontend need to know why an sdist was not created? > > Frontend is asking the backend, given the current state of the world, to > either produce an sdist, or not. Sans ahead-of-time knowledge (see below), > I would expect build_sdist to make some sanity checks about the world, then > make a binary choice about whether sdist creation is a valid goal. If not > possible, return None or NotImplemented or False or dict-of-reasons or > whatever. Only if creation was *attempted*, and in the exceptional event it > then failed, would I expect an Exception. We don't have structured > exceptions sadly so they can't really carry much useful information from a > protocol perspective above and beyond a simple None or the like anyway. > > I'd personally like to see some parity between build_sdist and build_wheel > in this regard. Maybe the disconnect here is we have a way to specify hard > reqs for building a wheel, statically or dynamically, and build_wheel is > expected to never fail, but no way to specify hard reqs needed for > build_sdist, necessitating this optional signaling path? > > If we had some definitive way for the frontend to know ahead of time if > build_sdist is even expected to work, it could be called with more > confidence. > > This could be a new sdist-related key in [build-system], a new table like > [sdist-system].requires, or making the get_requires_for_* less optional, > and defaulting to None instead of [ ]. > > Frontend is responsible for prepping the world, so if it can't get a list > of reqs, somehow, for build_sdist, it knows it can't work. Same for > build_wheel, because you have to specify the backend itself, so there is at > least one requirement! > > Thus if you are a backend that can produce an sdist without additional > requirements beyond build reqs, you should explicitly return empty list > from get_requires_for_build_sdist. > > -- > > C Anthony > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 26 17:06:44 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 16:06:44 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: I also think that Guido pretty much ruled out Notimplemented. On Aug 26, 2017 4:04 PM, "xoviat" wrote: > Why does the frontend need to know why an sdist was not created? > > I was of the opinion that such a distinction is not necessary because > building a source distribution doesn't take that much time. However Donald > thought that there needed to be a distinction because of the wasted time in > attempting to build a wheel that was going to fail anyway. One of the > things to consider is that site cythonizing takes time and maybe called for > building source distribution. However since I think we're of the agreement > that a source distribution should be as close to a checkout as possible, > that may not be an issue because cythonizing may not be required to build > the sdist. > > On Aug 26, 2017 3:47 PM, "C Anthony Risinger" > wrote: > >> On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: >> >>> [removed Guido from CC] >>> >>> On Aug 26, 2017 02:29, "Paul Moore" wrote: >>> >>> On 26 August 2017 at 03:17, Guido van Rossum wrote: >>> > In pretty much any other context, if you have an operation that >>> returns an >>> > regular value or an error value, the error value should be None. >>> (Exceptions >>> > include e.g. returning a non-negative int or -1 for errors, or True for >>> > success and False for errors.) >>> >>> So, given that build_sdist returns the path of the newly built sdist, >>> the correct way to signal "I didn't manage to build a sdist" would be >>> to return None. >>> >>> Now that it's put this way, it seems glaringly obvious to me that this >>> is the correct thing to do. >>> >>> >>> Eh... I would really prefer something that's (a) more explicit about >>> what specifically went wrong, and (b) harder to return by accident. It's >>> not at all obvious that if the list of requirements is 'None' that means >>> 'this build supports making sdists in general but cannot make them from >>> this source tree but might still be able to make a wheel'. And if you >>> forget to put in a return statement, then python returns None for you, >>> which seems like it could lead to some super confusing error modes. >>> >> >> Why does the frontend need to know why an sdist was not created? >> >> Frontend is asking the backend, given the current state of the world, to >> either produce an sdist, or not. Sans ahead-of-time knowledge (see below), >> I would expect build_sdist to make some sanity checks about the world, then >> make a binary choice about whether sdist creation is a valid goal. If not >> possible, return None or NotImplemented or False or dict-of-reasons or >> whatever. Only if creation was *attempted*, and in the exceptional event it >> then failed, would I expect an Exception. We don't have structured >> exceptions sadly so they can't really carry much useful information from a >> protocol perspective above and beyond a simple None or the like anyway. >> >> I'd personally like to see some parity between build_sdist and >> build_wheel in this regard. Maybe the disconnect here is we have a way to >> specify hard reqs for building a wheel, statically or dynamically, and >> build_wheel is expected to never fail, but no way to specify hard reqs >> needed for build_sdist, necessitating this optional signaling path? >> >> If we had some definitive way for the frontend to know ahead of time if >> build_sdist is even expected to work, it could be called with more >> confidence. >> >> This could be a new sdist-related key in [build-system], a new table like >> [sdist-system].requires, or making the get_requires_for_* less optional, >> and defaulting to None instead of [ ]. >> >> Frontend is responsible for prepping the world, so if it can't get a list >> of reqs, somehow, for build_sdist, it knows it can't work. Same for >> build_wheel, because you have to specify the backend itself, so there is at >> least one requirement! >> >> Thus if you are a backend that can produce an sdist without additional >> requirements beyond build reqs, you should explicitly return empty list >> from get_requires_for_build_sdist. >> >> -- >> >> C Anthony >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 26 17:11:43 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 16:11:43 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: Nathaniel: We're not talking about signaling failure in get_requires* we're talking about signaling failure in build*. On Aug 26, 2017 3:42 PM, "Nathaniel Smith" wrote: > On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore wrote: > > On 26 August 2017 at 20:17, Nathaniel Smith wrote: > >> Eh... I would really prefer something that's (a) more explicit about > what > >> specifically went wrong, and (b) harder to return by accident. It's not > at > >> all obvious that if the list of requirements is 'None' that means 'this > >> build supports making sdists in general but cannot make them from this > >> source tree but might still be able to make a wheel'. And if you forget > to > >> put in a return statement, then python returns None for you, which seems > >> like it could lead to some super confusing error modes. > > > > Well, we've had an extensive discussion about how frontends need to > > trust backends to get things right. I don't really see it as > > reasonable to now argue that backends might "forget" to return the > > right value - they might just as well "forget" to properly isolate > > builds... > > It's not about division of responsibilities, it's about handling > errors gracefully when they happen. There are three bins: > > - creating an sdist succeeded > - creating an sdist failed for expected reasons, and a clever frontend > might be able to handle the problem automatically if it understands > what the problem is (sdist creation isn't supported in this case) and > understands its goals (just trying to build a wheel really, so the > sdist isn't crucial) > - creating an sdist failed for unexpected reasons, that need a human > to sort out (due to a broken system, or bugs ? hey, they happen ? or > ...) > > The whole discussion has been about how we can most reliably > distinguish between the second and third categories, and give good > error messages for the third category. The argument for NotImplemented > is that it avoids cases where some internal call raises > NotImplementedError and it "leaks out" accidentally, causing a > unexpected error to be incorrectly treated as expected error -- we > don't want pip to be hiding real bugs in backend code. The argument > for NotImplementedError is that it produces better error messages on > buggy frontends. 'return None' is kind of the worst of both worlds, in > that it's an easy thing to return accidentally, and it gives confusing > error messages if the frontend fails to handle it properly. (Even more > confusing, actually, because 'NoneType object has no attribute ...' is > even harder to track down than 'NotImplementedType object has no > attribute ...'.) > > > As regards an explicit description of what went wrong, why can't we > > just use the same reporting methods that we will for any other build > > issue (backends simply report the problem on stdout/stderr)? I don't > > see why the backend has to package up its error information and send > > it to the frontend to report, when we already have a perfectly > > effective way for backends to report errors and/or warnings to the > > user. If you're worried that the frontend might suppress the > > information (maybe because it's planning on falling back to a direct > > wheel build) then isn't that just the converse - backends need to > > trust frontends to do the right thing? > > What I mean is more, if you're some random user and you see this in a > build backend, what do you guess it means? > > def get_requires_for_build_sdist(config_settings=None): > return None > > Now how about these? > > def get_requires_for_build_sdist(config_settings=None): > return NotImplemented > > def get_requires_for_build_sdist(config_settings=None): > raise NotImplementedError > > def get_requires_for_build_sdist(config_settings=None): > raise SdistBuildNotSupported > > I mean, obviously return None will work. Basically anything that's > different from "return a list or string" will work :-). That's what > makes this a bikeshed topic, and I still think we're mostly just > spinning our wheels here until Nick and Donald have a chance to hash > something out that they both can agree on. But I really don't see any > advantages to 'return None' compared to the other options that have > been discussed > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Aug 26 17:56:21 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 26 Aug 2017 14:56:21 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Sat, Aug 26, 2017 at 2:06 PM, xoviat wrote: > I also think that Guido pretty much ruled out Notimplemented. As I've said, I don't think it matters a huge deal whether we use NotImplemented or not. But please don't treat Guido as some kind of pronouncement generating machine where you hurl out-of-context questions at him and then use his response as a club to beat down discussion. It's rude to Guido, it's rude to Nick and Donald (to whom Guido has explicitly delegated his BDFL authority in packaging-related matters), and it's rude to everyone trying to discuss proposals on their merits. -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Sat Aug 26 18:03:24 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 17:03:24 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: As I said, I don't care what the particular solution is on this issue. However I'm simply trying to anticipate and resolve potential disagreements that could drag this out for a significant period of time. It is clear that Nick and Donald have a disagreement on this issue which is actually not in and of itself packaging related. The clear way to resolve this was to get a pronouncement. On Aug 26, 2017 4:56 PM, "Nathaniel Smith" wrote: > On Sat, Aug 26, 2017 at 2:06 PM, xoviat wrote: > > I also think that Guido pretty much ruled out Notimplemented. > > As I've said, I don't think it matters a huge deal whether we use > NotImplemented or not. But please don't treat Guido as some kind of > pronouncement generating machine where you hurl out-of-context > questions at him and then use his response as a club to beat down > discussion. It's rude to Guido, it's rude to Nick and Donald (to whom > Guido has explicitly delegated his BDFL authority in packaging-related > matters), and it's rude to everyone trying to discuss proposals on > their merits. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Aug 26 18:05:18 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 17:05:18 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: With respect to this issue, if everyone told me that I was wrong then I would say I'm obviously wrong. But some people are saying one thing and other people are saying something else. On Aug 26, 2017 5:03 PM, "xoviat" wrote: > As I said, I don't care what the particular solution is on this issue. > However I'm simply trying to anticipate and resolve potential disagreements > that could drag this out for a significant period of time. It is clear that > Nick and Donald have a disagreement on this issue which is actually not in > and of itself packaging related. The clear way to resolve this was to get a > pronouncement. > > On Aug 26, 2017 4:56 PM, "Nathaniel Smith" wrote: > >> On Sat, Aug 26, 2017 at 2:06 PM, xoviat wrote: >> > I also think that Guido pretty much ruled out Notimplemented. >> >> As I've said, I don't think it matters a huge deal whether we use >> NotImplemented or not. But please don't treat Guido as some kind of >> pronouncement generating machine where you hurl out-of-context >> questions at him and then use his response as a club to beat down >> discussion. It's rude to Guido, it's rude to Nick and Donald (to whom >> Guido has explicitly delegated his BDFL authority in packaging-related >> matters), and it's rude to everyone trying to discuss proposals on >> their merits. >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Aug 26 18:13:11 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 26 Aug 2017 15:13:11 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger wrote: > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: >> >> [removed Guido from CC] >> >> On Aug 26, 2017 02:29, "Paul Moore" wrote: >> >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> > an >> > regular value or an error value, the error value should be None. >> > (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> >> >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. > > > Why does the frontend need to know why an sdist was not created? This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-) > Frontend is asking the backend, given the current state of the world, to > either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I > would expect build_sdist to make some sanity checks about the world, then > make a binary choice about whether sdist creation is a valid goal. If not > possible, return None or NotImplemented or False or dict-of-reasons or > whatever. Only if creation was *attempted*, and in the exceptional event it > then failed, would I expect an Exception. We don't have structured > exceptions sadly so they can't really carry much useful information from a > protocol perspective above and beyond a simple None or the like anyway. > > I'd personally like to see some parity between build_sdist and build_wheel > in this regard. Maybe the disconnect here is we have a way to specify hard > reqs for building a wheel, statically or dynamically, and build_wheel is > expected to never fail, but no way to specify hard reqs needed for > build_sdist, necessitating this optional signaling path? Not sure what you mean about hard reqs. The reason for the lack of parity is that we don't currently have any use cases where build_wheel is expected to fail, but this is expected in some sense (not sure what that would even mean), and there's some fallback that the frontend may want to invoke instead. -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Sat Aug 26 18:21:50 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 17:21:50 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: Nathaniel: Just to clarify, we're talking about returning none for the build function not the get requirements function. The get requirements function is always expected to succeed and is optional. If while the backend is discovering requirements it finds that it cannot build, then it can return an empty list. It cannot however signal in the get requirements function that it cannot build. At least with what is being proposed. The reason that the proposal works is that the build function can never return none on success. On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger > wrote: > > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: > >> > >> [removed Guido from CC] > >> > >> On Aug 26, 2017 02:29, "Paul Moore" wrote: > >> > >> On 26 August 2017 at 03:17, Guido van Rossum wrote: > >> > In pretty much any other context, if you have an operation that > returns > >> > an > >> > regular value or an error value, the error value should be None. > >> > (Exceptions > >> > include e.g. returning a non-negative int or -1 for errors, or True > for > >> > success and False for errors.) > >> > >> So, given that build_sdist returns the path of the newly built sdist, > >> the correct way to signal "I didn't manage to build a sdist" would be > >> to return None. > >> > >> Now that it's put this way, it seems glaringly obvious to me that this > >> is the correct thing to do. > >> > >> > >> Eh... I would really prefer something that's (a) more explicit about > what > >> specifically went wrong, and (b) harder to return by accident. It's not > at > >> all obvious that if the list of requirements is 'None' that means 'this > >> build supports making sdists in general but cannot make them from this > >> source tree but might still be able to make a wheel'. And if you forget > to > >> put in a return statement, then python returns None for you, which seems > >> like it could lead to some super confusing error modes. > > > > > > Why does the frontend need to know why an sdist was not created? > > This whole discussion is about handling a specific case: suppose you > have a frontend like pip that when given a source directory and asked > to build a wheel, wants to implement that as: > - build sdist > - unpack sdist > - build wheel from unpacked sdist > > And suppose you have a backend like flit, that can build sdists from > some source directories (e.g. VCS checkouts) but not others (e.g. > unpacked sdists). We need some way for pip and flit to negotiate that > even though pip *normally* would implement its build-a-wheel operation > by first building an sdist, in this case it's ok to silently fall back > to some other strategy (like building the wheel directly in the source > tree, or manually copying the source tree somewhere else and then > building a wheel in it). > > But, we don't want this fallback behavior to hide real bugs. So if the > backend says "look, I just can't do sdists here, and that's an > expected thing, it's not something where the user needs to take any > particular action like filing a bug report or fixing their system or > anything like that, so if you have an alternative way to accomplish > what you're trying to do then you should just silently discard this > error and try that", ...cool. But if it doesn't explicitly say that, > then we don't want to silently discard the error and do something > else. > > It's taken a *lot* of back and forth to reach consensus that all we > need here is some special error signal from the *_sdist operations. > Let's focus on resolving that :-) > > > Frontend is asking the backend, given the current state of the world, to > > either produce an sdist, or not. Sans ahead-of-time knowledge (see > below), I > > would expect build_sdist to make some sanity checks about the world, then > > make a binary choice about whether sdist creation is a valid goal. If not > > possible, return None or NotImplemented or False or dict-of-reasons or > > whatever. Only if creation was *attempted*, and in the exceptional event > it > > then failed, would I expect an Exception. We don't have structured > > exceptions sadly so they can't really carry much useful information from > a > > protocol perspective above and beyond a simple None or the like anyway. > > > > I'd personally like to see some parity between build_sdist and > build_wheel > > in this regard. Maybe the disconnect here is we have a way to specify > hard > > reqs for building a wheel, statically or dynamically, and build_wheel is > > expected to never fail, but no way to specify hard reqs needed for > > build_sdist, necessitating this optional signaling path? > > Not sure what you mean about hard reqs. The reason for the lack of > parity is that we don't currently have any use cases where build_wheel > is expected to fail, but this is expected in some sense (not sure what > that would even mean), and there's some fallback that the frontend may > want to invoke instead. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Sat Aug 26 21:30:44 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Sat, 26 Aug 2017 20:30:44 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger wrote: > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: >> >> [removed Guido from CC] >> >> On Aug 26, 2017 02:29, "Paul Moore" wrote: >> >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> > an >> > regular value or an error value, the error value should be None. >> > (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> >> >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. > > > Why does the frontend need to know why an sdist was not created? This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-) Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written. What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake. Consider instead, if we make the following small changes: 1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary. 2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty). 3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support). 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static) 6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported. 7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself). This arrangement allows pip to know ahead-of-time if `build_sdist` is appropriate. If no sdist reqs are explicitly acknowledged, then no sdist can be created. Full stop. Example usages: * Backeds that only support wheel creation do not implement `get_requires_for_build_sdist` at all. * Backeds that conditionally support sdist creation implement `get_requires_for_build_sdist` and return None if unsupported. This is where flit signals "impossible" from an unpacked sdist and [...] from VCS. * Backeds that always support sdist creation (setuptools) implement `get_requires_for_build_sdist` and return [...] unconditionally. So for pip's sdist -> unpack -> wheel path, it knows right away if it should build an sdist or skip right to building a wheel because the backend will inform via `get_requires_for_build_sdist` instead of overloading `build_sdist`. I also think this achieves nice parity across sdist and wheel operations, keeps the same hooks optional, adds no special cases, and maybe even less overhead. What say you? -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Sat Aug 26 21:40:02 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Sat, 26 Aug 2017 20:40:02 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger wrote: > On Aug 26, 2017 2:17 PM, "Nathaniel Smith" wrote: >> >> [removed Guido from CC] >> >> On Aug 26, 2017 02:29, "Paul Moore" wrote: >> >> On 26 August 2017 at 03:17, Guido van Rossum wrote: >> > In pretty much any other context, if you have an operation that returns >> > an >> > regular value or an error value, the error value should be None. >> > (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> >> >> Eh... I would really prefer something that's (a) more explicit about what >> specifically went wrong, and (b) harder to return by accident. It's not at >> all obvious that if the list of requirements is 'None' that means 'this >> build supports making sdists in general but cannot make them from this >> source tree but might still be able to make a wheel'. And if you forget to >> put in a return statement, then python returns None for you, which seems >> like it could lead to some super confusing error modes. > > > Why does the frontend need to know why an sdist was not created? This whole discussion is about handling a specific case: suppose you have a frontend like pip that when given a source directory and asked to build a wheel, wants to implement that as: - build sdist - unpack sdist - build wheel from unpacked sdist And suppose you have a backend like flit, that can build sdists from some source directories (e.g. VCS checkouts) but not others (e.g. unpacked sdists). We need some way for pip and flit to negotiate that even though pip *normally* would implement its build-a-wheel operation by first building an sdist, in this case it's ok to silently fall back to some other strategy (like building the wheel directly in the source tree, or manually copying the source tree somewhere else and then building a wheel in it). But, we don't want this fallback behavior to hide real bugs. So if the backend says "look, I just can't do sdists here, and that's an expected thing, it's not something where the user needs to take any particular action like filing a bug report or fixing their system or anything like that, so if you have an alternative way to accomplish what you're trying to do then you should just silently discard this error and try that", ...cool. But if it doesn't explicitly say that, then we don't want to silently discard the error and do something else. It's taken a *lot* of back and forth to reach consensus that all we need here is some special error signal from the *_sdist operations. Let's focus on resolving that :-) Sure sure, I understand all that, and why we think we need some special error signal from `build_sdist`, as currently written. What I'm suggesting, is maybe calling `build_sdist` without knowing if it can succeed is already a mistake. Consider instead, if we make the following small changes: 1. `get_requires_for_build_*` is passed the sdist and wheel directories, just like `build_*`, giving them the chance to actually look at tree before deciding what other reqs might be necessary. 2. `get_requires_for_build_*` returns None to signal `build_*` is unsupported (superceded by static reqs defined in TOML) and [...] to signal support (can be empty). 3. `get_requires_for_build_*` assumed to return None if missing (so optional and implies no support). 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + `build-system.requires` (static) 6. If no reqs are found for sdist (no declared reqs in TOML and `get_requires_for_build_sdist` is missing or returns None), then `build_sdist` is unsupported. 7. If no reqs are found for wheel (no declared reqs in TOML and `get_requires_for_build_wheel` is missing or returns None), then `build_wheel` is unsupported. This one is a spec violation because at least one req is expected here (the backed itself). This arrangement allows pip to know ahead-of-time if `build_sdist` is appropriate. If no sdist reqs are explicitly acknowledged, then no sdist can be created. Full stop. Example usages: * Backeds that only support wheel creation do not implement `get_requires_for_build_sdist` at all. * Backeds that conditionally support sdist creation implement `get_requires_for_build_sdist` and return None if unsupported. This is where flit signals "impossible" from an unpacked sdist and [...] from VCS. * Backeds that always support sdist creation (setuptools) implement `get_requires_for_build_sdist` and return [...] unconditionally. So for pip's sdist -> unpack -> wheel path, it knows right away if it should build an sdist or skip right to building a wheel because the backend will inform via `get_requires_for_build_sdist` instead of overloading `build_sdist`. I also think this achieves nice parity across sdist and wheel operations, keeps the same hooks optional, adds no special cases, and maybe even less overhead. What say you? -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Aug 26 22:00:51 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 26 Aug 2017 19:00:51 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger wrote: > On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: > > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger > wrote: > > Sure sure, I understand all that, and why we think we need some special > error signal from `build_sdist`, as currently written. > > What I'm suggesting, is maybe calling `build_sdist` without knowing if it > can succeed is already a mistake. > > Consider instead, if we make the following small changes: > > 1. `get_requires_for_build_*` is passed the sdist and wheel directories, > just like `build_*`, giving them the chance to actually look at tree before > deciding what other reqs might be necessary. That's not a change, that's how it works :-). > 2. `get_requires_for_build_*` returns None to signal `build_*` is > unsupported (superceded by static reqs defined in TOML) and [...] to signal > support (can be empty). > > 3. `get_requires_for_build_*` assumed to return None if missing (so optional > and implies no support). This is what I originally proposed, except you use None where I use NotImplemented, which has the disadvantages I noted earlier. Also, people didn't like the missing get_requires_for_build_* being treated as no-support, which makes sense, since we expect that get_requires_for_build_* won't be used very often. But one can switch the default here without affecting much else. The reason we want to let build_sdist report failure is just for convenience of backends who don't have any other reason to implement get_requires_for_build_sdist. > 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) > > 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + > `build-system.requires` (static) build-system.requires contains the requirements that are always installed before we even try importing the backend, so they're available to all backend hooks equally. > 6. If no reqs are found for sdist (no declared reqs in TOML and > `get_requires_for_build_sdist` is missing or returns None), then > `build_sdist` is unsupported. > > 7. If no reqs are found for wheel (no declared reqs in TOML and > `get_requires_for_build_wheel` is missing or returns None), then > `build_wheel` is unsupported. This one is a spec violation because at least > one req is expected here (the backed itself). The TOML requires aren't really useful as a signal about whether sdist specifically is supported. Plus I think we probably want to leave no-requires-in-TOML as a valid option for saying "I don't need anything installed" (maybe because the backend is shipped inside the source tree) rather than overloading it to have extra meanings. -n -- Nathaniel J. Smith -- https://vorpus.org From c at anthonyrisinger.com Sun Aug 27 00:05:54 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Sat, 26 Aug 2017 23:05:54 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith wrote: > On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger > wrote: > > On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: > > > > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger > > wrote: > > > > Sure sure, I understand all that, and why we think we need some special > > error signal from `build_sdist`, as currently written. > > > > What I'm suggesting, is maybe calling `build_sdist` without knowing if it > > can succeed is already a mistake. > > > > Consider instead, if we make the following small changes: > > > > 1. `get_requires_for_build_*` is passed the sdist and wheel directories, > > just like `build_*`, giving them the chance to actually look at tree > before > > deciding what other reqs might be necessary. > > That's not a change, that's how it works :-). > Is that a change I missed from this thread? I'm reading here: https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel and they do not appear to receive the source or wheel directories. > > 2. `get_requires_for_build_*` returns None to signal `build_*` is > > unsupported (superceded by static reqs defined in TOML) and [...] to > signal > > support (can be empty). > > > > 3. `get_requires_for_build_*` assumed to return None if missing (so > optional > > and implies no support). > > This is what I originally proposed, except you use None where I use > NotImplemented, which has the disadvantages I noted earlier. Also, > people didn't like the missing get_requires_for_build_* being treated > as no-support, which makes sense, since we expect that > get_requires_for_build_* won't be used very often. But one can switch > the default here without affecting much else. The reason we want to > let build_sdist report failure is just for convenience of backends who > don't have any other reason to implement get_requires_for_build_sdist. > Oh OK, good good. Well in that case I agree with you and missed the suggestion. I personally prefer NotImplemented as well here but None seemed mostly just as good and did not elicit as much pushback. It's not too big of deal either way. However, a missing `get_requires_for_build_wheel` technically signaling "unsupported" makes good sense to me because it's always supplemented by the static *and mandatory* `build-system.requires` list. There is no proper way (without breaking the spec) to signal "unsupported" for `build_wheel` since the backend itself (setuptools, wheel, flit) is specified here. "Unsupported" is only signaled when *both* the static and dynamic requires are None (or NotImplemented as mentioned). The kicker here in my offering, is that the presence of `build-system.requires` *does not in any way imply* `build_sdist` support. As written, there is no way to statically set the requirements for sdist support (though this could be changed of course with a new TOML key/table), so you must explicitly signal it with something like: def get_requires_for_build_sdist(*args, **kwds): return [] This means, by default, `build_wheel` is "supported" and `build_sdist` is "unsupported", and both are no fail operations. If called, any exception is fatal to the entire process. If a backend goes through the work of supporting sdist creation, is it really a problem to relay this support with a 3 line function definition? Wheel (could also define nothing at all): def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None Flit: def get_requires_for_build_sdist(source_dir, ...): # I can output an sdist if the source directory is a VCS checkout I understand. requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_dir) return requires Setuptools: def get_requires_for_build_sdist(source_dir, ...): # I'm going to successfully create an sdist or die trying! # There is literally no directory I can't handle so I don't even look. return [] This seems pretty straightforward to me and avoids overloading `build_sdist`, keeping it no fail like `build_wheel`. Semantically, it's really the job of the requirements discovery mechanism to decide one of: a) Zero or more requirements are needed to transform target source_dir into an sdist. b) No requirement enables my backend to transform target source_dir into an sdist. This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely? > > 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) > > > > 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + > > `build-system.requires` (static) > > build-system.requires contains the requirements that are always > installed before we even try importing the backend, so they're > available to all backend hooks equally. > Yes absolutely. If said backend wants to also signal unconditional sdist support, it simply needs to: def get_requires_for_build_sdist(source_dir, ...): return [] and call it a day. The point of my `sdist reqs =` comment was to stress that while the presence of `build-system.requires` inherently signals `build_wheel` support, it *does not* signal `build_sdist` support. > > 6. If no reqs are found for sdist (no declared reqs in TOML and > > `get_requires_for_build_sdist` is missing or returns None), then > > `build_sdist` is unsupported. > > > > 7. If no reqs are found for wheel (no declared reqs in TOML and > > `get_requires_for_build_wheel` is missing or returns None), then > > `build_wheel` is unsupported. This one is a spec violation because at > least > > one req is expected here (the backed itself). > > The TOML requires aren't really useful as a signal about whether sdist > specifically is supported. Plus I think we probably want to leave > no-requires-in-TOML as a valid option for saying "I don't need > anything installed" (maybe because the backend is shipped inside the > source tree) rather than overloading it to have extra meanings. > The spec currently states that the `requires` key is mandatory, so `build_wheel` will unconditionally signal readiness. I was more referring to if we decided to add a separate key, like `sdist-requires`, to statically signal `build_sdist` readiness via TOML. You could use such a key to pull in a plugin for the chosen build backend, such as a hypothetical `flit.sdist` package, instead of baking the support into flit. Even without the addition of an `sdist-requires` TOML key, build backends can always signal sdist support dynamically by defining a basic `get_requires_for_build_sdist` returning an empty list. The most basic build backend is still one that defines only `build_wheel` and nothing else. If it also defines `build_sdist`, well great, but pip and friends will not use it unless it also defines an appropriate `get_requires_for_build_sdist`. This is an extremely low bar to me and achieves commonality in signature and behavior across all get_requires_* and build_* hooks. -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sun Aug 27 00:18:28 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 26 Aug 2017 23:18:28 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: > and they do not appear to receive the source or wheel directories. The source directory is the current directory, if I am not mistaken. > This lets `build_*` focus purely on building things straight away. There is a difference between "no more reqs are needed to do X" and "no possible req will achieve X" even though both add zero requirements. Why not let this hook relay it's decision more completely? Not trying to speak on behalf of flit here, but if I understand correctly, flit requires git to build a source distribution. Flit knows its build requirements, so it can just return them when get_requires is called. However, it needs to attempt to invoke git to find out whether it can build a source distribution, which if I understand correctly, is a lengthy operation (whether that's actually true is not actually relevant because we are discussion potentially any backend/operation). It's more efficient if git is only invoked once, and if a failure occurs, then under the proposal "None" would be returned from build_sdist rather than the name of the built source distribution. def get_requires_for_build_sdist(source_dir, ...): # I have no interest in sdists and I never will. # GO AWAY. return None I have to say that the above example is not a good idea. Perhaps I was a bit muddled on that point earlier: the reason that "return None" is a good idea for build sdist is, well: def build_sdist(...): # Indicates success return "x.tar.gz" def build_sdist(...): # Indicates failure return None Those are not truthily equivalent, which is important because it means that someone is unlikely to make a mistake on that matter. 2017-08-26 23:05 GMT-05:00 C Anthony Risinger : > On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith wrote: > >> On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger >> wrote: >> > On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: >> > >> > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger >> > wrote: >> > >> > Sure sure, I understand all that, and why we think we need some special >> > error signal from `build_sdist`, as currently written. >> > >> > What I'm suggesting, is maybe calling `build_sdist` without knowing if >> it >> > can succeed is already a mistake. >> > >> > Consider instead, if we make the following small changes: >> > >> > 1. `get_requires_for_build_*` is passed the sdist and wheel directories, >> > just like `build_*`, giving them the chance to actually look at tree >> before >> > deciding what other reqs might be necessary. >> >> That's not a change, that's how it works :-). >> > > Is that a change I missed from this thread? I'm reading here: > > https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 > https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 > https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist > https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel > > and they do not appear to receive the source or wheel directories. > > >> > 2. `get_requires_for_build_*` returns None to signal `build_*` is >> > unsupported (superceded by static reqs defined in TOML) and [...] to >> signal >> > support (can be empty). >> > >> > 3. `get_requires_for_build_*` assumed to return None if missing (so >> optional >> > and implies no support). >> >> This is what I originally proposed, except you use None where I use >> NotImplemented, which has the disadvantages I noted earlier. Also, >> people didn't like the missing get_requires_for_build_* being treated >> as no-support, which makes sense, since we expect that >> get_requires_for_build_* won't be used very often. But one can switch >> the default here without affecting much else. The reason we want to >> let build_sdist report failure is just for convenience of backends who >> don't have any other reason to implement get_requires_for_build_sdist. >> > > Oh OK, good good. Well in that case I agree with you and missed the > suggestion. I personally prefer NotImplemented as well here but None seemed > mostly just as good and did not elicit as much pushback. It's not too big > of deal either way. > > However, a missing `get_requires_for_build_wheel` technically signaling > "unsupported" makes good sense to me because it's always supplemented by > the static *and mandatory* `build-system.requires` list. There is no proper > way (without breaking the spec) to signal "unsupported" for `build_wheel` > since the backend itself (setuptools, wheel, flit) is specified here. > "Unsupported" is only signaled when *both* the static and dynamic requires > are None (or NotImplemented as mentioned). The kicker here in my offering, > is that the presence of `build-system.requires` *does not in any way imply* > `build_sdist` support. As written, there is no way to statically set the > requirements for sdist support (though this could be changed of course with > a new TOML key/table), so you must explicitly signal it with something like: > > def get_requires_for_build_sdist(*args, **kwds): return [] > > This means, by default, `build_wheel` is "supported" and `build_sdist` is > "unsupported", and both are no fail operations. If called, any exception is > fatal to the entire process. If a backend goes through the work of > supporting sdist creation, is it really a problem to relay this support > with a 3 line function definition? > > Wheel (could also define nothing at all): > > def get_requires_for_build_sdist(source_dir, ...): > # I have no interest in sdists and I never will. > # GO AWAY. > return None > > Flit: > > def get_requires_for_build_sdist(source_dir, ...): > # I can output an sdist if the source directory is a VCS checkout I > understand. > requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_ > dir) > return requires > > Setuptools: > > def get_requires_for_build_sdist(source_dir, ...): > # I'm going to successfully create an sdist or die trying! > # There is literally no directory I can't handle so I don't even look. > return [] > > This seems pretty straightforward to me and avoids overloading > `build_sdist`, keeping it no fail like `build_wheel`. > > Semantically, it's really the job of the requirements discovery mechanism > to decide one of: > > a) Zero or more requirements are needed to transform target source_dir > into an sdist. > b) No requirement enables my backend to transform target source_dir into > an sdist. > > This lets `build_*` focus purely on building things straight away. There > is a difference between "no more reqs are needed to do X" and "no possible > req will achieve X" even though both add zero requirements. Why not let > this hook relay it's decision more completely? > > >> > 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) >> > >> > 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + >> > `build-system.requires` (static) >> >> build-system.requires contains the requirements that are always >> installed before we even try importing the backend, so they're >> available to all backend hooks equally. >> > > Yes absolutely. If said backend wants to also signal unconditional sdist > support, it simply needs to: > > def get_requires_for_build_sdist(source_dir, ...): > return [] > > and call it a day. The point of my `sdist reqs =` comment was to stress > that while the presence of `build-system.requires` inherently signals > `build_wheel` support, it *does not* signal `build_sdist` support. > > >> > 6. If no reqs are found for sdist (no declared reqs in TOML and >> > `get_requires_for_build_sdist` is missing or returns None), then >> > `build_sdist` is unsupported. >> > >> > 7. If no reqs are found for wheel (no declared reqs in TOML and >> > `get_requires_for_build_wheel` is missing or returns None), then >> > `build_wheel` is unsupported. This one is a spec violation because at >> least >> > one req is expected here (the backed itself). >> >> The TOML requires aren't really useful as a signal about whether sdist >> specifically is supported. Plus I think we probably want to leave >> no-requires-in-TOML as a valid option for saying "I don't need >> anything installed" (maybe because the backend is shipped inside the >> source tree) rather than overloading it to have extra meanings. >> > > The spec currently states that the `requires` key is mandatory, so > `build_wheel` will unconditionally signal readiness. I was more referring > to if we decided to add a separate key, like `sdist-requires`, to > statically signal `build_sdist` readiness via TOML. You could use such a > key to pull in a plugin for the chosen build backend, such as a > hypothetical `flit.sdist` package, instead of baking the support into flit. > > Even without the addition of an `sdist-requires` TOML key, build backends > can always signal sdist support dynamically by defining a basic > `get_requires_for_build_sdist` returning an empty list. > > The most basic build backend is still one that defines only `build_wheel` > and nothing else. If it also defines `build_sdist`, well great, but pip and > friends will not use it unless it also defines an appropriate > `get_requires_for_build_sdist`. This is an extremely low bar to me and > achieves commonality in signature and behavior across all get_requires_* > and build_* hooks. > > -- > > C Anthony > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Sun Aug 27 00:21:06 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Sat, 26 Aug 2017 23:21:06 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Sat, Aug 26, 2017 at 11:05 PM, C Anthony Risinger wrote: > On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith wrote: > >> On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger >> wrote: >> > On Aug 26, 2017 5:13 PM, "Nathaniel Smith" wrote: >> > >> > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger >> > wrote: >> > >> > Sure sure, I understand all that, and why we think we need some special >> > error signal from `build_sdist`, as currently written. >> > >> > What I'm suggesting, is maybe calling `build_sdist` without knowing if >> it >> > can succeed is already a mistake. >> > >> > Consider instead, if we make the following small changes: >> > >> > 1. `get_requires_for_build_*` is passed the sdist and wheel directories, >> > just like `build_*`, giving them the chance to actually look at tree >> before >> > deciding what other reqs might be necessary. >> >> That's not a change, that's how it works :-). >> > > Is that a change I missed from this thread? I'm reading here: > > https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301 > https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254 > https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist > https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel > > and they do not appear to receive the source or wheel directories. > > >> > 2. `get_requires_for_build_*` returns None to signal `build_*` is >> > unsupported (superceded by static reqs defined in TOML) and [...] to >> signal >> > support (can be empty). >> > >> > 3. `get_requires_for_build_*` assumed to return None if missing (so >> optional >> > and implies no support). >> >> This is what I originally proposed, except you use None where I use >> NotImplemented, which has the disadvantages I noted earlier. Also, >> people didn't like the missing get_requires_for_build_* being treated >> as no-support, which makes sense, since we expect that >> get_requires_for_build_* won't be used very often. But one can switch >> the default here without affecting much else. The reason we want to >> let build_sdist report failure is just for convenience of backends who >> don't have any other reason to implement get_requires_for_build_sdist. >> > > Oh OK, good good. Well in that case I agree with you and missed the > suggestion. I personally prefer NotImplemented as well here but None seemed > mostly just as good and did not elicit as much pushback. It's not too big > of deal either way. > > However, a missing `get_requires_for_build_wheel` technically signaling > "unsupported" makes good sense to me because it's always supplemented by > the static *and mandatory* `build-system.requires` list. There is no proper > way (without breaking the spec) to signal "unsupported" for `build_wheel` > since the backend itself (setuptools, wheel, flit) is specified here. > "Unsupported" is only signaled when *both* the static and dynamic requires > are None (or NotImplemented as mentioned). The kicker here in my offering, > is that the presence of `build-system.requires` *does not in any way imply* > `build_sdist` support. As written, there is no way to statically set the > requirements for sdist support (though this could be changed of course with > a new TOML key/table), so you must explicitly signal it with something like: > > def get_requires_for_build_sdist(*args, **kwds): return [] > > This means, by default, `build_wheel` is "supported" and `build_sdist` is > "unsupported", and both are no fail operations. If called, any exception is > fatal to the entire process. If a backend goes through the work of > supporting sdist creation, is it really a problem to relay this support > with a 3 line function definition? > > Wheel (could also define nothing at all): > > def get_requires_for_build_sdist(source_dir, ...): > # I have no interest in sdists and I never will. > # GO AWAY. > return None > > Flit: > > def get_requires_for_build_sdist(source_dir, ...): > # I can output an sdist if the source directory is a VCS checkout I > understand. > requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_ > dir) > return requires > > Setuptools: > > def get_requires_for_build_sdist(source_dir, ...): > # I'm going to successfully create an sdist or die trying! > # There is literally no directory I can't handle so I don't even look. > return [] > > This seems pretty straightforward to me and avoids overloading > `build_sdist`, keeping it no fail like `build_wheel`. > > Semantically, it's really the job of the requirements discovery mechanism > to decide one of: > > a) Zero or more requirements are needed to transform target source_dir > into an sdist. > b) No requirement enables my backend to transform target source_dir into > an sdist. > > This lets `build_*` focus purely on building things straight away. There > is a difference between "no more reqs are needed to do X" and "no possible > req will achieve X" even though both add zero requirements. Why not let > this hook relay it's decision more completely? > > >> > 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static) >> > >> > 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) + >> > `build-system.requires` (static) >> >> build-system.requires contains the requirements that are always >> installed before we even try importing the backend, so they're >> available to all backend hooks equally. >> > > Yes absolutely. If said backend wants to also signal unconditional sdist > support, it simply needs to: > > def get_requires_for_build_sdist(source_dir, ...): > return [] > > and call it a day. The point of my `sdist reqs =` comment was to stress > that while the presence of `build-system.requires` inherently signals > `build_wheel` support, it *does not* signal `build_sdist` support. > > >> > 6. If no reqs are found for sdist (no declared reqs in TOML and >> > `get_requires_for_build_sdist` is missing or returns None), then >> > `build_sdist` is unsupported. >> > >> > 7. If no reqs are found for wheel (no declared reqs in TOML and >> > `get_requires_for_build_wheel` is missing or returns None), then >> > `build_wheel` is unsupported. This one is a spec violation because at >> least >> > one req is expected here (the backed itself). >> >> The TOML requires aren't really useful as a signal about whether sdist >> specifically is supported. Plus I think we probably want to leave >> no-requires-in-TOML as a valid option for saying "I don't need >> anything installed" (maybe because the backend is shipped inside the >> source tree) rather than overloading it to have extra meanings. >> > > The spec currently states that the `requires` key is mandatory, so > `build_wheel` will unconditionally signal readiness. I was more referring > to if we decided to add a separate key, like `sdist-requires`, to > statically signal `build_sdist` readiness via TOML. You could use such a > key to pull in a plugin for the chosen build backend, such as a > hypothetical `flit.sdist` package, instead of baking the support into flit. > > Even without the addition of an `sdist-requires` TOML key, build backends > can always signal sdist support dynamically by defining a basic > `get_requires_for_build_sdist` returning an empty list. > > The most basic build backend is still one that defines only `build_wheel` > and nothing else. If it also defines `build_sdist`, well great, but pip and > friends will not use it unless it also defines an appropriate > `get_requires_for_build_sdist`. This is an extremely low bar to me and > achieves commonality in signature and behavior across all get_requires_* > and build_* hooks. > Also, as I want to avoid focusing too much on the prospect of introducing a new TOML key for sdist requirements (which is likely unnecessary) flit could just as well use `get_requires_for_build_sdist` to dynamically decide if sdist creation is possible with something like: def get_requires_for_build_sdist(source_dir, ...): try: import flit.sdist except ImportError: return None requires = flit.sdist.get_requires_for_vcs_checkout_or_signal_unsupported(source_dir) return requires This `pyproject.toml` supports sdists: [build-system] requires = ["flit", "flit.sdist"] This `pyproject.toml` only supports wheels: [build-system] requires = ["flit"] -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Sun Aug 27 01:07:51 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Sun, 27 Aug 2017 00:07:51 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Sat, Aug 26, 2017 at 11:18 PM, xoviat wrote: > > and they do not appear to receive the source or wheel directories. > > The source directory is the current directory, if I am not mistaken. > Oh right right, for some reason I was reading the `build_*` hooks as receiving the input directory rather than the output directory. I don't see it stated anywhere that the PWD will be the source directory however. Might be a good addition. In this case, `get_requires_for_*` hooks should also be able to depend on PWD being the applicable source directory (which of course might be an unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel pipeline). > > This lets `build_*` focus purely on building things straight away. > There is a difference between "no more reqs are needed to do X" and "no > possible req will achieve X" even though both add zero requirements. Why > not let this hook relay it's decision more completely? > > Not trying to speak on behalf of flit here, but if I understand correctly, > flit requires git to build a source distribution. Flit knows its build > requirements, so it can just return them when get_requires is called. > However, it needs to attempt to invoke git to find out whether it can build > a source distribution, which if I understand correctly, is a lengthy > operation (whether that's actually true is not actually relevant because we > are discussion potentially any backend/operation). It's more efficient if > git is only invoked once, and if a failure occurs, then under the proposal > "None" would be returned from build_sdist rather than the name of the built > source distribution. > What is the benefit in making `build_sdist` handle this assertion? Requirements checking is only performed once, earlier than building. Somewhere, somehow, something must look to see if git exists. Is it not more appropriate to check early, while asserting python requirements, vs. actually being told to build an sdist? That is already the expectation on the wheel side of things since it would also fail if necessary system binaries are missing. Checking for git is conflating system reqs and python reqs as "requirements", but I don't see why the get_requires_* hooks can't do both if they so choose. > def get_requires_for_build_sdist(source_dir, ...): > # I have no interest in sdists and I never will. > # GO AWAY. > return None > > I have to say that the above example is not a good idea. Perhaps I was a > bit muddled on that point earlier: the reason that "return None" is a good > idea for build sdist is, well: > > def build_sdist(...): # Indicates success > return "x.tar.gz" > > def build_sdist(...): # Indicates failure > return None > > > Those are not truthily equivalent, which is important because it means > that someone is unlikely to make a mistake on that matter. > What is the difference? Both equally inform the frontend that building an sdist is not going to happen. At the end of the day, I'm not too partial to any of the solutions I've read thus far, and do not have any desire to derail progress. I just don't like the variation in `build_sdist` vs. `build_wheel` because it feels like a kludge to me. I thought perhaps defaulting sdists to "unsupported" and requiring a backend to opt-in was both more elegant and more explicit, allowing tools like flit to better express their conditional sdist support. -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sun Aug 27 01:28:45 2017 From: xoviat at gmail.com (xoviat) Date: Sun, 27 Aug 2017 00:28:45 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: The difference is that an empty list for requires would indicate that building is possible but none would not. Those two values are truthily equivalent. For building, the name returned indicates building was possible, but it's not truthily equivalent to none. On Aug 27, 2017 12:08 AM, "C Anthony Risinger" wrote: > On Sat, Aug 26, 2017 at 11:18 PM, xoviat wrote: > >> > and they do not appear to receive the source or wheel directories. >> >> The source directory is the current directory, if I am not mistaken. >> > > Oh right right, for some reason I was reading the `build_*` hooks as > receiving the input directory rather than the output directory. I don't > see it stated anywhere that the PWD will be the source directory however. > Might be a good addition. > > In this case, `get_requires_for_*` hooks should also be able to depend on > PWD being the applicable source directory (which of course might be an > unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel > pipeline). > > >> > This lets `build_*` focus purely on building things straight away. >> There is a difference between "no more reqs are needed to do X" and "no >> possible req will achieve X" even though both add zero requirements. Why >> not let this hook relay it's decision more completely? >> >> Not trying to speak on behalf of flit here, but if I understand >> correctly, flit requires git to build a source distribution. Flit knows its >> build requirements, so it can just return them when get_requires is called. >> However, it needs to attempt to invoke git to find out whether it can build >> a source distribution, which if I understand correctly, is a lengthy >> operation (whether that's actually true is not actually relevant because we >> are discussion potentially any backend/operation). It's more efficient if >> git is only invoked once, and if a failure occurs, then under the proposal >> "None" would be returned from build_sdist rather than the name of the built >> source distribution. >> > > What is the benefit in making `build_sdist` handle this assertion? > Requirements checking is only performed once, earlier than building. > Somewhere, somehow, something must look to see if git exists. Is it not > more appropriate to check early, while asserting python requirements, vs. > actually being told to build an sdist? That is already the expectation on > the wheel side of things since it would also fail if necessary system > binaries are missing. Checking for git is conflating system reqs and python > reqs as "requirements", but I don't see why the get_requires_* hooks can't > do both if they so choose. > > >> def get_requires_for_build_sdist(source_dir, ...): >> # I have no interest in sdists and I never will. >> # GO AWAY. >> return None >> >> I have to say that the above example is not a good idea. Perhaps I was a >> bit muddled on that point earlier: the reason that "return None" is a good >> idea for build sdist is, well: >> >> def build_sdist(...): # Indicates success >> return "x.tar.gz" >> >> def build_sdist(...): # Indicates failure >> return None >> >> >> Those are not truthily equivalent, which is important because it means >> that someone is unlikely to make a mistake on that matter. >> > > What is the difference? Both equally inform the frontend that building an > sdist is not going to happen. > > At the end of the day, I'm not too partial to any of the solutions I've > read thus far, and do not have any desire to derail progress. I just don't > like the variation in `build_sdist` vs. `build_wheel` because it feels like > a kludge to me. I thought perhaps defaulting sdists to "unsupported" and > requiring a backend to opt-in was both more elegant and more explicit, > allowing tools like flit to better express their conditional sdist support. > > -- > > C Anthony > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Aug 27 05:59:01 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 27 Aug 2017 10:59:01 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On 26 August 2017 at 23:13, Nathaniel Smith wrote: > This whole discussion is about handling a specific case: suppose you > have a frontend like pip that when given a source directory and asked > to build a wheel, wants to implement that as: > - build sdist > - unpack sdist > - build wheel from unpacked sdist While I can't speak for Donald or the other pip devs, I'd like to point out that in https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html I confirmed that I'm basically persuaded now that pip should just "trust the backend" and use build_wheel directly. So this whole debate isn't relevant for pip. We'll need to wait for other currently-hypothetical frontends like tox or twine which might use the build_wheel hook, before we'll have any practical experience to confirm if there's an issue with "return None" as the way for backends to signal that they weren't able to build a wheel. Paul From p.f.moore at gmail.com Sun Aug 27 06:37:17 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 27 Aug 2017 11:37:17 +0100 Subject: [Distutils] PEP 517 - a plan of action Message-ID: I feel that the discussion on PEP 517 is nearing an end. The outstanding questions are pretty detail-level, and there's little progress being made on them because the situations involved are mostly theoretical in the absence of actual implementations. So I propose the following plan of action: 1. Accept PEP 517 (at least provisionally). 2. Thomas to complete and publish his library of helper functions for PEP 517 consumers (optional - but I think he said he had such a thing in progress). 3. Build PEP 517 interfaces for flit and setuptools. 4. Implement PEP 517 support in pip. 5. Review what we learned from the above. 6. If needed, revise the PEP in view of actual implementation experience, and finalise it. Further frontends and backends can wait for the results of this process if they are concerned about tracking a provisional PEP. Hopefully, it shouldn't be too long a process (I'd like to see PEP 517 in the next release of pip, if possible). Notes ----- 1. On any remaining questions for the PEP, let's just go for the simplest solution for now. Let the PEP authors decide what that is, with Nick as BDFL arbitrating if there's disagreement. 2. In my view, the PEP 517 support in pip should fully embrace the principle in the PEP that it's the backend's job to ensure that sdist->wheel and build_wheel produce equivalent results (when both are possible), and should therefore *only* use build_wheel, at least until experience demonstrates that we have to protect users against badly written backends. 3. My suspicion is that the setuptools backend will pretty clearly demonstrate whether it's hard or easy to conform to the requirements in the PEP, which can inform pip's stance on a "pure build_wheel" approach. 4. If pip goes for direct wheel builds, this does mean that the build_sdist and get_requires_for_build_sdist hooks won't be exercised in any actual code during this process. IMO, that's OK, as those hooks cover important (if currently theoretical) cases for validating the "publish my project" workflow, and tools that want to address those cases (such as tox) are likely to need the sdist hooks later. If this seems like a reasonable plan, I'd suggest that people add links to issues or PRs for PEP 517 implementations in flit, setuptools and pip to this thread - as I'm sure all the participants here will want to review the concrete implementations. Please understand - I'm not trying to steamroller the process here, or force particular decisions. It's just that I know I'm burned out on the discussions, to the extent that I'm skimming mails and largely ignoring many of the hypothetical arguments, and I'm sure others are too. So we need to move to a point where we can discuss actual, in-development code rather than possibilities, and I believe the above plan gets us to that point as fast as possible. Paul From thomas at kluyver.me.uk Sun Aug 27 07:03:19 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sun, 27 Aug 2017 12:03:19 +0100 Subject: [Distutils] PEP 517 - a plan of action In-Reply-To: References: Message-ID: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> Thanks Paul for providing us with a way to move forwards. I'm in favour of what you propose. On Sun, Aug 27, 2017, at 11:37 AM, Paul Moore wrote: > 2. Thomas to complete and publish his library of helper functions for > PEP 517 consumers (optional - but I think he said he had such a thing > in progress). I do. It's not entirely up-to-date with the spec, but it can be found here: https://github.com/takluyver/pep517 My plan is to provide two levels of API: 1. Lower-level machinery to easily call PEP 517 hooks in a subprocess (this is what I've been writing so far) 2. A higher-level 'give me a wheel' interface which identifies the build dependencies, installs them into a build environment, and calls the build_wheel hook. Alternatively, this could be a separate package. I'd like to transfer this repo to PyPA once it's a bit more ready - assuming PyPA wants it. > 3. Build PEP 517 interfaces for flit and setuptools. The work for flit is well out of date, but it can be found here: https://github.com/takluyver/flit/tree/toml-config > 1. On any remaining questions for the PEP, let's just go for the > simplest solution for now. Let the PEP authors decide what that is, > with Nick as BDFL arbitrating if there's disagreement. I believe that the remaining questions are: - srcdir-on-sys.path - Notimplemented/Error (/None) Unfortunately, it's not clear to me what's the 'simplest' solution on either, and we can't easily change whichever decision we make. Nathaniel & Nick still disagree on srcdir-on-sys.path, I believe; I'm not sure how we resolve that. I still think the NotImplemented/Error issue is a prime example of bikeshedding. There clearly isn't one 'right' way to do it, because intelligent people continue to disagree about it. I'd like somebody to make the decision and end the argument, but I'm not sure who is 'above the fray' enough to be able to do that. > Please understand - I'm not trying to steamroller the process here, or > force particular decisions. It's just that I know I'm burned out on > the discussions, to the extent that I'm skimming mails and largely > ignoring many of the hypothetical arguments, and I'm sure others are > too. +1 From jeremy.kloth at gmail.com Sun Aug 27 08:41:58 2017 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Sun, 27 Aug 2017 06:41:58 -0600 Subject: [Distutils] PEP 517 - a plan of action In-Reply-To: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> References: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> Message-ID: On Sun, Aug 27, 2017 at 5:03 AM, Thomas Kluyver wrote: > I believe that the remaining questions are: > - srcdir-on-sys.path > - Notimplemented/Error (/None) > > Unfortunately, it's not clear to me what's the 'simplest' solution on > either, and we can't easily change whichever decision we make. Nathaniel > & Nick still disagree on srcdir-on-sys.path, I believe; I'm not sure how > we resolve that. > > I still think the NotImplemented/Error issue is a prime example of > bikeshedding. There clearly isn't one 'right' way to do it, because > intelligent people continue to disagree about it. I'd like somebody to > make the decision and end the argument, but I'm not sure who is 'above > the fray' enough to be able to do that. I just want to bring an additional point to the NotImplemented(Error)/None debate. It seems to me that this PEP is akin to the DB-API (https://www.python.org/dev/peps/pep-0249/). In DB-API, structured exceptions *are* used, just defined by the drivers themselves. It doesn't seem to be any problem for the driver implementations to provide them or for clients to handle them. That said, I would also lean towards its approach towards optional features. To quote: As with all DB API optional features, the database module authors are free to not implement these additional attributes and methods (using them will then result in an AttributeError) or to raise a NotSupportedError in case the availability can only be checked at run-time. Again, I see what is being developed here (PEP 517) as not that different from what has already been done with DB-API (PEP 249). If databases can achieve consensus, I'm sure that build tools can too. :) -- Jeremy Kloth From p.f.moore at gmail.com Sun Aug 27 09:00:22 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 27 Aug 2017 14:00:22 +0100 Subject: [Distutils] PEP 517 - a plan of action In-Reply-To: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> References: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> Message-ID: On 27 August 2017 at 12:03, Thomas Kluyver wrote: > I believe that the remaining questions are: > - srcdir-on-sys.path > - Notimplemented/Error (/None) > > Unfortunately, it's not clear to me what's the 'simplest' solution on > either, and we can't easily change whichever decision we make. Nathaniel > & Nick still disagree on srcdir-on-sys.path, I believe; I'm not sure how > we resolve that. > > I still think the NotImplemented/Error issue is a prime example of > bikeshedding. There clearly isn't one 'right' way to do it, because > intelligent people continue to disagree about it. I'd like somebody to > make the decision and end the argument, but I'm not sure who is 'above > the fray' enough to be able to do that. My feeling on this one (and I know, I'm biased here) is that Guido's suggestion of returning None to mean "couldn't build a sdist" is the simplest solution, at least for now. Initial frontend code will be sdist_path = build_sdist(...) if sdist_path is None: deal with no sdist and that's easy enough to modify to either sdist_path = build_sdist(...) if sdist_path is None or sdist_path is NotImplemented: deal with no sdist or try: sdist_path = build_sdist(...) # Backward compatibility if sdist_path is None: raise NotImplementedError except NotImplementedError: deal with no sdist Add to that the fact that I'm suggesting that there mightn't even be any call to build_sdist in pip, at least in the first instance, and it doesn't seem worth doing anything more complex for now. On the srcdir-on-sys.path question, I don't really have a good answer. It's part of the discussion I've been skimming/skipping - and the fact that I have been is one of the things that prompted me to make this suggestion. If you want, I'll go back and re-read the posts and see what my view is - but you may be happier if I don't add yet another viewpoint to confuse things further ;-) Paul From p.f.moore at gmail.com Sun Aug 27 09:04:57 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 27 Aug 2017 14:04:57 +0100 Subject: [Distutils] PEP 517 - a plan of action In-Reply-To: References: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> Message-ID: On 27 August 2017 at 13:41, Jeremy Kloth wrote: > Again, I see what is being developed here (PEP 517) as not that > different from what has already been done with DB-API (PEP 249). If > databases can achieve consensus, I'm sure that build tools can too. :) One difference with the DB-API is that DB exceptions are intended to be seen by the user, to be caught and reported in application code. The case we're discussing here is where a hook wants to report an unusual, but normal, condition (that the backend discovers *at runtime* that it can't do what has been requested of it) and the error is to be handled by "the other end of the protocol" (the front end) and is not for reporting to the user (except in the limited sense that we're considering whether the return value/exception needs to carry "why this failed" information that the frontend can report to the user if it wishes to). Paul From jeremy.kloth at gmail.com Sun Aug 27 09:23:35 2017 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Sun, 27 Aug 2017 07:23:35 -0600 Subject: [Distutils] PEP 517 - a plan of action In-Reply-To: References: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> Message-ID: On Sun, Aug 27, 2017 at 7:04 AM, Paul Moore wrote: > On 27 August 2017 at 13:41, Jeremy Kloth wrote: >> Again, I see what is being developed here (PEP 517) as not that >> different from what has already been done with DB-API (PEP 249). If >> databases can achieve consensus, I'm sure that build tools can too. :) > > One difference with the DB-API is that DB exceptions are intended to > be seen by the user, to be caught and reported in application code. > The case we're discussing here is where a hook wants to report an > unusual, but normal, condition (that the backend discovers *at > runtime* that it can't do what has been requested of it) and the error > is to be handled by "the other end of the protocol" (the front end) > and is not for reporting to the user (except in the limited sense that > we're considering whether the return value/exception needs to carry > "why this failed" information that the frontend can report to the user > if it wishes to). I guess to enhance my point, the correlation I see with DB-API is this: pip (frontend) --> sqlalchemy flit|setuptools|scons|make|my-awesome-build-tool (backend) --> mysql/ psycopg/sqlite In my usage of DB-API backends, I've certainly never thought of their exceptions as "user facing". Meaning that if a DB-API defined exception occurred, it was my responsibility as a developer to handle it; wrap it in an application-specific exception, format it for logging, retry the operation, what-have-you. Just never let it escape to the user of my application (in this case, the frontend, aka pip). Just attempting to bring a different perspective to the conversation. -- Jeremy Kloth From dholth at gmail.com Sun Aug 27 09:58:42 2017 From: dholth at gmail.com (Daniel Holth) Date: Sun, 27 Aug 2017 13:58:42 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: Return none makes sense, but I don't think the outcome of that or the "configurable src path" decisions will affect the success of the proposal. On Sun, Aug 27, 2017, 05:59 Paul Moore wrote: > On 26 August 2017 at 23:13, Nathaniel Smith wrote: > > This whole discussion is about handling a specific case: suppose you > > have a frontend like pip that when given a source directory and asked > > to build a wheel, wants to implement that as: > > - build sdist > > - unpack sdist > > - build wheel from unpacked sdist > > While I can't speak for Donald or the other pip devs, I'd like to > point out that in > https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html > I confirmed that I'm basically persuaded now that pip should just > "trust the backend" and use build_wheel directly. So this whole debate > isn't relevant for pip. We'll need to wait for other > currently-hypothetical frontends like tox or twine which might use the > build_wheel hook, before we'll have any practical experience to > confirm if there's an issue with "return None" as the way for backends > to signal that they weren't able to build a wheel. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Aug 27 10:35:29 2017 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 27 Aug 2017 16:35:29 +0200 Subject: [Distutils] Documentation link on PyPI.org Message-ID: Hi, Is it possible to create a documentation in the section with ?Project Links? on the page describing a release on PyPI.org ? That is, can I add some metadata to an sdist or wheel that would add an entry to that list, and if so how? I?m using setuptools to create the distribution artefacts. The reason I ask is that I currently use an index.html on pythonhosted.org that redirects to the real documentation because that adds a link to the documentation when looking at the release on pypi.python.org. That doesn?t work pypi.org. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Aug 27 11:33:54 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 27 Aug 2017 10:33:54 -0500 Subject: [Distutils] Documentation link on PyPI.org In-Reply-To: References: Message-ID: You can add a URL in the long_description (e.g. to https://read-the-docs.readthedocs.org/ ) It'd be useful to mention that here: https://packaging.python.org/tutorials/distributing-packages/ It would sure be great to be able to express with structured data that a project has [Sphinx] docs at a particular URL. It'd then be possible to discover and search docs for each project automatically. http://markmail.org/search/?q=project-url+documentation+list%3Aorg.python.distutils-sig "[Distutils] Provide separate development and documentation URLs in PyPI metadata?" http://markmail.org/search/?q=project-url+documentation+list%3Aorg.python.distutils-sig#query:project-url%20documentation%20list%3Aorg.python.distutils-sig+page:1+mid:e2ilbaevv2s3fl2j+state:results On Sunday, August 27, 2017, Ronald Oussoren wrote: > Hi, > > Is it possible to create a documentation in the section with ?Project > Links? on the page describing a release on PyPI.org? That is, can I add > some metadata to an sdist or wheel that would add an entry to that list, > and if so how? I?m using setuptools to create the distribution artefacts. > > The reason I ask is that I currently use an index.html on pythonhosted.org > that redirects to the real documentation because that adds a link to the > documentation when looking at the release on pypi.python.org. That > doesn?t work pypi.org. > > Ronald > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Aug 27 12:00:29 2017 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 27 Aug 2017 18:00:29 +0200 Subject: [Distutils] Documentation link on PyPI.org In-Reply-To: References: Message-ID: <7E1ABA81-9C79-4D02-8135-6AA409B1A540@mac.com> > On 27 Aug 2017, at 17:33, Wes Turner wrote: > > You can add a URL in the long_description (e.g. to https://read-the-docs.readthedocs.org/ ) I know, but that adds the URL to the project description and not the sidebar. PEP 345 appears to specify metadata that could be used for this, and warehouse seems to contain to make use of this (or at least a database model that allows for specifying more links than just the default one). Adding links to the sidebar would be nicer with the new design on PyPI.org , but there doesn?t seem to be a clean way to add ?Project-URL? metadata as described in PEP 345 to a wheel file. I could tweak my setup.py files to add Project-URL metadata, but before I do that I?d like to confirm that (a) that metadata is actually used by warehouse and (b) there is no cleaner way to add this metadata using setuptools. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Aug 27 18:12:11 2017 From: brett at python.org (Brett Cannon) Date: Sun, 27 Aug 2017 22:12:11 +0000 Subject: [Distutils] Documentation link on PyPI.org In-Reply-To: <7E1ABA81-9C79-4D02-8135-6AA409B1A540@mac.com> References: <7E1ABA81-9C79-4D02-8135-6AA409B1A540@mac.com> Message-ID: If you search the archive of this mailing list you will notice I asked this exact question about a month or 2 ago (I think). The answer I got was it is used on PyPI.org, but I don't know how to set it with setuptools (flit will probably add support after PEP 517). On Sun, Aug 27, 2017, 13:01 Ronald Oussoren wrote: > On 27 Aug 2017, at 17:33, Wes Turner wrote: > > You can add a URL in the long_description (e.g. to > https://read-the-docs.readthedocs.org/ ) > > > I know, but that adds the URL to the project description and not the > sidebar. PEP 345 appears to specify metadata that could be used for this, > and warehouse seems to contain to make use of this (or at least a database > model that allows for specifying more links than just the default one). > > Adding links to the sidebar would be nicer with the new design on PyPI.org, > but there doesn?t seem to be a clean way to add ?Project-URL? metadata as > described in PEP 345 to a wheel file. I could tweak my setup.py files to > add Project-URL metadata, but before I do that I?d like to confirm that (a) > that metadata is actually used by warehouse and (b) there is no cleaner way > to add this metadata using setuptools. > > Ronald > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Aug 27 19:27:23 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 28 Aug 2017 11:27:23 +1200 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: <59A3555B.5010603@canterbury.ac.nz> Nathaniel Smith wrote: > - creating an sdist failed for unexpected reasons, that need a human > to sort out (due to a broken system, or bugs ? hey, they happen ? or > ...) I think that should still be reported via an exception. Returning None should only be for the specific case that the backend doesn't support the requested operation. -- Greg From njs at pobox.com Sun Aug 27 19:47:03 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 27 Aug 2017 16:47:03 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: <59A3555B.5010603@canterbury.ac.nz> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> Message-ID: On Sun, Aug 27, 2017 at 4:27 PM, Greg Ewing wrote: > Nathaniel Smith wrote: >> >> - creating an sdist failed for unexpected reasons, that need a human >> to sort out (due to a broken system, or bugs ? hey, they happen ? or >> ...) > > > I think that should still be reported via an exception. Returning > None should only be for the specific case that the backend doesn't > support the requested operation. Well, you can't exactly say "if your code is buggy, then you should signal that by doing this well defined thing" :-). One of my objections to None is that it's very easy to return accidentally, i.e., buggy code *will* sometimes return None no matter what the spec says. -n -- Nathaniel J. Smith -- https://vorpus.org From thomas at kluyver.me.uk Mon Aug 28 04:43:48 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 28 Aug 2017 09:43:48 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> Message-ID: <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> If pip does uses build_wheel directly, as Paul now prefers, I think we can leave the NotImplemented/Error/None question for a later date. We only want some way to signal "I can't do that" because a frontend might try sdist->wheel with a fallback to making a wheel directly. If no frontend is actually planning to do that, we can leave specifying it until a frontend wants it. Donald, what do you think? IIRC, you were most keen on going sdist->wheel where possible, and I don't think you've commented on Paul's suggestion yet (apologies if I've overlooked a response). Thomas On Mon, Aug 28, 2017, at 12:47 AM, Nathaniel Smith wrote: > On Sun, Aug 27, 2017 at 4:27 PM, Greg Ewing > wrote: > > Nathaniel Smith wrote: > >> > >> - creating an sdist failed for unexpected reasons, that need a human > >> to sort out (due to a broken system, or bugs ? hey, they happen ? or > >> ...) > > > > > > I think that should still be reported via an exception. Returning > > None should only be for the specific case that the backend doesn't > > support the requested operation. > > Well, you can't exactly say "if your code is buggy, then you should > signal that by doing this well defined thing" :-). One of my > objections to None is that it's very easy to return accidentally, > i.e., buggy code *will* sometimes return None no matter what the spec > says. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From guettliml at thomas-guettler.de Mon Aug 28 07:05:07 2017 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Mon, 28 Aug 2017 13:05:07 +0200 Subject: [Distutils] PBR/distutils2 with pep517 Support Was: Conditionless setup.py Message-ID: <76c6aac9-113c-f9a2-86df-064cea942319@thomas-guettler.de> Am 25.08.2017 um 15:00 schrieb Paul Moore: > > One thought - are the PBR and/or distutils2 teams looking at providing > PEP 517 support? Assuming they are, have they had a change to review > the PEP to ensure it suits their needs? And if they aren't, what is it > about the PEP that makes them unwilling to do so? Hi Paul, I think this is a very good question! I started a new thread to make this more visible. Are there PBR/distutils2 hackers on this list here? If yes: Do you support this pep? If no: where can you meet PBR/distutils2 hackers? Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From fungi at yuggoth.org Mon Aug 28 07:46:25 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 28 Aug 2017 11:46:25 +0000 Subject: [Distutils] PBR/distutils2 with pep517 Support Was: Conditionless setup.py In-Reply-To: <76c6aac9-113c-f9a2-86df-064cea942319@thomas-guettler.de> References: <76c6aac9-113c-f9a2-86df-064cea942319@thomas-guettler.de> Message-ID: <20170828114625.GE2409@yuggoth.org> On 2017-08-28 13:05:07 +0200 (+0200), Thomas G?ttler wrote: [...] > Are there PBR/distutils2 hackers on this list here? > > If yes: Do you support this pep? > > If no: where can you meet PBR/distutils2 hackers? See: https://mail.python.org/pipermail/distutils-sig/2017-August/031315.html -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From donald at stufft.io Mon Aug 28 12:21:52 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 28 Aug 2017 12:21:52 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> Message-ID: <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> > On Aug 28, 2017, at 4:43 AM, Thomas Kluyver wrote: > > If pip does uses build_wheel directly, as Paul now prefers, I think we > can leave the NotImplemented/Error/None question for a later date. We > only want some way to signal "I can't do that" because a frontend might > try sdist->wheel with a fallback to making a wheel directly. If no > frontend is actually planning to do that, we can leave specifying it > until a frontend wants it. > > Donald, what do you think? IIRC, you were most keen on going > sdist->wheel where possible, and I don't think you've commented on > Paul's suggestion yet (apologies if I've overlooked a response). > I still think it should, and prefer pip to attempt to build a sdist prior to building a wheel when we?re coming from a VCS directory. I think that is going to be the most robust mechanism with the least amount of surprising behavior for end users. I also think it is important to define this fallback now, because doing it now means we have flexibility to adjust the strategy that frontends use without going back through the PEP process and waiting on backends to update to the latest version of the PEP. I am okay with an exception if it?s a specific exception, the idea of having an exception class exported as part of the API is fine with me. I think it?s still a worse API to use an exception for a non-exceptional case, but if it?s a specific exception it at least prevents bubbling up unrelated errors and having them be treated as asking for the fallback behavior if possible. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Aug 28 12:25:46 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Aug 2017 09:25:46 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On Sun, Aug 27, 2017 at 2:59 AM, Paul Moore wrote: > > Suppose you > > have a frontend like pip that when given a source directory and asked > > to build a wheel, wants to implement that as: > > - build sdist > > - unpack sdist > > - build wheel from unpacked sdist > > ... I'd like to > point out that in > https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html > I confirmed that I'm basically persuaded now that pip should just > "trust the backend" and use build_wheel directly. So this whole debate > isn't relevant for pip. Nor should it be for any other system. Before I saw this post I was about to make a point about not contorting ourselves to make things works the same way as pip and setuptools currently work -- pip was kind of hacked together to support setuptools, and setuptools grew kinda-organically from distutils (while also shoehorning in other not-really-building functionality). So let's make the "new" API clean and adapt the current tools to that. Specifically, I'm really confused about the sdist concept -- sure, it's a nice idea to have a standard way to create a source distributions, and I can see that tools that, for instance, create uploads for PyPi and the like will want to do that, but I don't see it as an inherent part of building a binary distribution (i.e. wheel), or building and installing a package directly. So: If you want a wheel, the build system should be asked to build a wheel. If you want an sdist, the build system should be asked to build an sdist -- and it can politely decline. Whether the build system first makes an sdist, and then builds a wheel from that it entirely up to the build system. Is there any reason to make this more complicated? We'll need to wait for other > currently-hypothetical frontends like tox or twine which might use the > build_wheel hook, before we'll have any practical experience to > confirm if there's an issue with "return None" as the way for backends > to signal that they weren't able to build a wheel. > Sure -- though I'm with Nathaniel on this one (I think -- kinda hard to keep th threads straight!) getting a NOne can happen for alot of reasons, better to have a more information-rich way to know that something failed. And isn't "be able to build a wheel" a requirement for ANY supported back-end? In which case, failure to build one is a failure that could have been caused by any number of issues -- shouldn't it simply fail with an Exception -- and the back end will hopefully generate nice exceptions for the common failures? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Aug 28 12:29:44 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Aug 2017 09:29:44 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> Message-ID: On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft wrote: > Donald, what do you think? IIRC, you were most keen on going > sdist->wheel where possible, and I don't think you've commented on > Paul's suggestion yet (apologies if I've overlooked a response). > > > I still think it should, and prefer pip to attempt to build a sdist prior > to building a wheel when we?re coming from a VCS directory. I think that is > going to be the most robust mechanism with the least amount of surprising > behavior for end users. > why? even as it stands, doesn't setuptools dump everything into the build dir anyway? The "Creating of an sdist" really seems like a build-system problem, not a package manager problem to me. > "when we?re coming from a VCS directory." whether this is a VCS directory or some other source seems to me like something pip should not need to know... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Aug 28 12:43:04 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Aug 2017 09:43:04 -0700 Subject: [Distutils] PBR/distutils2 with pep517 Support Was: Conditionless setup.py In-Reply-To: <76c6aac9-113c-f9a2-86df-064cea942319@thomas-guettler.de> References: <76c6aac9-113c-f9a2-86df-064cea942319@thomas-guettler.de> Message-ID: I've thought for ages that we could transition to a more sane system by convention: "your setup.py, after being imported, will have a "setup_params" attribute that is a dict that can be passed to setup()." So tools that want metadata, etc. without actually running an install could do; import setup name = setup.setup_params['name'] requirements = setup.setup_params['install_requires'] etc... i.e -- it does need to run the code in setup.py, but it does NOT need to run setup install, or build, or whatever else. A typical simple setup.py with static meta-data would be: from setuptools import setup setup_params = {name:'sample', version:'1.2.0', description:'A sample Python project', url:'https://github.com/pypa/sampleproject', author:'The Python Packaging Authority', license:'MIT', packages:['the_package'], install_requires=['peppercorn'], ... } setup(**setup_params) But a setup.py could still have arbitrary code to build up the dict of parameters if need be. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Aug 28 13:15:23 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 28 Aug 2017 13:15:23 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> Message-ID: > On Aug 28, 2017, at 12:29 PM, Chris Barker wrote: > > On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft > wrote: >> Donald, what do you think? IIRC, you were most keen on going >> sdist->wheel where possible, and I don't think you've commented on >> Paul's suggestion yet (apologies if I've overlooked a response). > > > I still think it should, and prefer pip to attempt to build a sdist prior to building a wheel when we?re coming from a VCS directory. I think that is going to be the most robust mechanism with the least amount of surprising behavior for end users. > > why? even as it stands, doesn't setuptools dump everything into the build dir anyway? The "Creating of an sdist" really seems like a build-system problem, not a package manager problem to me. > > > "when we?re coming from a VCS directory." > > whether this is a VCS directory or some other source seems to me like something pip should not need to know... > Differences between what files are in a sdist and what files are in a VCS directory will lead to different behaviors on install which makes ``pip install .`` and ``build-a-sdist && pip install the-sdist.tar.gz`` behave differently. Attempting to funnel everything through the same VCS -> sdist -> wheel path makes it less likely for these kinds of issues to occur. This is not a fully resolvable problem, and it is going to happen basically anytime you have two independent lists of what files get put into a sdist and what files get installed. It is not unique to setuptools nor is it a result of the way distutils/setuptools works? they only expose it more obviously because of their relevant APIs. Infact, both enscons and flit have this same problem (although flit has gone to some length to minimize the issue, so it?s somewhat hard, but not impossible, to actually trigger it). ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Aug 28 13:48:40 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 28 Aug 2017 17:48:40 +0000 Subject: [Distutils] PBR/distutils2 with pep517 Support Was: Conditionless setup.py In-Reply-To: References: <76c6aac9-113c-f9a2-86df-064cea942319@thomas-guettler.de> Message-ID: Imo PBR is entirely a setuptools creature, without special concerns to operate in pep517 land. If I were them I'd rewrite it to not require setup.py and call that pbr2. On Mon, Aug 28, 2017, 12:44 Chris Barker wrote: > I've thought for ages that we could transition to a more sane system by > convention: > > "your setup.py, after being imported, will have a "setup_params" attribute > that is a dict that can be passed to setup()." > > So tools that want metadata, etc. without actually running an install > could do; > > import setup > > name = setup.setup_params['name'] > requirements = setup.setup_params['install_requires'] > > etc... > > i.e -- it does need to run the code in setup.py, but it does NOT need to > run setup install, or build, or whatever else. > > > A typical simple setup.py with static meta-data would be: > > from setuptools import setup > > setup_params = {name:'sample', > version:'1.2.0', > description:'A sample Python project', > url:'https://github.com/pypa/sampleproject', > author:'The Python Packaging Authority', > license:'MIT', > packages:['the_package'], > install_requires=['peppercorn'], > ... > } > > setup(**setup_params) > > But a setup.py could still have arbitrary code to build up the dict of > parameters if need be. > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Aug 28 14:50:53 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 28 Aug 2017 13:50:53 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> Message-ID: Personally, my plan for the setuptools backend will be to build a source distribution (essentially using the command-line interface), extract into a tmpdir, and then build a wheel (essentially using the command line interface). So if pip calls build_sdist and then build_wheel, there will be two source distributions built (one by pip and one by setuptools) before building a wheel. There is not another way to do this that will comply with the specification because setuptools cannot currently be trusted to build a wheel directly. 2017-08-28 12:15 GMT-05:00 Donald Stufft : > > On Aug 28, 2017, at 12:29 PM, Chris Barker wrote: > > On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft wrote: > >> Donald, what do you think? IIRC, you were most keen on going >> sdist->wheel where possible, and I don't think you've commented on >> Paul's suggestion yet (apologies if I've overlooked a response). >> >> >> I still think it should, and prefer pip to attempt to build a sdist prior >> to building a wheel when we?re coming from a VCS directory. I think that is >> going to be the most robust mechanism with the least amount of surprising >> behavior for end users. >> > > why? even as it stands, doesn't setuptools dump everything into the build > dir anyway? The "Creating of an sdist" really seems like a build-system > problem, not a package manager problem to me. > > > "when we?re coming from a VCS directory." > > whether this is a VCS directory or some other source seems to me like > something pip should not need to know... > > > > Differences between what files are in a sdist and what files are in a VCS > directory will lead to different behaviors on install which makes ``pip > install .`` and ``build-a-sdist && pip install the-sdist.tar.gz`` behave > differently. Attempting to funnel everything through the same VCS -> sdist > -> wheel path makes it less likely for these kinds of issues to occur. > > This is not a fully resolvable problem, and it is going to happen > basically anytime you have two independent lists of what files get put into > a sdist and what files get installed. It is not unique to setuptools nor is > it a result of the way distutils/setuptools works? they only expose it more > obviously because of their relevant APIs. Infact, both enscons and flit > have this same problem (although flit has gone to some length to minimize > the issue, so it?s somewhat hard, but not impossible, to actually trigger > it). > > > ? > Donald Stufft > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Aug 28 14:53:09 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Aug 2017 19:53:09 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <65773AAE-2B84-4ACA-9316-11A5E027D691@stufft.io> <1503679767.3611126.1085057000.0D77BD5E@webmail.messagingengine.com> <66B87A34-56FD-4535-AFF7-B6300FE682CF@stufft.io> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> Message-ID: On 28 August 2017 at 17:25, Chris Barker wrote: > Nor should it be for any other system. Before I saw this post I was about to > make a point about not contorting ourselves to make things works the same > way as pip and setuptools currently work -- pip was kind of hacked together > to support setuptools, and setuptools grew kinda-organically from distutils > (while also shoehorning in other not-really-building functionality). > > So let's make the "new" API clean and adapt the current tools to that. > > Specifically, I'm really confused about the sdist concept -- sure, it's a > nice idea to have a standard way to create a source distributions, and I can > see that tools that, for instance, create uploads for PyPi and the like will > want to do that, but I don't see it as an inherent part of building a binary > distribution (i.e. wheel), or building and installing a package directly. This is *precisely* why I remain uncomfortable about the "just call build_wheel" approach. I agree (as I said) that going sdist->wheel in theory should make no difference, assuming that the backend follows the PEP. And so I'm in that sense persuaded that pip doesn't need to go sdist->wheel. However, I am *strongly* against allowing backends to skip the concept of a sdist altogether. Sufficiently so that I refused to use or support flit prior to it gaining support for building sdists. There's a problem of timing at the moment, in that we have not yet defined a new sdist format, so a sdist is currently loosely defined based on "what distutils makes, with some adhoc modifications to incorporate pyproject.toml. That's deliberate - we wanted to avoid PEP 517 getting bogged down in "sdist 2.0" discussions. But none of that matters to me - Python is an open source project, and a packaging infrastructure that isn't built around publishing sources is not something I want to support. And part of that is that we should be expecting build systems to support source publication, and expect projects to publish sources on PyPI (I do *not* consider a link to a project's github site, or other source repository, as sufficient - people can close their github accounts, projects can be abandoned - but PyPI should *always* contain what's needed to reproduce the distributions on there - and that means sources). So while I concede that pip might not *need* to build wheels via sdist, that does *not* mean I am in any way in favour of omitting "build a sdist" from the PEP 517 spec. I'm uncomfortable enough with allowing backends to have the ability to build a wheel when they can't build a sdist (and yes, that means I have reservations about how flit supports sdists). But for me, having a hook to build a sdist is non-negotiable. We've had debates in the past about whether building a sdist is in scope for the PEP - I do *not* want the discussion to go back to the point where we open up that question again. Paul From p.f.moore at gmail.com Mon Aug 28 15:20:28 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Aug 2017 20:20:28 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> Message-ID: On 28 August 2017 at 19:50, xoviat wrote: > Personally, my plan for the setuptools backend will be to build a source > distribution (essentially using the command-line interface), extract into a > tmpdir, and then build a wheel (essentially using the command line > interface). So if pip calls build_sdist and then build_wheel, there will be > two source distributions built (one by pip and one by setuptools) before > building a wheel. There is not another way to do this that will comply with > the specification because setuptools cannot currently be trusted to build a > wheel directly. This is precisely the question - whose responsibility is it to ensure that sdist->wheel vs build_wheel equivalence is maintained? The PEP says it's the backend's responsibility, so pip shouldn't need to do anything. But, we're currently having debates about "return None" which hinge on "if backends have bugs, return None might not work well". Sure - but if backends have bugs, they may not maintain wheel equivalence. I find this discrepancy immensely frustrating, as in one breath we're being told "trust the backend", and in the next we're being told "the backend may have bugs". To that extent, I'm with Donald - pip going sdist->wheel protects the user against the known-to-be-an-issue bug that the backend doesn't ensure wheel equivalence. But if we go with that, conscientious backend developers like xoviat suffer, because they take extra steps to do things correctly and as a result suffer bad performance. I've said I'm happy to trust the backend. But I'm starting to wonder if I should change that position: * Chris Barker has pointed out that backends have no reason to support sdists now. * Nathaniel is pushing a means of notifying "I can't build a sdist" that protects against backends accidentally not following the spec. * Donald has reservations (I don't fully understand them, except in the broad sense of "he wants pip to protect the user from problem cases" - which basically means "he doesn't trust the backend" - but in general I think they are valid) Should we trust the backend or not? Backends *will* have bugs - part of "trust the backend" is simply telling the user that the problem behaviour they found is not pip's issue and should be reported to the backend. Is that a sufficiently bad user experience that we should try to improve it (experience with setuptools says that it is - why are we assuming that developers of new backends will be so much more conscientious and careful than the setuptools developers? *I* certainly don't feel that the setuptools developers are unusually bad - quite the opposite!) Having a more robust means of saying "I can't build a sdist" than "return None" is protecting the user from issues with the backend. So is building via sdist. Donald's position is consistent - build via sdist and use a robust return. So was mine - build_wheel and return None. I don't want the PEP to end up defining an inconsistent position. And I'm starting to feel that conceding on build_wheel (and the "trust the backend" principle) was not seen in that context, but as conceding just the one item, leaving other aspects of trusting the backend open to debate. Maybe we go fully to Nick's proposal that we don't mandate any sort of consistency constraints in the PEP. That would mean pip *has* to go sdist->wheel (because pip does need consistent behaviour), and xoviat's setuptools backend can skip building a sdist on the way to building a wheel. It also means that the build_wheel hook is essentially unreliable in isolation, and that all frontends will likely have to do the build_sdist->build_wheel with fallback to inplace build_wheel dance that pip does. But we expect many less frontends than backends, so maybe that's the right trade-off? Paul From thomas at kluyver.me.uk Mon Aug 28 15:32:44 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 28 Aug 2017 20:32:44 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> Message-ID: <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote: > Maybe we go fully to Nick's proposal that we don't mandate any sort of > consistency constraints in the PEP. That would mean pip *has* to go > sdist->wheel (because pip does need consistent behaviour), and > xoviat's setuptools backend can skip building a sdist on the way to > building a wheel. It also means that the build_wheel hook is > essentially unreliable in isolation, and that all frontends will > likely have to do the build_sdist->build_wheel with fallback to > inplace build_wheel dance that pip does. But we expect many less > frontends than backends, so maybe that's the right trade-off? I've mentioned this before, but I have little faith in our ability to predict that one side of an interface will be far more numerous than the other. This is partly from my experience with Jupyter, where we got that prediction completely wrong. But I also think it's quite plausible that many frontend tools will want to use this interface (to argue against myself, most will be Python tools, so they could theoretically share a common wrapper module - but I have some doubts about whether they will). I'm not saying we shouldn't do this - as Donald is firmly in favour and you're wavering, it seems the easier option to wrap the discussion up. But I'm suspicious of the rationale that *there will be fewer frontends so they should have more responsibility*. Thomas From donald at stufft.io Mon Aug 28 15:47:58 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 28 Aug 2017 15:47:58 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> Message-ID: <49B5CAAA-5F89-457D-A16E-2FFF843ED2F7@stufft.io> > On Aug 28, 2017, at 3:32 PM, Thomas Kluyver wrote: > > On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote: >> Maybe we go fully to Nick's proposal that we don't mandate any sort of >> consistency constraints in the PEP. That would mean pip *has* to go >> sdist->wheel (because pip does need consistent behaviour), and >> xoviat's setuptools backend can skip building a sdist on the way to >> building a wheel. It also means that the build_wheel hook is >> essentially unreliable in isolation, and that all frontends will >> likely have to do the build_sdist->build_wheel with fallback to >> inplace build_wheel dance that pip does. But we expect many less >> frontends than backends, so maybe that's the right trade-off? > > I've mentioned this before, but I have little faith in our ability to > predict that one side of an interface will be far more numerous than the > other. This is partly from my experience with Jupyter, where we got that > prediction completely wrong. But I also think it's quite plausible that > many frontend tools will want to use this interface (to argue against > myself, most will be Python tools, so they could theoretically share a > common wrapper module - but I have some doubts about whether they will). > > I'm not saying we shouldn't do this - as Donald is firmly in favour and > you're wavering, it seems the easier option to wrap the discussion up. > But I'm suspicious of the rationale that *there will be fewer frontends > so they should have more responsibility*. > I don?t see anything wrong with saying *both* sides should be doing this when they can. Backends should attempt to be consistent where possible? while recognizing that is not 100% possible in every situation (for instance, flit when you?re in an archive from github.com w/o .git). Likewise front ends should decide how important that is to them, which can range from ?not important at all, YOLO? that just blindly calls build_wheel, to ?We?d like it, but we aren?t going to mandate it? that calls build_sdist and falls back to build_wheel, and ?We think this is a hard requirement? that calls build_sdist and fails if it can?t. My proposal isn?t exactly that I don?t trust the backends, it is that we don?t really know what the landscape is going to look at. Thus this proposal allows us to implement all three possible options without changing anything without mandating something that isn?t feasible (all backends must always be 100% consistent all the time). It would then be my desire that pip starts off with a pretty safe middle ground, try to build a sdist and fall back to directly building wheels that (A) helps to prevent issues where backends may have bugs or inadvertently are violating the ideal of a consistent build and (B) succeeds in most cases, even if (A) can?t be satisfied. Since we have flexibility we can then watch what happens, and if we find that a lot of problems are cropping up from inconsistent builds, then we can start talking about moving to mandating going via sdist OR if we find that the crop of build tools do a really good job and we think that we don?t really need the sanity check, migrate to just going straight to wheel. I think this is pretty important too, because how you want to handle not being able to create a sdist is going to be fairly task specific. Obviously if you can?t produce a sdist for a hypothetical ``twine sdist`` or ``pip sdist`` command, that is going to be a hard failure, but for ``pip install`` or ``pip wheel``, then there is a pretty reasonable argument that a fallback or straight to wheel is the correct answer. I also believe it is fundamentally impossible for the backends to guarantee consistency if they have a separate list for what gets installed vs what gets put into a sdist without literally building a sdist (or something similar)? and as I understand it one of the issues with that is that the tools don?t want to mandate being able to do that same logic in all situations. So I don?t think it works to say ?You *must* ensure a consistent output, and I think the only thing we can do is say that you *SHOULD* try to be consistent, and leave it up to front ends to decide how seriously they take that as a requirement. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Aug 28 15:48:18 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 28 Aug 2017 14:48:18 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> Message-ID: > But I'm suspicious of the rationale that *there will be fewer frontends so they should have more responsibility*. To be fair, pip is currently struggling to keep up with project requirements as it is, which has caused some frustration in the community (not that the frustration isn't wrong, but I don't think it's not going to resolve any problems). I don't see how having even more frontends will resolve these manpower issues. 2017-08-28 14:32 GMT-05:00 Thomas Kluyver : > On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote: > > Maybe we go fully to Nick's proposal that we don't mandate any sort of > > consistency constraints in the PEP. That would mean pip *has* to go > > sdist->wheel (because pip does need consistent behaviour), and > > xoviat's setuptools backend can skip building a sdist on the way to > > building a wheel. It also means that the build_wheel hook is > > essentially unreliable in isolation, and that all frontends will > > likely have to do the build_sdist->build_wheel with fallback to > > inplace build_wheel dance that pip does. But we expect many less > > frontends than backends, so maybe that's the right trade-off? > > I've mentioned this before, but I have little faith in our ability to > predict that one side of an interface will be far more numerous than the > other. This is partly from my experience with Jupyter, where we got that > prediction completely wrong. But I also think it's quite plausible that > many frontend tools will want to use this interface (to argue against > myself, most will be Python tools, so they could theoretically share a > common wrapper module - but I have some doubts about whether they will). > > I'm not saying we shouldn't do this - as Donald is firmly in favour and > you're wavering, it seems the easier option to wrap the discussion up. > But I'm suspicious of the rationale that *there will be fewer frontends > so they should have more responsibility*. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Aug 28 15:50:01 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Aug 2017 20:50:01 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> Message-ID: On 28 August 2017 at 20:32, Thomas Kluyver wrote: > On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote: >> Maybe we go fully to Nick's proposal that we don't mandate any sort of >> consistency constraints in the PEP. That would mean pip *has* to go >> sdist->wheel (because pip does need consistent behaviour), and >> xoviat's setuptools backend can skip building a sdist on the way to >> building a wheel. It also means that the build_wheel hook is >> essentially unreliable in isolation, and that all frontends will >> likely have to do the build_sdist->build_wheel with fallback to >> inplace build_wheel dance that pip does. But we expect many less >> frontends than backends, so maybe that's the right trade-off? > > I've mentioned this before, but I have little faith in our ability to > predict that one side of an interface will be far more numerous than the > other. This is partly from my experience with Jupyter, where we got that > prediction completely wrong. But I also think it's quite plausible that > many frontend tools will want to use this interface (to argue against > myself, most will be Python tools, so they could theoretically share a > common wrapper module - but I have some doubts about whether they will). > > I'm not saying we shouldn't do this - as Donald is firmly in favour and > you're wavering, it seems the easier option to wrap the discussion up. > But I'm suspicious of the rationale that *there will be fewer frontends > so they should have more responsibility*. Me too. At the moment, I only expect two backends of any substance - your flit backend and xoviat's setuptools one. But I only know of one frontend, namely pip - and talk of projects like tox or twine acting as frontends never seems to get any traction. The problem is that xoviat has to do a lot of work to provide guarantees that flit gets essentially for free, but flit can't provide the guaranteed sdist->wheel route that would let pip just go sdist->wheel cleanly. So there's a mismatch of requirements and *someone* needs to plaster over the cracks. At the moment, the debate is essentially whether that's xoviat or pip. Donald wants it to be pip because we can't guarantee that the next backend developer will be aware of all of the issues we debated here, and so we have to protect against that. I was persuaded[1] that we could document the requirements in the spec and expect that to be enough. I do worry that there's a risk that we're crippling the PEP with sufficient complexity that no-one will bother writing alternative frontends *or* additional backends (although I hope your support library might ease that problem somewhat). My main motivation for wavering is that I thought agreeing to trust the backend would simplify many of the decisions, and it's immensely frustrating to me that we're still debating the same question in the "return None" thread. Paul [1] I don't say "I thought" because I *never* thought backends would be sufficiently reliable. I just accepted that my concerns are only hypothetical. I wish others could do the same :-( From dholth at gmail.com Mon Aug 28 15:58:23 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 28 Aug 2017 19:58:23 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> Message-ID: On Mon, Aug 28, 2017 at 3:48 PM xoviat wrote: > > But I'm suspicious of the rationale that *there will be fewer frontends so > they should have more responsibility*. > > To be fair, pip is currently struggling to keep up with project > requirements as it is, which has caused some frustration in the community > (not that the frustration isn't wrong, but I don't think it's not going to > resolve any problems). I don't see how having even more frontends will > resolve these manpower issues. > Simple. It would be different people. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Aug 28 15:58:59 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Aug 2017 20:58:59 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: <49B5CAAA-5F89-457D-A16E-2FFF843ED2F7@stufft.io> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <49B5CAAA-5F89-457D-A16E-2FFF843ED2F7@stufft.io> Message-ID: On 28 August 2017 at 20:47, Donald Stufft wrote: > I also believe it is fundamentally impossible for the backends to guarantee > consistency if they have a separate list for what gets installed vs what > gets put into a sdist without literally building a sdist (or something > similar)? and as I understand it one of the issues with that is that the > tools don?t want to mandate being able to do that same logic in all > situations. And yet that's precisely what xoviat is intending to do, based on what the PEP says is needed. Are you saying he shouldn't? And the setuptools should simply expose the same consistency problems we've been dealing with until now? And the solution is for pip to switch to sdist->wheel build as we've always planned in order to resolve this issue? And any other future frontends will have to go through the same cycle that pip did? In which case, isn't the issue here that we've failed to argue sufficiently persuasively that all backends must provide a non-optional build_sdist hook, so we can do that cleanly? > So I don?t think it works to say ?You *must* ensure a consistent > output, and I think the only thing we can do is say that you *SHOULD* try to > be consistent, and leave it up to front ends to decide how seriously they > take that as a requirement. But how do frontends decide that? They have to base the decision on experience with backends. And it looks like the current crop of backends are split down the middle. Remember that if pip decides to go sdist->wheel, then that strongly motivates xoviat to abandon his current approach to ensure consistency, as it's duplicating the work pip will do. At which point the setuptools backend can't be assumed to be consistent, and the flit backend must be assumed to be (because we can't guarantee it can make a sdist). Paul From thomas at kluyver.me.uk Mon Aug 28 15:59:21 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 28 Aug 2017 20:59:21 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> Message-ID: <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> On Mon, Aug 28, 2017, at 08:50 PM, Paul Moore wrote: > My main motivation for wavering is that I thought agreeing to trust > the backend would simplify many of the decisions, and it's immensely > frustrating to me that we're still debating the same question in the > "return None" thread. The difference I see with the "return None" question is that there we have an alternative (return NotImplemented) which is just as simple for both sides, but avoids the identified issue with a buggy backend. The only argument there seems to be for using None is about semantics - and that's not a great argument, because 'practicality beats purity'. With the questions over sdist/wheel consistency, there's a complexity cost, for the spec and for frontends, in deciding that they can't. So we're weighing a trade-off: do we force ourselves to resolve the notimplemented question so that frontends can do sdist-wheel+fallback, or do we leave it up to frontends and risk some bugs which we might otherwise have prevented. Thomas From dholth at gmail.com Mon Aug 28 16:13:31 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 28 Aug 2017 20:13:31 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> Message-ID: Then end the debate by letting the PEP authors decide the return type, and write a paragraph explaining why the other options were rejected. It is not going to make a big difference. On Mon, Aug 28, 2017 at 3:59 PM Thomas Kluyver wrote: > On Mon, Aug 28, 2017, at 08:50 PM, Paul Moore wrote: > > My main motivation for wavering is that I thought agreeing to trust > > the backend would simplify many of the decisions, and it's immensely > > frustrating to me that we're still debating the same question in the > > "return None" thread. > > The difference I see with the "return None" question is that there we > have an alternative (return NotImplemented) which is just as simple for > both sides, but avoids the identified issue with a buggy backend. The > only argument there seems to be for using None is about semantics - and > that's not a great argument, because 'practicality beats purity'. > > With the questions over sdist/wheel consistency, there's a complexity > cost, for the spec and for frontends, in deciding that they can't. So > we're weighing a trade-off: do we force ourselves to resolve the > notimplemented question so that frontends can do sdist-wheel+fallback, > or do we leave it up to frontends and risk some bugs which we might > otherwise have prevented. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Aug 28 16:20:25 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 28 Aug 2017 16:20:25 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <49B5CAAA-5F89-457D-A16E-2FFF843ED2F7@stufft.io> Message-ID: <991735CE-7FB3-4EE5-8846-6545C505E352@stufft.io> > On Aug 28, 2017, at 3:58 PM, Paul Moore wrote: > > On 28 August 2017 at 20:47, Donald Stufft wrote: >> I also believe it is fundamentally impossible for the backends to guarantee >> consistency if they have a separate list for what gets installed vs what >> gets put into a sdist without literally building a sdist (or something >> similar)? and as I understand it one of the issues with that is that the >> tools don?t want to mandate being able to do that same logic in all >> situations. > > And yet that's precisely what xoviat is intending to do, based on what > the PEP says is needed. Are you saying he shouldn't? And the > setuptools should simply expose the same consistency problems we've > been dealing with until now? And the solution is for pip to switch to > sdist->wheel build as we've always planned in order to resolve this > issue? And any other future frontends will have to go through the same > cycle that pip did? I had some non-OSS stuff come up so I lost track of the discussion part way through so maybe I missed this recommendation, but I think that is going to be a bad path to go down because it is intermixing things in a way that I feel is going to result in a less optimal outcome. The problem is the backend doesn?t really know why we?re asking it to build a wheel to know whether we care or not about consistency guarantees. Future front ends can make their own decisions as it makes sense for their workflow. Something like Debian might choose to skip the sdist option (when they go straight from a VCS archive or w/e) because they have a human being sitting there ensuring that the install looks correct, and they are unlikely to want to go through sdist. > > In which case, isn't the issue here that we've failed to argue > sufficiently persuasively that all backends must provide a > non-optional build_sdist hook, so we can do that cleanly? I?m not sure what you mean by non-optional? Do you mean a build_Sdist hook that does not have a ?not implemented? flag? That would probably be my preference, but I?m happy to compromise on the not implemented return value/exception/whatever as a middle ground. Honestly this PEP has been drawn out enough with enough hypotheticals and corner cases that one of the main things I?m trying to optimize for is flexibility so we can handle different possible scenarios without going through this again. > >> So I don?t think it works to say ?You *must* ensure a consistent >> output, and I think the only thing we can do is say that you *SHOULD* try to >> be consistent, and leave it up to front ends to decide how seriously they >> take that as a requirement. > > But how do frontends decide that? They have to base the decision on > experience with backends. And it looks like the current crop of > backends are split down the middle. Remember that if pip decides to go > sdist->wheel, then that strongly motivates xoviat to abandon his > current approach to ensure consistency, as it's duplicating the work > pip will do. At which point the setuptools backend can't be assumed to > be consistent, and the flit backend must be assumed to be (because we > can't guarantee it can make a sdist). > > Well I don?t think it?s solely based on their experience with backends, but also what the general goal of their tool is. Something that is designed to run in CI might take a very strict stance to be extra sure to catch errors, something that runs in cases where you care more about performance than correctness in edge cases or specialized tools where they?ve done extra work and aren?t processing arbitrary Packages might just skip straight to building wheels. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Mon Aug 28 16:21:44 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 28 Aug 2017 20:21:44 +0000 Subject: [Distutils] PBR/distutils2 with pep517 Support Was: Conditionless setup.py In-Reply-To: References: <76c6aac9-113c-f9a2-86df-064cea942319@thomas-guettler.de> Message-ID: <20170828202143.GG2409@yuggoth.org> On 2017-08-28 17:48:40 +0000 (+0000), Daniel Holth wrote: > Imo PBR is entirely a setuptools creature, without special > concerns to operate in pep517 land. If I were them I'd rewrite it > to not require setup.py and call that pbr2. [...] While it's true that PBR relies on setuptools entrypoints _today_, the expectation is that it will also grow a PEP 517 build-system.build-backend compliant interface once the spec is officially approved. At least one problem with "just rewrite it" is that we expect to support people using old (pre-517) versions of pip for years to come, so will likely end up with some sort of hybrid implementation for as long as backwards compatibility needs to be maintained. It may also end up coupled with some manner of pre-build wrapper to consume/transform existing static metadata and emit a suitable pyproject.toml file, though that's for social reasons as much as anything. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From donald at stufft.io Mon Aug 28 16:23:52 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 28 Aug 2017 16:23:52 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> Message-ID: > On Aug 28, 2017, at 3:59 PM, Thomas Kluyver wrote: > > The difference I see with the "return None" question is that there we > have an alternative (return NotImplemented) which is just as simple for > both sides, but avoids the identified issue with a buggy backend. The > only argument there seems to be for using None is about semantics - and > that's not a great argument, because 'practicality beats purity'. > > With the questions over sdist/wheel consistency, there's a complexity > cost, for the spec and for frontends, in deciding that they can't. So > we're weighing a trade-off: do we force ourselves to resolve the > notimplemented question so that frontends can do sdist-wheel+fallback, > or do we leave it up to frontends and risk some bugs which we might > otherwise have prevented. I think the best option is to just export an exception that a frontend can catch. That gives like 95% of the benefit of either mechanism for an extra 5% of work. If a backend is unwilling to add a: class UnsupportedOperation(Exception): pass To their backend, then they?re likely not a backend that we should be spending a whole lot of time or effort trying to support. Two copy/pasteable lines of code is a tiny price to pay. I haven?t talked to Nick, but I?d be surprised if he was against this, since it?s basically what he wanted with a more specific exception (which is good practice anyways). ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Mon Aug 28 16:27:52 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 28 Aug 2017 21:27:52 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> Message-ID: <1503952072.3847891.1087759120.3E764075@webmail.messagingengine.com> On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote: > Then end the debate by letting the PEP authors decide the return type, > and write a paragraph explaining why the other options were rejected. > It is not going to make a big difference. Will that work now? Are we all so tired of this endless war that people will sign a peace treaty written by the people whose names are on the PEP (Nathaniel & me)? If so, let the trumpets sound, and the heralds declare that "return NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's preference right ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Aug 28 16:30:10 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 28 Aug 2017 15:30:10 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <1503952072.3847891.1087759120.3E764075@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> <1503952072.3847891.1087759120.3E764075@webmail.messagingengine.com> Message-ID: > If so, let the trumpets sound, and the heralds declare that "return NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's preference right ;-) For better or for worse, the trumpets seem to be sounding against this idea (Nathaniel seemed okay with whatever Donald and Nick thought was appropriate). 2017-08-28 15:27 GMT-05:00 Thomas Kluyver : > On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote: > > Then end the debate by letting the PEP authors decide the return type, and > write a paragraph explaining why the other options were rejected. It is not > going to make a big difference. > > > Will that work now? Are we all so tired of this endless war that people > will sign a peace treaty written by the people whose names are on the PEP > (Nathaniel & me)? > > If so, let the trumpets sound, and the heralds declare that "return > NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's > preference right ;-) > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Aug 28 19:28:19 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 28 Aug 2017 16:28:19 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: <1503952072.3847891.1087759120.3E764075@webmail.messagingengine.com> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> <1503952072.3847891.1087759120.3E764075@webmail.messagingengine.com> Message-ID: On Mon, Aug 28, 2017 at 1:27 PM, Thomas Kluyver wrote: > On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote: > > > Then end the debate by letting the PEP authors decide the return type, and > > write a paragraph explaining why the other options were rejected. It is not > > going to make a big difference. > > > Will that work now? Are we all so tired of this endless war that people will > sign a peace treaty written by the people whose names are on the PEP > (Nathaniel & me)? > > If so, let the trumpets sound, and the heralds declare that "return > NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's > preference right ;-) Fine with me, though if it turns out Donald and Nick prefer the version where the backend has to export an exception class then I'm fine with that too. (I'm basing this on -- Donald has said he likes it, and Nick hasn't commented yet but AFAICT it does address his concerns with NotImplemented, so it seem like a plausible outcome.) I hope Nick had a good weekend at the beach or whatever, because this is going to be a heck of an email backlog to come back to... -n -- Nathaniel J. Smith -- https://vorpus.org From chris.barker at noaa.gov Mon Aug 28 20:10:07 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Aug 2017 17:10:07 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> Message-ID: On Mon, Aug 28, 2017 at 12:20 PM, Paul Moore wrote: > > interface). So if pip calls build_sdist and then build_wheel, there will > be > > two source distributions built (one by pip and one by setuptools) before > > building a wheel. exactly why pip should NOT concern itself with the sdist -- either the back end needs to do it anyway, or it doesn't need to do it at all, and would only be doing so to satisfy pip. > This is precisely the question - whose responsibility is it to ensure > that sdist->wheel vs build_wheel equivalence is maintained? The PEP > says it's the backend's responsibility, so pip shouldn't need to do > anything. > makes sense to me. But, we're currently having debates about "return None" which hinge on > "if backends have bugs, return None might not work well". Sure - but > if backends have bugs, they may not maintain wheel equivalence. If backends have bugs (and they will), any number of things can go wrong. Making the spec so that front ends can have a better idea what went wrong is a fine idea, but it can only go so far. The only thing we really need to distinguish is " that functionality is not implemented" from "something went wrong" And not even that if we don't have pip trying to go through some machinations about what do if the back-end does not produce a sdist. > To that extent, I'm with Donald - pip going > sdist->wheel protects the user against the known-to-be-an-issue bug > that the backend doesn't ensure wheel equivalence. pip can not protect the user from a poorly written back-end. And it shouldn't try. * Chris Barker has pointed out that backends have no reason to support > sdists now. > not quite -- the reason to support sdist is because you want an sdist, not because it's a necessary part of the path to a wheel. I don't quite understand your (Paul's) point about the importance of sdists to open source (as opposed to a plain old tarball of the source, like what gitHub produces when you do a release. * Nathaniel is pushing a means of notifying "I can't build a sdist" > that protects against backends accidentally not following the spec. > "I can't do that" respond to any possible request makes sense. Unless "I can't build a sdist" violates the protocol -- which I don't think it should. > Should we trust the backend or not? Backends *will* have bugs - part > of "trust the backend" is simply telling the user that the problem > behaviour they found is not pip's issue and should be reported to the > backend. Is that a sufficiently bad user experience that we should try > to improve it (experience with setuptools says that it is - why are we > assuming that developers of new backends will be so much more > conscientious and careful than the setuptools developers? *I* > certainly don't feel that the setuptools developers are unusually bad > - quite the opposite!) > If that is a sufficiently bad user experience then the only other option is for "us" by some definition of "us" to build the back-end, too -- which is where this all started with setuptool. But my understanding is that goal is was not to force the whole stack to maintained together. And really, the end-user will need to report the problem to the package maintainer, not the build tool maintainer jsu tlike they should no if something doesn't pip install. Having a more robust means of saying "I can't build a sdist" than > "return None" is protecting the user from issues with the backend. So > is building via sdist. > but only one particular set of issues -- when there are an infinite number of possible issues... One other thought -- the easiest way to make an sdist it to simply copy the source tree. So: 1) not a big deal to require back-ends to do it -- it's easy to do but 2) back ends may do it the easy and sloppy way, and get build artifacts in the sdist, and then you are back where you started. so requiring the sdist step in no way saves you from poorly written back-ends. It may make it worse, as you _think_ you are getting something clean. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Aug 28 20:13:10 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Aug 2017 17:13:10 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> Message-ID: On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore wrote: > Me too. At the moment, I only expect two backends of any substance - > your flit backend and xoviat's setuptools one. But I only know of one > frontend, namely pip - and talk of projects like tox or twine acting > as frontends never seems to get any traction. > If the build API is clean and supported enough, I can see conda using it. just sayin' NOTE: it wouldn't be conda per-se using it -- but individual conda recipes, which generally call setuptools directly now. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Aug 29 03:03:14 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Aug 2017 17:03:14 +1000 Subject: [Distutils] PEP 517 - a plan of action In-Reply-To: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> References: <1503831799.2068119.1086348248.457297BE@webmail.messagingengine.com> Message-ID: On 27 August 2017 at 21:03, Thomas Kluyver wrote: > Thanks Paul for providing us with a way to move forwards. I'm in favour > of what you propose. > > On Sun, Aug 27, 2017, at 11:37 AM, Paul Moore wrote: >> 2. Thomas to complete and publish his library of helper functions for >> PEP 517 consumers (optional - but I think he said he had such a thing >> in progress). > > I do. It's not entirely up-to-date with the spec, but it can be found > here: > https://github.com/takluyver/pep517 > > My plan is to provide two levels of API: > 1. Lower-level machinery to easily call PEP 517 hooks in a subprocess > (this is what I've been writing so far) > 2. A higher-level 'give me a wheel' interface which identifies the build > dependencies, installs them into a build environment, and calls the > build_wheel hook. Alternatively, this could be a separate package. > > I'd like to transfer this repo to PyPA once it's a bit more ready - > assuming PyPA wants it. > >> 3. Build PEP 517 interfaces for flit and setuptools. > > The work for flit is well out of date, but it can be found here: > https://github.com/takluyver/flit/tree/toml-config > >> 1. On any remaining questions for the PEP, let's just go for the >> simplest solution for now. Let the PEP authors decide what that is, >> with Nick as BDFL arbitrating if there's disagreement. > > I believe that the remaining questions are: > - srcdir-on-sys.path > - Notimplemented/Error (/None) > > Unfortunately, it's not clear to me what's the 'simplest' solution on > either, and we can't easily change whichever decision we make. Nathaniel > & Nick still disagree on srcdir-on-sys.path, I believe; I'm not sure how > we resolve that. I've actually found the arguments in favour of leaving the source directory off sys.path by default fairly persuasive, as: 1. We have enough problems with Python's default sys.path[0] handling that I'm entirely sympathetic to the desire to omit it 2. If omitting it is genuinely a problem, we'll likely find out soon enough as part of implementing a setup.py backend 3. The `tools` section in pyproject.toml already offers a way to enable bootstrapping of a meta-backend that does `sys.path.insert(0, os.getcwd())` before loading a bundled backend So I think we can deem this one resolved in favour of "Frontends must ensure the current directory is *NOT* on sys.path before importing the designated backend", as starting there will mean we maximise our chances of learning something new as part of the initial rollout of the provisionally accepted API. > I still think the NotImplemented/Error issue is a prime example of > bikeshedding. There clearly isn't one 'right' way to do it, because > intelligent people continue to disagree about it. I'd like somebody to > make the decision and end the argument, but I'm not sure who is 'above > the fray' enough to be able to do that. ------- Note: TL;DR of the below recap is that I'd accept a PEP that used any of the following three designs: - backends optionally define an AnticipatedBuildFailure exception for frontends to catch (preferred) - anticipated build failures are indicated by raising NotImplementedError - anticipated build failures are indicated by returning None I *wouldn't* accept a return value based design that used NotImplemented as the special return value So the choice amongst those 3 options then devolves to the folks actually planning to do the work - while it's my least preferred of the three, an implemented None-return based API is nevertheless immeasurably more useful than a purely theoretical exception based API. ------- Taking another go at recapping this particular design question from my perspective: - the main point of calling the PEP 517 artifact creation hooks is for their side effect of actually creating the artifact, *not* their Python return value - as such, if they fail to create the requested artifact, they should raise an exception, just like any other state mutation operation - however, it's possible for artifact creation to fail for anticipated reasons, which can be signaled by raising a specific kind of exception describing that reason - we don't want to couple the API specification to any particular library, so we either need to choose an existing builtin exception, or else let the backend declare the exception it will use In other words, if I call an API to say "create this artifact", and I don't get an artifact out the other end, then that operation failed. It may have an anticipated reason for failing, and I may want to handle those anticipated failures differently from unexpected ones (so it's worth defining a protocol for distinguishing them), but it still failed. In the standard library, probably the best exemplar of this API design pattern would be the use of UnsupportedOperation in the `io` module base classes: https://docs.python.org/3/library/io.html#io.UnsupportedOperation - from the point of view of a running application, what matters about UnsupportedOperation is that it's an exception: it indicates that the requested operation failed and can be caught using the regular exception handlers. - from the point of view of a developer debugging a failed application, what matters about UnsupportedOperation is that it indicates that there's a program bug rather than an environmental problem. For PEP 517, the situation is somewhat similar: - from the point of view of a build pipeline, what matters about NotImplementedError is that it's an exception: it indicates that the artifact failed to be created. - from the point of view of defining fallbacks in a build pipeline, what matters about NotImplementedError is that it indicates an anticipated failure mode (such as missing VCS metadata or missing external dependencies), rather than a generic "we tried it and something went wrong". Whether or not that distinction matters will depend on the specific pipeline (e.g. if you're building for publication to PyPI, the distinction is irrelevant, and anticipated failures should be treated the same as any other failure, while if you're building purely for local installation purposes, you may decide that if an sdist build fails for anticipated reasons, you will instead fall back to building a wheel directly) So I'd still approve a version of the PEP that uses NotImplementedError to indicate anticipated failures. I'd also approve a version of the PEP that works similarly to the DB API (as Jeremy Kloth suggested): rather than picking a reasonable builtin exception type, we'd instead define a new backend attribute for "AnticipatedBuildFailure" (which would be required to be a subclass of "Exception"), and then frontends would catch that for anticipated failures. While it's slightly more work for frontends to handle compared to just picking a builtin exception type, this approach does allow backends to more easily avoid the risk of the libraries they call inadvertently reporting a nominally anticipated failure that is actually nothing of the sort. One added bonus of the exception type declaration approach is that it would allow backends to explicitly indicate that there *aren't* any anticipated build failures by skipping defining the AnticipatedBuildFailure attribute (frontends can substitute in an empty tuple in that case). If I'm understanding Donald's perspective correctly, he's defining failure slightly differently from the way I am: rather then defining it in terms of "Was the end user's goal (i.e. artifact creation) achieved?", he's instead defining it in terms of "Is this outcome in compliance with the interface specification between the API client and the API implementation?", and requesting that all spec compliant outcomes be indicated by a return value, and all spec non-complaint outcomes be indicated by an exception. While there are cases where that approach does make sense (using None as a default value for dict.get, using the NotImplemented singleton to speed up binary operand coercion while still allowing None as the result of a binary operation), I don't believe those rationales apply here - the subprocess invocations and any actual artifact creation will swamp any possible performance implications from raising and catching an exception, and we don't have any particular need to avoid requiring wrapping the API call in a try/except statement if an API client wants to special case the anticipated failures. That said, while it wouldn't be my preferred outcome, I *would* accept a version of the PEP that used "None" as an error indicator for the artifact creation APIs - that's then just a conventional "Optional[str]" return type, and while I think a structured exception would be a *better* way of indicating failure to create the requested artifact (since they provide a way for the backend to provide information on how to resolve the problem), I wouldn't find the use of None for this purpose objectionable the way I do the notion of using the NotImplemented singleton. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Aug 29 04:20:35 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Aug 2017 09:20:35 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> Message-ID: On 29 August 2017 at 01:13, Chris Barker wrote: > On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore wrote: >> >> Me too. At the moment, I only expect two backends of any substance - >> your flit backend and xoviat's setuptools one. But I only know of one >> frontend, namely pip - and talk of projects like tox or twine acting >> as frontends never seems to get any traction. > > > If the build API is clean and supported enough, I can see conda using it. > > just sayin' Thanks - that's really good to know. So *is* the build API clean enough for you? Specifically: 1. build_sdist can be missing, or can report back (somehow) that it failed to produce a sdist, and you should fall back to build_wheel. 2. build_wheel must always succeed and must always exist. 3. All hooks must be run in a separate subprocess There's also error handling, which I don't recall the details of, but I think boils down to the backend code can write what it likes to stdout/stderr and may raise an exception indicating "something blew up". Those are basically the key points at the moment. Add to that, does conda mind if build_wheel might result in a different wheel than build_sdist->build_wheel would produce? Paul From brett at python.org Tue Aug 29 12:51:20 2017 From: brett at python.org (Brett Cannon) Date: Tue, 29 Aug 2017 16:51:20 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <1138F66F-8F3B-4E3D-BACC-C4E694BF5E42@stufft.io> <59A0C81B.2010303@canterbury.ac.nz> <59A0CEAE.4060800@canterbury.ac.nz> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <1503950361.3841925.1087727096.3BEE060C@webmail.messagingengine.com> <1503952072.3847891.1087759120.3E764075@webmail.messagingengine.com> Message-ID: On Mon, 28 Aug 2017 at 16:29 Nathaniel Smith wrote: > On Mon, Aug 28, 2017 at 1:27 PM, Thomas Kluyver > wrote: > > On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote: > > > > > Then end the debate by letting the PEP authors decide the return type, > and > > > write a paragraph explaining why the other options were rejected. It > is not > > > going to make a big difference. > > > > > > Will that work now? Are we all so tired of this endless war that people > will > > sign a peace treaty written by the people whose names are on the PEP > > (Nathaniel & me)? > > > > If so, let the trumpets sound, and the heralds declare that "return > > NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's > > preference right ;-) > > Fine with me, though if it turns out Donald and Nick prefer the > version where the backend has to export an exception class then I'm > fine with that too. (I'm basing this on -- Donald has said he likes > it, and Nick hasn't commented yet but AFAICT it does address his > concerns with NotImplemented, so it seem like a plausible outcome.) > I loathe to weigh in on this and add yet another voice in this discussion, but the exported exception seems like the best solution for everyone involved from my lurking perspective. For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution). And I was fine with NotImplementedError but if people want something more specific and None is out due to worries of accidental bare returns, then the exported exception comes the closest to making everyone happy (it does tick the EIBTI box :) . -------------- next part -------------- An HTML attachment was scrubbed... URL: From paustin at eoas.ubc.ca Tue Aug 29 12:22:10 2017 From: paustin at eoas.ubc.ca (Phil Austin) Date: Tue, 29 Aug 2017 09:22:10 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> Message-ID: <24c16712-f45c-e66d-6bb9-709fc8876f63@eoas.ubc.ca> On 2017-08-29 01:20, Paul Moore wrote: > On 29 August 2017 at 01:13, Chris Barker wrote: >> On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore wrote: >>> >>> Me too. At the moment, I only expect two backends of any substance - >>> your flit backend and xoviat's setuptools one. But I only know of one >>> frontend, namely pip - and talk of projects like tox or twine acting >>> as frontends never seems to get any traction. scikit-build also expressed some interest: https://www.mail-archive.com/distutils-sig at python.org/msg26553.html although I don't see the pull request. >> >> >> If the build API is clean and supported enough, I can see conda using it. >> >> just sayin' > > Thanks - that's really good to know. So *is* the build API clean > enough for you? Specifically: > > 1. build_sdist can be missing, or can report back (somehow) that it > failed to produce a sdist, and you should fall back to build_wheel. > 2. build_wheel must always succeed and must always exist. > 3. All hooks must be run in a separate subprocess > > There's also error handling, which I don't recall the details of, but > I think boils down to the backend code can write what it likes to > stdout/stderr and may raise an exception indicating "something blew > up". > > Those are basically the key points at the moment. Add to that, does > conda mind if build_wheel might result in a different wheel than > build_sdist->build_wheel would produce? > conda-build doesn't produce or consume wheels. It works by creating a clean conda environment and running a shell script to install the python package into that environment (including its own versions of /usr/lib /usr/etc /usr/bin /urs/include /sbin -- which is why it can cope with non-python packages like g++, julia, R, git etc.). All files produced by the install script are then put into a tar file, which is conda's version of a wheel. From p.f.moore at gmail.com Tue Aug 29 14:13:57 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Aug 2017 19:13:57 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: <24c16712-f45c-e66d-6bb9-709fc8876f63@eoas.ubc.ca> References: <1503580261.2846467.1083603256.5479C27A@webmail.messagingengine.com> <59A3555B.5010603@canterbury.ac.nz> <1503909828.2697695.1087081360.4E01475F@webmail.messagingengine.com> <868DCABF-93AE-495D-AFA3-022D1F0C18BE@stufft.io> <1503948764.3836039.1087693736.1CD31012@webmail.messagingengine.com> <24c16712-f45c-e66d-6bb9-709fc8876f63@eoas.ubc.ca> Message-ID: On 29 August 2017 at 17:22, Phil Austin wrote: > On 2017-08-29 01:20, Paul Moore wrote: >> >> On 29 August 2017 at 01:13, Chris Barker wrote: >>> If the build API is clean and supported enough, I can see conda using it. >>> >>> just sayin' >> >> >> Thanks - that's really good to know. So *is* the build API clean >> enough for you? Specifically: >> >> 1. build_sdist can be missing, or can report back (somehow) that it >> failed to produce a sdist, and you should fall back to build_wheel. >> 2. build_wheel must always succeed and must always exist. >> 3. All hooks must be run in a separate subprocess >> >> There's also error handling, which I don't recall the details of, but >> I think boils down to the backend code can write what it likes to >> stdout/stderr and may raise an exception indicating "something blew >> up". >> >> Those are basically the key points at the moment. Add to that, does >> conda mind if build_wheel might result in a different wheel than >> build_sdist->build_wheel would produce? > > conda-build doesn't produce or consume wheels. It works by creating a clean > conda environment and running a shell script to install the python package > into that environment (including its own versions of /usr/lib /usr/etc > /usr/bin /urs/include /sbin -- which is why it can cope with non-python > packages like g++, julia, R, git etc.). All files produced by the install > script are then put into a tar file, which is conda's version of a wheel. OK, so I don't see how conda would use PEP 517 the way Chris suggested. But never mind - if it doesn't help conda that's fine. Paul From chris.barker at noaa.gov Tue Aug 29 14:45:26 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 29 Aug 2017 11:45:26 -0700 Subject: [Distutils] PEP 517 again Message-ID: > conda-build doesn't produce or consume wheels. It works by creating a > clean > > conda environment and running a shell script to install the python > package > > into that environment To be really clear -- conda build doesn't directly use ANY build/install system. All files produced by the install > > script are then put into a tar file, which is conda's version of a wheel. > right -- conda-build simply runs the script it is told to run (by the recipe), and then looks and sees what files were installed. In theory, one could build a conda package directly out of a binary wheel -- after all, as mentioned, they are actually pretty similar, but most folks would probably just use pip inside their conda-build recipe. In fact, that is already being done -- most python packages are built with "setup.py install", but some are built by using pip to install a wheel. But I didn't think that the goal of PEP 517 was to ensconce pip as the only way to install a package in Python. So if a build system can install (like setuptools currently does), then conda recipes may well call it directly. I don't think this really has any bearing on the the PEP, though -- As I understand it, the goal of the PEP is to formalize the interface between build systems and package managers -- and that can only be a good thing for other package managers. NOTE: For the most part, folks want to use a pip-compatible install in conda-build, because users tend to also directly use pip to install stuff in conda environments e.g. if there is not a conda package available. If a conda package were built with, say, plain distutils, then pip won't find the meta data it's looking for, and might falsely see an unresolved dependency. Personally, I've always avoided pip installing packages in conda -- preferring to make conda packages for all the deps I need, but conda-pip compatibility has come a long way, so it does work fine for many users. But it would be nice if packages didn't NEED to be pip installed -- but I think that's an issue with the wheel spec (that may well be solved). -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Aug 29 15:07:10 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Aug 2017 20:07:10 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: On 29 August 2017 at 19:45, Chris Barker wrote: > But it would be nice if packages didn't NEED to be pip installed -- but I > think that's an issue with the wheel spec (that may well be solved). Wheels don't need pip to install them - the spec has always been clear on how to "manually" install a wheel (unzip it and move a few things around), and the wheel package itself has a simple "wheel install" command. If you want to create wrapper scripts for entry points, you need to do a bit more work, but distlib offers that functionality (that's what pip uses). One point about PEP 517 is that it provides a standard way to ask *any* build system to make a wheel. So it's in theory possible to install any package without pip (of course, if you try to handle all the various corner cases, extra quirks, etc that come up in the real world, you end up reinventing pip ;-)). Paul From xoviat at gmail.com Tue Aug 29 15:30:32 2017 From: xoviat at gmail.com (xoviat) Date: Tue, 29 Aug 2017 14:30:32 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: > For me, using NotImplemented is a misuse of the singleton since I know what it's meant to be used for (and so I cringe every time I hear it brought up as a solution). I hate to reiterate too much, but I think that you are correct here: NotImplemented is not Pythonic. It's a complete hack that I would think people would like to quarantine to the limited use case of binary operations. Even with the poorly contextualized email that Guido received, I think it was clear that he thought it wasn't appropriate outside of that use case either (I don't think better context would have changed that). Personally, I was willing to overlook that for the sake of a speedy resolution. But reading the email chain, it seems that only Thomas and Daniel are currently in favor of using NotImplemented, which is a shift from the earlier situation. So now it seems that NotImplemented is not likely to be used. 2017-08-29 14:07 GMT-05:00 Paul Moore : > On 29 August 2017 at 19:45, Chris Barker wrote: > > But it would be nice if packages didn't NEED to be pip installed -- but I > > think that's an issue with the wheel spec (that may well be solved). > > Wheels don't need pip to install them - the spec has always been clear > on how to "manually" install a wheel (unzip it and move a few things > around), and the wheel package itself has a simple "wheel install" > command. > > If you want to create wrapper scripts for entry points, you need to do > a bit more work, but distlib offers that functionality (that's what > pip uses). > > One point about PEP 517 is that it provides a standard way to ask > *any* build system to make a wheel. So it's in theory possible to > install any package without pip (of course, if you try to handle all > the various corner cases, extra quirks, etc that come up in the real > world, you end up reinventing pip ;-)). > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Tue Aug 29 19:51:13 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 30 Aug 2017 00:51:13 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: Hi, On Tue, Aug 29, 2017 at 8:30 PM, xoviat wrote: >> For me, using NotImplemented is a misuse of the singleton since I know >> what it's meant to be used for (and so I cringe every time I hear it brought >> up as a solution). > > I hate to reiterate too much, but I think that you are correct here: > NotImplemented is not Pythonic. It's a complete hack that I would think > people would like to quarantine to the limited use case of binary > operations. Even with the poorly contextualized email that Guido received, I > think it was clear that he thought it wasn't appropriate outside of that use > case either (I don't think better context would have changed that). > > Personally, I was willing to overlook that for the sake of a speedy > resolution. But reading the email chain, it seems that only Thomas and > Daniel are currently in favor of using NotImplemented, which is a shift from > the earlier situation. So now it seems that NotImplemented is not likely to > be used. Just for the record, I thought Nathaniel's NotImplemented suggestion was a good one, and his explanation of why, was clear and convincing, but he's already accepted the alternative, of a custom error, so I don't think there's much point on going back over that. Also just for the record, I'd like to warmly thank y'all for your great patience in this long discussion. Cheers, Matthew From xoviat at gmail.com Wed Aug 30 19:29:41 2017 From: xoviat at gmail.com (xoviat) Date: Wed, 30 Aug 2017 18:29:41 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: So while I was working on pip, I noticed a parameter called 'autobuilding' in wheel.py. What this does (I think) is that if pip decides that it needs to be enabled (which is an implementation detail not relevant here), it adds the following to the "setup.py bdist_wheel" call: ["--python-tag", python_tag] This switches the output of the bdist_wheel filename to include something like "cp36" rather than "py36," which indicates the Python implementation. The question is this: this behavior isn't captured (I think) in the build_wheel function, possibly (probably) for good reason. So that means once everything goes through the PEP 517 protocol, pip will not be able to let setuptools know about this information. Which means that we need to decide: Should the PEP 517 backend for setuptools assume "cp36" in python_tag OR Should the PEP 517 backend for setuptools assume "py36" in python_tag 2017-08-29 18:51 GMT-05:00 Matthew Brett : > Hi, > > On Tue, Aug 29, 2017 at 8:30 PM, xoviat wrote: > >> For me, using NotImplemented is a misuse of the singleton since I know > >> what it's meant to be used for (and so I cringe every time I hear it > brought > >> up as a solution). > > > > I hate to reiterate too much, but I think that you are correct here: > > NotImplemented is not Pythonic. It's a complete hack that I would think > > people would like to quarantine to the limited use case of binary > > operations. Even with the poorly contextualized email that Guido > received, I > > think it was clear that he thought it wasn't appropriate outside of that > use > > case either (I don't think better context would have changed that). > > > > Personally, I was willing to overlook that for the sake of a speedy > > resolution. But reading the email chain, it seems that only Thomas and > > Daniel are currently in favor of using NotImplemented, which is a shift > from > > the earlier situation. So now it seems that NotImplemented is not likely > to > > be used. > > Just for the record, I thought Nathaniel's NotImplemented suggestion > was a good one, and his explanation of why, was clear and convincing, > but he's already accepted the alternative, of a custom error, so I > don't think there's much point on going back over that. > > Also just for the record, I'd like to warmly thank y'all for your > great patience in this long discussion. > > Cheers, > > Matthew > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 31 00:22:40 2017 From: xoviat at gmail.com (xoviat) Date: Wed, 30 Aug 2017 23:22:40 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: This appears to be related to https://github.com/pypa/pip/pull/3225, where pip would generate overbroad wheel names in the cache. So to proceed on this issue, setuptools will assume that generated wheels are valid for the specific python implementation only (eg. wheels generated for CPython will not be valid for pypy). Again, let me repeat that: wheels generated using setuptools are valid for CPython only if build on CPython. This is not the current setuptools behavior but will be for all setuptools build backend calls (I assume legacy will remain the same). Jacob, Paul, Donald, is everyone okay with that? And does everyone understand the reason for this? 2017-08-30 18:29 GMT-05:00 xoviat : > So while I was working on pip, I noticed a parameter called 'autobuilding' > in wheel.py. What this does (I think) is that if pip decides that it needs > to be enabled (which is an implementation detail not relevant here), it > adds the following to the "setup.py bdist_wheel" call: > > ["--python-tag", python_tag] > > This switches the output of the bdist_wheel filename to include something > like "cp36" rather than "py36," which indicates the Python implementation. > The question is this: this behavior isn't captured (I think) in the > build_wheel function, possibly (probably) for good reason. So that means > once everything goes through the PEP 517 protocol, pip will not be able to > let setuptools know about this information. Which means that we need to > decide: > > Should the PEP 517 backend for setuptools assume "cp36" in python_tag > OR > Should the PEP 517 backend for setuptools assume "py36" in python_tag > > > 2017-08-29 18:51 GMT-05:00 Matthew Brett : > >> Hi, >> >> On Tue, Aug 29, 2017 at 8:30 PM, xoviat wrote: >> >> For me, using NotImplemented is a misuse of the singleton since I know >> >> what it's meant to be used for (and so I cringe every time I hear it >> brought >> >> up as a solution). >> > >> > I hate to reiterate too much, but I think that you are correct here: >> > NotImplemented is not Pythonic. It's a complete hack that I would think >> > people would like to quarantine to the limited use case of binary >> > operations. Even with the poorly contextualized email that Guido >> received, I >> > think it was clear that he thought it wasn't appropriate outside of >> that use >> > case either (I don't think better context would have changed that). >> > >> > Personally, I was willing to overlook that for the sake of a speedy >> > resolution. But reading the email chain, it seems that only Thomas and >> > Daniel are currently in favor of using NotImplemented, which is a shift >> from >> > the earlier situation. So now it seems that NotImplemented is not >> likely to >> > be used. >> >> Just for the record, I thought Nathaniel's NotImplemented suggestion >> was a good one, and his explanation of why, was clear and convincing, >> but he's already accepted the alternative, of a custom error, so I >> don't think there's much point on going back over that. >> >> Also just for the record, I'd like to warmly thank y'all for your >> great patience in this long discussion. >> >> Cheers, >> >> Matthew >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Aug 31 00:56:00 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 31 Aug 2017 14:56:00 +1000 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: On 31 August 2017 at 14:22, xoviat wrote: > This appears to be related to https://github.com/pypa/pip/pull/3225, where > pip would generate overbroad wheel names in the cache. So to proceed on this > issue, setuptools will assume that generated wheels are valid for the > specific python implementation only (eg. wheels generated for CPython will > not be valid for pypy). > > Again, let me repeat that: wheels generated using setuptools are valid for > CPython only if build on CPython. This is not the current setuptools > behavior but will be for all setuptools build backend calls (I assume legacy > will remain the same). While I do think your proposal would work (on the assumption that folks aren't use "pip wheel" to generate their wheel files for upload), an alternative approach with a lower risk of unintended side effects would be for *pip* to either rename the autobuilt file before adding it to the cache, or else to adjust its caching strategy a bit to internally separate a shared wheel download cache from per-interpreter-compatibility-tag caches for locally built wheel files. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Thu Aug 31 01:36:55 2017 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 30 Aug 2017 22:36:55 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan wrote: > On 31 August 2017 at 14:22, xoviat wrote: >> Again, let me repeat that: wheels generated using setuptools are valid for >> CPython only if build on CPython. This is not the current setuptools >> behavior but will be for all setuptools build backend calls (I assume legacy >> will remain the same). > > While I do think your proposal would work (on the assumption that > folks aren't use "pip wheel" to generate their wheel files for > upload), I use 'pip wheel' to generate wheel files for upload... (I like to generate an sdist and then build a wheel from that, and 'pip wheel sdist.tar.gz' is more convenient than manually unpacking and running bdist_wheel. ) > an alternative approach with a lower risk of unintended side > effects would be for *pip* to either rename the autobuilt file before > adding it to the cache, or else to adjust its caching strategy a bit > to internally separate a shared wheel download cache from > per-interpreter-compatibility-tag caches for locally built wheel > files. +1 -n -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Thu Aug 31 02:37:20 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 31 Aug 2017 01:37:20 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: Will do. Better to hash this out sooner than later. On Aug 31, 2017 12:36 AM, "Nathaniel Smith" wrote: > On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan wrote: > > On 31 August 2017 at 14:22, xoviat wrote: > >> Again, let me repeat that: wheels generated using setuptools are valid > for > >> CPython only if build on CPython. This is not the current setuptools > >> behavior but will be for all setuptools build backend calls (I assume > legacy > >> will remain the same). > > > > While I do think your proposal would work (on the assumption that > > folks aren't use "pip wheel" to generate their wheel files for > > upload), > > I use 'pip wheel' to generate wheel files for upload... (I like to > generate an sdist and then build a wheel from that, and 'pip wheel > sdist.tar.gz' is more convenient than manually unpacking and running > bdist_wheel. ) > > > an alternative approach with a lower risk of unintended side > > effects would be for *pip* to either rename the autobuilt file before > > adding it to the cache, or else to adjust its caching strategy a bit > > to internally separate a shared wheel download cache from > > per-interpreter-compatibility-tag caches for locally built wheel > > files. > > +1 > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Aug 31 04:00:22 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Aug 2017 09:00:22 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: On 31 August 2017 at 06:36, Nathaniel Smith wrote: > On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan wrote: >> On 31 August 2017 at 14:22, xoviat wrote: >>> Again, let me repeat that: wheels generated using setuptools are valid for >>> CPython only if build on CPython. This is not the current setuptools >>> behavior but will be for all setuptools build backend calls (I assume legacy >>> will remain the same). >> >> While I do think your proposal would work (on the assumption that >> folks aren't use "pip wheel" to generate their wheel files for >> upload), > > I use 'pip wheel' to generate wheel files for upload... (I like to > generate an sdist and then build a wheel from that, and 'pip wheel > sdist.tar.gz' is more convenient than manually unpacking and running > bdist_wheel. ) Agreed. The wheels built by "pip wheel" mustn't change (and that means that neither pip nor the setuptools backend should not change the tags it applies to wheels by default). The fact that pip artificially restricts wheels it builds as part of its internal use to be platform-specific is a workaround for an underlying problem, which is that it cannot guarantee that an internally-built wheel will be the same as one that the project later publishes for that version (it's a variation on the wheel equivalency problem we've been discussing). Rather than artificially modifying the compatibility tags, pip should cache internally-built wheels differently, so that wheels published by the upstream project have priority over pip-built wheels. That's a change that purely affects pip and is unrelated to PEP 517 directly, but it's something pip probably needs to do to prepare for switching to PEP 517. (xoviat - sorry I didn't think of this when commenting on the PR - it was Nick's suggestion here that made the situation clearer to me). >> an alternative approach with a lower risk of unintended side >> effects would be for *pip* to either rename the autobuilt file before >> adding it to the cache, or else to adjust its caching strategy a bit >> to internally separate a shared wheel download cache from >> per-interpreter-compatibility-tag caches for locally built wheel >> files. > > +1 Yes. Renaming is a hack, to be honest, and improving the caching strategy would be my preference. Paul From chris.barker at noaa.gov Thu Aug 31 11:41:16 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 31 Aug 2017 08:41:16 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: <-6173407393987566310@unknownmsgid> > that neither pip nor the setuptools backend should not change the tags > it applies to wheels by default). I'm a bit confused -- are we talking about the backwards compatible path to the future -- or the end-game? In short -- I'm sure we'll have to do some hacky stuff to keep backwards compatibility, but the end game should be a clean separation of concerns : Pip (or any front end) should never "build a wheel", and it certainly shouldn't have to know enough about what's in a wheel to be re-naming it for generic python vs cpython. The package manager should manage the package, not built it, or change it. Surely the build system should know how to correctly name the wheel it builds. As to using pip to build wheels -- there is good reason to do that now, but in s post PEP 517 world, one would call the build system directly to build a wheel-- after all, all pip should be doing is calling the build system anyway. -CHB From p.f.moore at gmail.com Thu Aug 31 11:57:43 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Aug 2017 16:57:43 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: <-6173407393987566310@unknownmsgid> References: <-6173407393987566310@unknownmsgid> Message-ID: On 31 August 2017 at 16:41, Chris Barker - NOAA Federal wrote: > As to using pip to build wheels -- there is good reason to do that > now, but in s post PEP 517 world, one would call the build system > directly to build a wheel-- after all, all pip should be doing is > calling the build system anyway. I disagree - "pip wheel" will still be useful, for example to obtain a wheel from PyPI either by downloading or download & build. Also just to have a unified interface that works regardless of the project backend - if a project switches from setuptools to flit, for example, it would be good if deployment and test scripts didn't have to change. So I still expect pip wheel to be useful in a post-PEP 517 world. Paul From dholth at gmail.com Thu Aug 31 11:57:36 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 31 Aug 2017 15:57:36 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: <-6173407393987566310@unknownmsgid> References: <-6173407393987566310@unknownmsgid> Message-ID: I'm not against renaming a wheel to adjust the tags, but it is a strange caching strategy. wheel is due for a 'retag' feature that does a more rigorous job of changing tags. The build system should name its wheels correctly. Normally the pyNN tag would be used for a pure distribution and the cpNN tag when there are extensions. If you want to affect the tagging when called through PEP 517, edit the source's pyproject.toml / setup.cfg / setup.py. It will probably always make sense to use pip to build wheels when you want to build wheels for a project and all its dependencies. Then you can keep these wheels in a folder for repeatable offline deployment. On Thu, Aug 31, 2017 at 11:48 AM Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > > that neither pip nor the setuptools backend should not change the tags > > it applies to wheels by default). > > I'm a bit confused -- are we talking about the backwards compatible > path to the future -- or the end-game? > > In short -- I'm sure we'll have to do some hacky stuff to keep > backwards compatibility, but the end game should be a clean separation > of concerns : > > Pip (or any front end) should never "build a wheel", and it certainly > shouldn't have to know enough about what's in a wheel to be re-naming > it for generic python vs cpython. > > The package manager should manage the package, not built it, or change it. > > Surely the build system should know how to correctly name the wheel it > builds. > > As to using pip to build wheels -- there is good reason to do that > now, but in s post PEP 517 world, one would call the build system > directly to build a wheel-- after all, all pip should be doing is > calling the build system anyway. > > -CHB > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Aug 31 12:04:58 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 31 Aug 2017 12:04:58 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: <-6173407393987566310@unknownmsgid> References: <-6173407393987566310@unknownmsgid> Message-ID: <188DE1F5-936C-4036-8029-BCC18648C7E0@stufft.io> > On Aug 31, 2017, at 11:41 AM, Chris Barker - NOAA Federal wrote: > >> that neither pip nor the setuptools backend should not change the tags >> it applies to wheels by default). > > I'm a bit confused -- are we talking about the backwards compatible > path to the future -- or the end-game? > > In short -- I'm sure we'll have to do some hacky stuff to keep > backwards compatibility, but the end game should be a clean separation > of concerns : > > Pip (or any front end) should never "build a wheel", and it certainly > shouldn't have to know enough about what's in a wheel to be re-naming > it for generic python vs cpython. > > The package manager should manage the package, not built it, or change it. > > Surely the build system should know how to correctly name the wheel it builds. > > As to using pip to build wheels -- there is good reason to do that > now, but in s post PEP 517 world, one would call the build system > directly to build a wheel-- after all, all pip should be doing is > calling the build system anyway. > Pip purposely overrides what setuptools tags the wheel with in order to make it extremely specific to the Python it is currently being run on. Now, a big part of why it needs to do this, is because a lot of setup.py were not written with wheel in mind, so they produce broken wheels by default. One such example is a setup.py that produces a pure Python wheel for PyPy, and a C-extension wheel for Python. If someone installs that project on PyPy first and they only have a sdist available on PyPI, then pip will cache a pure Python wheel for that project, and by default (without this bit to force it) that wheel would be acceptable for CPython too and would have been used instead of building a wheel with a compiled extension for CPython. You could argue that the new crop of build backends should handle this better? and hopefully you?re correct? but I think that the two desires are at odd with one another. Most build backends are going to want to, by default, tag the wheel with as general of a tag as they think will work given the information available to them, whereas with pip?s internal wheel cache, we generally want that wheel to be as specific to the current version of Python as possible. Another possible solution is to stop using the wheel tagging format to encode this information for pip?s internal wheel cache and to encode it into the path, so that instead of having PyPy and CPython share a cache directory, they each have their own. That isn?t an unreasonable mechanism for doing that and which one gets used is an implementation detail of pip. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Aug 31 12:50:04 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 31 Aug 2017 09:50:04 -0700 Subject: [Distutils] PEP 517 again Message-ID: On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore wrote: > > As to using pip to build wheels -- there is good reason to do that > > now, but in s post PEP 517 world, one would call the build system > > directly to build a wheel-- after all, all pip should be doing is > > calling the build system anyway. > > I disagree - "pip wheel" will still be useful, for example to obtain a > wheel from PyPI either by downloading or download & build. Also just > to have a unified interface that works regardless of the project > backend - if a project switches from setuptools to flit, for example, > it would be good if deployment and test scripts didn't have to change. > Isn't that why we have PEP 517? I unified interface to build tools? So I still expect pip wheel to be useful in a post-PEP 517 world. Maybe so -- but all pip should be doing is passing off the job to the back-end. Again, the package manager, well, manages the packages. It shouldn't be concerned with how to build them. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Aug 31 12:58:18 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 31 Aug 2017 09:58:18 -0700 Subject: [Distutils] PEP 517 again Message-ID: On Thu, Aug 31, 2017 at 9:04 AM, Donald Stufft wrote: > I'm a bit confused -- are we talking about the backwards compatible > path to the future -- or the end-game? > > > Pip purposely overrides what setuptools tags the wheel with in order to > make it extremely specific to the Python it is currently being run on. Now, > a big part of why it needs to do this, is because a lot of setup.py were > not written with wheel in mind, so they produce broken wheels by default. > right -- just so we're clear that this is hack to maintain backward compatibility. Though that seems like a hack that setuptools should have, not pip. > One such example is a setup.py that produces a pure Python wheel for PyPy, > and a C-extension wheel for Python. If someone installs that project on > PyPy first and they only have a sdist available on PyPI, then pip will > cache a pure Python wheel for that project, and by default (without this > bit to force it) that wheel would be acceptable for CPython too and would > have been used instead of building a wheel with a compiled extension for > CPython. > Frankly, I'm not sure that pip should be sharing a cache between PyPy and cPython anyway.. (I've had other issues with pip's fairly aggressive caching in the past...) But if it does, then pip should always look to see if there is a more "specific" version of a package available anyway. But I see your point -- if there is a cached pure-python wheel, pip can go look on PyPi for a cpython wheel, but if it isn't there, and there is only an sdist, it would have to download and build it to find out if there is a cPython version. Which of course, would then happen (and be wasted effort) essentially all the time, as most packages are pure python. But the proper solution to this is for packages that have both a pure python and cpython version available to put the wheels up on pypi. After all, this is the unusual case. In fact, I see PEP 517 as a step toward more wheels, and fewer sdists for distribution. -CHB > You could argue that the new crop of build backends should handle this > better? and hopefully you?re correct? but I think that the two desires are > at odd with one another. Most build backends are going to want to, by > default, tag the wheel with as general of a tag as they think will work > given the information available to them, whereas with pip?s internal wheel > cache, we generally want that wheel to be as specific to the current > version of Python as possible. > > Another possible solution is to stop using the wheel tagging format to > encode this information for pip?s internal wheel cache and to encode it > into the path, so that instead of having PyPy and CPython share a cache > directory, they each have their own. That isn?t an unreasonable mechanism > for doing that and which one gets used is an implementation detail of pip. > > ? > Donald Stufft > > > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Aug 31 13:11:05 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 31 Aug 2017 13:11:05 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: > On Aug 31, 2017, at 12:58 PM, Chris Barker wrote: > > In fact, I see PEP 517 as a step toward more wheels, and fewer sdists for distribution. Everything should ideally still have a sdist. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Aug 31 13:04:29 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Aug 2017 18:04:29 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: On 31 August 2017 at 17:50, Chris Barker wrote: >> So I still expect pip wheel to be useful in a post-PEP 517 world. > > Maybe so -- but all pip should be doing is passing off the job to the > back-end. It also handles locating the appropriate version on PyPI, downloading (including caching the downloads), building via sdist to avoid discrepancies caused by out of date intermediates (yes, the backend should be doing this, let's not open up the "trust the backend" argument again ;-)) Pip also handles *installing* the appropriate backends into the build environment. Certainly "pip install ." is mostly redundant compared to "flit build" (although again, pip provides a unified command line interface - you *could* read pyproject.toml in your deployment script, then install the backend and call the build_wheel hook, but why do that rather than using at least *some* frontend, if not pip? > Again, the package manager, well, manages the packages. It shouldn't be > concerned with how to build them. It isn't - it just manages organising the process of getting the build tools and running them. A simpler competitor to pip that *only* works with wheels and knows nothing about the build process is entirely possible under PEP 517. Indeed such a tool probably doesn't even need PEP 517. But there hasn't been enough demand for such a tool to be built yet, so I doubt people will suddenly be crying out for one when PEP 517 is implemented... Paul From dholth at gmail.com Thu Aug 31 13:12:28 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 31 Aug 2017 17:12:28 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: Message-ID: On Thu, Aug 31, 2017 at 12:58 PM Chris Barker wrote: > On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore wrote: > >> > As to using pip to build wheels -- there is good reason to do that >> > now, but in s post PEP 517 world, one would call the build system >> > directly to build a wheel-- after all, all pip should be doing is >> > calling the build system anyway. >> >> I disagree - "pip wheel" will still be useful, for example to obtain a >> wheel from PyPI either by downloading or download & build. Also just >> to have a unified interface that works regardless of the project >> backend - if a project switches from setuptools to flit, for example, >> it would be good if deployment and test scripts didn't have to change. >> > > Isn't that why we have PEP 517? I unified interface to build tools? > > So I still expect pip wheel to be useful in a post-PEP 517 world. > > > Maybe so -- but all pip should be doing is passing off the job to the > back-end. > > Again, the package manager, well, manages the packages. It shouldn't be > concerned with how to build them. > pip has always been under appreciated as a build tool. Suppose the next version of pip dropped support for sdists and could only install wheels. Then it would not be a build tool. Very little changes when it switches to supporting sdists from its own setup.py machinery to a more pluggable mechanism. It is a little controversial that it can produce wheels for re-use but the feature is very practical. You could put 'pip wheel' under a different command line tool but why bother. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Aug 31 14:03:50 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 31 Aug 2017 11:03:50 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: <-6173407393987566310@unknownmsgid> References: <-6173407393987566310@unknownmsgid> Message-ID: On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal wrote: > The package manager should manage the package, not built it, or change it. > > Surely the build system should know how to correctly name the wheel it builds. It's probably worth mentioning the specific problem that motivated pip to start doing this. It used to be standard, and is still quite common, for setup.py scripts to contain stuff like: install_requires = [...] if sys.version_info < (3, 4): install_requires += [...] if platform.python_implementation() == "PyPy": install_requires += [...] setup(..., install_requires=install_requires) This kind of logic in setup.py worked fine in the old days when all you did was 'setup.py install', but then wheels came along and retroactively turned lots of setup.py scripts from working into broken. The problem is that with this kind of setup.py, setuptools has *no idea* that the install_requires you gave it would have been different if you had run setup.py with a different version of Python, so when it has to assign Python tags to a built wheel it guesses wrong and uses ones that are too general. The right way to do this is to use PEP 508 environment markers: https://www.python.org/dev/peps/pep-0508/#environment-markers or the non-standard extras hack: https://wheel.readthedocs.io/en/latest/#defining-conditional-dependencies Both of these let you export the whole requirements-choosing logic into the wheel metadata, so that it can be evaluated at install time instead of build time. But it will take a while for existing setup.py files transition to using those, and in the mean time pip can't assume that a random wheel generated by 'setup.py bdist_wheel' has accurate Python tags. Hopefully new legacy-free backends will get this right from the start. For example flit makes it impossible to get this wrong. -n -- Nathaniel J. Smith -- https://vorpus.org From dholth at gmail.com Thu Aug 31 14:19:20 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 31 Aug 2017 18:19:20 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: Don't be afraid of extras. They have not been PEP'd but are required for a useful Python packaging system. install_requires is just a shorthand for extras with the "" empty string as their name. Too bad we couldn't agree on ; vs : as the delimiter though. On Thu, Aug 31, 2017 at 2:04 PM Nathaniel Smith wrote: > On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal > wrote: > > The package manager should manage the package, not built it, or change > it. > > > > Surely the build system should know how to correctly name the wheel it > builds. > > It's probably worth mentioning the specific problem that motivated pip > to start doing this. > > It used to be standard, and is still quite common, for setup.py > scripts to contain stuff like: > > install_requires = [...] > if sys.version_info < (3, 4): > install_requires += [...] > if platform.python_implementation() == "PyPy": > install_requires += [...] > > setup(..., install_requires=install_requires) > > This kind of logic in setup.py worked fine in the old days when all > you did was 'setup.py install', but then wheels came along and > retroactively turned lots of setup.py scripts from working into > broken. The problem is that with this kind of setup.py, setuptools has > *no idea* that the install_requires you gave it would have been > different if you had run setup.py with a different version of Python, > so when it has to assign Python tags to a built wheel it guesses wrong > and uses ones that are too general. > > The right way to do this is to use PEP 508 environment markers: > https://www.python.org/dev/peps/pep-0508/#environment-markers > or the non-standard extras hack: > > https://wheel.readthedocs.io/en/latest/#defining-conditional-dependencies > Both of these let you export the whole requirements-choosing logic > into the wheel metadata, so that it can be evaluated at install time > instead of build time. > > But it will take a while for existing setup.py files transition to > using those, and in the mean time pip can't assume that a random wheel > generated by 'setup.py bdist_wheel' has accurate Python tags. > > Hopefully new legacy-free backends will get this right from the start. > For example flit makes it impossible to get this wrong. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 31 15:09:33 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 31 Aug 2017 14:09:33 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: There are two points that I would note here: 1. setuptools handling of environment markers in wheels was actually broken until recently. 2. Even now, people still prefer runtime detection of environment markers to work around pip's upgrade behavior and old setuptools versions. There needs to be a public effort to change this behavior. For #2, it's really not a clean solution, but it would be more effective to scan the bytecode of setup.py and then warn users that this behavior is deprecated. I expect that you would get a lot of complaints from package authors but that is the point. 2017-08-31 13:03 GMT-05:00 Nathaniel Smith : > On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal > wrote: > > The package manager should manage the package, not built it, or change > it. > > > > Surely the build system should know how to correctly name the wheel it > builds. > > It's probably worth mentioning the specific problem that motivated pip > to start doing this. > > It used to be standard, and is still quite common, for setup.py > scripts to contain stuff like: > > install_requires = [...] > if sys.version_info < (3, 4): > install_requires += [...] > if platform.python_implementation() == "PyPy": > install_requires += [...] > > setup(..., install_requires=install_requires) > > This kind of logic in setup.py worked fine in the old days when all > you did was 'setup.py install', but then wheels came along and > retroactively turned lots of setup.py scripts from working into > broken. The problem is that with this kind of setup.py, setuptools has > *no idea* that the install_requires you gave it would have been > different if you had run setup.py with a different version of Python, > so when it has to assign Python tags to a built wheel it guesses wrong > and uses ones that are too general. > > The right way to do this is to use PEP 508 environment markers: > https://www.python.org/dev/peps/pep-0508/#environment-markers > or the non-standard extras hack: > https://wheel.readthedocs.io/en/latest/#defining- > conditional-dependencies > Both of these let you export the whole requirements-choosing logic > into the wheel metadata, so that it can be evaluated at install time > instead of build time. > > But it will take a while for existing setup.py files transition to > using those, and in the mean time pip can't assume that a random wheel > generated by 'setup.py bdist_wheel' has accurate Python tags. > > Hopefully new legacy-free backends will get this right from the start. > For example flit makes it impossible to get this wrong. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Aug 31 17:29:41 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 31 Aug 2017 14:29:41 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith wrote: > > Surely the build system should know how to correctly name the wheel it > builds. > > It's probably worth mentioning the specific problem that motivated pip > to start doing this. > > It used to be standard, and is still quite common, for setup.py > scripts to contain stuff like: > > install_requires = [...] > if sys.version_info < (3, 4): > install_requires += [...] > if platform.python_implementation() == "PyPy": > install_requires += [...] > > setup(..., install_requires=install_requires) > > This kind of logic in setup.py worked fine in the old days when all > you did was 'setup.py install', but then wheels came along And indeed, setuptools originally used easy_install, which was part of setuptools... and > retroactively turned lots of setup.py scripts from working into > broken. The problem is that with this kind of setup.py, > > But it will take a while for existing setup.py files transition to > using those, and in the mean time pip can't assume that a random wheel > generated by 'setup.py bdist_wheel' has accurate Python tags. > This was my original point -- I understand that we want "pip install" to continue to work for, hopefully, everything it works for now. But I do think we should be clear about what is a hack for backward compatibility, and what is part of the designed functionality. Sorry to be poking at all this from the fringes (Not having been all that involved in a very long discussion), it's just that the whole distutils--setuptools--pip--distribute--setuptools--pip stack has a LOT of legacy cruft, and I'm concerned that the efforts for backward compatibility may end up leading us to another poorly de-coupled design. > Hopefully new legacy-free backends will get this right from the start. > exactly -- let's keep the "backward compatibility hack" labels clear! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Thu Aug 31 17:48:59 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 31 Aug 2017 16:48:59 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: I don't buy the argument that pip needs to keep separate caches for different implementations. According to the wheel specification, a wheel is valid for the Python implementations that it specifies. This is therefore a blatant violation by setuptools of an existing specification that, in theory, should be ignored by pip. That said, setuptools is currently the de facto build system, meaning that we cannot simply break the status quo. So here's my proposal: 1. For now, wheels are renamed if and only if the backend is setuptools. 2. In the future, we introduce a new setup() function that is imported from a different module path that requires compliance with the specification. 3. Once the vast majority of projects are using the new setup function, we can adjust the pep 517 setuptools build backend to use the most restrictive tag for the old function 4. We can finally remove autobuilding from pip. One thing to keep in mind is that there are quite a few projects on pypa with pure python source distributions uploaded that will not be updated and people may still desire to use. We want pip to be able to still build and install them. 2017-08-31 16:29 GMT-05:00 Chris Barker : > On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith wrote: > >> > Surely the build system should know how to correctly name the wheel it >> builds. >> >> It's probably worth mentioning the specific problem that motivated pip >> to start doing this. >> >> It used to be standard, and is still quite common, for setup.py >> scripts to contain stuff like: >> >> install_requires = [...] >> if sys.version_info < (3, 4): >> install_requires += [...] >> if platform.python_implementation() == "PyPy": >> install_requires += [...] >> >> setup(..., install_requires=install_requires) >> >> This kind of logic in setup.py worked fine in the old days when all >> you did was 'setup.py install', but then wheels came along > > > And indeed, setuptools originally used easy_install, which was part of > setuptools... > > and >> retroactively turned lots of setup.py scripts from working into >> broken. The problem is that with this kind of setup.py, >> > > >> But it will take a while for existing setup.py files transition to >> using those, and in the mean time pip can't assume that a random wheel >> generated by 'setup.py bdist_wheel' has accurate Python tags. >> > > This was my original point -- I understand that we want "pip install" to > continue to work for, hopefully, everything it works for now. > > But I do think we should be clear about what is a hack for backward > compatibility, and what is part of the designed functionality. > > Sorry to be poking at all this from the fringes (Not having been all that > involved in a very long discussion), it's just that the whole > > distutils--setuptools--pip--distribute--setuptools--pip > > stack has a LOT of legacy cruft, and I'm concerned that the efforts for > backward compatibility may end up leading us to another poorly de-coupled > design. > > >> Hopefully new legacy-free backends will get this right from the start. >> > > exactly -- let's keep the "backward compatibility hack" labels clear! > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Aug 31 18:49:01 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 31 Aug 2017 15:49:01 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: <9102857541681617612@unknownmsgid> One thing to keep in mind is that there are quite a few projects on pypa with pure python source distributions uploaded that will not be updated and people may still desire to use. We want pip to be able to still build and install them. That is the challenge! But the motivating use case here was a package with a c extension that was optionally built. That's the rare case. Regular old pure-python packages are the usual case, and a lot easier. -CHB 2017-08-31 16:29 GMT-05:00 Chris Barker : > On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith wrote: > >> > Surely the build system should know how to correctly name the wheel it >> builds. >> >> It's probably worth mentioning the specific problem that motivated pip >> to start doing this. >> >> It used to be standard, and is still quite common, for setup.py >> scripts to contain stuff like: >> >> install_requires = [...] >> if sys.version_info < (3, 4): >> install_requires += [...] >> if platform.python_implementation() == "PyPy": >> install_requires += [...] >> >> setup(..., install_requires=install_requires) >> >> This kind of logic in setup.py worked fine in the old days when all >> you did was 'setup.py install', but then wheels came along > > > And indeed, setuptools originally used easy_install, which was part of > setuptools... > > and >> retroactively turned lots of setup.py scripts from working into >> broken. The problem is that with this kind of setup.py, >> > > >> But it will take a while for existing setup.py files transition to >> using those, and in the mean time pip can't assume that a random wheel >> generated by 'setup.py bdist_wheel' has accurate Python tags. >> > > This was my original point -- I understand that we want "pip install" to > continue to work for, hopefully, everything it works for now. > > But I do think we should be clear about what is a hack for backward > compatibility, and what is part of the designed functionality. > > Sorry to be poking at all this from the fringes (Not having been all that > involved in a very long discussion), it's just that the whole > > distutils--setuptools--pip--distribute--setuptools--pip > > stack has a LOT of legacy cruft, and I'm concerned that the efforts for > backward compatibility may end up leading us to another poorly de-coupled > design. > > >> Hopefully new legacy-free backends will get this right from the start. >> > > exactly -- let's keep the "backward compatibility hack" labels clear! > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: