From ncoghlan at gmail.com Fri Sep 1 00:23:31 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Sep 2017 14:23:31 +1000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: On 1 September 2017 at 07:29, Chris Barker wrote: > But I do think we should be clear about what is a hack for backward > compatibility, and what is part of the designed functionality. Right, and I think it's pretty clear that the problem xoviat raised is with the way pip's implicit local artifact cache works, since it doesn't reliably distinguish between "this cached wheel was downloaded from a repository" and "this wheel was generated locally with a particular version of Python". Currently pip is working around that problem by overriding the compatibility tags produced by setup.py to ensure the generated wheel files are interpreter specific, but PEP 517 deliberately doesn't let frontends do that as part of the initial build process (instead, if they want to adjust the tags, they need to do it as a post-processing step). Since PEP 517 breaks the current workaround for the caching scheme being inaccurate, the most suitable response is to instead fix pip's caching scheme to use a two tier local cache: 1. Wheel download cache (if this gets a hit, don't even check the package repo) 2. Check the package repo for a wheel file (and if you find one, add it to the download cache) 3. Local build cache (make this interpreter implementation specific) 4. Build a local wheel that's specific to the running interpreter implementation Independently of that though, it's also important to note that pip probably *won't* switch to invoking legacy setup.py files through the PEP 517 backend - instead, it will only do that if there's a pyproject.toml file that says "Use the setuptools PEP 517 backend". In such cases, it's reasonable to expect that the project will also be using environment markers appropriately, and not putting conditional dependency installation logic directly into setup.py. Autobuilding support definitely *won't* be removed from pip, as there will not only always be a large number of projects that don't provide their own prebuilt wheel files, we also have plenty of PyPI users that explicitly *opt out* of using publisher-provided pre-built binaries. While Linux distributions are the most common example (see [1] for Fedora's policy, for example), we're not the only ones that have that kind of rule in place. Cheers, Nick. [1] https://fedoraproject.org/wiki/Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Fri Sep 1 01:29:17 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 1 Sep 2017 01:29:17 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: > On Sep 1, 2017, at 12:23 AM, Nick Coghlan wrote: > > 1. Wheel download cache (if this gets a hit, don't even check the package repo) > 2. Check the package repo for a wheel file (and if you find one, add > it to the download cache) > 3. Local build cache (make this interpreter implementation specific) > 4. Build a local wheel that's specific to the running interpreter implementation This is basically what exists now. We just force a compatibility tag to handle (4). From opensource at ronnypfannschmidt.de Fri Sep 1 13:32:45 2017 From: opensource at ronnypfannschmidt.de (RonnyPfannschmidt) Date: Fri, 1 Sep 2017 19:32:45 +0200 Subject: [Distutils] why do we still put version numbers into dist-info folders Message-ID: <80c9daa6-32ee-9775-8e34-39f7663d32d7@ronnypfannschmidt.de> Hi everyone, a while now i have wondered - why put version numbers into the dist info folders not only makes it lookup more expensive (need to search for a distribution->folder map) its also not serving a purpose, as the rest of the package is unversioned at the root of site-packages. in addition all those versions in folder names, that serve no purpose, and create duplication/conflicts now creep into other peps as well, please lets take a note at setuptools - for editable installs it wont put a version in to the egg info folder and imho thats the correct way to deal with data thats not actually versioned at that level where its being stored. -- Ronny From chris.barker at noaa.gov Fri Sep 1 14:30:19 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 01 Sep 2017 18:30:19 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan wrote: since it > doesn't reliably distinguish between "this cached wheel was downloaded > from a repository" and "this wheel was generated locally with a > particular version of Python". It shouldn't have to. sigh. > PEP 517 deliberately doesn't let > frontends do that as part of the initial build process (instead, if > they want to adjust the tags, they need to do it as a post-processing > step). > > Since PEP 517 breaks the current workaround for the caching scheme > being inaccurate, the most suitable response is to instead fix pip's > caching scheme to use a two tier local cache: I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools? If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has? I'm trying to think through the use case presented the other day: A package can be build either for all python's or for cpython in particular. That selection is (currently) done at build time, by custom code in setup.py. In this case, the build system SHOULD be able to build the proper wheel, and label it correctly. But there might need to be a way to tell the build system which you want. That logic isn't in existing setup.py files, but there is only so much you can do! Then there is the caching problem. If pip finds a pure-python wheel that matches the spec in the cache, it won't know to go looking for a cpython specific one. I can't see how separating the downloaded cache from the locally built cache will solve that problem: If a pure-python wheel was already downloaded, then that'll be in the downloaded cache anyway. I think the core problem here is having, for instance, pypy and cpython sharing a wheel cache. Is it really that important to support that? If so, this is a total kludge, but maybe the pip cache could keep a record-- I looked for a cpython specific wheel, and it doesn't exist, so I never need to look again. Or -- maybe more kludgy -- every time pip puts a wheel in the cache via cpython, it makes a link with the cpython name. Independently of that though, it's also important to note that pip > probably *won't* switch to invoking legacy setup.py files through the > PEP 517 backend - instead, it will only do that if there's a > pyproject.toml file that says "Use the setuptools PEP 517 backend". That's probably a good way to keep backward compatibility while moving forward. Which means pip can have two modes -- legacy mode and pep517 mode. Which may encourage me to stop being such a noodge -- As long as we're talking legacy mode, we can be as hacky as we need to be. Autobuilding support definitely *won't* be removed from pip, I'm not suggesting it should be (and I think I may be the only one that might have been) However, I do think auto-building support does not have to support all the complex use cases. we also have plenty of PyPI users that > explicitly *opt out* of using publisher-provided pre-built binaries. > While Linux distributions are the most common example (see [1] for > Fedora's policy, for example), we're not the only ones that have that > kind of rule in place. But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them. > Condo-forge, for example, almost always builds from source -- sometimes an sdist from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort). Do the Linux distros use pip to build their packages? I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds. So I ( and the entire community, as far as I can tell) stuck with calling setuptools directly - via setup.py, but with setuptools-specific flags) As pip is a package management tool, it makes sense that other package managers wouldn't need to use it. NOTE: I do use pip to build conda packages from wheels in a couple cases -- but that's considered a work around for my pathetic build skills -- and even then, I don't have pip download the wheel -- that's conda-build's job. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Sep 1 14:45:33 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 01 Sep 2017 18:45:33 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: I don't think this is a big deal. Compatibility is a guess, wheel tags do not and are not intended to guarantee compatibility especially when the wheel was built locally for caching purposes and not intentionally published. Anything can happen during a build. PEP 517 won't change this. So right now pip has an odd way of segregating its wheel cache per interpreter that involves obscure setup.py arguments. That can be fixed without re-tagging or asking the build system to re-tag wheels or otherwise interacting with the new PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Fri Sep 1 16:31:40 2017 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Fri, 1 Sep 2017 23:31:40 +0300 Subject: [Distutils] why do we still put version numbers into dist-info folders In-Reply-To: <80c9daa6-32ee-9775-8e34-39f7663d32d7@ronnypfannschmidt.de> References: <80c9daa6-32ee-9775-8e34-39f7663d32d7@ronnypfannschmidt.de> Message-ID: <5f25cb43-5ef9-3f55-e6cc-a847fe56d1e4@nextday.fi> +1 for getting rid of version numbers in the dist-info folders. RonnyPfannschmidt kirjoitti 01.09.2017 klo 20:32: > Hi everyone, > > a while now i have wondered - why put version numbers into the dist info > folders > not only makes it lookup more expensive (need to search for a > distribution->folder map) > its also not serving a purpose, as the rest of the package is > unversioned at the root of site-packages. > > in addition all those versions in folder names, that serve no purpose, > and create duplication/conflicts now creep into other peps as well, > > please lets take a note at setuptools - for editable installs it wont > put a version in to the egg info folder > > and imho thats the correct way to deal with data thats not actually > versioned at that level where its being stored. > > > -- Ronny > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From dholth at gmail.com Fri Sep 1 16:38:43 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 01 Sep 2017 20:38:43 +0000 Subject: [Distutils] why do we still put version numbers into dist-info folders In-Reply-To: <5f25cb43-5ef9-3f55-e6cc-a847fe56d1e4@nextday.fi> References: <80c9daa6-32ee-9775-8e34-39f7663d32d7@ronnypfannschmidt.de> <5f25cb43-5ef9-3f55-e6cc-a847fe56d1e4@nextday.fi> Message-ID: A long time ago pkg_resources was able to sometimes get the version number only from the .egg-info folder name, and skip opening the metadata at all when that data was not needed. We might always do listdir() for path in sys.path and parse every METADATA on the first pkg_resources import. On Fri, Sep 1, 2017 at 4:32 PM Alex Gr?nholm wrote: > +1 for getting rid of version numbers in the dist-info folders. > > > RonnyPfannschmidt kirjoitti 01.09.2017 klo 20:32: > > Hi everyone, > > > > a while now i have wondered - why put version numbers into the dist info > > folders > > not only makes it lookup more expensive (need to search for a > > distribution->folder map) > > its also not serving a purpose, as the rest of the package is > > unversioned at the root of site-packages. > > > > in addition all those versions in folder names, that serve no purpose, > > and create duplication/conflicts now creep into other peps as well, > > > > please lets take a note at setuptools - for editable installs it wont > > put a version in to the egg info folder > > > > and imho thats the correct way to deal with data thats not actually > > versioned at that level where its being stored. > > > > > > -- Ronny > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Fri Sep 1 16:46:26 2017 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Fri, 1 Sep 2017 23:46:26 +0300 Subject: [Distutils] why do we still put version numbers into dist-info folders In-Reply-To: References: <80c9daa6-32ee-9775-8e34-39f7663d32d7@ronnypfannschmidt.de> <5f25cb43-5ef9-3f55-e6cc-a847fe56d1e4@nextday.fi> Message-ID: <9411df5b-ca1a-0748-3983-61079ad497a5@nextday.fi> The question is, if we get rid of version numbers in the folder names, will it break anything we care about? Daniel Holth kirjoitti 01.09.2017 klo 23:38: > A long time ago pkg_resources was able to sometimes get the version > number only from the .egg-info folder name, and skip opening the > metadata at all when that data was not needed. We might always do > listdir() for path in sys.path and parse every METADATA on the first > pkg_resources import. > > On Fri, Sep 1, 2017 at 4:32 PM Alex Gr?nholm > wrote: > > +1 for getting rid of version numbers in the dist-info folders. > > > RonnyPfannschmidt kirjoitti 01.09.2017 klo 20:32: > > Hi everyone, > > > > a while now i have wondered - why put version numbers into the > dist info > > folders > > not only makes it lookup more expensive (need to search for a > > distribution->folder map) > > its also not serving a purpose, as the rest of the package is > > unversioned at the root of site-packages. > > > > in addition all those versions in folder names, that serve no > purpose, > > and create duplication/conflicts now creep into other peps as well, > > > > please lets take a note at setuptools - for editable installs it > wont > > put a version in to the egg info folder > > > > and imho thats the correct way to deal with data thats not actually > > versioned at that level where its being stored. > > > > > > -- Ronny > > > > _______________________________________________ > > Distutils-SIG maillist? - Distutils-SIG at python.org > > > https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist? - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Sep 1 20:58:58 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 1 Sep 2017 17:58:58 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: On Fri, Sep 1, 2017 at 11:30 AM, Chris Barker wrote: > I'm still confused -- if setuptools ( invoked by pip) is producing > incorrectly named wheels -- surely that's a bug-fix/workaround that should > go into setuptools? > > If the build is being run by pip, then doesn't setuptools have all the info > about the system that pip has? Some setup.py files are written by project authors who want to use them to generate wheels for uploading to PyPI, so they're carefully written to generate good wheels, the wheels go through QA, etc. They rely on setuptools's current (and fairly sensible) defaults for how to tag wheels, and if those go wrong, then they take the responsibility for fixing things. Other setup.py files were written by project authors who never considered any possibility outside of 'setup.py install', and haven't changed since. For them, setuptools's defaults are not so great, but the authors don't care, because they never guaranteed that it would work. Setuptools can't tell which kind of setup.py is calling it. But pip can make a pretty good guess: if it found a wheel on pypi, then someone had to have uploaded it, and it's that person's job to make sure the tags are correct. OTOH if it's taking some random setup.py file it found in an sdist, then it could be the second type, so better to play it safe and use a more restrictive wheel tag. It's a bit quirky and annoying, but that's life. And in the grand scheme of things this isn't a big deal. The only program that has to care about this is pip, and pip can always change to a different heuristic if the situation changes. -n -- Nathaniel J. Smith -- https://vorpus.org From p.f.moore at gmail.com Sat Sep 2 05:06:29 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 2 Sep 2017 10:06:29 +0100 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: On 1 September 2017 at 19:30, Chris Barker wrote: > I'm still confused -- if setuptools ( invoked by pip) is producing > incorrectly named wheels -- surely that's a bug-fix/workaround that should > go into setuptools? > > If the build is being run by pip, then doesn't setuptools have all the info > about the system that pip has? Simple answer - for historical reasons, and because it's essentially baked into the design of distutils, setuptools cannot guarantee that wheels created by two different Python interpreters on a given machine will NOT be:given the same tags and yet not usable by both. The bug isn't directly in setuptools - it's typically because the project is doing unsafe things in setup.py. This is known to happen in real life - pip has had bug reports for it. In practice, it's sufficiently difficult to debug the situation, and sufficiently hard to get fixes made in all of the upstream projects that might need fixing, that we need to do something in pip. The solution used in pip (which is a heuristic based on the bug reports we've seen) is to segregate wheels that we build internally by the Python interpreter that was used in the build. We don't do that for downloaded wheels, as we trust that they have been packaged correctly, with suitable tags. This sort of "it's really a problem further up the chain, but pip can't afford to assume it'll get fixed there" problem is typical of the reason the pp developers have been uncomfortable with the "trust the backend" approach in the PEP - but they are very much about legacy issues with the design of distutils that setuptools inherits, and I imagine that any new build systems and backends will tend to avoid such problems in their design. So yes, to address another characterisation of this issue that was mentioned, it's a workaround for legacy issues. But that legacy will be with us for a long time to come, in practice. Sorry - history lesson over ;-) Paul From g.rodola at gmail.com Sat Sep 2 01:34:20 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 2 Sep 2017 13:34:20 +0800 Subject: [Distutils] pythonhosted.org doc upload no longer works Message-ID: I know it was deprecated long ago in favor of readthedocs but I kept postponing it and my doc is still hosted on https://pythonhosted.org/psutil/ . I just took a look at: https://pypi.python.org/pypi?:action=pkg_edit&name=psutil ...and the form to upload the doc in zip format is gone. The only available option is the "destroy documentation" button. I would like to at least upload a static HTML page redirecting to the new doc which I will soon move on readthedocs. Is that possible? Thanks in advance. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nassim.daoud.algerie at gmail.com Sat Sep 2 07:05:45 2017 From: nassim.daoud.algerie at gmail.com (nassim daoud) Date: Sat, 2 Sep 2017 12:05:45 +0100 Subject: [Distutils] PEP 541 Message-ID: Hello, There is a friend of mine wondering, if abandoned PyPI projects can be claimed. And if so what the procedure in order to do that. My best regards, Daoud Nassim -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Sep 2 15:42:21 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 2 Sep 2017 15:42:21 -0400 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> Message-ID: <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> > On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: > > On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan > wrote: > > since it > doesn't reliably distinguish between "this cached wheel was downloaded > from a repository" and "this wheel was generated locally with a > particular version of Python". > > It shouldn't have to. sigh. > > PEP 517 deliberately doesn't let > frontends do that as part of the initial build process (instead, if > they want to adjust the tags, they need to do it as a post-processing > step). > > Since PEP 517 breaks the current workaround for the caching scheme > being inaccurate, the most suitable response is to instead fix pip's > caching scheme to use a two tier local cache: > > I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools? > > If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has? > > Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we?re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues. > > we also have plenty of PyPI users that > explicitly *opt out* of using publisher-provided pre-built binaries. > While Linux distributions are the most common example (see [1] for > Fedora's policy, for example), we're not the only ones that have that > kind of rule in place. > > But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them. > > Condo-forge, for example, almost always builds from source -- sometimes an sdist > from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort). Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don?t have a wheel compatibility tag defined for them, and for which we?re unlikely to ever have any wheels published for (much less wide spread). It?s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long. Pip works in all these situations, and it does so by relying on building from source. > > Do the Linux distros use pip to build their packages? Not that I am aware of. > > I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds. Adding ?no-cache-dir disables all of pip?s caching. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Sep 2 19:25:26 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 2 Sep 2017 18:25:26 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: One more issue that has come up is that "--no-user-cfg" seems to be passed to the egg_info invocation if the "isolated" parameter is enabled. I don't understand what this does, but it is again not defined in the PEP 517 interface. Should we always pass this parameter or should we never pass it? 2017-09-02 14:42 GMT-05:00 Donald Stufft : > > On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: > > On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan wrote: > > since it >> doesn't reliably distinguish between "this cached wheel was downloaded >> from a repository" and "this wheel was generated locally with a >> particular version of Python". > > > It shouldn't have to. sigh. > >> > PEP 517 deliberately doesn't let >> frontends do that as part of the initial build process (instead, if >> they want to adjust the tags, they need to do it as a post-processing >> step). >> >> Since PEP 517 breaks the current workaround for the caching scheme >> being inaccurate, the most suitable response is to instead fix pip's >> caching scheme to use a two tier local cache: > > > I'm still confused -- if setuptools ( invoked by pip) is producing > incorrectly named wheels -- surely that's a bug-fix/workaround that should > go into setuptools? > > If the build is being run by pip, then doesn't setuptools have all the > info about the system that pip has? > > > > > Someone building a wheel for distribution is likely intimately aware of > that project, and can take care to ensure that the wheel is built in such a > way that it is giving people the most optimal behavior. Pip is auto > building wheels without human intervention, and as such there is nobody > there to make sure that we?re not accidentally creating a too-broad wheel, > so we want to ensure that we have some mechanism in place for not re-using > the wheel across boundaries that might cause issues. > > > > we also have plenty of PyPI users that >> explicitly *opt out* of using publisher-provided pre-built binaries. >> While Linux distributions are the most common example (see [1] for >> Fedora's policy, for example), we're not the only ones that have that >> kind of rule in place. > > > But this is an argument for why pypi should host sdists, and the build > tools should build sdists, but not why pip should auto-build them. > >> > Condo-forge, for example, almost always builds from source -- sometimes an > sdist > from pypi, sometimes a source distribution from github or wherever the > package is hosted. And sometimes from a git tag ( last resort). > > > > Pip supports more systems than Conda does, and we do that by relying on > auto building support. Pip supports systems that don?t have a wheel > compatibility tag defined for them, and for which we?re unlikely to ever > have any wheels published for (much less wide spread). It?s pretty easy to > cover Windows/macOS/Some Linux systems, but when you start talking about > FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets > extremely long. > > Pip works in all these situations, and it does so by relying on building > from source. > > > > Do the Linux distros use pip to build their packages? > > > > Not that I am aware of. > > > I tried to do that with conda-packages, and failed due to pip's caching > behavior-- it probably would have worked fine in production, but when I > was developing the build script, I couldn't reliably get pip to ignore > cached wheels from previous experimental builds. > > > > Adding ?no-cache-dir disables all of pip?s caching. > > ? > Donald Stufft > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Sep 2 19:26:47 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 2 Sep 2017 18:26:47 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: Donald, This was your work in https://github.com/pypa/pip/pull/2169. Unfortunately the comments were quite sparse. 2017-09-02 18:25 GMT-05:00 xoviat : > One more issue that has come up is that "--no-user-cfg" seems to be passed > to the egg_info invocation if the "isolated" parameter is enabled. I don't > understand what this does, but it is again not defined in the PEP 517 > interface. Should we always pass this parameter or should we never pass it? > > 2017-09-02 14:42 GMT-05:00 Donald Stufft : > >> >> On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: >> >> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan wrote: >> >> since it >>> doesn't reliably distinguish between "this cached wheel was downloaded >>> from a repository" and "this wheel was generated locally with a >>> particular version of Python". >> >> >> It shouldn't have to. sigh. >> >>> >> PEP 517 deliberately doesn't let >>> frontends do that as part of the initial build process (instead, if >>> they want to adjust the tags, they need to do it as a post-processing >>> step). >>> >>> Since PEP 517 breaks the current workaround for the caching scheme >>> being inaccurate, the most suitable response is to instead fix pip's >>> caching scheme to use a two tier local cache: >> >> >> I'm still confused -- if setuptools ( invoked by pip) is producing >> incorrectly named wheels -- surely that's a bug-fix/workaround that should >> go into setuptools? >> >> If the build is being run by pip, then doesn't setuptools have all the >> info about the system that pip has? >> >> >> >> >> Someone building a wheel for distribution is likely intimately aware of >> that project, and can take care to ensure that the wheel is built in such a >> way that it is giving people the most optimal behavior. Pip is auto >> building wheels without human intervention, and as such there is nobody >> there to make sure that we?re not accidentally creating a too-broad wheel, >> so we want to ensure that we have some mechanism in place for not re-using >> the wheel across boundaries that might cause issues. >> >> >> >> we also have plenty of PyPI users that >>> explicitly *opt out* of using publisher-provided pre-built binaries. >>> While Linux distributions are the most common example (see [1] for >>> Fedora's policy, for example), we're not the only ones that have that >>> kind of rule in place. >> >> >> But this is an argument for why pypi should host sdists, and the build >> tools should build sdists, but not why pip should auto-build them. >> >>> >> Condo-forge, for example, almost always builds from source -- sometimes >> an sdist >> from pypi, sometimes a source distribution from github or wherever the >> package is hosted. And sometimes from a git tag ( last resort). >> >> >> >> Pip supports more systems than Conda does, and we do that by relying on >> auto building support. Pip supports systems that don?t have a wheel >> compatibility tag defined for them, and for which we?re unlikely to ever >> have any wheels published for (much less wide spread). It?s pretty easy to >> cover Windows/macOS/Some Linux systems, but when you start talking about >> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets >> extremely long. >> >> Pip works in all these situations, and it does so by relying on building >> from source. >> >> >> >> Do the Linux distros use pip to build their packages? >> >> >> >> Not that I am aware of. >> >> >> I tried to do that with conda-packages, and failed due to pip's caching >> behavior-- it probably would have worked fine in production, but when I >> was developing the build script, I couldn't reliably get pip to ignore >> cached wheels from previous experimental builds. >> >> >> >> Adding ?no-cache-dir disables all of pip?s caching. >> >> ? >> Donald Stufft >> >> >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Sep 2 20:16:32 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 2 Sep 2017 19:16:32 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: Whatever it was, removing it seems to have had no effect on the tests. I will remove it unless someone has an objection. 2017-09-02 18:26 GMT-05:00 xoviat : > Donald, > > > This was your work in https://github.com/pypa/pip/pull/2169. > Unfortunately the comments were quite sparse. > > 2017-09-02 18:25 GMT-05:00 xoviat : > >> One more issue that has come up is that "--no-user-cfg" seems to be >> passed to the egg_info invocation if the "isolated" parameter is enabled. I >> don't understand what this does, but it is again not defined in the PEP 517 >> interface. Should we always pass this parameter or should we never pass it? >> >> 2017-09-02 14:42 GMT-05:00 Donald Stufft : >> >>> >>> On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: >>> >>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan wrote: >>> >>> since it >>>> doesn't reliably distinguish between "this cached wheel was downloaded >>>> from a repository" and "this wheel was generated locally with a >>>> particular version of Python". >>> >>> >>> It shouldn't have to. sigh. >>> >>>> >>> PEP 517 deliberately doesn't let >>>> frontends do that as part of the initial build process (instead, if >>>> they want to adjust the tags, they need to do it as a post-processing >>>> step). >>>> >>>> Since PEP 517 breaks the current workaround for the caching scheme >>>> being inaccurate, the most suitable response is to instead fix pip's >>>> caching scheme to use a two tier local cache: >>> >>> >>> I'm still confused -- if setuptools ( invoked by pip) is producing >>> incorrectly named wheels -- surely that's a bug-fix/workaround that should >>> go into setuptools? >>> >>> If the build is being run by pip, then doesn't setuptools have all the >>> info about the system that pip has? >>> >>> >>> >>> >>> Someone building a wheel for distribution is likely intimately aware of >>> that project, and can take care to ensure that the wheel is built in such a >>> way that it is giving people the most optimal behavior. Pip is auto >>> building wheels without human intervention, and as such there is nobody >>> there to make sure that we?re not accidentally creating a too-broad wheel, >>> so we want to ensure that we have some mechanism in place for not re-using >>> the wheel across boundaries that might cause issues. >>> >>> >>> >>> we also have plenty of PyPI users that >>>> explicitly *opt out* of using publisher-provided pre-built binaries. >>>> While Linux distributions are the most common example (see [1] for >>>> Fedora's policy, for example), we're not the only ones that have that >>>> kind of rule in place. >>> >>> >>> But this is an argument for why pypi should host sdists, and the build >>> tools should build sdists, but not why pip should auto-build them. >>> >>>> >>> Condo-forge, for example, almost always builds from source -- sometimes >>> an sdist >>> from pypi, sometimes a source distribution from github or wherever the >>> package is hosted. And sometimes from a git tag ( last resort). >>> >>> >>> >>> Pip supports more systems than Conda does, and we do that by relying on >>> auto building support. Pip supports systems that don?t have a wheel >>> compatibility tag defined for them, and for which we?re unlikely to ever >>> have any wheels published for (much less wide spread). It?s pretty easy to >>> cover Windows/macOS/Some Linux systems, but when you start talking about >>> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets >>> extremely long. >>> >>> Pip works in all these situations, and it does so by relying on building >>> from source. >>> >>> >>> >>> Do the Linux distros use pip to build their packages? >>> >>> >>> >>> Not that I am aware of. >>> >>> >>> I tried to do that with conda-packages, and failed due to pip's caching >>> behavior-- it probably would have worked fine in production, but when I >>> was developing the build script, I couldn't reliably get pip to ignore >>> cached wheels from previous experimental builds. >>> >>> >>> >>> Adding ?no-cache-dir disables all of pip?s caching. >>> >>> ? >>> Donald Stufft >>> >>> >>> >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Sat Sep 2 20:51:49 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sun, 03 Sep 2017 00:51:49 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: On Sat, Sep 2, 2017 at 5:17 PM xoviat wrote: > Whatever it was, removing it seems to have had no effect on the tests. I > will remove it unless someone has an objection. > Just FYI, I wouldn't take the tests still passing as a major signal. I've noticed there are even common code paths / functional scenarios that aren't under test. --Chris > 2017-09-02 18:26 GMT-05:00 xoviat : > >> Donald, >> >> >> This was your work in https://github.com/pypa/pip/pull/2169. >> Unfortunately the comments were quite sparse. >> >> 2017-09-02 18:25 GMT-05:00 xoviat : >> >>> One more issue that has come up is that "--no-user-cfg" seems to be >>> passed to the egg_info invocation if the "isolated" parameter is enabled. I >>> don't understand what this does, but it is again not defined in the PEP 517 >>> interface. Should we always pass this parameter or should we never pass it? >>> >>> 2017-09-02 14:42 GMT-05:00 Donald Stufft : >>> >>>> >>>> On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: >>>> >>>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan >>>> wrote: >>>> >>>> since it >>>>> doesn't reliably distinguish between "this cached wheel was downloaded >>>>> from a repository" and "this wheel was generated locally with a >>>>> particular version of Python". >>>> >>>> >>>> It shouldn't have to. sigh. >>>> >>>>> >>>> PEP 517 deliberately doesn't let >>>>> frontends do that as part of the initial build process (instead, if >>>>> they want to adjust the tags, they need to do it as a post-processing >>>>> step). >>>>> >>>>> Since PEP 517 breaks the current workaround for the caching scheme >>>>> being inaccurate, the most suitable response is to instead fix pip's >>>>> caching scheme to use a two tier local cache: >>>> >>>> >>>> I'm still confused -- if setuptools ( invoked by pip) is producing >>>> incorrectly named wheels -- surely that's a bug-fix/workaround that should >>>> go into setuptools? >>>> >>>> If the build is being run by pip, then doesn't setuptools have all the >>>> info about the system that pip has? >>>> >>>> >>>> >>>> >>>> Someone building a wheel for distribution is likely intimately aware of >>>> that project, and can take care to ensure that the wheel is built in such a >>>> way that it is giving people the most optimal behavior. Pip is auto >>>> building wheels without human intervention, and as such there is nobody >>>> there to make sure that we?re not accidentally creating a too-broad wheel, >>>> so we want to ensure that we have some mechanism in place for not re-using >>>> the wheel across boundaries that might cause issues. >>>> >>>> >>>> >>>> we also have plenty of PyPI users that >>>>> explicitly *opt out* of using publisher-provided pre-built binaries. >>>>> While Linux distributions are the most common example (see [1] for >>>>> Fedora's policy, for example), we're not the only ones that have that >>>>> kind of rule in place. >>>> >>>> >>>> But this is an argument for why pypi should host sdists, and the build >>>> tools should build sdists, but not why pip should auto-build them. >>>> >>>>> >>>> Condo-forge, for example, almost always builds from source -- sometimes >>>> an sdist >>>> from pypi, sometimes a source distribution from github or wherever the >>>> package is hosted. And sometimes from a git tag ( last resort). >>>> >>>> >>>> >>>> Pip supports more systems than Conda does, and we do that by relying on >>>> auto building support. Pip supports systems that don?t have a wheel >>>> compatibility tag defined for them, and for which we?re unlikely to ever >>>> have any wheels published for (much less wide spread). It?s pretty easy to >>>> cover Windows/macOS/Some Linux systems, but when you start talking about >>>> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets >>>> extremely long. >>>> >>>> Pip works in all these situations, and it does so by relying on >>>> building from source. >>>> >>>> >>>> >>>> Do the Linux distros use pip to build their packages? >>>> >>>> >>>> >>>> Not that I am aware of. >>>> >>>> >>>> I tried to do that with conda-packages, and failed due to pip's caching >>>> behavior-- it probably would have worked fine in production, but when I >>>> was developing the build script, I couldn't reliably get pip to ignore >>>> cached wheels from previous experimental builds. >>>> >>>> >>>> >>>> Adding ?no-cache-dir disables all of pip?s caching. >>>> >>>> ? >>>> Donald Stufft >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>> >>>> >>> >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sun Sep 3 14:10:18 2017 From: xoviat at gmail.com (xoviat) Date: Sun, 3 Sep 2017 13:10:18 -0500 Subject: [Distutils] PEP 541 In-Reply-To: References: Message-ID: I would actually like to transfer the "pywin32" project to Paul, Glyph, myself and friends because nothing has been uploaded there, but I thought it wasn't possible. Maybe it is? 2017-09-02 6:05 GMT-05:00 nassim daoud : > Hello, > > There is a friend of mine wondering, if abandoned PyPI projects can be > claimed. And if so what the procedure in order to do that. > > My best regards, > > Daoud Nassim > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sun Sep 3 14:14:37 2017 From: xoviat at gmail.com (xoviat) Date: Sun, 3 Sep 2017 13:14:37 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: Just an update for everyone here: 1. We're currently waiting on the implementation of the 'dist_info" command in the wheel project. 2. Once that is done we can switch pip over to reading dist-info rather than egg_info. 3. Then we can move the backend over to setuptools. Because Jacob has a much more efficient release system than pip, I anticipate having a release of setuptools first and then we can switch pip over to requiring a newer setuptools via PEP 518. 2017-09-02 19:51 GMT-05:00 Chris Jerdonek : > On Sat, Sep 2, 2017 at 5:17 PM xoviat wrote: > >> Whatever it was, removing it seems to have had no effect on the tests. I >> will remove it unless someone has an objection. >> > > Just FYI, I wouldn't take the tests still passing as a major signal. I've > noticed there are even common code paths / functional scenarios that aren't > under test. > > --Chris > > > > >> 2017-09-02 18:26 GMT-05:00 xoviat : >> >>> Donald, >>> >>> >>> This was your work in https://github.com/pypa/pip/pull/2169. >>> Unfortunately the comments were quite sparse. >>> >>> 2017-09-02 18:25 GMT-05:00 xoviat : >>> >>>> One more issue that has come up is that "--no-user-cfg" seems to be >>>> passed to the egg_info invocation if the "isolated" parameter is enabled. I >>>> don't understand what this does, but it is again not defined in the PEP 517 >>>> interface. Should we always pass this parameter or should we never pass it? >>>> >>>> 2017-09-02 14:42 GMT-05:00 Donald Stufft : >>>> >>>>> >>>>> On Sep 1, 2017, at 2:30 PM, Chris Barker >>>>> wrote: >>>>> >>>>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan >>>>> wrote: >>>>> >>>>> since it >>>>>> doesn't reliably distinguish between "this cached wheel was downloaded >>>>>> from a repository" and "this wheel was generated locally with a >>>>>> particular version of Python". >>>>> >>>>> >>>>> It shouldn't have to. sigh. >>>>> >>>>>> >>>>> PEP 517 deliberately doesn't let >>>>>> frontends do that as part of the initial build process (instead, if >>>>>> they want to adjust the tags, they need to do it as a post-processing >>>>>> step). >>>>>> >>>>>> Since PEP 517 breaks the current workaround for the caching scheme >>>>>> being inaccurate, the most suitable response is to instead fix pip's >>>>>> caching scheme to use a two tier local cache: >>>>> >>>>> >>>>> I'm still confused -- if setuptools ( invoked by pip) is producing >>>>> incorrectly named wheels -- surely that's a bug-fix/workaround that should >>>>> go into setuptools? >>>>> >>>>> If the build is being run by pip, then doesn't setuptools have all the >>>>> info about the system that pip has? >>>>> >>>>> >>>>> >>>>> >>>>> Someone building a wheel for distribution is likely intimately aware >>>>> of that project, and can take care to ensure that the wheel is built in >>>>> such a way that it is giving people the most optimal behavior. Pip is auto >>>>> building wheels without human intervention, and as such there is nobody >>>>> there to make sure that we?re not accidentally creating a too-broad wheel, >>>>> so we want to ensure that we have some mechanism in place for not re-using >>>>> the wheel across boundaries that might cause issues. >>>>> >>>>> >>>>> >>>>> we also have plenty of PyPI users that >>>>>> explicitly *opt out* of using publisher-provided pre-built binaries. >>>>>> While Linux distributions are the most common example (see [1] for >>>>>> Fedora's policy, for example), we're not the only ones that have that >>>>>> kind of rule in place. >>>>> >>>>> >>>>> But this is an argument for why pypi should host sdists, and the build >>>>> tools should build sdists, but not why pip should auto-build them. >>>>> >>>>>> >>>>> Condo-forge, for example, almost always builds from source -- >>>>> sometimes an sdist >>>>> from pypi, sometimes a source distribution from github or wherever the >>>>> package is hosted. And sometimes from a git tag ( last resort). >>>>> >>>>> >>>>> >>>>> Pip supports more systems than Conda does, and we do that by relying >>>>> on auto building support. Pip supports systems that don?t have a wheel >>>>> compatibility tag defined for them, and for which we?re unlikely to ever >>>>> have any wheels published for (much less wide spread). It?s pretty easy to >>>>> cover Windows/macOS/Some Linux systems, but when you start talking about >>>>> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets >>>>> extremely long. >>>>> >>>>> Pip works in all these situations, and it does so by relying on >>>>> building from source. >>>>> >>>>> >>>>> >>>>> Do the Linux distros use pip to build their packages? >>>>> >>>>> >>>>> >>>>> Not that I am aware of. >>>>> >>>>> >>>>> I tried to do that with conda-packages, and failed due to pip's >>>>> caching behavior-- it probably would have worked fine in production, but >>>>> when I was developing the build script, I couldn't reliably get pip to >>>>> ignore cached wheels from previous experimental builds. >>>>> >>>>> >>>>> >>>>> Adding ?no-cache-dir disables all of pip?s caching. >>>>> >>>>> ? >>>>> Donald Stufft >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>>> >>>>> >>>> >>> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cyanrave at gmail.com Sat Sep 2 18:04:34 2017 From: cyanrave at gmail.com (cyanrave at gmail.com) Date: Sat, 2 Sep 2017 15:04:34 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: Great discussions on this PEP over the proposal, here and in the archives. I've just now gotten plugged in to the distutils distro, at a seemingly perfect time... somewhat new to Python packaging and still learning. Many of the issues observed by users in the mail chain are reasons why we had marching orders at the organization I work for to internally build all whls, on the platforms we support by using 'pip wheel' with specific flags. One of these flags is to pull source only (pip download, no-binary) and direct 'pip wheel' to build with that source dir. For the caching issue we ended up using Docker, spinning up a new instance per build with the platforms we support, running the build, and uploading those artifacts to our internal repo for pulls by the users, then thrashing that container wholly. In a similar way, I see the 'venv' requirement of 517 mirroring this process. The presupposition of this pipeline we built is that metadata and source repo structures are trusted to produce accurate binaries... the more I learn about how projects / packages have set up their build instructions, the more horrified I become in the variance of the implementations out in the wild. 517 and 518 to me seems like a way to reign in some of these variances and formalize them, especially around setup.py and metadata... Is that correct or am I way off base with the discussion and proposal so far? Best regards, C. Schmautz > On Sep 2, 2017, at 12:42 PM, Donald Stufft wrote: > > >> On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: >> >>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan wrote: >>> >>> since it >>> doesn't reliably distinguish between "this cached wheel was downloaded >>> from a repository" and "this wheel was generated locally with a >>> particular version of Python". >> >> It shouldn't have to. sigh. >> >>> PEP 517 deliberately doesn't let >>> frontends do that as part of the initial build process (instead, if >>> they want to adjust the tags, they need to do it as a post-processing >>> step). >>> >>> Since PEP 517 breaks the current workaround for the caching scheme >>> being inaccurate, the most suitable response is to instead fix pip's >>> caching scheme to use a two tier local cache: >> >> I'm still confused -- if setuptools ( invoked by pip) is producing incorrectly named wheels -- surely that's a bug-fix/workaround that should go into setuptools? >> >> If the build is being run by pip, then doesn't setuptools have all the info about the system that pip has? >> >> > > > Someone building a wheel for distribution is likely intimately aware of that project, and can take care to ensure that the wheel is built in such a way that it is giving people the most optimal behavior. Pip is auto building wheels without human intervention, and as such there is nobody there to make sure that we?re not accidentally creating a too-broad wheel, so we want to ensure that we have some mechanism in place for not re-using the wheel across boundaries that might cause issues. > > >> >>> we also have plenty of PyPI users that >>> explicitly *opt out* of using publisher-provided pre-built binaries. >>> While Linux distributions are the most common example (see [1] for >>> Fedora's policy, for example), we're not the only ones that have that >>> kind of rule in place. >> >> But this is an argument for why pypi should host sdists, and the build tools should build sdists, but not why pip should auto-build them. >> >> Condo-forge, for example, almost always builds from source -- sometimes an sdist >> from pypi, sometimes a source distribution from github or wherever the package is hosted. And sometimes from a git tag ( last resort). > > > Pip supports more systems than Conda does, and we do that by relying on auto building support. Pip supports systems that don?t have a wheel compatibility tag defined for them, and for which we?re unlikely to ever have any wheels published for (much less wide spread). It?s pretty easy to cover Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long. > > Pip works in all these situations, and it does so by relying on building from source. > > >> >> Do the Linux distros use pip to build their packages? > > > Not that I am aware of. > >> >> I tried to do that with conda-packages, and failed due to pip's caching behavior-- it probably would have worked fine in production, but when I was developing the build script, I couldn't reliably get pip to ignore cached wheels from previous experimental builds. > > > Adding ?no-cache-dir disables all of pip?s caching. > > ? > Donald Stufft > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Sep 3 22:00:58 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 3 Sep 2017 19:00:58 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: > Just an update for everyone here: > > 1. We're currently waiting on the implementation of the 'dist_info" command > in the wheel project. > 2. Once that is done we can switch pip over to reading dist-info rather than > egg_info. > 3. Then we can move the backend over to setuptools. Because Jacob has a much > more efficient release system than pip, I anticipate having a release of > setuptools first and then we can switch pip over to requiring a newer > setuptools via PEP 518. I don't think pip actually has any use for the PEP 517 prepare_wheel_metadata hook right now though? Historically 'setup.py egg-info' was needed to kluge around unwanted behavior in 'setup.py install', but with a PEP 517 backend that's irrelevant because 'setup.py install' is never used. And in the future when pip has a real resolver, then prepare_wheel_metadata should allow some optimizations. But right now, prepare_wheel_metadata is completely useless AFAIK. So why is 'setup.py dist_info' a blocker for things? -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Mon Sep 4 09:00:26 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Sep 2017 23:00:26 +1000 Subject: [Distutils] PEP 517 again In-Reply-To: <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: On 3 September 2017 at 05:42, Donald Stufft wrote: > On Sep 1, 2017, at 2:30 PM, Chris Barker wrote: >> Do the Linux distros use pip to build their packages? > > Not that I am aware of. Fedora's build macros for Python projects currently rely on running setup.py directly, but we've been considering switching to pip instead since 2013 or so. PEP 517 is likely to provide the impetus to switch from "maybe we should do that" to "we need to do that, at least if setup.py is missing, and potentially always, so we get more consistent installation metadata" Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Sep 4 09:33:33 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Sep 2017 23:33:33 +1000 Subject: [Distutils] Migrating interoperability specs to packaging.python.org Message-ID: Some time ago, I started the process [1] of adjusting how distutils-sig uses the PEP process so that the reference specifications will live on packaging.python.org, and we use the PEP process to manage *changes* to those specifications, rather than serving as the specifications themselves (that is, adopting a process closer to the URL-centric way the Python language reference is managed, rather than using the RFCstyle PEP-number-centric model the way we do now). I never actually finished that work, and as a result, it's currently thoroughly unclear [2] that Description-Content-Type and Provides-Extra are defined at https://packaging.python.org/specifications/#core-metadata rather than in a PEP. I'm currently at the CPython core development sprint in San Francisco, and I'm thinking that finalising that migration [3] and updating the affected PEPs accordingly (most notably, PEP 345) is likely to be a good use of my time. However, I'm also wondering if it may still be worthwhile writing a metadata 1.3 PEP that does the following things: 1. Explicitly notes the addition of the two new fields 2. Describes the process change for packaging interoperability specifications 3. Defines a canonical transformation between the human-readable key:value format and a more automation friendly JSON format That PEP would then essentially be the first one to use the new process: it would supersede PEP 345 as the latest metadata specification, but it would *also* redirect readers to the relevant URL on packaging.python.org as the canonical source of the specification, rather than being the reference documentation in its own right. Cheers, Nick. [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 [2] https://github.com/python/peps/issues/388 P.S. Daniel, if you're currently thinking "I proposed defining an incremental metadata 1.3 tweak years ago!", aye, you did. My subsequent additions to PEP 426 were a classic case of second-system syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we suspected long ago, hence that PEP's original deferral) Fortunately, the disciplining effect of working with a primarily volunteer contributor base has prevented my over-engineered change-for-change's-sake-rather-than-to-solve-actual-user-problems version from becoming reality ;) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Sep 4 09:56:41 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Sep 2017 23:56:41 +1000 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: On 2 September 2017 at 15:34, Giampaolo Rodola' wrote: > I know it was deprecated long ago in favor of readthedocs but I kept > postponing it and my doc is still hosted on > https://pythonhosted.org/psutil/. While we've talked about deprecating it, it *hasn't* been deprecated. Looking at https://github.com/pypa/pypi-legacy/commits/production, I'm not seeing anything obvious that would have caused problems with docs management, but that's probably still the best issue tracker to use to report the bug. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Mon Sep 4 10:03:22 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 04 Sep 2017 14:03:22 +0000 Subject: [Distutils] Migrating interoperability specs to packaging.python.org In-Reply-To: References: Message-ID: Some people enjoy using metadata.json which tracked pep 426 but I have been meaning to take it out, and perhaps keep the key/value to json converter as a command. On Mon, Sep 4, 2017, 09:33 Nick Coghlan wrote: > Some time ago, I started the process [1] of adjusting how > distutils-sig uses the PEP process so that the reference > specifications will live on packaging.python.org, and we use the PEP > process to manage *changes* to those specifications, rather than > serving as the specifications themselves (that is, adopting a process > closer to the URL-centric way the Python language reference is > managed, rather than using the RFCstyle PEP-number-centric model the > way we do now). > > I never actually finished that work, and as a result, it's currently > thoroughly unclear [2] that Description-Content-Type and > Provides-Extra are defined at > https://packaging.python.org/specifications/#core-metadata rather than > in a PEP. > > I'm currently at the CPython core development sprint in San Francisco, > and I'm thinking that finalising that migration [3] and updating the > affected PEPs accordingly (most notably, PEP 345) is likely to be a > good use of my time. > > However, I'm also wondering if it may still be worthwhile writing a > metadata 1.3 PEP that does the following things: > > 1. Explicitly notes the addition of the two new fields > 2. Describes the process change for packaging interoperability > specifications > 3. Defines a canonical transformation between the human-readable > key:value format and a more automation friendly JSON format > > That PEP would then essentially be the first one to use the new > process: it would supersede PEP 345 as the latest metadata > specification, but it would *also* redirect readers to the relevant > URL on packaging.python.org as the canonical source of the > specification, rather than being the reference documentation in its > own right. > > Cheers, > Nick. > > [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 > [2] https://github.com/python/peps/issues/388 > > P.S. Daniel, if you're currently thinking "I proposed defining an > incremental metadata 1.3 tweak years ago!", aye, you did. My > subsequent additions to PEP 426 were a classic case of second-system > syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we > suspected long ago, hence that PEP's original deferral) > > Fortunately, the disciplining effect of working with a primarily > volunteer contributor base has prevented my over-engineered > change-for-change's-sake-rather-than-to-solve-actual-user-problems > version from becoming reality ;) > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Mon Sep 4 10:06:16 2017 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Mon, 4 Sep 2017 17:06:16 +0300 Subject: [Distutils] Migrating interoperability specs to packaging.python.org In-Reply-To: References: Message-ID: Yes, I see the inclusion of a metadata file which conforms to an unaccepted PEP as potentially dangerous. Perhaps I should disable it in the next release? Daniel Holth kirjoitti 04.09.2017 klo 17:03: > > Some people enjoy using metadata.json which tracked pep 426 but I have > been meaning to take it out, and perhaps keep the key/value to json > converter as a command. > > > On Mon, Sep 4, 2017, 09:33 Nick Coghlan > wrote: > > Some time ago, I started the process [1] of adjusting how > distutils-sig uses the PEP process so that the reference > specifications will live on packaging.python.org > , and we use the PEP > process to manage *changes* to those specifications, rather than > serving as the specifications themselves (that is, adopting a process > closer to the URL-centric way the Python language reference is > managed, rather than using the RFCstyle PEP-number-centric model the > way we do now). > > I never actually finished that work, and as a result, it's currently > thoroughly unclear [2] that Description-Content-Type and > Provides-Extra are defined at > https://packaging.python.org/specifications/#core-metadata rather than > in a PEP. > > I'm currently at the CPython core development sprint in San Francisco, > and I'm thinking that finalising that migration [3] and updating the > affected PEPs accordingly (most notably, PEP 345) is likely to be a > good use of my time. > > However, I'm also wondering if it may still be worthwhile writing a > metadata 1.3 PEP that does the following things: > > 1. Explicitly notes the addition of the two new fields > 2. Describes the process change for packaging interoperability > specifications > 3. Defines a canonical transformation between the human-readable > key:value format and a more automation friendly JSON format > > That PEP would then essentially be the first one to use the new > process: it would supersede PEP 345 as the latest metadata > specification, but it would *also* redirect readers to the relevant > URL on packaging.python.org as the > canonical source of the > specification, rather than being the reference documentation in its > own right. > > Cheers, > Nick. > > [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 > [2] https://github.com/python/peps/issues/388 > > P.S. Daniel, if you're currently thinking "I proposed defining an > incremental metadata 1.3 tweak years ago!", aye, you did. My > subsequent additions to PEP 426 were a classic case of second-system > syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we > suspected long ago, hence that PEP's original deferral) > > Fortunately, the disciplining effect of working with a primarily > volunteer contributor base has prevented my over-engineered > change-for-change's-sake-rather-than-to-solve-actual-user-problems > version from becoming reality ;) > > -- > Nick Coghlan? ?| ncoghlan at gmail.com > ?|? ?Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist? - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Sep 4 10:48:26 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 04 Sep 2017 14:48:26 +0000 Subject: [Distutils] Migrating interoperability specs to packaging.python.org In-Reply-To: References: Message-ID: Well, none of the metadata generated by bdist wheel conforms to an accepted pep. But if you rely on the json file then you won't be interoperable with wheels from any other generator. On Mon, Sep 4, 2017, 10:06 Alex Gr?nholm wrote: > Yes, I see the inclusion of a metadata file which conforms to an > unaccepted PEP as potentially dangerous. > > Perhaps I should disable it in the next release? > > Daniel Holth kirjoitti 04.09.2017 klo 17:03: > > Some people enjoy using metadata.json which tracked pep 426 but I have > been meaning to take it out, and perhaps keep the key/value to json > converter as a command. > > On Mon, Sep 4, 2017, 09:33 Nick Coghlan wrote: > >> Some time ago, I started the process [1] of adjusting how >> distutils-sig uses the PEP process so that the reference >> specifications will live on packaging.python.org, and we use the PEP >> process to manage *changes* to those specifications, rather than >> serving as the specifications themselves (that is, adopting a process >> closer to the URL-centric way the Python language reference is >> managed, rather than using the RFCstyle PEP-number-centric model the >> way we do now). >> >> I never actually finished that work, and as a result, it's currently >> thoroughly unclear [2] that Description-Content-Type and >> Provides-Extra are defined at >> https://packaging.python.org/specifications/#core-metadata rather than >> in a PEP. >> >> I'm currently at the CPython core development sprint in San Francisco, >> and I'm thinking that finalising that migration [3] and updating the >> affected PEPs accordingly (most notably, PEP 345) is likely to be a >> good use of my time. >> >> However, I'm also wondering if it may still be worthwhile writing a >> metadata 1.3 PEP that does the following things: >> >> 1. Explicitly notes the addition of the two new fields >> 2. Describes the process change for packaging interoperability >> specifications >> 3. Defines a canonical transformation between the human-readable >> key:value format and a more automation friendly JSON format >> >> That PEP would then essentially be the first one to use the new >> process: it would supersede PEP 345 as the latest metadata >> specification, but it would *also* redirect readers to the relevant >> URL on packaging.python.org as the canonical source of the >> specification, rather than being the reference documentation in its >> own right. >> >> Cheers, >> Nick. >> >> [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 >> [2] https://github.com/python/peps/issues/388 >> >> P.S. Daniel, if you're currently thinking "I proposed defining an >> incremental metadata 1.3 tweak years ago!", aye, you did. My >> subsequent additions to PEP 426 were a classic case of second-system >> syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we >> suspected long ago, hence that PEP's original deferral) >> >> Fortunately, the disciplining effect of working with a primarily >> volunteer contributor base has prevented my over-engineered >> change-for-change's-sake-rather-than-to-solve-actual-user-problems >> version from becoming reality ;) >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.orghttps://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Mon Sep 4 10:51:13 2017 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Mon, 4 Sep 2017 17:51:13 +0300 Subject: [Distutils] Migrating interoperability specs to packaging.python.org In-Reply-To: References: Message-ID: <3672fda6-4077-b76a-d70e-058abe13a4d5@nextday.fi> Yeah, so we should maybe start using the upcoming v1.3 metadata when the related PEP is accepted...? Daniel Holth kirjoitti 04.09.2017 klo 17:48: > > Well, none of the metadata generated by bdist wheel conforms to an > accepted pep. But if you rely on the json file then you won't be > interoperable with wheels from any other generator. > > > On Mon, Sep 4, 2017, 10:06 Alex Gr?nholm > wrote: > > Yes, I see the inclusion of a metadata file which conforms to an > unaccepted PEP as potentially dangerous. > > Perhaps I should disable it in the next release? > > > Daniel Holth kirjoitti 04.09.2017 klo 17:03: >> >> Some people enjoy using metadata.json which tracked pep 426 but I >> have been meaning to take it out, and perhaps keep the key/value >> to json converter as a command. >> >> >> On Mon, Sep 4, 2017, 09:33 Nick Coghlan > > wrote: >> >> Some time ago, I started the process [1] of adjusting how >> distutils-sig uses the PEP process so that the reference >> specifications will live on packaging.python.org >> , and we use the PEP >> process to manage *changes* to those specifications, rather than >> serving as the specifications themselves (that is, adopting a >> process >> closer to the URL-centric way the Python language reference is >> managed, rather than using the RFCstyle PEP-number-centric >> model the >> way we do now). >> >> I never actually finished that work, and as a result, it's >> currently >> thoroughly unclear [2] that Description-Content-Type and >> Provides-Extra are defined at >> https://packaging.python.org/specifications/#core-metadata >> rather than >> in a PEP. >> >> I'm currently at the CPython core development sprint in San >> Francisco, >> and I'm thinking that finalising that migration [3] and >> updating the >> affected PEPs accordingly (most notably, PEP 345) is likely >> to be a >> good use of my time. >> >> However, I'm also wondering if it may still be worthwhile >> writing a >> metadata 1.3 PEP that does the following things: >> >> 1. Explicitly notes the addition of the two new fields >> 2. Describes the process change for packaging >> interoperability specifications >> 3. Defines a canonical transformation between the human-readable >> key:value format and a more automation friendly JSON format >> >> That PEP would then essentially be the first one to use the new >> process: it would supersede PEP 345 as the latest metadata >> specification, but it would *also* redirect readers to the >> relevant >> URL on packaging.python.org as >> the canonical source of the >> specification, rather than being the reference documentation >> in its >> own right. >> >> Cheers, >> Nick. >> >> [1] >> https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 >> [2] https://github.com/python/peps/issues/388 >> >> P.S. Daniel, if you're currently thinking "I proposed defining an >> incremental metadata 1.3 tweak years ago!", aye, you did. My >> subsequent additions to PEP 426 were a classic case of >> second-system >> syndrome: https://en.wikipedia.org/wiki/Second-system_effect >> (which we >> suspected long ago, hence that PEP's original deferral) >> >> Fortunately, the disciplining effect of working with a primarily >> volunteer contributor base has prevented my over-engineered >> change-for-change's-sake-rather-than-to-solve-actual-user-problems >> version from becoming reality ;) >> >> -- >> Nick Coghlan? ?| ncoghlan at gmail.com >> ?|? ?Brisbane, Australia >> _______________________________________________ >> Distutils-SIG maillist? - Distutils-SIG at python.org >> >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> >> _______________________________________________ >> Distutils-SIG maillist -Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist? - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Sep 4 10:46:12 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 04 Sep 2017 14:46:12 +0000 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: Wheel has always implemented dist info by converting from egg info. It is necessary to invoke all of the egg info writers provided by setuptools or tons of packages will break. But so far dist info has only been generated as part of building a wheel. On Sun, Sep 3, 2017, 22:01 Nathaniel Smith wrote: > On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: > > Just an update for everyone here: > > > > 1. We're currently waiting on the implementation of the 'dist_info" > command > > in the wheel project. > > 2. Once that is done we can switch pip over to reading dist-info rather > than > > egg_info. > > 3. Then we can move the backend over to setuptools. Because Jacob has a > much > > more efficient release system than pip, I anticipate having a release of > > setuptools first and then we can switch pip over to requiring a newer > > setuptools via PEP 518. > > I don't think pip actually has any use for the PEP 517 > prepare_wheel_metadata hook right now though? Historically 'setup.py > egg-info' was needed to kluge around unwanted behavior in 'setup.py > install', but with a PEP 517 backend that's irrelevant because > 'setup.py install' is never used. And in the future when pip has a > real resolver, then prepare_wheel_metadata should allow some > optimizations. But right now, prepare_wheel_metadata is completely > useless AFAIK. > > So why is 'setup.py dist_info' a blocker for things? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Sep 4 20:09:36 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 4 Sep 2017 19:09:36 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: Nathaniel: Pip requires egg_info to discover dependencies of source distributions so that it can build wheels all at once after downloading the requirements. I need to move pip off of egg_info as soon as possible and dist_info is required to do that. 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : > On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: > > Just an update for everyone here: > > > > 1. We're currently waiting on the implementation of the 'dist_info" > command > > in the wheel project. > > 2. Once that is done we can switch pip over to reading dist-info rather > than > > egg_info. > > 3. Then we can move the backend over to setuptools. Because Jacob has a > much > > more efficient release system than pip, I anticipate having a release of > > setuptools first and then we can switch pip over to requiring a newer > > setuptools via PEP 518. > > I don't think pip actually has any use for the PEP 517 > prepare_wheel_metadata hook right now though? Historically 'setup.py > egg-info' was needed to kluge around unwanted behavior in 'setup.py > install', but with a PEP 517 backend that's irrelevant because > 'setup.py install' is never used. And in the future when pip has a > real resolver, then prepare_wheel_metadata should allow some > optimizations. But right now, prepare_wheel_metadata is completely > useless AFAIK. > > So why is 'setup.py dist_info' a blocker for things? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Sep 4 20:11:54 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 4 Sep 2017 19:11:54 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: Also if someone with pip write access could please discuss and hopefully merge my initial PR on pip, I would very much appreciate it. Paul seems to be short on time. 2017-09-04 19:09 GMT-05:00 xoviat : > Nathaniel: > > Pip requires egg_info to discover dependencies of source distributions so > that it can build wheels all at once after downloading the requirements. I > need to move pip off of egg_info as soon as possible and dist_info is > required to do that. > > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : > >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: >> > Just an update for everyone here: >> > >> > 1. We're currently waiting on the implementation of the 'dist_info" >> command >> > in the wheel project. >> > 2. Once that is done we can switch pip over to reading dist-info rather >> than >> > egg_info. >> > 3. Then we can move the backend over to setuptools. Because Jacob has a >> much >> > more efficient release system than pip, I anticipate having a release of >> > setuptools first and then we can switch pip over to requiring a newer >> > setuptools via PEP 518. >> >> I don't think pip actually has any use for the PEP 517 >> prepare_wheel_metadata hook right now though? Historically 'setup.py >> egg-info' was needed to kluge around unwanted behavior in 'setup.py >> install', but with a PEP 517 backend that's irrelevant because >> 'setup.py install' is never used. And in the future when pip has a >> real resolver, then prepare_wheel_metadata should allow some >> optimizations. But right now, prepare_wheel_metadata is completely >> useless AFAIK. >> >> So why is 'setup.py dist_info' a blocker for things? >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Sep 4 20:34:27 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 4 Sep 2017 17:34:27 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: > Nathaniel: > > Pip requires egg_info to discover dependencies of source distributions so > that it can build wheels all at once after downloading the requirements. I > need to move pip off of egg_info as soon as possible and dist_info is > required to do that. "Requires" is a strong word -- AFAIK this is just a historical artifact. I don't really know what you're talking about in the second sentence. The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. (In fact I can't see any reason why pip would ever call it before the resolver lands.) So either (a) there's some other reason you want a dist_info command, (b) there's some reason I'm missing why prepare_wheel_metadata matters, or (c) one of us is misunderstanding something :-). -n > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : >> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: >> > Just an update for everyone here: >> > >> > 1. We're currently waiting on the implementation of the 'dist_info" >> > command >> > in the wheel project. >> > 2. Once that is done we can switch pip over to reading dist-info rather >> > than >> > egg_info. >> > 3. Then we can move the backend over to setuptools. Because Jacob has a >> > much >> > more efficient release system than pip, I anticipate having a release of >> > setuptools first and then we can switch pip over to requiring a newer >> > setuptools via PEP 518. >> >> I don't think pip actually has any use for the PEP 517 >> prepare_wheel_metadata hook right now though? Historically 'setup.py >> egg-info' was needed to kluge around unwanted behavior in 'setup.py >> install', but with a PEP 517 backend that's irrelevant because >> 'setup.py install' is never used. And in the future when pip has a >> real resolver, then prepare_wheel_metadata should allow some >> optimizations. But right now, prepare_wheel_metadata is completely >> useless AFAIK. >> >> So why is 'setup.py dist_info' a blocker for things? >> >> -n >> >> -- >> Nathaniel J. Smith -- https://vorpus.org > > -- Nathaniel J. Smith -- https://vorpus.org From xoviat at gmail.com Mon Sep 4 20:51:26 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 4 Sep 2017 19:51:26 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: > The only reason I can think of that setuptools would need a dist_info command would be to implement the PEP 517 prepare_wheel_metadata hook. Yes. That is absolutely correct. > But this hook is optional and in fact provides no value right now, so it can't be a blocker for anything. The simplest way to start on this issue is to replace egg_info in pip with prepare_metadata_for_build_wheel. It is absolutely a historical artifact but I need to work on one issue at a time. The next issue will be replacing egg_info in pip with prepare_metadata_for_build_wheel. 2017-09-04 19:34 GMT-05:00 Nathaniel Smith : > On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: > > Nathaniel: > > > > Pip requires egg_info to discover dependencies of source distributions so > > that it can build wheels all at once after downloading the requirements. > I > > need to move pip off of egg_info as soon as possible and dist_info is > > required to do that. > > "Requires" is a strong word -- AFAIK this is just a historical > artifact. I don't really know what you're talking about in the second > sentence. > > The only reason I can think of that setuptools would need a dist_info > command would be to implement the PEP 517 prepare_wheel_metadata hook. > But this hook is optional and in fact provides no value right now, so > it can't be a blocker for anything. (In fact I can't see any reason > why pip would ever call it before the resolver lands.) So either (a) > there's some other reason you want a dist_info command, (b) there's > some reason I'm missing why prepare_wheel_metadata matters, or (c) one > of us is misunderstanding something :-). > > -n > > > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : > >> > >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: > >> > Just an update for everyone here: > >> > > >> > 1. We're currently waiting on the implementation of the 'dist_info" > >> > command > >> > in the wheel project. > >> > 2. Once that is done we can switch pip over to reading dist-info > rather > >> > than > >> > egg_info. > >> > 3. Then we can move the backend over to setuptools. Because Jacob has > a > >> > much > >> > more efficient release system than pip, I anticipate having a release > of > >> > setuptools first and then we can switch pip over to requiring a newer > >> > setuptools via PEP 518. > >> > >> I don't think pip actually has any use for the PEP 517 > >> prepare_wheel_metadata hook right now though? Historically 'setup.py > >> egg-info' was needed to kluge around unwanted behavior in 'setup.py > >> install', but with a PEP 517 backend that's irrelevant because > >> 'setup.py install' is never used. And in the future when pip has a > >> real resolver, then prepare_wheel_metadata should allow some > >> optimizations. But right now, prepare_wheel_metadata is completely > >> useless AFAIK. > >> > >> So why is 'setup.py dist_info' a blocker for things? > >> > >> -n > >> > >> -- > >> Nathaniel J. Smith -- https://vorpus.org > > > > > > > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Mon Sep 4 21:08:30 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 4 Sep 2017 20:08:30 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: In any case, we're going to need this for prepare_metadata, so the question you should ask is: what are the reasons for *not* merging this? I haven't heard any so far but that doesn't mean that they don't exist. If there are none, then I don't see why we cannot merge my wheel PR and do a release. 2017-09-04 19:51 GMT-05:00 xoviat : > > The only reason I can think of that setuptools would need a dist_info > command would be to implement the PEP 517 prepare_wheel_metadata hook. > > Yes. That is absolutely correct. > > > But this hook is optional and in fact provides no value right now, so > it can't be a blocker for anything. > > The simplest way to start on this issue is to replace egg_info in pip with > prepare_metadata_for_build_wheel. It is absolutely a historical artifact > but I need to work on one issue at a time. The next issue will be replacing > egg_info in pip with prepare_metadata_for_build_wheel. > > 2017-09-04 19:34 GMT-05:00 Nathaniel Smith : > >> On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: >> > Nathaniel: >> > >> > Pip requires egg_info to discover dependencies of source distributions >> so >> > that it can build wheels all at once after downloading the >> requirements. I >> > need to move pip off of egg_info as soon as possible and dist_info is >> > required to do that. >> >> "Requires" is a strong word -- AFAIK this is just a historical >> artifact. I don't really know what you're talking about in the second >> sentence. >> >> The only reason I can think of that setuptools would need a dist_info >> command would be to implement the PEP 517 prepare_wheel_metadata hook. >> But this hook is optional and in fact provides no value right now, so >> it can't be a blocker for anything. (In fact I can't see any reason >> why pip would ever call it before the resolver lands.) So either (a) >> there's some other reason you want a dist_info command, (b) there's >> some reason I'm missing why prepare_wheel_metadata matters, or (c) one >> of us is misunderstanding something :-). >> >> -n >> >> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : >> >> >> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: >> >> > Just an update for everyone here: >> >> > >> >> > 1. We're currently waiting on the implementation of the 'dist_info" >> >> > command >> >> > in the wheel project. >> >> > 2. Once that is done we can switch pip over to reading dist-info >> rather >> >> > than >> >> > egg_info. >> >> > 3. Then we can move the backend over to setuptools. Because Jacob >> has a >> >> > much >> >> > more efficient release system than pip, I anticipate having a >> release of >> >> > setuptools first and then we can switch pip over to requiring a newer >> >> > setuptools via PEP 518. >> >> >> >> I don't think pip actually has any use for the PEP 517 >> >> prepare_wheel_metadata hook right now though? Historically 'setup.py >> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py >> >> install', but with a PEP 517 backend that's irrelevant because >> >> 'setup.py install' is never used. And in the future when pip has a >> >> real resolver, then prepare_wheel_metadata should allow some >> >> optimizations. But right now, prepare_wheel_metadata is completely >> >> useless AFAIK. >> >> >> >> So why is 'setup.py dist_info' a blocker for things? >> >> >> >> -n >> >> >> >> -- >> >> Nathaniel J. Smith -- https://vorpus.org >> > >> > >> >> >> >> -- >> Nathaniel J. Smith -- https://vorpus.org >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Sep 4 21:18:51 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 4 Sep 2017 18:18:51 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: On Mon, Sep 4, 2017 at 5:51 PM, xoviat wrote: >> The only reason I can think of that setuptools would need a dist_info > command would be to implement the PEP 517 prepare_wheel_metadata hook. > > Yes. That is absolutely correct. > >> But this hook is optional and in fact provides no value right now, so > it can't be a blocker for anything. > > The simplest way to start on this issue is to replace egg_info in pip with > prepare_metadata_for_build_wheel. It is absolutely a historical artifact but > I need to work on one issue at a time. The next issue will be replacing > egg_info in pip with prepare_metadata_for_build_wheel. This still doesn't make sense to me. To support PEP 517, pip HAS to support backends that don't provide prepare_metadata_for_build_wheel. You will have to handle this before PEP 517 support can ship. So what's your plan for after you replace egg_info with prepare_metadata_for_build_wheel? Turning around and deleting the code you just wrote? -n > 2017-09-04 19:34 GMT-05:00 Nathaniel Smith : >> >> On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: >> > Nathaniel: >> > >> > Pip requires egg_info to discover dependencies of source distributions >> > so >> > that it can build wheels all at once after downloading the requirements. >> > I >> > need to move pip off of egg_info as soon as possible and dist_info is >> > required to do that. >> >> "Requires" is a strong word -- AFAIK this is just a historical >> artifact. I don't really know what you're talking about in the second >> sentence. >> >> The only reason I can think of that setuptools would need a dist_info >> command would be to implement the PEP 517 prepare_wheel_metadata hook. >> But this hook is optional and in fact provides no value right now, so >> it can't be a blocker for anything. (In fact I can't see any reason >> why pip would ever call it before the resolver lands.) So either (a) >> there's some other reason you want a dist_info command, (b) there's >> some reason I'm missing why prepare_wheel_metadata matters, or (c) one >> of us is misunderstanding something :-). >> >> -n >> >> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : >> >> >> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: >> >> > Just an update for everyone here: >> >> > >> >> > 1. We're currently waiting on the implementation of the 'dist_info" >> >> > command >> >> > in the wheel project. >> >> > 2. Once that is done we can switch pip over to reading dist-info >> >> > rather >> >> > than >> >> > egg_info. >> >> > 3. Then we can move the backend over to setuptools. Because Jacob has >> >> > a >> >> > much >> >> > more efficient release system than pip, I anticipate having a release >> >> > of >> >> > setuptools first and then we can switch pip over to requiring a newer >> >> > setuptools via PEP 518. >> >> >> >> I don't think pip actually has any use for the PEP 517 >> >> prepare_wheel_metadata hook right now though? Historically 'setup.py >> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py >> >> install', but with a PEP 517 backend that's irrelevant because >> >> 'setup.py install' is never used. And in the future when pip has a >> >> real resolver, then prepare_wheel_metadata should allow some >> >> optimizations. But right now, prepare_wheel_metadata is completely >> >> useless AFAIK. >> >> >> >> So why is 'setup.py dist_info' a blocker for things? >> >> >> >> -n >> >> >> >> -- >> >> Nathaniel J. Smith -- https://vorpus.org >> > >> > >> >> >> >> -- >> Nathaniel J. Smith -- https://vorpus.org > > -- Nathaniel J. Smith -- https://vorpus.org From chris.jerdonek at gmail.com Mon Sep 4 21:19:13 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Mon, 4 Sep 2017 18:19:13 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: On Mon, Sep 4, 2017 at 6:08 PM, xoviat wrote: > In any case, we're going to need this for prepare_metadata, so the question > you should ask is: what are the reasons for *not* merging this? I haven't > heard any so far but that doesn't mean that they don't exist. If there are > none, then I don't see why we cannot merge my wheel PR and do a release. FYI, you're not the only one waiting for PR's to be merged. There are 56 other PR's, some of which have already been approved by repo members with commit access. I would try to be a little more patient. Otherwise, everyone can be emailing the list saying the same thing and asking why their PR isn't being merged. --Chris > > 2017-09-04 19:51 GMT-05:00 xoviat : >> >> > The only reason I can think of that setuptools would need a dist_info >> command would be to implement the PEP 517 prepare_wheel_metadata hook. >> >> Yes. That is absolutely correct. >> >> > But this hook is optional and in fact provides no value right now, so >> it can't be a blocker for anything. >> >> The simplest way to start on this issue is to replace egg_info in pip with >> prepare_metadata_for_build_wheel. It is absolutely a historical artifact but >> I need to work on one issue at a time. The next issue will be replacing >> egg_info in pip with prepare_metadata_for_build_wheel. >> >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith : >>> >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: >>> > Nathaniel: >>> > >>> > Pip requires egg_info to discover dependencies of source distributions >>> > so >>> > that it can build wheels all at once after downloading the >>> > requirements. I >>> > need to move pip off of egg_info as soon as possible and dist_info is >>> > required to do that. >>> >>> "Requires" is a strong word -- AFAIK this is just a historical >>> artifact. I don't really know what you're talking about in the second >>> sentence. >>> >>> The only reason I can think of that setuptools would need a dist_info >>> command would be to implement the PEP 517 prepare_wheel_metadata hook. >>> But this hook is optional and in fact provides no value right now, so >>> it can't be a blocker for anything. (In fact I can't see any reason >>> why pip would ever call it before the resolver lands.) So either (a) >>> there's some other reason you want a dist_info command, (b) there's >>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one >>> of us is misunderstanding something :-). >>> >>> -n >>> >>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : >>> >> >>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: >>> >> > Just an update for everyone here: >>> >> > >>> >> > 1. We're currently waiting on the implementation of the 'dist_info" >>> >> > command >>> >> > in the wheel project. >>> >> > 2. Once that is done we can switch pip over to reading dist-info >>> >> > rather >>> >> > than >>> >> > egg_info. >>> >> > 3. Then we can move the backend over to setuptools. Because Jacob >>> >> > has a >>> >> > much >>> >> > more efficient release system than pip, I anticipate having a >>> >> > release of >>> >> > setuptools first and then we can switch pip over to requiring a >>> >> > newer >>> >> > setuptools via PEP 518. >>> >> >>> >> I don't think pip actually has any use for the PEP 517 >>> >> prepare_wheel_metadata hook right now though? Historically 'setup.py >>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py >>> >> install', but with a PEP 517 backend that's irrelevant because >>> >> 'setup.py install' is never used. And in the future when pip has a >>> >> real resolver, then prepare_wheel_metadata should allow some >>> >> optimizations. But right now, prepare_wheel_metadata is completely >>> >> useless AFAIK. >>> >> >>> >> So why is 'setup.py dist_info' a blocker for things? >>> >> >>> >> -n >>> >> >>> >> -- >>> >> Nathaniel J. Smith -- https://vorpus.org >>> > >>> > >>> >>> >>> >>> -- >>> Nathaniel J. Smith -- https://vorpus.org >> >> > From xoviat at gmail.com Mon Sep 4 21:41:59 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 4 Sep 2017 20:41:59 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: The PR that I am taking about is not for pip but for the wheel project. On Sep 4, 2017 8:19 PM, "Chris Jerdonek" wrote: > On Mon, Sep 4, 2017 at 6:08 PM, xoviat wrote: > > In any case, we're going to need this for prepare_metadata, so the > question > > you should ask is: what are the reasons for *not* merging this? I haven't > > heard any so far but that doesn't mean that they don't exist. If there > are > > none, then I don't see why we cannot merge my wheel PR and do a release. > > FYI, you're not the only one waiting for PR's to be merged. There are > 56 other PR's, some of which have already been approved by repo > members with commit access. I would try to be a little more patient. > Otherwise, everyone can be emailing the list saying the same thing and > asking why their PR isn't being merged. > > --Chris > > > > > 2017-09-04 19:51 GMT-05:00 xoviat : > >> > >> > The only reason I can think of that setuptools would need a dist_info > >> command would be to implement the PEP 517 prepare_wheel_metadata hook. > >> > >> Yes. That is absolutely correct. > >> > >> > But this hook is optional and in fact provides no value right now, so > >> it can't be a blocker for anything. > >> > >> The simplest way to start on this issue is to replace egg_info in pip > with > >> prepare_metadata_for_build_wheel. It is absolutely a historical > artifact but > >> I need to work on one issue at a time. The next issue will be replacing > >> egg_info in pip with prepare_metadata_for_build_wheel. > >> > >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith : > >>> > >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: > >>> > Nathaniel: > >>> > > >>> > Pip requires egg_info to discover dependencies of source > distributions > >>> > so > >>> > that it can build wheels all at once after downloading the > >>> > requirements. I > >>> > need to move pip off of egg_info as soon as possible and dist_info is > >>> > required to do that. > >>> > >>> "Requires" is a strong word -- AFAIK this is just a historical > >>> artifact. I don't really know what you're talking about in the second > >>> sentence. > >>> > >>> The only reason I can think of that setuptools would need a dist_info > >>> command would be to implement the PEP 517 prepare_wheel_metadata hook. > >>> But this hook is optional and in fact provides no value right now, so > >>> it can't be a blocker for anything. (In fact I can't see any reason > >>> why pip would ever call it before the resolver lands.) So either (a) > >>> there's some other reason you want a dist_info command, (b) there's > >>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one > >>> of us is misunderstanding something :-). > >>> > >>> -n > >>> > >>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : > >>> >> > >>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: > >>> >> > Just an update for everyone here: > >>> >> > > >>> >> > 1. We're currently waiting on the implementation of the > 'dist_info" > >>> >> > command > >>> >> > in the wheel project. > >>> >> > 2. Once that is done we can switch pip over to reading dist-info > >>> >> > rather > >>> >> > than > >>> >> > egg_info. > >>> >> > 3. Then we can move the backend over to setuptools. Because Jacob > >>> >> > has a > >>> >> > much > >>> >> > more efficient release system than pip, I anticipate having a > >>> >> > release of > >>> >> > setuptools first and then we can switch pip over to requiring a > >>> >> > newer > >>> >> > setuptools via PEP 518. > >>> >> > >>> >> I don't think pip actually has any use for the PEP 517 > >>> >> prepare_wheel_metadata hook right now though? Historically 'setup.py > >>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py > >>> >> install', but with a PEP 517 backend that's irrelevant because > >>> >> 'setup.py install' is never used. And in the future when pip has a > >>> >> real resolver, then prepare_wheel_metadata should allow some > >>> >> optimizations. But right now, prepare_wheel_metadata is completely > >>> >> useless AFAIK. > >>> >> > >>> >> So why is 'setup.py dist_info' a blocker for things? > >>> >> > >>> >> -n > >>> >> > >>> >> -- > >>> >> Nathaniel J. Smith -- https://vorpus.org > >>> > > >>> > > >>> > >>> > >>> > >>> -- > >>> Nathaniel J. Smith -- https://vorpus.org > >> > >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Mon Sep 4 22:00:49 2017 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Mon, 4 Sep 2017 19:00:49 -0700 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: On Mon, Sep 4, 2017 at 6:41 PM, xoviat wrote: > The PR that I am taking about is not for pip but for the wheel project. Okay, well you started the thread asking something similar for your pip PR (see below). I'm sure similar considerations hold for the wheel project. On Mon, Sep 4, 2017 at 5:11 PM, xoviat wrote: > Also if someone with pip write access could please discuss and hopefully > merge my initial PR on pip, I would very much appreciate it. Paul seems to > be short on time. --Chris > > On Sep 4, 2017 8:19 PM, "Chris Jerdonek" wrote: >> >> On Mon, Sep 4, 2017 at 6:08 PM, xoviat wrote: >> > In any case, we're going to need this for prepare_metadata, so the >> > question >> > you should ask is: what are the reasons for *not* merging this? I >> > haven't >> > heard any so far but that doesn't mean that they don't exist. If there >> > are >> > none, then I don't see why we cannot merge my wheel PR and do a release. >> >> FYI, you're not the only one waiting for PR's to be merged. There are >> 56 other PR's, some of which have already been approved by repo >> members with commit access. I would try to be a little more patient. >> Otherwise, everyone can be emailing the list saying the same thing and >> asking why their PR isn't being merged. >> >> --Chris >> >> > >> > 2017-09-04 19:51 GMT-05:00 xoviat : >> >> >> >> > The only reason I can think of that setuptools would need a dist_info >> >> command would be to implement the PEP 517 prepare_wheel_metadata hook. >> >> >> >> Yes. That is absolutely correct. >> >> >> >> > But this hook is optional and in fact provides no value right now, so >> >> it can't be a blocker for anything. >> >> >> >> The simplest way to start on this issue is to replace egg_info in pip >> >> with >> >> prepare_metadata_for_build_wheel. It is absolutely a historical >> >> artifact but >> >> I need to work on one issue at a time. The next issue will be replacing >> >> egg_info in pip with prepare_metadata_for_build_wheel. >> >> >> >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith : >> >>> >> >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: >> >>> > Nathaniel: >> >>> > >> >>> > Pip requires egg_info to discover dependencies of source >> >>> > distributions >> >>> > so >> >>> > that it can build wheels all at once after downloading the >> >>> > requirements. I >> >>> > need to move pip off of egg_info as soon as possible and dist_info >> >>> > is >> >>> > required to do that. >> >>> >> >>> "Requires" is a strong word -- AFAIK this is just a historical >> >>> artifact. I don't really know what you're talking about in the second >> >>> sentence. >> >>> >> >>> The only reason I can think of that setuptools would need a dist_info >> >>> command would be to implement the PEP 517 prepare_wheel_metadata hook. >> >>> But this hook is optional and in fact provides no value right now, so >> >>> it can't be a blocker for anything. (In fact I can't see any reason >> >>> why pip would ever call it before the resolver lands.) So either (a) >> >>> there's some other reason you want a dist_info command, (b) there's >> >>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one >> >>> of us is misunderstanding something :-). >> >>> >> >>> -n >> >>> >> >>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : >> >>> >> >> >>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat wrote: >> >>> >> > Just an update for everyone here: >> >>> >> > >> >>> >> > 1. We're currently waiting on the implementation of the >> >>> >> > 'dist_info" >> >>> >> > command >> >>> >> > in the wheel project. >> >>> >> > 2. Once that is done we can switch pip over to reading dist-info >> >>> >> > rather >> >>> >> > than >> >>> >> > egg_info. >> >>> >> > 3. Then we can move the backend over to setuptools. Because Jacob >> >>> >> > has a >> >>> >> > much >> >>> >> > more efficient release system than pip, I anticipate having a >> >>> >> > release of >> >>> >> > setuptools first and then we can switch pip over to requiring a >> >>> >> > newer >> >>> >> > setuptools via PEP 518. >> >>> >> >> >>> >> I don't think pip actually has any use for the PEP 517 >> >>> >> prepare_wheel_metadata hook right now though? Historically >> >>> >> 'setup.py >> >>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py >> >>> >> install', but with a PEP 517 backend that's irrelevant because >> >>> >> 'setup.py install' is never used. And in the future when pip has a >> >>> >> real resolver, then prepare_wheel_metadata should allow some >> >>> >> optimizations. But right now, prepare_wheel_metadata is completely >> >>> >> useless AFAIK. >> >>> >> >> >>> >> So why is 'setup.py dist_info' a blocker for things? >> >>> >> >> >>> >> -n >> >>> >> >> >>> >> -- >> >>> >> Nathaniel J. Smith -- https://vorpus.org >> >>> > >> >>> > >> >>> >> >>> >> >>> >> >>> -- >> >>> Nathaniel J. Smith -- https://vorpus.org >> >> >> >> >> > From xoviat at gmail.com Mon Sep 4 22:04:36 2017 From: xoviat at gmail.com (xoviat) Date: Mon, 4 Sep 2017 21:04:36 -0500 Subject: [Distutils] PEP 517 again In-Reply-To: References: <-6173407393987566310@unknownmsgid> <135C9921-2072-41FB-B2B5-ABFFC5A8B3CF@stufft.io> Message-ID: Supposedly there is some meeting tomorrow concerning the wheel project that will determine the fate of dist_info. So that is why I bought it up. On Sep 4, 2017 9:00 PM, "Chris Jerdonek" wrote: > On Mon, Sep 4, 2017 at 6:41 PM, xoviat wrote: > > The PR that I am taking about is not for pip but for the wheel project. > > Okay, well you started the thread asking something similar for your > pip PR (see below). I'm sure similar considerations hold for the wheel > project. > > On Mon, Sep 4, 2017 at 5:11 PM, xoviat wrote: > > Also if someone with pip write access could please discuss and hopefully > > merge my initial PR on pip, I would very much appreciate it. Paul seems > to > > be short on time. > > --Chris > > > > > > On Sep 4, 2017 8:19 PM, "Chris Jerdonek" > wrote: > >> > >> On Mon, Sep 4, 2017 at 6:08 PM, xoviat wrote: > >> > In any case, we're going to need this for prepare_metadata, so the > >> > question > >> > you should ask is: what are the reasons for *not* merging this? I > >> > haven't > >> > heard any so far but that doesn't mean that they don't exist. If there > >> > are > >> > none, then I don't see why we cannot merge my wheel PR and do a > release. > >> > >> FYI, you're not the only one waiting for PR's to be merged. There are > >> 56 other PR's, some of which have already been approved by repo > >> members with commit access. I would try to be a little more patient. > >> Otherwise, everyone can be emailing the list saying the same thing and > >> asking why their PR isn't being merged. > >> > >> --Chris > >> > >> > > >> > 2017-09-04 19:51 GMT-05:00 xoviat : > >> >> > >> >> > The only reason I can think of that setuptools would need a > dist_info > >> >> command would be to implement the PEP 517 prepare_wheel_metadata > hook. > >> >> > >> >> Yes. That is absolutely correct. > >> >> > >> >> > But this hook is optional and in fact provides no value right now, > so > >> >> it can't be a blocker for anything. > >> >> > >> >> The simplest way to start on this issue is to replace egg_info in pip > >> >> with > >> >> prepare_metadata_for_build_wheel. It is absolutely a historical > >> >> artifact but > >> >> I need to work on one issue at a time. The next issue will be > replacing > >> >> egg_info in pip with prepare_metadata_for_build_wheel. > >> >> > >> >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith : > >> >>> > >> >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat wrote: > >> >>> > Nathaniel: > >> >>> > > >> >>> > Pip requires egg_info to discover dependencies of source > >> >>> > distributions > >> >>> > so > >> >>> > that it can build wheels all at once after downloading the > >> >>> > requirements. I > >> >>> > need to move pip off of egg_info as soon as possible and dist_info > >> >>> > is > >> >>> > required to do that. > >> >>> > >> >>> "Requires" is a strong word -- AFAIK this is just a historical > >> >>> artifact. I don't really know what you're talking about in the > second > >> >>> sentence. > >> >>> > >> >>> The only reason I can think of that setuptools would need a > dist_info > >> >>> command would be to implement the PEP 517 prepare_wheel_metadata > hook. > >> >>> But this hook is optional and in fact provides no value right now, > so > >> >>> it can't be a blocker for anything. (In fact I can't see any reason > >> >>> why pip would ever call it before the resolver lands.) So either (a) > >> >>> there's some other reason you want a dist_info command, (b) there's > >> >>> some reason I'm missing why prepare_wheel_metadata matters, or (c) > one > >> >>> of us is misunderstanding something :-). > >> >>> > >> >>> -n > >> >>> > >> >>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith : > >> >>> >> > >> >>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat > wrote: > >> >>> >> > Just an update for everyone here: > >> >>> >> > > >> >>> >> > 1. We're currently waiting on the implementation of the > >> >>> >> > 'dist_info" > >> >>> >> > command > >> >>> >> > in the wheel project. > >> >>> >> > 2. Once that is done we can switch pip over to reading > dist-info > >> >>> >> > rather > >> >>> >> > than > >> >>> >> > egg_info. > >> >>> >> > 3. Then we can move the backend over to setuptools. Because > Jacob > >> >>> >> > has a > >> >>> >> > much > >> >>> >> > more efficient release system than pip, I anticipate having a > >> >>> >> > release of > >> >>> >> > setuptools first and then we can switch pip over to requiring a > >> >>> >> > newer > >> >>> >> > setuptools via PEP 518. > >> >>> >> > >> >>> >> I don't think pip actually has any use for the PEP 517 > >> >>> >> prepare_wheel_metadata hook right now though? Historically > >> >>> >> 'setup.py > >> >>> >> egg-info' was needed to kluge around unwanted behavior in > 'setup.py > >> >>> >> install', but with a PEP 517 backend that's irrelevant because > >> >>> >> 'setup.py install' is never used. And in the future when pip has > a > >> >>> >> real resolver, then prepare_wheel_metadata should allow some > >> >>> >> optimizations. But right now, prepare_wheel_metadata is > completely > >> >>> >> useless AFAIK. > >> >>> >> > >> >>> >> So why is 'setup.py dist_info' a blocker for things? > >> >>> >> > >> >>> >> -n > >> >>> >> > >> >>> >> -- > >> >>> >> Nathaniel J. Smith -- https://vorpus.org > >> >>> > > >> >>> > > >> >>> > >> >>> > >> >>> > >> >>> -- > >> >>> Nathaniel J. Smith -- https://vorpus.org > >> >> > >> >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Mon Sep 4 22:52:13 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 5 Sep 2017 10:52:13 +0800 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: Thanks a lot Nick. I filed a bug: https://github.com/pypa/pypi-legacy/issues/700 On Mon, Sep 4, 2017 at 9:56 PM, Nick Coghlan wrote: > On 2 September 2017 at 15:34, Giampaolo Rodola' > wrote: > > I know it was deprecated long ago in favor of readthedocs but I kept > > postponing it and my doc is still hosted on > > https://pythonhosted.org/psutil/. > > While we've talked about deprecating it, it *hasn't* been deprecated. > Looking at https://github.com/pypa/pypi-legacy/commits/production, I'm > not seeing anything obvious that would have caused problems with docs > management, but that's probably still the best issue tracker to use to > report the bug. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From berker.peksag at gmail.com Tue Sep 5 00:19:32 2017 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Tue, 5 Sep 2017 07:19:32 +0300 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan wrote: > On 2 September 2017 at 15:34, Giampaolo Rodola' wrote: >> I know it was deprecated long ago in favor of readthedocs but I kept >> postponing it and my doc is still hosted on >> https://pythonhosted.org/psutil/. > > While we've talked about deprecating it, it *hasn't* been deprecated. > Looking at https://github.com/pypa/pypi-legacy/commits/production, I'm > not seeing anything obvious that would have caused problems with docs > management, but that's probably still the best issue tracker to use to > report the bug. See the 'doc_upload' handler at https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393891f6c6bcbf84c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804 I've collected all information I found at https://github.com/pypa/pypi-legacy/issues/672#issuecomment-316125918 Please correct me if I missed anything. --Berker From g.rodola at gmail.com Tue Sep 5 00:43:16 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 5 Sep 2017 12:43:16 +0800 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: I think it's wise to revert that commit. It seems pythonhosted only suggested to migrate to RTD but there never was an official shutdown date or warning (either via direct email or message on the web page). On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? wrote: > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan wrote: > > On 2 September 2017 at 15:34, Giampaolo Rodola' > wrote: > >> I know it was deprecated long ago in favor of readthedocs but I kept > >> postponing it and my doc is still hosted on > >> https://pythonhosted.org/psutil/. > > > > While we've talked about deprecating it, it *hasn't* been deprecated. > > Looking at https://github.com/pypa/pypi-legacy/commits/production, I'm > > not seeing anything obvious that would have caused problems with docs > > management, but that's probably still the best issue tracker to use to > > report the bug. > > See the 'doc_upload' handler at > https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393891f6c6bcbf84 > c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804 > I've collected all information I found at > https://github.com/pypa/pypi-legacy/issues/672#issuecomment-316125918 > Please correct me if I missed anything. > > --Berker > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Tue Sep 5 00:49:39 2017 From: noah at coderanger.net (Noah Kantrowitz) Date: Mon, 4 Sep 2017 21:49:39 -0700 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: This was discussed several years ago in https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html and a few other threads. The final phases went out earlier this year. I don't think there is any plan to re-enable uploads to pythonhosted at this time. If you want a one-off redirect change or just having the old files removed, we can probably do that though, but I very much defer to Donald and others on that :) --Noah > On Sep 4, 2017, at 9:43 PM, Giampaolo Rodola' wrote: > > I think it's wise to revert that commit. It seems pythonhosted only suggested to migrate to RTD but there never was an official shutdown date or warning (either via direct email or message on the web page). > > On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? wrote: > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan wrote: > > On 2 September 2017 at 15:34, Giampaolo Rodola' wrote: > >> I know it was deprecated long ago in favor of readthedocs but I kept > >> postponing it and my doc is still hosted on > >> https://pythonhosted.org/psutil/. > > > > While we've talked about deprecating it, it *hasn't* been deprecated. > > Looking at https://github.com/pypa/pypi-legacy/commits/production, I'm > > not seeing anything obvious that would have caused problems with docs > > management, but that's probably still the best issue tracker to use to > > report the bug. > > See the 'doc_upload' handler at > https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393891f6c6bcbf84c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804 > I've collected all information I found at > https://github.com/pypa/pypi-legacy/issues/672#issuecomment-316125918 > Please correct me if I missed anything. > > --Berker > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > > -- > Giampaolo - http://grodola.blogspot.com > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From njs at pobox.com Tue Sep 5 02:33:01 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 4 Sep 2017 23:33:01 -0700 Subject: [Distutils] string types for paths in PEP 517 Message-ID: Hi all, Quick question about an arcane topic: currently, PEP 517 says that paths are always represented as unicode strings. For example, when the frontend calls build_wheel, it has to create a temporary dir to hold the output wheel, and it passes this in as an absolute path represented as a unicode string. In Python 3 I think this is totally fine, because the surrogate-escape system means that all paths can be represented as unicode strings, even on systems like Linux where you can have paths that are invalid according to Python's idea of the filesystem encoding. In Python 2, if I understand correctly (and I'm not super confident that I do), then there is no surrogate-escape, and it's possible to have paths that can't be represented as a unicode object. For example, if someone's home directory is /home/st?fan in UTF-8 but Python thinks that the locale is C, and a frontend tries to make a tmpdir in $HOME/.local/tmp/ and pass it to a backend then... everything blows up, I guess? So I guess this is a question for those unfortunate souls who understand these details better than me (hi Nick!): is this actually a problem, and is there anything we can/should do differently? -n -- Nathaniel J. Smith -- https://vorpus.org From g.rodola at gmail.com Tue Sep 5 03:31:56 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 5 Sep 2017 15:31:56 +0800 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: A bare redirect would also be fine. It's a shame setup.py classifiers / metadata does not have a "documentation" field otherwise pythonhosted.org/{project_name} could have automatically redirected to the project doc. Perhaps https://pypi.python.org/pypi?:action=pkg_edit&name={project_name} could provide a form where this can be specified. What do you think? On Tue, Sep 5, 2017 at 12:49 PM, Noah Kantrowitz wrote: > This was discussed several years ago in https://mail.python.org/ > pipermail/distutils-sig/2015-May/026381.html and a few other threads. The > final phases went out earlier this year. I don't think there is any plan to > re-enable uploads to pythonhosted at this time. If you want a one-off > redirect change or just having the old files removed, we can probably do > that though, but I very much defer to Donald and others on that :) > > --Noah > > > On Sep 4, 2017, at 9:43 PM, Giampaolo Rodola' > wrote: > > > > I think it's wise to revert that commit. It seems pythonhosted only > suggested to migrate to RTD but there never was an official shutdown date > or warning (either via direct email or message on the web page). > > > > On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? > wrote: > > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan wrote: > > > On 2 September 2017 at 15:34, Giampaolo Rodola' > wrote: > > >> I know it was deprecated long ago in favor of readthedocs but I kept > > >> postponing it and my doc is still hosted on > > >> https://pythonhosted.org/psutil/. > > > > > > While we've talked about deprecating it, it *hasn't* been deprecated. > > > Looking at https://github.com/pypa/pypi-legacy/commits/production, I'm > > > not seeing anything obvious that would have caused problems with docs > > > management, but that's probably still the best issue tracker to use to > > > report the bug. > > > > See the 'doc_upload' handler at > > https://github.com/pypa/pypi-legacy/commit/ > 1598e6ea0f7fb0393891f6c6bcbf84c191834a0e#diff- > 19fadc30e1b17100568adbd8c6c3cc13R2804 > > I've collected all information I found at > > https://github.com/pypa/pypi-legacy/issues/672#issuecomment-316125918 > > Please correct me if I missed anything. > > > > --Berker > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > > > > > -- > > Giampaolo - http://grodola.blogspot.com > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Tue Sep 5 04:00:48 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 05 Sep 2017 09:00:48 +0100 Subject: [Distutils] string types for paths in PEP 517 In-Reply-To: References: Message-ID: <1504598448.3904789.1095432616.2CFC0003@webmail.messagingengine.com> I considered this. It's *potentially* a problem, but I think we should not try to deal with it for now: - Normally, temp files will go in /tmp - so it should be fine to construct paths of entirely ascii characters. - Frontends that want the wheel to end up elsewhere can ask for it in a tmp directory first and then move it, so there's a workaround if it becomes an issue. - We already have workarounds for the commonest case of UTF-8 paths + C locale: ignore the locale and treat paths as UTF-8. - The 'right' way to deal with it on Unix is to make all paths bytes, which would introduce a similar issue on Windows. If paths have to be bytes in some situations and unicode in others, both frontends and backends need extra complexity to handle that. - If your non-ascii username breaks stuff on Python 2... Python 3 is ready to make your life easier. Thomas On Tue, Sep 5, 2017, at 07:33 AM, Nathaniel Smith wrote: > Hi all, > > Quick question about an arcane topic: currently, PEP 517 says that > paths are always represented as unicode strings. For example, when the > frontend calls build_wheel, it has to create a temporary dir to hold > the output wheel, and it passes this in as an absolute path > represented as a unicode string. > > In Python 3 I think this is totally fine, because the surrogate-escape > system means that all paths can be represented as unicode strings, > even on systems like Linux where you can have paths that are invalid > according to Python's idea of the filesystem encoding. > > In Python 2, if I understand correctly (and I'm not super confident > that I do), then there is no surrogate-escape, and it's possible to > have paths that can't be represented as a unicode object. For example, > if someone's home directory is /home/st?fan in UTF-8 but Python thinks > that the locale is C, and a frontend tries to make a tmpdir in > $HOME/.local/tmp/ and pass it to a backend then... everything blows > up, I guess? > > So I guess this is a question for those unfortunate souls who > understand these details better than me (hi Nick!): is this actually a > problem, and is there anything we can/should do differently? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From p.f.moore at gmail.com Tue Sep 5 04:21:31 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 5 Sep 2017 09:21:31 +0100 Subject: [Distutils] string types for paths in PEP 517 In-Reply-To: <1504598448.3904789.1095432616.2CFC0003@webmail.messagingengine.com> References: <1504598448.3904789.1095432616.2CFC0003@webmail.messagingengine.com> Message-ID: On 5 September 2017 at 09:00, Thomas Kluyver wrote: > I considered this. It's *potentially* a problem, but I think we should > not try to deal with it for now: > > - Normally, temp files will go in /tmp - so it should be fine to > construct paths of entirely ascii characters. > - Frontends that want the wheel to end up elsewhere can ask for it in a > tmp directory first and then move it, so there's a workaround if it > becomes an issue. > - We already have workarounds for the commonest case of UTF-8 paths + C > locale: ignore the locale and treat paths as UTF-8. > - The 'right' way to deal with it on Unix is to make all paths bytes, > which would introduce a similar issue on Windows. If paths have to be > bytes in some situations and unicode in others, both frontends and > backends need extra complexity to handle that. > - If your non-ascii username breaks stuff on Python 2... Python 3 is > ready to make your life easier. +1 on this Paul From njs at pobox.com Tue Sep 5 04:36:11 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 5 Sep 2017 01:36:11 -0700 Subject: [Distutils] string types for paths in PEP 517 In-Reply-To: <1504598448.3904789.1095432616.2CFC0003@webmail.messagingengine.com> References: <1504598448.3904789.1095432616.2CFC0003@webmail.messagingengine.com> Message-ID: On Tue, Sep 5, 2017 at 1:00 AM, Thomas Kluyver wrote: > I considered this. It's *potentially* a problem, but I think we should > not try to deal with it for now: > > - Normally, temp files will go in /tmp - so it should be fine to > construct paths of entirely ascii characters. Does pip in fact use /tmp for temporary directories? (It's not always the right choice, because /tmp has limited space on some systems, e.g. b/c it's on a ramdisk. If we still had build_directory= then this could be an issue, since build directories can be arbitrarily large; maybe it's not a big deal now that we only need the tmpdir to handle a single sdist/wheel/dist-info.) > - Frontends that want the wheel to end up elsewhere can ask for it in a > tmp directory first and then move it, so there's a workaround if it > becomes an issue. > - We already have workarounds for the commonest case of UTF-8 paths + C > locale: ignore the locale and treat paths as UTF-8. Only in 3.7, I think? Or do you mean, backends should be doing this manually on Python 2? (To be clear, I think the current text is potentially fine, I just want to make sure I/we understand the full consequences instead of discovering them a year from now when we're stuck with them :-).) -n -- Nathaniel J. Smith -- https://vorpus.org From donald at stufft.io Tue Sep 5 09:08:06 2017 From: donald at stufft.io (Donald Stufft) Date: Tue, 5 Sep 2017 09:08:06 -0400 Subject: [Distutils] string types for paths in PEP 517 In-Reply-To: References: <1504598448.3904789.1095432616.2CFC0003@webmail.messagingengine.com> Message-ID: <235EE4FF-B31A-4D54-B915-3A38E2FC9B49@stufft.io> > > On Sep 5, 2017, at 4:36 AM, Nathaniel Smith wrote: > > Does pip in fact use /tmp for temporary directories? (It's not always > the right choice, because /tmp has limited space on some systems, e.g. > b/c it's on a ramdisk. If we still had build_directory= then this > could be an issue, since build directories can be arbitrarily large; > maybe it's not a big deal now that we only need the tmpdir to handle a > single sdist/wheel/dist-info.) It does by default yes. It just uses the tempfile module so it respects $TMPDIR directory and everything if people want to point it to a different directory. From ncoghlan at gmail.com Tue Sep 5 10:15:10 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 5 Sep 2017 07:15:10 -0700 Subject: [Distutils] string types for paths in PEP 517 In-Reply-To: References: <1504598448.3904789.1095432616.2CFC0003@webmail.messagingengine.com> Message-ID: On 5 September 2017 at 01:36, Nathaniel Smith wrote: > On Tue, Sep 5, 2017 at 1:00 AM, Thomas Kluyver wrote: >> - We already have workarounds for the commonest case of UTF-8 paths + C >> locale: ignore the locale and treat paths as UTF-8. > > Only in 3.7, I think? Or do you mean, backends should be doing this > manually on Python 2? The frontend controls the locale that the backend runs in, so the frontend can set C.UTF-8 even if the frontend itself is launched in the C locale. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From xoviat at gmail.com Tue Sep 5 16:08:17 2017 From: xoviat at gmail.com (xoviat) Date: Tue, 5 Sep 2017 15:08:17 -0500 Subject: [Distutils] string types for paths in PEP 517 In-Reply-To: References: <1504598448.3904789.1095432616.2CFC0003@webmail.messagingengine.com> Message-ID: +1 2017-09-05 3:21 GMT-05:00 Paul Moore : > On 5 September 2017 at 09:00, Thomas Kluyver wrote: > > I considered this. It's *potentially* a problem, but I think we should > > not try to deal with it for now: > > > > - Normally, temp files will go in /tmp - so it should be fine to > > construct paths of entirely ascii characters. > > - Frontends that want the wheel to end up elsewhere can ask for it in a > > tmp directory first and then move it, so there's a workaround if it > > becomes an issue. > > - We already have workarounds for the commonest case of UTF-8 paths + C > > locale: ignore the locale and treat paths as UTF-8. > > - The 'right' way to deal with it on Unix is to make all paths bytes, > > which would introduce a similar issue on Windows. If paths have to be > > bytes in some situations and unicode in others, both frontends and > > backends need extra complexity to handle that. > > - If your non-ascii username breaks stuff on Python 2... Python 3 is > > ready to make your life easier. > > +1 on this > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Sep 5 16:37:49 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 5 Sep 2017 13:37:49 -0700 Subject: [Distutils] PEP 517 again Message-ID: On Mon, Sep 4, 2017 at 6:00 AM, Nick Coghlan wrote: > >> Do the Linux distros use pip to build their packages? > > > > Not that I am aware of. > > Fedora's build macros for Python projects currently rely on running > setup.py directly, but we've been considering switching to pip instead > since 2013 or so. PEP 517 is likely to provide the impetus to switch > from "maybe we should do that" to "we need to do that, at least if > setup.py is missing, and potentially always, so we get more consistent > installation metadata" > which is exactly why I tried to do it in conda. In a post PEP 517 world, that may be a good way to go. I'm looking forward to it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Thu Sep 7 04:10:33 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Thu, 7 Sep 2017 16:10:33 +0800 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: OK I updated the new doc at http://psutil.readthedocs.io/ Is it possible to redirect the old doc here? On Tue, Sep 5, 2017 at 3:31 PM, Giampaolo Rodola' wrote: > A bare redirect would also be fine. It's a shame setup.py classifiers / > metadata does not have a "documentation" field otherwise > pythonhosted.org/{project_name} > could have automatically > redirected to the project doc. > Perhaps https://pypi.python.org/pypi?:action=pkg_edit&name={project_name} > could provide a form where this can be specified. > What do you think? > > On Tue, Sep 5, 2017 at 12:49 PM, Noah Kantrowitz > wrote: > >> This was discussed several years ago in https://mail.python.org/piperm >> ail/distutils-sig/2015-May/026381.html and a few other threads. The >> final phases went out earlier this year. I don't think there is any plan to >> re-enable uploads to pythonhosted at this time. If you want a one-off >> redirect change or just having the old files removed, we can probably do >> that though, but I very much defer to Donald and others on that :) >> >> --Noah >> >> > On Sep 4, 2017, at 9:43 PM, Giampaolo Rodola' >> wrote: >> > >> > I think it's wise to revert that commit. It seems pythonhosted only >> suggested to migrate to RTD but there never was an official shutdown date >> or warning (either via direct email or message on the web page). >> > >> > On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? >> wrote: >> > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan >> wrote: >> > > On 2 September 2017 at 15:34, Giampaolo Rodola' >> wrote: >> > >> I know it was deprecated long ago in favor of readthedocs but I kept >> > >> postponing it and my doc is still hosted on >> > >> https://pythonhosted.org/psutil/. >> > > >> > > While we've talked about deprecating it, it *hasn't* been deprecated. >> > > Looking at https://github.com/pypa/pypi-legacy/commits/production, >> I'm >> > > not seeing anything obvious that would have caused problems with docs >> > > management, but that's probably still the best issue tracker to use to >> > > report the bug. >> > >> > See the 'doc_upload' handler at >> > https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393 >> 891f6c6bcbf84c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804 >> > I've collected all information I found at >> > https://github.com/pypa/pypi-legacy/issues/672#issuecomment-316125918 >> > Please correct me if I missed anything. >> > >> > --Berker >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> > >> > >> > >> > -- >> > Giampaolo - http://grodola.blogspot.com >> > >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > > > -- > Giampaolo - http://grodola.blogspot.com > > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Thu Sep 7 05:28:54 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Thu, 7 Sep 2017 17:28:54 +0800 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: ...and also http://pyftpdlib.readthedocs.io/ if possible; thanks a lot! On Thu, Sep 7, 2017 at 4:10 PM, Giampaolo Rodola' wrote: > OK I updated the new doc at http://psutil.readthedocs.io/ > Is it possible to redirect the old doc here? > > On Tue, Sep 5, 2017 at 3:31 PM, Giampaolo Rodola' > wrote: > >> A bare redirect would also be fine. It's a shame setup.py classifiers / >> metadata does not have a "documentation" field otherwise >> pythonhosted.org/{project_name} >> could have automatically >> redirected to the project doc. >> Perhaps https://pypi.python.org/pypi?:action=pkg_edit&name={project_name} >> could provide a form where this can be specified. >> What do you think? >> >> On Tue, Sep 5, 2017 at 12:49 PM, Noah Kantrowitz >> wrote: >> >>> This was discussed several years ago in https://mail.python.org/piperm >>> ail/distutils-sig/2015-May/026381.html and a few other threads. The >>> final phases went out earlier this year. I don't think there is any plan to >>> re-enable uploads to pythonhosted at this time. If you want a one-off >>> redirect change or just having the old files removed, we can probably do >>> that though, but I very much defer to Donald and others on that :) >>> >>> --Noah >>> >>> > On Sep 4, 2017, at 9:43 PM, Giampaolo Rodola' >>> wrote: >>> > >>> > I think it's wise to revert that commit. It seems pythonhosted only >>> suggested to migrate to RTD but there never was an official shutdown date >>> or warning (either via direct email or message on the web page). >>> > >>> > On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? < >>> berker.peksag at gmail.com> wrote: >>> > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan >>> wrote: >>> > > On 2 September 2017 at 15:34, Giampaolo Rodola' >>> wrote: >>> > >> I know it was deprecated long ago in favor of readthedocs but I kept >>> > >> postponing it and my doc is still hosted on >>> > >> https://pythonhosted.org/psutil/. >>> > > >>> > > While we've talked about deprecating it, it *hasn't* been deprecated. >>> > > Looking at https://github.com/pypa/pypi-legacy/commits/production, >>> I'm >>> > > not seeing anything obvious that would have caused problems with docs >>> > > management, but that's probably still the best issue tracker to use >>> to >>> > > report the bug. >>> > >>> > See the 'doc_upload' handler at >>> > https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393 >>> 891f6c6bcbf84c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804 >>> > I've collected all information I found at >>> > https://github.com/pypa/pypi-legacy/issues/672#issuecomment-316125918 >>> > Please correct me if I missed anything. >>> > >>> > --Berker >>> > _______________________________________________ >>> > Distutils-SIG maillist - Distutils-SIG at python.org >>> > https://mail.python.org/mailman/listinfo/distutils-sig >>> > >>> > >>> > >>> > -- >>> > Giampaolo - http://grodola.blogspot.com >>> > >>> > _______________________________________________ >>> > Distutils-SIG maillist - Distutils-SIG at python.org >>> > https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >> >> >> >> -- >> Giampaolo - http://grodola.blogspot.com >> >> > > > -- > Giampaolo - http://grodola.blogspot.com > > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Thu Sep 7 05:29:27 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Thu, 7 Sep 2017 17:29:27 +0800 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: Whops! I meant: http://pythonhosted.org/pyftpdlib/ On Thu, Sep 7, 2017 at 5:28 PM, Giampaolo Rodola' wrote: > ...and also http://pyftpdlib.readthedocs.io/ > if possible; thanks a lot! > > On Thu, Sep 7, 2017 at 4:10 PM, Giampaolo Rodola' > wrote: > >> OK I updated the new doc at http://psutil.readthedocs.io/ >> Is it possible to redirect the old doc here? >> >> On Tue, Sep 5, 2017 at 3:31 PM, Giampaolo Rodola' >> wrote: >> >>> A bare redirect would also be fine. It's a shame setup.py classifiers / >>> metadata does not have a "documentation" field otherwise >>> pythonhosted.org/{project_name} >>> could have automatically >>> redirected to the project doc. >>> Perhaps https://pypi.python.org/pypi?:action=pkg_edit&name={ >>> project_name} could provide a form where this can be specified. >>> What do you think? >>> >>> On Tue, Sep 5, 2017 at 12:49 PM, Noah Kantrowitz >>> wrote: >>> >>>> This was discussed several years ago in https://mail.python.org/piperm >>>> ail/distutils-sig/2015-May/026381.html and a few other threads. The >>>> final phases went out earlier this year. I don't think there is any plan to >>>> re-enable uploads to pythonhosted at this time. If you want a one-off >>>> redirect change or just having the old files removed, we can probably do >>>> that though, but I very much defer to Donald and others on that :) >>>> >>>> --Noah >>>> >>>> > On Sep 4, 2017, at 9:43 PM, Giampaolo Rodola' >>>> wrote: >>>> > >>>> > I think it's wise to revert that commit. It seems pythonhosted only >>>> suggested to migrate to RTD but there never was an official shutdown date >>>> or warning (either via direct email or message on the web page). >>>> > >>>> > On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? < >>>> berker.peksag at gmail.com> wrote: >>>> > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan >>>> wrote: >>>> > > On 2 September 2017 at 15:34, Giampaolo Rodola' >>>> wrote: >>>> > >> I know it was deprecated long ago in favor of readthedocs but I >>>> kept >>>> > >> postponing it and my doc is still hosted on >>>> > >> https://pythonhosted.org/psutil/. >>>> > > >>>> > > While we've talked about deprecating it, it *hasn't* been >>>> deprecated. >>>> > > Looking at https://github.com/pypa/pypi-legacy/commits/production, >>>> I'm >>>> > > not seeing anything obvious that would have caused problems with >>>> docs >>>> > > management, but that's probably still the best issue tracker to use >>>> to >>>> > > report the bug. >>>> > >>>> > See the 'doc_upload' handler at >>>> > https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393 >>>> 891f6c6bcbf84c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804 >>>> > I've collected all information I found at >>>> > https://github.com/pypa/pypi-legacy/issues/672#issuecomment-316125918 >>>> > Please correct me if I missed anything. >>>> > >>>> > --Berker >>>> > _______________________________________________ >>>> > Distutils-SIG maillist - Distutils-SIG at python.org >>>> > https://mail.python.org/mailman/listinfo/distutils-sig >>>> > >>>> > >>>> > >>>> > -- >>>> > Giampaolo - http://grodola.blogspot.com >>>> > >>>> > _______________________________________________ >>>> > Distutils-SIG maillist - Distutils-SIG at python.org >>>> > https://mail.python.org/mailman/listinfo/distutils-sig >>>> >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>> >>> >>> >>> >>> -- >>> Giampaolo - http://grodola.blogspot.com >>> >>> >> >> >> -- >> Giampaolo - http://grodola.blogspot.com >> >> > > > -- > Giampaolo - http://grodola.blogspot.com > > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From opensource at ronnypfannschmidt.de Thu Sep 7 09:26:34 2017 From: opensource at ronnypfannschmidt.de (RonnyPfannschmidt) Date: Thu, 7 Sep 2017 15:26:34 +0200 Subject: [Distutils] supporting editable installs by PEP-517/518 by shim wheels - proposing a common implementation Message-ID: Hi Everyone, i'd like to propose a common tooling for installing packages in editable mode which is based on generating an actual wheel, which includes shim files for the python packages allows to sanely instal/uninstall editable packages a while back i implemented a prototype in gumby-elf which serves as my example. i'd like to start a discussion on this, and hopefully ensure that tools like flit and setuptools adapt a similar mechanism, ideally sharing a implementation once this discussion is finished i'd also like to implement it for setuptools -- Ronny From p.f.moore at gmail.com Thu Sep 7 11:43:47 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 7 Sep 2017 16:43:47 +0100 Subject: [Distutils] supporting editable installs by PEP-517/518 by shim wheels - proposing a common implementation In-Reply-To: References: Message-ID: On 7 September 2017 at 14:26, RonnyPfannschmidt wrote: > Hi Everyone, > > i'd like to propose a common tooling for installing packages in editable > mode > which is based on generating an actual wheel, which includes shim files > for the python packages > allows to sanely instal/uninstall editable packages So pip install -e ., instead of installing a .pth file which included the current directory on sys.path, would install a set of .py files which loaded the actual code from the current directory? That sounds like a plausible approach, and it would certainly stop editable installs being quite as much of a special case as they currently are. One thought - how would you handle C extensions? In principle, though, this sounds like a good idea. Paul From opensource at ronnypfannschmidt.de Thu Sep 7 12:17:43 2017 From: opensource at ronnypfannschmidt.de (RonnyPfannschmidt) Date: Thu, 7 Sep 2017 18:17:43 +0200 Subject: [Distutils] supporting editable installs by PEP-517/518 by shim wheels - proposing a common implementation In-Reply-To: References: Message-ID: <3e457e17-d377-9e91-e3f5-89f6e82e17e8@ronnypfannschmidt.de> Am 07.09.2017 um 17:43 schrieb Paul Moore: > On 7 September 2017 at 14:26, RonnyPfannschmidt > wrote: >> Hi Everyone, >> >> i'd like to propose a common tooling for installing packages in editable >> mode >> which is based on generating an actual wheel, which includes shim files >> for the python packages >> allows to sanely instal/uninstall editable packages > > So pip install -e ., instead of installing a .pth file which included > the current directory on sys.path, would install a set of .py files > which loaded the actual code from the current directory? That sounds > like a plausible approach, and it would certainly stop editable > installs being quite as much of a special case as they currently are. > One thought - how would you handle C extensions? > for c extension im not quite sure, is python finally able to load them from folders belonging to a package? else something far more painful might be necessary but since c extensions need a rebuild anyway, it may be reasonably inexpensive to just put them into the wheel in the other case but id strongly prefer just having them inside of the package discovered via __path__ -- Ronny > In principle, though, this sounds like a good idea. > > Paul > From ncoghlan at gmail.com Thu Sep 7 17:30:41 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Sep 2017 14:30:41 -0700 Subject: [Distutils] supporting editable installs by PEP-517/518 by shim wheels - proposing a common implementation In-Reply-To: References: Message-ID: On 7 September 2017 at 08:43, Paul Moore wrote: > So pip install -e ., instead of installing a .pth file which included > the current directory on sys.path, would install a set of .py files > which loaded the actual code from the current directory? That sounds > like a plausible approach, and it would certainly stop editable > installs being quite as much of a special case as they currently are. It sounds like a recipe for obscure bugs to me, since there's no standard mechanism for one Python file to transparently pretend to be a different one (other than symlinking them). By contrast, manipulating sys.path to include additional directories is entirely normal - it can just have surprising name shadowing effects if the directory has other files and directories in it (this is one reason for the popularity of "src/" directories that *only* include the "for import" files). So for systems with reliable user level symlink support, the most robust approach will be along the lines of: 1. Generate an "_install/" directory 2. Generate proper wheel metadata inside that directory 3. Symlink from that directory to any Python source files, package directories and built extension modules based on the manifest in a fully built wheel file 4. Install a *.pth file into the virtual environment that adds the generated directory to sys.path (and refuse to run if no virtualenv is active) Doing things this way means that hot reloading in things like web frameworks will still work properly, but you also won't inadvertently be able to import random other files that happen to be lying around in the source directory Unfortunately, that approach *doesn't* work reliably on Windows, since the symlinks in step 3 can't be generated with ordinary user permissions. Instead, the installed *.pth needs to reference the given source directory directly, which means you won't have the freedom to rearrange things to match the layout in the wheel manifest. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Sep 7 17:32:40 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 7 Sep 2017 14:32:40 -0700 Subject: [Distutils] supporting editable installs by PEP-517/518 by shim wheels - proposing a common implementation In-Reply-To: <3e457e17-d377-9e91-e3f5-89f6e82e17e8@ronnypfannschmidt.de> References: <3e457e17-d377-9e91-e3f5-89f6e82e17e8@ronnypfannschmidt.de> Message-ID: On 7 September 2017 at 09:17, RonnyPfannschmidt wrote: > for c extension im not quite sure, is python finally able to load them > from folders belonging to a package? As far as I'm aware, it's always been able to do this. Is there a particular bug you're thinking of where it couldn't? (You can't currently have *builtin* package hierarchies, but extension modules as submodules is fine). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Thu Sep 7 17:50:00 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 7 Sep 2017 16:50:00 -0500 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: Here's an http-equiv REFRESH index.html: https://github.com/westurner/resume/blob/master/index.html Is there a Sphinx extension to generate redirect pages? On Thursday, September 7, 2017, Giampaolo Rodola' wrote: > Whops! I meant: http://pythonhosted.org/pyftpdlib/ > > On Thu, Sep 7, 2017 at 5:28 PM, Giampaolo Rodola' > wrote: > >> ...and also http://pyftpdlib.readthedocs.io/ >> if possible; thanks a lot! >> >> On Thu, Sep 7, 2017 at 4:10 PM, Giampaolo Rodola' > > wrote: >> >>> OK I updated the new doc at http://psutil.readthedocs.io/ >>> Is it possible to redirect the old doc here? >>> >>> On Tue, Sep 5, 2017 at 3:31 PM, Giampaolo Rodola' >> > wrote: >>> >>>> A bare redirect would also be fine. It's a shame setup.py classifiers / >>>> metadata does not have a "documentation" field otherwise >>>> pythonhosted.org/{project_name} >>>> could have automatically >>>> redirected to the project doc. >>>> Perhaps https://pypi.python.org/pypi?:action=pkg_edit&name={ >>>> project_name} could provide a form where this can be specified. >>>> What do you think? >>>> >>>> On Tue, Sep 5, 2017 at 12:49 PM, Noah Kantrowitz >>> > wrote: >>>> >>>>> This was discussed several years ago in https://mail.python.org/piperm >>>>> ail/distutils-sig/2015-May/026381.html and a few other threads. The >>>>> final phases went out earlier this year. I don't think there is any plan to >>>>> re-enable uploads to pythonhosted at this time. If you want a one-off >>>>> redirect change or just having the old files removed, we can probably do >>>>> that though, but I very much defer to Donald and others on that :) >>>>> >>>>> --Noah >>>>> >>>>> > On Sep 4, 2017, at 9:43 PM, Giampaolo Rodola' >>>> > wrote: >>>>> > >>>>> > I think it's wise to revert that commit. It seems pythonhosted only >>>>> suggested to migrate to RTD but there never was an official shutdown date >>>>> or warning (either via direct email or message on the web page). >>>>> > >>>>> > On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? < >>>>> berker.peksag at gmail.com >>>>> > wrote: >>>>> > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan >>>> > wrote: >>>>> > > On 2 September 2017 at 15:34, Giampaolo Rodola' < >>>>> g.rodola at gmail.com >>>>> > wrote: >>>>> > >> I know it was deprecated long ago in favor of readthedocs but I >>>>> kept >>>>> > >> postponing it and my doc is still hosted on >>>>> > >> https://pythonhosted.org/psutil/. >>>>> > > >>>>> > > While we've talked about deprecating it, it *hasn't* been >>>>> deprecated. >>>>> > > Looking at https://github.com/pypa/pypi-legacy/commits/production, >>>>> I'm >>>>> > > not seeing anything obvious that would have caused problems with >>>>> docs >>>>> > > management, but that's probably still the best issue tracker to >>>>> use to >>>>> > > report the bug. >>>>> > >>>>> > See the 'doc_upload' handler at >>>>> > https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393 >>>>> 891f6c6bcbf84c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804 >>>>> > I've collected all information I found at >>>>> > https://github.com/pypa/pypi-legacy/issues/672#issuecomment- >>>>> 316125918 >>>>> > Please correct me if I missed anything. >>>>> > >>>>> > --Berker >>>>> > _______________________________________________ >>>>> > Distutils-SIG maillist - Distutils-SIG at python.org >>>>> >>>>> > https://mail.python.org/mailman/listinfo/distutils-sig >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > Giampaolo - http://grodola.blogspot.com >>>>> > >>>>> > _______________________________________________ >>>>> > Distutils-SIG maillist - Distutils-SIG at python.org >>>>> >>>>> > https://mail.python.org/mailman/listinfo/distutils-sig >>>>> >>>>> _______________________________________________ >>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>> >>>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>>> >>>> >>>> >>>> >>>> -- >>>> Giampaolo - http://grodola.blogspot.com >>>> >>>> >>> >>> >>> -- >>> Giampaolo - http://grodola.blogspot.com >>> >>> >> >> >> -- >> Giampaolo - http://grodola.blogspot.com >> >> > > > -- > Giampaolo - http://grodola.blogspot.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ntshkelkar at gmail.com Tue Sep 5 08:13:19 2017 From: ntshkelkar at gmail.com (Natasha Kelkar) Date: Tue, 5 Sep 2017 17:43:19 +0530 Subject: [Distutils] (no subject) Message-ID: Respected sir/madam I am trying to install jpredapi in python 2.1 using command python -m pip install jpreadapi but getting an error Can you please help me in this? -- Natasha Kelkar Institute of Bioinformatics and Biotechnology(iBB), Savitribai Phule Pune University. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Fri Sep 8 08:31:29 2017 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Fri, 8 Sep 2017 15:31:29 +0300 Subject: [Distutils] (no subject) In-Reply-To: References: Message-ID: <2d90de0d-5c00-137a-bb53-b120f0277bf6@nextday.fi> Python 2.1 is very, very old and has not been supported for many years. It is doubtful that any currently available Python package will work with it. For future reference, when you ask for help with an error, always include the error message in your request. It would also be prudent to include a subject in your email. Natasha Kelkar kirjoitti 05.09.2017 klo 15:13: > Respected sir/madam > I am trying to install jpredapi in python 2.1 using command python -m > pip install jpreadapi but getting an error > > Can you please help me in this? > > -- > Natasha Kelkar > > Institute of Bioinformatics and Biotechnology(iBB), > Savitribai Phule Pune University. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Fri Sep 8 08:54:00 2017 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 8 Sep 2017 14:54:00 +0200 Subject: [Distutils] Install psutil without gcc? Message-ID: <01fa6007-8aa2-999d-1b2f-ebc0a152cfe5@thomas-guettler.de> It would be great, if I could install psutil without the need for a gcc on linux. The maintainer (Giampaolo Rodola) is interested, but the roadmap is not clear yet. He said: > It's something which needs to be investigated. A quick Google searched brought me here: > https://github.com/joerick/cibuildwheel > ...so it appears it's definitively possible. Is this the current state of the art, or is there a better way? Regards, Thomas G?ttler Related issue: https://github.com/giampaolo/psutil/issues/824 -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From matthew.brett at gmail.com Fri Sep 8 09:06:13 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 8 Sep 2017 14:06:13 +0100 Subject: [Distutils] Install psutil without gcc? In-Reply-To: <01fa6007-8aa2-999d-1b2f-ebc0a152cfe5@thomas-guettler.de> References: <01fa6007-8aa2-999d-1b2f-ebc0a152cfe5@thomas-guettler.de> Message-ID: Hi, On Fri, Sep 8, 2017 at 1:54 PM, Thomas G?ttler wrote: > It would be great, if I could install psutil without the need for a gcc on > linux. > > The maintainer (Giampaolo Rodola) is interested, but the roadmap is not > clear yet. > > He said: > >> It's something which needs to be investigated. A quick Google searched >> brought me here: >> https://github.com/joerick/cibuildwheel >> ...so it appears it's definitively possible. > > Is this the current state of the art, or is there a better way? I haven't looked at CIBuildwheel in much detail, but my impression is that it will work well for simple wheels, but I didn't investigate far to see how well it would cope for more substantial builds. My own project for the same thing is somewhat less streamlined, but is relatively flexible: https://github.com/matthew-brett/multibuild I did a lot of wheel-build bootstrapping. so there are a lot of projects that use it - numpy, scipy, matplotlib, pandas, Cython, h5py, Pillow, scikit-learn, scikit-image, astropy ... Of course that density is largely historical, but it does at least mean that the project is battle tested and relatively well maintained. I'm happy to help you get set up. You might also look at Scikit-Build - https://github.com/scikit-build/scikit-build - I believe that project is particularly good if you are using CMake. By the way, you might prefer the wheel-builders list for this discussion - https://mail.python.org/mailman/listinfo/wheel-builders Cheers, Matthew From wes.turner at gmail.com Fri Sep 8 09:42:12 2017 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 8 Sep 2017 08:42:12 -0500 Subject: [Distutils] Install psutil without gcc? In-Reply-To: References: <01fa6007-8aa2-999d-1b2f-ebc0a152cfe5@thomas-guettler.de> Message-ID: Is there a reason that psutil cannot be built as a manylinux1 wheel? - manylinux1_x86_64 - manylinux1_i686 On Friday, September 8, 2017, Matthew Brett wrote: > Hi, > > On Fri, Sep 8, 2017 at 1:54 PM, Thomas G?ttler > > wrote: > > It would be great, if I could install psutil without the need for a gcc > on > > linux. > > > > The maintainer (Giampaolo Rodola) is interested, but the roadmap is not > > clear yet. > > > > He said: > > > >> It's something which needs to be investigated. A quick Google searched > >> brought me here: > >> https://github.com/joerick/cibuildwheel > >> ...so it appears it's definitively possible. > > > > Is this the current state of the art, or is there a better way? > > I haven't looked at CIBuildwheel in much detail, but my impression is > that it will work well for simple wheels, but I didn't investigate far > to see how well it would cope for more substantial builds. > > My own project for the same thing is somewhat less streamlined, but is > relatively flexible: > > https://github.com/matthew-brett/multibuild > > I did a lot of wheel-build bootstrapping. so there are a lot of > projects that use it - numpy, scipy, matplotlib, pandas, Cython, h5py, > Pillow, scikit-learn, scikit-image, astropy ... Of course that > density is largely historical, but it does at least mean that the > project is battle tested and relatively well maintained. I'm happy > to help you get set up. > > You might also look at Scikit-Build - > https://github.com/scikit-build/scikit-build - I believe that project > is particularly good if you are using CMake. > > By the way, you might prefer the wheel-builders list for this > discussion - https://mail.python.org/mailman/listinfo/wheel-builders > > Cheers, > > Matthew > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Sep 8 09:47:19 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 8 Sep 2017 14:47:19 +0100 Subject: [Distutils] Install psutil without gcc? In-Reply-To: References: <01fa6007-8aa2-999d-1b2f-ebc0a152cfe5@thomas-guettler.de> Message-ID: Hi, On Fri, Sep 8, 2017 at 2:42 PM, Wes Turner wrote: > Is there a reason that psutil cannot be built as a manylinux1 wheel? > > - manylinux1_x86_64 > - manylinux1_i686 Thomas can correct me if I'm wrong, but I think he was looking for an easy automated way to do this. Cheers, Matthew From lukasz at langa.pl Fri Sep 8 15:13:34 2017 From: lukasz at langa.pl (Lukasz Langa) Date: Fri, 8 Sep 2017 12:13:34 -0700 Subject: [Distutils] Accepting PEP 541? Message-ID: <7660AC24-CAFF-415E-BBDE-C6C28018A1D6@langa.pl> I gathered and responded to feedback on this early this year. I think the PEP is ready to be accepted. https://www.python.org/dev/peps/pep-0541/ - ? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP URL: From wes.turner at gmail.com Fri Sep 8 16:28:46 2017 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 8 Sep 2017 15:28:46 -0500 Subject: [Distutils] Install psutil without gcc? In-Reply-To: References: <01fa6007-8aa2-999d-1b2f-ebc0a152cfe5@thomas-guettler.de> Message-ID: I should've read this issue before responding: https://github.com/giampaolo/psutil/issues/824 On Friday, September 8, 2017, Matthew Brett wrote: > Hi, > > On Fri, Sep 8, 2017 at 2:42 PM, Wes Turner > wrote: > > Is there a reason that psutil cannot be built as a manylinux1 wheel? > > > > - manylinux1_x86_64 > > - manylinux1_i686 > > Thomas can correct me if I'm wrong, but I think he was looking for an > easy automated way to do this. > > Cheers, > > Matthew > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Sun Sep 10 18:21:58 2017 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Mon, 11 Sep 2017 01:21:58 +0300 Subject: [Distutils] New wheel release (v0.30.0) Message-ID: I have now made my first wheel release as the new maintainer. This release contains bug fixes and documentation updates. I decided to make a release now to get a fix out for some long standing issues. A proper overhaul of the documentation is coming later. The project has also been migrated from Bitbucket to Github . -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Sep 11 01:24:19 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 11 Sep 2017 15:24:19 +1000 Subject: [Distutils] New wheel release (v0.30.0) In-Reply-To: References: Message-ID: On 11 September 2017 at 08:21, Alex Gr?nholm wrote: > I have now made my first wheel release as the new maintainer. > > This release contains bug fixes and documentation updates. I decided to make > a release now to get a fix out for some long standing issues. Thank you for taking this on, and thanks to Daniel for being open to granting you the power to make new releases! :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guettliml at thomas-guettler.de Mon Sep 11 05:13:36 2017 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Mon, 11 Sep 2017 11:13:36 +0200 Subject: [Distutils] Install psutil without gcc? In-Reply-To: <01fa6007-8aa2-999d-1b2f-ebc0a152cfe5@thomas-guettler.de> References: <01fa6007-8aa2-999d-1b2f-ebc0a152cfe5@thomas-guettler.de> Message-ID: <11ba78cf-1566-e206-5ea6-c5d4d0b3f7f2@thomas-guettler.de> Am 08.09.2017 um 14:54 schrieb Thomas G?ttler: > It would be great, if I could install psutil without the need for a gcc on linux. > > The maintainer (Giampaolo Rodola) is interested, but the roadmap is not clear yet. > > He said: > > > It's something which needs to be investigated. A quick Google searched brought me here: > > https://github.com/joerick/cibuildwheel > > ...so it appears it's definitively possible. > > Is this the current state of the art, or is there a better way? > > Regards, > Thomas G?ttler > > Related issue: https://github.com/giampaolo/psutil/issues/824 A very big "thank you" for all experts, who jumped from here to the above issue of psutils. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines From thomas at kluyver.me.uk Mon Sep 11 06:36:13 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 11 Sep 2017 11:36:13 +0100 Subject: [Distutils] PEP 517 - is it ready? Message-ID: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> Discussion on PEP 517 seems to have settled down, and I believe that Nick said he was about ready to accept it. Is everyone involved satisfied with the current state? Or is there anything else you think should be considered before accepting it? https://www.python.org/dev/peps/pep-0517/ From p.f.moore at gmail.com Mon Sep 11 06:43:15 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 11 Sep 2017 11:43:15 +0100 Subject: [Distutils] PEP 517 - is it ready? In-Reply-To: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> References: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> Message-ID: Go for it. On 11 September 2017 at 11:36, Thomas Kluyver wrote: > Discussion on PEP 517 seems to have settled down, and I believe that > Nick said he was about ready to accept it. Is everyone involved > satisfied with the current state? Or is there anything else you think > should be considered before accepting it? > > https://www.python.org/dev/peps/pep-0517/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From chris.barker at noaa.gov Mon Sep 11 11:39:21 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 11 Sep 2017 08:39:21 -0700 Subject: [Distutils] Accepting PEP 541? In-Reply-To: <7660AC24-CAFF-415E-BBDE-C6C28018A1D6@langa.pl> References: <7660AC24-CAFF-415E-BBDE-C6C28018A1D6@langa.pl> Message-ID: <-5006877152662658552@unknownmsgid> This looks great: thanks for moving it along! Minor notes (all copy editing): It could use some editing to bring it into the present: """ Existing package repositories such as CPAN [3] , NPM [4] , and GitHub [5] will be investigated as prior art in this field. """ Probably need a time scale on: """" - no activity from the owner on the project's home page. """ Minor edit: "the candidate is able to demonstrate own failed attempts to contact the existing owner" To "the candidate is able to demonstrate their own failed attempts to contact the existing owner" (Adding "their" -- or his/her) Same under "abandoned". -CHB Sent from my iPhone On Sep 8, 2017, at 12:14 PM, Lukasz Langa wrote: I gathered and responded to feedback on this early this year. I think the PEP is ready to be accepted. https://www.python.org/dev/peps/pep-0541/ - ? _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasulumar at hotmail.com Sat Sep 9 12:22:33 2017 From: rasulumar at hotmail.com (Umar Rasul) Date: Sat, 9 Sep 2017 16:22:33 +0000 Subject: [Distutils] (no subject) Message-ID: How can I install pandas on Python 2.7 Sent from Windows Mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From dimaqq at gmail.com Tue Sep 12 23:17:34 2017 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 13 Sep 2017 11:17:34 +0800 Subject: [Distutils] Accepting PEP 541? In-Reply-To: <7660AC24-CAFF-415E-BBDE-C6C28018A1D6@langa.pl> References: <7660AC24-CAFF-415E-BBDE-C6C28018A1D6@langa.pl> Message-ID: Looks good to me! On 9 September 2017 at 03:13, Lukasz Langa wrote: > I gathered and responded to feedback on this early this year. I think the > PEP is ready to be accepted. > > https://www.python.org/dev/peps/pep-0541/ > > - ? > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From g.rodola at gmail.com Wed Sep 13 08:39:21 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 13 Sep 2017 20:39:21 +0800 Subject: [Distutils] pythonhosted.org doc upload no longer works In-Reply-To: References: Message-ID: Thanks. Can somebody please upload this HTML page on https://pythonhosted.org/psutil/? Redirecting to http://psutil.readthedocs.io

This page is outdated

Redirecting to http://psutil.readthedocs.io

On Fri, Sep 8, 2017 at 5:50 AM, Wes Turner wrote: > Here's an http-equiv REFRESH index.html: > > https://github.com/westurner/resume/blob/master/index.html > > Is there a > Sphinx extension to generate redirect pages? > > On Thursday, September 7, 2017, Giampaolo Rodola' > wrote: > >> Whops! I meant: http://pythonhosted.org/pyftpdlib/ >> >> On Thu, Sep 7, 2017 at 5:28 PM, Giampaolo Rodola' >> wrote: >> >>> ...and also http://pyftpdlib.readthedocs.io/ >>> if possible; thanks a lot! >>> >>> On Thu, Sep 7, 2017 at 4:10 PM, Giampaolo Rodola' >>> wrote: >>> >>>> OK I updated the new doc at http://psutil.readthedocs.io/ >>>> Is it possible to redirect the old doc here? >>>> >>>> On Tue, Sep 5, 2017 at 3:31 PM, Giampaolo Rodola' >>>> wrote: >>>> >>>>> A bare redirect would also be fine. It's a shame setup.py classifiers >>>>> / metadata does not have a "documentation" field otherwise >>>>> pythonhosted.org/{project_name} >>>>> could have automatically >>>>> redirected to the project doc. >>>>> Perhaps https://pypi.python.org/pypi?:action=pkg_edit&name={ >>>>> project_name} could provide a form where this can be specified. >>>>> What do you think? >>>>> >>>>> On Tue, Sep 5, 2017 at 12:49 PM, Noah Kantrowitz >>>>> wrote: >>>>> >>>>>> This was discussed several years ago in >>>>>> https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html >>>>>> and a few other threads. The final phases went out earlier this year. I >>>>>> don't think there is any plan to re-enable uploads to pythonhosted at this >>>>>> time. If you want a one-off redirect change or just having the old files >>>>>> removed, we can probably do that though, but I very much defer to Donald >>>>>> and others on that :) >>>>>> >>>>>> --Noah >>>>>> >>>>>> > On Sep 4, 2017, at 9:43 PM, Giampaolo Rodola' >>>>>> wrote: >>>>>> > >>>>>> > I think it's wise to revert that commit. It seems pythonhosted only >>>>>> suggested to migrate to RTD but there never was an official shutdown date >>>>>> or warning (either via direct email or message on the web page). >>>>>> > >>>>>> > On Tue, Sep 5, 2017 at 12:19 PM, Berker Peksa? < >>>>>> berker.peksag at gmail.com> wrote: >>>>>> > On Mon, Sep 4, 2017 at 4:56 PM, Nick Coghlan >>>>>> wrote: >>>>>> > > On 2 September 2017 at 15:34, Giampaolo Rodola' < >>>>>> g.rodola at gmail.com> wrote: >>>>>> > >> I know it was deprecated long ago in favor of readthedocs but I >>>>>> kept >>>>>> > >> postponing it and my doc is still hosted on >>>>>> > >> https://pythonhosted.org/psutil/. >>>>>> > > >>>>>> > > While we've talked about deprecating it, it *hasn't* been >>>>>> deprecated. >>>>>> > > Looking at https://github.com/pypa/pypi-legacy/commits/production, >>>>>> I'm >>>>>> > > not seeing anything obvious that would have caused problems with >>>>>> docs >>>>>> > > management, but that's probably still the best issue tracker to >>>>>> use to >>>>>> > > report the bug. >>>>>> > >>>>>> > See the 'doc_upload' handler at >>>>>> > https://github.com/pypa/pypi-legacy/commit/1598e6ea0f7fb0393 >>>>>> 891f6c6bcbf84c191834a0e#diff-19fadc30e1b17100568adbd8c6c3cc13R2804 >>>>>> > I've collected all information I found at >>>>>> > https://github.com/pypa/pypi-legacy/issues/672#issuecomment- >>>>>> 316125918 >>>>>> > Please correct me if I missed anything. >>>>>> > >>>>>> > --Berker >>>>>> > _______________________________________________ >>>>>> > Distutils-SIG maillist - Distutils-SIG at python.org >>>>>> > https://mail.python.org/mailman/listinfo/distutils-sig >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Giampaolo - http://grodola.blogspot.com >>>>>> > >>>>>> > _______________________________________________ >>>>>> > Distutils-SIG maillist - Distutils-SIG at python.org >>>>>> > https://mail.python.org/mailman/listinfo/distutils-sig >>>>>> >>>>>> _______________________________________________ >>>>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>>>> https://mail.python.org/mailman/listinfo/distutils-sig >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Giampaolo - http://grodola.blogspot.com >>>>> >>>>> >>>> >>>> >>>> -- >>>> Giampaolo - http://grodola.blogspot.com >>>> >>>> >>> >>> >>> -- >>> Giampaolo - http://grodola.blogspot.com >>> >>> >> >> >> -- >> Giampaolo - http://grodola.blogspot.com >> >> -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From baptista.carlos.ab at gmail.com Wed Sep 13 06:59:42 2017 From: baptista.carlos.ab at gmail.com (Carlos Baptista) Date: Wed, 13 Sep 2017 12:59:42 +0200 Subject: [Distutils] Questions about Python 3.6.2 installation Message-ID: Greetings, I'm currently using Linux Mint 18.2 "Sonya" (Cinnamon-64) and Python there're two versions installed by default on my system: *Python 2.7.12* and *Python 3.5.2* (picture bellow). [image: Imagem inline 2] I would like to install the latest Python version (3.6.2). As far as I know, I can't install it over Python 2. However, I would like to know if I can install the 3.6.2 over 3.5.2. If that's possible, how can I do that? Thanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Python versions.png Type: image/png Size: 371128 bytes Desc: not available URL: From leorochael at gmail.com Fri Sep 15 08:48:06 2017 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Fri, 15 Sep 2017 09:48:06 -0300 Subject: [Distutils] Questions about Python 3.6.2 installation In-Reply-To: References: Message-ID: Hi Carlos, Distutils is not the exact forum for these kinds of questions, since we mostly deal with the packaging of Python code for redistribution, not with redistribution / installation of Python itself. For general help with Python we suggest the python-help mailing list: https://mail.python.org/mailman/listinfo/python-help That said, you can install Python 3.6.2 while still maintaining Python 3.5.2 installed. For Linux Mint, which is Ubuntu based (as far as I remember), I suggest you use the Dead Snakes PPA: https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa In any case, I recommend against uninstalling the Pythons that came with your system, as other packages might depend on them. Alternatively, if you don't want to mess with system packages at all, you can use `pyenv`: https://github.com/pyenv/pyenv Best regards, Leo On 13 September 2017 at 07:59, Carlos Baptista wrote: > Greetings, > > > I'm currently using Linux Mint 18.2 "Sonya" (Cinnamon-64) and Python > there're two versions installed by default on my system: *Python 2.7.12* > and *Python 3.5.2* (picture bellow). > > [image: Imagem inline 2] > > I would like to install the latest Python version (3.6.2). As far as I > know, I can't install it over Python 2. However, I would like to know if I > can install the 3.6.2 over 3.5.2. If that's possible, how can I do that? > > Thanks in advance. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Python versions.png Type: image/png Size: 371128 bytes Desc: not available URL: From brett at python.org Thu Sep 21 12:54:32 2017 From: brett at python.org (Brett Cannon) Date: Thu, 21 Sep 2017 16:54:32 +0000 Subject: [Distutils] PEP 517 - is it ready? In-Reply-To: References: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> Message-ID: Are we waiting on anything then? Just Nick officially accepting the PEP? Or did I miss something while catching up on over 2 weeks of emails? I ask not only to help close this long chapter of packaging behind us, but because I'm leading a PyLadies workshop next month on packaging and it would be awesome to say "this is the future" (or "this is the bleeding edge" if Thomas gets flit updated before the workshop) vs. "this is *probably* be the future". :) On Mon, 11 Sep 2017 at 03:43 Paul Moore wrote: > Go for it. > > On 11 September 2017 at 11:36, Thomas Kluyver > wrote: > > Discussion on PEP 517 seems to have settled down, and I believe that > > Nick said he was about ready to accept it. Is everyone involved > > satisfied with the current state? Or is there anything else you think > > should be considered before accepting it? > > > > https://www.python.org/dev/peps/pep-0517/ > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Sep 21 22:03:47 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 22 Sep 2017 12:03:47 +1000 Subject: [Distutils] PEP 517 - is it ready? In-Reply-To: References: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> Message-ID: On 22 September 2017 at 02:54, Brett Cannon wrote: > Are we waiting on anything then? Just Nick officially accepting the PEP? Or > did I miss something while catching up on over 2 weeks of emails? I'm waiting for explicit sign-off (either public or private) from a couple of folks. They're aware I'm waiting to hear back from them, but they're also busy people :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From minesh444 at gmail.com Sat Sep 23 13:45:15 2017 From: minesh444 at gmail.com (Minesh Parikh) Date: Sat, 23 Sep 2017 23:15:15 +0530 Subject: [Distutils] how to install Message-ID: how to install setup.py file on my website -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Sep 25 12:39:20 2017 From: brett at python.org (Brett Cannon) Date: Mon, 25 Sep 2017 16:39:20 +0000 Subject: [Distutils] how to install In-Reply-To: References: Message-ID: The tutorial on how to install packages can be found at https://packaging.python.org/tutorials/installing-packages/. On Mon, 25 Sep 2017 at 05:01 Minesh Parikh wrote: > how to install setup.py file on my website > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Sep 26 09:44:17 2017 From: dholth at gmail.com (Daniel Holth) Date: Tue, 26 Sep 2017 13:44:17 +0000 Subject: [Distutils] New wheel release (v0.30.0) In-Reply-To: References: Message-ID: Thanks. This release also drops support for Python 2.6. It seems hardly anyone noticed. On Mon, Sep 11, 2017, 01:24 Nick Coghlan wrote: > On 11 September 2017 at 08:21, Alex Gr?nholm > wrote: > > I have now made my first wheel release as the new maintainer. > > > > This release contains bug fixes and documentation updates. I decided to > make > > a release now to get a fix out for some long standing issues. > > Thank you for taking this on, and thanks to Daniel for being open to > granting you the power to make new releases! :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Tue Sep 26 22:30:34 2017 From: xoviat at gmail.com (xoviat) Date: Tue, 26 Sep 2017 21:30:34 -0500 Subject: [Distutils] Extracting distutils into setuptools Message-ID: This was a comment by @zooba (Steve Dower): > (FWIW, I think it makes *much* more sense for setuptools to fix this by simply forking all of distutils and never looking back. But since we don't live in that world yet, it went into distutils.) And here is my response: > Since you mention it, I agree with that proposal. But currently we have core developers contributing to distutils and @jaraco contributing to setuptools. @jaraco is quite competent, but I doubt that he would be able to maintain an independent fork of distutils by himself. > In short, I think your proposal is a good one, but how can we allocate manpower? (issue31595 on bugs.python.org) So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 27 04:42:46 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 27 Sep 2017 21:42:46 +1300 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: On Wed, Sep 27, 2017 at 3:30 PM, xoviat wrote: > This was a comment by @zooba (Steve Dower): > > > (FWIW, I think it makes *much* more sense for setuptools to fix this by > simply forking all of distutils and never looking back. But since we don't > live in that world yet, it went into distutils.) > > And here is my response: > > > Since you mention it, I agree with that proposal. But currently we have > core developers contributing to distutils and @jaraco contributing to > setuptools. @jaraco is quite competent, but I doubt that he would be able > to maintain an independent fork of distutils by himself. > > > In short, I think your proposal is a good one, but how can we allocate > manpower? > > (issue31595 on bugs.python.org) > > So what do others think of this? My sense of things is that people are > open to the idea, but there isn't a plan to make it happen. > My $2c: I'd only be a very occasional contributor, but it makes a lot of sense from the point of view of packaging/distributing changes to distutils. Also setuptools is a lot better maintained than distutils, and using the bug tracker is a much better experience. So many reasons to do it, and I'd certainly be more likely to report bugs and/or fix them. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 27 05:37:53 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 27 Sep 2017 19:37:53 +1000 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: On 27 September 2017 at 12:30, xoviat wrote: >> In short, I think your proposal is a good one, but how can we allocate >> manpower? > > (issue31595 on bugs.python.org) > > So what do others think of this? My sense of things is that people are open > to the idea, but there isn't a plan to make it happen. As long as Jason is amenable to the idea on the setuptools side, and pip's trick of injecting setuptools into a process to change the behaviour of distutils imports continues to work, then this seems like a reasonable idea to me. There will still be folks that want to patch distutils itself, but I don't see that as being substantially different from folks still patching the standard library Tcl/Tk bindings, even though there are a number of popular non-stdlib GUI toolkits. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Sep 27 09:35:41 2017 From: donald at stufft.io (Donald Stufft) Date: Wed, 27 Sep 2017 09:35:41 -0400 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: On Sep 27, 2017, at 5:37 AM, Nick Coghlan wrote: > > pip's trick of injecting setuptools into a process to change the > behaviour of distutils imports continues to work I suspect this might be difficult, but I haven?t actually tried it. Our trick (and really setup tools itself relies on this too) is that setuptools monkey patches distutils at runtime to make distutils into setuptools. It only has to patch a few bits and pieces right now, but I suspect that if this happened it?d have to monkey patch the entire namespace. It would probably also break people who mix and match imports from setuptools and distutils. Of course, maybe it won?t! Won?t know until we try it I suppose. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 27 12:27:37 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Sep 2017 02:27:37 +1000 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: On 27 September 2017 at 23:35, Donald Stufft wrote: > I suspect this might be difficult, but I haven?t actually tried it. Our > trick (and really setup tools itself relies on this too) is that setuptools > monkey patches distutils at runtime to make distutils into setuptools. It > only has to patch a few bits and pieces right now, but I suspect that if > this happened it?d have to monkey patch the entire namespace. It would > probably also break people who mix and match imports from setuptools and > distutils. Right, my assumption was that pip would need to ensure that setuptools took over the distutils namespace before any of the code in setup.py ran, so that any monkeypatches and state modifications in other code would all take place in the setuptools provided copy, and the standard library's copy would never even get loaded. However, I don't think that's a new requirement: I believe you already need to do that, since the first line might be "from distutils import setup". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From xoviat at gmail.com Wed Sep 27 12:34:33 2017 From: xoviat at gmail.com (xoviat) Date: Wed, 27 Sep 2017 11:34:33 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: So since PEP 517 is going into setuptools anyway, we can sys.path.insert(0, NEW_DISTUTILS_LOCATION) and then import distutils in the interface before setup.py is loaded. Seems simple enough. 2017-09-27 11:27 GMT-05:00 Nick Coghlan : > On 27 September 2017 at 23:35, Donald Stufft wrote: > > I suspect this might be difficult, but I haven?t actually tried it. Our > > trick (and really setup tools itself relies on this too) is that > setuptools > > monkey patches distutils at runtime to make distutils into setuptools. It > > only has to patch a few bits and pieces right now, but I suspect that if > > this happened it?d have to monkey patch the entire namespace. It would > > probably also break people who mix and match imports from setuptools and > > distutils. > > Right, my assumption was that pip would need to ensure that setuptools > took over the distutils namespace before any of the code in setup.py > ran, so that any monkeypatches and state modifications in other code > would all take place in the setuptools provided copy, and the standard > library's copy would never even get loaded. > > However, I don't think that's a new requirement: I believe you already > need to do that, since the first line might be "from distutils import > setup". > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Wed Sep 27 12:36:37 2017 From: xoviat at gmail.com (xoviat) Date: Wed, 27 Sep 2017 11:36:37 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: /cc @jaraco 2017-09-27 11:34 GMT-05:00 xoviat : > So since PEP 517 is going into setuptools anyway, we can > sys.path.insert(0, NEW_DISTUTILS_LOCATION) and then import distutils in the > interface before setup.py is loaded. Seems simple enough. > > 2017-09-27 11:27 GMT-05:00 Nick Coghlan : > >> On 27 September 2017 at 23:35, Donald Stufft wrote: >> > I suspect this might be difficult, but I haven?t actually tried it. Our >> > trick (and really setup tools itself relies on this too) is that >> setuptools >> > monkey patches distutils at runtime to make distutils into setuptools. >> It >> > only has to patch a few bits and pieces right now, but I suspect that if >> > this happened it?d have to monkey patch the entire namespace. It would >> > probably also break people who mix and match imports from setuptools and >> > distutils. >> >> Right, my assumption was that pip would need to ensure that setuptools >> took over the distutils namespace before any of the code in setup.py >> ran, so that any monkeypatches and state modifications in other code >> would all take place in the setuptools provided copy, and the standard >> library's copy would never even get loaded. >> >> However, I don't think that's a new requirement: I believe you already >> need to do that, since the first line might be "from distutils import >> setup". >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed Sep 27 13:31:02 2017 From: steve.dower at python.org (Steve Dower) Date: Wed, 27 Sep 2017 10:31:02 -0700 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: <12845c21-2057-a490-990f-055ca1ec2185@python.org> On 26Sep2017 1930, xoviat wrote: > This was a comment by @zooba (Steve Dower): > > > (FWIW, I think it makes *much* more sense for setuptools to fix this > by simply forking all of distutils and never looking back. But since we > don't live in that world yet, it went into distutils.) > > And here is my response: > >> Since you mention it, I agree with that proposal. But currently we have core developers contributing to distutils and @jaraco contributing to setuptools. @jaraco is quite competent, but I doubt that he would be able to maintain an independent fork of distutils by himself. > >> In short, I think your proposal is a good one, but how can we allocate manpower? > > (issue31595 on bugs.python.org ) > > So what do others think of this? My sense of things is that people are > open to the idea, but there isn't a plan to make it happen. Just to put it out there, I know this is asking a lot of Jason and the setuptools maintainers. I don't make the proposal lightly. But especially in the world of PEP 517 I would rather not have an array of new libraries depending on distutils for their ability to compile extension modules. As long as there is "core" functionality to do this, we won't ever get any innovation out there. For an example of innovation, I've already published a library that can find VS 2015 and VS 2017 installs on Windows, generate the correct project (make) files and build an extension.[1] It's far more reliable and customisable than distutils - and I've put barely any effort into it! - but as long as people have to choose between a dependency and a core library nobody will ever get to take advantage of it. (And, as occurred, I'll end up porting pieces into core CPython so that we don't break the world by *really* deprecating distutils.) setuptools is totally welcome in my book to simply copy the compiler infrastructure we already have from core and never look back. It really does need to be maintained separately from CPython, especially on Windows where we continue to get innovation in the targeted tools. I know it's a big ask, and it's one that I can't personally commit real time to (though I obviously will as much as possible), but I do think it is necessary for our ecosystem to not be tied to CPython release cycles. Cheers, Steve [1]: The library is at https://pypi.org/project/pyfindvs/ and the docs do not refer to the compilation part at all. Look at https://github.com/zooba/pyfindvs/tree/master/pyfindvs/msbuildcompiler to see it. From nad at python.org Wed Sep 27 14:51:47 2017 From: nad at python.org (Ned Deily) Date: Wed, 27 Sep 2017 14:51:47 -0400 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: <12845c21-2057-a490-990f-055ca1ec2185@python.org> References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> Message-ID: <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> On Sep 27, 2017, at 13:31, Steve Dower wrote: > setuptools is totally welcome in my book to simply copy the compiler infrastructure we already have from core and never look back. It really does need to be maintained separately from CPython, especially on Windows where we continue to get innovation in the targeted tools. I know it's a big ask, and it's one that I can't personally commit real time to (though I obviously will as much as possible), but I do think it is necessary for our ecosystem to not be tied to CPython release cycles. Whatever is done, keep in mind that currently distutils is required to build Python itself, e.g. the standard library. And that at least one important project, numpy, already subclasses distutils. -- Ned Deily nad at python.org -- [] From xoviat at gmail.com Wed Sep 27 15:46:20 2017 From: xoviat at gmail.com (xoviat) Date: Wed, 27 Sep 2017 14:46:20 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> Message-ID: No. Setuptools is what projects without a build_backend in pyproject.toml get. Not distutils. We should make it clear now that the distutils namespace belongs to setuptools except for when building cpython. On Sep 27, 2017 2:33 PM, "Ned Deily" wrote: > On Sep 27, 2017, at 13:31, Steve Dower wrote: > > setuptools is totally welcome in my book to simply copy the compiler > infrastructure we already have from core and never look back. It really > does need to be maintained separately from CPython, especially on Windows > where we continue to get innovation in the targeted tools. I know it's a > big ask, and it's one that I can't personally commit real time to (though I > obviously will as much as possible), but I do think it is necessary for our > ecosystem to not be tied to CPython release cycles. > > Whatever is done, keep in mind that currently distutils is required to > build Python itself, e.g. the standard library. And that at least one > important project, numpy, already subclasses distutils. > > -- > Ned Deily > nad at python.org -- [] > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Wed Sep 27 16:00:56 2017 From: xoviat at gmail.com (xoviat) Date: Wed, 27 Sep 2017 15:00:56 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> Message-ID: That's actually an interesting idea though: for Python 3.7 distutils -> _distutils (and then setuptools is required for building). For/against? 2017-09-27 14:46 GMT-05:00 xoviat : > No. Setuptools is what projects without a build_backend in pyproject.toml > get. Not distutils. We should make it clear now that the distutils > namespace belongs to setuptools except for when building cpython. > > On Sep 27, 2017 2:33 PM, "Ned Deily" wrote: > >> On Sep 27, 2017, at 13:31, Steve Dower wrote: >> > setuptools is totally welcome in my book to simply copy the compiler >> infrastructure we already have from core and never look back. It really >> does need to be maintained separately from CPython, especially on Windows >> where we continue to get innovation in the targeted tools. I know it's a >> big ask, and it's one that I can't personally commit real time to (though I >> obviously will as much as possible), but I do think it is necessary for our >> ecosystem to not be tied to CPython release cycles. >> >> Whatever is done, keep in mind that currently distutils is required to >> build Python itself, e.g. the standard library. And that at least one >> important project, numpy, already subclasses distutils. >> >> -- >> Ned Deily >> nad at python.org -- [] >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Sep 27 16:22:13 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 28 Sep 2017 09:22:13 +1300 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> Message-ID: On Thu, Sep 28, 2017 at 8:46 AM, xoviat wrote: > No. Setuptools is what projects without a build_backend in pyproject.toml > get. Not distutils. We should make it clear now that the distutils > namespace belongs to setuptools except for when building cpython. > > On Sep 27, 2017 2:33 PM, "Ned Deily" wrote: > >> On Sep 27, 2017, at 13:31, Steve Dower wrote: >> > setuptools is totally welcome in my book to simply copy the compiler >> infrastructure we already have from core and never look back. It really >> does need to be maintained separately from CPython, especially on Windows >> where we continue to get innovation in the targeted tools. I know it's a >> big ask, and it's one that I can't personally commit real time to (though I >> obviously will as much as possible), but I do think it is necessary for our >> ecosystem to not be tied to CPython release cycles. >> >> Whatever is done, keep in mind that currently distutils is required to >> build Python itself, e.g. the standard library. And that at least one >> important project, numpy, already subclasses distutils. >> > For numpy that seems fixable (if it even breaks, it may not). As long as the setuptools maintainers are willing to keep numpy.distutils compatibility, I'm happy to make the necessary changes in numpy. FYI, it has happened twice (IIRC) in the last five years that a new setuptools release broke numpy.distutils. This was fixed very quickly with good collaboration between the projects. Ralf > >> -- >> Ned Deily >> nad at python.org -- [] >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 27 23:36:32 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 28 Sep 2017 13:36:32 +1000 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> Message-ID: On 28 September 2017 at 06:00, xoviat wrote: > That's actually an interesting idea though: for Python 3.7 distutils -> > _distutils (and then setuptools is required for building). For/against? distutils works fine for its original purpose (building components for the system Python in Linux distros), so we still need to avoid breaking that. setuptools is only essential if you want full support for modern *Python* level packaging features (PEP 376 install metadata, venv compatibility, wheel files, etc), and a lot of Linux system components simply don't worry about those things, and rely on their system level equivalents instead (e.g. the RPM/deb databases, chroots and containers, RPM/deb files) However, what *could* be interesting is a proposal to move distutils to the "ensurepip" model, where rather than maintaining distutils directly as part of CPython, the CPython build process instead runs setuptools/distutils from a bundled wheel file. Doing that would entail having setuptools actually start installing a copy of distutils into site-packages: older CPython releases would ignore it by default (since the stdlib version would shadow it), while 3.7+ would offer either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" (bikeshed to be painted via the PEP process) to install the setuptools-provided version. I'm not claiming actually doing that would be particularly easy - I just think it's the most viable path to get us away from the current version coupling between the build infrastructure in distutils and the runtime support infrastructure in the rest of the standard library, and to avoid maintaining two distinct copies of distutils indefinitely (one in the stdlib, one in setuptools). That approach wouldn't even entail any *new* bundling at the CPython level, as while it's currently formally an implementation detail (pending potential removal in a post-PEP-517 world), setuptools is already bundled as part of the support infrastructure for ensurepip: https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Thu Sep 28 10:55:17 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 28 Sep 2017 14:55:17 +0000 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> Message-ID: Enscons uses parts of distutils to get compiler flags and so on but does not use Extension() to do the actual compiling. There might be a cleaner way to do it that I was not able to find. There could be a cleaner separation between parts of distutils related to how Python itself was compiled and the part that does the actual compiling. The sysconfig module attempts to do at least part of this. distutils also knows about some compiler tricks like generating symbol tables for Windows, that are not easy to re-use or invoke separately. Someone could catalog these tricks and techniques and refactor the useful ones into a reusable library. I don't have a great sense of how to do it. https://bitbucket.org/dholth/enscons/src/tip/enscons/cpyext.py?at=default&fileviewer=file-view-default On Wed, Sep 27, 2017 at 11:36 PM Nick Coghlan wrote: > On 28 September 2017 at 06:00, xoviat wrote: > > That's actually an interesting idea though: for Python 3.7 distutils -> > > _distutils (and then setuptools is required for building). For/against? > > distutils works fine for its original purpose (building components for > the system Python in Linux distros), so we still need to avoid > breaking that. setuptools is only essential if you want full support > for modern *Python* level packaging features (PEP 376 install > metadata, venv compatibility, wheel files, etc), and a lot of Linux > system components simply don't worry about those things, and rely on > their system level equivalents instead (e.g. the RPM/deb databases, > chroots and containers, RPM/deb files) > > However, what *could* be interesting is a proposal to move distutils > to the "ensurepip" model, where rather than maintaining distutils > directly as part of CPython, the CPython build process instead runs > setuptools/distutils from a bundled wheel file. Doing that would > entail having setuptools actually start installing a copy of distutils > into site-packages: older CPython releases would ignore it by default > (since the stdlib version would shadow it), while 3.7+ would offer > either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" > (bikeshed to be painted via the PEP process) to install the > setuptools-provided version. > > I'm not claiming actually doing that would be particularly easy - I > just think it's the most viable path to get us away from the current > version coupling between the build infrastructure in distutils and the > runtime support infrastructure in the rest of the standard library, > and to avoid maintaining two distinct copies of distutils indefinitely > (one in the stdlib, one in setuptools). > > That approach wouldn't even entail any *new* bundling at the CPython > level, as while it's currently formally an implementation detail > (pending potential removal in a post-PEP-517 world), setuptools is > already bundled as part of the support infrastructure for ensurepip: > https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Sep 28 11:11:14 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Sep 2017 16:11:14 +0100 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> Message-ID: I use distutils to build a C program that embeds Python, for making standalone launchers. I'm not at all clear how I'd get all the information I needed to build a compatible Python (including the "find the appropriate C compiler" dance) manually - I'd probably end up with something that didn't work except with specific Python versions or setups. Having said that, I assume that the relevant APIs or equivalents would be exposed via a future setuptools that included distutils (as it would need them internally to build extensions). Alternatively, I'd support refactoring these aspects of distutils out into a standalone library. Paul On 28 September 2017 at 15:55, Daniel Holth wrote: > Enscons uses parts of distutils to get compiler flags and so on but does not > use Extension() to do the actual compiling. There might be a cleaner way to > do it that I was not able to find. There could be a cleaner separation > between parts of distutils related to how Python itself was compiled and the > part that does the actual compiling. The sysconfig module attempts to do at > least part of this. > > distutils also knows about some compiler tricks like generating symbol > tables for Windows, that are not easy to re-use or invoke separately. > Someone could catalog these tricks and techniques and refactor the useful > ones into a reusable library. I don't have a great sense of how to do it. > > https://bitbucket.org/dholth/enscons/src/tip/enscons/cpyext.py?at=default&fileviewer=file-view-default > > On Wed, Sep 27, 2017 at 11:36 PM Nick Coghlan wrote: >> >> On 28 September 2017 at 06:00, xoviat wrote: >> > That's actually an interesting idea though: for Python 3.7 distutils -> >> > _distutils (and then setuptools is required for building). For/against? >> >> distutils works fine for its original purpose (building components for >> the system Python in Linux distros), so we still need to avoid >> breaking that. setuptools is only essential if you want full support >> for modern *Python* level packaging features (PEP 376 install >> metadata, venv compatibility, wheel files, etc), and a lot of Linux >> system components simply don't worry about those things, and rely on >> their system level equivalents instead (e.g. the RPM/deb databases, >> chroots and containers, RPM/deb files) >> >> However, what *could* be interesting is a proposal to move distutils >> to the "ensurepip" model, where rather than maintaining distutils >> directly as part of CPython, the CPython build process instead runs >> setuptools/distutils from a bundled wheel file. Doing that would >> entail having setuptools actually start installing a copy of distutils >> into site-packages: older CPython releases would ignore it by default >> (since the stdlib version would shadow it), while 3.7+ would offer >> either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" >> (bikeshed to be painted via the PEP process) to install the >> setuptools-provided version. >> >> I'm not claiming actually doing that would be particularly easy - I >> just think it's the most viable path to get us away from the current >> version coupling between the build infrastructure in distutils and the >> runtime support infrastructure in the rest of the standard library, >> and to avoid maintaining two distinct copies of distutils indefinitely >> (one in the stdlib, one in setuptools). >> >> That approach wouldn't even entail any *new* bundling at the CPython >> level, as while it's currently formally an implementation detail >> (pending potential removal in a post-PEP-517 world), setuptools is >> already bundled as part of the support infrastructure for ensurepip: >> https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From p.f.moore at gmail.com Thu Sep 28 11:12:13 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 28 Sep 2017 16:12:13 +0100 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> Message-ID: I meant to include a link, sorry. https://github.com/pfmoore/pylaunch/blob/master/build_zastub.py On 28 September 2017 at 16:11, Paul Moore wrote: > I use distutils to build a C program that embeds Python, for making > standalone launchers. I'm not at all clear how I'd get all the > information I needed to build a compatible Python (including the "find > the appropriate C compiler" dance) manually - I'd probably end up with > something that didn't work except with specific Python versions or > setups. > > Having said that, I assume that the relevant APIs or equivalents would > be exposed via a future setuptools that included distutils (as it > would need them internally to build extensions). Alternatively, I'd > support refactoring these aspects of distutils out into a standalone > library. > > Paul > > On 28 September 2017 at 15:55, Daniel Holth wrote: >> Enscons uses parts of distutils to get compiler flags and so on but does not >> use Extension() to do the actual compiling. There might be a cleaner way to >> do it that I was not able to find. There could be a cleaner separation >> between parts of distutils related to how Python itself was compiled and the >> part that does the actual compiling. The sysconfig module attempts to do at >> least part of this. >> >> distutils also knows about some compiler tricks like generating symbol >> tables for Windows, that are not easy to re-use or invoke separately. >> Someone could catalog these tricks and techniques and refactor the useful >> ones into a reusable library. I don't have a great sense of how to do it. >> >> https://bitbucket.org/dholth/enscons/src/tip/enscons/cpyext.py?at=default&fileviewer=file-view-default >> >> On Wed, Sep 27, 2017 at 11:36 PM Nick Coghlan wrote: >>> >>> On 28 September 2017 at 06:00, xoviat wrote: >>> > That's actually an interesting idea though: for Python 3.7 distutils -> >>> > _distutils (and then setuptools is required for building). For/against? >>> >>> distutils works fine for its original purpose (building components for >>> the system Python in Linux distros), so we still need to avoid >>> breaking that. setuptools is only essential if you want full support >>> for modern *Python* level packaging features (PEP 376 install >>> metadata, venv compatibility, wheel files, etc), and a lot of Linux >>> system components simply don't worry about those things, and rely on >>> their system level equivalents instead (e.g. the RPM/deb databases, >>> chroots and containers, RPM/deb files) >>> >>> However, what *could* be interesting is a proposal to move distutils >>> to the "ensurepip" model, where rather than maintaining distutils >>> directly as part of CPython, the CPython build process instead runs >>> setuptools/distutils from a bundled wheel file. Doing that would >>> entail having setuptools actually start installing a copy of distutils >>> into site-packages: older CPython releases would ignore it by default >>> (since the stdlib version would shadow it), while 3.7+ would offer >>> either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" >>> (bikeshed to be painted via the PEP process) to install the >>> setuptools-provided version. >>> >>> I'm not claiming actually doing that would be particularly easy - I >>> just think it's the most viable path to get us away from the current >>> version coupling between the build infrastructure in distutils and the >>> runtime support infrastructure in the rest of the standard library, >>> and to avoid maintaining two distinct copies of distutils indefinitely >>> (one in the stdlib, one in setuptools). >>> >>> That approach wouldn't even entail any *new* bundling at the CPython >>> level, as while it's currently formally an implementation detail >>> (pending potential removal in a post-PEP-517 world), setuptools is >>> already bundled as part of the support infrastructure for ensurepip: >>> https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled >>> >>> Cheers, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> From chris.barker at noaa.gov Thu Sep 28 12:08:14 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 28 Sep 2017 09:08:14 -0700 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> Message-ID: <-8238039093373882742@unknownmsgid> >>>> distutils works fine for its original purpose (building components for >>>> the system Python in Linux distros), What does Linux have to with it? In the eagle days, I found it most helpful for Windows, actually. And it's very helpful for OS-X as well. It was also great for pure python bundling and installing. But regardless of original intent, it's clear that distutils has not turned out to be as cleanly extendible as is needed. Setuptools has its issues as well -- personally it has always driven me crazy with its mingling of run-time vs build-time vs install time functionality. I think it would be helpful to be explicit about the final goals before talking about whether and how to fork distutils. What I'd love to see is a build tool that is compatible with the "modern" packaging paradigm-- pip, pypi, wheels. ( and other systems like rpm, conda, ....) Essentially setuptools without the cruft. ( no easy install, pkg_resources, ...) This could be maintained as a third party package, like the current setuptools. If/when it becomes ubiquitous, an "ensure" style module could be added to the stdlib. So how do we get there? I would think a fork and merge of both setuptools and distutils would make sense. Preserve the building part of the API, so folks could simply replace the import line an their setup.py files to use the new tool. Distutils and setuptools remain in maintainable only mode for probably a long time :-( -CHB >>>> so we still need to avoid >>>> breaking that. setuptools is only essential if you want full support >>>> for modern *Python* level packaging features (PEP 376 install >>>> metadata, venv compatibility, wheel files, etc), and a lot of Linux >>>> system components simply don't worry about those things, and rely on >>>> their system level equivalents instead (e.g. the RPM/deb databases, >>>> chroots and containers, RPM/deb files) >>>> >>>> However, what *could* be interesting is a proposal to move distutils >>>> to the "ensurepip" model, where rather than maintaining distutils >>>> directly as part of CPython, the CPython build process instead runs >>>> setuptools/distutils from a bundled wheel file. Doing that would >>>> entail having setuptools actually start installing a copy of distutils >>>> into site-packages: older CPython releases would ignore it by default >>>> (since the stdlib version would shadow it), while 3.7+ would offer >>>> either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" >>>> (bikeshed to be painted via the PEP process) to install the >>>> setuptools-provided version. >>>> >>>> I'm not claiming actually doing that would be particularly easy - I >>>> just think it's the most viable path to get us away from the current >>>> version coupling between the build infrastructure in distutils and the >>>> runtime support infrastructure in the rest of the standard library, >>>> and to avoid maintaining two distinct copies of distutils indefinitely >>>> (one in the stdlib, one in setuptools). >>>> >>>> That approach wouldn't even entail any *new* bundling at the CPython >>>> level, as while it's currently formally an implementation detail >>>> (pending potential removal in a post-PEP-517 world), setuptools is >>>> already bundled as part of the support infrastructure for ensurepip: >>>> https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled >>>> >>>> Cheers, >>>> Nick. >>>> >>>> -- >>>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>>> _______________________________________________ >>>> Distutils-SIG maillist - Distutils-SIG at python.org >>>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >>> >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From xoviat at gmail.com Thu Sep 28 15:14:05 2017 From: xoviat at gmail.com (xoviat) Date: Thu, 28 Sep 2017 14:14:05 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: <-8238039093373882742@unknownmsgid> References: <12845c21-2057-a490-990f-055ca1ec2185@python.org> <71B04DA7-F9FF-4E25-A9DD-9479622A90D3@python.org> <-8238039093373882742@unknownmsgid> Message-ID: > So how do we get there? I would think a fork and merge of both setuptools and distutils would make sense. Preserve the building part of the API, so folks could simply replace the import line an their setup.py files to use the new tool. There's no need to even replace the line in setup.py. Setuptools currently has the full capability to override the distutils namespace when it's imported first. I think if we extract distutils into setuptools, it's not difficult to list setuptools in install_requires or otherwise on Python 3.7 and above. And I think it will eventually be better for setuptools because the monkeypatch can be avoided. This might even solve the bootstrapping problem. 2017-09-28 11:08 GMT-05:00 Chris Barker - NOAA Federal < chris.barker at noaa.gov>: > >>>> distutils works fine for its original purpose (building components for > >>>> the system Python in Linux distros), > > What does Linux have to with it? In the eagle days, I found it most > helpful for Windows, actually. And it's very helpful for OS-X as well. > > It was also great for pure python bundling and installing. > > But regardless of original intent, it's clear that distutils has not > turned out to be as cleanly extendible as is needed. > > Setuptools has its issues as well -- personally it has always driven > me crazy with its mingling of run-time vs build-time vs install time > functionality. > > I think it would be helpful to be explicit about the final goals > before talking about whether and how to fork distutils. > > What I'd love to see is a build tool that is compatible with the > "modern" packaging paradigm-- pip, pypi, wheels. ( and other systems > like rpm, conda, ....) > > Essentially setuptools without the cruft. ( no easy install, > pkg_resources, ...) > > This could be maintained as a third party package, like the current > setuptools. If/when it becomes ubiquitous, an "ensure" style module > could be added to the stdlib. > > So how do we get there? I would think a fork and merge of both > setuptools and distutils would make sense. Preserve the building part > of the API, so folks could simply replace the import line an their > setup.py files to use the new tool. > > Distutils and setuptools remain in maintainable only mode for probably > a long time :-( > > -CHB > > > > >>>> so we still need to avoid > >>>> breaking that. setuptools is only essential if you want full support > >>>> for modern *Python* level packaging features (PEP 376 install > >>>> metadata, venv compatibility, wheel files, etc), and a lot of Linux > >>>> system components simply don't worry about those things, and rely on > >>>> their system level equivalents instead (e.g. the RPM/deb databases, > >>>> chroots and containers, RPM/deb files) > >>>> > >>>> However, what *could* be interesting is a proposal to move distutils > >>>> to the "ensurepip" model, where rather than maintaining distutils > >>>> directly as part of CPython, the CPython build process instead runs > >>>> setuptools/distutils from a bundled wheel file. Doing that would > >>>> entail having setuptools actually start installing a copy of distutils > >>>> into site-packages: older CPython releases would ignore it by default > >>>> (since the stdlib version would shadow it), while 3.7+ would offer > >>>> either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" > >>>> (bikeshed to be painted via the PEP process) to install the > >>>> setuptools-provided version. > >>>> > >>>> I'm not claiming actually doing that would be particularly easy - I > >>>> just think it's the most viable path to get us away from the current > >>>> version coupling between the build infrastructure in distutils and the > >>>> runtime support infrastructure in the rest of the standard library, > >>>> and to avoid maintaining two distinct copies of distutils indefinitely > >>>> (one in the stdlib, one in setuptools). > >>>> > >>>> That approach wouldn't even entail any *new* bundling at the CPython > >>>> level, as while it's currently formally an implementation detail > >>>> (pending potential removal in a post-PEP-517 world), setuptools is > >>>> already bundled as part of the support infrastructure for ensurepip: > >>>> https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled > >>>> > >>>> Cheers, > >>>> Nick. > >>>> > >>>> -- > >>>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > >>>> _______________________________________________ > >>>> Distutils-SIG maillist - Distutils-SIG at python.org > >>>> https://mail.python.org/mailman/listinfo/distutils-sig > >>> > >>> > >>> _______________________________________________ > >>> Distutils-SIG maillist - Distutils-SIG at python.org > >>> https://mail.python.org/mailman/listinfo/distutils-sig > >>> > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From doko at ubuntu.com Thu Sep 28 17:42:51 2017 From: doko at ubuntu.com (Matthias Klose) Date: Thu, 28 Sep 2017 23:42:51 +0200 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: On 27.09.2017 11:37, Nick Coghlan wrote: > On 27 September 2017 at 12:30, xoviat wrote: >>> In short, I think your proposal is a good one, but how can we allocate >>> manpower? >> >> (issue31595 on bugs.python.org) >> >> So what do others think of this? My sense of things is that people are open >> to the idea, but there isn't a plan to make it happen. > > As long as Jason is amenable to the idea on the setuptools side, and > pip's trick of injecting setuptools into a process to change the > behaviour of distutils imports continues to work, then this seems like > a reasonable idea to me. > > There will still be folks that want to patch distutils itself, but I > don't see that as being substantially different from folks still > patching the standard library Tcl/Tk bindings, even though there are a > number of popular non-stdlib GUI toolkits. In this case, we should deprecate distutils in python3.7 and remove it in 3.8. From ncoghlan at gmail.com Fri Sep 29 02:31:20 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 29 Sep 2017 16:31:20 +1000 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: On 29 September 2017 at 07:42, Matthias Klose wrote: > On 27.09.2017 11:37, Nick Coghlan wrote: >> On 27 September 2017 at 12:30, xoviat wrote: >>>> In short, I think your proposal is a good one, but how can we allocate >>>> manpower? >>> >>> (issue31595 on bugs.python.org) >>> >>> So what do others think of this? My sense of things is that people are open >>> to the idea, but there isn't a plan to make it happen. >> >> As long as Jason is amenable to the idea on the setuptools side, and >> pip's trick of injecting setuptools into a process to change the >> behaviour of distutils imports continues to work, then this seems like >> a reasonable idea to me. >> >> There will still be folks that want to patch distutils itself, but I >> don't see that as being substantially different from folks still >> patching the standard library Tcl/Tk bindings, even though there are a >> number of popular non-stdlib GUI toolkits. > > In this case, we should deprecate distutils in python3.7 and remove it in 3.8. Aye, that seems reasonable to me as well (especially if 3.7+ also offers an "ensuresetuptools" module), and, looking at the recent(-ish) commit history at https://github.com/python/cpython/commits/master/Lib/distutils, I'd suggest we ask Steve Dower if he'd be willing to be BDFL-Delegate for the related PEP. This is one of those changes where the longer term process improvement benefits are already reasonably clear to the folks involved in maintaining the software (i.e. we think it will provide a better end user experience overall if distutils switches to being updated and versioned based on the setuptools release cycle rather than the reference interpreter & standard library one), so the main thing the PEP process will need to ensure is that we're providing a sufficiently non-disruptive transition plan for getting from the current state to the more desirable state. Between Steve ("Don't break Windows"), you ("Don't break Debian (et al)"), me ("Don't break Fedora (et al)"), Donald ("Don't break pip"), and Jason ("Don't break setuptools"/"Don't lumber setuptools with an unreasonable ongoing maintenance burden"), along with everyone else on distutils-sig, python-ideas, and python-dev, I think we'll be able to make a reasonable assessment of what counts as "too disruptive". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Sep 29 02:38:18 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 29 Sep 2017 16:38:18 +1000 Subject: [Distutils] PEP 517 - is it ready? In-Reply-To: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> References: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> Message-ID: On 11 September 2017 at 20:36, Thomas Kluyver wrote: > Discussion on PEP 517 seems to have settled down, and I believe that > Nick said he was about ready to accept it. Is everyone involved > satisfied with the current state? Or is there anything else you think > should be considered before accepting it? > > https://www.python.org/dev/peps/pep-0517/ I've heard back off-list from the folks I was waiting to hear from, and I'm very happy to say that I'm provisionally accepting PEP 517, as defined in https://www.pypa.io/en/latest/specifications/#provisional-acceptance That means folks are now free to go start updating the reference tooling based on the PEP, but for the time being, we reserve the right to make further changes to the specification based on our real world experience with those initial implementations. Thank you Thomas and Nathanial for your persistence in working through all the feedback received on the proposal, and thus enabling a significant step forward for the overall Python packaging ecosystem :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Sep 29 03:40:08 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 29 Sep 2017 08:40:08 +0100 Subject: [Distutils] PEP 517 - is it ready? In-Reply-To: References: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> Message-ID: On 29 September 2017 at 07:38, Nick Coghlan wrote: > Thank you Thomas and Nathanial for your persistence in working through > all the feedback received on the proposal, and thus enabling a > significant step forward for the overall Python packaging ecosystem :) Yay! This has been a long and difficult discussion for everyone. Thanks to all involved for their patience and willingness to compromise :-) This is a major achievement. Paul From ben+python at benfinney.id.au Fri Sep 29 04:07:24 2017 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 29 Sep 2017 18:07:24 +1000 Subject: [Distutils] PEP 517 - is it ready? References: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> Message-ID: <85mv5e3p37.fsf@benfinney.id.au> Nick Coghlan writes: > I've heard back off-list from the folks I was waiting to hear from, > and I'm very happy to say that I'm provisionally accepting PEP 517, as > defined in > https://www.pypa.io/en/latest/specifications/#provisional-acceptance Good news. You might want to update the PEP document again, though; has the title ?Rrovisional acceptance? :-) -- \ ?I know you believe you understood what you think I said, but I | `\ am not sure you realize that what you heard is not what I | _o__) meant.? ?Robert J. McCloskey | Ben Finney From brett at python.org Fri Sep 29 12:25:43 2017 From: brett at python.org (Brett Cannon) Date: Fri, 29 Sep 2017 16:25:43 +0000 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: Actually, you can't remove any module in the stdlib that exists in both Python 2.7 and 3.5 until after Python 2.7 is no longer supported: https://www.python.org/dev/peps/pep-0004/#for-modules-existing-in-both-python-2-7-and-python-3-5 . On Thu, 28 Sep 2017 at 23:31 Nick Coghlan wrote: > On 29 September 2017 at 07:42, Matthias Klose wrote: > > On 27.09.2017 11:37, Nick Coghlan wrote: > >> On 27 September 2017 at 12:30, xoviat wrote: > >>>> In short, I think your proposal is a good one, but how can we allocate > >>>> manpower? > >>> > >>> (issue31595 on bugs.python.org) > >>> > >>> So what do others think of this? My sense of things is that people are > open > >>> to the idea, but there isn't a plan to make it happen. > >> > >> As long as Jason is amenable to the idea on the setuptools side, and > >> pip's trick of injecting setuptools into a process to change the > >> behaviour of distutils imports continues to work, then this seems like > >> a reasonable idea to me. > >> > >> There will still be folks that want to patch distutils itself, but I > >> don't see that as being substantially different from folks still > >> patching the standard library Tcl/Tk bindings, even though there are a > >> number of popular non-stdlib GUI toolkits. > > > > In this case, we should deprecate distutils in python3.7 and remove it > in 3.8. > > Aye, that seems reasonable to me as well (especially if 3.7+ also > offers an "ensuresetuptools" module), and, looking at the recent(-ish) > commit history at > https://github.com/python/cpython/commits/master/Lib/distutils, I'd > suggest we ask Steve Dower if he'd be willing to be BDFL-Delegate for > the related PEP. > > This is one of those changes where the longer term process improvement > benefits are already reasonably clear to the folks involved in > maintaining the software (i.e. we think it will provide a better end > user experience overall if distutils switches to being updated and > versioned based on the setuptools release cycle rather than the > reference interpreter & standard library one), so the main thing the > PEP process will need to ensure is that we're providing a sufficiently > non-disruptive transition plan for getting from the current state to > the more desirable state. > > Between Steve ("Don't break Windows"), you ("Don't break Debian (et > al)"), me ("Don't break Fedora (et al)"), Donald ("Don't break pip"), > and Jason ("Don't break setuptools"/"Don't lumber setuptools with an > unreasonable ongoing maintenance burden"), along with everyone else on > distutils-sig, python-ideas, and python-dev, I think we'll be able to > make a reasonable assessment of what counts as "too disruptive". > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Sep 29 12:30:07 2017 From: brett at python.org (Brett Cannon) Date: Fri, 29 Sep 2017 16:30:07 +0000 Subject: [Distutils] PEP 517 - is it ready? In-Reply-To: <85mv5e3p37.fsf@benfinney.id.au> References: <1505126173.902033.1101961672.4FD9027B@webmail.messagingengine.com> <85mv5e3p37.fsf@benfinney.id.au> Message-ID: Spelling mistake fixed! And a huge thank you to everyone for trudging forward with this and reaching this point. I think this will be a real boon for the community. On Fri, 29 Sep 2017 at 01:08 Ben Finney wrote: > Nick Coghlan writes: > > > I've heard back off-list from the folks I was waiting to hear from, > > and I'm very happy to say that I'm provisionally accepting PEP 517, as > > defined in > > https://www.pypa.io/en/latest/specifications/#provisional-acceptance > > Good news. You might want to update the PEP document again, though; > > has the title ?Rrovisional acceptance? :-) > > -- > \ ?I know you believe you understood what you think I said, but I | > `\ am not sure you realize that what you heard is not what I | > _o__) meant.? ?Robert J. McCloskey | > Ben Finney > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Fri Sep 29 14:16:13 2017 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Fri, 29 Sep 2017 11:16:13 -0700 Subject: [Distutils] Reproducible builds (Sdist) Message-ID: Hello there, I'm going to ask questions about Reproducible Builds, a previous thread have been started in March[1], but does not cover some of the questions I have. In particular I'm interested in the reproducible build of an _sdist_. That is to say the process of going from a given commit to the corresponding TGZ file. It is my understanding that setting SOURCE_DATE_EPOCH (SDE for short) should allow a reproducible building of an Sdist; And by reproducible I mean that the tgz itself is the same byte for byte; (the unpacked-content being the same is a weaker form I'm less interested in). Is this assumption correct? In particular I cannot seem to be able to do that without unpacking and repacking the tgz myself; because the copy_tree-taring and the gziping by default embed the current timestamp of when these functions were ran. Am I missing something ? Second; is there a convention to store the SDE value ? I don't seem to be able to find one. It is nice to have reproducible build; but if it's a pain for reproducers to find the SDE value that highly decrease the value of SDE build. Also congrats for pep 517 and thanks for everyone who participated; Thanks -- Matthias 1: https://mail.python.org/pipermail/distutils-sig/2017-March/030284.html From xoviat at gmail.com Fri Sep 29 14:24:40 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 29 Sep 2017 13:24:40 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: I understand the motivation behind that PEP, but we're not really deprecating distutils, just moving it out into setuptools. The goal should be "not to break anyone's code." 2017-09-29 11:25 GMT-05:00 Brett Cannon : > Actually, you can't remove any module in the stdlib that exists in both > Python 2.7 and 3.5 until after Python 2.7 is no longer supported: > https://www.python.org/dev/peps/pep-0004/#for-modules- > existing-in-both-python-2-7-and-python-3-5 . > > On Thu, 28 Sep 2017 at 23:31 Nick Coghlan wrote: > >> On 29 September 2017 at 07:42, Matthias Klose wrote: >> > On 27.09.2017 11:37, Nick Coghlan wrote: >> >> On 27 September 2017 at 12:30, xoviat wrote: >> >>>> In short, I think your proposal is a good one, but how can we >> allocate >> >>>> manpower? >> >>> >> >>> (issue31595 on bugs.python.org) >> >>> >> >>> So what do others think of this? My sense of things is that people >> are open >> >>> to the idea, but there isn't a plan to make it happen. >> >> >> >> As long as Jason is amenable to the idea on the setuptools side, and >> >> pip's trick of injecting setuptools into a process to change the >> >> behaviour of distutils imports continues to work, then this seems like >> >> a reasonable idea to me. >> >> >> >> There will still be folks that want to patch distutils itself, but I >> >> don't see that as being substantially different from folks still >> >> patching the standard library Tcl/Tk bindings, even though there are a >> >> number of popular non-stdlib GUI toolkits. >> > >> > In this case, we should deprecate distutils in python3.7 and remove it >> in 3.8. >> >> Aye, that seems reasonable to me as well (especially if 3.7+ also >> offers an "ensuresetuptools" module), and, looking at the recent(-ish) >> commit history at >> https://github.com/python/cpython/commits/master/Lib/distutils, I'd >> suggest we ask Steve Dower if he'd be willing to be BDFL-Delegate for >> the related PEP. >> >> This is one of those changes where the longer term process improvement >> benefits are already reasonably clear to the folks involved in >> maintaining the software (i.e. we think it will provide a better end >> user experience overall if distutils switches to being updated and >> versioned based on the setuptools release cycle rather than the >> reference interpreter & standard library one), so the main thing the >> PEP process will need to ensure is that we're providing a sufficiently >> non-disruptive transition plan for getting from the current state to >> the more desirable state. >> >> Between Steve ("Don't break Windows"), you ("Don't break Debian (et >> al)"), me ("Don't break Fedora (et al)"), Donald ("Don't break pip"), >> and Jason ("Don't break setuptools"/"Don't lumber setuptools with an >> unreasonable ongoing maintenance burden"), along with everyone else on >> distutils-sig, python-ideas, and python-dev, I think we'll be able to >> make a reasonable assessment of what counts as "too disruptive". >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Sep 29 16:02:16 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 29 Sep 2017 21:02:16 +0100 Subject: [Distutils] Reproducible builds (Sdist) In-Reply-To: References: Message-ID: <1506715336.3233190.1122879608.5840B3BD@webmail.messagingengine.com> On Fri, Sep 29, 2017, at 07:16 PM, Matthias Bussonnier wrote: > Second; is there a convention to store the SDE value ? I don't seem to > be able to find one. It is nice to have reproducible build; but if > it's a pain for reproducers to find the SDE value that highly decrease > the value of SDE build. Does it make sense to add a new optional metadata field to store the value of SOURCE_DATE_EPOCH if it's set when a distribution is built? I guess it could cause problems if unpacking & repacking a tarball means that its metadata is no longer accurate, though. Thomas From donald at stufft.io Fri Sep 29 16:06:19 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 29 Sep 2017 16:06:19 -0400 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: > On Sep 29, 2017, at 2:31 AM, Nick Coghlan wrote: > > This is one of those changes where the longer term process improvement > benefits are already reasonably clear to the folks involved in > maintaining the software (i.e. we think it will provide a better end > user experience overall if distutils switches to being updated and > versioned based on the setuptools release cycle rather than the > reference interpreter & standard library one), so the main thing the > PEP process will need to ensure is that we're providing a sufficiently > non-disruptive transition plan for getting from the current state to > the more desirable state. I?m not sure that it?s going to be helpful to take distutils out of the standard library. It has many problems, and being in the standard library is one of them sure, but the more insidious problems are inherent in it?s entire design which we can?t really fix without breaking all of the things. The main blocker to improvement is primarily just that after two decades of existence tons of random setup.py scripts out there have reached in and messed with sometimes even super internal parts of distutils. There are bits of distutils that are super useful (Steve indicated the compiler support as one of them), and I think a better path forward is to do like we?ve done with the packaging library. Make a compiler package that can handle those bits and recommend that projects that want to handle compilation just reuse them (of course not mandate it!). Then we can focus first on creating a good API in that package without inheriting the problems of distutils, and then we can maybe work on bundling that inside of distutils and turn the distutils API into a shim over that (or just leaving it alone). This leaves the question of what happens to setuptools since it is intimately tied to distutils. I don?t really have a good answer for that. Probably it?s best to keep setuptools moving along as it?s doing now without making major architectural changes to it. It inherited a lot of the problems of distutils where people reach in and muck around with it?s internals so it?s easy to break things if you make too many changes. Therefore we might be better to just mostly leave distutils/setuptools progressing as they are today and focus on a brighter future in the greenfield landscape of PEP 517. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Fri Sep 29 17:16:35 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 29 Sep 2017 14:16:35 -0700 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: <7714677f-0443-2cef-b465-4e600be11133@python.org> On 29Sep2017 1306, Donald Stufft wrote: > There are bits of distutils that are super useful (Steve indicated the > compiler support as one of them), and I think a better path forward is > to do like we?ve done with the packaging library. Make a compiler > package that can handle those bits and recommend that projects that want > to handle compilation just reuse them (of course not mandate it!). Then > we can focus first on creating a good API in that package without > inheriting the problems of distutils, and then we can maybe work on > bundling that inside of distutils and turn the distutils API into a shim > over that (or just leaving it alone). I'm happy enough with this approach, my only problem with it is that I don't want to be maintaining two versions (the new one that we want people to change to, and the old one so that people keep working with new versions of Python without having to change to the new one). If we don't make a very clear statement that "distutils is no longer being maintained, it probably won't build valid extensions on 3.[7/8/whatever], and we mean it this time" then it doesn't count. I insist on at least removing the distutils tests from the test suite, since as long as those need to pass I can never stop maintaining the stdlib version. And that puts this in the realm of python-dev, which means a PEP to formally deprecate distutils (that I'm happy to co-author) and recommend an alternative, as well as the alternative path needed to build the core extension modules on non-Windows platforms. Whatever breaks setuptools's dependency on distutils helps drastically with this, as then we can recommend setuptools as the alternative path. Perhaps we can extract distutils into an installable package that will shadow the stdlib one, and setuptools can continue monkey patching while also taking a build/install-time dependency on it. I honestly don't know what the best approach here is, which is why I raise the question. I'm fairly convinced that the best approach is *not* to have a whole lot of PEP 517 build tools that depend on distutils for core functionality. Cheers, Steve From xoviat at gmail.com Fri Sep 29 17:32:08 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 29 Sep 2017 16:32:08 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: Message-ID: Distutils has many problems as you said, but by direct implication, if setuptools is "intimately tied" to distutils, then setuptools has these problems as well (and I think that it does). > I don?t really have a good answer for that. Probably it?s best to keep setuptools moving along as it?s doing now without making major architectural changes to it. The answer (I think) is clear. Setuptools should take over the distutils namespace, even on older Python versions, and even when there is already a distutils in the standard library. If setuptools is imported first, then it can rewrite the import paths such that the setuptools version of distutils is used. This would extract and consolidate what has become a de-facto single project (distutils + setuptools) into one repository, reducing duplication of effort. 2017-09-29 15:06 GMT-05:00 Donald Stufft : > > > On Sep 29, 2017, at 2:31 AM, Nick Coghlan wrote: > > This is one of those changes where the longer term process improvement > benefits are already reasonably clear to the folks involved in > maintaining the software (i.e. we think it will provide a better end > user experience overall if distutils switches to being updated and > versioned based on the setuptools release cycle rather than the > reference interpreter & standard library one), so the main thing the > PEP process will need to ensure is that we're providing a sufficiently > non-disruptive transition plan for getting from the current state to > the more desirable state. > > > I?m not sure that it?s going to be helpful to take distutils out of the > standard library. It has many problems, and being in the standard library > is one of them sure, but the more insidious problems are inherent in it?s > entire design which we can?t really fix without breaking all of the things. > The main blocker to improvement is primarily just that after two decades of > existence tons of random setup.py scripts out there have reached in and > messed with sometimes even super internal parts of distutils. > > There are bits of distutils that are super useful (Steve indicated the > compiler support as one of them), and I think a better path forward is to > do like we?ve done with the packaging library. Make a compiler package that > can handle those bits and recommend that projects that want to handle > compilation just reuse them (of course not mandate it!). Then we can focus > first on creating a good API in that package without inheriting the > problems of distutils, and then we can maybe work on bundling that inside > of distutils and turn the distutils API into a shim over that (or just > leaving it alone). > > This leaves the question of what happens to setuptools since it is > intimately tied to distutils. I don?t really have a good answer for that. > Probably it?s best to keep setuptools moving along as it?s doing now > without making major architectural changes to it. It inherited a lot of the > problems of distutils where people reach in and muck around with it?s > internals so it?s easy to break things if you make too many changes. > > Therefore we might be better to just mostly leave distutils/setuptools > progressing as they are today and focus on a brighter future in the > greenfield landscape of PEP 517. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Fri Sep 29 18:05:24 2017 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Fri, 29 Sep 2017 15:05:24 -0700 Subject: [Distutils] Reproducible builds (Sdist) In-Reply-To: <1506715336.3233190.1122879608.5840B3BD@webmail.messagingengine.com> References: <1506715336.3233190.1122879608.5840B3BD@webmail.messagingengine.com> Message-ID: > Does it make sense to add a new optional metadata field to store the > value of SOURCE_DATE_EPOCH if it's set when a distribution is built? I > guess it could cause problems if unpacking & repacking a tarball means > that its metadata is no longer accurate, though. That make sens ? and that would be useful, but then that mean you need to have the sdist to reproduce the sdist... I was more thinking of a location in the source-tree/commit; for example in pyproject.toml's tool section. So if I give you only that you can tell me "When I build the sdist I get this sha256", and I can do the same independently. -- M On Fri, Sep 29, 2017 at 1:02 PM, Thomas Kluyver wrote: > On Fri, Sep 29, 2017, at 07:16 PM, Matthias Bussonnier wrote: >> Second; is there a convention to store the SDE value ? I don't seem to >> be able to find one. It is nice to have reproducible build; but if >> it's a pain for reproducers to find the SDE value that highly decrease >> the value of SDE build. > > Does it make sense to add a new optional metadata field to store the > value of SOURCE_DATE_EPOCH if it's set when a distribution is built? I > guess it could cause problems if unpacking & repacking a tarball means > that its metadata is no longer accurate, though. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From jwilk at jwilk.net Fri Sep 29 15:04:48 2017 From: jwilk at jwilk.net (Jakub Wilk) Date: Fri, 29 Sep 2017 21:04:48 +0200 Subject: [Distutils] Reproducible builds (Sdist) In-Reply-To: References: Message-ID: <20170929190448.ndprtp6p2d3epk2v@jwilk.net> * Matthias Bussonnier , 2017-09-29, 11:16: >I'm interested in the reproducible build of an _sdist_. >That is to say the process of going from a given commit to the >corresponding TGZ file. It is my understanding that setting >SOURCE_DATE_EPOCH (SDE for short) should allow a reproducible building >of an Sdist; It not enough to normalize timestamps. You need to normalize permissions and ownership, too. (I'm using https://pypi.python.org/pypi/distutils644 for normalizing permissions/ownership in my own packages.) >I cannot seem to be able to do that without unpacking and repacking the >tgz myself; Yeah, I don't believe distutils honors SOURCE_DATE_EPOCH at the moment. >Second; is there a convention to store the SDE value ? In the changelog. -- Jakub Wilk From donald at stufft.io Fri Sep 29 23:15:16 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 29 Sep 2017 23:15:16 -0400 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: <7714677f-0443-2cef-b465-4e600be11133@python.org> References: <7714677f-0443-2cef-b465-4e600be11133@python.org> Message-ID: <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> > On Sep 29, 2017, at 5:16 PM, Steve Dower wrote: > > I'm happy enough with this approach, my only problem with it is that I don't want to be maintaining two versions (the new one that we want people to change to, and the old one so that people keep working with new versions of Python without having to change to the new one). > > If we don't make a very clear statement that "distutils is no longer being maintained, it probably won't build valid extensions on 3.[7/8/whatever], and we mean it this time" then it doesn't count. I insist on at least removing the distutils tests from the test suite, since as long as those need to pass I can never stop maintaining the stdlib version. And that puts this in the realm of python-dev, which means a PEP to formally deprecate distutils (that I'm happy to co-author) and recommend an alternative, as well as the alternative path needed to build the core extension modules on non-Windows platforms. As long as CPython itself depends on distutils I don?t think we can break or remove distutils, but we can move make it a shim over something that is better architecturally and easier to maintain. The tests should still pass in this case because they?d essentially be testing that our shim is functioning correctly. I also don?t think we can spin distutils out of the stdlib as long as CPython depends on it. I see two paths forward: 1) We come up with something better *in the standard library*, implement it, move CPython to using that in the build process, and then deprecate/remove distutils from the stdlib and have it maintained outside. 2) We come up with something better outside of the standard library, implement it, vendor it into distutils/_whatevercoollib, and turn the relevant distutils APIs into shims over the new API. Of the two, I think that (2) is going to be the least painful for users. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Fri Sep 29 23:21:04 2017 From: xoviat at gmail.com (xoviat) Date: Fri, 29 Sep 2017 22:21:04 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> References: <7714677f-0443-2cef-b465-4e600be11133@python.org> <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> Message-ID: I don't think writing shims for distutils is as easy as you make it sound. In fact, I'd venture to say that it's an intractable problem because of the difficulty of knowing the number of distutils hacks that are in the wild. 2017-09-29 22:15 GMT-05:00 Donald Stufft : > > > On Sep 29, 2017, at 5:16 PM, Steve Dower wrote: > > I'm happy enough with this approach, my only problem with it is that I > don't want to be maintaining two versions (the new one that we want people > to change to, and the old one so that people keep working with new versions > of Python without having to change to the new one). > > If we don't make a very clear statement that "distutils is no longer being > maintained, it probably won't build valid extensions on 3.[7/8/whatever], > and we mean it this time" then it doesn't count. I insist on at least > removing the distutils tests from the test suite, since as long as those > need to pass I can never stop maintaining the stdlib version. And that puts > this in the realm of python-dev, which means a PEP to formally deprecate > distutils (that I'm happy to co-author) and recommend an alternative, as > well as the alternative path needed to build the core extension modules on > non-Windows platforms. > > > As long as CPython itself depends on distutils I don?t think we can break > or remove distutils, but we can move make it a shim over something that is > better architecturally and easier to maintain. The tests should still pass > in this case because they?d essentially be testing that our shim is > functioning correctly. > > I also don?t think we can spin distutils out of the stdlib as long as > CPython depends on it. > > I see two paths forward: > > 1) We come up with something better *in the standard library*, implement > it, move CPython to using that in the build process, and then > deprecate/remove distutils from the stdlib and have it maintained outside. > > 2) We come up with something better outside of the standard library, > implement it, vendor it into distutils/_whatevercoollib, and turn the > relevant distutils APIs into shims over the new API. > > > Of the two, I think that (2) is going to be the least painful for users. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Sep 29 23:43:09 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 29 Sep 2017 23:43:09 -0400 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <7714677f-0443-2cef-b465-4e600be11133@python.org> <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> Message-ID: <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io> > On Sep 29, 2017, at 11:21 PM, xoviat wrote: > > I don't think writing shims for distutils is as easy as you make it sound. In fact, I'd venture to say that it's an intractable problem because of the difficulty of knowing the number of distutils hacks that are in the wild. > For the compiler code? It?s probably pretty tractable, which is I think the main thing that needs maintenance. From xoviat at gmail.com Sat Sep 30 15:52:31 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 30 Sep 2017 14:52:31 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io> References: <7714677f-0443-2cef-b465-4e600be11133@python.org> <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io> Message-ID: I don't know whether this will convince anyone otherwise, but at least from my perspective, a build-time dependency is different than a run-time dependency. Assuming that other parts of CPython don't have a run-time dependency on distutils, I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools). 2017-09-29 22:43 GMT-05:00 Donald Stufft : > > > > On Sep 29, 2017, at 11:21 PM, xoviat wrote: > > > > I don't think writing shims for distutils is as easy as you make it > sound. In fact, I'd venture to say that it's an intractable problem because > of the difficulty of knowing the number of distutils hacks that are in the > wild. > > > > > For the compiler code? It?s probably pretty tractable, which is I think > the main thing that needs maintenance. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Sep 30 20:59:58 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 30 Sep 2017 20:59:58 -0400 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <7714677f-0443-2cef-b465-4e600be11133@python.org> <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io> Message-ID: <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io> > On Sep 30, 2017, at 3:52 PM, xoviat wrote: > > I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools). As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled. That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don?t think that?s a great idea. I think Python should not depend on Python to build. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Sep 30 21:09:09 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 30 Sep 2017 20:09:09 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io> References: <7714677f-0443-2cef-b465-4e600be11133@python.org> <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io> <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io> Message-ID: I have personally not built Python myself (though I've built many an extension), but what I can say is that I got the idea from Larry Hastings. According to him (this if for the Gilectomy fork): "Second, as you hack on the Gilectomy you may break your "python" executable rather badly. This is of course expected. However, the python Makefile itself depends on having a working local python interpreter, so when you break that you often break your build too." 2017-09-30 19:59 GMT-05:00 Donald Stufft : > > > On Sep 30, 2017, at 3:52 PM, xoviat wrote: > > I don't think CPython needs to bundle all of its build-time dependencies. > That principle doesn't really apply to other Python programs nor most other > programs in general. AFAIK, CPython already has a build-time dependency on > another, external, Python, so it wouldn't be too much to require the > external Python to have setuptools installed with something like > pyproject.toml (other programming languages usually bootstrap themselves > with previous versions of the language along with some associated build > tools). > > > As far as I can tell, CPython does *not* have a build time dependency on > having Python available. I just spun up a bare alpine linux container and > compiled CPython `master` branch on it. As far as I can tell the only > Python that exists in this container is the one I just compiled. > > That means that in order for CPython to depend on distutils to build as > you indicate, it would also need to start depending on an existing version > of Python being available. I don?t think that?s a great idea. I think > Python should not depend on Python to build. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xoviat at gmail.com Sat Sep 30 21:11:50 2017 From: xoviat at gmail.com (xoviat) Date: Sat, 30 Sep 2017 20:11:50 -0500 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <7714677f-0443-2cef-b465-4e600be11133@python.org> <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io> <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io> Message-ID: It would be nice to know whether this information is correct, or whether I hold an invalid belief. 2017-09-30 20:09 GMT-05:00 xoviat : > I have personally not built Python myself (though I've built many an > extension), but what I can say is that I got the idea from Larry Hastings. > According to him (this if for the Gilectomy fork): > > "Second, as you hack on the Gilectomy you may break your "python" > executable rather badly. This is of course expected. However, the python > Makefile itself depends on having a working local python interpreter, so > when you break that you often break your build too." > > 2017-09-30 19:59 GMT-05:00 Donald Stufft : > >> >> >> On Sep 30, 2017, at 3:52 PM, xoviat wrote: >> >> I don't think CPython needs to bundle all of its build-time dependencies. >> That principle doesn't really apply to other Python programs nor most other >> programs in general. AFAIK, CPython already has a build-time dependency on >> another, external, Python, so it wouldn't be too much to require the >> external Python to have setuptools installed with something like >> pyproject.toml (other programming languages usually bootstrap themselves >> with previous versions of the language along with some associated build >> tools). >> >> >> As far as I can tell, CPython does *not* have a build time dependency on >> having Python available. I just spun up a bare alpine linux container and >> compiled CPython `master` branch on it. As far as I can tell the only >> Python that exists in this container is the one I just compiled. >> >> That means that in order for CPython to depend on distutils to build as >> you indicate, it would also need to start depending on an existing version >> of Python being available. I don?t think that?s a great idea. I think >> Python should not depend on Python to build. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Sep 30 21:14:43 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 30 Sep 2017 21:14:43 -0400 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <7714677f-0443-2cef-b465-4e600be11133@python.org> <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io> <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io> Message-ID: I think that the CPython builds a python executable, then uses that built executable to finish the installation. > On Sep 30, 2017, at 9:11 PM, xoviat wrote: > > It would be nice to know whether this information is correct, or whether I hold an invalid belief. > > 2017-09-30 20:09 GMT-05:00 xoviat >: > I have personally not built Python myself (though I've built many an extension), but what I can say is that I got the idea from Larry Hastings. According to him (this if for the Gilectomy fork): > > "Second, as you hack on the Gilectomy you may break your "python" executable rather badly. This is of course expected. However, the python Makefile itself depends on having a working local python interpreter, so when you break that you often break your build too." > > 2017-09-30 19:59 GMT-05:00 Donald Stufft >: > > >> On Sep 30, 2017, at 3:52 PM, xoviat > wrote: >> >> I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools). > > As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled. > > That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don?t think that?s a great idea. I think Python should not depend on Python to build. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sat Sep 30 22:32:35 2017 From: steve.dower at python.org (Steve Dower) Date: Sat, 30 Sep 2017 19:32:35 -0700 Subject: [Distutils] Extracting distutils into setuptools In-Reply-To: References: <7714677f-0443-2cef-b465-4e600be11133@python.org> <98F10EB7-60CF-4C06-9CB4-B68B32B8D4B1@stufft.io> <97EEEF28-6C43-4325-90A3-EC5F6395BFEF@stufft.io> <7EB1D0E7-941E-4261-A38F-4904251EA4FC@stufft.io> Message-ID: It must, or it couldn?t have a build time dependency on distutils :) Top-posted from my Windows phone From: Donald Stufft Sent: Saturday, September 30, 2017 18:15 To: xoviat Cc: Steve Dower; DistUtils mailing list Subject: Re: [Distutils] Extracting distutils into setuptools I think that the CPython builds a python executable, then uses that built executable to finish the installation. On Sep 30, 2017, at 9:11 PM, xoviat wrote: It would be nice to know whether this information is correct, or whether I hold an invalid belief. 2017-09-30 20:09 GMT-05:00 xoviat : I have personally not built Python myself (though I've built many an extension), but what I can say is that I got the idea from Larry Hastings. According to him (this if for the Gilectomy fork): "Second, as you hack on the Gilectomy you may break your "python" executable rather badly. This is of course expected. However, the python Makefile itself depends on having a working local python interpreter, so when you break that you often break your build too." 2017-09-30 19:59 GMT-05:00 Donald Stufft : On Sep 30, 2017, at 3:52 PM, xoviat wrote: I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools). As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled. That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don?t think that?s a great idea. I think Python should not depend on Python to build. -------------- next part -------------- An HTML attachment was scrubbed... URL: